fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load) #1833

Merged
HumanBean merged 4 commits from Jer9/Longshot:fix/astc-upload-chunk-submit into main 2026-11-20 06:17:09 +00:00
Contributor

Summary

Large GPU ASTC texture-array uploads previously packed nearly 2,000 compute dispatches into one queue submission, exceeding Windows' ~2 s WDDM timeout and crashing with VK_ERROR_DEVICE_LOST across many integrated GPUs and handheld APUs (#1040, #1730). Splitting uploads across bounded submissions (RX_VK_ASTC_CHUNK_MPX, default 16 million decoded texels per chunk, ~125 ms per submit on Radeon 860M) stops those Windows 2 s TDR resets — fixing VK_ERROR_DEVICE_LOST on Windows iGPUs and removing every chip- and CPU-specific astc_on_cpu workaround from device_policy.h so all GPUs share the unified GPU ASTC decode (BC3/BC7) path. On iGPUs previously forced onto the CPU fallback, returning to GPU BC3 decode cuts Kokiri Forest terrain upload time by 85.4% (15,239 ms -> 2,228 ms, 6.84x faster) and Zora's Domain load time by 54.5% (11.33 s -> 5.16 s, 2.20x faster), restores normal BC3 texture footprint (5,416 MB -> 1,474 MB decoded ASTC; 7,888 MB -> 3,702 MB peak VRAM on Radeon 860M), and lifts GPU-bound 1080p Native steady-state FPS in Zora's Domain by +10.2%.

We helped create that PCI-ID and CPU-graphics exception list ourselves — routing Radeon 860M (0x1114) to CPU decode in #1114, then adding Ryzen 7000 Raphael (0x164E) and Ryzen 9000 Granite Ridge (0x13C0) desktop CPU graphics plus a 6-8 GB iGPU texture-budget exception in #1559, while unlisted RDNA 3 APUs like 780M/890M (#1040, #1730) kept crashing. Hiding individual chips on CPU decode traded TDR resets for ~15 s area-entry freezes and ~5.4 GB of uncompressed RGBA8 textures that still break 16 GB RAM laptops. Bounded upload flushing fixes the shared root cause in upload_astc_gpu once across all hardware, inspired by yuzu's bounded ASTC compute submission splitting (https://github.com/yuzu-emu/yuzu/pull/10398, carried into Eden's MaxTexelsPerSubmission) and DXVK's bounded upload/compute submission splitting for Windows WDDM timeouts.

Reproducing on main: On a Windows AMD iGPU (e.g., Radeon 860M 0x1114), set RX_VK_ASTC_CPU=0 (or run tests/test_astc_upload_chunking.py on 30545573c9 / 068fb8ebe2) and enter Kokiri Forest. The two 1365x1365 x90 terrain arrays (~1,980 compute dispatches, ~223M decoded texels across mip cascades) are recorded into a single r->upload submission (flushes=0 submits=1), stalling the GPU past the ~2 s WDDM timeout (VK_ERROR_DEVICE_LOST), while default astc_on_cpu=true freezes ~15.2 s decoding on the CPU and bloats texture memory to ~7.2 GB.

Where reviewers should look first:

  • gpu/vk/vk_replay.cpp (replay_flush_upload) & gpu/vk/vk_internal.h (up_astc_texels): ends, submits (under dev.present_mu, forwarding any pending sparse_bind_sem), waits on r->fence, resets r->upload_pool, and re-begins r->upload in place while keeping begin_recording() and submit_recording() unchanged.
  • gpu/vk/vk_texture.cpp (upload_astc_gpu, astc_chunk_texels): tracks cumulative decoded texels (with a 64K-texel per-dispatch floor for mip tails), calls replay_flush_upload() before exceeding RX_VK_ASTC_CHUNK_MPX (16 MPx default, 0 disables), clears last_barrier_cmd so the next vk_full_barrier() starts the reset command buffer cleanly, and rebinds astc_pipe and descriptors on both BC and RGBA8 paths.
  • gpu/vk/device_policy.h, gpu/vk/switches.def, docs/dev/reference/switches.md: drops the Windows AMD iGPU PCI-ID rule (p.astc_on_cpu = false across all GPUs), demotes RX_VK_ASTC_CPU to a kill override (=1 forces CPU decode), and documents RX_VK_ASTC_CHUNK_MPX.
  • tests/test_astc_upload_chunking.py & tests/: game-free test failing on main (30545573c9: flushes=0 submits=1) and passing here (5e7dfbeed8: flushes=17 submits=18), plus updated source-cutting/policy checks (tests/test_astc_cpu_860m.py, tests/test_device_policy.py, tests/parallel_rb_refresh_test.cpp, tests/test_sparse_mirror.py).

Fixes #1040, See #1730, See #1114, See #1559

Testing

Tested:

  • Windows 11 (build 26200), AMD Ryzen AI 7 350, Radeon 860M (1002:1114, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (LLPC), standalone longshot_vk.dll (built in 79 s, 0 failed):
    • Interactive playtest (pre-rebase branch build, identical in executable gpu/vk/ code to rebased 5e7dfbeed8): 8+ minutes in Kokiri Forest (21,120+ presents, 30.0-48.2 fps, 1,228 pipelines, RX_VK_PRECOMPILE=0) with 0 VK_ERROR_DEVICE_LOST; textures stayed 3,011-3,495 MB inside the 4,096 MB iGPU budget.
    • Headless interleaved A/B runs (0de25c3e4a vs pre-rebase fix build, 13 runs: 3 pairs at Auto Minimum, 3 pairs at 1080p Native, 1 gprof run): 0 VK_ERROR_DEVICE_LOST, 0 write-back timeouts, 0 evicted textures, 0 quick pipelines, compiled 0 warm cache.
    • Game-free checks on rebased 5e7dfbeed8 (on main 068fb8ebe2): tests/test_astc_upload_chunking.py --cxx clang++ failed before the fix (30545573c9, flushes=0 submits=1, exit 1) and passed all 9 modes on 5e7dfbeed8 (flushes=17 submits=18 max_texels=15930497); tests/test_switch_table.py, tests/test_astc_cpu_860m.py, tests/test_device_policy.py, tests/test_parallel_rb_refresh.py --cxx clang++, tests/test_sparse_mirror.py, tests/test_doc_anchors.py, tests/test_utf8_paths.py, tests/test_log_levels.py, tests/test_thinlto_tls.py, tests/test_program_stamp_knobs.py, and scripts/check_third_party.py all passed.
    • Pre-#1802 full suite parity (before rebasing onto 068fb8ebe2, against 0de25c3e4a): tests/run_all.py hit 772 passed, 50 failed (49 FAIL + 1 TIMEOUT tests/test_vbytes.py), 54 skipped (windows, 16 at once); all 49 non-timeout failures also failed on 0de25c3e4a (0 regressions), and #1802/#1807 (0de25c3e4a..068fb8ebe2) changed 0 executable gpu/vk/ C++ lines.

Still to test:

  • Windows AMD iGPUs (windows+amd+igpu — Raphael 0x164E, Granite Ridge 0x13C0, 780M/880M/890M APUs): enter Kokiri Forest and Zora's Domain on default settings without RX_VK_ASTC_CPU, expecting GPU ASTC decode with zero VK_ERROR_DEVICE_LOST TDR resets (@Beanstalk test Windows AMD iGPUs (windows+amd+igpu) on Kokiri Forest and Zora's Domain)
  • Tier 1 discrete GPUs and Steam Deck (windows+nvidia, windows+amd+dgpu, linux+radv, steamdeck): sanity-run Kokiri Forest and Zora's Domain to confirm unchanged GPU ASTC upload stability (@Beanstalk test Windows NVIDIA, Windows AMD dGPU, Linux RADV, and Steam Deck on Kokiri Forest and Zora's Domain)
  • Linux Clang/GCC build and full test suite on rebased 5e7dfbeed8 after #1802 (by hand)

Measurements

Verdict: Fixes GPU WDDM TDR resets on Radeon 860M with default GPU ASTC decode, cuts Kokiri terrain upload time by 85.4% (15,239 ms -> 2,228 ms, 6.84x faster), cuts Zora's Domain load time by 54.5% (11.33 s -> 5.16 s, 2.20x faster), cuts peak VRAM by 53.1% (7,888 MB -> 3,702 MB) and decoded ASTC memory by 72.8% (5,416 MB -> 1,474 MB), improves Kokiri init fps by +57.5%, and keeps steady-state Auto fps within the spread.

Switches: RX_VK_ASTC_CHUNK_MPX (tune) ships at its default 16 million decoded texels per chunk (choosing ~125 ms compute+BC submissions well under the ~2 s WDDM TDR limit without excessive fence round-trips: 18 submits / 2.23 s total on Kokiri vs 0 disabled); RX_VK_ASTC_CPU (kill) ships off by default on all GPUs, returning Windows AMD iGPUs to GPU decode while preserving =1 as a runtime override.

Baseline: main at 0de25c3e4a vs the identical pre-rebase fix build (rebased onto 068fb8ebe2 as 5e7dfbeed8; 0de25c3e4a..068fb8ebe2 touched 0 executable gpu/vk/ C++ lines, and standalone longshot_vk.dll + tests/test_astc_upload_chunking.py re-verified on 5e7dfbeed8).

Setup: Windows 11 (26200), AMD Ryzen AI 7 350, Radeon 860M (1002:1114, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (LLPC); Quality preset Auto (Minimum, 768x432 internal -> 1536x864 window crop, FSR1 sharpness 30, AF 8x, vsync=0, RX_VK_PRECOMPILE=0, warm cache compiled 0); 3 interleaved pairs (low_0/2/4 vs low_1/3/5), median reported.

Metric Before: main @ 0de25c3e4a After: this PR @ 5e7dfbeed8 Δ
VK_ERROR_DEVICE_LOST / WB timeouts / evicted 0 / 0 / 0 (crashes if RX_VK_ASTC_CPU=0) 0 / 0 / 0 keeps working on GPU decode
Kokiri 2x 1365x1365 x90 upload (ms) 15,239 (14,910-15,997) 2,228 (2,199-2,238) -85.4% (6.84x faster)
Kokiri max slow submit (ms) 15,395.4 (15,070.4-16,161.2) 2,303.6 (2,271.7-2,313.5) -85.0%
Zora's Domain load time (s) 11.33 (10.71-11.80) 5.16 (5.06-5.27) -54.5% (2.20x faster)
Peak VRAM used (MB) 7,888 (7,696-7,952) 3,702 (3,638-3,766) -53.1% (-4,186 MB)
Peak textures (MB) 7,215 (7,024-7,351) 3,131 (3,118-3,256) -56.6% (-4,084 MB)
Peak decoded ASTC memory (MB) 5,416 (5,230-5,533) 1,474 (1,472-1,502) -72.8% (-3,942 MB)
Kokiri init steady wall_fps (GPU ms) 28.16 (24.15-28.89, 17.6 ms) 44.35 (41.05-46.15, 17.1 ms) +57.5%
Zora's Domain steady wall_fps (GPU ms) 44.11 (42.65-44.36, 18.7 ms) 42.06 (37.98-50.33, 18.1 ms) within the spread
Kokiri revisit steady wall_fps (GPU ms) 48.22 (46.39-48.75, 17.0 ms) 47.59 (46.85-48.32, 17.3 ms) within the spread

At 1080p Native (GPU-bound, 3 interleaved pairs high_0/2/4 vs high_1/3/5), the upload and memory wins replicate (15,143 -> 2,227 ms Kokiri upload, 13.60 -> 5.64 s Zora load, 7,743 -> 3,862 MB peak VRAM, 5,177 -> 1,462 MB peak decoded ASTC, and Kokiri init 18.56 -> 23.06 fps), while Kokiri revisit stays within the spread (22.85 [22.63-22.92] vs 23.09 [22.16-23.57]). Zora's Domain steady-state fps rises cleanly without overlap from 19.59 [19.29-19.81] (45.9 ms GPU) to 21.59 fps [20.57-21.75] (42.1 ms GPU, +10.2% / -3.8 ms), because storing decoded ASTC terrain in ~1.46 GB BC3 rather than ~5.18 GB RGBA8 eases shared UMA memory bandwidth pressure.

# Before (main @ 0de25c3e4a, low_2_main_b): CPU ASTC decode -> 15.2 s stall + 7.95 GB VRAM (7.24 GB textures, 5.53 GB decoded ASTC)
[hos Presentation Thread] vk: a frame's uploads took 15239 ms: on the CPU 83 (157.6 MB, 15239 ms); [1365x1365 x90 NVN 0x80 on the CPU 56.5 MB 7846 ms] [1365x1365 x90 NVN 0x8e on the CPU 56.5 MB 7381 ms] [260x260 x6 NVN 0x4f on the CPU 0.8 MB 9 ms]
[hos Presentation Thread] vk: VRAM: 7952 MB used of 12212 MB allowed (309 MB of it free in the blocks); textures 7237 MB (budget 8192 MB), 2198 textures
[hos Presentation Thread] vk: textures: decoded ASTC 5533 MB, rendered 645 MB (108 drawn in one frame, kept: 38368 KB), other 495 MB; used in the last 8 presents 2963 MB
# After (pre-rebase build identical in gpu/vk/ code to 5e7dfbeed8, low_3_fix_b + rebased_gprof): chunked GPU ASTC decode -> 2.23 s across 18 submits + 3.64 GB VRAM (3.13 GB textures, 1.47 GB decoded ASTC)
[hos Presentation Thread] vk: slow submit: 2303.6 ms: draw 2288.8 descriptors 2278.1 texture lookups 2276.9 uploads 2227.4 programs 7.7 commands 2.6 queue submit 8.4
[hos Presentation Thread] vk: a frame's uploads took 2228 ms: ASTC on the GPU 2 (112.9 MB, 2223 ms); on the CPU 79 (44.0 MB, 4 ms); [1365x1365 x90 NVN 0x8e ASTC on the GPU 56.5 MB 1121 ms] [1365x1365 x90 NVN 0x80 ASTC on the GPU 56.5 MB 1102 ms]
[hos Presentation Thread] vk: VRAM: 3638 MB used of 12212 MB allowed (193 MB of it free in the blocks); textures 3131 MB (budget 4096 MB), 2145 textures
[hos Presentation Thread] vk: textures: decoded ASTC 1474 MB, rendered 637 MB (108 drawn in one frame, kept: 35755 KB), other 497 MB; used in the last 8 presents 1308 MB
[hos Presentation Thread] vk:   upload command buffers: 4.87 ms in 0.3 a frame (0.03 ms in spans, 4.83 ms between them: texture uploads and decodes, the page table, glue)
# Game-free check (tests/test_astc_upload_chunking.py --cxx clang++ on 5e7dfbeed8):
mode=terrain_bc3_default flushes=17 submits=18 total_dispatches=1980 max_dispatches=488 max_texels=15930497 errors=0

Risks

  • replay_flush_upload() ends, submits, and fence-waits r->upload under dev.present_mu mid-recording: safe only on the single replay upload path (upload_astc_gpu() explicitly skips chunk splitting when called outside the main upload recording, tl_seg_index >= 0).
  • Weak 2-CU Windows AMD desktop iGPUs (Raphael 0x164E, Granite Ridge 0x13C0) and 780M/890M APUs have not been re-tested directly here yet; at ~1/4 the CUs of the 860M, 16M-texel chunks could take longer per submission. Players can lower RX_VK_ASTC_CHUNK_MPX=4 or restore CPU decoding immediately with RX_VK_ASTC_CPU=1, or a clean revert restores prior behavior.
  • Player update impact: none — saves, longshot_options.ini, and shader/program caches are unchanged; Windows AMD iGPUs previously forced onto CPU decode regain normal hardware texture budgets (4,096 MB on 860M) and ~5 GB lower RAM/VRAM pressure.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • A switch added or changed: it ships at the default the measurements chose, with its row in gpu/vk/switches.def.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths.
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary Large GPU ASTC texture-array uploads previously packed nearly 2,000 compute dispatches into one queue submission, exceeding Windows' ~2 s WDDM timeout and crashing with `VK_ERROR_DEVICE_LOST` across many integrated GPUs and handheld APUs (#1040, #1730). Splitting uploads across bounded submissions (`RX_VK_ASTC_CHUNK_MPX`, default `16` million decoded texels per chunk, ~125 ms per submit on Radeon 860M) stops those Windows 2 s TDR resets — **fixing `VK_ERROR_DEVICE_LOST` on Windows iGPUs and removing every chip- and CPU-specific `astc_on_cpu` workaround from `device_policy.h` so all GPUs share the unified GPU ASTC decode (`BC3`/`BC7`) path**. On iGPUs previously forced onto the CPU fallback, returning to GPU `BC3` decode cuts Kokiri Forest terrain upload time by **85.4% (`15,239 ms -> 2,228 ms`, `6.84x` faster)** and Zora's Domain load time by **54.5% (`11.33 s -> 5.16 s`, `2.20x` faster)**, restores normal `BC3` texture footprint (`5,416 MB -> 1,474 MB` decoded ASTC; `7,888 MB -> 3,702 MB` peak VRAM on Radeon 860M), and lifts GPU-bound `1080p Native` steady-state FPS in Zora's Domain by **+10.2%**. We helped create that PCI-ID and CPU-graphics exception list ourselves — routing Radeon 860M (`0x1114`) to CPU decode in #1114, then adding Ryzen 7000 Raphael (`0x164E`) and Ryzen 9000 Granite Ridge (`0x13C0`) desktop CPU graphics plus a 6-8 GB iGPU texture-budget exception in #1559, while unlisted RDNA 3 APUs like 780M/890M (#1040, #1730) kept crashing. Hiding individual chips on CPU decode traded TDR resets for ~15 s area-entry freezes and ~5.4 GB of uncompressed RGBA8 textures that still break 16 GB RAM laptops. Bounded upload flushing fixes the shared root cause in `upload_astc_gpu` once across all hardware, inspired by yuzu's bounded ASTC compute submission splitting (https://github.com/yuzu-emu/yuzu/pull/10398, carried into Eden's `MaxTexelsPerSubmission`) and DXVK's bounded upload/compute submission splitting for Windows WDDM timeouts. **Reproducing on `main`:** On a Windows AMD iGPU (e.g., Radeon 860M `0x1114`), set `RX_VK_ASTC_CPU=0` (or run `tests/test_astc_upload_chunking.py` on `30545573c9` / `068fb8ebe2`) and enter Kokiri Forest. The two `1365x1365 x90` terrain arrays (~1,980 compute dispatches, ~223M decoded texels across mip cascades) are recorded into a single `r->upload` submission (`flushes=0 submits=1`), stalling the GPU past the ~2 s WDDM timeout (`VK_ERROR_DEVICE_LOST`), while default `astc_on_cpu=true` freezes ~15.2 s decoding on the CPU and bloats texture memory to ~7.2 GB. **Where reviewers should look first:** - `gpu/vk/vk_replay.cpp` (`replay_flush_upload`) & `gpu/vk/vk_internal.h` (`up_astc_texels`): ends, submits (under `dev.present_mu`, forwarding any pending `sparse_bind_sem`), waits on `r->fence`, resets `r->upload_pool`, and re-begins `r->upload` in place while keeping `begin_recording()` and `submit_recording()` unchanged. - `gpu/vk/vk_texture.cpp` (`upload_astc_gpu`, `astc_chunk_texels`): tracks cumulative decoded texels (with a `64K`-texel per-dispatch floor for mip tails), calls `replay_flush_upload()` before exceeding `RX_VK_ASTC_CHUNK_MPX` (`16` MPx default, `0` disables), clears `last_barrier_cmd` so the next `vk_full_barrier()` starts the reset command buffer cleanly, and rebinds `astc_pipe` and descriptors on both BC and RGBA8 paths. - `gpu/vk/device_policy.h`, `gpu/vk/switches.def`, `docs/dev/reference/switches.md`: drops the Windows AMD iGPU PCI-ID rule (`p.astc_on_cpu = false` across all GPUs), demotes `RX_VK_ASTC_CPU` to a `kill` override (`=1` forces CPU decode), and documents `RX_VK_ASTC_CHUNK_MPX`. - `tests/test_astc_upload_chunking.py` & `tests/`: game-free test failing on `main` (`30545573c9`: `flushes=0 submits=1`) and passing here (`5e7dfbeed8`: `flushes=17 submits=18`), plus updated source-cutting/policy checks (`tests/test_astc_cpu_860m.py`, `tests/test_device_policy.py`, `tests/parallel_rb_refresh_test.cpp`, `tests/test_sparse_mirror.py`). Fixes #1040, See #1730, See #1114, See #1559 ## Testing **Tested:** - Windows 11 (build 26200), AMD Ryzen AI 7 350, Radeon 860M (`1002:1114`, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (`LLPC`), standalone `longshot_vk.dll` (built in 79 s, 0 failed): - **Interactive playtest** (pre-rebase branch build, identical in executable `gpu/vk/` code to rebased `5e7dfbeed8`): 8+ minutes in Kokiri Forest (`21,120+` presents, `30.0-48.2 fps`, `1,228` pipelines, `RX_VK_PRECOMPILE=0`) with `0` `VK_ERROR_DEVICE_LOST`; textures stayed `3,011-3,495 MB` inside the `4,096 MB` iGPU budget. - **Headless interleaved A/B runs** (`0de25c3e4a` vs pre-rebase fix build, 13 runs: 3 pairs at `Auto` Minimum, 3 pairs at `1080p Native`, 1 `gprof` run): `0` `VK_ERROR_DEVICE_LOST`, `0` write-back timeouts, `0` evicted textures, `0` quick pipelines, `compiled 0` warm cache. - **Game-free checks on rebased `5e7dfbeed8`** (on `main` `068fb8ebe2`): `tests/test_astc_upload_chunking.py --cxx clang++` failed before the fix (`30545573c9`, `flushes=0 submits=1`, exit `1`) and passed all 9 modes on `5e7dfbeed8` (`flushes=17 submits=18 max_texels=15930497`); `tests/test_switch_table.py`, `tests/test_astc_cpu_860m.py`, `tests/test_device_policy.py`, `tests/test_parallel_rb_refresh.py --cxx clang++`, `tests/test_sparse_mirror.py`, `tests/test_doc_anchors.py`, `tests/test_utf8_paths.py`, `tests/test_log_levels.py`, `tests/test_thinlto_tls.py`, `tests/test_program_stamp_knobs.py`, and `scripts/check_third_party.py` all passed. - **Pre-`#1802` full suite parity** (before rebasing onto `068fb8ebe2`, against `0de25c3e4a`): `tests/run_all.py` hit `772 passed, 50 failed (49 FAIL + 1 TIMEOUT tests/test_vbytes.py), 54 skipped (windows, 16 at once)`; all 49 non-timeout failures also failed on `0de25c3e4a` (`0` regressions), and `#1802`/`#1807` (`0de25c3e4a..068fb8ebe2`) changed `0` executable `gpu/vk/` C++ lines. **Still to test:** - Windows AMD iGPUs (`windows+amd+igpu` — Raphael `0x164E`, Granite Ridge `0x13C0`, 780M/880M/890M APUs): enter Kokiri Forest and Zora's Domain on default settings without `RX_VK_ASTC_CPU`, expecting GPU ASTC decode with zero `VK_ERROR_DEVICE_LOST` TDR resets (@Beanstalk test Windows AMD iGPUs (`windows+amd+igpu`) on Kokiri Forest and Zora's Domain) - Tier 1 discrete GPUs and Steam Deck (`windows+nvidia`, `windows+amd+dgpu`, `linux+radv`, `steamdeck`): sanity-run Kokiri Forest and Zora's Domain to confirm unchanged GPU ASTC upload stability (@Beanstalk test Windows NVIDIA, Windows AMD dGPU, Linux RADV, and Steam Deck on Kokiri Forest and Zora's Domain) - Linux Clang/GCC build and full test suite on rebased `5e7dfbeed8` after `#1802` (by hand) ## Measurements **Verdict:** Fixes GPU WDDM TDR resets on Radeon 860M with default GPU ASTC decode, cuts Kokiri terrain upload time by 85.4% (`15,239 ms -> 2,228 ms`, `6.84x` faster), cuts Zora's Domain load time by 54.5% (`11.33 s -> 5.16 s`, `2.20x` faster), cuts peak VRAM by 53.1% (`7,888 MB -> 3,702 MB`) and decoded ASTC memory by 72.8% (`5,416 MB -> 1,474 MB`), improves Kokiri init fps by `+57.5%`, and keeps steady-state `Auto` fps within the spread. Switches: `RX_VK_ASTC_CHUNK_MPX` (`tune`) ships at its default `16` million decoded texels per chunk (choosing ~125 ms compute+BC submissions well under the ~2 s WDDM TDR limit without excessive fence round-trips: 18 submits / `2.23 s` total on Kokiri vs `0` disabled); `RX_VK_ASTC_CPU` (`kill`) ships `off` by default on all GPUs, returning Windows AMD iGPUs to GPU decode while preserving `=1` as a runtime override. Baseline: `main` at `0de25c3e4a` vs the identical pre-rebase fix build (rebased onto `068fb8ebe2` as `5e7dfbeed8`; `0de25c3e4a..068fb8ebe2` touched `0` executable `gpu/vk/` C++ lines, and standalone `longshot_vk.dll` + `tests/test_astc_upload_chunking.py` re-verified on `5e7dfbeed8`). Setup: Windows 11 (26200), AMD Ryzen AI 7 350, Radeon 860M (`1002:1114`, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (`LLPC`); Quality preset `Auto` (`Minimum`, `768x432` internal -> `1536x864` window crop, FSR1 sharpness 30, AF 8x, vsync=0, `RX_VK_PRECOMPILE=0`, warm cache `compiled 0`); 3 interleaved pairs (`low_0/2/4` vs `low_1/3/5`), median reported. | Metric | Before: main @ 0de25c3e4a | After: this PR @ 5e7dfbeed8 | Δ | |---|---|---|---| | `VK_ERROR_DEVICE_LOST` / WB timeouts / evicted | `0 / 0 / 0` (crashes if `RX_VK_ASTC_CPU=0`) | `0 / 0 / 0` | keeps working on GPU decode | | Kokiri 2x `1365x1365 x90` upload (`ms`) | `15,239` (`14,910-15,997`) | `2,228` (`2,199-2,238`) | `-85.4%` (`6.84x` faster) | | Kokiri max `slow submit` (`ms`) | `15,395.4` (`15,070.4-16,161.2`) | `2,303.6` (`2,271.7-2,313.5`) | `-85.0%` | | Zora's Domain load time (`s`) | `11.33` (`10.71-11.80`) | `5.16` (`5.06-5.27`) | `-54.5%` (`2.20x` faster) | | Peak VRAM used (`MB`) | `7,888` (`7,696-7,952`) | `3,702` (`3,638-3,766`) | `-53.1%` (`-4,186 MB`) | | Peak textures (`MB`) | `7,215` (`7,024-7,351`) | `3,131` (`3,118-3,256`) | `-56.6%` (`-4,084 MB`) | | Peak decoded ASTC memory (`MB`) | `5,416` (`5,230-5,533`) | `1,474` (`1,472-1,502`) | `-72.8%` (`-3,942 MB`) | | Kokiri init steady `wall_fps` (`GPU ms`) | `28.16` (`24.15-28.89`, `17.6 ms`) | `44.35` (`41.05-46.15`, `17.1 ms`) | `+57.5%` | | Zora's Domain steady `wall_fps` (`GPU ms`) | `44.11` (`42.65-44.36`, `18.7 ms`) | `42.06` (`37.98-50.33`, `18.1 ms`) | within the spread | | Kokiri revisit steady `wall_fps` (`GPU ms`) | `48.22` (`46.39-48.75`, `17.0 ms`) | `47.59` (`46.85-48.32`, `17.3 ms`) | within the spread | At `1080p Native` (GPU-bound, 3 interleaved pairs `high_0/2/4` vs `high_1/3/5`), the upload and memory wins replicate (`15,143 -> 2,227 ms` Kokiri upload, `13.60 -> 5.64 s` Zora load, `7,743 -> 3,862 MB` peak VRAM, `5,177 -> 1,462 MB` peak decoded ASTC, and Kokiri init `18.56 -> 23.06 fps`), while Kokiri revisit stays within the spread (`22.85` `[22.63-22.92]` vs `23.09` `[22.16-23.57]`). Zora's Domain steady-state fps rises cleanly without overlap from `19.59` `[19.29-19.81]` (`45.9 ms` GPU) to `21.59 fps` `[20.57-21.75]` (`42.1 ms` GPU, `+10.2%` / `-3.8 ms`), because storing decoded ASTC terrain in ~1.46 GB BC3 rather than ~5.18 GB RGBA8 eases shared UMA memory bandwidth pressure. ```text # Before (main @ 0de25c3e4a, low_2_main_b): CPU ASTC decode -> 15.2 s stall + 7.95 GB VRAM (7.24 GB textures, 5.53 GB decoded ASTC) [hos Presentation Thread] vk: a frame's uploads took 15239 ms: on the CPU 83 (157.6 MB, 15239 ms); [1365x1365 x90 NVN 0x80 on the CPU 56.5 MB 7846 ms] [1365x1365 x90 NVN 0x8e on the CPU 56.5 MB 7381 ms] [260x260 x6 NVN 0x4f on the CPU 0.8 MB 9 ms] [hos Presentation Thread] vk: VRAM: 7952 MB used of 12212 MB allowed (309 MB of it free in the blocks); textures 7237 MB (budget 8192 MB), 2198 textures [hos Presentation Thread] vk: textures: decoded ASTC 5533 MB, rendered 645 MB (108 drawn in one frame, kept: 38368 KB), other 495 MB; used in the last 8 presents 2963 MB # After (pre-rebase build identical in gpu/vk/ code to 5e7dfbeed8, low_3_fix_b + rebased_gprof): chunked GPU ASTC decode -> 2.23 s across 18 submits + 3.64 GB VRAM (3.13 GB textures, 1.47 GB decoded ASTC) [hos Presentation Thread] vk: slow submit: 2303.6 ms: draw 2288.8 descriptors 2278.1 texture lookups 2276.9 uploads 2227.4 programs 7.7 commands 2.6 queue submit 8.4 [hos Presentation Thread] vk: a frame's uploads took 2228 ms: ASTC on the GPU 2 (112.9 MB, 2223 ms); on the CPU 79 (44.0 MB, 4 ms); [1365x1365 x90 NVN 0x8e ASTC on the GPU 56.5 MB 1121 ms] [1365x1365 x90 NVN 0x80 ASTC on the GPU 56.5 MB 1102 ms] [hos Presentation Thread] vk: VRAM: 3638 MB used of 12212 MB allowed (193 MB of it free in the blocks); textures 3131 MB (budget 4096 MB), 2145 textures [hos Presentation Thread] vk: textures: decoded ASTC 1474 MB, rendered 637 MB (108 drawn in one frame, kept: 35755 KB), other 497 MB; used in the last 8 presents 1308 MB [hos Presentation Thread] vk: upload command buffers: 4.87 ms in 0.3 a frame (0.03 ms in spans, 4.83 ms between them: texture uploads and decodes, the page table, glue) # Game-free check (tests/test_astc_upload_chunking.py --cxx clang++ on 5e7dfbeed8): mode=terrain_bc3_default flushes=17 submits=18 total_dispatches=1980 max_dispatches=488 max_texels=15930497 errors=0 ``` ## Risks - `replay_flush_upload()` ends, submits, and fence-waits `r->upload` under `dev.present_mu` mid-recording: safe only on the single replay upload path (`upload_astc_gpu()` explicitly skips chunk splitting when called outside the main upload recording, `tl_seg_index >= 0`). - Weak 2-CU Windows AMD desktop iGPUs (Raphael `0x164E`, Granite Ridge `0x13C0`) and 780M/890M APUs have not been re-tested directly here yet; at ~1/4 the CUs of the 860M, 16M-texel chunks could take longer per submission. Players can lower `RX_VK_ASTC_CHUNK_MPX=4` or restore CPU decoding immediately with `RX_VK_ASTC_CPU=1`, or a clean revert restores prior behavior. - **Player update impact:** none — saves, `longshot_options.ini`, and shader/program caches are unchanged; Windows AMD iGPUs previously forced onto CPU decode regain normal hardware texture budgets (`4,096 MB` on 860M) and ~5 GB lower RAM/VRAM pressure. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [x] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] A switch added or changed: it ships at the default the measurements chose, with its row in `gpu/vk/switches.def`. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
Cut upload_astc_gpu(), bc_encode_into(), bc_encode_resized(), level_geo(), and the upload command-buffer hooks from gpu/vk/vk_texture.cpp into a game-free Vulkan spy harness.

Checks that a 1365 x 1365 x 90 11-mip BC3 or RGBA8 ASTC array upload flushes across bounded submissions via replay_flush_upload() (~16M texels per submit by default), encodes every (level, layer) once in order, clears last_barrier_cmd and re-emits the initial full barrier and rebinds astc_pipe after each in-place command-buffer reset, accumulates up_astc_texels across uploads within a recording and resets it in texture_begin_recording(), honours RX_VK_ASTC_CHUNK_MPX=0 and negative/custom values, and leaves parallel replay untouched.

Assisted-by: Antigravity (Gemini 4 Carbon)
fix(vk): split large GPU ASTC uploads across bounded submissions
Some checks reported errors
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 5s
beanstalk/approval approved by HumanBean (reviewed)
beanstalk/windows+amd+igpu 0/1 pass, 2 running, 0 queued
beanstalk/windows+nvidia 0/1 pass, 2 error
beanstalk/windows+amd+dgpu 0/1 pass, 1 running, 1 queued
beanstalk/linux+radv 0/1 pass, 2 running, 0 queued
beanstalk/steamdeck 0/1 pass, 2 running, 0 queued
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
5e7dfbeed8
Large ASTC texture-array uploads (such as 1365x1365x90 11-mip BC3 terrain arrays) previously packed nearly 2,000 ASTC compute decode + BC encode dispatches and barriers into a single upload command buffer, tripping the ~2 s Windows WDDM TDR timeout on integrated Radeon GPUs (860M/780M/Raphael) when GPU ASTC decode was enabled.

- Add replay_flush_upload() in gpu/vk/vk_replay.cpp to end, submit, and fence-wait the recording's active upload command buffer (r->upload with r->fence) under dev.present_mu (carrying any pending sparse-bind semaphore), then reset r->upload_pool and re-begin r->upload in place while leaving begin_recording() and submit_recording() unchanged.
- Track cumulative decoded texels (up_astc_texels, with a 64K-texel per-run floor so small-mip cascades also bound dispatch count) across upload_astc_gpu() calls within a recording, flushing via replay_flush_upload() before exceeding RX_VK_ASTC_CHUNK_MPX (default 16M texels; 0 disables) on both BC and RGBA8 paths, clearing last_barrier_cmd to emit the initial full barrier, and rebinding astc_pipe on the reset command buffer.
- Default device_policy.h astc_on_cpu to false across Windows AMD iGPUs now that GPU ASTC uploads stay within WDDM TDR budgets, while keeping RX_VK_ASTC_CPU=1 as an explicit override, mark RX_VK_ASTC_CPU as a kill switch in gpu/vk/switches.def, and update device-policy and replay-cutting tests and docs/dev/reference/switches.md.

Assisted-by: Antigravity (Gemini 4 Carbon)
Author
Contributor

Tested in game: Windows 11 (26200), AMD Ryzen AI 7 350, Radeon 860M (1002:1114, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (LLPC): 8+ min interactive playtest in Kokiri Forest (21,120+ presents, 30.0-48.2 fps, 0 VK_ERROR_DEVICE_LOST, textures 3,011-3,495 MB within 4,096 MB budget) plus 13 warm-cache headless interleaved A/B runs across Kokiri Forest and Zora's Domain (Kokiri terrain upload 15,239 ms -> 2,228 ms, Zora load 11.33 s -> 5.16 s, 0 VK_ERROR_DEVICE_LOST, 0 write-back timeouts).

Tested in game: Windows 11 (26200), AMD Ryzen AI 7 350, Radeon 860M (1002:1114, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (LLPC): 8+ min interactive playtest in Kokiri Forest (21,120+ presents, 30.0-48.2 fps, 0 VK_ERROR_DEVICE_LOST, textures 3,011-3,495 MB within 4,096 MB budget) plus 13 warm-cache headless interleaved A/B runs across Kokiri Forest and Zora's Domain (Kokiri terrain upload 15,239 ms -> 2,228 ms, Zora load 11.33 s -> 5.16 s, 0 VK_ERROR_DEVICE_LOST, 0 write-back timeouts).
Member

@Jer9 only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@Jer9 only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Member

Possibly related, for context:

  • issue #1234 Game: Radeon 880M has device-lost crashes and severe stutter — Open Radeon 880M device-loss report; possibly addressed by the shared ASTC upload fix.
Possibly related, for context: - issue #1234 Game: Radeon 880M has device-lost crashes and severe stutter — Open Radeon 880M device-loss report; possibly addressed by the shared ASTC upload fix.
Author
Contributor

@Beanstalk test Windows AMD iGPUs (windows+amd+igpu), Windows NVIDIA (windows+nvidia), Windows AMD dGPU (windows+amd+dgpu), Linux RADV (linux+radv), and Steam Deck (steamdeck) on Kokiri Forest and Zora's Domain — verify GPU ASTC terrain upload completes with zero VK_ERROR_DEVICE_LOST resets and fast load times

@Beanstalk test Windows AMD iGPUs (windows+amd+igpu), Windows NVIDIA (windows+nvidia), Windows AMD dGPU (windows+amd+dgpu), Linux RADV (linux+radv), and Steam Deck (steamdeck) on Kokiri Forest and Zora's Domain — verify GPU ASTC terrain upload completes with zero VK_ERROR_DEVICE_LOST resets and fast load times
Member

@Jer9 only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@Jer9 only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Contributor

#1843 changes the same loop in upload_astc_gpu. At levels of 1024 or less it runs eight layers at a time, so each of your chunks should hold about an eighth of the runs and barriers at those levels and finish sooner, for the same texels decoded. That part is untested: I haven't run the two together or on an iGPU. The two fit together. Whichever merges second needs a small rebase, and I'm happy to do it on mine.

#1843 changes the same loop in `upload_astc_gpu`. At levels of 1024 or less it runs eight layers at a time, so each of your chunks should hold about an eighth of the runs and barriers at those levels and finish sooner, for the same texels decoded. That part is untested: I haven't run the two together or on an iGPU. The two fit together. Whichever merges second needs a small rebase, and I'm happy to do it on mine.
Member
Beanstalk test farm · an earlier run: commit 5e7dfbe (the newer results are in a comment below)

Commit 5e7dfbe · tier reviewed · approved by HumanBean (👍 on Jer9's request) · dashboard

What will be tested · AI plan (gpt-6.1-terra)

The plan compares default ASTC uploads on main and the pull request in Kokiri Forest and Zora's Domain across the five requested Windows and Linux hardware groups. Each tester runs one pass per build, taking about 11 minutes, with load times, frame times, VRAM, logs, and terrain screenshots recorded. A crash, device-loss or upload-failure log, CPU ASTC fallback, significant load-time or performance regression, or excessive screenshot difference fails the pull request. The available tag counts do not establish whether every requested tag combination is online; any unmatched target will wait for a tester. No tester online now matches windows+amd+igpu: that target waits for one.

  • Where: windows+amd+igpu (1-2 reports), windows+nvidia (1-2 reports), windows+amd+dgpu (1-2 reports), linux+radv (1-2 reports), steamdeck (1-2 reports).
  • terrain-uploads: from the repository's save kokiri-start, then teleport to Kokiri Forest, take the picture kokiri-terrain to compare with main, measure kokiri for 20 s (after 5 s warm-up), teleport to Zora's Domain, take the picture zora-terrain to compare with main, measure zora for 20 s (after 5 s warm-up); 1 run on each side.
  • Measured: crashes, load times, frame times, stutters and hitches, VRAM use, the game's log.
  • The PR fails if: it crashes; or a picture differs from main by more than 15%; or peak VRAM is more than 15% above main; or longest load is more than 25% above main; or frame time (p99) is more than 20% above main; or forbidden log lines is above 0.
  • Time: about 13 min of game runs per tester (2 game starts), plus the two builds.

Started at 05:15 UTC.

Estimate: done in about 45 min, from the queue and the testers online now.

Target Reports Crash Frame p99 (ms) vs main VRAM peak (MB) vs main Result
windows+amd+igpu 0/1 — — — running
windows+nvidia · 1: windows+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0; 2: windows+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 4080, 621.1.0 2/1 1 2 none — — error
windows+amd+dgpu 0/1 — — — running
linux+radv 0/1 — — — running
steamdeck 0/1 — — — running

Measurements · windows+nvidia

Verdict: could not be judged on windows+nvidia (2/1 report).

Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0 AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM; 2: windows+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 4080, 621.1.0 AMD Ryzen 7 9800X3D 8-Core Processor, 32 GB RAM
Scene: terrain-uploads, from the repository's save kokiri-start; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); terrain-uploads: 1 game start a side, kokiri 20 s after 5 s warm-up, zora 20 s after 5 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (@Beanstalk on this PR runs it again)

Nothing was measured: no metric came back.

Flags and settings

None: the plan varied no setting, switch or build option.

Markdown for the PR description
## Measurements

**Verdict:** could not be judged on windows+nvidia (2/1 report).

Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0` `AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM`; 2: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 4080, 621.1.0` `AMD Ryzen 7 9800X3D 8-Core Processor, 32 GB RAM`  
Scene: terrain-uploads, from the repository's save kokiri-start; each cell's settings and switches as the table names them  
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); terrain-uploads: 1 game start a side, kokiri 20 s after 5 s warm-up, zora 20 s after 5 s warm-up; each number the median of the runs, then of 2 reports  
How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again)

Nothing was measured: no metric came back.

### Flags and settings

None: the plan varied no setting, switch or build option.
<!-- beanstalk:previous --> <details><summary>Beanstalk test farm · an earlier run: commit <code>5e7dfbe</code> (the newer results are in a comment below)</summary> Commit `5e7dfbe` · tier **reviewed** · approved by HumanBean (👍 on Jer9's request) · [dashboard](https://beanstalk.magicbean.dev/prs/1833) **What will be tested** · AI plan (`gpt-6.1-terra`) The plan compares default ASTC uploads on main and the pull request in Kokiri Forest and Zora's Domain across the five requested Windows and Linux hardware groups. Each tester runs one pass per build, taking about 11 minutes, with load times, frame times, VRAM, logs, and terrain screenshots recorded. A crash, device-loss or upload-failure log, CPU ASTC fallback, significant load-time or performance regression, or excessive screenshot difference fails the pull request. The available tag counts do not establish whether every requested tag combination is online; any unmatched target will wait for a tester. No tester online now matches windows+amd+igpu: that target waits for one. - **Where:** windows+amd+igpu (1-2 reports), windows+nvidia (1-2 reports), windows+amd+dgpu (1-2 reports), linux+radv (1-2 reports), steamdeck (1-2 reports). - **terrain-uploads:** from the repository's save `kokiri-start`, then teleport to Kokiri Forest, take the picture kokiri-terrain to compare with main, measure kokiri for 20 s (after 5 s warm-up), teleport to Zora's Domain, take the picture zora-terrain to compare with main, measure zora for 20 s (after 5 s warm-up); 1 run on each side. - **Measured:** crashes, load times, frame times, stutters and hitches, VRAM use, the game's log. - **The PR fails if:** it crashes; or a picture differs from main by more than 15%; or peak VRAM is more than 15% above main; or longest load is more than 25% above main; or frame time (p99) is more than 20% above main; or forbidden log lines is above 0. - **Time:** about 13 min of game runs per tester (2 game starts), plus the two builds. Started at 05:15 UTC. **Estimate:** done in about 45 min, from the queue and the testers online now. | Target | Reports | Crash | Frame p99 (ms) vs main | VRAM peak (MB) vs main | Result | | --- | --- | --- | --- | --- | --- | | windows+amd+igpu | 0/1 | — | — | — | running | | windows+nvidia · 1: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0`; 2: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 4080, 621.1.0` | 2/1 [1](https://beanstalk.magicbean.dev/reports/rp_8d7ef87f760b6461) [2](https://beanstalk.magicbean.dev/reports/rp_de59a24cb964151d) | none | — | — | error | | windows+amd+dgpu | 0/1 | — | — | — | running | | linux+radv | 0/1 | — | — | — | running | | steamdeck | 0/1 | — | — | — | running | #### Measurements · windows+nvidia **Verdict:** could not be judged on windows+nvidia (2/1 report). Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0` `AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM`; 2: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 4080, 621.1.0` `AMD Ryzen 7 9800X3D 8-Core Processor, 32 GB RAM` Scene: terrain-uploads, from the repository's save kokiri-start; each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); terrain-uploads: 1 game start a side, kokiri 20 s after 5 s warm-up, zora 20 s after 5 s warm-up; each number the median of the runs, then of 2 reports How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) Nothing was measured: no metric came back. ### Flags and settings None: the plan varied no setting, switch or build option. <details><summary>Markdown for the PR description</summary> ```markdown ## Measurements **Verdict:** could not be judged on windows+nvidia (2/1 report). Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0` `AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM`; 2: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 4080, 621.1.0` `AMD Ryzen 7 9800X3D 8-Core Processor, 32 GB RAM` Scene: terrain-uploads, from the repository's save kokiri-start; each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); terrain-uploads: 1 game start a side, kokiri 20 s after 5 s warm-up, zora 20 s after 5 s warm-up; each number the median of the runs, then of 2 reports How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) Nothing was measured: no metric came back. ### Flags and settings None: the plan varied no setting, switch or build option. ``` </details> </details>
fix(vk): RX_VK_ASTC_CPU's text says =1, and a failed upload chunk end is logged
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 11s
beanstalk/approval not approved yet
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Website build and checks (pull_request) Has been cancelled
85de75720c
The switch row and the CPU-decode log line no longer say 0 forces the GPU
(the GPU decoder is the default everywhere now); vkEndCommandBuffer failing
in replay_flush_upload is logged like a failed submit; a comment says why a
level over the chunk bound takes one layer a dispatch.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Beanstalk test farm

Commit afecbae · tier untrusted · approved by nobody yet · dashboard

A member of BigBeans can review this commit and comment @Beanstalk to test it.

<!-- beanstalk:sticky --> ### Beanstalk test farm Commit `afecbae` · tier **untrusted** · approved by nobody yet · [dashboard](https://beanstalk.magicbean.dev/prs/1833) A member of `BigBeans` can review this commit and comment `@Beanstalk` to test it.
HumanBean approved these changes 2026-11-20 05:54:18 +00:00
Dismissed
HumanBean left a comment

Recommendation: Approve. It fixes the Windows iGPU VK_ERROR_DEVICE_LOST at Kokiri at its cause, one submission past the 2 s TDR, for every GPU at once. It replaces the per-chip CPU-decode list. The flush code is correct across fences, the batch queue, sparse binds, the recorder and parallel replay. Merged ahead of the farm's results at the maintainer's call: the farm run on this PR is still the check for the other iGPUs and for load time on dGPUs.

For players on AMD iGPUs, Kokiri no longer resets the GPU, and areas load several times faster with a quarter of the texture memory. On other GPUs, an area load now waits on the GPU at each 16 Mtexel chunk.

Does it help: Yes, on the Radeon 860M: 3 interleaved pairs at two presets, and the ranges don't overlap.

  • Kokiri terrain upload: 15,239 → 2,228 ms.
  • Zora's Domain load: 11.33 → 5.16 s.
  • Peak VRAM: 7,888 → 3,702 MB. Decoded ASTC: 5,416 → 1,474 MB.
  • No device lost in 8 minutes of play.

Unproven on the other iGPUs (780M/880M/890M from #1040, #1730 and #1234; Raphael 0x164E and Granite Ridge 0x13C0) until the farm's windows+amd+igpu run reports.

Other systems:

  • dGPUs and the Deck: chunking also runs there at area loads (Kokiri is ~14 flushes). Each flush waits on the CPU with replay_mu held, so the load hitch grows by about the GPU decode time. Not measured; the farm's longest-load criterion will show it.
  • Parallel replay (par_active, tl_seg_index >= 0) and the RGBA8 all path keep one submission.
  • The recorder modes are fine: every call in the flush is a recorder boundary.
  • Pass timing: a span that crosses a flush includes the CPU wait.
  • iGPU texture cap: it returns to 4,096 MB, now that BC3 is used instead of RGBA8. That fits on the 860M (3.0–3.5 GB).

Switches:

  • RX_VK_ASTC_CHUNK_MPX (tune, default 16; 0 sends one submission): fine for the 860M (~125 ms a chunk) and estimated ~500 ms on a 2-CU part, so still under 2 s. The dGPU cost isn't measured.
  • RX_VK_ASTC_CPU (now kill, default off, =1 forces CPU decode): a good way back.

Reviewed: 5e7dfbee against 068fb8eb.

  • Read in full: both commits; replay_flush_upload and the replay code it relies on (Recording, recording_wait, begin/submit_recording, queue_flush_locked, segment submission, gpu_time and fo/gprof timestamps); the recorder's boundary list; memory_sparse_wait; all of upload_astc_gpu, upload_cmd, full_barrier and astc_chunk_texels; the device_policy rule; all five test files.
  • Read only for what they cut: the other checks that cut upload_astc_gpu.
  • Nothing run (a code review).

Fixed in this review: 85de7572, pushed to the head branch.

  • RX_VK_ASTC_CPU's row and the CPU-decode log line no longer say "0 forces the GPU".
  • A failed vkEndCommandBuffer in replay_flush_upload is now logged, like a failed submit.
  • A comment says why a level over the chunk bound takes one layer a dispatch.

Checked and fine:

  • The fence: r->fence is unsignaled at each use and reset after it.
  • The recording lookup: it needs !submitted, so a batch-held buffer is never reset.
  • Ordering: queue_flush_locked keeps held batches in order. The sparse wait is forwarded, and the main submission still waits on it.
  • Lifetimes: staging and mirror lifetimes follow the timeline value, so running early only finishes sooner. The decoder uses push descriptors, and the pipeline and push constants are rebound every run.
  • Timeline and timestamps: no timeline value is signalled by a chunk, and the timestamp queries were written in the first chunk.
  • last_barrier_cmd: clearing it is needed and stays inside the anonymous namespace.
  • The test: it fails on 30545573 with flushes=0 submits=1. It also fails if any of these is undone: the flush call, the last_barrier_cmd reset, or the per-run rebind.

Suggestions (follow-ups, not blocking)

  • Don't wait on the CPU at each chunk. Separate submissions are enough to stay under the TDR, as in yuzu's https://github.com/yuzu-emu/yuzu/pull/10398 and DXVK. Instead, submit each chunk without a fence into a fresh command buffer from upload_pool, and let the final fence cover them all. That keeps the GPU decode overlapped with recording on dGPUs. Worth doing if the farm shows longer loads.
  • Size chunks by time, not a fixed 16 Mtexels, e.g. clamp(chunk * 250 ms / measured, 4M, 64M). Then drop the advice in Risks that players lower RX_VK_ASTC_CHUNK_MPX: a default right for everyone, no player knob.
  • tests/test_astc_cpu_860m.py now checks a constant (p.astc_on_cpu = false;). Delete it or rename it, since test_device_policy covers the default and RX_VK_ASTC_CPU=0/1.
  • Add a source check on replay_flush_upload (srccut.block): the order queue_flush_locked → sparse wait → submit on r->fence → wait → reset fence → reset pool → begin, plus the par_active guard. The chunking test checks a stub of it, not the real function.

Testing:

  • The 860M runs are in the PR.
  • The Beanstalk run on this PR, across 5 targets, was still running when this merged.
  • The Linux test suite on the rebased head is still to run.

Overlaps and merge order:

  • #1843 rebases onto this. Its GprofSpan and this PR's chunk_texels both go right after const uint64_t limit = ... in upload_astc_gpu. The two fit together: step_texels counts its 8-layer runs.
  • #1847 touches the repack line next to this PR's all. It should check that its prep->buffer lifetime follows the recording's value.
  • #521 shadows all once it rebases.
**Recommendation: Approve.** It fixes the Windows iGPU `VK_ERROR_DEVICE_LOST` at Kokiri at its cause, one submission past the 2 s TDR, for every GPU at once. It replaces the per-chip CPU-decode list. The flush code is correct across fences, the batch queue, sparse binds, the recorder and parallel replay. Merged ahead of the farm's results at the maintainer's call: the farm run on this PR is still the check for the other iGPUs and for load time on dGPUs. For players on AMD iGPUs, Kokiri no longer resets the GPU, and areas load several times faster with a quarter of the texture memory. On other GPUs, an area load now waits on the GPU at each 16 Mtexel chunk. **Does it help:** Yes, on the Radeon 860M: 3 interleaved pairs at two presets, and the ranges don't overlap. - Kokiri terrain upload: 15,239 → 2,228 ms. - Zora's Domain load: 11.33 → 5.16 s. - Peak VRAM: 7,888 → 3,702 MB. Decoded ASTC: 5,416 → 1,474 MB. - No device lost in 8 minutes of play. Unproven on the other iGPUs (780M/880M/890M from #1040, #1730 and #1234; Raphael 0x164E and Granite Ridge 0x13C0) until the farm's windows+amd+igpu run reports. **Other systems:** - **dGPUs and the Deck:** chunking also runs there at area loads (Kokiri is ~14 flushes). Each flush waits on the CPU with replay_mu held, so the load hitch grows by about the GPU decode time. Not measured; the farm's longest-load criterion will show it. - **Parallel replay** (`par_active`, `tl_seg_index >= 0`) and the RGBA8 `all` path keep one submission. - **The recorder modes** are fine: every call in the flush is a recorder boundary. - **Pass timing:** a span that crosses a flush includes the CPU wait. - **iGPU texture cap:** it returns to 4,096 MB, now that BC3 is used instead of RGBA8. That fits on the 860M (3.0–3.5 GB). **Switches:** - `RX_VK_ASTC_CHUNK_MPX` (tune, default 16; 0 sends one submission): fine for the 860M (~125 ms a chunk) and estimated ~500 ms on a 2-CU part, so still under 2 s. The dGPU cost isn't measured. - `RX_VK_ASTC_CPU` (now kill, default off, `=1` forces CPU decode): a good way back. **Reviewed:** 5e7dfbee against 068fb8eb. - Read in full: both commits; `replay_flush_upload` and the replay code it relies on (Recording, recording_wait, begin/submit_recording, queue_flush_locked, segment submission, gpu_time and fo/gprof timestamps); the recorder's boundary list; `memory_sparse_wait`; all of `upload_astc_gpu`, `upload_cmd`, `full_barrier` and `astc_chunk_texels`; the device_policy rule; all five test files. - Read only for what they cut: the other checks that cut `upload_astc_gpu`. - Nothing run (a code review). **Fixed in this review:** 85de7572, pushed to the head branch. - `RX_VK_ASTC_CPU`'s row and the CPU-decode log line no longer say "0 forces the GPU". - A failed `vkEndCommandBuffer` in `replay_flush_upload` is now logged, like a failed submit. - A comment says why a level over the chunk bound takes one layer a dispatch. **Checked and fine:** - **The fence:** `r->fence` is unsignaled at each use and reset after it. - **The recording lookup:** it needs `!submitted`, so a batch-held buffer is never reset. - **Ordering:** `queue_flush_locked` keeps held batches in order. The sparse wait is forwarded, and the main submission still waits on it. - **Lifetimes:** staging and mirror lifetimes follow the timeline value, so running early only finishes sooner. The decoder uses push descriptors, and the pipeline and push constants are rebound every run. - **Timeline and timestamps:** no timeline value is signalled by a chunk, and the timestamp queries were written in the first chunk. - **`last_barrier_cmd`:** clearing it is needed and stays inside the anonymous namespace. - **The test:** it fails on 30545573 with `flushes=0 submits=1`. It also fails if any of these is undone: the flush call, the `last_barrier_cmd` reset, or the per-run rebind. **Suggestions** (follow-ups, not blocking) - **Don't wait on the CPU at each chunk.** Separate submissions are enough to stay under the TDR, as in yuzu's https://github.com/yuzu-emu/yuzu/pull/10398 and DXVK. Instead, submit each chunk without a fence into a fresh command buffer from `upload_pool`, and let the final fence cover them all. That keeps the GPU decode overlapped with recording on dGPUs. Worth doing if the farm shows longer loads. - **Size chunks by time, not a fixed 16 Mtexels**, e.g. `clamp(chunk * 250 ms / measured, 4M, 64M)`. Then drop the advice in Risks that players lower `RX_VK_ASTC_CHUNK_MPX`: a default right for everyone, no player knob. - **`tests/test_astc_cpu_860m.py` now checks a constant** (`p.astc_on_cpu = false;`). Delete it or rename it, since test_device_policy covers the default and `RX_VK_ASTC_CPU=0/1`. - **Add a source check on `replay_flush_upload`** (`srccut.block`): the order queue_flush_locked → sparse wait → submit on `r->fence` → wait → reset fence → reset pool → begin, plus the `par_active` guard. The chunking test checks a stub of it, not the real function. **Testing:** - The 860M runs are in the PR. - The Beanstalk run on this PR, across 5 targets, was still running when this merged. - The Linux test suite on the rebased head is still to run. **Overlaps and merge order:** - [#1843](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1843) rebases onto this. Its `GprofSpan` and this PR's `chunk_texels` both go right after `const uint64_t limit = ...` in `upload_astc_gpu`. The two fit together: `step_texels` counts its 8-layer runs. - [#1847](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1847) touches the `repack` line next to this PR's `all`. It should check that its `prep->buffer` lifetime follows the recording's value. - [#521](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/521) shadows `all` once it rebases.
Jer9 changed title from fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load) to WIP: fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load) 2026-11-20 05:55:44 +00:00
Jer9 changed title from WIP: fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load) to fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load) 2026-11-20 05:55:53 +00:00
HumanBean force-pushed fix/astc-upload-chunk-submit from 85de75720c
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 11s
beanstalk/approval not approved yet
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Website build and checks (pull_request) Has been cancelled
to afecbaee75
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 11s
beanstalk/approval not approved yet
CI / Website build and checks (pull_request) Failing after 4m8s
2026-11-20 05:57:23 +00:00
Compare
HumanBean dismissed HumanBean's review 2026-11-20 05:57:23 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

HumanBean approved these changes 2026-11-20 06:17:06 +00:00
HumanBean left a comment

Recommendation: Approve, again on afecbaee: the branch was rebased onto main after #1763 (the threads rewrite). The fix is the same, and only what #1763 removed changed.

Fixed in this review: afecbaee, on top of the rebased commits.

  • replay_flush_upload and upload_astc_gpu lose their guards against the parallel replay. It is gone on main: there is no par_active or tl_seg_index, and test_threaded_rendering refuses either name.
  • test_astc_upload_chunking.py drops its segment-worker case, and its cut now ends at the staging marker. The cut holds the same code as before.
  • test_sparse_mirror counts three sparse waits: the flush, an upload chunk and a submit.
  • The rebase conflicts were resolved: tl_seg is dropped, up_astc_texels is kept, and tests/parallel_rb_refresh_test.cpp stays deleted, as main has it.

Checked:

  • On main, recordings are still submitted directly, in order, under dev.present_mu, so the chunk's submit can't overtake an earlier recording. rec_cur, recordings, queue_flush_locked and memory_sparse_wait are unchanged.
  • vk_replay.cpp and vk_texture.cpp compile clean (-fsyntax-only) against main's headers with the build's flags.
  • The tests were not run.
**Recommendation: Approve,** again on afecbaee: the branch was rebased onto main after #1763 (the threads rewrite). The fix is the same, and only what #1763 removed changed. **Fixed in this review:** afecbaee, on top of the rebased commits. - `replay_flush_upload` and `upload_astc_gpu` lose their guards against the parallel replay. It is gone on main: there is no `par_active` or `tl_seg_index`, and `test_threaded_rendering` refuses either name. - `test_astc_upload_chunking.py` drops its segment-worker case, and its cut now ends at the staging marker. The cut holds the same code as before. - `test_sparse_mirror` counts three sparse waits: the flush, an upload chunk and a submit. - The rebase conflicts were resolved: `tl_seg` is dropped, `up_astc_texels` is kept, and `tests/parallel_rb_refresh_test.cpp` stays deleted, as main has it. **Checked:** - On main, recordings are still submitted directly, in order, under `dev.present_mu`, so the chunk's submit can't overtake an earlier recording. `rec_cur`, `recordings`, `queue_flush_locked` and `memory_sparse_wait` are unchanged. - `vk_replay.cpp` and `vk_texture.cpp` compile clean (`-fsyntax-only`) against main's headers with the build's flags. - The tests were not run.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1833
No description provided.