fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load) #1833
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1833
Loading…
Reference in a new issue
No description provided.
Delete branch "Jer9/Longshot:fix/astc-upload-chunk-submit"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Large GPU ASTC texture-array uploads previously packed nearly 2,000 compute dispatches into one queue submission, exceeding Windows' ~2 s WDDM timeout and crashing with
VK_ERROR_DEVICE_LOSTacross many integrated GPUs and handheld APUs (#1040, #1730). Splitting uploads across bounded submissions (RX_VK_ASTC_CHUNK_MPX, default16million decoded texels per chunk, ~125 ms per submit on Radeon 860M) stops those Windows 2 s TDR resets — fixingVK_ERROR_DEVICE_LOSTon Windows iGPUs and removing every chip- and CPU-specificastc_on_cpuworkaround fromdevice_policy.hso all GPUs share the unified GPU ASTC decode (BC3/BC7) path. On iGPUs previously forced onto the CPU fallback, returning to GPUBC3decode cuts Kokiri Forest terrain upload time by 85.4% (15,239 ms -> 2,228 ms,6.84xfaster) and Zora's Domain load time by 54.5% (11.33 s -> 5.16 s,2.20xfaster), restores normalBC3texture footprint (5,416 MB -> 1,474 MBdecoded ASTC;7,888 MB -> 3,702 MBpeak VRAM on Radeon 860M), and lifts GPU-bound1080p Nativesteady-state FPS in Zora's Domain by +10.2%.We helped create that PCI-ID and CPU-graphics exception list ourselves — routing Radeon 860M (
0x1114) to CPU decode in #1114, then adding Ryzen 7000 Raphael (0x164E) and Ryzen 9000 Granite Ridge (0x13C0) desktop CPU graphics plus a 6-8 GB iGPU texture-budget exception in #1559, while unlisted RDNA 3 APUs like 780M/890M (#1040, #1730) kept crashing. Hiding individual chips on CPU decode traded TDR resets for ~15 s area-entry freezes and ~5.4 GB of uncompressed RGBA8 textures that still break 16 GB RAM laptops. Bounded upload flushing fixes the shared root cause inupload_astc_gpuonce across all hardware, inspired by yuzu's bounded ASTC compute submission splitting (https://github.com/yuzu-emu/yuzu/pull/10398, carried into Eden'sMaxTexelsPerSubmission) and DXVK's bounded upload/compute submission splitting for Windows WDDM timeouts.Reproducing on
main: On a Windows AMD iGPU (e.g., Radeon 860M0x1114), setRX_VK_ASTC_CPU=0(or runtests/test_astc_upload_chunking.pyon30545573c9/068fb8ebe2) and enter Kokiri Forest. The two1365x1365 x90terrain arrays (~1,980 compute dispatches, ~223M decoded texels across mip cascades) are recorded into a singler->uploadsubmission (flushes=0 submits=1), stalling the GPU past the ~2 s WDDM timeout (VK_ERROR_DEVICE_LOST), while defaultastc_on_cpu=truefreezes ~15.2 s decoding on the CPU and bloats texture memory to ~7.2 GB.Where reviewers should look first:
gpu/vk/vk_replay.cpp(replay_flush_upload) &gpu/vk/vk_internal.h(up_astc_texels): ends, submits (underdev.present_mu, forwarding any pendingsparse_bind_sem), waits onr->fence, resetsr->upload_pool, and re-beginsr->uploadin place while keepingbegin_recording()andsubmit_recording()unchanged.gpu/vk/vk_texture.cpp(upload_astc_gpu,astc_chunk_texels): tracks cumulative decoded texels (with a64K-texel per-dispatch floor for mip tails), callsreplay_flush_upload()before exceedingRX_VK_ASTC_CHUNK_MPX(16MPx default,0disables), clearslast_barrier_cmdso the nextvk_full_barrier()starts the reset command buffer cleanly, and rebindsastc_pipeand descriptors on both BC and RGBA8 paths.gpu/vk/device_policy.h,gpu/vk/switches.def,docs/dev/reference/switches.md: drops the Windows AMD iGPU PCI-ID rule (p.astc_on_cpu = falseacross all GPUs), demotesRX_VK_ASTC_CPUto akilloverride (=1forces CPU decode), and documentsRX_VK_ASTC_CHUNK_MPX.tests/test_astc_upload_chunking.py&tests/: game-free test failing onmain(30545573c9:flushes=0 submits=1) and passing here (5e7dfbeed8:flushes=17 submits=18), plus updated source-cutting/policy checks (tests/test_astc_cpu_860m.py,tests/test_device_policy.py,tests/parallel_rb_refresh_test.cpp,tests/test_sparse_mirror.py).Fixes #1040, See #1730, See #1114, See #1559
Testing
Tested:
1002:1114, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (LLPC), standalonelongshot_vk.dll(built in 79 s, 0 failed):gpu/vk/code to rebased5e7dfbeed8): 8+ minutes in Kokiri Forest (21,120+presents,30.0-48.2 fps,1,228pipelines,RX_VK_PRECOMPILE=0) with0VK_ERROR_DEVICE_LOST; textures stayed3,011-3,495 MBinside the4,096 MBiGPU budget.0de25c3e4avs pre-rebase fix build, 13 runs: 3 pairs atAutoMinimum, 3 pairs at1080p Native, 1gprofrun):0VK_ERROR_DEVICE_LOST,0write-back timeouts,0evicted textures,0quick pipelines,compiled 0warm cache.5e7dfbeed8(onmain068fb8ebe2):tests/test_astc_upload_chunking.py --cxx clang++failed before the fix (30545573c9,flushes=0 submits=1, exit1) and passed all 9 modes on5e7dfbeed8(flushes=17 submits=18 max_texels=15930497);tests/test_switch_table.py,tests/test_astc_cpu_860m.py,tests/test_device_policy.py,tests/test_parallel_rb_refresh.py --cxx clang++,tests/test_sparse_mirror.py,tests/test_doc_anchors.py,tests/test_utf8_paths.py,tests/test_log_levels.py,tests/test_thinlto_tls.py,tests/test_program_stamp_knobs.py, andscripts/check_third_party.pyall passed.#1802full suite parity (before rebasing onto068fb8ebe2, against0de25c3e4a):tests/run_all.pyhit772 passed, 50 failed (49 FAIL + 1 TIMEOUT tests/test_vbytes.py), 54 skipped (windows, 16 at once); all 49 non-timeout failures also failed on0de25c3e4a(0regressions), and#1802/#1807(0de25c3e4a..068fb8ebe2) changed0executablegpu/vk/C++ lines.Still to test:
windows+amd+igpu— Raphael0x164E, Granite Ridge0x13C0, 780M/880M/890M APUs): enter Kokiri Forest and Zora's Domain on default settings withoutRX_VK_ASTC_CPU, expecting GPU ASTC decode with zeroVK_ERROR_DEVICE_LOSTTDR resets (@Beanstalk test Windows AMD iGPUs (windows+amd+igpu) on Kokiri Forest and Zora's Domain)windows+nvidia,windows+amd+dgpu,linux+radv,steamdeck): sanity-run Kokiri Forest and Zora's Domain to confirm unchanged GPU ASTC upload stability (@Beanstalk test Windows NVIDIA, Windows AMD dGPU, Linux RADV, and Steam Deck on Kokiri Forest and Zora's Domain)5e7dfbeed8after#1802(by hand)Measurements
Verdict: Fixes GPU WDDM TDR resets on Radeon 860M with default GPU ASTC decode, cuts Kokiri terrain upload time by 85.4% (
15,239 ms -> 2,228 ms,6.84xfaster), cuts Zora's Domain load time by 54.5% (11.33 s -> 5.16 s,2.20xfaster), cuts peak VRAM by 53.1% (7,888 MB -> 3,702 MB) and decoded ASTC memory by 72.8% (5,416 MB -> 1,474 MB), improves Kokiri init fps by+57.5%, and keeps steady-stateAutofps within the spread.Switches:
RX_VK_ASTC_CHUNK_MPX(tune) ships at its default16million decoded texels per chunk (choosing ~125 ms compute+BC submissions well under the ~2 s WDDM TDR limit without excessive fence round-trips: 18 submits /2.23 stotal on Kokiri vs0disabled);RX_VK_ASTC_CPU(kill) shipsoffby default on all GPUs, returning Windows AMD iGPUs to GPU decode while preserving=1as a runtime override.Baseline:
mainat0de25c3e4avs the identical pre-rebase fix build (rebased onto068fb8ebe2as5e7dfbeed8;0de25c3e4a..068fb8ebe2touched0executablegpu/vk/C++ lines, and standalonelongshot_vk.dll+tests/test_astc_upload_chunking.pyre-verified on5e7dfbeed8).Setup: Windows 11 (26200), AMD Ryzen AI 7 350, Radeon 860M (
1002:1114, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (LLPC); Quality presetAuto(Minimum,768x432internal ->1536x864window crop, FSR1 sharpness 30, AF 8x, vsync=0,RX_VK_PRECOMPILE=0, warm cachecompiled 0); 3 interleaved pairs (low_0/2/4vslow_1/3/5), median reported.0de25c3e4a5e7dfbeed8VK_ERROR_DEVICE_LOST/ WB timeouts / evicted0 / 0 / 0(crashes ifRX_VK_ASTC_CPU=0)0 / 0 / 01365x1365 x90upload (ms)15,239(14,910-15,997)2,228(2,199-2,238)-85.4%(6.84xfaster)slow submit(ms)15,395.4(15,070.4-16,161.2)2,303.6(2,271.7-2,313.5)-85.0%s)11.33(10.71-11.80)5.16(5.06-5.27)-54.5%(2.20xfaster)MB)7,888(7,696-7,952)3,702(3,638-3,766)-53.1%(-4,186 MB)MB)7,215(7,024-7,351)3,131(3,118-3,256)-56.6%(-4,084 MB)MB)5,416(5,230-5,533)1,474(1,472-1,502)-72.8%(-3,942 MB)wall_fps(GPU ms)28.16(24.15-28.89,17.6 ms)44.35(41.05-46.15,17.1 ms)+57.5%wall_fps(GPU ms)44.11(42.65-44.36,18.7 ms)42.06(37.98-50.33,18.1 ms)wall_fps(GPU ms)48.22(46.39-48.75,17.0 ms)47.59(46.85-48.32,17.3 ms)At
1080p Native(GPU-bound, 3 interleaved pairshigh_0/2/4vshigh_1/3/5), the upload and memory wins replicate (15,143 -> 2,227 msKokiri upload,13.60 -> 5.64 sZora load,7,743 -> 3,862 MBpeak VRAM,5,177 -> 1,462 MBpeak decoded ASTC, and Kokiri init18.56 -> 23.06 fps), while Kokiri revisit stays within the spread (22.85[22.63-22.92]vs23.09[22.16-23.57]). Zora's Domain steady-state fps rises cleanly without overlap from19.59[19.29-19.81](45.9 msGPU) to21.59 fps[20.57-21.75](42.1 msGPU,+10.2%/-3.8 ms), because storing decoded ASTC terrain in ~1.46 GB BC3 rather than ~5.18 GB RGBA8 eases shared UMA memory bandwidth pressure.Risks
replay_flush_upload()ends, submits, and fence-waitsr->uploadunderdev.present_mumid-recording: safe only on the single replay upload path (upload_astc_gpu()explicitly skips chunk splitting when called outside the main upload recording,tl_seg_index >= 0).0x164E, Granite Ridge0x13C0) and 780M/890M APUs have not been re-tested directly here yet; at ~1/4 the CUs of the 860M, 16M-texel chunks could take longer per submission. Players can lowerRX_VK_ASTC_CHUNK_MPX=4or restore CPU decoding immediately withRX_VK_ASTC_CPU=1, or a clean revert restores prior behavior.longshot_options.ini, and shader/program caches are unchanged; Windows AMD iGPUs previously forced onto CPU decode regain normal hardware texture budgets (4,096 MBon 860M) and ~5 GB lower RAM/VRAM pressure.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.gpu/vk/switches.def.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Tested in game: Windows 11 (26200), AMD Ryzen AI 7 350, Radeon 860M (1002:1114, 6 GB UMA, 26 GB usable RAM), Adrenalin 26.10.1 (LLPC): 8+ min interactive playtest in Kokiri Forest (21,120+ presents, 30.0-48.2 fps, 0 VK_ERROR_DEVICE_LOST, textures 3,011-3,495 MB within 4,096 MB budget) plus 13 warm-cache headless interleaved A/B runs across Kokiri Forest and Zora's Domain (Kokiri terrain upload 15,239 ms -> 2,228 ms, Zora load 11.33 s -> 5.16 s, 0 VK_ERROR_DEVICE_LOST, 0 write-back timeouts).
@Jer9 only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.Possibly related, for context:
@Beanstalk test Windows AMD iGPUs (windows+amd+igpu), Windows NVIDIA (windows+nvidia), Windows AMD dGPU (windows+amd+dgpu), Linux RADV (linux+radv), and Steam Deck (steamdeck) on Kokiri Forest and Zora's Domain — verify GPU ASTC terrain upload completes with zero VK_ERROR_DEVICE_LOST resets and fast load times
@Jer9 only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.#1843 changes the same loop in
upload_astc_gpu. At levels of 1024 or less it runs eight layers at a time, so each of your chunks should hold about an eighth of the runs and barriers at those levels and finish sooner, for the same texels decoded. That part is untested: I haven't run the two together or on an iGPU. The two fit together. Whichever merges second needs a small rebase, and I'm happy to do it on mine.Beanstalk test farm · an earlier run: commit
5e7dfbe(the newer results are in a comment below)Commit
5e7dfbe· tier reviewed · approved by HumanBean (👍 on Jer9's request) · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)The plan compares default ASTC uploads on main and the pull request in Kokiri Forest and Zora's Domain across the five requested Windows and Linux hardware groups. Each tester runs one pass per build, taking about 11 minutes, with load times, frame times, VRAM, logs, and terrain screenshots recorded. A crash, device-loss or upload-failure log, CPU ASTC fallback, significant load-time or performance regression, or excessive screenshot difference fails the pull request. The available tag counts do not establish whether every requested tag combination is online; any unmatched target will wait for a tester. No tester online now matches windows+amd+igpu: that target waits for one.
kokiri-start, then teleport to Kokiri Forest, take the picture kokiri-terrain to compare with main, measure kokiri for 20 s (after 5 s warm-up), teleport to Zora's Domain, take the picture zora-terrain to compare with main, measure zora for 20 s (after 5 s warm-up); 1 run on each side.Started at 05:15 UTC.
Estimate: done in about 45 min, from the queue and the testers online now.
NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0; 2: windows+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 4080, 621.1.0Measurements · windows+nvidia
Verdict: could not be judged on windows+nvidia (2/1 report).
Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu
NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM; 2: windows+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 4080, 621.1.0AMD Ryzen 7 9800X3D 8-Core Processor, 32 GB RAMScene: terrain-uploads, from the repository's save kokiri-start; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); terrain-uploads: 1 game start a side, kokiri 20 s after 5 s warm-up, zora 20 s after 5 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)Nothing was measured: no metric came back.
Flags and settings
None: the plan varied no setting, switch or build option.
Markdown for the PR description
Beanstalk test farm
Commit
afecbae· tier untrusted · approved by nobody yet · dashboardA member of
BigBeanscan review this commit and comment@Beanstalkto test it.Recommendation: Approve. It fixes the Windows iGPU
VK_ERROR_DEVICE_LOSTat Kokiri at its cause, one submission past the 2 s TDR, for every GPU at once. It replaces the per-chip CPU-decode list. The flush code is correct across fences, the batch queue, sparse binds, the recorder and parallel replay. Merged ahead of the farm's results at the maintainer's call: the farm run on this PR is still the check for the other iGPUs and for load time on dGPUs.For players on AMD iGPUs, Kokiri no longer resets the GPU, and areas load several times faster with a quarter of the texture memory. On other GPUs, an area load now waits on the GPU at each 16 Mtexel chunk.
Does it help: Yes, on the Radeon 860M: 3 interleaved pairs at two presets, and the ranges don't overlap.
Unproven on the other iGPUs (780M/880M/890M from #1040, #1730 and #1234; Raphael 0x164E and Granite Ridge 0x13C0) until the farm's windows+amd+igpu run reports.
Other systems:
par_active,tl_seg_index >= 0) and the RGBA8allpath keep one submission.Switches:
RX_VK_ASTC_CHUNK_MPX(tune, default 16; 0 sends one submission): fine for the 860M (~125 ms a chunk) and estimated ~500 ms on a 2-CU part, so still under 2 s. The dGPU cost isn't measured.RX_VK_ASTC_CPU(now kill, default off,=1forces CPU decode): a good way back.Reviewed:
5e7dfbeeagainst068fb8eb.replay_flush_uploadand the replay code it relies on (Recording, recording_wait, begin/submit_recording, queue_flush_locked, segment submission, gpu_time and fo/gprof timestamps); the recorder's boundary list;memory_sparse_wait; all ofupload_astc_gpu,upload_cmd,full_barrierandastc_chunk_texels; the device_policy rule; all five test files.upload_astc_gpu.Fixed in this review:
85de7572, pushed to the head branch.RX_VK_ASTC_CPU's row and the CPU-decode log line no longer say "0 forces the GPU".vkEndCommandBufferinreplay_flush_uploadis now logged, like a failed submit.Checked and fine:
r->fenceis unsignaled at each use and reset after it.!submitted, so a batch-held buffer is never reset.queue_flush_lockedkeeps held batches in order. The sparse wait is forwarded, and the main submission still waits on it.last_barrier_cmd: clearing it is needed and stays inside the anonymous namespace.30545573withflushes=0 submits=1. It also fails if any of these is undone: the flush call, thelast_barrier_cmdreset, or the per-run rebind.Suggestions (follow-ups, not blocking)
upload_pool, and let the final fence cover them all. That keeps the GPU decode overlapped with recording on dGPUs. Worth doing if the farm shows longer loads.clamp(chunk * 250 ms / measured, 4M, 64M). Then drop the advice in Risks that players lowerRX_VK_ASTC_CHUNK_MPX: a default right for everyone, no player knob.tests/test_astc_cpu_860m.pynow checks a constant (p.astc_on_cpu = false;). Delete it or rename it, since test_device_policy covers the default andRX_VK_ASTC_CPU=0/1.replay_flush_upload(srccut.block): the order queue_flush_locked → sparse wait → submit onr->fence→ wait → reset fence → reset pool → begin, plus thepar_activeguard. The chunking test checks a stub of it, not the real function.Testing:
Overlaps and merge order:
GprofSpanand this PR'schunk_texelsboth go right afterconst uint64_t limit = ...inupload_astc_gpu. The two fit together:step_texelscounts its 8-layer runs.repackline next to this PR'sall. It should check that itsprep->bufferlifetime follows the recording's value.allonce it rebases.fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load)to WIP: fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load)WIP: fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load)to fix(vk): chunk GPU ASTC uploads to fix iGPU VK_ERROR_DEVICE_LOST (Windows 2s TDR) and drop per-chip CPU fallbacks (15.2 s to 2.2 s Kokiri load)85de75720cafecbaee75New commits pushed, approval review dismissed automatically according to repository settings
Recommendation: Approve, again on
afecbaee: the branch was rebased onto main after #1763 (the threads rewrite). The fix is the same, and only what #1763 removed changed.Fixed in this review:
afecbaee, on top of the rebased commits.replay_flush_uploadandupload_astc_gpulose their guards against the parallel replay. It is gone on main: there is nopar_activeortl_seg_index, andtest_threaded_renderingrefuses either name.test_astc_upload_chunking.pydrops its segment-worker case, and its cut now ends at the staging marker. The cut holds the same code as before.test_sparse_mirrorcounts three sparse waits: the flush, an upload chunk and a submit.tl_segis dropped,up_astc_texelsis kept, andtests/parallel_rb_refresh_test.cppstays deleted, as main has it.Checked:
dev.present_mu, so the chunk's submit can't overtake an earlier recording.rec_cur,recordings,queue_flush_lockedandmemory_sparse_waitare unchanged.vk_replay.cppandvk_texture.cppcompile clean (-fsyntax-only) against main's headers with the build's flags.futondestroyer referenced this pull request2026-11-20 06:35:08 +00:00