fix(vk): free on the GPU's progress, not on a time or a frame count: read-back timeouts and the upscalers' frees #788
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8015c9a
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!788
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:fix/async-safe-frees"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: macOS 26.7, Apple M4 Max (game-free checks only, arm64 and x86_64) · Needs: Windows and Linux, any GPU
Why
When the GPU runs late, the renderer can free or reset things the GPU is still using. Two places go by a clock or a frame count instead of the GPU's progress:
HOS_HEADLESS), the upscalers' images are freed 4 frames after they were last used. Nothing in that mode waits for the GPU each frame, so the GPU can still be using them.Either can end in a GPU fault or a lost device, on any GPU. Neither has been seen in a game; they were found by reading the source, and the new test shows each one on main's code. The issue tracker's titles (snapshot of 2026-11-14 20:01) have nothing that fits: #130 and #583 are device losses with other causes, #241 is the read-back's cost.
How to reproduce
Seen on: not seen in a game; shown on main's own functions by
tests/test_async_safe_frees.pyandtests/test_ui_texture_upload.py(caseheadless), on any OSSettings and switches: for the read-back: any settings, a GPU stall of more than 5 s during a texture write-back (
vk: a batch of N texture write-backs did not complete in time); for the frees:HOS_HEADLESS=1with DLSS, FSR 3.1 or FSR 4 as the upscaler orRX_VK_REACTIVE=1, orRX_VK_SHOT_MENU=1Steps: run
python -I -B tests/test_async_safe_frees.py --headers <Vulkan-Headers/include>on main; in a game run, setRX_VK_VALIDATION=1with the settings above and play while the GPU is busy (an area load)Expected: nothing reset, begun or destroyed while a GPU command may still use it
Actual: the test reports a pool and a fence reset, and a command buffer begun, under a read-back that has not finished, and each of the three free lists freeing before the GPU passed the work that used it. The validation layer would name it (for example
VUID-vkResetCommandPool-commandPool-00040,VUID-vkDestroyImage-image-01000) when the GPU is that far behindWhat changed
The read-back waits for a submit whose 5 s wait ran out before its pool or fence is reset. The free lists counted in frames also wait for the timeline value that followed the work, as
vk_fsr3.cpp(drop_image,free_dead) already does. Nothing is freed earlier than before; it is only freed later where the GPU had not finished.gpu/vk/vk_texture.cpp:readback_begin(main:3077) resetrb_pool, andreadback_submit_wait(:3111) resetrb_fence, with no wait after the 5 s wait (:3117) had given up. Vulkan forbids both resets while the command buffer or the submit is pending, and drivers reuse a reset pool's command memory.rb_pendingstays set from the submit until its fence is seen signalled, andreadback_settle()waits for it without a limit before either reset.flush_textures's timeout branch (:3337) already kept the batch's staging buffers until the next submission (:3348). The images the copies read were left free: a texture's death waits only for itslast_valueon the timeline (:4671,:4684), and the read-back's own submit is not on it. Each copied texture'slast_valueis now raised to that same submission.gpu/vk/vk_reactive.cpp,gpu/vk/vk_temporal.cpp,gpu/vk/vk_fsr3_inputs.cpp:laterandfree_dead(mainvk_reactive.cpp:76,:81;vk_temporal.cpp:291,:296;vk_fsr3_inputs.cpp:98,:103) freed an image, view, buffer or released feature 4 frames on. This includes DLSS's and FSR 4's, throughtemporal_later(vk_temporal.cpp:821).do_presentreturns beforepresent_image(gpu/vk/vk_replay.cpp:4499), so nothing waits for the present ring's fences. Only the 48-slot recording ring (vk_replay.cpp:176) limits how far behind the GPU is, and that is many frames when a frame has few recordings.vk_retirement_value()from when it was let go. It is freed oncevk_timeline_completed()has passed that value, and never on a value of 0.gpu/vk/vk_imgui.cpp:reap(main:197,:200) frees 4 counts on. With no window the overlays are drawn into the replay's recordings (RX_VK_SHOT_MENU), andidle()(:477) counts frames without waiting for the GPU.vk_draw.cpp, 120 frames). With no window every frame records its offscreen copy, so 120 frames are more recordings than the ring holds.Measurements
Verdict: nothing to measure. A normal run does the same work at the same time; only a GPU that runs late changes when something is freed.
Flags and settings
None
Testing
Tested: game-free, on macOS 26.7, Apple M4 Max, arm64 and x86_64 (Rosetta):
tests/test_async_safe_frees.py(new) andtests/test_ui_texture_upload.py(caseheadlessadded) fail on mainaa95d35fand pass on this PR, on both architectures.vk_texture.cpp,vk_reactive.cpp,vk_temporal.cpp,vk_fsr3_inputs.cpp,vk_imgui.cpp,vk_replay.cpp) withnx2/build.py's macOS commands, on both architectures: every compile exits 0 with no warnings, as on main.vk_retirement_valueandvk_timeline_completed, both defined invk_replay.o.python -I -B tests/run_all.pyon this Mac, where main's runner takes macOS for Linux: 544 passed, 33 failed, 17 skipped. Mainaa95d35fgives 543 passed, 33 failed, 17 skipped, and by name every result is the same apart from the new test (pass). The 33 fail the same way on main on this Mac: test_appimage_package_retry, test_appimage_player_build, test_collect_wide, test_control, test_cpm_discovery, test_crash_dialog, test_crash_record, test_env_on, test_game_layout, test_lift_cleanup_error, test_linux_preflight, test_longshot_ctl, test_low_memory_notice, test_make_game_cache, test_mem_gate, test_mem_gate_children, test_mirror_trim, test_oot_state_tap, test_options_budget_once, test_options_path, test_pcache_header_read, test_pipeline_cache_file, test_pipeline_cache_save, test_pipeline_play_cache, test_pipeline_tour, test_prof_symbols, test_program_cache_cleanup, test_program_cache_one_variant, test_readback_late, test_report_run, test_split_giants, test_vk_pgo, test_walk_key.Not tested: in the game, on any platform (neither hazard can be produced on demand: it needs a GPU stall over 5 s, or the GPU more than 4 frames behind with no window). Not built on Windows (llvm-mingw) or Linux. The code has no platform branch, so it runs the same on both.
Must be tested:
90f6c3e3) and mainaa95d35fthe same way and runpython -I -B tests/run_all.py. Then start each withHOS_HEADLESS=1 RX_VK_FSR=1 RX_VK_REACTIVE=1 RX_VK_VALIDATION=1(no window, FSR 3.1, the reactive mask) and play to the title and into the field. Expect every check to pass, and no validation error on this PR that main does not also show. A read-back that runs out of its 5 s wait cannot be made on demand (by hand: test request T-NTFW)Risks
What could break: a read-back after a GPU stall now waits for the stall to end instead of going on. A real hang still ends at the GPU hang watch (
RX_VK_GPU_HANG_SEC, 30 s). The upscalers' images and the headless overlays are kept until the GPU has passed them, at most a few frames longer than before.Turning it off: revert the commit
Compatibility: Nothing changes
Checklist
Everyone
type(scope): description) of the status line's type.main(mainaa95d35f, head90f6c3e3).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.WIP: fix(vk): free on the GPU's progress, not on a time or a frame count: read-back timeouts and the upscalers' freesto fix(vk): free on the GPU's progress, not on a time or a frame count: read-back timeouts and the upscalers' freesThe frees counted in frames (reactive mask, temporal/DLSS/FSR 4, FSR 3 inputs, headless overlays) read right to me: they copy the
vk_retirement_value()/vk_timeline_completed()rulevk_fsr3.cppalready uses, never free on 0, andfree_dead(all)invk_temporal.cppstill frees everything on teardown. Novk_api.hchange, no generated files. That half I would approve.The read-back half has to change before merge, because #553 and #554 are open for exactly the same two defects (a read-back reset under a pending submit; a timed-out batch's source images freed under its copies), and they were run on a Steam Deck (non-regression, no timeout hit). This PR does not mention them. As it stands:
gpu/vk/vk_texture.cpp: therb_pendingdeclaration (553 declaresbool rb_pending = false, rb_lost = false;at the same place),readback_submit_wait's last lines, and thefor (Flush& f : fs)timeout loop (554 edits the same lines to addf.t->last_value).readback_beginreturn false while the fence is unsignalled (a 0 ns wait) and keeps the batch's textures resident for a later pass, with a device-lost state. This PR instead waits withUINT64_MAXinreadback_settleand then goes on, evicting the batch's textures with no copy of their texels (the loss 553 removes). The new tests also assert opposite behaviour (test_readback_timeout_reuse.pyexpectsreadback_begin()to fail while pending;test_async_safe_frees.pyexpects it to begin after an unbounded wait), so both cannot pass on one tree.last_valueraise is the same fix as 554 (if (f.t)here, unguarded there).flush_texture/texture_frame: a stall that never reports device lost holds that thread until the 30 s hang watch ends the process. 553's non-blocking path avoids this.Please drop the read-back parts (the
vk_texture.cpphunks,F5_READBACKand thetexturechecks intests/test_async_safe_frees.py, its docstring lines) and keep this PR to the frame-counted frees, rebased after 553 and 554 land. If you think the unbounded wait is better than 553's design, argue that on 553 instead.Smaller:
Dead::value = retired_value()invk_imgui.cppis a default member initializer, so it is taken wherever aDeadis constructed. That is right for the currentpush_back({presents, ...})sites; a one-line comment saying so would help.@ -3062,0 +3063,4 @@// GPU may still be running it, and a pool whose command buffer is pending, or a fence of a pending submit, must not be reset (a// driver reuses a reset pool's command memory). So rb_pool and rb_fence wait for it, without a limit, before they are reset; they// were reset with no wait, its command buffer begun again under itbool rb_pending = false;Duplicates and conflicts with #553 (same
rb_pending, opposite design: 553 returns false from readback_begin while pending and keeps the textures; this waits unbounded then evicts without the copy). Drop the read-back part and rebase after 553/554.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.