fix(vk): a read-back after a timed-out one waits for the GPU, and its textures are kept #553
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: needs approval
beanstalk: running
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!553
Loading…
Reference in a new issue
No description provided.
Delete branch "zeldaootr/Longshot:readback-timeout-reuse"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: medium · Tested on: Linux x86-64 (game-free); Steam Deck OLED, SteamOS 3.9.5, RADV (a tester's run) · Needs: none
Why
When the GPU was stuck long enough for a texture read-back to time out (5 s), the next read-back reused its command buffer and fence while the GPU was still running them, which Vulkan forbids and can turn a stall into a crash or a device lost; and the textures waiting on it were evicted with nothing copied, so they came back empty. The tracker has no issue for it (searched).
How to reproduce
Seen on: the source, found by reading
gpu/vk/vk_texture.cpp; a 5 s GPU stall cannot be made on demand in the game, so the game-free test reproduces it against a stub VulkanSettings and switches: defaults
Steps: at the test-only commit
08514708(mainaa95d35fplus the test), runpython3 -I -B tests/test_readback_timeout_reuse.py --cxx g++Expected: a read-back after a timed-out one waits for the GPU to finish the old one, the textures waiting on it are kept, and nothing calls Vulkan after a device lost
Actual: on main the next read-back resets and resubmits the command buffer the GPU still runs (3 Vulkan rule violations), the textures are evicted with no copy, and after a device lost read-backs still call Vulkan (10 calls, 2 waits of 5 s)
What changed
A read-back now remembers that the one before it timed out or lost the device, and does not reuse its command buffer and fence until the GPU is past them. Textures whose write-back could not run because of it stay resident for a later pass instead of being evicted empty.
readback_submit_wait()ingpu/vk/vk_texture.cppnotes a wait that timed out (rb_pending) or failed otherwise (rb_lost, the device lost).readback_begin()then checks the fence without waiting: while the old read-back still runs, no new one starts; once it has signalled, read-backs carry on; after a device lost, none starts and none is waited for.flush_textures()marks the write-backs that could not run because a read-back is still pending (Flush::busy), andtexture_frame()keeps those textures (flush_later) for a later pass instead of evicting them with no copy; textures that need no write-back still go.tests/test_readback_timeout_reuse.pycuts the read-back state and functions,flush_texturesand the end oftexture_frame's eviction out of the source and runs them against a stub Vulkan that tracks what is pending: a read-back after a timed-out one, an eviction pass whose first batch times out, and a device lost.Measurements
Verdict: pass. The fix is proven by the game-free test (3 of 3 cases fail on main, 0 with this PR). In the game it is a non-regression result: on a tester's Steam Deck OLED the timed-out read-back never happened (no
did not complete in timeline in 4 runs), and fps, faults and texture evictions were the same as main within main's own run-to-run spread.Hardware: Steam Deck OLED, AMD Custom APU 0932 (RADV VANGOGH), 16 GB, Mesa 26.2.0-devel git-5c65d4fc31, SteamOS 3.9.5 (build 20261023.100), kernel 7.2.7-valve2, Game Mode, docked, stock clocks (dukenheimer's run, test request T-4H95); game-free: Linux x86-64 server, no GPU or game
Scene: a route through the control bridge: Continue into Kokiri Forest, then 10 debug teleports with about 45 s in each area (Hyrule Field, Kakariko, the Market, Hyrule Field, the Lost Woods, Goron City, Death Mountain Trail, Zora's River, Gerudo Valley, Kokiri Forest), about 11 minutes a run; the Minimum (4GB) preset (render 50% of 1280x720, a 2.2 GB VRAM budget, Handheld profile), the game's own upscaler, frame generation off
Method: fps from the bridge's status every 5 s, averaged over the time in play; the log lines from each run's
longshot_log.txt; the kernel's amdgpu lines; one picture at each arrival. Four runs, interleaved (this PR, main, main, this PR), one test folder with warm caches and the binaries swapped; the median of each build's two runs. Game-free:python3 -I -B tests/test_readback_timeout_reuse.py --cxx g++at08514708and at3b245fa7(rebased onto mainaa95d35f), with g++ 13.3 and with clang 23.1.5; before that, at9fc90117and515a3fdfon main53f96d78, and before the first rebase atecacc39cand93c285b7, twice each, alternatingHow to reproduce: build main
7ff5256eand the combined build (main7ff5256ewith #550 to #558 merged in order, this PR at93c285b7), both x86-64-v3 with the release play profile and LLVM 23.1.5; choose the Minimum (4GB) preset, then drive the route above; for the game-free check, run the test command at each commit7ff5256e93c285b7(in the combined build)did not complete in timelinesevicted, made againlinesBy area, the two builds' means are within 2 fps everywhere, in both directions (this PR +1.9 in Goron City, -1.6 in Gerudo Valley); main's own two runs differ by more than the builds do. The textures were under pressure in every run: 2.7 to 2.8 GB against a 2,560 MB texture budget, VRAM in use peaking at 4.5 to 5.1 GB of the 5.3 to 6.1 GB allowed, about 4.3 GB swapped, MemAvailable down to 281-471 MB, and two
written back into their memorylines a run (8,640 KB each, 12 to 64 ms). The kernel's amdgpu lines were only the display'sLosing delta precision while programming shaper LUTnotices (6 a start, 12 in run 1): no reset, timeout, hang or fault. The tester's table, the fps by area and the pictures: #553 (comment)aa95d35f(with the test,08514708)3b245fa7Flags and settings
None.
Testing
Tested: Linux x86-64, game-free. The new test fails on the test-only commit
08514708and passes at3b245fa7, this branch rebased onto mainaa95d35f, with g++ 13.3 and with clang 23.1.5 (before, it did the same at9fc90117and515a3fdfon main53f96d78, and atecacc39cand93c285b7, twice each, alternating) (python3 -I -B tests/test_readback_timeout_reuse.py --cxx g++). This PR's own tests and the checks its fix commit edits ran at both commits, and once more onland/integration27553c83 (main7ff5256ewith this PR and twelve other fixes merged), where the thirteen fixes' tests and edited checks pass, 16 of 16; the fullpython3 -I -B tests/run_all.pyran on that branch too (see the checklist).In the game: dukenheimer ran test request T-4H95 on a Steam Deck OLED (RADV VANGOGH, Mesa 26.2.0-devel, SteamOS 3.9.5, Game Mode, docked) and reported a pass on 2026-11-15 at 03:12 UTC (#553 (comment)). The build was the combined build of main
7ff5256ewith #550 to #558 merged in order, this PR at its head then,93c285b7(3b245fa7after the rebases, the same change bygit patch-id), against main7ff5256e: four runs of the 11-minute route above, interleaved. Nodid not complete in timeline in any run, so the timed-out read-back this PR changes never happened there: it is a non-regression run, not a run of the fix. No crash, device lost, device fault or GPU stall, no kernel reset, timeout, hang or fault, and no garbled or rainbow textures in the 44 arrival pictures (they differ only in timing: an arrival still fading in, Navi's place, the area title part-way through its dissolve, which main shows the same way). fps was the same within main's own spread.The branch has been rebased since the tester's run, onto main
aa95d35f, with its commits re-authored under the forge's noreply address; each commit's change is identical to before bygit patch-id, so the commits named above are the ones tested and the result stands for this head (a6ad0e0e).Not tested: A real 5 s GPU stall or device lost, in the game or on any GPU: the fixed paths are proven only against the stub, since a stall that long cannot be made on demand. Windows in the game (the renderer was built for Windows, see the checklist).
Must be tested: None
Risks
What could break: While a read-back is pending, the textures that need a write-back stay resident and are tried again on a later pass; a GPU that hangs for good without reporting device-lost keeps them resident for the session (more VRAM held, not lost texels). Write-backs of virtual pages that cannot run while a read-back is pending are still lost, as on main. After a device lost, every later write-back fails at once.
readback-timeout-keeps-source-textures(#554) changes the lines next to these inflush_textures(the timed-out branch); the two merge without a conflict in either order and are meant to land together or one after the other.Turning it off: revert the commit
Compatibility: Nothing changes
Checklist
Everyone
type(scope): description) of the status line's type.main. Onto main94abdd57(headbb02fd2b) with no conflict, with the commit subjects made Conventional Commits, which CI's lint job now checks for every commit, on a base that has CI's config (#417); each commit's change is the same as before bygit patch-id. The commits named in this description are from before this rebase:08514708is now3531263a,3b245fa7is now9c2110b0,a6ad0e0eis nowbb02fd2b. Re-run on94abdd57:test_readback_timeout_reusefails at the test-only commit3531263aand passes at the head, with g++ 13.3 and clang 23.1.5; of the 37 checks that are this PR's or read its files (g++ 13.3), 35 pass and the 2 that fail fail on main94abdd57too (test_bc_storage_views,test_texture_init_barrier).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Run on this branch atcbe9b9ceon main53f96d78, before this rebase (a6ad0e0enow, the same patches bygit patch-id; the Vulkan and SPIR-V headers from the CPM cache, 8 at once, on a loaded Linux x86-64 server): 530 passed, 16 failed, 48 skipped. 15 of them fail on main53f96d78in the same run:test_appimage_player_build,test_bake_portability,test_cpm_discovery,test_crash_dialog,test_env_on,test_lift_cache,test_lift_cleanup_error,test_make_game_cache,test_pipeline_tour,test_readback_late,test_vk_pgo,test_vram_budget_row,test_walk_key,nx2/test_mesh_first_types.pyandnx2/test_texture_init_barrier.py. The 16th,test_pipeline_cache_save(a timing check:wait_until(play_saves, 3)), passed 2 of 2 when run alone on this branch, as on main; main's run timed outtest_fpx_stickyinstead, which passed here and passes alone on main. After the rebase onto mainaa95d35f, ata6ad0e0e: 35 checks, this PR's test and those that read its changed files, with g++ 13.3 and with clang 23.1.5: 33 pass; the 2 that fail fail on mainaa95d35ftoo (run alone there, both compilers):nx2/test_bc_storage_views.py(noclang++on this PC's path),nx2/test_texture_init_barrier.py(#628).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each new measured run (format indocs/tested-configurations.md).docs/tested/2026-11-15-readback-timeout-dukenheimer-t-4h95.md, the tester's run;scripts/tested_configurations.pyandtests/test_tested_configurations.pypass with it. The same file, byte for byte, is in #554: the two branches merge into main together without a conflict.mainand passing here.RX_VK_ONLY=1 RX_RELEASE=1 RX_TARGET=windowswith llvm-mingw 20261104 linkedlongshot_vk.dll(sha256 25117493...), and the same withoutRX_TARGETlinkedlongshot_vk.so; the changed file has no new warning under-Wall -Wextraon either platform againstmain. The tester's combined build linked for Linux (x86-64-v3, LLVM 23.1.5). After the rebase onto mainaa95d35f,gpu/vk/vk_texture.cppata6ad0e0echecked whole (-fsyntax-only -Wall -Wextra, clang 23.1.5 for Linux and llvm-mingw 20261104 for Windows): the same warnings as main aa95d35f's, none new.rb_cmdandrb_fenceuntil the GPU is past them, and the textures waiting on itFilled in: the full game-free suite on the combined branch, the Windows and Linux builds, and the test request T-4H95. Still work in progress until a Steam Deck run is in.
Test request T-4H95: pass
Built in a combined build of main
7ff5256ewith #550 to #558 merged in order, this PR at its head93c285b7; the comparison is main7ff5256e. Setup and method below.Result
that this PR changes never happened here: this is a non-regression result.
two builds' means are within 2 fps everywhere, in both directions: the PR is ahead in Goron City (+1.9) and behind in Gerudo
Valley (-1.6) (table below).
evicted, made againlines).written back into their memorylines a run (8,640 KB each, 12 to 64 ms), in all four runs.They differ only in timing:
out the same way on main: a quick series of pictures through that arrival on each build caught both at those stages
(attached, with the four runs' own pictures of it below).
shaper LUT" notices (6 a start; 12 in run 1). No reset, timeout, hang or fault.
Setup
kernel 7.2.7-valve2. Game Mode, docked, stock clocks.
7ff5256e, and the combined build: main + #550 to #558, merged in order, all clean. Both for x86-64-v3 with therelease play profile and LLVM 23.1.5. One test folder (warm caches) with the binaries swapped.
options file.
each area: Hyrule Field, Kakariko, the Market, Hyrule Field, the Lost Woods, Goron City, Death Mountain Trail, Zora's River, Gerudo
Valley, Kokiri Forest. About 11 minutes of play a run.
picture at each arrival. The log lines are from each run's
longshot_log.txt.fps by area (the mean of each build's two runs)
Thank you, dukenheimer, for the Steam Deck runs of T-4H95 (#553 (comment)).
Rebased onto main
53f96d78with no conflict: the test and fix commits are the same as before (git range-diff:ecacc39cis now9fc90117,93c285b7is now515a3fdf), and the run is added asdocs/tested/2026-11-15-readback-timeout-dukenheimer-t-4h95.mdin a third commit,cbe9b9ce(the same file as in #554; the two merge into main together without a conflict). The description now ticks the rebase anddocs/tested/boxes.Tests run again after the rebase:
tests/test_readback_timeout_reuse.pyfails at9fc90117(begin, later, lost) and passes at the tip, with g++ 13.3 and with clang 23.1.5. The fullpython3 -I -B tests/run_all.pyon this branch: 530 passed, 16 failed, 48 skipped; 15 of them fail on main53f96d78in the same run (named in the checklist), and the 16th,test_pipeline_cache_save, a timing check, passed 2 of 2 run alone, as on main.gpu/vk/vk_texture.cppchecked whole for Linux and Windows: the same warnings as main's, none new.Why the Deck result still holds: main changed nothing in the lines this PR changes. Since the tested base
7ff5256e, main changedgpu/vk/vk_texture.cppin one commit,b2911d33(weak Intel laptops):system_ram_bytes(), the low-VRAM mip rule (an integrated GPU with 16 GB of RAM or less now counts as small at Minimum's budgets, which includes the Deck at the preset the tester used), a lower squeeze line on integrated GPUs and a squeeze after the driver refuses pipelines for want of memory, intexture_frame, and the integrated GPU budget's log line. None of it is inreadback_begin,readback_submit_wait,flush_texturesor the eviction lines this PR keeps textures in. Those rules change how many textures the Deck keeps and evicts on main and on this PR alike, so the fps and eviction counts measured on7ff5256eare 7ff5256e's; the result itself, a non-regression run in which no read-back timed out, does not depend on them, and the fix stays proven by the game-free test.93c285b7c4cbe9b9ced6WIP: fix(vk): a read-back after a timed-out one waits for the GPU, and its textures are keptto fix(vk): a read-back after a timed-out one waits for the GPU, and its textures are keptcbe9b9ced6a6ad0e0e66Rebased onto
mainaa95d35f(heada6ad0e0e); the commits are re-authored with the forge's noreply address: no code change, the same patches bygit patch-id.Tests re-run on the new base, g++ 13.3 and clang 23.1.5:
test_readback_timeout_reusefails at the test-only commit08514708(3 violations; 10 calls and 2 waits after a device lost) and passes at3b245fa7; of the 35 checks that are this PR's or read its files, 33 pass and the 2 that fail fail on main too. The Deck run (T-4H95) stands; the description names the new commits.Read the diff and the surrounding
flush_textures/texture_framecode. The fix is sound:readback_beginno longer resets the pool or fence while the timed-out submit may still run, the later batches of a pass do not each wait 5 s, and the textures whose write-back is waiting stay resident instead of being evicted with no copy (thecontinuebeforedestroy_later/gone.insertleaves the entry intextures, so it is chosen again next pass). Device lost is handled (rb_lost), and thebusyexpression keeps a submit failure with no pending fence on the old path. The test cuts the real functions out of the source and checks begin/later/lost.It fits with #554: after a timeout the batch's textures are kept here, and 554 covers the case where one is evicted anyway while the copy may still run (an eviction with
e.flushfalse, or aflush_texturecaller whose timeout leaves its image alive but later evictable). The two merge cleanly in either order, as the description says.For #788: it overlaps this PR and 554 (same
rb_pending, same timeout loop) with an incompatible design (unbounded wait, then evict without the copy), so I asked for its read-back half to be dropped. Its frame-counted frees are independent of this PR.Nit, not blocking: a wait that fails with something other than success or timeout (for example out of host memory) latches
rb_lostfor the session; fine for a lost device, but the name suggests more than the code checks.a6ad0e0e66bb02fd2b36New commits pushed, approval review dismissed automatically according to repository settings
Rebased onto
main94abdd57(headbb02fd2b), with no conflict; the commit subjects are now Conventional Commits, which CircleCI's lint job checks for every commit of a pull request (#417), and the base has the CI's config, which a fork's pull request needs at its own head. No other change: each commit's change is the same as before bygit patch-id.Re-run on
94abdd57:test_readback_timeout_reusefails at the test-only commit3531263aand passes at the head, with g++ 13.3 and clang 23.1.5; of the 37 checks that are this PR's or read its files (g++ 13.3), 35 pass and the 2 that fail fail on main94abdd57too (test_bc_storage_views,test_texture_init_barrier). The description names the new commits.