fix(vk): a timed-out read-back's source images are kept until the GPU is past the copy #554
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!554
Loading…
Reference in a new issue
No description provided.
Delete branch "zeldaootr/Longshot:readback-timeout-keeps-source-textures"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux x86-64 (game-free); Steam Deck OLED, SteamOS 3.9.5, RADV (a tester's run) · Needs: none
Why
When a texture read-back timed out (the GPU stuck for 5 s), the textures it copied from were freed, or handed to a new render target, while the copy could still run on the GPU: a read from freed memory, which can corrupt a picture or crash the driver. The tracker has no issue for it (searched).
How to reproduce
Seen on: the source, found by reading
gpu/vk/vk_texture.cpp; a 5 s GPU stall cannot be made on demand in the game, so the game-free test reproduces it against stub Vulkan, VMA and pool callsSettings and switches: defaults
Steps: at the test-only commit
1b907d6b(mainaa95d35fplus the test), runpython3 -I -B tests/test_readback_timeout_source.py --cxx g++Expected: the image a timed-out read-back copies from outlives the copy, then goes
Actual: on main the source image is destroyed (a texture) or pooled (a render target) at completed 10, before the GPU is past the copy, while its staging buffer waits for 11
What changed
After a timed-out read-back, the textures it copies from now live until the GPU is past that copy, the same rule its staging buffers already followed. Nothing changes when no read-back times out.
flush_textures()ingpu/vk/vk_texture.cpp, on a timed-out read-back, raises each source texture'slast_valueto the submission after it (rec_value + 1), so the eviction'sdestroy_laterkeeps the image until the GPU is past the copy.tests/test_readback_timeout_source.pycuts the read-back state and functions,Dead,Flush,flush_textures,destroy_laterandtexture_frame's loop over the dead list out of the source and runs them against stub Vulkan, VMA and pool calls: the image must outlive the copy, then go.Measurements
Verdict: pass. The fix is proven by the game-free test (2 of 2 cases fail on main, 0 with this PR). In the game it is a non-regression result: on a tester's Steam Deck OLED the timed-out read-back never happened (no
did not complete in timeline in 4 runs), and fps, faults and texture evictions were the same as main within main's own run-to-run spread.Hardware: Steam Deck OLED, AMD Custom APU 0932 (RADV VANGOGH), 16 GB, Mesa 26.2.0-devel git-5c65d4fc31, SteamOS 3.9.5 (build 20261023.100), kernel 7.2.7-valve2, Game Mode, docked, stock clocks (dukenheimer's run, test request T-GQD8, the same runs as T-4H95 for #553); game-free: Linux x86-64 server, no GPU or game
Scene: a route through the control bridge: Continue into Kokiri Forest, then 10 debug teleports with about 45 s in each area (Hyrule Field, Kakariko, the Market, Hyrule Field, the Lost Woods, Goron City, Death Mountain Trail, Zora's River, Gerudo Valley, Kokiri Forest), about 11 minutes a run; the Minimum (4GB) preset (render 50% of 1280x720, a 2.2 GB VRAM budget, Handheld profile), the game's own upscaler, frame generation off
Method: fps from the bridge's status every 5 s, averaged over the time in play; the log lines from each run's
longshot_log.txt; the kernel's amdgpu lines (journalctl -k, each start to its end); one picture at each arrival. Four runs, interleaved (this PR, main, main, this PR), one test folder with warm caches and the binaries swapped; the median of each build's two runs. Game-free:python3 -I -B tests/test_readback_timeout_source.py --cxx g++at1b907d6band at9287f171(rebased onto mainaa95d35f), with g++ 13.3 and with clang 23.1.5; before that, at4cf946a1andfa5b6da6on main53f96d78, and before the first rebase atf5bfead9anda235b180, twice each, alternatingHow to reproduce: build main
7ff5256eand the combined build (main7ff5256ewith #550 to #558 merged in order), both x86-64-v3 with the release play profile and LLVM 23.1.5; choose the Minimum (4GB) preset, then drive the route above; for the game-free check, run the test command at each commit7ff5256e6c76d0eb(in the combined build)did not complete in timelinesevicted, made againlinesThe Deck ran this PR at
6c76d0eb. Its fix is now9287f171(fa5b6da6on main53f96d78, anda235b180before that, the same change bygit patch-id), whose code is that of6c76d0eb: it differs only intests/test_readback_timeout_source.py(5 lines that check the fixture's precondition), and that test passes with LLVM 23.1.5. By area, the two builds' means are within 2 fps everywhere, in both directions (Goron City +1.9, Gerudo Valley -1.6); main's own two runs differ by more than the builds do. The kernel's amdgpu lines were only the display's[drm] Losing delta precision while programming shaper LUTnotices. The textures were under pressure in every run: 2.7 to 2.8 GB against a 2,560 MB texture budget, VRAM in use peaking at 4.5 to 5.1 GB of the 5.3 to 6.1 GB allowed, about 4.3 GB swapped, MemAvailable down to 281-471 MB. The tester's report: #554 (comment)aa95d35f(with the test,1b907d6b)9287f171Flags and settings
None.
Testing
Tested: Linux x86-64, game-free. The new test fails on the test-only commit
1b907d6band passes at9287f171, this branch rebased onto mainaa95d35f, with g++ 13.3 and with clang 23.1.5 (before, it did the same at4cf946a1andfa5b6da6on main53f96d78, and atf5bfead9anda235b180, twice each, alternating) (python3 -I -B tests/test_readback_timeout_source.py --cxx g++); it also builds and passes with clang 23.1.5 under its own-Wall -Wextra -Werrorat both commits. This PR's own tests and the checks its fix commit edits ran at both commits, and once more onland/integration27553c83 (main7ff5256ewith this PR and twelve other fixes merged), where the thirteen fixes' tests and edited checks pass, 16 of 16; the fullpython3 -I -B tests/run_all.pyran on that branch too (see the checklist).In the game: dukenheimer ran test request T-GQD8 on a Steam Deck OLED (RADV VANGOGH, Mesa 26.2.0-devel, SteamOS 3.9.5, Game Mode, docked) and reported a pass on 2026-11-15 at 03:12 UTC (#554 (comment)). These are the same four runs as T-4H95 for #553: the combined build of main
7ff5256ewith #550 to #558 merged in order, this PR at6c76d0eb, against main7ff5256e, the 11-minute route above, interleaved. Nodid not complete in timeline in any run, so the timed-out read-back whose source images this PR keeps never happened there: it is a non-regression run, not a run of the fix. No crash, device lost, device fault or GPU stall; no GPU reset, ring timeout, hang or page fault in the kernel log; no garbled or rainbow textures in the 44 arrival pictures (the sheet is on #553). fps was the same within main's own spread.The branch has been rebased since the tester's run, onto main
aa95d35f, with its commits re-authored under the forge's noreply address; each commit's change is identical to before bygit patch-id, so the commits named above are the ones tested and the result stands for this head (68606017).Not tested: A real 5 s GPU stall, in the game or on any GPU: the fixed path is proven only against the stubs, since a stall that long cannot be made on demand. The head in the game (its code is that of the
6c76d0ebthe Deck ran). Windows in the game (the renderer was built for Windows, see the checklist).Must be tested: None
Risks
What could break: This is a separate defect on main from the read-back reuse one (#553), and touches the lines next to that fix in
flush_textures(the timed-out branch): the two merge without a conflict (git's automatic merge ofgpu/vk/vk_texture.cpp), and onland/integrationthey compose, with both tests passing there (a texture kept for a later pass is not evicted at all). After a timeout, the source images now live one submission longer.Turning it off: revert the commit
Compatibility: Nothing changes
Checklist
Everyone
type(scope): description) of the status line's type.main. Onto main94abdd57(head77a84d96) with no conflict, with the commit subjects made Conventional Commits, which CI's lint job now checks for every commit, on a base that has CI's config (#417); each commit's change is the same as before bygit patch-id. The commits named in this description are from before this rebase:1b907d6bis now6a5dfdf7,9287f171is nowead1b345,68606017is now77a84d96. Re-run on94abdd57:test_readback_timeout_sourcefails at the test-only commit6a5dfdf7and passes at the head, with g++ 13.3 and clang 23.1.5; of the 37 checks that are this PR's or read its files (g++ 13.3), 35 pass and the 2 that fail fail on main94abdd57too (test_bc_storage_views,test_texture_init_barrier).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Run on this branch at0dfefc41on main53f96d78, before this rebase (68606017now, the same patches bygit patch-id; the Vulkan and SPIR-V headers from the CPM cache, 8 at once, on a loaded Linux x86-64 server): 530 passed, 16 failed, 48 skipped. 15 of them fail on main53f96d78in the same run:test_appimage_player_build,test_bake_portability,test_cpm_discovery,test_crash_dialog,test_env_on,test_lift_cache,test_lift_cleanup_error,test_make_game_cache,test_pipeline_tour,test_readback_late,test_vk_pgo,test_vram_budget_row,test_walk_key,nx2/test_mesh_first_types.pyandnx2/test_texture_init_barrier.py. The 16th,test_pipeline_cache_save(a timing check:wait_until(play_saves, 3)), passed 2 of 2 when run alone on this branch, as on main; main's run timed outtest_fpx_stickyinstead, which passed here and passes alone on main. After the rebase onto mainaa95d35f, at68606017: 35 checks, this PR's test and those that read its changed files, with g++ 13.3 and with clang 23.1.5: 33 pass; the 2 that fail fail on mainaa95d35ftoo (run alone there, both compilers):nx2/test_bc_storage_views.py(noclang++on this PC's path),nx2/test_texture_init_barrier.py(#628).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each new measured run (format indocs/tested-configurations.md).docs/tested/2026-11-15-readback-timeout-dukenheimer-t-4h95.md, the tester's run;scripts/tested_configurations.pyandtests/test_tested_configurations.pypass with it. The same file, byte for byte, is in #553: the two branches merge into main together without a conflict.mainand passing here.RX_VK_ONLY=1 RX_RELEASE=1 RX_TARGET=windowswith llvm-mingw 20261104 linkedlongshot_vk.dll(sha256 25117493...), and the same withoutRX_TARGETlinkedlongshot_vk.so; the changed file has no new warning under-Wall -Wextraon either platform againstmain. The tester's combined build linked for Linux (x86-64-v3, LLVM 23.1.5). After the rebase onto mainaa95d35f,gpu/vk/vk_texture.cppat68606017checked whole (-fsyntax-only -Wall -Wextra, clang 23.1.5 for Linux and llvm-mingw 20261104 for Windows): the same warnings as main aa95d35f's, none new.Filled in: the full game-free suite on the combined branch, the Windows and Linux builds, and the test request T-GQD8. Still work in progress until a Steam Deck run is in.
Thank you for catching this. The fixture's destroy_gen is now used (the test checks that the eviction really retired the texture before it looks at the dead list), so it compiles with clang 23.1.5 under the test's own -Wall -Wextra -Werror; checked here with that clang at both commits. The fix commit is unchanged. #553's test had no such warning.
6c76d0eb90a235b18035Test request T-GQD8: pass
Built in a combined build of main
7ff5256ewith #550 to #558 merged in order, this PR at6c76d0eb. The comparison is main7ff5256e.This PR's head is now
a235b180, whose code is the same: it differs from6c76d0ebonly intests/test_readback_timeout_source.py(5 lines added). That test passes with LLVM 23.1.5. The same runs as for #553 (T-4H95). Setup and method below.
Result
journalctl -k | grep -i amdgpu, from each start to its end): the lines were only the display's[drm] Losing delta precision while programming shaper LUTnotices, 6 a start (12 in run 1). No GPU reset, ring timeout, hang orpage fault in any run.
whose source images this PR keeps never happened here: this is a non-regression result.
two builds' means are within 2 fps everywhere, in both directions (Goron City +1.9, Gerudo Valley -1.6).
evicted, made againlines).arrival still fading in, Navi's place, and when the area title showed (it dissolves in and out, the same on main).
Setup
kernel 7.2.7-valve2. Game Mode, docked, stock clocks.
7ff5256e, and the combined build: main + #550 to #558, merged in order, all clean. Both for x86-64-v3 with therelease play profile and LLVM 23.1.5. One test folder (warm caches) with the binaries swapped.
options file.
each area: Hyrule Field, Kakariko, the Market, Hyrule Field, the Lost Woods, Goron City, Death Mountain Trail, Zora's River, Gerudo
Valley, Kokiri Forest. About 11 minutes of play a run; every teleport was taken on the first try.
longshot_log.txt.Thank you, dukenheimer, for T-GQD8 on the Steam Deck (#554 (comment)).
Rebased onto main
53f96d78with no conflict: the test and fix commits are the same as before (git range-diff:f5bfead9is now4cf946a1,a235b180is nowfa5b6da6), and the run is added asdocs/tested/2026-11-15-readback-timeout-dukenheimer-t-4h95.mdin a third commit,0dfefc41(the same file as in #553; the two merge into main together without a conflict). The description now ticks the rebase anddocs/tested/boxes.Tests run again after the rebase:
tests/test_readback_timeout_source.pyfails at4cf946a1(texture, render target) and passes at the tip, with g++ 13.3 and with clang 23.1.5. The fullpython3 -I -B tests/run_all.pyon this branch: 530 passed, 16 failed, 48 skipped; 15 of them fail on main53f96d78in the same run (named in the checklist), and the 16th,test_pipeline_cache_save, a timing check, passed 2 of 2 run alone, as on main.gpu/vk/vk_texture.cppchecked whole for Linux and Windows: the same warnings as main's, none new.Why the Deck result still holds: main changed nothing in the lines this PR changes, and the Deck's
6c76d0ebhas the same code as the head (a235b180added 5 lines to the test only). Since the tested base7ff5256e, main changedgpu/vk/vk_texture.cppin one commit,b2911d33(weak Intel laptops):system_ram_bytes(), the low-VRAM mip rule (an integrated GPU with 16 GB of RAM or less now counts as small at Minimum's budgets, which includes the Deck at the preset the tester used), a lower squeeze line on integrated GPUs and a squeeze after the driver refuses pipelines for want of memory, intexture_frame, and the integrated GPU budget's log line. None of it is inflush_textures' timed-out branch,destroy_lateror the dead list's loop this PR relies on. Those rules change how many textures the Deck keeps and evicts on main and on this PR alike, so the fps and eviction counts measured on7ff5256eare 7ff5256e's; the result itself, a non-regression run in which no read-back timed out, does not depend on them, and the fix stays proven by the game-free test.a235b180350dfefc41baWIP: fix(vk): a timed-out read-back's source images are kept until the GPU is past the copyto fix(vk): a timed-out read-back's source images are kept until the GPU is past the copy0dfefc41ba686060178dRebased onto
mainaa95d35f(head68606017); the commits are re-authored with the forge's noreply address: no code change, the same patches bygit patch-id.Tests re-run on the new base, g++ 13.3 and clang 23.1.5:
test_readback_timeout_sourcefails at the test-only commit1b907d6b(texture, target) and passes at9287f171; of the 35 checks that are this PR's or read its files, 33 pass and the 2 that fail fail on main too. The Deck run (T-GQD8) stands; the description names the new commits.Read the diff and the surrounding code. When a batch's 5 s wait runs out, the staging buffers already waited for
rec_value + 1, but the source images only waited for their oldlast_value, so an eviction could destroy or pool them under the copy. Raising each texture'slast_valuetorec_value + 1gives them the same lifetime.Flush::tis always set where the batch is built, so the unguarded deref is fine, and thelast_value > completedchecks in the eviction pick just keep the texture off the candidate list until then.It fits with #553 (which keeps the batch's textures resident while the read-back is pending; this PR still matters for an eviction without a write-back and for
texture_flush_to_guest's single flushes). The two merge cleanly in either order. The test cuts the realflush_textures,destroy_laterand the dead-list loop out of the source and checks the texture and render-target cases.#788 makes the same
last_valuechange in the same lines plus a conflicting read-back design; I asked for that part to be dropped there.686060178d77a84d96ddNew commits pushed, approval review dismissed automatically according to repository settings
Rebased onto
main94abdd57(head77a84d96), with no conflict; the commit subjects are now Conventional Commits, which CircleCI's lint job checks for every commit of a pull request (#417), and the base has the CI's config, which a fork's pull request needs at its own head. No other change: each commit's change is the same as before bygit patch-id.Re-run on
94abdd57:test_readback_timeout_sourcefails at the test-only commit6a5dfdf7and passes at the head, with g++ 13.3 and clang 23.1.5; of the 37 checks that are this PR's or read its files (g++ 13.3), 35 pass and the 2 that fail fail on main94abdd57too (test_bc_storage_views,test_texture_init_barrier). The description names the new commits.