perf(vk): RX_VK_READBACK_EARLY, a read of GPU results written in place waits for its writer's part, not the whole submission #303

Open

dukenheimer wants to merge 8 commits from dukenheimer/Longshot:opus/early-readback into main

pull from: dukenheimer/Longshot:opus/early-readback

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 10 Commits 8 Files changed 16 +832 -18

dukenheimer commented 2026-11-13 20:52:11 +00:00

First-time contributor

• Copy link
• Copy Markdown

Why

On the Steam Deck the game's job threads spent ~18 ms of every frame sleep-polling for a GPU result (the culling camera block at +0x3980) that the GPU had finished computing long before: the read waited for the end of the whole 22-26 ms submission, not for the dispatch that wrote it. This lets such a read go on once its writer is done, opt-in for now.

What changed

• vk (diagnostics): RX_VK_LOG_RB_WRITERS=1 (with RX_VK_LOG_READBACK_WAITS=1): for each read of GPU results that waits, what wrote its pages (the op, its program, its part and draw in the submission), when the last writer's part was done into the wait, and how many bytes changed after it; a summary by size every 3000 reads. memory_writer_note, memory_count_work, memory_rb_writers in gpu/vk/vk_memory.cpp; rb_diag_* and the call in invalidate in gpu/vk/vk_replay.cpp. It is how the writers below were found. Its wait comes after RX_VK_READBACK_LATE's rule and only for a read that waits; a submission's span is closed on every path that finishes its recording, a burst's end included (rb_span_close).
• vk: RX_VK_READBACK_EARLY=1 (opt-in), for pages the GPU writes in place (no VRAM mirror: integrated GPUs, the Steam Deck):
  • once a range's read has waited (memory_rb_waited, at most 64 ranges), every GPU write in place to it keeps its submission and the part it was recorded in, whatever its size (early_in_place_write, called from note_direct_write);
  • a later read waits for that part (memory_in_place_writer_part; the newest of it and a mirror's part), when every page of the range that the read's submission wrote in place has that write seen, else for the whole submission as before (a range widened over a page written before, or a page tracked later); a write to any byte of a page a waited range reaches is kept for the page, and only a write of the waited bytes ends its part (invalidate, before the usual choice, which it replaces only when it waited);
  • such a write ends its part right after it (split_part(cmd, true) in replay), at least RX_VK_READBACK_EARLY_GAP draws (16) into the part, so writers in a row end one part; maybe_split is split_part(cmd, false), unchanged;
  • a draw of a program that writes storage buffers builds its descriptors (same_desc in gpu/vk/vk_draw.cpp), so each such write is noted with its part;
  • left off with RX_VK_PARALLEL ≥ 2 (logged): a parallel replay's segments are numbered as parts but never put on the recordings' timeline, so a read could wait for a part that never comes; ballotstomper also found the part before a command buffer's segments written back late there (team room);
  • with RX_VK_READBACK_LATE (#238, on main since): a range its rule names is read late as before, with no wait, and the early wait applies only to the other reads. Both switches act on the same waits in invalidate; whether they stay two switches, become one, or this one closes is asked on #238 (#238 (comment)) and is being measured on the Deck (off / newest / this / both, one build).
• tests: tests/test_readback_early.py (new): the tracking cut from vk_memory.cpp and run with the switch on, off and with RX_VK_PARALLEL=3; invalidate's early wait cut out and run against stubs; the wiring in replay, split_part, run_submit, note_direct_write and the descriptor reuse. a range RX_VK_READBACK_LATE names takes no early wait. tests/test_parallel_small_cb.py and tests/test_split_ub_flush.py follow maybe_split into split_part; tests/direct_buffer_writes_test.cpp stubs early_in_place_write; tests/test_readback_part.py stubs the early wait (off) in its cut of invalidate; tests/test_late_fix_build.py declares the writers' hook in wb_pages_note. From the review: a range widened over a page the same submission wrote before, two ranges in one page, a write really over 1 MiB, and a burst's span (burst_end and the spans cut out and run); tests/test_readback_part.py runs the late rule with the diagnostics on.
• docs: docs/development.md has both switches and the order with RX_VK_READBACK_LATE; four run files in docs/tested/ (the Deck runs below).

No translator file changes: the shader caches stay valid.

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
RX_VK_READBACK_EARLY=1	A read of GPU results written in place waits for its writer's part, which ends right after the writer	off	on for the Steam Deck (a follow-up in the Deck profile, #225), unless the A/B against RX_VK_READBACK_LATE=newest says otherwise; off elsewhere until measured	Steam Deck (+3.0 real fps at stock clocks, +2.2 at GPU 1600 MHz, interleaved)	dGPUs with VRAM mirrors (their pages are written back by part already: nearly inert); RX_VK_PARALLEL ≥ 2 (left off)	other APUs (ROG Ally, Legion Go, Ryzen laptops), Windows iGPUs
RX_VK_READBACK_EARLY_GAP=<draws>	The least draws into a part before an early end	16	keep	-	-	-
RX_VK_LOG_RB_WRITERS=1	Diagnostics: the readbacks' writers (above)	off	for diagnosis only: it waits for the writer's part first, timed	-	-	-

Measurements

Steam Deck with the review fixes (52306c0c), the player's build, two runs per arm: off 11.5 / 11.1 real fps, on 13.5 / 13.2 (+2.05, +18%), no read falling back to its whole submission. Before them, current main with #238, two usable runs per arm: off 10.8 / 11.4 real fps, RX_VK_READBACK_EARLY=1 12.6 / 12.8 (+1.6, +14% on the means), against #238's RX_VK_READBACK_LATE=newest 12.0 / 11.6. Both switches together: 12.7 / 13.0, level with this one alone. The earlier tables below (+2.3 to +3.0) have one usable off run on the head, or on runs 16.8-19.5 wide: supporting only. RTX 5090: idle at 98cfaf0, which is before #250's sparse mirrors; with them a few pages are written in place on a discrete card too, and that path is not measured there.

main: cf605c85 (2026-11-13, with #238, #250, #252)
this PR: 9e50672 (the code of 025d8e98) in the Deck build 22253fd = main cf605c85 + #225 Deck profile (d9b3551) + #223 (52a61f4) + this PR; one build for every arm. Measured before the review fixes (52306c0c: a part known only when every write of the submission to the range's pages was seen; the diagnostics' wait after the late rule; the diagnostics' span closed at a burst's end); the next Deck A/B re-measures off against on with them.
Baseline: A: this PR @ 9e50672, nothing set (with no switch set this PR's code is idle: main's path; one build isolates the switches from #225 and #223, which the Deck build needs).
Hardware: Steam Deck OLED (AMD Custom GPU 0932, RADV VANGOGH, Mesa 26.2.0-devel), 4C/8T Zen 2, 16 GB, SteamOS 3.9.5, stock clocks.
Settings: the player's own: handheld profile, the game drawn at 1536x864 (a crop of 2560x1440) at 100% render, a 1280x720 FIFO swapchain, frame generation 3x; no longshot_env.txt; RX_VK_LOG_READBACK_WAITS=1 in every arm.
Scene and method: Kokiri Forest, the moving route (walk and camera pans, 125 s, measured 30-110 s), warm, interleaved A B C D D C B A; real fps from the harness's frame log (the game's frames; "shown" with frame generation); 15 route shots and 4 at the start a run, compared at the same points; one unmeasured flash-detector run each for A, B and D (RX_VK_FLASH=1).

Steam Deck OLED, stock	A: this PR @ 9e50672, nothing set	B: this PR @ 9e50672, RX_VK_READBACK_LATE=newest	C: this PR @ 9e50672, RX_VK_READBACK_EARLY=1	D: this PR @ 9e50672, both
real fps (runs)	10.8, 11.4	12.0, 11.6	12.6, 12.8	12.7, 13.0
mean, against A	11.1	11.8 (+0.7)	12.7 (+1.6, +14%)	12.85 (+1.75)
shown fps	27.5, 29.1	31.8, 29.9	34.0, 35.3	34.2, 35.2
p50 / p95 / p99 ms	16.9 / 127.6, 125.2 / 180.9, 174.4	16.8 / 114.7, 121.9 / 163.4, 185.4	16.8-16.9 / 106.3, 80.7 / 161.9, 156.2	16.8 / 102.2, 98.7 / 155.8, 156.3
frames over 100 ms	203, 207	165, 179	147, 125	148, 138
a readback that waits (of 3000)	268 x 23.3 ms, 221 x 23.6	303 x 14.9, 408 x 13.3	1023 x 1.55, 1029 x 1.36	999 x 0.88, 1003 x 1.12
captures waiting for late pages (of 600)	1, 3	0, 10	92, 90	82, 91
replay memory sync, ms a frame	2.3, 1.9	1.2, 2.3	19.6, 21.6	14.8, 20.2
GPU busy (route average)	71%	74-75%	80-81%	82-85%

• This switch is the faster one here, and its values are exact: a read waits for its writer's part (1.4-1.6 ms) rather than being read without a wait. newest lets the newest submission's ranges go late but every other read still waits for its whole submission (13-15 ms each).
• Both together add nothing measurable over this switch alone (12.85 against 12.7, within the spread).
• What this switch costs: the game, let go at the writer's part, writes the camera block's page again while the submission runs, and the next submit's capture waits for the rest of it (82-92 of 600 captures, memory sync ~20 ms a frame against ~2). A follow-up makes those captures wait for the writer's part too; it is being measured on the Deck (off / on / on with it). newest does not have this cost.
• Picture: the route shots match at the same points in every arm (the only dark frames are the camera inside foliage, in A and D alike); no white frames. Flash detector, one run each: A 1 (Navi's light next to the camera), B 0, D 2 (Navi's light again; a Kokiri boy drawn white for one present, Navi at his head the present before; one run cannot say whether D's frame-old values played a part). C had no flash run.
• The GPU takes 50-61 ms a frame at these settings in every arm, so the frame is close to GPU-bound; the switches fill the GPU (71% busy -> 80-85%).

With the review fixes, on the player's build (deck-user-1113b @ 832a8a12 = main cf605c85 + #227 with #225 + #223 + #232 + this PR @ 025d8e98, plus the review fixes 52306c0c and a local follow-up whose switch, RX_VK_READBACK_EARLY_CAPTURE, is off in both columns here: its code then only calls an empty hook; longshot_vk.so 4756e5e7, one build; the same settings and route, interleaved A B (C C) B A, run opusab):

Steam Deck OLED, stock	the player's build + this PR @ 52306c0c, RX_VK_READBACK_EARLY=0	the player's build + this PR @ 52306c0c, RX_VK_READBACK_EARLY=1	Δ
real fps (runs)	11.5, 11.1	13.5, 13.2	+2.05 on the means (+18%)
shown fps	29.9, 27.9	38.9, 35.6
p95 / p99 ms	119.0, 127.9 / 171.3, 175.6	61.0, 89.5 / 132.2, 150.4
frames over 100 ms	178, 216	82, 131
a readback that waits (of 3000)	277 x 22.4 ms, 236 x 27.3	904 x 1.1, 1002 x 1.4; 0 for their whole submission
captures waiting for late pages (of 600)	2, 1	69, 84
GPU busy	73%, 74%	85%, 83%

The follow-up (C, on in two runs between them) made those captures wait for their late pages' writers' part: it engaged on every one (89, 92) but each still waited ~20 ms, the writers' parts ending about where the submissions do, and ran 12.7 / 12.8: no gain, so it is not proposed. Route shots normal in every arm.

Earlier runs (supporting, before main had #238):

main: 514591a (2026-11-13)
this PR: opus/early-readback @ 5b31bc0 (the Deck build: ae1cffd = main 514591a + #225 Deck profile + this PR @ 5b31bc0 + #252 movie input limit); the PC pair @ 98cfaf0
Baseline: this PR @ 5b31bc0, RX_VK_READBACK_EARLY unset. With the switch unset this change's code is idle, which is main's path; one build for both arms isolates the switch from #225 and #252, which the Deck build needs and main does not have. 98cfaf0 (after the Deck build) only keeps the switch idle where no page is written in place: on the Deck every page is, so its path there is 5b31bc0's.
Hardware: Steam Deck OLED (AMD Custom GPU 0932, RADV VANGOGH, Mesa 26.2.0-devel git-5c65d4fc31), 4C/8T Zen 2, 16 GB (swapping in every run), SteamOS; docked to 3440x1440, game output 2048x1152, FSR 3.1 at ~50%, frame generation 2x (the Deck profile). PC: RTX 5090, Windows 11, the PC runner's bench folder.
Scene: Kokiri Forest, the moving route (walk and camera pans, 125 s, measured 30-110 s); the Deck: the user's options, no longshot_env.txt; the PC: M1152 (2048x1152 window) and MP165 (5120x2160 at 165 Hz)
Method: the Deck agent's A/B harness and the PC runner's bench; runs interleaved off / on / off / on after a cold start that prepared the shaders; RX_VK_LOG_READBACK_WAITS=1 in both arms; fps and frame times from the harness's frame log ("real": the game's frames; "shown": with frame generation); screenshots through the route; flashes and luma on the PC
How to reproduce: RX_VK_READBACK_EARLY=1 in longshot_env.txt, Kokiri Forest; the log's readbacks waited for their writer's part line every 3000 reads

Steam Deck OLED, stock clocks	this PR @ 5b31bc0, RX_VK_READBACK_EARLY unset	this PR @ 5b31bc0, RX_VK_READBACK_EARLY=1	Δ
real fps (runs)	13.9; 11.0 (swapped heavily: 78,710 pages in, against 45-50k in the others)	16.5, 16.0 (median 16.25)	+2.3 against 13.9 (+17%); +3.8 against the median of both (12.45)
shown fps	25.9; 17.6	31.6, 31.3
p50 / p95 / p99 frame ms	25.6 / 84.5 / 164.6; 34.9 / 160.9 / 352.6	24.2-24.7 / 71.0-73.3 / 120.1-136.5
frames over 100 ms	77; 157	70, 44
draws a frame	1,509; 1,815	1,378, 1,576
readback waits (a log line of 3000 reads)	193 x 21.6 ms, 246 x 16.7 ms	964 x 1.01 ms, 988 x 1.66 ms

The cold start before them (switch unset, shaders prepared in it) ran 15.0 real fps on a lighter stretch (1,264 draws a frame); not counted. All four runs stayed on the outdoor route; the picture is normal in every shot. readbacks waited for their writer's part (RX_VK_READBACK_EARLY): 749-988, 0.98-2.29 ms each; 0 for their whole submission (writer not known); 800-863 parts ended after a readback's writer.

RTX 5090, Windows	this PR @ 98cfaf0, RX_VK_READBACK_EARLY unset	this PR @ 98cfaf0, RX_VK_READBACK_EARLY=1	Δ
M1152 fps (runs)	74.80, 73.72	74.86, 73.22	+0.06, -0.50 (noise)
M1152 p99	19.99, 20.39 ms	19.54, 19.79 ms
MP165 fps (runs)	68.29, 66.54	67.96, 71.58	-0.33, +5.04 (noise)
MP165 p99	23.32, 23.66 ms	22.71, 20.95 ms
replay submit / descriptors, ms a frame	6.10-9.47 / 1.08-1.44	6.11-9.48 / 1.07-1.37
flashes per 1,000 frames	38.1, 41.1 (M1152); 20.5, 14.8 (MP165)	50.6, 34.4; 15.0, 10.8	noise

On the PC the switch logged 0, 0.00 ms each; 0 for their whole submission (writer not known); 0 parts ended in every run: idle, as it should be with VRAM mirrors. The first PC pair, on 8f6ac8f before the idle check, cost ~2 fps at M1152 (73.66 -> 71.71) from bookkeeping that never acted; 98cfaf0 removed it (comment of 2026-11-13).

Supporting: the Steam Deck user build, interleaved (deck-longshot-user 92640867 = main e7ed2f7a + #225 + memory pressure + #227 + #232, with this change as 996e76b, pushed as opus/early-readback-deck-measured; one build for both arms):

stock clocks	user build, switch unset @ 996e76b	user build, RX_VK_READBACK_EARLY=1 @ 996e76b	Δ
real fps (runs)	15.5, 14.8 (median 15.2)	16.8, 19.5 (median 18.2)	+3.0 (+20%)
p95 frame ms	74.4, 79.6	73.5, 49.4
frames over 100 ms	59, 72	51, 31

GPU at 1600 MHz	user build, switch unset @ 996e76b	user build, RX_VK_READBACK_EARLY=1 @ 996e76b	Δ
real fps (runs)	19.8 (the other run left the route: void)	21.1, 22.8 (median 22.0)	+2.2
p95 frame ms	45.5	46.2, 41.1

Earlier single runs (with HOS_CHAIN), why it helps:

	switch off (chain-lu)	RX_VK_READBACK_EARLY=1 (EA)
real fps	14.7	17.1
HOS_CHAIN critical path a frame	72.4 ms	57.6 ms
of it, the job threads' poll at lr 0x8009a478	18.0 ms	5.3 ms
0x3980 read wait, each	18.9 ms	1.7 ms
replay submit / of it memory sync, ms a frame	27.6 / 1.4	35.9 / 7.1

The frame's path is now the render thread (the inline replay and the game's render work, 32 of 58 ms), then the main thread.

(this build + RX_VK_LOG_RB_WRITERS, stock)
vk: readback writers, size 0x3980: 68 waits, 15.54 ms each; 68 with the last writer's part known: done 4.49 ms into their 15.54 ms (11.04 ms of each wait after it); bytes the wait changed: 328.0 each, 0.0 of them after the part (0 of 68 waits); last writers: dispatch fd958faab99f20a2 x68
vk: readback writers 0x1139483000 +0x3980: last dispatch fd958faab99f20a2 (part 62659, draw 744 of 1384 in submission 28043), first dispatch 7984161db3e053b5 (part 62657, draw 580), 4 pages; the submission's parts 62653..62664; its last writer's part done 0.67 ms into the wait of 12.48; bytes changed 0 by then, 0 after

What the diagnostic run showed (236 waits for +0x3980, 23 for +0x8b040/+0x9b040): the culling camera block (+0x3980, 1.1 reads a frame) is written by compute dispatch fd958faab99f20a2, its first writer 7984161db3e053b5 one or two parts before, at the median 49% of its submission; on average its part was done 6.4 ms into an 18.9 ms wait, and in no wait did a byte change after it. The +0x8b040/+0x9b040 rings are written by dispatches at ~43% (when +0x3980's wait was skipped outright in an earlier test, these took its place with 5.5-12 ms waits: here their writer's part ends early too).

The cost: each early part end is a part's end (the uniform updates, the write-back, a barrier, a submit) and the next one's start; replay submit time rose 8.3 ms a frame at stock (memory sync 1.4 -> 7.1), but only ~1 ms of it on the render thread's path. Most of it is not the part ends: the capture before each game submit waits for its "late pages" (the game writes the camera block again while the submission still runs), for the whole submission (36-71 of 600 takes, ~8 ms each). A follow-up PR makes those wait for their writers' part (being measured on the Deck).

Testing

Tested:

• python -I -B tests/run_all.py @ 7da13e56 (the review fixes; Linux also @ 52306c0c) and @ 025d8e9 on main cf605c8 (with #238, #250, #162), the same failures at each, Linux (Ubuntu 22.04 container, clang 23, the CPM Vulkan headers) and Windows (llvm-mingw): exactly main's failures. Linux 7, each failing on main @ cf605c8 in the same container (test_cpp_preflight, test_dev_packs, test_framegen_resources, test_module_ids, test_pad_layout, test_prof_clock, the byte-compile without .git), 523 passed against main's 522. Windows 7 (test_fpx_sticky, test_bake_portability, test_module_ids, test_pad_layout, test_spawn_emit, test_tested_configurations (its CRLF pipe check), test_oot_state_tap (named pipe)), each also failing on main here; 478 passed. On the way, main's newer test_split_ub_flush and test_late_fix_build cut maybe_split and wb_pages_note; both follow them in this PR. The Linux renderer (RX_VK_ONLY) builds clean.
• tests/test_readback_early.py with gcc 13 (WSL) and clang 23 (container, llvm-mingw).
• In the game: Steam Deck OLED, Kokiri Forest moving route: this PR on main (4 runs, 2 with the switch), the user build's interleaved confirmation (8 runs, 4 with the switch) and two earlier single runs (stock with HOS_CHAIN; GPU at 1600 MHz); 15 screenshots along the route and up to 4 in the conversation each, the switch's 8 runs checked: exposure, fog, lighting, HUD and minimap normal, no flashes, white or dark frames, no glowing objects, the conversation's camera right.
• RTX 5090, Windows: the switch off and on @ 98cfaf0, M1152 and MP165, 2 interleaved rounds (above): idle, no fps or p99 change, flashes within main's range, 0 bad log lines (no screenshots in that pair: a bench bug left them off on both arms).

Not tested:

• This head on a discrete GPU: the RTX 5090 pair was @ 98cfaf0, before #250's sparse mirrors made a few pages written in place there too.
• Other APUs and Windows iGPUs, where pools may be unmirrored too.
• Other areas than Kokiri Forest; long sessions.

Needs other testers:

• ROG Ally / Legion Go / a Ryzen laptop on Linux: Kokiri Forest with the switch, expecting the readbacks waited for their writer's part line and fps not lower (T-346Y; claimable once a maintainer adds the testing wanted label)

Screenshots

Nothing changes on screen by design; the picture check is under Testing (38 shots on the Deck, normal).

Risk and rollback

The reading job goes on while the rest of the submission is still on the GPU, as on the console, whose GPU is at that point when the game polls. The parts after the writer may still read guest memory in place, which the game, let go, may write: it reads only the writer's results and the game's own fences still wait for whole submissions, and no byte changed after the writer in the diagnostic run, but a game whose read is followed by writes to memory that same submission still reads would see them early (issue 997 was reads let go before their writer, which this doesn't do). Off by default; RX_VK_READBACK_EARLY unset or 0 is the code path as before (the tracking and the splits are behind it); left off with a parallel replay.

Reviewer notes

Read early_in_place_write and memory_in_place_writer_part in gpu/vk/vk_memory.cpp, then the early wait in invalidate and the split in replay (gpu/vk/vk_replay.cpp). The per-part cost is left for a follow-up on purpose. With RX_VK_READBACK_LATE the order is in invalidate: late first, then the early wait, then the usual choice.

Issues

See #225 (the Deck profile, where a default would go).

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
• Tested in the game, or what wasn't is named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• Flags noted, and the switch reference updated (docs/development.md, the README).
• A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md; no appended row or assembled view).
• A game-free test added for the fix, failing on main and passing here.
• A check under tests/ that reads changed source updated in this PR, not weakened.
• Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
• Docs updated.

## Why On the Steam Deck the game's job threads spent ~18 ms of every frame sleep-polling for a GPU result (the culling camera block at +0x3980) that the GPU had finished computing long before: the read waited for the end of the whole 22-26 ms submission, not for the dispatch that wrote it. This lets such a read go on once its writer is done, opt-in for now. ## What changed - vk (diagnostics): `RX_VK_LOG_RB_WRITERS=1` (with `RX_VK_LOG_READBACK_WAITS=1`): for each read of GPU results that waits, what wrote its pages (the op, its program, its part and draw in the submission), when the last writer's part was done into the wait, and how many bytes changed after it; a summary by size every 3000 reads. `memory_writer_note`, `memory_count_work`, `memory_rb_writers` in `gpu/vk/vk_memory.cpp`; `rb_diag_*` and the call in `invalidate` in `gpu/vk/vk_replay.cpp`. It is how the writers below were found. Its wait comes after `RX_VK_READBACK_LATE`'s rule and only for a read that waits; a submission's span is closed on every path that finishes its recording, a burst's end included (`rb_span_close`). - vk: `RX_VK_READBACK_EARLY=1` (opt-in), for pages the GPU writes in place (no VRAM mirror: integrated GPUs, the Steam Deck): - once a range's read has waited (`memory_rb_waited`, at most 64 ranges), every GPU write in place to it keeps its submission and the part it was recorded in, whatever its size (`early_in_place_write`, called from `note_direct_write`); - a later read waits for that part (`memory_in_place_writer_part`; the newest of it and a mirror's part), when every page of the range that the read's submission wrote in place has that write seen, else for the whole submission as before (a range widened over a page written before, or a page tracked later); a write to any byte of a page a waited range reaches is kept for the page, and only a write of the waited bytes ends its part (`invalidate`, before the usual choice, which it replaces only when it waited); - such a write ends its part right after it (`split_part(cmd, true)` in `replay`), at least `RX_VK_READBACK_EARLY_GAP` draws (16) into the part, so writers in a row end one part; `maybe_split` is `split_part(cmd, false)`, unchanged; - a draw of a program that writes storage buffers builds its descriptors (`same_desc` in `gpu/vk/vk_draw.cpp`), so each such write is noted with its part; - left off with `RX_VK_PARALLEL` ≥ 2 (logged): a parallel replay's segments are numbered as parts but never put on the recordings' timeline, so a read could wait for a part that never comes; ballotstomper also found the part before a command buffer's segments written back late there (team room); - with `RX_VK_READBACK_LATE` (#238, on main since): a range its rule names is read late as before, with no wait, and the early wait applies only to the other reads. Both switches act on the same waits in `invalidate`; whether they stay two switches, become one, or this one closes is asked on #238 (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/238#issuecomment-16652) and is being measured on the Deck (off / `newest` / this / both, one build). - tests: `tests/test_readback_early.py` (new): the tracking cut from `vk_memory.cpp` and run with the switch on, off and with `RX_VK_PARALLEL=3`; `invalidate`'s early wait cut out and run against stubs; the wiring in `replay`, `split_part`, `run_submit`, `note_direct_write` and the descriptor reuse. a range `RX_VK_READBACK_LATE` names takes no early wait. `tests/test_parallel_small_cb.py` and `tests/test_split_ub_flush.py` follow `maybe_split` into `split_part`; `tests/direct_buffer_writes_test.cpp` stubs `early_in_place_write`; `tests/test_readback_part.py` stubs the early wait (off) in its cut of `invalidate`; `tests/test_late_fix_build.py` declares the writers' hook in `wb_pages_note`. From the review: a range widened over a page the same submission wrote before, two ranges in one page, a write really over 1 MiB, and a burst's span (`burst_end` and the spans cut out and run); `tests/test_readback_part.py` runs the late rule with the diagnostics on. - docs: `docs/development.md` has both switches and the order with `RX_VK_READBACK_LATE`; four run files in `docs/tested/` (the Deck runs below). No translator file changes: the shader caches stay valid. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `RX_VK_READBACK_EARLY=1` | A read of GPU results written in place waits for its writer's part, which ends right after the writer | off | on for the Steam Deck (a follow-up in the Deck profile, #225), unless the A/B against `RX_VK_READBACK_LATE=newest` says otherwise; off elsewhere until measured | Steam Deck (+3.0 real fps at stock clocks, +2.2 at GPU 1600 MHz, interleaved) | dGPUs with VRAM mirrors (their pages are written back by part already: nearly inert); `RX_VK_PARALLEL` ≥ 2 (left off) | other APUs (ROG Ally, Legion Go, Ryzen laptops), Windows iGPUs | | `RX_VK_READBACK_EARLY_GAP=<draws>` | The least draws into a part before an early end | 16 | keep | - | - | - | | `RX_VK_LOG_RB_WRITERS=1` | Diagnostics: the readbacks' writers (above) | off | for diagnosis only: it waits for the writer's part first, timed | - | - | - | ## Measurements **Steam Deck with the review fixes (52306c0c), the player's build, two runs per arm: off 11.5 / 11.1 real fps, on 13.5 / 13.2 (+2.05, +18%), no read falling back to its whole submission.** Before them, **current main with #238, two usable runs per arm: off 10.8 / 11.4 real fps, `RX_VK_READBACK_EARLY=1` 12.6 / 12.8 (+1.6, +14% on the means), against #238's `RX_VK_READBACK_LATE=newest` 12.0 / 11.6.** Both switches together: 12.7 / 13.0, level with this one alone. The earlier tables below (+2.3 to +3.0) have one usable off run on the head, or on runs 16.8-19.5 wide: supporting only. **RTX 5090: idle at 98cfaf0**, which is before #250's sparse mirrors; with them a few pages are written in place on a discrete card too, and that path is not measured there. main: cf605c85 (2026-11-13, with #238, #250, #252) this PR: 9e50672 (the code of 025d8e98) in the Deck build 22253fd = main cf605c85 + #225 Deck profile (d9b3551) + #223 (52a61f4) + this PR; one build for every arm. Measured before the review fixes (52306c0c: a part known only when every write of the submission to the range's pages was seen; the diagnostics' wait after the late rule; the diagnostics' span closed at a burst's end); the next Deck A/B re-measures off against on with them. Baseline: A: this PR @ 9e50672, nothing set (with no switch set this PR's code is idle: main's path; one build isolates the switches from #225 and #223, which the Deck build needs). Hardware: Steam Deck OLED (AMD Custom GPU 0932, RADV VANGOGH, Mesa 26.2.0-devel), 4C/8T Zen 2, 16 GB, SteamOS 3.9.5, stock clocks. Settings: the player's own: handheld profile, the game drawn at 1536x864 (a crop of 2560x1440) at 100% render, a 1280x720 FIFO swapchain, frame generation 3x; no longshot_env.txt; `RX_VK_LOG_READBACK_WAITS=1` in every arm. Scene and method: Kokiri Forest, the moving route (walk and camera pans, 125 s, measured 30-110 s), warm, interleaved A B C D D C B A; real fps from the harness's frame log (the game's frames; "shown" with frame generation); 15 route shots and 4 at the start a run, compared at the same points; one unmeasured flash-detector run each for A, B and D (`RX_VK_FLASH=1`). | Steam Deck OLED, stock | A: this PR @ 9e50672, nothing set | B: this PR @ 9e50672, `RX_VK_READBACK_LATE=newest` | C: this PR @ 9e50672, `RX_VK_READBACK_EARLY=1` | D: this PR @ 9e50672, both | |---|---|---|---|---| | real fps (runs) | 10.8, 11.4 | 12.0, 11.6 | 12.6, 12.8 | 12.7, 13.0 | | mean, against A | 11.1 | 11.8 (+0.7) | 12.7 (+1.6, +14%) | 12.85 (+1.75) | | shown fps | 27.5, 29.1 | 31.8, 29.9 | 34.0, 35.3 | 34.2, 35.2 | | p50 / p95 / p99 ms | 16.9 / 127.6, 125.2 / 180.9, 174.4 | 16.8 / 114.7, 121.9 / 163.4, 185.4 | 16.8-16.9 / 106.3, 80.7 / 161.9, 156.2 | 16.8 / 102.2, 98.7 / 155.8, 156.3 | | frames over 100 ms | 203, 207 | 165, 179 | 147, 125 | 148, 138 | | a readback that waits (of 3000) | 268 x 23.3 ms, 221 x 23.6 | 303 x 14.9, 408 x 13.3 | 1023 x 1.55, 1029 x 1.36 | 999 x 0.88, 1003 x 1.12 | | captures waiting for late pages (of 600) | 1, 3 | 0, 10 | 92, 90 | 82, 91 | | replay memory sync, ms a frame | 2.3, 1.9 | 1.2, 2.3 | 19.6, 21.6 | 14.8, 20.2 | | GPU busy (route average) | 71% | 74-75% | 80-81% | 82-85% | - **This switch is the faster one here, and its values are exact:** a read waits for its writer's part (1.4-1.6 ms) rather than being read without a wait. `newest` lets the newest submission's ranges go late but every other read still waits for its whole submission (13-15 ms each). - **Both together add nothing measurable** over this switch alone (12.85 against 12.7, within the spread). - **What this switch costs:** the game, let go at the writer's part, writes the camera block's page again while the submission runs, and the next submit's capture waits for the rest of it (82-92 of 600 captures, memory sync ~20 ms a frame against ~2). A follow-up makes those captures wait for the writer's part too; it is being measured on the Deck (off / on / on with it). `newest` does not have this cost. - **Picture:** the route shots match at the same points in every arm (the only dark frames are the camera inside foliage, in A and D alike); no white frames. Flash detector, one run each: A 1 (Navi's light next to the camera), B 0, D 2 (Navi's light again; a Kokiri boy drawn white for one present, Navi at his head the present before; one run cannot say whether D's frame-old values played a part). C had no flash run. - The GPU takes 50-61 ms a frame at these settings in every arm, so the frame is close to GPU-bound; the switches fill the GPU (71% busy -> 80-85%). **With the review fixes, on the player's build** (deck-user-1113b @ 832a8a12 = main cf605c85 + #227 with #225 + #223 + #232 + this PR @ 025d8e98, plus the review fixes 52306c0c and a local follow-up whose switch, `RX_VK_READBACK_EARLY_CAPTURE`, is off in both columns here: its code then only calls an empty hook; longshot_vk.so 4756e5e7, one build; the same settings and route, interleaved A B (C C) B A, run opusab): | Steam Deck OLED, stock | the player's build + this PR @ 52306c0c, `RX_VK_READBACK_EARLY=0` | the player's build + this PR @ 52306c0c, `RX_VK_READBACK_EARLY=1` | Δ | |---|---|---|---| | real fps (runs) | 11.5, 11.1 | 13.5, 13.2 | +2.05 on the means (+18%) | | shown fps | 29.9, 27.9 | 38.9, 35.6 | | | p95 / p99 ms | 119.0, 127.9 / 171.3, 175.6 | 61.0, 89.5 / 132.2, 150.4 | | | frames over 100 ms | 178, 216 | 82, 131 | | | a readback that waits (of 3000) | 277 x 22.4 ms, 236 x 27.3 | 904 x 1.1, 1002 x 1.4; 0 for their whole submission | | | captures waiting for late pages (of 600) | 2, 1 | 69, 84 | | | GPU busy | 73%, 74% | 85%, 83% | | The follow-up (C, on in two runs between them) made those captures wait for their late pages' writers' part: it engaged on every one (89, 92) but each still waited ~20 ms, the writers' parts ending about where the submissions do, and ran 12.7 / 12.8: no gain, so it is not proposed. Route shots normal in every arm. **Earlier runs (supporting, before main had #238):** main: 514591a (2026-11-13) this PR: opus/early-readback @ 5b31bc0 (the Deck build: ae1cffd = main 514591a + #225 Deck profile + this PR @ 5b31bc0 + #252 movie input limit); the PC pair @ 98cfaf0 Baseline: this PR @ 5b31bc0, RX_VK_READBACK_EARLY unset. With the switch unset this change's code is idle, which is main's path; one build for both arms isolates the switch from #225 and #252, which the Deck build needs and main does not have. 98cfaf0 (after the Deck build) only keeps the switch idle where no page is written in place: on the Deck every page is, so its path there is 5b31bc0's. Hardware: Steam Deck OLED (AMD Custom GPU 0932, RADV VANGOGH, Mesa 26.2.0-devel git-5c65d4fc31), 4C/8T Zen 2, 16 GB (swapping in every run), SteamOS; docked to 3440x1440, game output 2048x1152, FSR 3.1 at ~50%, frame generation 2x (the Deck profile). PC: RTX 5090, Windows 11, the PC runner's bench folder. Scene: Kokiri Forest, the moving route (walk and camera pans, 125 s, measured 30-110 s); the Deck: the user's options, no longshot_env.txt; the PC: M1152 (2048x1152 window) and MP165 (5120x2160 at 165 Hz) Method: the Deck agent's A/B harness and the PC runner's bench; runs interleaved off / on / off / on after a cold start that prepared the shaders; `RX_VK_LOG_READBACK_WAITS=1` in both arms; fps and frame times from the harness's frame log ("real": the game's frames; "shown": with frame generation); screenshots through the route; flashes and luma on the PC How to reproduce: `RX_VK_READBACK_EARLY=1` in longshot_env.txt, Kokiri Forest; the log's `readbacks waited for their writer's part` line every 3000 reads | Steam Deck OLED, stock clocks | this PR @ 5b31bc0, RX_VK_READBACK_EARLY unset | this PR @ 5b31bc0, RX_VK_READBACK_EARLY=1 | Δ | |---|---|---|---| | real fps (runs) | 13.9; 11.0 (swapped heavily: 78,710 pages in, against 45-50k in the others) | 16.5, 16.0 (median 16.25) | +2.3 against 13.9 (+17%); +3.8 against the median of both (12.45) | | shown fps | 25.9; 17.6 | 31.6, 31.3 | | | p50 / p95 / p99 frame ms | 25.6 / 84.5 / 164.6; 34.9 / 160.9 / 352.6 | 24.2-24.7 / 71.0-73.3 / 120.1-136.5 | | | frames over 100 ms | 77; 157 | 70, 44 | | | draws a frame | 1,509; 1,815 | 1,378, 1,576 | | | readback waits (a log line of 3000 reads) | 193 x 21.6 ms, 246 x 16.7 ms | 964 x 1.01 ms, 988 x 1.66 ms | | The cold start before them (switch unset, shaders prepared in it) ran 15.0 real fps on a lighter stretch (1,264 draws a frame); not counted. All four runs stayed on the outdoor route; the picture is normal in every shot. `readbacks waited for their writer's part (RX_VK_READBACK_EARLY): 749-988, 0.98-2.29 ms each; 0 for their whole submission (writer not known); 800-863 parts ended after a readback's writer`. | RTX 5090, Windows | this PR @ 98cfaf0, RX_VK_READBACK_EARLY unset | this PR @ 98cfaf0, RX_VK_READBACK_EARLY=1 | Δ | |---|---|---|---| | M1152 fps (runs) | 74.80, 73.72 | 74.86, 73.22 | +0.06, -0.50 (noise) | | M1152 p99 | 19.99, 20.39 ms | 19.54, 19.79 ms | | | MP165 fps (runs) | 68.29, 66.54 | 67.96, 71.58 | -0.33, +5.04 (noise) | | MP165 p99 | 23.32, 23.66 ms | 22.71, 20.95 ms | | | replay submit / descriptors, ms a frame | 6.10-9.47 / 1.08-1.44 | 6.11-9.48 / 1.07-1.37 | | | flashes per 1,000 frames | 38.1, 41.1 (M1152); 20.5, 14.8 (MP165) | 50.6, 34.4; 15.0, 10.8 | noise | On the PC the switch logged `0, 0.00 ms each; 0 for their whole submission (writer not known); 0 parts ended` in every run: idle, as it should be with VRAM mirrors. The first PC pair, on 8f6ac8f before the idle check, cost ~2 fps at M1152 (73.66 -> 71.71) from bookkeeping that never acted; 98cfaf0 removed it (comment of 2026-11-13). **Supporting: the Steam Deck user build, interleaved** (deck-longshot-user 92640867 = main e7ed2f7a + #225 + memory pressure + #227 + #232, with this change as 996e76b, pushed as opus/early-readback-deck-measured; one build for both arms): | stock clocks | user build, switch unset @ 996e76b | user build, RX_VK_READBACK_EARLY=1 @ 996e76b | Δ | |---|---|---|---| | real fps (runs) | 15.5, 14.8 (median 15.2) | 16.8, 19.5 (median 18.2) | +3.0 (+20%) | | p95 frame ms | 74.4, 79.6 | 73.5, 49.4 | | | frames over 100 ms | 59, 72 | 51, 31 | | | GPU at 1600 MHz | user build, switch unset @ 996e76b | user build, RX_VK_READBACK_EARLY=1 @ 996e76b | Δ | |---|---|---|---| | real fps (runs) | 19.8 (the other run left the route: void) | 21.1, 22.8 (median 22.0) | +2.2 | | p95 frame ms | 45.5 | 46.2, 41.1 | | **Earlier single runs (with HOS_CHAIN), why it helps:** | | switch off (chain-lu) | RX_VK_READBACK_EARLY=1 (EA) | |---|---|---| | real fps | 14.7 | 17.1 | | HOS_CHAIN critical path a frame | 72.4 ms | **57.6 ms** | | of it, the job threads' poll at lr 0x8009a478 | 18.0 ms | **5.3 ms** | | 0x3980 read wait, each | 18.9 ms | **1.7 ms** | | replay submit / of it memory sync, ms a frame | 27.6 / 1.4 | 35.9 / 7.1 | The frame's path is now the render thread (the inline replay and the game's render work, 32 of 58 ms), then the main thread. ``` (this build + RX_VK_LOG_RB_WRITERS, stock) vk: readback writers, size 0x3980: 68 waits, 15.54 ms each; 68 with the last writer's part known: done 4.49 ms into their 15.54 ms (11.04 ms of each wait after it); bytes the wait changed: 328.0 each, 0.0 of them after the part (0 of 68 waits); last writers: dispatch fd958faab99f20a2 x68 vk: readback writers 0x1139483000 +0x3980: last dispatch fd958faab99f20a2 (part 62659, draw 744 of 1384 in submission 28043), first dispatch 7984161db3e053b5 (part 62657, draw 580), 4 pages; the submission's parts 62653..62664; its last writer's part done 0.67 ms into the wait of 12.48; bytes changed 0 by then, 0 after ``` What the diagnostic run showed (236 waits for +0x3980, 23 for +0x8b040/+0x9b040): the culling camera block (+0x3980, 1.1 reads a frame) is written by compute dispatch fd958faab99f20a2, its first writer 7984161db3e053b5 one or two parts before, at the median 49% of its submission; on average its part was done 6.4 ms into an 18.9 ms wait, and in no wait did a byte change after it. The +0x8b040/+0x9b040 rings are written by dispatches at ~43% (when +0x3980's wait was skipped outright in an earlier test, these took its place with 5.5-12 ms waits: here their writer's part ends early too). The cost: each early part end is a part's end (the uniform updates, the write-back, a barrier, a submit) and the next one's start; replay submit time rose 8.3 ms a frame at stock (memory sync 1.4 -> 7.1), but only ~1 ms of it on the render thread's path. Most of it is not the part ends: the capture before each game submit waits for its "late pages" (the game writes the camera block again while the submission still runs), for the whole submission (36-71 of 600 takes, ~8 ms each). A follow-up PR makes those wait for their writers' part (being measured on the Deck). ## Testing **Tested:** - `python -I -B tests/run_all.py` @ 7da13e56 (the review fixes; Linux also @ 52306c0c) and @ 025d8e9 on main cf605c8 (with #238, #250, #162), the same failures at each, Linux (Ubuntu 22.04 container, clang 23, the CPM Vulkan headers) and Windows (llvm-mingw): exactly main's failures. Linux 7, each failing on main @ cf605c8 in the same container (`test_cpp_preflight`, `test_dev_packs`, `test_framegen_resources`, `test_module_ids`, `test_pad_layout`, `test_prof_clock`, the byte-compile without `.git`), 523 passed against main's 522. Windows 7 (`test_fpx_sticky`, `test_bake_portability`, `test_module_ids`, `test_pad_layout`, `test_spawn_emit`, `test_tested_configurations` (its CRLF pipe check), `test_oot_state_tap` (named pipe)), each also failing on main here; 478 passed. On the way, main's newer `test_split_ub_flush` and `test_late_fix_build` cut `maybe_split` and `wb_pages_note`; both follow them in this PR. The Linux renderer (`RX_VK_ONLY`) builds clean. - `tests/test_readback_early.py` with gcc 13 (WSL) and clang 23 (container, llvm-mingw). - In the game: Steam Deck OLED, Kokiri Forest moving route: this PR on main (4 runs, 2 with the switch), the user build's interleaved confirmation (8 runs, 4 with the switch) and two earlier single runs (stock with HOS_CHAIN; GPU at 1600 MHz); 15 screenshots along the route and up to 4 in the conversation each, the switch's 8 runs checked: exposure, fog, lighting, HUD and minimap normal, no flashes, white or dark frames, no glowing objects, the conversation's camera right. - RTX 5090, Windows: the switch off and on @ 98cfaf0, M1152 and MP165, 2 interleaved rounds (above): idle, no fps or p99 change, flashes within main's range, 0 bad log lines (no screenshots in that pair: a bench bug left them off on both arms). **Not tested:** - This head on a discrete GPU: the RTX 5090 pair was @ 98cfaf0, before #250's sparse mirrors made a few pages written in place there too. - Other APUs and Windows iGPUs, where pools may be unmirrored too. - Other areas than Kokiri Forest; long sessions. **Needs other testers:** - ROG Ally / Legion Go / a Ryzen laptop on Linux: Kokiri Forest with the switch, expecting the `readbacks waited for their writer's part` line and fps not lower (T-346Y; claimable once a maintainer adds the `testing wanted` label) ## Screenshots Nothing changes on screen by design; the picture check is under Testing (38 shots on the Deck, normal). ## Risk and rollback The reading job goes on while the rest of the submission is still on the GPU, as on the console, whose GPU is at that point when the game polls. The parts after the writer may still read guest memory in place, which the game, let go, may write: it reads only the writer's results and the game's own fences still wait for whole submissions, and no byte changed after the writer in the diagnostic run, but a game whose read is followed by writes to memory that same submission still reads would see them early (issue 997 was reads let go before their writer, which this doesn't do). Off by default; `RX_VK_READBACK_EARLY` unset or 0 is the code path as before (the tracking and the splits are behind it); left off with a parallel replay. ## Reviewer notes Read `early_in_place_write` and `memory_in_place_writer_part` in `gpu/vk/vk_memory.cpp`, then the early wait in `invalidate` and the split in `replay` (`gpu/vk/vk_replay.cpp`). The per-part cost is left for a follow-up on purpose. With `RX_VK_READBACK_LATE` the order is in `invalidate`: `late` first, then the early wait, then the usual choice. ## Issues See https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/225 (the Deck profile, where a default would go). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`; no appended row or assembled view). - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] Docs updated.

dukenheimer added 2 commits 2026-11-13 20:52:11 +00:00

feat(vk): RX_VK_LOG_RB_WRITERS, what wrote the pages a readback waits for ... f17a931a37

Diagnostics, off by default. The replay notes the op it records (and a storage buffer's
program) and counts its draws and dispatches; each page a write-back is noted for keeps the
first and last such writer of its newest submission. A read of GPU results that waits then
logs (with RX_VK_LOG_READBACK_WAITS) its writers, their part and draw in the submission, how
long into the wait the last writer's part was done (it waits for that part first, then for the
rest), and the bytes the wait changed before and after that part; every 3000 reads, a summary
by size. On the Steam Deck it named the dispatch that writes the camera block the game's jobs
poll, halfway through its submission, with the data final 12.5 of 18.9 ms before the wait ended.

Co-authored-by: Copilot App <[email protected]>

perf(vk): RX_VK_READBACK_EARLY, a read of pages written in place waits for its writer's part ... 8f6ac8f320

Opt-in. On a GPU whose pools have no VRAM mirror (the Steam Deck: every page) a CPU read of
GPU results waited for the whole submission that wrote it. With RX_VK_READBACK_EARLY=1, once a
range's read has waited, every GPU write in place to it keeps the part it was recorded in (any
size), a later read waits for that part when its newest write is the range's newest (else the
whole submission, as before), and the write ends its part right after it (at least
RX_VK_READBACK_EARLY_GAP draws, 16, into the part). A draw of a program that writes storage
buffers builds its descriptors so each write is noted.

Steam Deck, Kokiri Forest, stock clocks: the camera-block reads waited 18.9 ms each, 1.7 now;
the frame's critical path 72.4 -> 57.6 ms; 14-15 -> 17.1 real fps (preliminary, one run).

tests/test_readback_early.py; test_parallel_small_cb and the direct-write fixture follow
maybe_split through split_part and the note in note_direct_write. docs/development.md.

Co-authored-by: Copilot App <[email protected]>

dukenheimer added 1 commit 2026-11-13 22:09:26 +00:00

docs(tested): RX_VK_READBACK_EARLY on the Steam Deck, interleaved, stock and GPU 1600 MHz ... 5b31bc0c5d

Co-authored-by: Copilot App <[email protected]>

dukenheimer commented 2026-11-13 22:09:28 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

Interleaved confirmation on the Steam Deck OLED (one build for both arms, only RX_VK_READBACK_EARLY differs, Kokiri Forest moving route, the user's options):

• Stock clocks: median real fps 15.2 -> 18.2 (+3.0, +20%), shown 28.7 -> 34.8. Runs: off 15.5, 14.8; on 16.8, 19.5. The switch's runs also have the better p95 (49-74 against 74-80 ms) and fewer frames over 100 ms (31-51 against 59-72).
• GPU at 1600 MHz: 19.8 -> 22.0 real fps (the other switch-off run left the route and is void).
• Readback waits: 11.7-12.8 ms each before, 1.5-1.8 ms with the switch.
• Picture: normal in every run with the switch, along the route and in the conversation.

The description's Measurements section now has both tables; the earlier single runs and the HOS_CHAIN split are kept below them. The device is a Steam Deck OLED, not LCD as the first version said. docs/tested-configurations.md has the two runs (5b31bc0). The PC no-regression pair (RTX 5090) is still running.

Interleaved confirmation on the Steam Deck OLED (one build for both arms, only `RX_VK_READBACK_EARLY` differs, Kokiri Forest moving route, the user's options): - **Stock clocks:** median real fps 15.2 -> **18.2 (+3.0, +20%)**, shown 28.7 -> 34.8. Runs: off 15.5, 14.8; on 16.8, 19.5. The switch's runs also have the better p95 (49-74 against 74-80 ms) and fewer frames over 100 ms (31-51 against 59-72). - **GPU at 1600 MHz:** 19.8 -> **22.0** real fps (the other switch-off run left the route and is void). - **Readback waits:** 11.7-12.8 ms each before, 1.5-1.8 ms with the switch. - **Picture:** normal in every run with the switch, along the route and in the conversation. The description's Measurements section now has both tables; the earlier single runs and the HOS_CHAIN split are kept below them. The device is a Steam Deck **OLED**, not LCD as the first version said. `docs/tested-configurations.md` has the two runs (5b31bc0). The PC no-regression pair (RTX 5090) is still running.

dukenheimer added 1 commit 2026-11-13 22:43:51 +00:00

perf(vk): RX_VK_READBACK_EARLY idle until a read of pages written in place waits ... 98cfaf00b8

On an RTX 5090 (VRAM mirrors: no page written in place) the switch waited early 0 times and cost
~2 fps at M1152 (1.2 at MP165): every waiting read noted its range and walked the ranges under
wb_pages_mu, which made every storage-writing draw rebuild its descriptors, and every op the
replay recorded did an atomic exchange. Now a range is noted only by a read with a page written
in place pending, the descriptor rebuild waits for one, and the per-op check is a load.

Co-authored-by: Copilot App <[email protected]>

dukenheimer commented 2026-11-13 22:45:17 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

PC no-regression pair (RTX 5090), first result on 8f6ac8f: the switch never acted there, but cost ~2 fps (M1152 73.66 → 71.71, −1.84 and −2.05 in the two rounds; p99 +0.9 ms; MP165 round 1 −1.21). The B arm logged 0 early waits, which is right: with VRAM mirrors no page is written in place. The cost was its bookkeeping, which ran anyway:

• every waiting read noted its range and walked the ranges under wb_pages_mu;
• that turned on the per-draw descriptor rebuild for storage-writing programs;
• every op the replay recorded did an atomic exchange.

98cfaf0 (pushed) makes the switch idle until a read of a page written in place waits:

• a range is noted only then;
• the descriptor rebuild waits for that;
• the per-op check is a load.

On the Steam Deck every page is written in place, so its path and the numbers above are unchanged. The pair is running again on 98cfaf0. tests/test_readback_early.py checks the idle state (nothing noted for a read of mirrored pages only; no tracking before an in-place read waited).

**PC no-regression pair (RTX 5090), first result on 8f6ac8f: the switch never acted there, but cost ~2 fps** (M1152 73.66 → 71.71, −1.84 and −2.05 in the two rounds; p99 +0.9 ms; MP165 round 1 −1.21). The B arm logged 0 early waits, which is right: with VRAM mirrors no page is written in place. The cost was its bookkeeping, which ran anyway: - every waiting read noted its range and walked the ranges under `wb_pages_mu`; - that turned on the per-draw descriptor rebuild for storage-writing programs; - every op the replay recorded did an atomic exchange. **98cfaf0** (pushed) makes the switch idle until a read of a page written in place waits: - a range is noted only then; - the descriptor rebuild waits for that; - the per-op check is a load. On the Steam Deck every page is written in place, so its path and the numbers above are unchanged. The pair is running again on 98cfaf0. `tests/test_readback_early.py` checks the idle state (nothing noted for a read of mirrored pages only; no tracking before an in-place read waited).

proci referenced this pull request 2026-11-13 23:58:38 +00:00

fix(vk): HUD labels the GPU drew no longer go blank after a zone change #325

proci referenced this pull request 2026-11-14 00:01:43 +00:00

WIP: perf(vk): a mirror sync's pages from the captures go in one compute dispatch, not one copy region each #326

dukenheimer commented 2026-11-14 00:17:02 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

Measured on the head, and ready for review.

• Steam Deck OLED, this PR on main 514591a (build ae1cffd = main + #225 + this PR @ 5b31bc0 + #252; switch unset vs on, interleaved, stock clocks):
  • real fps: 13.9 → 16.5, 16.0 with the switch (+2.3, +17%), p95 85 → 71–73 ms.
  • The other switch-off run (11.0): it swapped heavily (78.7k pages in, against 45–50k in the others), so it isn't used for the delta. Counting it, the median gain is +3.8.
  • Readback waits: 16.7–21.6 ms each → 1.0–1.7 ms. Picture normal in every shot.
• RTX 5090 @ 98cfaf0: the switch is idle (0 early waits, 0 fallbacks, 0 parts ended in all 243 log lines). M1152 74.26 → 74.04 fps (+0.06, −0.50), MP165 67.42 → 69.77 (noise), p99 not worse, flashes within main's range. The −2 fps of 8f6ac8f is gone.
• Suites @ 98cfaf0: Linux 500 passed, Windows 456 passed, failing exactly main's 5 / 6.

The user-build confirmation (+3.0 at stock, +2.2 at 1600 MHz) stays as supporting evidence. T-346Y (other AMD APUs) waits for the testing wanted label.

**Measured on the head, and ready for review.** - **Steam Deck OLED, this PR on main 514591a** (build ae1cffd = main + #225 + this PR @ 5b31bc0 + #252; switch unset vs on, interleaved, stock clocks): - **real fps:** 13.9 → **16.5, 16.0** with the switch (+2.3, +17%), p95 85 → 71–73 ms. - **The other switch-off run (11.0):** it swapped heavily (78.7k pages in, against 45–50k in the others), so it isn't used for the delta. Counting it, the median gain is +3.8. - **Readback waits:** 16.7–21.6 ms each → 1.0–1.7 ms. Picture normal in every shot. - **RTX 5090 @ 98cfaf0:** the switch is idle (0 early waits, 0 fallbacks, 0 parts ended in all 243 log lines). M1152 74.26 → 74.04 fps (+0.06, −0.50), MP165 67.42 → 69.77 (noise), p99 not worse, flashes within main's range. The −2 fps of 8f6ac8f is gone. - **Suites @ 98cfaf0:** Linux 500 passed, Windows 456 passed, failing exactly main's 5 / 6. The user-build confirmation (+3.0 at stock, +2.2 at 1600 MHz) stays as supporting evidence. T-346Y (other AMD APUs) waits for the `testing wanted` label.

dukenheimer changed title from WIP: perf(vk): RX_VK_READBACK_EARLY, a read of GPU results written in place waits for its writer's part, not the whole submission to perf(vk): RX_VK_READBACK_EARLY, a read of GPU results written in place waits for its writer's part, not the whole submission 2026-11-14 00:17:14 +00:00

HumanBean referenced this pull request from a commit 2026-11-14 01:28:04 +00:00

fix(vk): HUD labels the GPU drew no longer go blank after a zone change (#325)

mooki added the

area/renderer

gpu/igpu

kind

performance

platform/steam-deck

labels 2026-11-14 01:46:02 +00:00

dukenheimer referenced this pull request 2026-11-14 02:06:06 +00:00

perf(vk): RX_VK_READBACK_LATE, the readbacks a rule names are read without their wait (opt-in) #238

dukenheimer force-pushed opus/early-readback from 98cfaf00b8 to 9e5067211b 2026-11-14 02:18:52 +00:00 Compare

dukenheimer commented 2026-11-14 02:19:45 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

Rebased onto main 76699aa, over #238 (RX_VK_READBACK_LATE). Conflicts were in invalidate: a range #238's rule names is read late as before, with no wait, and RX_VK_READBACK_EARLY applies only to the other reads. Main's newer checks (test_split_ub_flush, test_late_fix_build) follow split_part and the writers' hook. The Deck runs are in docs/tested/ (the frozen table). How the two switches combine is asked on #238; a Deck A/B of off / newest / early / both is requested. Head 9e50672.

Rebased onto main 76699aa, over #238 (`RX_VK_READBACK_LATE`). Conflicts were in `invalidate`: a range #238's rule names is read late as before, with no wait, and `RX_VK_READBACK_EARLY` applies only to the other reads. Main's newer checks (`test_split_ub_flush`, `test_late_fix_build`) follow `split_part` and the writers' hook. The Deck runs are in `docs/tested/` (the frozen table). How the two switches combine is asked on #238; a Deck A/B of off / newest / early / both is requested. Head 9e50672.

dukenheimer force-pushed opus/early-readback from 9e5067211b to 025d8e98d2 2026-11-14 02:38:10 +00:00 Compare

dukenheimer commented 2026-11-14 02:39:20 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

Rebased again onto main cf605c8 (#162, #250, #252); only the docs conflicted. Head 025d8e9. One note on #250: with sparse mirrors, a dGPU write to a block that has no memory yet goes in place for that submission, so on a discrete card the early wait can act on those pages. The docs and the comment in invalidate now say so (the RTX 5090 pair predates #250). Suites: exactly main's failures on Linux and Windows. The Deck A/B (off / newest / early / both) is being built from main cf605c8 + #225 + #223 + this.

Rebased again onto main cf605c8 (#162, #250, #252); only the docs conflicted. Head 025d8e9. One note on #250: with sparse mirrors, a dGPU write to a block that has no memory yet goes in place for that submission, so on a discrete card the early wait can act on those pages. The docs and the comment in `invalidate` now say so (the RTX 5090 pair predates #250). Suites: exactly main's failures on Linux and Windows. The Deck A/B (off / newest / early / both) is being built from main cf605c8 + #225 + #223 + this.

mooki added the

priority

high

label 2026-11-14 03:55:00 +00:00

mooki requested changes 2026-11-14 04:23:57 +00:00

Dismissed

mooki left a comment

• Copy link
• Copy Markdown

Reviewed head 025d8e98 against main cf605c85: the full diff, surrounding readback and submission paths, all comments and the discussion on #238; the six added or changed test drivers and seven related drivers pass here, including GCC and Clang for every compiled fixture. No game or renderer build was run.

What's good

• The cost exists in main: a pending write in place makes the read wait for its submission rather than its writer's part. Reusing the recordings' timeline is a small way to shorten that wait without deliberately reading an older result.
• The switch remains opt-in, refuses parallel replay, and starts tracking only after an in-place read waits. The existing part-wait, uniform-flush and direct-write checks retain their assertions.

Must change

1. Require complete writer coverage before returning an early part. gpu/vk/vk_memory.cpp, lines 2043–2051, skips pages missing from early_pages and accepts the range when any tracked page has the submission value whole. For example, first learn page A alone. In submission 11, write A in part 7 and an untracked page B in part 8. A first read covering A and B gets whole == 11; adding that wider range in memory_rb_waited does not recover B's earlier write. This function returns part 7, and invalidate releases the CPU before B's writer has finished. Main waits for the submission instead. Check that every pending in-place portion of the requested range has complete writer history, otherwise keep the whole wait. Add a case where a learned range is widened after another part wrote the newly included bytes, including two subranges of one page.

2. Apply the late rule before the diagnostic wait. gpu/vk/vk_replay.cpp, lines 3232–3233, calls rb_diag_wait before computing late. With RX_VK_LOG_RB_WRITERS=1 and a matching RX_VK_READBACK_LATE rule, that function already waits for the writer's part. The read is no longer wait-free, despite the later !late guards, and the diagnostic can change both the values read and the timing being compared. Decide whether this read will wait first, and do not add a wait to a late read. Exercise the diagnostic-enabled path as well: the new choice fixture starts after this call and cannot catch it.

3. Finish the diagnostic span when a burst closes. gpu/vk/vk_replay.cpp, line 2583, calls rb_span_end only outside a burst. The normal event-ordered path enters run_submit with burst_open set, then finishes through burst_end and finish_recording, neither of which closes the span. Its last part and total work remain zero. The log then substitutes the last writer's work count for the total, reporting that writer as the end of the submission and printing a last part of zero. Close the span on the common finishing path and cover a normal burst with work after the writer. These fields are used to establish how early the writer finishes.

Measurement and smaller points

• The disclosed switch comparison is a valid choice of baseline, but the headline needs qualification. After excluding the heavily swapping run, the recent Deck table has only one usable off run, not two. The older stock pair gives the stated median gain, but its on runs span 16.8–19.5 fps, about 14.9% of their median. Treat the gain as preliminary until the promised interleaved off / newest / early / both comparison supplies two usable runs per arm and reports their spread. The discussion at #238 (comment) remains open; no naming or default decision is needed to fix the issues above.
• Keep the RTX 5090 result tied to its measured head. As the latest comment explains, sparse mirrors now allow in-place writes on discrete GPUs too, so the earlier idle result does not establish the cost of that newly active path.
• tests/test_readback_early.py, line 128, calls 64 pages a write beyond 1 MB. That is only 256 KiB. Use a range larger than 1 MiB for the claimed boundary case.

Not checked

No Steam Deck, Windows, other GPU, long-session or visual testing here. I checked the published tables and quoted lines, not the underlying captures or screenshots. The source checks pass, but do not cover the three cases above. The current source-check jobs are blocked, not green: https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/3821.

Reviewed head `025d8e98` against main `cf605c85`: the full diff, surrounding readback and submission paths, all comments and the discussion on https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/238; the six added or changed test drivers and seven related drivers pass here, including GCC and Clang for every compiled fixture. No game or renderer build was run. ### What's good - The cost exists in main: a pending write in place makes the read wait for its submission rather than its writer's part. Reusing the recordings' timeline is a small way to shorten that wait without deliberately reading an older result. - The switch remains opt-in, refuses parallel replay, and starts tracking only after an in-place read waits. The existing part-wait, uniform-flush and direct-write checks retain their assertions. ### Must change 1. **Require complete writer coverage before returning an early part.** `gpu/vk/vk_memory.cpp`, lines 2043–2051, skips pages missing from `early_pages` and accepts the range when any tracked page has the submission value `whole`. For example, first learn page A alone. In submission 11, write A in part 7 and an untracked page B in part 8. A first read covering A and B gets `whole == 11`; adding that wider range in `memory_rb_waited` does not recover B's earlier write. This function returns part 7, and `invalidate` releases the CPU before B's writer has finished. Main waits for the submission instead. Check that every pending in-place portion of the requested range has complete writer history, otherwise keep the whole wait. Add a case where a learned range is widened after another part wrote the newly included bytes, including two subranges of one page. 2. **Apply the late rule before the diagnostic wait.** `gpu/vk/vk_replay.cpp`, lines 3232–3233, calls `rb_diag_wait` before computing `late`. With `RX_VK_LOG_RB_WRITERS=1` and a matching `RX_VK_READBACK_LATE` rule, that function already waits for the writer's part. The read is no longer wait-free, despite the later `!late` guards, and the diagnostic can change both the values read and the timing being compared. Decide whether this read will wait first, and do not add a wait to a late read. Exercise the diagnostic-enabled path as well: the new choice fixture starts after this call and cannot catch it. 3. **Finish the diagnostic span when a burst closes.** `gpu/vk/vk_replay.cpp`, line 2583, calls `rb_span_end` only outside a burst. The normal event-ordered path enters `run_submit` with `burst_open` set, then finishes through `burst_end` and `finish_recording`, neither of which closes the span. Its last part and total work remain zero. The log then substitutes the last writer's work count for the total, reporting that writer as the end of the submission and printing a last part of zero. Close the span on the common finishing path and cover a normal burst with work after the writer. These fields are used to establish how early the writer finishes. ### Measurement and smaller points - The disclosed switch comparison is a valid choice of baseline, but the headline needs qualification. After excluding the heavily swapping run, the recent Deck table has only one usable off run, not two. The older stock pair gives the stated median gain, but its on runs span 16.8–19.5 fps, about 14.9% of their median. Treat the gain as preliminary until the promised interleaved off / `newest` / early / both comparison supplies two usable runs per arm and reports their spread. The discussion at https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/238#issuecomment-16652 remains open; no naming or default decision is needed to fix the issues above. - Keep the RTX 5090 result tied to its measured head. As the latest comment explains, sparse mirrors now allow in-place writes on discrete GPUs too, so the earlier idle result does not establish the cost of that newly active path. - `tests/test_readback_early.py`, line 128, calls 64 pages a write beyond 1 MB. That is only 256 KiB. Use a range larger than 1 MiB for the claimed boundary case. ### Not checked No Steam Deck, Windows, other GPU, long-session or visual testing here. I checked the published tables and quoted lines, not the underlying captures or screenshots. The source checks pass, but do not cover the three cases above. The current source-check jobs are blocked, not green: https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/3821.

dukenheimer added 2 commits 2026-11-14 07:01:01 +00:00

fix(vk): RX_VK_READBACK_EARLY needs every write of the submission seen; the diagnostics wait after the late rule ... 52306c0cc3

From mooki's review of #303:
- memory_in_place_writer_part skipped pages it had no write for: a range widened over a page the
  same submission wrote before (in a later part) was let go at an earlier part. Now every page of
  the range whose in-place write-back is the read's submission (as memory_writeback_value found
  it) needs that write seen, else the whole submission as before; and a write to any byte of a
  page a waited range reaches is kept for the page (two ranges in one page), while only a write
  of the waited bytes ends its part.
- invalidate: RX_VK_LOG_RB_WRITERS's wait for the writer's part came before the late rule
  (RX_VK_READBACK_LATE), so a read late waited with the diagnostics on. It comes after it now,
  and only for a read that waits.
- RX_VK_LOG_RB_WRITERS's submission span was closed only outside a burst: a burst's kept no last
  part or total work. rb_span_close before every finish_recording.
- tests: the widened range and two ranges of a page (test_readback_early.py), a write really
  over 1 MiB, the diagnostics with a late rule (test_readback_part.py), a burst's span (cut
  out and run).

Co-authored-by: Copilot App <[email protected]>

docs(tested): RX_VK_READBACK_EARLY against RX_VK_READBACK_LATE=newest and both on the Steam Deck, interleaved ... 7da13e5609

Co-authored-by: Copilot App <[email protected]>

dukenheimer commented 2026-11-14 07:01:18 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

@mooki, thanks. All three must-change points are fixed in 52306c0c; head 7da13e56.

1. Writer coverage. memory_in_place_writer_part now walks every page of the range. Each page whose in-place write-back is the read's submission (as memory_writeback_value found it, wb_pages / wb_parts) needs that write seen in early_pages, with the same value; otherwise the read waits for the whole submission, as main does. Pages an older submission wrote are covered by any part of this one, because the recordings' timeline is in order. Tracking is now per page: a write to any byte of a page that a waited range reaches is kept for the whole page, so two ranges in one page can't hide a later part. Only a write of the waited bytes still ends its part. New cases in tests/test_readback_early.py:

   • a range widened over a page the submission wrote unseen;
   • the same range once that page is written again (the later part);
   • two ranges of one page;
   • a range reaching into an unseen page.

   The previous code fails four of them (checked).

2. Late rule first. rb_diag_wait now comes after late and runs only for pending && !late. tests/test_readback_part.py now cuts that call with the choice and runs it with the diagnostics on: with a matching newest rule there is no wait, and without a rule the diagnostic wait comes first. A source check also fails if rb_diag_wait appears before the late rule.

3. Span at a burst's end. A new rb_span_close() runs before every finish_recording: run_submit's, burst_end's and a resumed queue's capture. burst_end and the span functions are cut out and run with a burst that does work after its writer: its span closes with the last part and the total work. A source check counts the finish_recording calls.

Smaller points

• The test's write over 1 MB is now 320 pages (1.25 MiB).
• The RTX 5090 result is tied to 98cfaf0 and listed under Not tested for this head, with #250's sparse mirrors named.

Measurement
The interleaved Deck comparison is in, on main cf605c85 with #238, two usable runs per arm (they were measured before these fixes):

arm	real fps
off	10.8, 11.4
RX_VK_READBACK_LATE=newest	12.0, 11.6
RX_VK_READBACK_EARLY=1	12.6, 12.8
both	12.7, 13.0

The headline is now this table, with the spread. The earlier +2.3 to +3.0 runs are marked as supporting only.

One cost is stated there too: with this switch the next submit's capture waits for late pages, about 20 ms of memory sync a frame. A follow-up for that is on the Deck now, in the same A/B that re-measures off against on with these fixes.

Suites at 7da13e56 fail exactly main's tests on Linux and Windows. Not tested: the fixes on the Deck (that A/B is queued).

@mooki, thanks. All three must-change points are fixed in 52306c0c; head 7da13e56. 1. **Writer coverage.** `memory_in_place_writer_part` now walks every page of the range. Each page whose in-place write-back is the read's submission (as `memory_writeback_value` found it, `wb_pages` / `wb_parts`) needs that write seen in `early_pages`, with the same value; otherwise the read waits for the whole submission, as main does. Pages an older submission wrote are covered by any part of this one, because the recordings' timeline is in order. Tracking is now per page: a write to any byte of a page that a waited range reaches is kept for the whole page, so two ranges in one page can't hide a later part. Only a write of the waited bytes still ends its part. New cases in `tests/test_readback_early.py`: - a range widened over a page the submission wrote unseen; - the same range once that page is written again (the later part); - two ranges of one page; - a range reaching into an unseen page. The previous code fails four of them (checked). 2. **Late rule first.** `rb_diag_wait` now comes after `late` and runs only for `pending && !late`. `tests/test_readback_part.py` now cuts that call with the choice and runs it with the diagnostics on: with a matching `newest` rule there is no wait, and without a rule the diagnostic wait comes first. A source check also fails if `rb_diag_wait` appears before the late rule. 3. **Span at a burst's end.** A new `rb_span_close()` runs before every `finish_recording`: run_submit's, `burst_end`'s and a resumed queue's capture. `burst_end` and the span functions are cut out and run with a burst that does work after its writer: its span closes with the last part and the total work. A source check counts the `finish_recording` calls. **Smaller points** - The test's write over 1 MB is now 320 pages (1.25 MiB). - The RTX 5090 result is tied to 98cfaf0 and listed under Not tested for this head, with #250's sparse mirrors named. **Measurement** The interleaved Deck comparison is in, on main cf605c85 with #238, two usable runs per arm (they were measured before these fixes): | arm | real fps | |---|---| | off | 10.8, 11.4 | | `RX_VK_READBACK_LATE=newest` | 12.0, 11.6 | | `RX_VK_READBACK_EARLY=1` | 12.6, 12.8 | | both | 12.7, 13.0 | The headline is now this table, with the spread. The earlier +2.3 to +3.0 runs are marked as supporting only. One cost is stated there too: with this switch the next submit's capture waits for late pages, about 20 ms of memory sync a frame. A follow-up for that is on the Deck now, in the same A/B that re-measures off against on with these fixes. Suites at 7da13e56 fail exactly main's tests on Linux and Windows. Not tested: the fixes on the Deck (that A/B is queued).

dukenheimer referenced this pull request 2026-11-14 07:02:00 +00:00

perf(vk): RX_VK_READBACK_LATE, the readbacks a rule names are read without their wait (opt-in) #238

mooki requested changes 2026-11-14 07:53:27 +00:00

mooki left a comment

• Copy link
• Copy Markdown

Read head 7da13e56, fix 52306c0c and its parent 025d8e98: the complete diff, updated description, all comments and the earlier review; ran both changed test drivers with GCC and Clang, their regression cases against the parent, three related checks, and the mixed-write case below. No game or renderer build was run.

What's good

The new tests catch the earlier small-write coverage, late-diagnostic and burst-span failures. The fix keeps the existing whole-submission fallback and does not change the switch's opt-in default.

Earlier must-change points

1. Writer coverage: partly settled. gpu/vk/vk_memory.cpp, lines 2025-2054, now follows writes to any byte of a learned page and rejects missing history for pages stamped with whole. The four regressions at tests/test_readback_early.py, lines 144, 155, 157 and 159, fail against 025d8e98 and pass at this head with both compilers. The later rewrite at line 147 also selects the later part. However, the new filter misses a current large write when the plain page table still names an older submission. That case is below.

2. Late rule before diagnostics: settled. gpu/vk/vk_replay.cpp, lines 3247-3249, now decides late before calling rb_diag_wait, guarded by pending && !late. The diagnostic-enabled cases at tests/test_readback_part.py, lines 240-247, pass with GCC and Clang. Against the parent, the new source guard fails; compiling the same fixture with the parent's actual diagnostic prefix included also fails the no-wait assertion. This checks the formerly missed call, not just the later choice.

3. Span at a burst's end: settled. gpu/vk/vk_replay.cpp, lines 2457, 2584 and 2612, closes the span before every recording finish; lines 2962-2964 use the still-live recording's value and part. The new burst fixture at tests/test_readback_early.py, lines 261-272, fails against the parent and passes here with both compilers. It retains the last part and total work after work following the writer. The finish-site source guards pass too.

The parent runs used the new fixture bodies. Their drivers stop first at source guards, so I also compiled the memory and span fixtures separately and included the old diagnostic call in the part fixture. These are behavioral failures, not missing-symbol failures.

Must change

1. Include large in-place writes when choosing the early part. gpu/vk/vk_memory.cpp, lines 2049-2053, skips a page before consulting early_pages when its wb_pages value differs from whole. But note_direct_write, lines 4136-4140, updates early_pages for every write and wb_pages only for writes at most 1 MiB. A current writer can therefore be skipped despite being known:

   • Learn adjacent pages A and B. Submission 10 previously wrote B through a small binding.
   • In submission 11, part 7 writes A through a small binding. Part 8 writes B through a 320-page binding starting at B, without touching A.
   • The plain table still has A = 11 and B = 10, while the early table has A = {11, 7} and B = {11, 8}. Reading both pages gives whole == 11.
   • The new lookup returns part 7. The parent returns part 8. invalidate, lines 3264-3278, accepts part 7 and skips the whole wait, so the CPU can read B before its writer finishes. Main waits for the whole submission 11 instead.

   Reproduced with GCC and Clang, using the actual note_direct_write, page tables and writer lookup cut from each revision. Both print v=11 whole=11 mirror_part=0; the parent prints early_part=8, this head early_part=7. This is a regression introduced by 52306c0c. Include the current early entry even when the plain table is stale, or fall back to the whole submission if coverage is uncertain. Add this mixed small/large-write case and require part 8 or that fallback. The new all-large-write case at tests/test_readback_early.py:131-134 does not cover it.

Bounds, repeated writes and smaller points

• The additional compiled checks pass for an unaligned read across a page-table chunk, an exact page end, zero size, the table's last page, and eviction of overlapping learned ranges. The limit remains 64 ranges, not 64 pages. No additional bounds defect found.
• Ordinary small writes spanning submissions select the newer part. Rewriting the same page in the same submission selects its later part, regardless of the payload values. These checks execute the tracking, not GPU payload writes. The mixed-size case above is the remaining failure.
• The 320-page test is now genuinely 1.25 MiB, and the RTX 5090 result is correctly tied to 98cfaf0.
• The new Deck table has named commits, one PC and settings, and two usable runs per arm in A B C D D C B A order. Its means are also its medians: off 11.1 fps, early 12.7, both 12.85. The +1.6 fps calculation holds for that pre-fix build. Off spans 10.8-11.4 fps, a 5.4% repeat spread; early spans 12.6-12.8, or 1.6%. Both adds less than that spread. Keep the gain tied to the measured build until the disclosed current-head comparison arrives.

Not checked

No Steam Deck, Windows execution, GPU payload correctness, screenshots, long session or current-head performance measurement here. The early-part regression is reproduced without a game; its gameplay frequency is unknown. The updated head merges cleanly with main 0caf0618. Source-check jobs remain blocked, not passed: https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/4461.

Read head `7da13e56`, fix `52306c0c` and its parent `025d8e98`: the complete diff, updated description, all comments and the earlier review; ran both changed test drivers with GCC and Clang, their regression cases against the parent, three related checks, and the mixed-write case below. No game or renderer build was run. ### What's good The new tests catch the earlier small-write coverage, late-diagnostic and burst-span failures. The fix keeps the existing whole-submission fallback and does not change the switch's opt-in default. ### Earlier must-change points 1. **Writer coverage: partly settled.** `gpu/vk/vk_memory.cpp`, lines 2025-2054, now follows writes to any byte of a learned page and rejects missing history for pages stamped with `whole`. The four regressions at `tests/test_readback_early.py`, lines 144, 155, 157 and 159, fail against `025d8e98` and pass at this head with both compilers. The later rewrite at line 147 also selects the later part. However, the new filter misses a current large write when the plain page table still names an older submission. That case is below. 2. **Late rule before diagnostics: settled.** `gpu/vk/vk_replay.cpp`, lines 3247-3249, now decides `late` before calling `rb_diag_wait`, guarded by `pending && !late`. The diagnostic-enabled cases at `tests/test_readback_part.py`, lines 240-247, pass with GCC and Clang. Against the parent, the new source guard fails; compiling the same fixture with the parent's actual diagnostic prefix included also fails the no-wait assertion. This checks the formerly missed call, not just the later choice. 3. **Span at a burst's end: settled.** `gpu/vk/vk_replay.cpp`, lines 2457, 2584 and 2612, closes the span before every recording finish; lines 2962-2964 use the still-live recording's value and part. The new burst fixture at `tests/test_readback_early.py`, lines 261-272, fails against the parent and passes here with both compilers. It retains the last part and total work after work following the writer. The finish-site source guards pass too. The parent runs used the new fixture bodies. Their drivers stop first at source guards, so I also compiled the memory and span fixtures separately and included the old diagnostic call in the part fixture. These are behavioral failures, not missing-symbol failures. ### Must change 1. **Include large in-place writes when choosing the early part.** `gpu/vk/vk_memory.cpp`, lines 2049-2053, skips a page before consulting `early_pages` when its `wb_pages` value differs from `whole`. But `note_direct_write`, lines 4136-4140, updates `early_pages` for every write and `wb_pages` only for writes at most 1 MiB. A current writer can therefore be skipped despite being known: - Learn adjacent pages A and B. Submission 10 previously wrote B through a small binding. - In submission 11, part 7 writes A through a small binding. Part 8 writes B through a 320-page binding starting at B, without touching A. - The plain table still has A = 11 and B = 10, while the early table has A = `{11, 7}` and B = `{11, 8}`. Reading both pages gives `whole == 11`. - The new lookup returns part 7. The parent returns part 8. `invalidate`, lines 3264-3278, accepts part 7 and skips the whole wait, so the CPU can read B before its writer finishes. Main waits for the whole submission 11 instead. **Reproduced with GCC and Clang**, using the actual `note_direct_write`, page tables and writer lookup cut from each revision. Both print `v=11 whole=11 mirror_part=0`; the parent prints `early_part=8`, this head `early_part=7`. This is a regression introduced by `52306c0c`. Include the current early entry even when the plain table is stale, or fall back to the whole submission if coverage is uncertain. Add this mixed small/large-write case and require part 8 or that fallback. The new all-large-write case at `tests/test_readback_early.py:131-134` does not cover it. ### Bounds, repeated writes and smaller points - The additional compiled checks pass for an unaligned read across a page-table chunk, an exact page end, zero size, the table's last page, and eviction of overlapping learned ranges. The limit remains 64 ranges, not 64 pages. No additional bounds defect found. - Ordinary small writes spanning submissions select the newer part. Rewriting the same page in the same submission selects its later part, regardless of the payload values. These checks execute the tracking, not GPU payload writes. The mixed-size case above is the remaining failure. - The 320-page test is now genuinely 1.25 MiB, and the RTX 5090 result is correctly tied to `98cfaf0`. - The new Deck table has named commits, one PC and settings, and two usable runs per arm in A B C D D C B A order. Its means are also its medians: off 11.1 fps, early 12.7, both 12.85. The +1.6 fps calculation holds for that pre-fix build. Off spans 10.8-11.4 fps, a 5.4% repeat spread; early spans 12.6-12.8, or 1.6%. Both adds less than that spread. Keep the gain tied to the measured build until the disclosed current-head comparison arrives. ### Not checked No Steam Deck, Windows execution, GPU payload correctness, screenshots, long session or current-head performance measurement here. The early-part regression is reproduced without a game; its gameplay frequency is unknown. The updated head merges cleanly with main `0caf0618`. Source-check jobs remain blocked, not passed: https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/4461.

dukenheimer added 1 commit 2026-11-14 09:11:55 +00:00

docs(tested): RX_VK_READBACK_EARLY with the review fixes on the Steam Deck player's build, interleaved ... 995aab1577

Co-authored-by: Copilot App <[email protected]>

dukenheimer commented 2026-11-14 09:11:55 +00:00

Author

First-time contributor

• Copy link
• Copy Markdown

Measured with the review fixes on the Steam Deck. This was the player's build plus 52306c0c: one build, the moving Kokiri route, interleaved, stock clocks. Off ran 11.5 / 11.1 real fps, on 13.5 / 13.2 (+18%); p95 went from 119 / 128 ms to 61 / 90 ms. Each early read waited 1.1–1.4 ms, and 0 reads fell back to their whole submission, so the stricter writer check costs nothing there. A follow-up that made the capture's late pages wait for their writers' part engaged but saved nothing (12.7 / 12.8), so it isn't proposed. Run file in docs/tested/.

Measured with the review fixes on the Steam Deck. This was the player's build plus 52306c0c: one build, the moving Kokiri route, interleaved, stock clocks. Off ran 11.5 / 11.1 real fps, on 13.5 / 13.2 (+18%); p95 went from 119 / 128 ms to 61 / 90 ms. Each early read waited 1.1–1.4 ms, and 0 reads fell back to their whole submission, so the stricter writer check costs nothing there. A follow-up that made the capture's late pages wait for their writers' part engaged but saved nothing (12.7 / 12.8), so it isn't proposed. Run file in docs/tested/.

suporte_de_cadeira commented 2026-11-14 13:29:28 +00:00

First-time contributor

• Copy link
• Copy Markdown

Windows compile and game-free checks for head 995aab15:

• longshot_vk.dll built with the pinned llvm-mingw 20261104 (RX_VK_ONLY=1 RX_RELEASE=1 RX_TARGET=windows, --jobs 6): 0 failed, linked.
• tests/test_readback_early.py, tests/test_readback_part.py, tests/test_split_ub_flush.py, tests/test_parallel_small_cb.py and tests/test_late_fix_build.py: pass.

No game or gameplay run on this machine, so the Deck numbers remain the performance evidence. The Windows build at least shows the new readback paths compile and link there.

Windows compile and game-free checks for head `995aab15`: - `longshot_vk.dll` built with the pinned llvm-mingw 20261104 (`RX_VK_ONLY=1 RX_RELEASE=1 RX_TARGET=windows`, `--jobs 6`): 0 failed, linked. - `tests/test_readback_early.py`, `tests/test_readback_part.py`, `tests/test_split_ub_flush.py`, `tests/test_parallel_small_cb.py` and `tests/test_late_fix_build.py`: pass. No game or gameplay run on this machine, so the Deck numbers remain the performance evidence. The Windows build at least shows the new readback paths compile and link there.

Some checks are pending

Hide checks

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Website / Website on Cloudflare (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.

Some workflows are waiting to be reviewed.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'opus/early-readback:dukenheimer-opus/early-readback'

git switch 'dukenheimer-opus/early-readback'

Sign in to join this conversation.

Reviewers

No reviewers

mooki

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Linux only

  

platform/steam-deck

Steam Deck

  

platform/windows

Windows only

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!303

No description provided.