perf(vk): RX_VK_READBACK_LATE, the readbacks a rule names are read without their wait (opt-in) #238
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!238
Loading…
Reference in a new issue
No description provided.
Delete branch "p0lux/Longshot:perf/readback-late"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
On a Steam Deck the frame spends several milliseconds waiting on read-backs: a CPU read of GPU results waits for the write-back of the submission that writes them, and the game asks for most of them right after submitting the work that fills them. Reading those without the wait takes a heavy view from 24.3 to 30.3 fps on a Steam Deck OLED, and Hyrule Field from 21.9 to 26.5. The starting point is camilobt's finding on Hookshot's issue 1000 (the two 0x3980-byte ranges).
What changed
invalidate()ingpu/vk/vk_replay.cppasksreadback_late_wanted(size, newest)before its wait. A range a rule names is read as guest memory holds it (the frame before's values); every other range waits as before.gpu/vk/readback_late.h,parseandwanted.newest: the range's writer is the newest submission handed out. A hex size: the ranges of exactly that many bytes. Either is enough.rb_range_noteandrb_ranges_loginvk_replay.cpp, the read-backs by range every 3,000 calls, behind their own switch. That log is how the ranges were found, and what shows below that the 0x3980 pair is no longer most of the wait.tests/test_readback_late.py(new) runs the rules and checks the wiring;tests/test_readback_part.pyfollowsinvalidate()'s text and runs the wait with and without a rule.docs/development.md, "Switches for testing and debugging", with what was measured; a row indocs/tested-configurations.md.Flags and settings
RX_VK_READBACK_LATE=<rule>[,<rule>...]newest, or a hex size such as3980newest(ornewest,3980) on a Steam Deck after more testing on other hardwareRX_VK_LOG_READBACK_RANGES=1Measurements
main:
fbd42024(2026-11-13)this PR: perf/readback-late @ 3a6fbc0c, the change on that main. The branch is now that change rebased onto main
514591a3(bcc2496d) and a docs commit (0f7148ac: the row, and a comment and a paragraph that said what Hookshot measured). The rebase conflicted in the docs only; the change's code applied as it was. Main moved 10 commits in between, one of them in the replay (#260, the occlusion counts): the numbers below are fbd42024's, and main514591a3against the PR is to be run again.Hardware: Steam Deck OLED (AMD Custom APU 0932, RADV VANGOGH, Mesa 26.2.0-devel), 16 GB, SteamOS 3.9.5, kernel 7.2.7-valve2; the GPU's clock fixed at 1600 MHz (Steam's performance menu), read back as 1600 MHz during every run
Scene: three saves, standing still: a heavy view (about 2,150 draws a frame), Hyrule Field (about 1,600), Kokiri Forest (about 2,600); no screen (headless), handheld profile, Minimum preset, the game's own upscaler, no frame generation, no cap
Method: the back end built alone from each commit, on a host and game folder of main
000f19fb(vk_api.his the same). Each run starts the game, loads the save by script, waits 3 log windows, then measures exactly 720 presents: fps over that span; the wait is the log's "readbacks of GPU results" line. Five arms alternated in each round: main; this PR with no rule (RX_VK_READBACK_LATE=0, which names none);newest;3980;newest,3980. Heavy view and Hyrule Field 3 rounds, Kokiri Forest 2; medians (Kokiri: the mean of its two)How to reproduce: a save standing in Hyrule Field,
RX_VK_READBACK_LATE=newestinlongshot_env.txtagainst unset, "N presents, X fps" inlongshot_log.txtonce the view has settled;RX_VK_LOG_READBACK_RANGES=1for the rangesfbd42024newest3980newest,3980Every run, fps:
The log line the switch moves (heavy view, first round; Hyrule Field with no rule and with both):
Which ranges wait, with no rule (
RX_VK_LOG_READBACK_RANGES=1, the heavy view, 3,000 calls): the two 0x3980-byte ranges are 971 ms of the 2,147 ms of the eight ranges that wait most (45%; 40% in Hyrule Field). The rest is six ranges of 0x8b040 and 0x9b040 bytes. On Hookshot the 0x3980 pair was 94% of the wait, which is why the rule by size is here; on main it isnewestthat carries the gain.Testing
Tested:
python -I -B tests/run_all.pyon macOS with the Vulkan headers given: 467 passed, 30 failed, 37 skipped on the rebased branch; the 30 fail the same way on main514591a3on that machine (no/proc, noposix_fallocate, the Linux launch scripts and the like).test_readback_late.pyfails on main (the header is not there) and passes here;test_readback_part.pypasses.RX_VK_FLASH=1, about 1,800 presents a run), Kokiri Forest and Hyrule Field: 0 flashes with no rule, 0 withnewest,3980.newest,3980) played on the Deck for two days on Hookshot and about an hour on Longshot, with no flicker noticed by the player.Not tested:
newestand 1,600 to 1,634 without: a small difference in what the game draws, not looked into.Needs other testers:
RX_VK_LOG_READBACK_RANGES=1, which ranges wait the most and for how long, then whetherRX_VK_READBACK_LATE=newestflickers or gains anything there, expecting no flicker (request to open)Screenshots
Nothing on screen.
Risk and rollback
With the switch unset nothing changes but one test of a constant a read-back (main and the PR with no rule measure the same above). With it set, a range read late holds the frame before's values: right for a range the game reads before its work can be done, wrong for one whose late read matters. The exposure meter is such a one (the white flashes the wait is there for), which is why this is a rule by range and off by default, not
RX_VK_NO_READBACK_WAIT. Unset the switch to turn it off.Reviewer notes
mirror_on()is off on an integrated GPU), so this switch is for the machines that one does not reach.3980alone is weak on main (24.3 to 24.9 fps). The rule by size is kept because it is what the log lets a tester try on other hardware; say if you would rather havenewestonly.Issues
None.
Checklist
For everyone
type(scope): description).main.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above).third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.docs/tested-configurations.mdfor each measured run.mainand passing here.tests/that reads changed source updated in this PR, not weakened.c23f844e350f7148aca3Measured on a Steam Deck OLED, and rebased onto main
514591a3(the rebase conflicted in the docs only).fbd42024; the change with no rule;newest;3980;newest,3980). With both rules: 24.3 to 30.3 fps on a heavy view, 21.9 to 26.5 in Hyrule Field, 23.4 to 26.8 in Kokiri Forest. With no rule the PR measures as main.newestcarries the gain (30.1 fps on the heavy view by itself);3980alone gives 24.9. On main the two 0x3980-byte ranges are 40 to 45% of the read-backs' wait, not the 94% they were on Hookshot. The header's comment and the docs said the latter; they now say what was measured.The description has the tables and the log lines,
docs/tested-configurations.mdthe row.WIP: perf(vk): RX_VK_READBACK_LATE, the readbacks a rule names are read without their wait (opt-in)to perf(vk): RX_VK_READBACK_LATE, the readbacks a rule names are read without their wait (opt-in)Review of PR 238 (
RX_VK_READBACK_LATE), head0f7148ac. Code review only: I ran no tests or builds.The change is clean and small. With the switch unset,
invalidate()does what it did before:lateis false, sopending && !lateis the oldv > completed(). The rules are easy to follow, both checks follow the new text, and the by-range log is behind its own switch. Two things still need settling before merge, both from the rules in CONTRIBUTING.md and AGENTS.md:pr status 238: "1 to fix"), and CI here has no Windows build job to settle it. AGENTS.md says a box that can't be true yet keeps the PR WIP. The diff only uses standard C++ (getenv,strtoull,std::mutex), so I expect the llvm-mingw build oflongshot_vk.dllto pass. Please run it and tick the box, or move the PR back to WIP until someone has.fbd42024/ 3a6fbc0c, and the branch now sits on main514591a3. The 10 commits in between include a replay change (#260, occlusion query numbering across queues). The description says "main514591a3against the PR is to be run again". CONTRIBUTING.md asks you to measure again or say why the numbers still hold. Either would do: a short re-run ofnewestagainst no rule in one scene, or a line on why #260 can't move the readback wait.Smaller points, in the inline comments:
parse()keeps any entry that starts with hex digits:allis read as 0xa,1as size 1,newestxasnewest.newestwith ranges the CPU writes back, on a GPU with no mirrors such as the Deck.The open test request for a desktop GPU: the
testing wantedlabel is on, butpr statusstill counts it as "1 to open". Please check the request card exists.@ -0,0 +28,4 @@Rules r;for (const char* p = text; p && *p;) {if (!strncmp(p, "newest", 6)) r.newest = true;else if (const uint64_t n = strtoull(p, nullptr, 16)) r.sizes.push_back(n);Nit: an entry is kept as soon as it starts with hex digits.
RX_VK_READBACK_LATE=allgives size 0xa,=1(someone switching it "on") gives size 1, andnewestfoomatchesneweston line 30 because of the prefix compare. Nothing breaks: the start-up line shows(newest 0, 1 sizes). Still, a mistyped rule quietly turns into a different rule. Suggest keeping an entry only when it is whole:newestfollowed by,or the end, orstrtoullwith an end pointer that stops at,/\0. Ajunk = parse("all,1x")case in test_readback_late.py would cover it.@ -2966,2 +3025,3 @@}if (v > completed()) {const bool pending = v > completed(), newest = pending && v >= f0;const bool late = pending && readback_late_wanted(size, newest); // (RX_VK_READBACK_LATE: read without its wait)Question on
newestwith ranges the CPU writes back. The exposure meter is read, then reset to 0 by the CPU. If a range like that is read late, the CPU's write can land before the pending submission's write-back, which then overwrites it: the 2x sum in the comment above. On the Deck there are no mirrors (whole, pages written in place), and the wait is all that orders the CPU's write against the GPU's.newestdecides by who wrote the range, not by whether the game writes it back. The flash detector found 0 flashes, so the meter wasn't "newest" in those runs. Is that guaranteed by how OoT submits, or only by timing? If it's timing, please say so where the switch is documented (development.md / readback_late.h), so desktop testers know what a flash withnewestmeans.@ -2994,1 +3055,4 @@}static const bool log_ranges = env_on("RX_VK_LOG_READBACK_RANGES");if (log_ranges)rb_range_note(gpu, size, w,Nit: a range read late can still have
w == truefrom the catch-up above (behind0 && !no_catchupsets it before the late check). Such a read is noted here as both waited and late, and it also counts in the 3000-call "waited for their write-back" line. The numbers stay right, but with a rule set the by-range log can show a late range as "N waited". Either note only the part wait aswaited, or say in the log's comment that the catch-up counts.0f7148aca368ea13561aRebased onto main
5b81a8a1(was on514591a3) to clear the merge conflict; nothing else in the PR changed.docs/tested-configurations.mdonly. Main added three 2026-11-13 rows at the top of the table (the 6 GB mirror budget, #237); this PR's Steam Deck OLED row is kept on top as before, main's three rows follow it unchanged.gpu/vk/vk_replay.cpp,gpu/vk/readback_late.h, the tests) anddocs/development.mdapplied without conflict.68ea1356(the two commits are still authored by p0lux).No tests, builds or compilers were run here.
@p0lux, a question before I combine this with #303 (
RX_VK_READBACK_EARLY). Both act on the same readback waits ininvalidate(), so I've rebased #303 over this PR and would like your view on how they should fit together. I haven't decided anything yet.How they differ
RX_VK_READBACK_LATEreads a range a rule names without waiting: the frame before's values.RX_VK_READBACK_EARLY(no VRAM mirrors, so the Deck) still waits, but only until the GPU finishes the part of the submission that wrote the range, so the values are exact. On the Deck, the 0x3980 block's writer was done 6.4 ms into an 18.9 ms wait, and 0 bytes changed after that part. On the moving Kokiri route at stock clocks it gave +2.2 to +3.0 real fps (15.2 → 18.2 on the user build, 13.9 → 16.5/16.0 on #303's head).The order in the rebase, for now
newestset, EARLY is left with the reads whose writer is an older submission, so I expect it to add little on top.Options
RX_VK_READBACK_LATE. Then a switch called LATE would sometimes read exact values, which seems confusing to me.newestis as fast on the Deck with an equally clean picture, close #303, or keep it only as the exact option for ranges where a frame-old value is wrong.I'm asking for one Deck A/B from a single build: off /
newest/ EARLY / both, on the moving Kokiri route at stock clocks, with the screenshots checked. The results will go on #303. Which would you prefer, or do you see another way to combine them?@p0lux, the Deck comparison I mentioned is in (details on #303: #303).
Setup: one build on main
cf605c85(with this PR), the player's settings (handheld, 1536x864 at 100%, FG 3x), stock clocks, the moving Kokiri route, interleaved, two runs per arm.newestRX_VK_READBACK_EARLY=1Picture: route shots match in every arm. In the flash runs (one per arm, unmeasured): nothing set 1 flash (Navi's light),
newest0, both 2 (Navi again, and a Kokiri boy white for one present).My reading: on the Deck,
newestadds nothing measurable on top of EARLY, so I'd keep the two switches as they are (one exact, one late) rather than merge them, and not changenewest. Your view is still welcome, as is the result of the desktop runs HumanBean mentioned. One difference worth knowing: EARLY makes the next capture wait for late pages (~20 ms of memory sync a frame), andnewestdoesn't. A follow-up for that is being measured.A follow-up after the merge, as announced above: the measurement again on a newer main, and the Windows build that was the one box left unticked.
Main
514591a3against the change rebased on it (0f7148ac), four arms alternated on the same three scenes, the same Steam Deck OLED and method (720 presents a run; heavy view 3 rounds, the others 2):514591a30f7148ac, no rule0f7148ac,newest0f7148ac,newest,3980So the gain holds there: +20% on the heavy view and in Hyrule Field, +10% in Kokiri Forest, and with no rule it measures as main. No error in 30 runs.
Windows:
longshot_vk.dllcross-builds from0f7148acwith llvm-mingw 20261104 on Linux (RX_TARGET=windows RX_VK_ONLY=1 RX_RELEASE=1 nx2/build.py --standalone), 0 failed. Not run in the game on Windows.