perf(vk): on the Steam Deck's APU, no idle send by default, and held submits go out while a readback waits #2422
Open
addisonbair wants to merge 2 commits from
addisonbair/deck-batch-idle into main
pull from: addisonbair/deck-batch-idle
merge into: MagicBeanDev:main
MagicBeanDev:main
MagicBeanDev:github/pr/2893
MagicBeanDev:github/pr/2890
MagicBeanDev:github/pr/2887
MagicBeanDev:github/pr/2878
MagicBeanDev:github/pr/2875
MagicBeanDev:github/pr/2874
MagicBeanDev:github/pr/2873
MagicBeanDev:github/pr/2871
MagicBeanDev:github/pr/2756
MagicBeanDev:github/pr/2897
MagicBeanDev:github/pr/2896
MagicBeanDev:github/pr/2718
MagicBeanDev:github/pr/2842
MagicBeanDev:github/pr/2691
MagicBeanDev:github/pr/2895
MagicBeanDev:github/pr/2743
MagicBeanDev:github/pr/2765
MagicBeanDev:github/pr/2708
MagicBeanDev:github/pr/2732
MagicBeanDev:github/pr/2755
MagicBeanDev:github/pr/2817
MagicBeanDev:github/pr/2876
MagicBeanDev:release-please--branches--main
MagicBeanDev:release-please--branches--main--release-notes
MagicBeanDev:github/pr/2778
MagicBeanDev:github/pr/2888
MagicBeanDev:github/pr/2866
MagicBeanDev:github/pr/2877
MagicBeanDev:github/pr/2800
MagicBeanDev:github/pr/2829
MagicBeanDev:github/pr/2850
MagicBeanDev:github/pr/2889
MagicBeanDev:github/pr/2882
MagicBeanDev:github/pr/2860
MagicBeanDev:github/pr/2849
MagicBeanDev:github/pr/2886
MagicBeanDev:github/pr/2794
MagicBeanDev:github/pr/2840
MagicBeanDev:github/pr/2830
MagicBeanDev:github/pr/2834
MagicBeanDev:github/pr/2719
MagicBeanDev:github/pr/2742
MagicBeanDev:github/pr/2741
MagicBeanDev:github/pr/2820
MagicBeanDev:github/pr/2771
MagicBeanDev:github/pr/2859
MagicBeanDev:github/pr/2858
MagicBeanDev:github/pr/2823
MagicBeanDev:github/pr/2757
MagicBeanDev:github/pr/2769
MagicBeanDev:github/pr/2844
MagicBeanDev:github/pr/2843
MagicBeanDev:github/pr/2673
MagicBeanDev:github/pr/2855
MagicBeanDev:github/pr/2739
MagicBeanDev:github/pr/2807
MagicBeanDev:github/pr/2851
MagicBeanDev:github/pr/2789
MagicBeanDev:github/pr/2814
MagicBeanDev:github/pr/2818
MagicBeanDev:github/pr/2774
MagicBeanDev:github/pr/2772
MagicBeanDev:github/pr/2750
MagicBeanDev:github/pr/2786
MagicBeanDev:github/pr/2785
MagicBeanDev:github/pr/2782
MagicBeanDev:github/pr/2779
MagicBeanDev:github/pr/2775
MagicBeanDev:github/pr/2706
MagicBeanDev:github/pr/2766
MagicBeanDev:github/pr/2747
MagicBeanDev:github/pr/2759
MagicBeanDev:github/pr/2745
MagicBeanDev:github/pr/2740
MagicBeanDev:github/pr/2733
MagicBeanDev:github/pr/2725
MagicBeanDev:github/pr/2727
MagicBeanDev:github/pr/2728
MagicBeanDev:github/pr/2729
MagicBeanDev:github/pr/2721
MagicBeanDev:github/pr/2722
MagicBeanDev:github/pr/2715
MagicBeanDev:github/pr/2714
MagicBeanDev:github/pr/2707
MagicBeanDev:github/pr/2703
MagicBeanDev:github/pr/2705
MagicBeanDev:github/pr/2697
MagicBeanDev:github/pr/2679
MagicBeanDev:github/pr/2676
MagicBeanDev:github/pr/2669
MagicBeanDev:docs/what-hookshot-connects-to
MagicBeanDev:fix/hdr-encode-test
MagicBeanDev:fix/main-tests
MagicBeanDev:fix/play-cache-files
MagicBeanDev:perf/game-barrier-order
MagicBeanDev:feat/heap-lazy
MagicBeanDev:feat/texture-detail
MagicBeanDev:perf/astc-opaque-bc1
MagicBeanDev:fix/fs-dir-lock
MagicBeanDev:fix/framegen-native-kept
MagicBeanDev:fix/update-no-downgrade
MagicBeanDev:port/k10-gcn-f16-mirrors
MagicBeanDev:fix/framegen-mult-cap
MagicBeanDev:chore/release-as-2.1.1
MagicBeanDev:perf/pcache-load-switch
MagicBeanDev:fix/prepare-progress-log
MagicBeanDev:fix/update-closes-stale-game
MagicBeanDev:fix/hdr-scaled-sampled
MagicBeanDev:perf/first-start-preset
MagicBeanDev:perf/zero-upload
MagicBeanDev:fix/vcore-escape-2s
MagicBeanDev:test/fake-gh-atomic
MagicBeanDev:perf/batch-idle-flush
MagicBeanDev:test/finish-stub-a
MagicBeanDev:chore/record-2.1.0
MagicBeanDev:fix/main-red-tests
MagicBeanDev:fix/test-display-startup-driver-cache
MagicBeanDev:feat/mesh-shader-mode
MagicBeanDev:fix/menu-probes-at-startup
MagicBeanDev:fix/fps-counter-classic
MagicBeanDev:fix/test-log-batch-flake
MagicBeanDev:fix/narrow-screen-hud
MagicBeanDev:ci/prereleases
MagicBeanDev:release/v2.0.15
MagicBeanDev:hotfix/v2.0.15
MagicBeanDev:fix/present-ahead-replay
MagicBeanDev:release/v2.0.13
MagicBeanDev:fix/bracket-paths-2013
MagicBeanDev:fix/hdr-overlays-2013
MagicBeanDev:fix/fps-counter-scale-2013
MagicBeanDev:fix/stick-seen-at-rest
MagicBeanDev:hotfix/menu-fits-screen
MagicBeanDev:fix/f1-always-opens-menu
MagicBeanDev:fix/ini-keep-lines
MagicBeanDev:claude/cranky-pike-7cf247
MagicBeanDev:perf/texture-answer-flat
MagicBeanDev:fix/stamp-test-modules
MagicBeanDev:fix/appimage-fonts
MagicBeanDev:fix/display-awake
MagicBeanDev:release/v2.0.12
MagicBeanDev:fix/discord-windows
MagicBeanDev:rc/v2.0.12-tests
MagicBeanDev:fix/dlss-keep-feature
MagicBeanDev:rc/v2.0.12
MagicBeanDev:button-icons
No reviewers
Labels
Clear labels
accessibility
Barrier affecting people with disabilities
autorelease: pending
autorelease: tagged
bug
Something isn't working
dependencies
Pull requests that update a dependency file
documentation
Improvements or additions to documentation
duplicate
This issue or pull request already exists
enhancement
New feature or request
github_actions
Pull requests that update GitHub Actions code
good first issue
Good for newcomers
help wanted
Extra attention is needed
invalid
This doesn't seem right
question
Further information is requested
testing wanted
wontfix
This will not be worked on
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!2422
Loading…
Reference in a new issue
No description provided.
Delete branch "addisonbair/deck-batch-idle"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
#1999 (from #1993) sends a held batch as soon as the GPU is idle. On an RX 9070 XT that took Kokiri Forest from 33-41 fps to 46-61. On a Steam Deck (Van Gogh, RADV) each submit costs ~1 ms of CPU, so the idle send costs more than it saves. Two commits:
0x163fLCD,0x1435OLED). Every other GPU, other APUs included (780M, 890M, 8060S), keeps it: none of them is measured.RX_VK_BATCH_IDLE=1or0still chooses, and the log says which (vk: idle send on/off).invalidate()sends what is held when the wait begins, then again every 8 held command buffers while it waits. The waits themselves are unchanged.RX_VK_READBACK_SEND=0turns it off;RX_VK_READBACK_SEND_CBSsets the count.Deck measurements on unmodified main (
eaef1460, which includes #2125 and #2222), default switches, standing Kokiri scene (~2,400 draws), GPU at 1200 MHz, deterministic headless runs, two passes:submitqueue submitCommit 1 alone saves ~9 ms of replay but loses fps: readback waits gate the frame, and holding batches makes them longer. That's why the two belong together. The Deck runs used the same change with a few off-by-default diagnostics compiled in; this branch drops them.
P16 (NVIDIA,
RX_VK_BATCH=1 RX_VK_BATCH_MAX=32to imitate RADV's batching): main 40.9 fps / 15.6 flushes a frame, commit 1 34.8 / 2.8, both 38.6 / 5.4. So on a GPU that keeps the idle send, commit 2 only acts while a readback waits.Testing:
tests/run_all.pyin an Ubuntu 22.04 builder: the same failures as main (tools missing in my container), plus a newtests/test_readback_send.py. The renderer builds.🤖 Generated with Claude Code
The code change is fine:
dev.propsis filled before the first submit, the function-local static is initialised once, andRX_VK_BATCH_IDLE=0/1still chooses either way. The two C++ fixtures that stubbatch_gpu_idle()(tests/draw_stream_visibility_test.cpp,tests/fsr_image_retirement_test.cpp) aren't affected. My questions are about the default's reach and the evidence behind it, not the code:question
gpu/vk/vk_replay.cpp:179-180: batching is on by default only with RADV (batch_on), so in practice this default lands on every RADV APU: the Deck, but also 780M/890M laptops and the Radeon 8060S from issue 413, the machine batching was introduced for. As you say yourself, the Deck is the only one measured. The 8060S is a much stronger GPU that is more likely to sit idle between flushes, which is the case #1993 was fixing. Could you either get one A/B run on another APU (anyone from issue 413 / #1000), or narrow the rule (Van Gogh bydeviceID, or a GPU-clock/CU threshold)? Otherwise, please state in the PR body that the 8060S is expected to change and nobody has checked.question: the Deck numbers were taken on main with #1375 reverted and
RX_VK_NO_READBACK_WAIT=1, so neither is what players run. #1375 changed how much the replay thread copies, which shifts exactly the CPU-vs-submit balance this default is about. Does the ~1.3 fps / ~4-5 ms of queue submit still show on unmodified main with default switches? With two runs per side the gap (18.6-18.8 vs 20.0-20.7) is plausible but small, and confirming it on the shipped configuration would make the default easy to accept.minor:
batch_on()logs its choice and why.batch_gpu_idle()now also depends on the device type but logs nothing, so a player'shookshot_log.txtwon't show whether the idle send was on. A one-timelogfbesidebatch_on's ("idle send off on an integrated GPU (RX_VK_BATCH_IDLE=1: on)") would make reports readable.Only the title check has run in CI on this PR so far. The game-free suite still has to run once the workflow is approved.
The code is correct (
vk_replay.cpp:179readsdev.propsonce, andRX_VK_BATCH_IDLEstill overrides). The rule is broader than the title, though. Batching is only on with RADV, so "integrated" hits every RADV APU (780M/890M laptops, Radeon 8060S), not just the Deck. The Deck numbers were also taken with #1375 reverted andRX_VK_NO_READBACK_WAIT=1, not the shipped configuration. Either narrow it to Van Gogh by deviceID or measure another APU. A log line showing whether the idle send is on would help with reports.Thanks for measuring this on a Deck, the per-submit cost on Van Gogh is a useful data point. I read the change and agree with Joey-Magnus's review: the code is correct (
dev.propsis filled before the first flush, the static is set once,RX_VK_BATCH_IDLE=0/1still overrides either way, and main hasn't touchedbatch_gpu_idle()since, so it merges cleanly).His open points still stand and I'd like them settled before this goes in:
deviceID.RX_VK_NO_READBACK_WAIT=1. One A/B pair on unmodified main with default switches would show the gain holds for what players actually run.logfof the idle-send choice next tobatch_on()'s, sohookshot_log.txtshows which path a report came from.CI: only the title check has run; the game-free suite still needs the workflow approved. No verdict from me until the reach question is answered.
nit: "current main" here isn't quite what was measured (main with #1375 reverted and
RX_VK_NO_READBACK_WAIT=1, per the PR body). Either say so, or re-measure on stock main and keep the wording.Thanks, both. All three points are addressed at
dc25850:0x163f,0x1435). Every other APU, the 8060S included, keeps the idle send.eaef1460) with default switches. This answer is less flattering: the idle-send change alone loses fps there (13.65 → 12.8). It saves ~9 ms of replay, but readback waits gate the frame, and holding batches makes those waits longer. So I added a second commit. While a readback waits, held submits go out, which brings it to 14.45 (two passes each). The table is in the description. The comment atvk_replay.cpp:182now quotes the stock-main numbers.vk: idle send on/off (RX_VK_BATCH_IDLE…)next tobatch_on()'s, and the readback send logs its choice too.The second commit touches the readback path from #2125, so it's worth a look by whoever reviewed that one. A new source check,
tests/test_readback_send.py, covers it.🤖 Generated with Claude Code
Confirmed on the Deck with this exact head (
dc25850, built against today's main), against main's stock renderer on the sameeaef1460game, with default switches and the standing Kokiri scene at a 1200 MHz GPU clock: main 13.5 / 13.5 fps, this PR 14.3 / 14.4. GPU time is unchanged (29.9 ms vs 29.8), so the gain is on the CPU side, as expected.🤖 Generated with Claude Code
Steam Deck results for this PR merged onto current main. They show a regression there, and I think it comes from its interaction with #2602.
Setup: Steam Deck OLED (Galileo, APU 0932, RADV, SteamOS 3.9.5 Game Mode, handheld profile, docked to a 1920x1080 gamescope screen). Linux x86-64-v3 builds with the release play profile. Kokiri Forest standing scene from the same save (~2,400 draws). Each start loads the save, then logs 30 s of frame times and takes 80 compositor screenshots.
Builds: main
6749ba21plus the Deck patches (UB snapshot without mirrors), with and without this PR merged (a clean merge). The two arms alternated start by start on the same caches.The logs confirm the PR's switches were active:
idle send off (... off on the Steam Deck)andheld submits go out while a readback waits: on. On start 3 the stall repeats this line:a readback's wait for write-back 35850 timed out after 250 ms (done 35849, submitted 35851, handed out 35851). That is the same one-short timeline that #2222 bounds. Without the idle send, readbacks wait far more often (~4x), and the waits run into that cap.The PR's own numbers were taken on
eaef1460, before #2602 (readbacks wait for the part that writes their range back), and #2602 changes when readbacks wait. On current main the same build without this PR is already at ~29-30 FPS on the Deck (b7e0e043, before #2602: ~25). So I'd re-measure the idle-send default on the Deck against current main before merging.A second Deck's numbers, in case they help. On mine this PR is slower than main at
c544828f, in two scenes.Setup. Steam Deck OLED (Van Gogh
0x1435), SteamOS 3.9.5, kernel 7.2.7-valve2, Mesa 26.2.0-devel (RADV). Game built from main atc544828f(v2.1.0-rc.3-248). Two renderers built the same way: main atc544828funchanged, and this PR's headdc258507merged ontoc544828f(it merges cleanly). Headless runs on a fixed save, standing still, 110 s measured per run, two alternating passes.Not default switches. This Deck runs with
HOS_HANDHELD=1,RX_VK_LOW_SPEC=1, texture caps (RX_VK_VRAM_MB=2000,RX_VK_TEX_MAX_MB=128,RX_VK_TEX_MAX_DIM=1024) and frame generation at 2x (RX_VK_FRAMEGEN_MULT=2,RX_VK_FSR_FG=0). I did not measure default switches, so this does not contradict the 13.5 to 14.4 fps above. It says the result depends on the configuration.c544828fdc258507onc544828fKokiri, per frame:
c544828fdc258507onc544828fsubmitThe log confirms the PR's paths were active (
idle send off (RX_VK_BATCH_IDLE, off on the Steam Deck),held submits go out while a readback waits: on).Two things I take from it:
c544828fthe idle send costs much less than it did. Queue submit is 4.0 ms a frame in Kokiri here; a build from main ate19d969(11 November, with local changes, so not a clean comparison) gave 12.2 ms on the same scene. So there is less left to save by sending less often.I also tried a middle way on
c544828f, holding the idle send until 12 command buffers or 2.5 ms. It lands between the two (22.12 / 22.32 fps in Kokiri, 20.76 / 20.78 in Hyrule Field), so still under unchanged main. Happy to run another configuration on this Deck if it helps.My earlier points are addressed: Van Gogh only by device ID (
gpu/vk/vk_replay.cpp:206), numbers from stock main, both choices logged.I merged it into current main and ran
tests/run_all.py. Two failures are this PR's:tests/test_draw_stream_visibility.py(red in CI too) andtests/test_fsr_image_retirement.pycutsubmit_recordingout, and their fixtures don't declarereadback_send_noworFR_READBACK. Both pass on main; please add the stubs.The bigger problem is the measurements since. Brett-D (Deck LCD, recent main) saw 30 fps drop to 9.6 and 24, and one start stalled with readbacks hitting the 250 ms cap. hclaria (Deck OLED, non-default switches) saw 22.9 to 18.5 fps in Kokiri. Both suspect #2602 changed when readbacks wait after your
eaef1460runs. Could you re-measure on current main?Smaller:
invalidate()(:2388) is on for every RADV GPU, not just the Deck, and that's unmeasured.RX_VK_READBACK_SENDis read withatoi(:192), so=onturns it off; useenv_value_off, andenv_intfor_CBS(:200).