MagicBeanDev/Longshot

Watch

Fork 99

Deck: the frame is CPU-bound in the submission path, not GPU-bound (16 fps at 50% internal, GPU-bound wait 0.4 ms) #402

New issue

Open

opened 2026-11-14 09:39:46 +00:00 by omgitsgela · 7 comments

omgitsgela commented

2026-11-14 09:39:46 +00:00

Contributor

Steam Deck OLED (RDNA2, 4-core), current main cf605c85, NVMe install, internal Resolution 50% (640x400 -> game's own upscale), Kokiri Forest, steady camera turn. The Deck sits at 16-19 fps here and it is not GPU-bound: the renderer's own per-frame stats, one window:

10920 presents, 16.3 fps; a frame: 1374 draws (0 skipped), 122 dispatches, 175 passes, 171 barriers, 6 submits, ... GPU-bound wait 0.4 ms
a frame's replay time (ms): submit 24.23, present 0.13, draw 15.39, descriptors 4.92, texture lookups 0.29, uploads 0.04,
  pipelines 0.58, programs 2.41, commands 6.44, dispatch 1.60, memory sync 1.15, queue submit 2.25, slot wait 0.41
batched submits: 1053 flushes, 2.3 command buffers each, over 120 presents; by reason: poll 1, wait 126, slot 0, present 119, cap 0, value 0, other 807

So the frame is ~50 ms wall while the GPU waits 0.4 ms for the submission: the cost is on the CPU/driver side recording and flushing the frame. The submission-side line items alone (submit 24 + commands 6 + descriptors 5) are about 36 of those 50 ms. Roughly 8.8 flushes a frame (1053/120), most of them "other" rather than present/wait/cap - many small command-buffer breaks a frame across 175 passes and 171 barriers.

Two things I ruled out on this machine, both negative, so they can be left out of the search:

Internal resolution barely matters here. Same scene, live switch via the Resolution row: 50% internal 18.6 fps vs 100% internal (1280x800) 13.7 fps. Pixels are not the wall; an FSR/render-scale fix cannot get the Deck to 30. (For contrast #263 shows DLSS slower than the game's own on an RTX 5090; on the Deck the game's own upscale is already the fast path.)
Storage, once on NVMe, is not it either; that is the separate #386 (SD installs re-inflate the draw count because asset streaming never settles).

Question for whoever knows this path: on RADV with RX_VK_BATCH already on, is the ~9-flushes-a-frame "other" pattern reducible for a scene with this many passes/barriers, or is there a known knob to hold a frame in fewer/larger submissions? This is the Deck's main frame-rate ceiling; the GPU is idle.

Steam Deck OLED (RDNA2, 4-core), current main cf605c85, NVMe install, internal Resolution 50% (640x400 -> game's own upscale), Kokiri Forest, steady camera turn. The Deck sits at 16-19 fps here and it is **not GPU-bound**: the renderer's own per-frame stats, one window: ``` 10920 presents, 16.3 fps; a frame: 1374 draws (0 skipped), 122 dispatches, 175 passes, 171 barriers, 6 submits, ... GPU-bound wait 0.4 ms a frame's replay time (ms): submit 24.23, present 0.13, draw 15.39, descriptors 4.92, texture lookups 0.29, uploads 0.04, pipelines 0.58, programs 2.41, commands 6.44, dispatch 1.60, memory sync 1.15, queue submit 2.25, slot wait 0.41 batched submits: 1053 flushes, 2.3 command buffers each, over 120 presents; by reason: poll 1, wait 126, slot 0, present 119, cap 0, value 0, other 807 ``` So the frame is ~50 ms wall while the GPU waits 0.4 ms for the submission: the cost is on the CPU/driver side recording and flushing the frame. The submission-side line items alone (submit 24 + commands 6 + descriptors 5) are about 36 of those 50 ms. Roughly 8.8 flushes a frame (1053/120), most of them "other" rather than present/wait/cap - many small command-buffer breaks a frame across 175 passes and 171 barriers. Two things I ruled out on this machine, both negative, so they can be left out of the search: - **Internal resolution barely matters here.** Same scene, live switch via the Resolution row: 50% internal 18.6 fps vs 100% internal (1280x800) 13.7 fps. Pixels are not the wall; an FSR/render-scale fix cannot get the Deck to 30. (For contrast #263 shows DLSS slower than the game's own on an RTX 5090; on the Deck the game's own upscale is already the fast path.) - **Storage**, once on NVMe, is not it either; that is the separate #386 (SD installs re-inflate the draw count because asset streaming never settles). Question for whoever knows this path: on RADV with RX_VK_BATCH already on, is the ~9-flushes-a-frame "other" pattern reducible for a scene with this many passes/barriers, or is there a known knob to hold a frame in fewer/larger submissions? This is the Deck's main frame-rate ceiling; the GPU is idle.

waryxxful referenced this issue

2026-11-14 19:15:26 +00:00

Idea: cut the per-draw cost of the compute fallback (the Market at 2,300 draws: GPU 22.7 ms plus 19 ms of replay submit on an RX 5600 XT) #549

p0lux referenced this issue

2026-11-14 21:29:26 +00:00

perf(vk): three small costs of the replay thread removed: unread bookkeeping without mirrors, 16 vertex bindings a change, a lock a sampled texture #590

p0lux referenced this issue

2026-11-14 21:31:17 +00:00

docs(tested): the committed play profile on a Steam Deck OLED, on the compute path: about 10% fewer instructions a frame #591

mooki referenced this issue

2026-11-15 02:29:03 +00:00

WIP: feat(menu): Don't wait for the GPU, a row for the game's present poll answered at once (off by default) #656

omgitsgela commented

2026-11-15 05:43:59 +00:00

Author

Contributor

More Deck arms for this issue (Steam Deck OLED, same save, Kokiri spawn). Build: main fad1a964 through the dev_packs pipeline (renderer f978d720), install on microSD, so read #386's ~15-20% storage penalty into every row. Options fixed across arms: cap 60, vsync off, AMD FSR 3.1, internal 50% (RX_VK_TEX_* memory switches on), RX_VK_MESH_OPT=1, RX_VK_READBACK_LATE=newest. Protocol fixed: load the save, ~10 idle samples, ~15 s full-speed right-stick spin, ~20 s walk, same script every arm; fps is the bridge's rolling average, min over samples where shown.

arm	idle	spin	walk
Threaded rendering Off, clocks auto (x2)	17.1 / 17.3	17.0 / 17.6	drift*
Threaded rendering On, clocks auto (x2)	16.6 / 15.8	15.6 / 16.7	drift*
GPU pinned high (`power_dpm_force_performance_level=high`), Threaded Off	21.3	21.0	30.0
GPU pinned high, Threaded On	21.3	17.5	26.8
real occlusion counts (`RX_VK_OCC_FIXED=-1`)	17.7 / 17.4	17.6 / 16.9	20.8 / 23.1

* the walk arms end at different headings, so walk columns across arms are not comparable; within-arm they're consistent.

What this says about the bottleneck in this issue:

Raising the GPU's clocks moves the ceiling +23% (17.2 idle to 21.3) even though the host wait on GPU work stays at ~0.1-0.4 ms a frame. So the frame is still CPU-shaped in the sense this issue measures, but the GPU's own clock state is a first-order term on this APU: at auto clocks the GPU takes long enough per frame that the submit thread's next frame arrives into a still-busy machine without ever being counted as a wait. Pinning is free (no visual change) and SteamOS's Game Mode already ships the mechanism; see the last paragraph.
Threaded rendering (#588) is not a Deck win at either clock state - Auto leaving it Off under 12 logical CPUs looks right, measured.
Real per-draw occlusion counts gain nothing here (17.7 vs 17.1-17.3 idle). Unlike the desktop reports, our draw counts do not fall with real counts at these scenes on the Deck (idle draws 2,287 vs 2,043-2,057 - if anything slightly up), which fits this issue's point: the cost is the submit path walking the commands, not the count of objects the GPU skips. A real occlusion pass still costs query buffers and resolves, so the desktop pattern should not be ported expecting a Deck number.
Test farm pending: whether SteamOS's gamemoderun defaults (CPU governor, io priority) recover any of the gap on top of this recipe; and pinned clocks measured against an internal-NVMe install to separate #386 from the clock effect.

One Deck-specific note for whoever reads the ceiling: the two levers this table isolates stack. This run's spin idle is 17.2 at auto clocks on SD; pinning multiplies it 1.23 (measured, this table) and the internal-NVMe install adds the ~1.19 from #386's measurement — so the same arm on internal storage at pinned clocks should land near 25 idle before anything in this issue's scope is touched. That arithmetic is mine, not a measurement; the pin stage against the NVMe install will replace it with a number. What remains after both levers is this issue's actual subject: the ~24 ms submit walk.

More Deck arms for this issue (Steam Deck OLED, same save, Kokiri spawn). Build: main `fad1a964` through the dev_packs pipeline (renderer `f978d720`), install on microSD, so read #386's ~15-20% storage penalty into every row. Options fixed across arms: cap 60, vsync off, AMD FSR 3.1, internal 50% (`RX_VK_TEX_*` memory switches on), `RX_VK_MESH_OPT=1`, `RX_VK_READBACK_LATE=newest`. Protocol fixed: load the save, ~10 idle samples, ~15 s full-speed right-stick spin, ~20 s walk, same script every arm; fps is the bridge's rolling average, min over samples where shown. | arm | idle | spin | walk | |---|---|---|---| | Threaded rendering Off, clocks auto (x2) | 17.1 / 17.3 | 17.0 / 17.6 | drift* | | Threaded rendering On, clocks auto (x2) | 16.6 / 15.8 | 15.6 / 16.7 | drift* | | **GPU pinned high** (`power_dpm_force_performance_level=high`), Threaded Off | **21.3** | **21.0** | 30.0 | | GPU pinned high, Threaded On | 21.3 | 17.5 | 26.8 | | real occlusion counts (`RX_VK_OCC_FIXED=-1`) | 17.7 / 17.4 | 17.6 / 16.9 | 20.8 / 23.1 | \* the walk arms end at different headings, so walk columns across arms are not comparable; within-arm they're consistent. What this says about the bottleneck in this issue: - **Raising the GPU's clocks moves the ceiling +23%** (17.2 idle to 21.3) even though the host wait on GPU work stays at ~0.1-0.4 ms a frame. So the frame is still CPU-shaped in the sense this issue measures, but the GPU's own clock state is a first-order term on this APU: at auto clocks the GPU takes long enough per frame that the submit thread's next frame arrives into a still-busy machine without ever being counted as a wait. Pinning is free (no visual change) and SteamOS's Game Mode already ships the mechanism; see the last paragraph. - **Threaded rendering (#588) is not a Deck win at either clock state** - Auto leaving it Off under 12 logical CPUs looks right, measured. - **Real per-draw occlusion counts gain nothing here** (17.7 vs 17.1-17.3 idle). Unlike the desktop reports, our draw counts do not fall with real counts at these scenes on the Deck (idle draws 2,287 vs 2,043-2,057 - if anything slightly up), which fits this issue's point: the cost is the submit path walking the commands, not the count of objects the GPU skips. A real occlusion pass still costs query buffers and resolves, so the desktop pattern should not be ported expecting a Deck number. - Test farm pending: whether SteamOS's `gamemoderun` defaults (CPU governor, io priority) recover any of the gap on top of this recipe; and pinned clocks measured against an internal-NVMe install to separate #386 from the clock effect. One Deck-specific note for whoever reads the ceiling: the two levers this table isolates stack. This run's spin idle is 17.2 at auto clocks on SD; pinning multiplies it 1.23 (measured, this table) and the internal-NVMe install adds the ~1.19 from #386's measurement — so the same arm on internal storage at pinned clocks should land near 25 idle before anything in this issue's scope is touched. That arithmetic is mine, not a measurement; the pin stage against the NVMe install will replace it with a number. What remains after both levers is this issue's actual subject: the ~24 ms submit walk.

omgitsgela commented

2026-11-15 06:08:27 +00:00

Author

Contributor

Finishing the launch-state levers on this issue (same build, save, protocol as the table above):

gamemoderun (SteamOS gamemode 1.8.2): idle 16.8 / 17.0 against 17.1 / 17.3 for plain launch - nothing. That gamemode build's own strings say "no cpu governors found" on this image, and it does not touch GPU clocks.
CPU governor performance (amd-pstate-epp, 8 policies, GPU left at auto): idle 17.9 / 18.0 - about +4%, inside noise on the spin arms (15.5 / 18.6 against 17.0 / 17.6). The submit walk this issue measures is single-threaded work per frame; the governor mainly changes how quickly a thread wakes into boost, not the walk itself.
Both were set and restored from desktop mode (sysfs power_dpm_force_performance_level / scaling_governor); the arms ran with the same wrapper as the rows above.

So on the Deck the frame's shape is: one first-order launch-state lever (GPU clock, +23% here and again +30% dukenheimer measured in Game Mode), a small second (governor), and this issue's ~24 ms submit walk underneath both. Filed separately as a Deck launch proposal with the numbers table.

Finishing the launch-state levers on this issue (same build, save, protocol as the table above): - `gamemoderun` (SteamOS gamemode 1.8.2): idle 16.8 / 17.0 against 17.1 / 17.3 for plain launch - **nothing**. That gamemode build's own strings say "no cpu governors found" on this image, and it does not touch GPU clocks. - CPU governor `performance` (amd-pstate-epp, 8 policies, GPU left at auto): idle 17.9 / 18.0 - **about +4%**, inside noise on the spin arms (15.5 / 18.6 against 17.0 / 17.6). The submit walk this issue measures is single-threaded work per frame; the governor mainly changes how quickly a thread wakes into boost, not the walk itself. - Both were set and restored from desktop mode (sysfs `power_dpm_force_performance_level` / `scaling_governor`); the arms ran with the same wrapper as the rows above. So on the Deck the frame's shape is: one first-order launch-state lever (GPU clock, +23% here and again +30% dukenheimer measured in Game Mode), a small second (governor), and this issue's ~24 ms submit walk underneath both. Filed separately as a Deck launch proposal with the numbers table.

omgitsgela referenced this issue

2026-11-15 06:09:41 +00:00

Deck: launching from desktop mode gets no performance treatment (GPU auto, governor powersave); pinning the GPU clock is +23% measured #716

omgitsgela commented

2026-11-15 06:09:54 +00:00

Author

Contributor

The launch-state proposal with this table is now #716 (Deck: launching from desktop mode gets no performance treatment).

omgitsgela referenced this issue

2026-11-15 06:31:39 +00:00

Deck: launching from desktop mode gets no performance treatment (GPU auto, governor powersave); pinning the GPU clock is +23% measured #716

omgitsgela commented

2026-11-15 07:46:05 +00:00

Author

Contributor

Correction on my Threaded rendering arms in the table above, plus the rerun done right. What happened: the ini row threaded_rendering= sits at its default and the options writer drops default rows, so my sed never applied - the four "On" arms logged Auto: off and were Off. That part of the table is void, sorry. The rerun used the env overrides instead (RX_VK_ASYNC=1 = the menu's On row exactly per vk_api.h:68; and RX_VK_RECORDER=thread on top, which the menu cannot select), four+ arms interleaved, same build/protocol/storage:

config	idle	spin	walk
Off (Auto), three arms	18.2 / 17.9 / 21.3	19.8 / 18.9 / 22.1	28.5 / 29.1 / 30.9
menu On: `RX_VK_ASYNC=1` only, two arms	18.6 / 18.1	19.2 / 20.3	31.9 / 32.5
async + recorder thread: `RX_VK_ASYNC=1 RX_VK_RECORDER=thread`, two arms	15.7 / 15.7	15.0 / 14.9	25.3 / 18.7

Reading: the menu's On row is parity at best on the Deck (idle 18.1-18.6 inside the Off arms' 17.9-21.3 spread; walk maybe a hair better, draws lean lower so call it a wash). The recorder-thread configuration is reliably worse: -14% idle/spin in both arms - on this APU, packing vkCmd* into a ring for a recorder thread costs more than it overlaps. That is worth keeping out of any Auto rule for 8-logical-CPU AMDs even as step 2 matures. Caveat on the Off spread itself: 17.9 to 21.3 across three identical arms is a warning that single arms here carry several fps of machine-state variance; the paired rerun of the clock-pin A/B (issue #716) is in flight before anyone quotes +23% as settled.

Correction on my Threaded rendering arms in the table above, plus the rerun done right. What happened: the ini row `threaded_rendering=` sits at its default and the options writer drops default rows, so my sed never applied - the four "On" arms logged `Auto: off` and were Off. That part of the table is void, sorry. The rerun used the env overrides instead (`RX_VK_ASYNC=1` = the menu's On row exactly per vk_api.h:68; and `RX_VK_RECORDER=thread` on top, which the menu cannot select), four+ arms interleaved, same build/protocol/storage: | config | idle | spin | walk | |---|---|---|---| | Off (Auto), three arms | 18.2 / 17.9 / 21.3 | 19.8 / 18.9 / 22.1 | 28.5 / 29.1 / 30.9 | | menu On: `RX_VK_ASYNC=1` only, two arms | 18.6 / 18.1 | 19.2 / 20.3 | 31.9 / 32.5 | | async + recorder thread: `RX_VK_ASYNC=1 RX_VK_RECORDER=thread`, two arms | 15.7 / 15.7 | 15.0 / 14.9 | 25.3 / 18.7 | Reading: the menu's On row is parity at best on the Deck (idle 18.1-18.6 inside the Off arms' 17.9-21.3 spread; walk maybe a hair better, draws lean lower so call it a wash). The **recorder-thread configuration is reliably worse: -14% idle/spin in both arms** - on this APU, packing vkCmd* into a ring for a recorder thread costs more than it overlaps. That is worth keeping out of any Auto rule for 8-logical-CPU AMDs even as step 2 matures. Caveat on the Off spread itself: 17.9 to 21.3 across three identical arms is a warning that single arms here carry several fps of machine-state variance; the paired rerun of the clock-pin A/B (issue #716) is in flight before anyone quotes +23% as settled.

p0lux referenced this issue

2026-11-15 09:05:46 +00:00

perf(vk): six small costs of a draw and of a submission's parts removed #391

omgitsgela commented

2026-11-15 09:22:05 +00:00

Author

Contributor

Milestone: best-playable Deck configuration, measured numbers. Angela asked for this written up for the devs. Deck LCD (Jupiter/Vangogh), game on the SD card, Kokiri Village load scenario (top save, ~2 min settle, idle → full-speed camera spin → forward walk).

Runtime: dev_packs runtime built from merged main a4821d5674bad9... (so #488 movie pacing + #399 skip-title + #588 + #238 all in), host aabc168d9ded, renderer f1e064e34a99. Settings, exactly as run: ini quality_preset=1 handheld_profile=1 resolution=4 upscaling=2 (FSR3.1, 50%) sharpness=30 frame_rate_cap=3 vsync=0 frame_pacing=1 threaded_rendering=0 dlss=0 anisotropic_filtering=4 vram_budget=1 at 1280x800; env RX_VK_READBACK_LATE=newest RX_VK_MESH_OPT=1 RX_VK_BC_LAYERS=8 RX_VK_TEX_MAX_DIM=512 RX_VK_TEX_SKIP_MB=8 HOS_SHORT_SLEEP_TIMER=1; plus the launch-state from issue #716: power_dpm_force_performance_level=high (sclk locked 1600 MHz) and cpu governors performance. Threaded confirmed Auto: off in each arm log.

Three same-config arms (two from the interleaved clock pair, both pinned):

arm	idle	spin	walk	frame delay idle/spin/walk (ms)
cp1	23.5 med (12.8-44.0)	28.4 med (24.4-30.4)	28.4 med (24.5-39.3)	1890 / 985 / 931
cp2	24.5 med (21.6-26.5)	25.0 med (20.6-27.5)	26.0 (23.6-28.4, n=2)	2126 / 1903 / 703
m2	25.4 med (20.5-26.4)	26.9 med (23.5-28.5)	37.2 med (33.3-38.3)	2122 / 1056 / 470

Steady-state (load-in samples with >2400 ms frame delay dropped): spin 17/21 samples >=25 fps, walk 29/32 >=25, idle medians 23.4-25.4 with p25 at 22.4+. So the steady 25-30 band is met for spin and walk on SD storage; idle sits at the bottom edge. Compare the same protocol on the pre-#488 sync runtime at auto clocks: idle ~17-18, spin ~19, walk ~28-31. Frame delay (game draw -> present backlog) drops from ~2 s idle to ~0.5-0.9 s under load, which matches the CPU-bound submit picture from this issue.

Honest caveats: byte-identical arms spread ±2-3 fps at idle and walk swings with streaming distance (draws 400-2500), so bands not points; lowest idle samples are load-in tail; all SD - #386's ~15-20% NVMe penalty still applies. One config note: the game logs vi: frame rate cap 144 for ini frame_rate_cap=3 - the menu row label and the ini value disagree with what I assumed for weeks (we have been quoting "cap 60" rows that were actually 144 + vsync off). Worth checking the menu mapping.

Movie check on this exact runtime: intro 5947 pictures ... 4.0 ms decode, 874 skipped to keep up with its sound - pacing code present in binary; dedicated skip-count arms rerunning now.

**Milestone: best-playable Deck configuration, measured numbers.** Angela asked for this written up for the devs. Deck **LCD** (Jupiter/Vangogh), game on the SD card, Kokiri Village load scenario (top save, ~2 min settle, idle → full-speed camera spin → forward walk). Runtime: dev_packs runtime built from merged main `a4821d5674bad9...` (so #488 movie pacing + #399 skip-title + #588 + #238 all in), host `aabc168d9ded`, renderer `f1e064e34a99`. Settings, exactly as run: ini `quality_preset=1 handheld_profile=1 resolution=4 upscaling=2 (FSR3.1, 50%) sharpness=30 frame_rate_cap=3 vsync=0 frame_pacing=1 threaded_rendering=0 dlss=0 anisotropic_filtering=4 vram_budget=1` at 1280x800; env `RX_VK_READBACK_LATE=newest RX_VK_MESH_OPT=1 RX_VK_BC_LAYERS=8 RX_VK_TEX_MAX_DIM=512 RX_VK_TEX_SKIP_MB=8 HOS_SHORT_SLEEP_TIMER=1`; plus the launch-state from issue #716: `power_dpm_force_performance_level=high` (sclk locked 1600 MHz) and cpu governors `performance`. Threaded confirmed `Auto: off` in each arm log. Three same-config arms (two from the interleaved clock pair, both pinned): | arm | idle | spin | walk | frame delay idle/spin/walk (ms) | |---|---|---|---|---| | cp1 | 23.5 med (12.8-44.0) | 28.4 med (24.4-30.4) | 28.4 med (24.5-39.3) | 1890 / 985 / 931 | | cp2 | 24.5 med (21.6-26.5) | 25.0 med (20.6-27.5) | 26.0 (23.6-28.4, n=2) | 2126 / 1903 / 703 | | m2 | 25.4 med (20.5-26.4) | 26.9 med (23.5-28.5) | 37.2 med (33.3-38.3) | 2122 / 1056 / 470 | Steady-state (load-in samples with >2400 ms frame delay dropped): spin 17/21 samples >=25 fps, walk 29/32 >=25, idle medians 23.4-25.4 with p25 at 22.4+. So the steady 25-30 band is met for spin and walk on SD storage; idle sits at the bottom edge. Compare the same protocol on the pre-#488 sync runtime at auto clocks: idle ~17-18, spin ~19, walk ~28-31. Frame delay (game draw -> present backlog) drops from ~2 s idle to ~0.5-0.9 s under load, which matches the CPU-bound submit picture from this issue. Honest caveats: byte-identical arms spread ±2-3 fps at idle and walk swings with streaming distance (draws 400-2500), so bands not points; lowest idle samples are load-in tail; all SD - #386's ~15-20% NVMe penalty still applies. One config note: the game logs `vi: frame rate cap 144` for ini `frame_rate_cap=3` - the menu row label and the ini value disagree with what I assumed for weeks (we have been quoting "cap 60" rows that were actually 144 + vsync off). Worth checking the menu mapping. Movie check on this exact runtime: intro `5947 pictures ... 4.0 ms decode, 874 skipped to keep up with its sound` - pacing code present in binary; dedicated skip-count arms rerunning now.

omgitsgela commented

2026-11-15 10:18:15 +00:00

Author

Contributor

Correction + mechanism, from the repo's own sampler on the Deck. The scene benchmark and a 60-second HOS_PROFILE=5 HOS_SAMPLE_ALL run on the exact Kokiri Village hot spot (rt-main a4821d56, GPU pinned 1600 MHz + governors performance, SD storage) change the picture: that spot is not "out of CPU" - it is bound on a serial wait chain around presentation, with ~7 of 8 cores idle.

The arm itself first (best Kokiri numbers this Deck has produced, and Angela watching live called it "excellent, even better than our last milestone"): idle mean 27.9 (10 samples), spin 27.4, walk 31.2 - the 23-25 idle band from my earlier arms was the same config, so treat pinned Kokiri as ~24-28 and this profile as the mechanism behind it.

Profile facts (278k raw stack lines, resolved against the exact rt-main binaries with RX_MAPS; classifier on request):

89% of all samples have a wait frame somewhere in the stack. The single dominant stack (213k weighted repeats) is syscall <- longshot:SleepConditionVariableSRW+0x7b <- hos_cv_sleep <- nn::os::ReceiveLightMessageQueue / queue_get: the game's own threads parked on message queues and events.
Total on-CPU across the whole process ≈ 1.1 cores of 8. Per-thread busy during ~28 fps idle: Presentation_Thread 39% of wall, sead::MainThread 36%, five ModuleSystemWorkers ~25% each, the game's BaseProcExecutor0-3 ~18% each. The renderer's own mkw_vk_worker, mkw_vk_capture_, mkw_vk_order, mkw_vk_gpu_hang threads sampled inside sem_trywait / Sleep / sync_watch / copy_helper_main - parked, waiting for work.
Top on-CPU leaves, in order: mkwvk::build_descriptors (300 samples), collect_in (207), mkwvk::draw (168), mkwvk::program_for (110), mkwvk::replay (96), then a long tail of lifted-game f_* singletons and a64_fpx_* exact-float helpers. Caller context below them: site_call, n_nvnCommandBufferEndRecording, n_nvnCommandBufferAddControlMemory, run_submit, memory_capture_submit.

So "CPU-bound submit path" needs a qualifier I should have used from the start: the submit path IS where the on-CPU time lives, and the micro-costs in this issue's scope are real (descriptor building and collect_in are literally the top leaves) - but no core is anywhere near saturated at 28 fps. The frame time is dominated by waits: the game's MainThread sleeping a large slice of every frame (the #73 chain analysis measured ~8.6 ms of its own loop as sleep - same signature), and a ~2 s deep present backlog draining at 28/s behind that cadence. Cutting per-draw costs buys headroom inside the serial budget; the structural win is wherever the frame's sleep/wait cadence is set.

Two things I can contribute from here: (1) testing p0lux's #567/#590 as runtimes on this exact save + protocol right now - they target exactly these top leaves, and the Hyrule Town spin stall (#750: 2-6 fps, GPU-written 691x432 texture re-uploaded per present, 714,844-page capture) is where the page-capture cost is gross, not marginal; (2) if anyone wants a same-window HOS_THREAD_TIMES or presents.csv cross-check before trusting one 60 s sample, ask and I'll run it on the Deck (idle ±1, walk ±7 noise band applies).

Hardware note for all of this: Steam Deck LCD (Jupiter, VANGOGH, LPDDR4X-4266), not OLED - draw/cost structures should match OLED, absolute fps and memory-bound margins will not.

Correction + mechanism, from the repo's own sampler on the Deck. The scene benchmark and a 60-second `HOS_PROFILE=5 HOS_SAMPLE_ALL` run on the exact Kokiri Village hot spot (rt-main `a4821d56`, GPU pinned 1600 MHz + governors performance, SD storage) change the picture: **that spot is not "out of CPU" - it is bound on a serial wait chain around presentation, with ~7 of 8 cores idle.** The arm itself first (best Kokiri numbers this Deck has produced, and Angela watching live called it "excellent, even better than our last milestone"): idle mean **27.9** (10 samples), spin 27.4, walk 31.2 - the 23-25 idle band from my earlier arms was the same config, so treat pinned Kokiri as ~24-28 and this profile as the mechanism behind it. Profile facts (278k raw stack lines, resolved against the exact rt-main binaries with `RX_MAPS`; classifier on request): - **89% of all samples have a wait frame somewhere in the stack.** The single dominant stack (213k weighted repeats) is `syscall` <- `longshot:SleepConditionVariableSRW+0x7b` <- `hos_cv_sleep` <- `nn::os::ReceiveLightMessageQueue` / `queue_get`: the game's own threads parked on message queues and events. - Total on-CPU across the whole process ≈ **1.1 cores of 8**. Per-thread busy during ~28 fps idle: `Presentation_Thread` 39% of wall, `sead::MainThread` 36%, five `ModuleSystemWorker`s ~25% each, the game's `BaseProcExecutor0-3` ~18% each. The renderer's own `mkw_vk_worker`, `mkw_vk_capture_`, `mkw_vk_order`, `mkw_vk_gpu_hang` threads sampled inside `sem_trywait` / `Sleep` / `sync_watch` / `copy_helper_main` - parked, waiting for work. - Top on-CPU leaves, in order: `mkwvk::build_descriptors` (300 samples), `collect_in` (207), `mkwvk::draw` (168), `mkwvk::program_for` (110), `mkwvk::replay` (96), then a long tail of lifted-game `f_*` singletons and `a64_fpx_*` exact-float helpers. Caller context below them: `site_call`, `n_nvnCommandBufferEndRecording`, `n_nvnCommandBufferAddControlMemory`, `run_submit`, `memory_capture_submit`. So "CPU-bound submit path" needs a qualifier I should have used from the start: the submit path IS where the on-CPU time lives, and the micro-costs in this issue's scope are real (descriptor building and collect_in are literally the top leaves) - but no core is anywhere near saturated at 28 fps. The frame time is dominated by waits: the game's MainThread sleeping a large slice of every frame (the #73 chain analysis measured ~8.6 ms of its own loop as sleep - same signature), and a ~2 s deep present backlog draining at 28/s behind that cadence. Cutting per-draw costs buys headroom inside the serial budget; the structural win is wherever the frame's sleep/wait cadence is set. Two things I can contribute from here: (1) testing p0lux's #567/#590 as runtimes on this exact save + protocol right now - they target exactly these top leaves, and the Hyrule Town spin stall (#750: 2-6 fps, GPU-written 691x432 texture re-uploaded per present, 714,844-page capture) is where the page-capture cost is gross, not marginal; (2) if anyone wants a same-window `HOS_THREAD_TIMES` or presents.csv cross-check before trusting one 60 s sample, ask and I'll run it on the Deck (idle ±1, walk ±7 noise band applies). Hardware note for all of this: **Steam Deck LCD** (Jupiter, VANGOGH, LPDDR4X-4266), not OLED - draw/cost structures should match OLED, absolute fps and memory-bound margins will not.

dukenheimer commented

2026-11-15 17:37:50 +00:00

Contributor

Our Deck data on this issue's question (Steam Deck OLED, stock clocks, docked Game Mode, the moving Kokiri route, 2 interleaved runs per arm unless noted).

Where the frame went (HOS_CHAIN, one run, the Longshot user build 92640867 = main e7ed2f7a + Deck patches; output 2048x1152, FSR 3.1 ~50%, frame generation 2x):

The critical path was 72.4 ms a frame (14.7 real fps).
18.0 ms of it was the game's job threads sleep-polling (lr 0x8009a478) for a GPU result, the culling camera block at +0x3980. Each read waited 18.9 ms, for the end of its whole submission.
The rest was the Presentation thread (the game's render work plus the inline replay) and sead::MainThread, in series. That's the same serial chain your sampler shows.

The readback switches (one build: main cf605c85 + #225 + #223 + #303; the player's settings: Handheld, 1536x864 at 100% render, frame generation 3x):

	nothing set	`RX_VK_READBACK_LATE=newest` (#238, on main)	`RX_VK_READBACK_EARLY=1` (#303)
real fps	10.8, 11.4	12.0, 11.6	12.6, 12.8
a readback that waits	~23 ms	13-15 ms	~1.5 ms
GPU busy	71%	74-75%	80-81%

EARLY isn't usable. Its correctness probe on the Deck found 2-4% of the checked camera-block reads came back before a later GPU write that main waits for (#303 (comment)). RX_VK_READBACK_LATE=newest is the switch to use.
Where EARLY's wait went: with it, sead::MainThread spent ~20 ms a frame on the path waiting in the capture for late pages (HOS_CHAIN on the installed build).
Taking that wait out (RX_VK_CAPTURE_WAIT=0) gave nothing: 13.35 → 12.85 real at these settings, and 14.4 → 14.5 at 1280x800 with the upscaler off. The readbacks waited instead: 2,000-2,500 of 3,000 against ~900.

On "GPU-bound wait 0.4 ms": that figure counts only waits for a free recording slot (the log says (recording slots only)).

Our GPU time lines read 46-55 ms of GPU a frame at 1280x800 native, and 53-59 ms at the player's 1536x864.
FSR 3.1 Quality at stock took the GPU from ~55 to 43-46 ms, and fps from 13.5 to 15.25 real. Below that, the CPU chain set the frame.
So at 100% render the Deck's GPU is close to the frame time, and at 50% the serial chain is what's left, as you found.

Our Deck data on this issue's question (Steam Deck OLED, stock clocks, docked Game Mode, the moving Kokiri route, 2 interleaved runs per arm unless noted). **Where the frame went** (`HOS_CHAIN`, one run, the Longshot user build 92640867 = main e7ed2f7a + Deck patches; output 2048x1152, FSR 3.1 ~50%, frame generation 2x): - The critical path was 72.4 ms a frame (14.7 real fps). - 18.0 ms of it was the game's job threads sleep-polling (lr 0x8009a478) for a GPU result, the culling camera block at +0x3980. Each read waited 18.9 ms, for the end of its whole submission. - The rest was the Presentation thread (the game's render work plus the inline replay) and `sead::MainThread`, in series. That's the same serial chain your sampler shows. **The readback switches** (one build: main cf605c85 + #225 + #223 + #303; the player's settings: Handheld, 1536x864 at 100% render, frame generation 3x): | | nothing set | `RX_VK_READBACK_LATE=newest` (#238, on main) | `RX_VK_READBACK_EARLY=1` (#303) | |---|---|---|---| | real fps | 10.8, 11.4 | 12.0, 11.6 | 12.6, 12.8 | | a readback that waits | ~23 ms | 13-15 ms | ~1.5 ms | | GPU busy | 71% | 74-75% | 80-81% | - **EARLY isn't usable.** Its correctness probe on the Deck found 2-4% of the checked camera-block reads came back before a later GPU write that main waits for (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/303#issuecomment-23944). `RX_VK_READBACK_LATE=newest` is the switch to use. - **Where EARLY's wait went:** with it, `sead::MainThread` spent ~20 ms a frame on the path waiting in the capture for late pages (`HOS_CHAIN` on the installed build). - **Taking that wait out** (`RX_VK_CAPTURE_WAIT=0`) gave nothing: 13.35 → 12.85 real at these settings, and 14.4 → 14.5 at 1280x800 with the upscaler off. The readbacks waited instead: 2,000-2,500 of 3,000 against ~900. **On "GPU-bound wait 0.4 ms":** that figure counts only waits for a free recording slot (the log says `(recording slots only)`). - Our `GPU time` lines read 46-55 ms of GPU a frame at 1280x800 native, and 53-59 ms at the player's 1536x864. - FSR 3.1 Quality at stock took the GPU from ~55 to 43-46 ms, and fps from 13.5 to 15.25 real. Below that, the CPU chain set the frame. - So at 100% render the Deck's GPU is close to the frame time, and at 50% the serial chain is what's left, as you found.