RADV (Radeon 8060S / Linux): each vkQueueSubmit2 costs ~2 ms on the replay thread, which limits heavy scenes to ~42 fps #789

Closed

opened 2026-11-08 21:21:00 +00:00 by myylkn · 1 comment

myylkn commented

2026-11-08 21:21:00 +00:00

(Migrated from github.com)

The idea

Follow-up to #413 (the batched submits). Thanks for v2.0.2: the capture wait (ea78039) removed the per-frame present wait without the flashes. Here are measurements on v2.0.2 (5c9b0bf).

System: GPD WIN 5, Ryzen AI Max+ 395 / Radeon 8060S (RDNA 3.5), Bazzite, Mesa/RADV 26.2.6, 80 W, power_dpm_force_performance_level=high.
env: RX_VK_IMPORT_CHUNK_MB=4, RX_VK_BATCH_POLL_US=2000, RX_VK_MESH_OPT=1, RX_VK_WAIT_GPU=1

What limits it

The Presentation Thread. In heavy Kokiri Forest scenes it runs ~42 fps, with ~11 ms of GPU time and ~22 ms per frame. Of that frame:

~6 ms recording (draw, descriptors, commands)
~6–7 ms in queue submit: ~2.6–3 batched flushes a frame (wait ≈1.7, present ≈1, poll ≈0.3), each ~2 ms in vkQueueSubmit2.

Resolution doesn't change the fps.

What we tried

RX_VK_BATCH_MAX=8 (mid-frame flushes) on honest-eta: the GPU starts earlier (present wait 11 → 6.6 ms), but 4 flushes × 2 ms cost more. 38.6 vs 41.1 fps.
Submitting from a separate thread (a local patch: the held batches go out on their own thread under a queue mutex, and every other queue user drains it first): the replay's submit cost dropped to ~1–2 ms a frame, and fps went up to 45–49. But it flashes whenever the game thread isn't held for the submit, or the GPU gets work mid-frame: a whole frame blown out to white, the HUD hearts black. That looks like the same CPU-ahead race as the exposure meter, on other pages. With the waiter held until its batch is sent there are no flashes, but the gain drops to ~+2 fps. I can share the patch for reference if useful; it is not meant to be merged as is.
RX_VK_MIRROR=1: 38 fps (66 MB written back a frame on an APU).
HOOKSHOT_OCCLUSION_OFF=1, CHUNK_MB=8, BATCH_MAX=64: within noise.
Without RX_VK_WAIT_GPU=1, v2.0.2 still shows occasional flicker on this machine. With it set to 1 the flicker is gone, at no measurable fps cost.

Questions

Is the ~2 ms per submit the amdgpu kernel revalidating the userptr imports (~90–100 live, ~900 MB)? Could fewer or larger imports, or a submit path that doesn't carry them each time, help?
Could RX_VK_WAIT_GPU=1 be the default on Linux/RADV, since the capture wait doesn't cover every page there?

Logs for each run are available on request.

Anything else

Investigation, measurements and the experimental patch were done with Claude (Anthropic's AI assistant).

### The idea Follow-up to #413 (the batched submits). Thanks for v2.0.2: the capture wait (ea78039) removed the per-frame present wait without the flashes. Here are measurements on v2.0.2 (5c9b0bf). **System:** GPD WIN 5, Ryzen AI Max+ 395 / Radeon 8060S (RDNA 3.5), Bazzite, Mesa/RADV 26.2.6, 80 W, `power_dpm_force_performance_level=high`. **env:** `RX_VK_IMPORT_CHUNK_MB=4`, `RX_VK_BATCH_POLL_US=2000`, `RX_VK_MESH_OPT=1`, `RX_VK_WAIT_GPU=1` ### What limits it The Presentation Thread. In heavy Kokiri Forest scenes it runs ~42 fps, with ~11 ms of GPU time and ~22 ms per frame. Of that frame: - ~6 ms recording (draw, descriptors, commands) - **~6–7 ms in `queue submit`**: ~2.6–3 batched flushes a frame (wait ≈1.7, present ≈1, poll ≈0.3), each **~2 ms** in vkQueueSubmit2. Resolution doesn't change the fps. ### What we tried - `RX_VK_BATCH_MAX=8` (mid-frame flushes) on honest-eta: the GPU starts earlier (present wait 11 → 6.6 ms), but 4 flushes × 2 ms cost more. 38.6 vs 41.1 fps. - **Submitting from a separate thread** (a local patch: the held batches go out on their own thread under a queue mutex, and every other queue user drains it first): the replay's submit cost dropped to ~1–2 ms a frame, and fps went up to **45–49**. **But it flashes** whenever the game thread isn't held for the submit, or the GPU gets work mid-frame: a whole frame blown out to white, the HUD hearts black. That looks like the same CPU-ahead race as the exposure meter, on other pages. With the waiter held until its batch is sent there are no flashes, but the gain drops to ~+2 fps. I can share the patch for reference if useful; it is not meant to be merged as is. - `RX_VK_MIRROR=1`: 38 fps (66 MB written back a frame on an APU). - `HOOKSHOT_OCCLUSION_OFF=1`, `CHUNK_MB=8`, `BATCH_MAX=64`: within noise. - Without `RX_VK_WAIT_GPU=1`, v2.0.2 still shows occasional flicker on this machine. With it set to 1 the flicker is gone, at no measurable fps cost. ### Questions 1. Is the ~2 ms per submit the amdgpu kernel revalidating the userptr imports (~90–100 live, ~900 MB)? Could fewer or larger imports, or a submit path that doesn't carry them each time, help? 2. Could `RX_VK_WAIT_GPU=1` be the default on Linux/RADV, since the capture wait doesn't cover every page there? Logs for each run are available on request. ### Anything else Investigation, measurements and the experimental patch were done with Claude (Anthropic's AI assistant).

NotRexed commented

2026-11-09 05:57:04 +00:00

(Migrated from github.com)

Tracked in #1000, follow there. That issue collects every report of Linux / Steam Deck GPU hangs, crashes, low fps and build failures, and this one is linked from it with its GPU, version and log details. If it still happens on v2.0.6 (https://github.com/NotRexed/Hookshot/releases/tag/v2.0.6; Update next to Play in the launcher), please add a comment there with your hookshot_log.txt.

Commenting is not possible because the repository is archived.