Perf: possible performance improvement leads #1020

Open
opened 2026-11-16 02:57:42 +00:00 by HumanBean · 3 comments
Owner

A place to collect leads for making frames faster: where profiling shows frame time going, what has been tried, and what has been ruled out. Comment with a lead (the area, the hardware, what the profile showed, and how much of the frame it is), or with a result when you try one. When someone takes a lead, they say so here and link the pull request.

The first leads, from profiling Kokiri Forest, Hyrule Field and the Market on Linux (RTX 3090, Ryzen 9 7900X), are in the first comment. Related: #410 (CPU-side bottleneck in Kokiri Forest), #766 (the first fix from it, merged).

A place to collect leads for making frames faster: where profiling shows frame time going, what has been tried, and what has been ruled out. Comment with a lead (the area, the hardware, what the profile showed, and how much of the frame it is), or with a result when you try one. When someone takes a lead, they say so here and link the pull request. The first leads, from profiling Kokiri Forest, Hyrule Field and the Market on Linux (RTX 3090, Ryzen 9 7900X), are in the first comment. Related: [#410](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/410) (CPU-side bottleneck in Kokiri Forest), [#766](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/766) (the first fix from it, merged).
Author
Owner

Leads from profiling on Linux (RTX 3090, NVIDIA 620.70.06, Ryzen 9 7900X with the game on 12 of its 24 threads, CachyOS; main a00c485c, Threaded rendering Auto (on), uncapped, 1280x720 output). Kokiri Forest, Hyrule Field and the Market are all limited by the CPU there: GPU time is 4.5 to 8.7 ms of a 9 to 12.5 ms frame. The render thread is not the limit either. The frame's critical path is the game's main thread waiting on its job workers (ModuleSystemWorkers, BaseProcExecutors).

Ranked by what I'd expect each to gain:

  1. The rest of the recordings' copy lock (copy_lock, host/nvn_native.c). After #766, the job workers still spend ~4% of their running samples waiting on it: about half under nvnCommandBufferEndRecording, about half under AddControlMemory → release_copies, both exclusive. On Linux the SRW shim (host/posix/windows.h) maps SRW locks to pthread_rwlock, which sleeps on the futex at once, where Windows' SRW lock spins first. A short spin in the shim would help every SRW lock on Linux. Holding the lock for less time in EndRecording would cut it further. Guess: 1-2% in CPU-bound areas.
  2. Renderer capture work on the game's threads: ~10% of the critical path in the Market (capture_make, copy_part, collect_in; ~0.2 ms per submit, ~6 submits a frame). The default 4 capture helper threads are already best: RX_VK_CAPTURE_THREADS=0 was 0.254 vs 0.195 ms per submit with lower fps, and 2 was slower too. The gain would have to come from copying less per submit or merging submits. It is the biggest item, but it is in the renderer and riskier.
  3. The presenter thread, where the render thread is the limit. On NVIDIA, vkQueuePresentKHR blocks the render thread for 3-4 ms a frame. RX_VK_PRESENT_THREAD=1 freed ~3 ms of it (idle 1.3 → 4.2 ms a frame) but moved fps by under 2% here, because this PC is limited by the game threads. With Threaded rendering off, on 8 threads or fewer, and maybe on the Steam Deck, it could matter more. It is off for swapchain external-sync and a 5 s drain; one idea is to drain the presenter before vkAcquireNextImageKHR in present_image. First step: measure with Threaded rendering off to see if it is worth it.
  4. obj_lookup: ~4.4% of the critical path in the Market. Not looked into yet: what it looks up, and whether it can be cached.

Ruled out:

  • The number of capture helper threads (above).
  • The game's short sleeps: on Linux a 20 µs sleep takes ~23.5 µs, so there is little to win there.
  • rx_nvn_invalidate → wait_rec_seq (~2.6% of the workers' time): a real wait for a GPU readback.
  • A hot list or PGO profile of the lifted game code made from the game binary: it is made from the game, so it can't ship to players (see #505).

How it was measured: HOS_SAMPLE_ALL (every game thread every 2 ms) and HOS_CHAIN for the critical path, frame times from RX_VK_FRAME_LOG with HOS_FPS_MAX=240; in each area 15 s standing and 20 s turning the camera at full speed. One gotcha: on Linux, prof.waiting doesn't count pthread_rwlock_*lock futex sleeps, so they show up as "running" libc time. That is how the copy lock waits were found.

**Leads from profiling on Linux** (RTX 3090, NVIDIA 620.70.06, Ryzen 9 7900X with the game on 12 of its 24 threads, CachyOS; main a00c485c, Threaded rendering Auto (on), uncapped, 1280x720 output). Kokiri Forest, Hyrule Field and the Market are all limited by the CPU there: GPU time is 4.5 to 8.7 ms of a 9 to 12.5 ms frame. The render thread is not the limit either. The frame's critical path is the game's main thread waiting on its job workers (ModuleSystemWorkers, BaseProcExecutors). Ranked by what I'd expect each to gain: 1. **The rest of the recordings' copy lock** (`copy_lock`, `host/nvn_native.c`). After [#766](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/766), the job workers still spend ~4% of their running samples waiting on it: about half under `nvnCommandBufferEndRecording`, about half under `AddControlMemory` → `release_copies`, both exclusive. On Linux the SRW shim (`host/posix/windows.h`) maps SRW locks to `pthread_rwlock`, which sleeps on the futex at once, where Windows' SRW lock spins first. A short spin in the shim would help every SRW lock on Linux. Holding the lock for less time in EndRecording would cut it further. Guess: 1-2% in CPU-bound areas. 2. **Renderer capture work on the game's threads**: ~10% of the critical path in the Market (`capture_make`, `copy_part`, `collect_in`; ~0.2 ms per submit, ~6 submits a frame). The default 4 capture helper threads are already best: `RX_VK_CAPTURE_THREADS=0` was 0.254 vs 0.195 ms per submit with lower fps, and 2 was slower too. The gain would have to come from copying less per submit or merging submits. It is the biggest item, but it is in the renderer and riskier. 3. **The presenter thread, where the render thread is the limit.** On NVIDIA, `vkQueuePresentKHR` blocks the render thread for 3-4 ms a frame. `RX_VK_PRESENT_THREAD=1` freed ~3 ms of it (idle 1.3 → 4.2 ms a frame) but moved fps by under 2% here, because this PC is limited by the game threads. With Threaded rendering off, on 8 threads or fewer, and maybe on the Steam Deck, it could matter more. It is off for swapchain external-sync and a 5 s drain; one idea is to drain the presenter before `vkAcquireNextImageKHR` in `present_image`. First step: measure with Threaded rendering off to see if it is worth it. 4. **`obj_lookup`**: ~4.4% of the critical path in the Market. Not looked into yet: what it looks up, and whether it can be cached. **Ruled out:** - The number of capture helper threads (above). - The game's short sleeps: on Linux a 20 µs sleep takes ~23.5 µs, so there is little to win there. - `rx_nvn_invalidate` → `wait_rec_seq` (~2.6% of the workers' time): a real wait for a GPU readback. - A hot list or PGO profile of the lifted game code made from the game binary: it is made from the game, so it can't ship to players (see [#505](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/505)). **How it was measured:** `HOS_SAMPLE_ALL` (every game thread every 2 ms) and `HOS_CHAIN` for the critical path, frame times from `RX_VK_FRAME_LOG` with `HOS_FPS_MAX=240`; in each area 15 s standing and 20 s turning the camera at full speed. One gotcha: on Linux, `prof.waiting` doesn't count `pthread_rwlock_*lock` futex sleeps, so they show up as "running" libc time. That is how the copy lock waits were found.
Collaborator

Possibly related, for context:

  • #410 [Performance] Severe CPU-side rendering bottleneck in Kokiri Forest (21–42 FPS, low GPU usage) — Prior profiling issue for CPU-side bottleneck in Kokiri Forest, explicitly linked by this profiling-leads tracker.
  • #402 Deck: the frame is CPU-bound in the submission path, not GPU-bound (16 fps at 50% internal, GPU-bound wait 0.4 ms) — Another measured CPU-side frame-time bottleneck with detailed profiling, though on Steam Deck.
Possibly related, for context: - #410 [Performance] Severe CPU-side rendering bottleneck in Kokiri Forest (21–42 FPS, low GPU usage) — Prior profiling issue for CPU-side bottleneck in Kokiri Forest, explicitly linked by this profiling-leads tracker. - #402 Deck: the frame is CPU-bound in the submission path, not GPU-bound (16 fps at 50% internal, GPU-bound wait 0.4 ms) — Another measured CPU-side frame-time bottleneck with detailed profiling, though on Steam Deck.
Author
Owner

More leads, and results since rc.4 (Linux, RTX 3090, NVIDIA 620.70.06, Ryzen 9 7900X on all 24 threads, CachyOS). This time at 2560x1440: the game's own upscaler at 100%, uncapped, V-Sync off, frame generation off. In the Market: 5 s walking forward, then 20 s walking with the camera turning. Builds are player builds from the Builder AppImage, with runs interleaved between versions.

Where main stands against rc.4 in the Market (medians of 4 clean runs; walking / walking + turning):

version fps frames over 33 ms (turning) GPU ms a frame (turning)
v3.0.0-rc.4 (53f96d78) 57.6 / 60.5 7-18 a run ~9.0
main 1764fed2 59.6 / 68.6 0-9 a run ~8.0
main 6fb13fe5 61.4 / 66.8 0-3 a run ~8.0

That's about +5% walking and +12% turning. The float helpers (a64_fpx_*) went from ~5.5% of the game threads' running samples to ~2.5%. #838 (6fb13fe5) moved nothing measurable on this PC. Turning numbers carry some noise, because the circle sees 1,000-1,400 draws from run to run.

New leads:

  1. The main thread waits on GPU fences (nvnSyncWait → mkwvk::wait) more than in rc.4. In the Market it's in 1.8% of the main thread's samples on rc.4, 2.8% on dbad7d12 and 3.9% on 6fb13fe5: it grows step by step. In a 6-minute play session on e4d035e9 (DLSS on, 55-70 fps) it was 6.7-8.8%. The main thread is the critical path, so this is frame time. First step: which syncs the game waits on, and which commit since rc.4 made it grow. Then whether they can be signalled at the CPU-side point where the work is known done, rather than at GPU completion.
  2. The capture copy on the main thread, measured again: capture_make → copy_split → copy_part is in 7.3-8.1% of the main thread's samples in the Market (rc.4 8.1%, main 7.3%), and 7-7.5% in the play session (lead 2 above, the same order of size at 1440p).
  3. The game waits for room in the render thread's queue at almost every present. In play on e4d035e9, the render thread's summary line shows presents at 3-8 ms a frame. The game waited for room at 528-598 of every 600 presents, 0.7-1.0 ms each, and 4.8 ms each in one heavy stretch. So the render thread does sit on the critical path some of the time on this PC, through the blocking present.

Ruled out:

  • RX_VK_PRESENT_THREAD=1 in the Market, on 24 threads at 1440p: no gain. 3 runs each way, interleaved on 6fb13fe5. Off: 61.2 / 67.7 fps; on: 60.0 / 70.4 fps. The higher turning median comes from runs that saw fewer draws (1,038-1,094 against ~1,300); the one pair with matching draws was 64.6 on against 67.7 off. The presenter thread starts on its own queue (family 0, queue 1, with VK_KHR_present_wait). So freeing the render thread from the present (lead 3 above) doesn't shorten the frame on its own here; the queue waits in lead 3 of this comment need another approach.

Side finding: main dbad7d12 (the main of earlier today) peaked at ~18.2 GB of the game process's RAM in the Market, against ~12.5 GB for rc.4, 1764fed2 and 6fb13fe5. On a 30 GB PC that pushed the rest of the PC into swap, with hitches up to 650 ms. One of the 31 commits up to 1764fed2 fixed it. The commit that caused it and the one that fixed it aren't pinned down yet.

How it was measured: HOS_SAMPLE_ALL while walking and turning, RX_VK_PASS_TIMING=1, RX_VK_FRAME_LOG, and the game's RSS and the PC's swap-ins every 2 s. Runs with heavy swapping from other programs (several GB of swap-ins during a run) are left out of the medians. The game code isn't named in any of this: the profile names only Longshot's own functions.

**More leads, and results since rc.4** (Linux, RTX 3090, NVIDIA 620.70.06, Ryzen 9 7900X **on all 24 threads**, CachyOS). This time at **2560x1440**: the game's own upscaler at 100%, uncapped, V-Sync off, frame generation off. In the Market: 5 s walking forward, then 20 s walking with the camera turning. Builds are player builds from the Builder AppImage, with runs interleaved between versions. **Where main stands against rc.4 in the Market** (medians of 4 clean runs; walking / walking + turning): | version | fps | frames over 33 ms (turning) | GPU ms a frame (turning) | |---|---|---|---| | v3.0.0-rc.4 (53f96d78) | 57.6 / 60.5 | 7-18 a run | ~9.0 | | main 1764fed2 | 59.6 / 68.6 | 0-9 a run | ~8.0 | | main 6fb13fe5 | 61.4 / 66.8 | 0-3 a run | ~8.0 | That's about +5% walking and +12% turning. The float helpers (`a64_fpx_*`) went from ~5.5% of the game threads' running samples to ~2.5%. [#838](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/838) (6fb13fe5) moved nothing measurable on this PC. Turning numbers carry some noise, because the circle sees 1,000-1,400 draws from run to run. **New leads:** 1. **The main thread waits on GPU fences (`nvnSyncWait` → `mkwvk::wait`) more than in rc.4.** In the Market it's in 1.8% of the main thread's samples on rc.4, 2.8% on dbad7d12 and 3.9% on 6fb13fe5: it grows step by step. In a 6-minute play session on e4d035e9 (DLSS on, 55-70 fps) it was 6.7-8.8%. The main thread is the critical path, so this is frame time. First step: which syncs the game waits on, and which commit since rc.4 made it grow. Then whether they can be signalled at the CPU-side point where the work is known done, rather than at GPU completion. 2. **The capture copy on the main thread, measured again:** `capture_make` → `copy_split` → `copy_part` is in 7.3-8.1% of the main thread's samples in the Market (rc.4 8.1%, main 7.3%), and 7-7.5% in the play session (lead 2 above, the same order of size at 1440p). 3. **The game waits for room in the render thread's queue at almost every present.** In play on e4d035e9, the render thread's summary line shows presents at 3-8 ms a frame. The game waited for room at **528-598 of every 600 presents, 0.7-1.0 ms each**, and 4.8 ms each in one heavy stretch. So the render thread does sit on the critical path some of the time on this PC, through the blocking present. **Ruled out:** - **`RX_VK_PRESENT_THREAD=1` in the Market, on 24 threads at 1440p: no gain.** 3 runs each way, interleaved on 6fb13fe5. Off: 61.2 / 67.7 fps; on: 60.0 / 70.4 fps. The higher turning median comes from runs that saw fewer draws (1,038-1,094 against ~1,300); the one pair with matching draws was 64.6 on against 67.7 off. The presenter thread starts on its own queue (`family 0, queue 1`, with `VK_KHR_present_wait`). So freeing the render thread from the present (lead 3 above) doesn't shorten the frame on its own here; the queue waits in lead 3 of this comment need another approach. **Side finding:** main dbad7d12 (the main of earlier today) peaked at **~18.2 GB** of the game process's RAM in the Market, against ~12.5 GB for rc.4, 1764fed2 and 6fb13fe5. On a 30 GB PC that pushed the rest of the PC into swap, with hitches up to 650 ms. One of the 31 commits up to 1764fed2 fixed it. The commit that caused it and the one that fixed it aren't pinned down yet. **How it was measured:** `HOS_SAMPLE_ALL` while walking and turning, `RX_VK_PASS_TIMING=1`, `RX_VK_FRAME_LOG`, and the game's RSS and the PC's swap-ins every 2 s. Runs with heavy swapping from other programs (several GB of swap-ins during a run) are left out of the medians. The game code isn't named in any of this: the profile names only Longshot's own functions.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1020
No description provided.