WIP: perf(vk): RX_VK_CAPTURE_SLOW_MS=<ms> logs a slow capture's parts (probe, off by default) #1667

Draft
Molier wants to merge 1 commit from Molier/Longshot:perf/capture-ready-probe into main
Contributor

Type: diag probe · Risk: low (off by default; additions only) · Tested on: serv (Linux renderer build, RX_VK_ONLY=1, 0 failed; tests/test_capture_spare_reuse.py fixtures) · Needs: a play session with RX_VK_CAPTURE_SLOW_MS set, against a live capture path, to confirm the log format and that the parts it reports add up sensibly

Why

We measured (on our own stack, not this commit) that in heavy loads the submit-time Vulkan capture (capture_make, copying every written guest page into a capture buffer) can run 50-290 ms on the game's own thread, and that most of that time is the memcpy itself. Without a probe we could not tell which part of the copy path (lock, scan, late-page wait, buffer, copy) a slow capture actually spent its time in, or whether its buffer was made ahead of time or on the game thread itself. We are contributing the probe (not a fix) first.

Narrowed since the first push

This PR originally also added RX_VK_CAP_READY=1, which touches a capture buffer's pages on the worker thread that made it ahead of time, to move first-touch page faults off the game thread. Our own later data (not from this commit) found that 112 of 116 slow captures in a measured battery made their buffer on the capturing (game) thread itself, not ahead of time on a worker — the path RX_VK_CAP_READY's pretouch targets. Pretouching a worker-made buffer cannot help a buffer that was never made ahead, so shipping RX_VK_CAP_READY as a fix candidate would have been an unsupported claim. We dropped it and narrowed this PR to the RX_VK_CAPTURE_SLOW_MS probe alone, which is what the next investigation actually needs.

How to reproduce

Seen on: main (2f1c0c9c), any GPU; this specific commit's correctness is a host-side unit test, not a live capture.
Settings and switches: unset (default): no new behaviour. RX_VK_CAPTURE_SLOW_MS=<ms>: log a capture that takes <ms> ms or more.
Steps: python -I -B tests/test_capture_spare_reuse.py.
Expected / actual: see Measurements.

What changed

  • gpu/vk/vk_memory.cpp: capture_slow_ms() reads RX_VK_CAPTURE_SLOW_MS once (through longshot_env.h's env_int, clamped to > 0); a capture whose wall time reaches that threshold gets one log line (parts: lock, scan, late-page wait, buffer, copy; page count; the thread's minor-fault delta from getrusage(RUSAGE_THREAD, ...) on Linux; whether its buffer was made on this thread).
  • gpu/vk/switches.def and docs/development.md: one row each for RX_VK_CAPTURE_SLOW_MS.
  • tests/test_capture_spare_reuse.py: the existing fixtures' stub for cap_made_on_thread is now thread_local, matching the real counter.
  • Off by default: no code path changes when the switch is unset beyond one cached int read.

Review fixes (this push)

  • cap_made_on_thread was a process-global std::atomic, so when two threads capture at once (main and Presentation Thread both do) one thread's buffer could be credited to the other's slow-capture log line. It is now thread_local.
  • thread_minflt() guarded its getrusage(RUSAGE_THREAD, ...) call with #ifndef _WIN32, a Windows-only exclusion, for a Linux-only constant (RUSAGE_THREAD does not exist on every non-Windows platform). Changed to #ifdef RUSAGE_THREAD.
  • capture_slow_ms() read RX_VK_CAPTURE_SLOW_MS with a raw getenv+atoi (not through longshot_env.h's one rule) and did not clamp a negative value, which would have logged every capture. Now reads through env_int and clamps to > 0.
  • Dropped RX_VK_CAP_READY, cap_ready_on(), cap_pretouch() and tests/test_capture_ready.py entirely (see "Narrowed since the first push" above), and the switches.def/docs rows for RX_VK_CAP_READY.

Measurements

Verdict: the probe is a diagnostic only; it is not presented as a fix. The counter and fault-delta fixes above are shown correct by inspection (thread_local scoping, the RUSAGE_THREAD guard, the env_int clamp) and by the renderer build (0 failed) and the existing fixture passing; no frame-time or capture-time number from this exact commit is claimed, since the probe's job is to produce numbers for the next PR, not to change behaviour itself.

BUILD_EXIT=0, compiled in 187s, 0 failed

tests/test_capture_spare_reuse.py passes (no output on success; its asserts cover cap_buf_get, cap_spare_make and the capture-buffer worker path with the new thread_local counter in place).

Not measured

  • The RX_VK_CAPTURE_SLOW_MS probe's own log format has not been exercised against a live capture outside our local (non-upstream) stack.
  • Whatever replaces RX_VK_CAP_READY as an actual fix for the 112-of-116-on-the-game-thread pattern: not part of this PR.
  • Multi-platform (Windows) minor-fault counting: the probe logs 0 for faults on platforms without RUSAGE_THREAD (documented, not a crash).
**Type:** diag probe · **Risk:** low (off by default; additions only) · **Tested on:** serv (Linux renderer build, `RX_VK_ONLY=1`, 0 failed; `tests/test_capture_spare_reuse.py` fixtures) · **Needs:** a play session with `RX_VK_CAPTURE_SLOW_MS` set, against a live capture path, to confirm the log format and that the parts it reports add up sensibly ## Why We measured (on our own stack, not this commit) that in heavy loads the submit-time Vulkan capture (`capture_make`, copying every written guest page into a capture buffer) can run 50-290 ms on the game's own thread, and that most of that time is the memcpy itself. Without a probe we could not tell which part of the copy path (lock, scan, late-page wait, buffer, copy) a slow capture actually spent its time in, or whether its buffer was made ahead of time or on the game thread itself. We are contributing the probe (not a fix) first. ### Narrowed since the first push This PR originally also added `RX_VK_CAP_READY=1`, which touches a capture buffer's pages on the worker thread that made it ahead of time, to move first-touch page faults off the game thread. Our own later data (not from this commit) found that **112 of 116 slow captures in a measured battery made their buffer on the capturing (game) thread itself**, not ahead of time on a worker — the path `RX_VK_CAP_READY`'s pretouch targets. Pretouching a worker-made buffer cannot help a buffer that was never made ahead, so shipping `RX_VK_CAP_READY` as a fix candidate would have been an unsupported claim. We dropped it and narrowed this PR to the `RX_VK_CAPTURE_SLOW_MS` probe alone, which is what the next investigation actually needs. ## How to reproduce **Seen on:** main (`2f1c0c9c`), any GPU; this specific commit's correctness is a host-side unit test, not a live capture. **Settings and switches:** unset (default): no new behaviour. `RX_VK_CAPTURE_SLOW_MS=<ms>`: log a capture that takes `<ms>` ms or more. **Steps:** `python -I -B tests/test_capture_spare_reuse.py`. **Expected / actual:** see Measurements. ## What changed - `gpu/vk/vk_memory.cpp`: `capture_slow_ms()` reads `RX_VK_CAPTURE_SLOW_MS` once (through `longshot_env.h`'s `env_int`, clamped to `> 0`); a capture whose wall time reaches that threshold gets one log line (parts: lock, scan, late-page wait, buffer, copy; page count; the thread's minor-fault delta from `getrusage(RUSAGE_THREAD, ...)` on Linux; whether its buffer was made on this thread). - `gpu/vk/switches.def` and `docs/development.md`: one row each for `RX_VK_CAPTURE_SLOW_MS`. - `tests/test_capture_spare_reuse.py`: the existing fixtures' stub for `cap_made_on_thread` is now `thread_local`, matching the real counter. - Off by default: no code path changes when the switch is unset beyond one cached int read. ### Review fixes (this push) - `cap_made_on_thread` was a process-global `std::atomic`, so when two threads capture at once (main and Presentation Thread both do) one thread's buffer could be credited to the other's slow-capture log line. It is now `thread_local`. - `thread_minflt()` guarded its `getrusage(RUSAGE_THREAD, ...)` call with `#ifndef _WIN32`, a Windows-only exclusion, for a Linux-only constant (`RUSAGE_THREAD` does not exist on every non-Windows platform). Changed to `#ifdef RUSAGE_THREAD`. - `capture_slow_ms()` read `RX_VK_CAPTURE_SLOW_MS` with a raw `getenv`+`atoi` (not through `longshot_env.h`'s one rule) and did not clamp a negative value, which would have logged every capture. Now reads through `env_int` and clamps to `> 0`. - Dropped `RX_VK_CAP_READY`, `cap_ready_on()`, `cap_pretouch()` and `tests/test_capture_ready.py` entirely (see "Narrowed since the first push" above), and the switches.def/docs rows for `RX_VK_CAP_READY`. ## Measurements **Verdict: the probe is a diagnostic only; it is not presented as a fix.** The counter and fault-delta fixes above are shown correct by inspection (thread_local scoping, the RUSAGE_THREAD guard, the env_int clamp) and by the renderer build (0 failed) and the existing fixture passing; **no frame-time or capture-time number from this exact commit is claimed**, since the probe's job is to produce numbers for the next PR, not to change behaviour itself. ``` BUILD_EXIT=0, compiled in 187s, 0 failed ``` `tests/test_capture_spare_reuse.py` passes (no output on success; its asserts cover `cap_buf_get`, `cap_spare_make` and the capture-buffer worker path with the new thread_local counter in place). ## Not measured - The `RX_VK_CAPTURE_SLOW_MS` probe's own log format has not been exercised against a live capture outside our local (non-upstream) stack. - Whatever replaces `RX_VK_CAP_READY` as an actual fix for the 112-of-116-on-the-game-thread pattern: not part of this PR. - Multi-platform (Windows) minor-fault counting: the probe logs 0 for faults on platforms without `RUSAGE_THREAD` (documented, not a crash).
Member

Possibly related, for context:

  • PR #1599 WIP: perf(vk): retain and refill capture buffers by size class — Open capture-buffer reuse/refill work overlaps the prewarm path and capture buffer lifecycle.
  • PR #1553 perf(runtime): reduce Market frame stalls and Vulkan capture overhead — Merged renderer work introduced capture-range reuse; this PR adds pretouch and timing diagnostics on that path.
Possibly related, for context: - PR #1599 WIP: perf(vk): retain and refill capture buffers by size class — Open capture-buffer reuse/refill work overlaps the prewarm path and capture buffer lifecycle. - PR #1553 perf(runtime): reduce Market frame stalls and Vulkan capture overhead — Merged renderer work introduced capture-range reuse; this PR adds pretouch and timing diagnostics on that path.
Molier force-pushed perf/capture-ready-probe from 765f921623
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to a871f3ff73
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 1m10s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-18 23:46:41 +00:00
Compare
Molier changed title from WIP: perf(vk): RX_VK_CAP_READY pretouch + RX_VK_CAPTURE_SLOW_MS probe for the submit-time capture copy to WIP: perf(vk): RX_VK_CAPTURE_SLOW_MS=<ms> logs a slow capture's parts (probe, off by default) 2026-11-18 23:48:36 +00:00
Contributor

Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. This PR's head a871f3ff and #1666's 4ec23e96, both cherry-picked onto their common base 2f1c0c9c, in one test build (longshot abc-probes (source 166c4da4)), with RX_VK_CAPTURE_SLOW_MS=50. No errors, hangs or device losses in any run.

The probe (this PR's Needs item): it works against a live capture path, and its parts add up: lock, scan, late pages, buffer and copy are within 0.3 ms of the total on all 24 lines. Lock, scan and late pages were 0.5 ms or less on every line; the time is the copy and the buffer.

The longest slow captures, both runs thread total buffer copy copied copy rate
run 2, out into the field sead::MainThread 773 ms 527 ms, made on this thread 245 ms 243 MB 0.99 GB/s
run 1, out into the field sead::MainThread 258 ms 11 ms, made on this thread 246 ms 252 MB 1.03 GB/s
run 2, out into the field sead::MainThread 226 ms 142 ms, made on this thread 84 ms 90 MB 1.07 GB/s
run 2, out into the field sead::MainThread 206 ms 141 ms, made on this thread 65 ms 66 MB 1.00 GB/s
run 1, out into the field Presentation Thread 163 ms 32 ms, made on this thread 130 ms 171 MB 1.31 GB/s
run 1, measured part sead::MainThread 122 ms 0 ms 122 ms 153 MB 1.26 GB/s
  • The copy runs at about 1 GB/s on every line (0.7 to 1.3 GB/s, one at 2.3). It's shared by the capturing thread and the 4 copy helpers (copy_split, 4 from 12 CPU threads), so that's about 200 MB/s a thread. Neither end is slow memory on paper: the capture buffer is cached host memory (create_buffer_cached: VMA_MEMORY_USAGE_AUTO_PREFER_HOST, random access), and the game's memory is imported in place as dma-bufs. Minor faults on the capturing thread are low (11, 4, 28), but the helpers do most of the copying and their faults and kernel time aren't in the line. Counting those (or the helpers' share of the pages and their time) would show where the rest goes.
  • 12 of the 24 made their buffer on the capturing thread, the same pattern as your 112 of 116. The worst frame of both runs, 825 ms on entering the field, is the 773 ms capture, 758 ms of it on a CPU (#1666's line for the same stretch).
  • The biggest captures can never take a spare: the background spares are 32, 64, 128 and 192 MB (cap_prewarm_index is -1 above 192 MB), and the field's two biggest captures needed 243 and 252 MB, so their buffer is always made on the capturing thread (527 ms for the 243 MB one). Those numbers are on #1599, which works on the spare classes.
  • In the two 120 s measured parts, 3 of the 4 frames of 100 ms or more (148, 118 and 103 ms) each hold one slow capture on sead::MainThread (122, 56 and 51 ms).
[hos sead::MainThread] vk: slow capture 773.0 ms: lock 0.1, scan 0.3, late pages 0.0, buffer 527.2 (made on this thread), copy 245.2; 62111 pages (242.6 MB), 28 minor faults, 0 late
[hos sead::MainThread] vk: slow capture 226.2 ms: lock 0.0, scan 0.1, late pages 0.0, buffer 141.6 (made on this thread), copy 84.4; 23126 pages (90.3 MB), 4 minor faults, 0 late
[hos sead::MainThread] vk: slow capture 205.9 ms: lock 0.0, scan 0.1, late pages 0.0, buffer 140.5 (made on this thread), copy 65.3; 16757 pages (65.5 MB), 2 minor faults, 0 late
[hos Presentation Thread] vk: slow capture 163.2 ms: lock 0.2, scan 0.3, late pages 0.0, buffer 32.1 (made on this thread), copy 130.4; 43783 pages (171.0 MB), 39 minor faults, 0 late
[hos sead::MainThread] vk: slow capture 122.0 ms: lock 0.1, scan 0.2, late pages 0.0, buffer 0.0, copy 121.6; 39229 pages (153.2 MB), 11 minor faults, 180 late

We'd like to look into the copy rate next, unless you're on it already: say if so, and we'll send you runs instead.

Not compared against main: the probe changes nothing while it's off, so these are its readings, not a measurement of this PR.

Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on). HOS_AWAY_LOG_MS=50 (#1666) and HOS_PROFILE=1 in every run.
Method: a scripted route through the control bridge from the repository save kokiri-start: the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps along Link's facing between 20 m and 320 m out with a 180° camera turn every 10 s (the trigger in #1720). A warm-up run first, then 2 runs; Link's hearts protected (HOS_OOT_PROTECT_HEARTS=1).

Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. This PR's head `a871f3ff` and #1666's `4ec23e96`, both cherry-picked onto their common base `2f1c0c9c`, in one test build (`longshot abc-probes (source 166c4da4)`), with `RX_VK_CAPTURE_SLOW_MS=50`. No errors, hangs or device losses in any run. **The probe (this PR's Needs item):** it works against a live capture path, and its parts add up: lock, scan, late pages, buffer and copy are within 0.3 ms of the total on all 24 lines. Lock, scan and late pages were 0.5 ms or less on every line; **the time is the copy and the buffer**. | The longest slow captures, both runs | thread | total | buffer | copy | copied | copy rate | |---|---|---|---|---|---|---| | run 2, out into the field | sead::MainThread | 773 ms | 527 ms, made on this thread | 245 ms | 243 MB | 0.99 GB/s | | run 1, out into the field | sead::MainThread | 258 ms | 11 ms, made on this thread | 246 ms | 252 MB | 1.03 GB/s | | run 2, out into the field | sead::MainThread | 226 ms | 142 ms, made on this thread | 84 ms | 90 MB | 1.07 GB/s | | run 2, out into the field | sead::MainThread | 206 ms | 141 ms, made on this thread | 65 ms | 66 MB | 1.00 GB/s | | run 1, out into the field | Presentation Thread | 163 ms | 32 ms, made on this thread | 130 ms | 171 MB | 1.31 GB/s | | run 1, measured part | sead::MainThread | 122 ms | 0 ms | 122 ms | 153 MB | 1.26 GB/s | - **The copy runs at about 1 GB/s** on every line (0.7 to 1.3 GB/s, one at 2.3). It's shared by the capturing thread and the 4 copy helpers (`copy_split`, 4 from 12 CPU threads), so that's about 200 MB/s a thread. Neither end is slow memory on paper: the capture buffer is cached host memory (`create_buffer_cached`: `VMA_MEMORY_USAGE_AUTO_PREFER_HOST`, random access), and the game's memory is imported in place as dma-bufs. Minor faults on the capturing thread are low (11, 4, 28), but the helpers do most of the copying and their faults and kernel time aren't in the line. Counting those (or the helpers' share of the pages and their time) would show where the rest goes. - **12 of the 24 made their buffer on the capturing thread**, the same pattern as your 112 of 116. The worst frame of both runs, 825 ms on entering the field, is the 773 ms capture, 758 ms of it on a CPU (#1666's line for the same stretch). - **The biggest captures can never take a spare:** the background spares are 32, 64, 128 and 192 MB (`cap_prewarm_index` is -1 above 192 MB), and the field's two biggest captures needed 243 and 252 MB, so their buffer is always made on the capturing thread (527 ms for the 243 MB one). Those numbers are on #1599, which works on the spare classes. - In the two 120 s measured parts, 3 of the 4 frames of 100 ms or more (148, 118 and 103 ms) each hold one slow capture on sead::MainThread (122, 56 and 51 ms). ``` [hos sead::MainThread] vk: slow capture 773.0 ms: lock 0.1, scan 0.3, late pages 0.0, buffer 527.2 (made on this thread), copy 245.2; 62111 pages (242.6 MB), 28 minor faults, 0 late [hos sead::MainThread] vk: slow capture 226.2 ms: lock 0.0, scan 0.1, late pages 0.0, buffer 141.6 (made on this thread), copy 84.4; 23126 pages (90.3 MB), 4 minor faults, 0 late [hos sead::MainThread] vk: slow capture 205.9 ms: lock 0.0, scan 0.1, late pages 0.0, buffer 140.5 (made on this thread), copy 65.3; 16757 pages (65.5 MB), 2 minor faults, 0 late [hos Presentation Thread] vk: slow capture 163.2 ms: lock 0.2, scan 0.3, late pages 0.0, buffer 32.1 (made on this thread), copy 130.4; 43783 pages (171.0 MB), 39 minor faults, 0 late [hos sead::MainThread] vk: slow capture 122.0 ms: lock 0.1, scan 0.2, late pages 0.0, buffer 0.0, copy 121.6; 39229 pages (153.2 MB), 11 minor faults, 180 late ``` We'd like to look into the copy rate next, unless you're on it already: say if so, and we'll send you runs instead. Not compared against main: the probe changes nothing while it's off, so these are its readings, not a measurement of this PR. Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on). `HOS_AWAY_LOG_MS=50` (#1666) and `HOS_PROFILE=1` in every run. Method: a scripted route through the control bridge from the repository save `kokiri-start`: the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps along Link's facing between 20 m and 320 m out with a 180° camera turn every 10 s (the trigger in #1720). A warm-up run first, then 2 runs; Link's hearts protected (`HOS_OOT_PROTECT_HEARTS=1`).
Contributor

Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. The same test build as above (this PR's a871f3ff and #1666's 4ec23e96 on 2f1c0c9c, longshot abc-probes (source 166c4da4)), with RX_VK_CAPTURE_SLOW_MS=50. No errors, hangs or device losses in any run.

A follow-up on the copy rate, with your probe. The copy is bound by page faults, and non-temporal stores (RX_VK_CAPTURE_NT=1) take most of them away. In five diagnostic runs, one capture thread copied as fast as five (0.97 against 1.02-1.07 GB/s), with 0.85 minor faults per copied 4 KB page. Then an interleaved A/B of RX_VK_CAPTURE_NT, 3 runs a side, in the order off, on, on, off, off, on after a warm-up:

RX_VK_CAPTURE_NT=0, the default RX_VK_CAPTURE_NT=1
Slow captures (50 ms or more), whole run 16, 21, 16 5, 6, 2
Copy rate of the slow captures (median) 1.05, 0.94, 1.03 GB/s 10.15, 10.16, 7.92 GB/s
Copy helpers: minor faults, kernel time 81597, 164061, 148658; 33%, 39%, 37% 21350, 23172, 25739; 8%, 12%, 8%
Measured 120 s: 1% low 19.2, 10.5, 16.9 fps 20.3, 28.1, 27.4 fps
Measured 120 s: worst frame 267, 2612, 797 ms 302, 56, 113 ms
Field entry: worst frame 1185, 570, 479 ms 255, 885, 287 ms
  • Every run with it on had fewer slow captures, fewer helper faults and a higher 1% low than every run with it off. The worst frame and the field entry overlap between the sides; the average fps hardly moves (54.0-54.7 against 52.4-54.3).
  • With it on, your probe shows what is left: the buffer. The copy is 1 to 5 ms, and most remaining slow captures are a capture buffer made on the capturing thread, 150 to 300 ms even for 8 to 40 MB. Two copies still took 156 and 217 ms for 28 and 43 MB with few faults; we can't explain those yet.
  • This is AMD on Linux only. RX_VK_CAPTURE_NT is opt-in because of the RTX desktop runs in the comment at capture_nt(), so it says nothing yet about a default elsewhere.
  • These runs are on 2f1c0c9c, before #1121 (merged today), which replays jobs as they come instead of whole frames; the copy itself shouldn't change with it, but the frame times may.
RX_VK_CAPTURE_NT=0:
[hos sead::MainThread] vk: slow capture 1061.6 ms: lock 0.1, scan 0.3, late pages 0.0, buffer 972.8 (made on this thread), copy 88.2; 64261 pages (251.0 MB), 59 minor faults, 0 late
RX_VK_CAPTURE_NT=1:
[hos sead::MainThread] vk: slow capture 306.7 ms: lock 0.0, scan 0.2, late pages 0.0, buffer 301.5 (made on this thread), copy 4.9; 10315 pages (40.3 MB), 39 minor faults, 0 late

Every run is on record in #1749 (docs/tested/, run01 to run11).

Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on). HOS_AWAY_LOG_MS=50 (#1666) and HOS_PROFILE=1 in every run; the copy helpers' faults and CPU time from /proc every 200 ms.
Method: the same scripted route as above: from the repository save kokiri-start, the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps between 20 m and 320 m out with a 180° camera turn every 10 s (the trigger in #1720). A warm-up run first, then 3 runs a side, interleaved; Link's hearts protected (HOS_OOT_PROTECT_HEARTS=1).

Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. The same test build as above (this PR's `a871f3ff` and #1666's `4ec23e96` on `2f1c0c9c`, `longshot abc-probes (source 166c4da4)`), with `RX_VK_CAPTURE_SLOW_MS=50`. No errors, hangs or device losses in any run. A follow-up on the copy rate, with your probe. **The copy is bound by page faults, and non-temporal stores (`RX_VK_CAPTURE_NT=1`) take most of them away.** In five diagnostic runs, one capture thread copied as fast as five (0.97 against 1.02-1.07 GB/s), with 0.85 minor faults per copied 4 KB page. Then an interleaved A/B of `RX_VK_CAPTURE_NT`, 3 runs a side, in the order off, on, on, off, off, on after a warm-up: | | `RX_VK_CAPTURE_NT=0`, the default | `RX_VK_CAPTURE_NT=1` | |---|---|---| | Slow captures (50 ms or more), whole run | 16, 21, 16 | 5, 6, 2 | | Copy rate of the slow captures (median) | 1.05, 0.94, 1.03 GB/s | 10.15, 10.16, 7.92 GB/s | | Copy helpers: minor faults, kernel time | 81597, 164061, 148658; 33%, 39%, 37% | 21350, 23172, 25739; 8%, 12%, 8% | | Measured 120 s: 1% low | 19.2, 10.5, 16.9 fps | 20.3, 28.1, 27.4 fps | | Measured 120 s: worst frame | 267, 2612, 797 ms | 302, 56, 113 ms | | Field entry: worst frame | 1185, 570, 479 ms | 255, 885, 287 ms | - Every run with it on had fewer slow captures, fewer helper faults and a higher 1% low than every run with it off. The worst frame and the field entry overlap between the sides; the average fps hardly moves (54.0-54.7 against 52.4-54.3). - **With it on, your probe shows what is left: the buffer.** The copy is 1 to 5 ms, and most remaining slow captures are a capture buffer made on the capturing thread, 150 to 300 ms even for 8 to 40 MB. Two copies still took 156 and 217 ms for 28 and 43 MB with few faults; we can't explain those yet. - This is AMD on Linux only. `RX_VK_CAPTURE_NT` is opt-in because of the RTX desktop runs in the comment at `capture_nt()`, so it says nothing yet about a default elsewhere. - These runs are on `2f1c0c9c`, before #1121 (merged today), which replays jobs as they come instead of whole frames; the copy itself shouldn't change with it, but the frame times may. ``` RX_VK_CAPTURE_NT=0: [hos sead::MainThread] vk: slow capture 1061.6 ms: lock 0.1, scan 0.3, late pages 0.0, buffer 972.8 (made on this thread), copy 88.2; 64261 pages (251.0 MB), 59 minor faults, 0 late RX_VK_CAPTURE_NT=1: [hos sead::MainThread] vk: slow capture 306.7 ms: lock 0.0, scan 0.2, late pages 0.0, buffer 301.5 (made on this thread), copy 4.9; 10315 pages (40.3 MB), 39 minor faults, 0 late ``` Every run is on record in #1749 (`docs/tested/`, run01 to run11). Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on). `HOS_AWAY_LOG_MS=50` (#1666) and `HOS_PROFILE=1` in every run; the copy helpers' faults and CPU time from `/proc` every 200 ms. Method: the same scripted route as above: from the repository save `kokiri-start`, the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps between 20 m and 320 m out with a 180° camera turn every 10 s (the trigger in #1720). A warm-up run first, then 3 runs a side, interleaved; Link's hearts protected (`HOS_OOT_PROTECT_HEARTS=1`).
Contributor

Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. A new test build on current main: 819a5967 (after #1121) with this PR's a871f3ff and #1666's 4ec23e96 cherry-picked (longshot abc-probes-main (source 80057f9b)), with RX_VK_CAPTURE_SLOW_MS=50. No errors, hangs or device losses in any run.

RX_VK_CAPTURE_NT=1 holds on main after #1121, and #1121's gate makes no clear difference. A 2x2 of RX_VK_FRAME_GATE (0 is main's default since #1121) and RX_VK_CAPTURE_NT, 2 runs a cell, in the order A B C D D C B A after a warm-up:

Each cell's two runs gate off, NT off (main) gate off, NT on gate on, NT off gate on, NT on
Slow captures (50 ms or more), whole run 20, 18 2, 0 21, 23 2, 0
Measured 120 s: 1% low 12.4, 19.2 fps 29.6, 30.1 fps 17.2, 9.6 fps 27.7, 28.2 fps
Measured 120 s: worst frame 942, 313 ms 50, 68 ms 727, 1444 ms 53, 67 ms
Measured 120 s: frames of 100 ms or more 5, 7 0, 0 4, 9 0, 0
Measured 120 s: fps 54.4, 56.9 57.3, 57.4 54.1, 52.0 55.6, 55.5
Field entry: worst frame 465, 282 ms 787, 333 ms 594, 328 ms 558, 275 ms
  • With NT on, no run had a frame of 100 ms or more in the measured part, and the 1% low was 27.7-30.1 fps against 9.6-19.2 with it off.
  • The field entry doesn't separate: its spikes are the GPU-sync waits (#1666's probe: 8-12 waits of 100 ms or more a run, up to 496 ms, in every cell), which NT doesn't touch.
  • 2 runs a cell, AMD on Linux only; the RTX desktop results that keep it opt-in are untouched by this.

Every run is on record in #1749 (docs/tested/, run12 to run19).

Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on). HOS_AWAY_LOG_MS=50 (#1666) and HOS_PROFILE=1 in every run.
Method: the same scripted route as above: from the repository save kokiri-start, the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps between 20 m and 320 m out with a 180° camera turn every 10 s. A warm-up run first, then 2 runs a cell, interleaved; Link's hearts protected (HOS_OOT_PROTECT_HEARTS=1).

Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. A new test build on current main: `819a5967` (after #1121) with this PR's `a871f3ff` and #1666's `4ec23e96` cherry-picked (`longshot abc-probes-main (source 80057f9b)`), with `RX_VK_CAPTURE_SLOW_MS=50`. No errors, hangs or device losses in any run. **`RX_VK_CAPTURE_NT=1` holds on main after #1121, and #1121's gate makes no clear difference.** A 2x2 of `RX_VK_FRAME_GATE` (0 is main's default since #1121) and `RX_VK_CAPTURE_NT`, 2 runs a cell, in the order A B C D D C B A after a warm-up: | Each cell's two runs | gate off, NT off (main) | gate off, NT on | gate on, NT off | gate on, NT on | |---|---|---|---|---| | Slow captures (50 ms or more), whole run | 20, 18 | 2, 0 | 21, 23 | 2, 0 | | Measured 120 s: 1% low | 12.4, 19.2 fps | 29.6, 30.1 fps | 17.2, 9.6 fps | 27.7, 28.2 fps | | Measured 120 s: worst frame | 942, 313 ms | 50, 68 ms | 727, 1444 ms | 53, 67 ms | | Measured 120 s: frames of 100 ms or more | 5, 7 | 0, 0 | 4, 9 | 0, 0 | | Measured 120 s: fps | 54.4, 56.9 | 57.3, 57.4 | 54.1, 52.0 | 55.6, 55.5 | | Field entry: worst frame | 465, 282 ms | 787, 333 ms | 594, 328 ms | 558, 275 ms | - With NT on, no run had a frame of 100 ms or more in the measured part, and the 1% low was 27.7-30.1 fps against 9.6-19.2 with it off. - The field entry doesn't separate: its spikes are the GPU-sync waits (#1666's probe: 8-12 waits of 100 ms or more a run, up to 496 ms, in every cell), which NT doesn't touch. - 2 runs a cell, AMD on Linux only; the RTX desktop results that keep it opt-in are untouched by this. Every run is on record in #1749 (`docs/tested/`, run12 to run19). Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on). `HOS_AWAY_LOG_MS=50` (#1666) and `HOS_PROFILE=1` in every run. Method: the same scripted route as above: from the repository save `kokiri-start`, the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps between 20 m and 320 m out with a 180° camera turn every 10 s. A warm-up run first, then 2 runs a cell, interleaved; Link's hearts protected (`HOS_OOT_PROTECT_HEARTS=1`).
Molier force-pushed perf/capture-ready-probe from a871f3ff73
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 1m10s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 68c28bf83c
Some checks are pending
CI / Website build and checks (pull_request) Blocked by required conditions
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 11s
2026-11-20 14:38:06 +00:00
Compare
Author
Contributor

Rebased onto main @ 89c3d21e. Only the docs row moved (#1802); switches.def, vk_memory.cpp and the test merged cleanly with #1763/#1848. tests/test_capture_spare_reuse.py passes and the renderer builds.

Staying WIP: I have a broader capture diagnostic (per-part CPU time and page faults, buffer memory type, per-capture wall time) that answers the same question, and I'd rather fold this threshold log into it than add two capture probes to switches.def. I'll update this PR when that is ready, or close it if #1599's shape makes it moot.

Rebased onto `main` @ `89c3d21e`. Only the docs row moved (#1802); `switches.def`, `vk_memory.cpp` and the test merged cleanly with #1763/#1848. `tests/test_capture_spare_reuse.py` passes and the renderer builds. Staying WIP: I have a broader capture diagnostic (per-part CPU time and page faults, buffer memory type, per-capture wall time) that answers the same question, and I'd rather fold this threshold log into it than add two capture probes to `switches.def`. I'll update this PR when that is ready, or close it if #1599's shape makes it moot.
Some checks are pending
CI / Website build and checks (pull_request) Blocked by required conditions
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 11s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/capture-ready-probe:Molier-perf/capture-ready-probe'
git switch 'Molier-perf/capture-ready-probe'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1667
No description provided.