WIP: docs(tested): capture and GPU-sync diagnostics in Hyrule Field on an RX 9070 XT (RADV) #1749

Closed
GrimVisage wants to merge 4 commits from GrimVisage/Longshot:docs/tested-capture-diagnostics-rdna4 into main
Contributor

Type: docs · Risk: low · Tested on: Linux (Fedora 45), Ryzen 5 5600X, RX 9070 XT (RADV 26.2.7) · Needs: none

TL;DR

Walking into Hyrule Field and turning the camera, the game drops to a few fps for up to a second (#1720). Runs with the probes from #1667 and #1666 point to two causes on an RX 9070 XT. Out in the field, the renderer's copy of freshly loaded memory is page-fault bound, and non-temporal copies (RX_VK_CAPTURE_NT=1) cut its long frames on every run, on main after #1121 too. At the field entry, the longest waits look like the GPU's ASTC work: ~450 ms at 100% GPU that the log's GPU time doesn't show, a third of it the BC3 re-encode; a few others are slow captures. This PR records the thirty-nine runs in docs/tested/; it changes no game code.

Breakdown

  • Who it helps: whoever works on the field's drops (#1720, #1696, #1735, #1667, #1666, #1599): the runs behind the comments are on record.
  • What was wrong: nothing in the code; the diagnostic runs weren't in the tested-configurations catalogue.
  • What changes: thirty-nine rows in docs/tested/, one per run.
  • What does not change: the game, the frozen table, every other docs file.
  • How it was checked: tests/test_tested_configurations.py passes; the full suite's results are under Testing.
  • What is still open: whether RX_VK_CAPTURE_NT=1 helps beyond AMD on Linux (it is opt-in for an RTX desktop's results); the capture buffers made on the capturing thread, which are most of what is left with it on; how to spread the ASTC decode and BC3 re-encode's GPU time at an area's entry (the upload budget bounds only the CPU's); a few long waits with the GPU idle, not looked into.
  • Risk: low: thirty-nine docs rows.

Why

Since the 3.0 release candidates, players see the frame rate collapse for a moment in Hyrule Field, worst when the field streams in or the camera turns (#1720 on Windows with an RTX 4070 Ti; the same here on Linux with an RX 9070 XT). Similar drops when circling or attacking are in #1696 and #1735; one player in #1735 dates them to rc.8, so the release that brought them is not settled. The probes in #1667 (RX_VK_CAPTURE_SLOW_MS) and #1666 (HOS_AWAY_LOG_MS) showed the main thread's two kinds of long waits on this route: the capture at a submit, and waiting on a GPU sync (results: #1667 (comment) and #1666 (comment)). Five runs here follow each one up with a switch, six more are an interleaved A/B of the one that looked best, eight more repeat it on main after #1121, twenty more follow the field entry's GPU-sync waits, and docs/tested/ is where a measured run goes (CONTRIBUTING.md, "Measurements").

What changed

Thirty-nine run files, following docs/tested/README.md, one a run:

  • docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run01.md: RX_VK_CAPTURE_THREADS=0.
  • docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run02.md: RX_VK_CAPTURE_NT=1.
  • docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run03.md: RX_VK_CAP_BLOCKS=1.
  • docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run04.md: HOS_NVN_SYNC_LOG=1.
  • docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run05.md: HOS_NVN_SYNC_LOG=2440.
  • docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run06.md to run11.md: the A/B, RX_VK_CAPTURE_NT=0 and =1 in the order off, on, on, off, off, on.
  • docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run12.md to run19.md: on main 819a5967 with the two probes, a 2x2 of RX_VK_FRAME_GATE and RX_VK_CAPTURE_NT, 2 runs a cell.
  • run20.md to run39.md: the field entry's GPU-sync waits on the same build, with RX_VK_CAPTURE_NT=1 and HOS_NVN_SYNC_LOG=1: two plain runs (run20-21); RX_VK_MIRROR_SPARSE on and off (run22-25); RX_VK_BATCH=0 and RX_VK_RECORDER=off (run26-29); then on a build with one more local probe line at a capture's timed-out write-back wait, not in any PR: two runs (run30-31), two with the GPU's busy percent sampled (run32-33), and RX_VK_ASTC_CPU=1 and RX_VK_ASTC_RGBA8=1 against the default (run34-39).

Measurements

Verdict: nothing to measure in the game for this PR: docs only. The files pass their format check, and the game-free suite gives the same result as on main (CONTRIBUTING.md, "Measurements": docs).

Before: main @ 868f6be8 After: this PR @ 3aa8225b
tests/test_tested_configurations.py OK OK, with the nineteen new files; again OK at 5da79b44 on main 0de25c3e with all thirty-nine
mise run test (tests/run_all.py --strict) 695 passed, 139 failed, 17 skipped 695 passed, 139 failed, 17 skipped: the same 139

Baseline: these are diagnostics, one run a switch, on one build (longshot abc-probes (source 166c4da4): main 2f1c0c9c with #1667 a871f3ff and #1666 4ec23e96), not a main-against-PR comparison; the two probe runs on #1667 are the same build with no switch.
Hardware: Fedora 45 (kernel 7.3.4), KDE Plasma 6.8.3 (Wayland), RX 9070 XT 16 GB, Mesa RADV 26.2.7, Ryzen 5 5600X (12 logical processors), 32 GB
Scene: Hyrule Field (child) at 12:00, out of the Castle Town gate; 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), frame generation off, no frame cap
Method: a scripted route through the control bridge from the repository save kokiri-start: the teleport to the Market, the clock at 12:00, out of the gate, then 120 s of laps between 20 m and 320 m with a 180° camera turn every 10 s (the trigger in #1720); frames from HOS_PROFILE=1's presents trace, the captures from RX_VK_CAPTURE_SLOW_MS=50, the waits from HOS_AWAY_LOG_MS=50, each thread's user and kernel CPU time and minor faults from /proc every 200 ms
How to reproduce: the same switches on any build carrying #1667 and #1666

What the runs show

From the field entry to the end of the run Default copy (run04 / run05) RX_VK_CAPTURE_THREADS=0 (run01) RX_VK_CAPTURE_NT=1 (run02) RX_VK_CAP_BLOCKS=1 (run03)
Copy rate of the slow captures (median) 1.02 / 1.04 GB/s 0.97 GB/s (one thread) no capture of 50 ms or more 1.05 GB/s
Copy helpers' CPU time in the kernel 38% / 37% (no helpers) 7% 40%
Minor faults, copy helpers 52859 / 73789 463605 on the capturing thread 7285 30542
Field entry: frames of 100 ms or more, worst 16 / 13, 454 / 597 ms 14, 455 ms 9, 272 ms 14, 477 ms
Measured 120 s: 1% low, worst 15.1 / 17.3 fps, 509 / 277 ms 15.3 fps, 677 ms 28.6 fps, 54 ms 14.7 fps, 1446 ms
  • The copy is bound by page faults. One thread copies as fast as five (0.97 against 1.02-1.07 GB/s), and with the helpers off the capturing thread takes 0.85 minor faults per copied page: nearly every 4 KB page faults. The capture buffer is cached host memory (create_buffer_cached) and the game's memory is imported in place as dma-bufs, so neither end is slow memory on paper.
  • Non-temporal stores cut the faults by 7 to 10 times and the helpers' kernel time from 37-40% to 7%, and that run had no slow capture at all. It's one run, and RX_VK_CAPTURE_NT is opt-in because four field runs on an RTX desktop showed no gain and more stutters with it (the comment at capture_nt() in gpu/vk/vk_memory.cpp): the A/B below repeats it.
  • The GPU-sync waits wait for the renderer. Each of run04's eight waits of 100 ms or more (125 to 461 ms) is on the frame before's fence, already handed over, with the replay 4 submissions behind it (6 once). In the probe runs the render thread sat idle at its whole-frame gate through such a wait and the Presentation Thread waited for the CPU's signal in present(): where the work is held is still open. #1121 (merged after these runs) turns that whole-frame gate off by default (RX_VK_FRAME_GATE=1 brings it back), so these waits were run on main again: they stay (below).
run01, RX_VK_CAPTURE_THREADS=0:
[hos sead::MainThread] vk: slow capture 264.7 ms: lock 0.1, scan 0.2, late pages 0.0, buffer 18.0 (made on this thread), copy 246.1; 59224 pages (231.3 MB), 48588 minor faults, 0 late
run04, HOS_NVN_SYNC_LOG=1:
[hos sead::MainThread] nvn: SyncWait 0x1010073b9020 fence 10172 timeout 18446744073709551615 (lr 0x8042cde0); latest 10175, done 10168
[hos sead::MainThread] away: sead::MainThread 460.9 ms (on a CPU 0.9 ms) from longshot+0x1f73a4b8, ended at 25725400343988 ns

The A/B of RX_VK_CAPTURE_NT

Six runs on the same build, after a warm-up run that isn't counted, in the order off, on, on, off, off, on (run06 to run11); each figure is a run's, in that order within a side.

RX_VK_CAPTURE_NT=0, the default (run06, run09, run10) RX_VK_CAPTURE_NT=1 (run07, run08, run11)
Slow captures (50 ms or more), whole run 16, 21, 16 5, 6, 2
Copy rate of the slow captures (median) 1.05, 0.94, 1.03 GB/s 10.15, 10.16, 7.92 GB/s
Copy helpers: minor faults, kernel time 81597, 164061, 148658; 33%, 39%, 37% 21350, 23172, 25739; 8%, 12%, 8%
Measured 120 s: 1% low 19.2, 10.5, 16.9 fps 20.3, 28.1, 27.4 fps
Measured 120 s: worst frame 267, 2612, 797 ms 302, 56, 113 ms
Measured 120 s: fps, p99 54.0, 52.4, 54.3; 31.0, 32.2, 33.0 ms 54.7, 54.7, 54.3; 30.0, 29.8, 29.6 ms
Field entry: worst frame, frames of 100 ms or more 1185, 570, 479 ms; 15, 17, 14 255, 885, 287 ms; 13, 14, 13
  • Separated on every run: fewer slow captures, 3.5 to 7 times fewer helper faults, and a higher 1% low with it on. The worst frame and the field entry overlap between the sides (medians 797 against 113 ms and 570 against 287 ms), and the average fps hardly moves.
  • What is left with it on: the copy is 1 to 5 ms; most remaining slow captures are a capture buffer made on the capturing thread, 150 to 300 ms even for 8 to 40 MB. Two copies still took 156 and 217 ms for 28 and 43 MB with few faults, not explained yet.
  • Limits: 3 runs a side, one PC (AMD on Linux), on a build before #1121. The field entry's spikes stay with it on, which fits the GPU-sync waits being a separate cause.
run06, RX_VK_CAPTURE_NT=0:
[hos sead::MainThread] vk: slow capture 1061.6 ms: lock 0.1, scan 0.3, late pages 0.0, buffer 972.8 (made on this thread), copy 88.2; 64261 pages (251.0 MB), 59 minor faults, 0 late
run08, RX_VK_CAPTURE_NT=1:
[hos sead::MainThread] vk: slow capture 306.7 ms: lock 0.0, scan 0.2, late pages 0.0, buffer 301.5 (made on this thread), copy 4.9; 10315 pages (40.3 MB), 39 minor faults, 0 late

On main after #1121: RX_VK_FRAME_GATE x RX_VK_CAPTURE_NT

A second test build, main 819a5967 (after #1121) with the same two probes (longshot abc-probes-main (source 80057f9b)); a warm-up run, then 2 runs a cell in the order A B C D D C B A (run12 to run19).

Each cell's two runs gate off, NT off (main) gate off, NT on gate on, NT off gate on, NT on
Slow captures (50 ms or more), whole run 20, 18 2, 0 21, 23 2, 0
Measured 120 s: 1% low 12.4, 19.2 fps 29.6, 30.1 fps 17.2, 9.6 fps 27.7, 28.2 fps
Measured 120 s: worst frame 942, 313 ms 50, 68 ms 727, 1444 ms 53, 67 ms
Measured 120 s: frames of 100 ms or more 5, 7 0, 0 4, 9 0, 0
Measured 120 s: fps 54.4, 56.9 57.3, 57.4 54.1, 52.0 55.6, 55.5
Field entry: worst frame 465, 282 ms 787, 333 ms 594, 328 ms 558, 275 ms
GPU-sync waits of 100 ms or more (longest) 10, 8 (496, 457 ms) 12, 10 (466, 457 ms) 10, 10 (444, 463 ms) 11, 9 (462, 466 ms)
  • NT holds on main: with it on, no run had a frame of 100 ms or more in the measured part, and the 1% low was 27.7-30.1 fps against 9.6-19.2.
  • #1121's gate makes no clear difference on this route, either way.
  • The field entry's spikes are the GPU-sync waits, in every cell; neither switch touches them.
  • Limits: 2 runs a cell, one PC (AMD on Linux).

The field entry's longest stalls: likely the GPU's ASTC work

Twenty runs on main 819a5967 with the two probes, RX_VK_CAPTURE_NT=1 (the capture copy out of the way) and HOS_NVN_SYNC_LOG=1 (run20 to run39).

  • The CPU side is fine; the GPU is behind. In the worst wait (~460 ms) the render thread waits 250 ms for a submission's write-back and times out. A local probe at that timeout showed the submission recorded and sent, nothing held in a batch, and the GPU 3 recordings short of it. RX_VK_MIRROR_SPARSE=0, RX_VK_BATCH=0 and RX_VK_RECORDER=off (1 run: the second was ended by the kernel's OOM killer, a memory jump not looked into) changed nothing.
  • The GPU is at 100% through those waits (amdgpu gpu_busy_percent every 50 ms), against a median of 34-42% through play; the log's GPU time line (longest submission about 20 ms) doesn't show it.
  • The ASTC work is the likely part. This card doesn't sample ASTC: it is decoded on the GPU, and the big textures are re-encoded to BC3 on the GPU too.
Each side's two runs default (run34, run39) RX_VK_ASTC_RGBA8=1: no BC3 re-encode (run36, run37) RX_VK_ASTC_CPU=1 (run35, run38)
Main-thread GPU-sync waits of 100 ms or more 8, 7 5, 6 74, 73
Longest 460, 463 ms 311, 326 ms 20.6, 20.7 s (the GPU idle)
A capture's write-back wait timed out at 250 ms 1, 1 0, 0 0, 0
Measured 120 s: fps, 1% low (median of 2) 57.8, 28.9 57.2, 29.7 39.9, 3.4
  • Measured: skipping the BC3 re-encode takes about 150 ms off the worst stall and removes the timeouts. Not measured: the remaining ~300 ms, the GPU still at 100%: the decode is the likely candidate, but no run took it off the GPU without moving it to the CPU, which is far worse here (that path is meant for integrated GPUs).
  • The upload budget doesn't spread it (from reading upload_defer on main, not tested): it bounds the CPU time in upload(), 12 ms a frame, not the decode and encode's GPU time, and never defers terrain textures (layer arrays, or over 1024x1024).
  • The waits with the GPU idle are slow captures. About 2.5 s before the measured part, stepping out of the gate, the main thread waits 205-784 ms on a capture copy (36-85 MB at about 110-180 MB/s with RX_VK_CAPTURE_NT=1), or on the Presentation Thread's. With 21-102 minor faults and half or more of the time off the CPU, they aren't the fault cost NT removes; pages read back in from swap (major faults) is a guess, not yet measured.
  • Limits: 2 runs a side (1 for the recorder off), one PC (AMD on Linux). The runs from run30 on were made in a systemd unit capped at 20 GB with no swap (the earlier runs used none); the cap never acted.

Flags and settings

None

Testing

Tested: the thirty-nine runs on this PC (above). On main 868f6be8 (the base then): python -I -B tests/test_tested_configurations.py: OK; mise run ci:smoke: OK (6 tests); mise run test (tests/run_all.py --strict): 695 passed, 139 failed, 17 skipped on this branch and on main 868f6be8, the same 139 checks on both. Rebased onto main 8b0ea786 (08756302): tests/test_tested_configurations.py and mise run ci:smoke OK again; the five newer commits on main touch no docs/tested/ file. Rebased onto main 819a5967 with the A/B's six files (d3ab4f66): tests/test_tested_configurations.py OK; onto main ad9fa48e with the main runs' eight (3aa8225b): tests/test_tested_configurations.py OK; onto main 0de25c3e with the twenty newer runs (5da79b44): tests/test_tested_configurations.py OK. Most of them are tools this PC doesn't have, which --strict fails instead of skipping (the Vulkan headers, clang, the SPIR-V headers); none reads docs/tested/ except tests/test_tested_configurations.py, which passes.

Not tested: nothing in the game: the PR changes no game files.

Must be tested:

  • None

Risks

What could break: nothing in the game; the catalogue page gains thirty-nine rows.

Turning it off: revert the commit.

Compatibility: Nothing changes.

Issues

See #1720
See #1667
See #1666
See #1599
See #1696
See #1735

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md).
  • Docs updated.

🤖 Generated with Claude Code

**Type:** docs · **Risk:** low · **Tested on:** Linux (Fedora 45), Ryzen 5 5600X, RX 9070 XT (RADV 26.2.7) · **Needs:** none ## TL;DR Walking into Hyrule Field and turning the camera, the game drops to a few fps for up to a second (#1720). Runs with the probes from #1667 and #1666 point to two causes on an RX 9070 XT. Out in the field, the renderer's copy of freshly loaded memory is page-fault bound, and non-temporal copies (`RX_VK_CAPTURE_NT=1`) cut its long frames on every run, on main after #1121 too. At the field entry, the longest waits look like the GPU's ASTC work: ~450 ms at 100% GPU that the log's GPU time doesn't show, a third of it the BC3 re-encode; a few others are slow captures. This PR records the thirty-nine runs in `docs/tested/`; it changes no game code. ## Breakdown - **Who it helps:** whoever works on the field's drops (#1720, #1696, #1735, #1667, #1666, #1599): the runs behind the comments are on record. - **What was wrong:** nothing in the code; the diagnostic runs weren't in the tested-configurations catalogue. - **What changes:** thirty-nine rows in `docs/tested/`, one per run. - **What does not change:** the game, the frozen table, every other docs file. - **How it was checked:** `tests/test_tested_configurations.py` passes; the full suite's results are under Testing. - **What is still open:** whether `RX_VK_CAPTURE_NT=1` helps beyond AMD on Linux (it is opt-in for an RTX desktop's results); the capture buffers made on the capturing thread, which are most of what is left with it on; how to spread the ASTC decode and BC3 re-encode's GPU time at an area's entry (the upload budget bounds only the CPU's); a few long waits with the GPU idle, not looked into. - **Risk:** low: thirty-nine docs rows. ## Why Since the 3.0 release candidates, players see the frame rate collapse for a moment in Hyrule Field, worst when the field streams in or the camera turns (#1720 on Windows with an RTX 4070 Ti; the same here on Linux with an RX 9070 XT). Similar drops when circling or attacking are in #1696 and #1735; one player in #1735 dates them to rc.8, so the release that brought them is not settled. The probes in #1667 (`RX_VK_CAPTURE_SLOW_MS`) and #1666 (`HOS_AWAY_LOG_MS`) showed the main thread's two kinds of long waits on this route: the capture at a submit, and waiting on a GPU sync (results: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1667#issuecomment-53871 and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1666#issuecomment-53875). Five runs here follow each one up with a switch, six more are an interleaved A/B of the one that looked best, eight more repeat it on main after #1121, twenty more follow the field entry's GPU-sync waits, and `docs/tested/` is where a measured run goes (CONTRIBUTING.md, "Measurements"). ## What changed Thirty-nine run files, following docs/tested/README.md, one a run: - `docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run01.md`: `RX_VK_CAPTURE_THREADS=0`. - `docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run02.md`: `RX_VK_CAPTURE_NT=1`. - `docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run03.md`: `RX_VK_CAP_BLOCKS=1`. - `docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run04.md`: `HOS_NVN_SYNC_LOG=1`. - `docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run05.md`: `HOS_NVN_SYNC_LOG=2440`. - `docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run06.md` to `run11.md`: the A/B, `RX_VK_CAPTURE_NT=0` and `=1` in the order off, on, on, off, off, on. - `docs/tested/2026-11-19-capture-diagnostics-field-grimvisage-run12.md` to `run19.md`: on main 819a5967 with the two probes, a 2x2 of `RX_VK_FRAME_GATE` and `RX_VK_CAPTURE_NT`, 2 runs a cell. - `run20.md` to `run39.md`: the field entry's GPU-sync waits on the same build, with `RX_VK_CAPTURE_NT=1` and `HOS_NVN_SYNC_LOG=1`: two plain runs (run20-21); `RX_VK_MIRROR_SPARSE` on and off (run22-25); `RX_VK_BATCH=0` and `RX_VK_RECORDER=off` (run26-29); then on a build with one more local probe line at a capture's timed-out write-back wait, not in any PR: two runs (run30-31), two with the GPU's busy percent sampled (run32-33), and `RX_VK_ASTC_CPU=1` and `RX_VK_ASTC_RGBA8=1` against the default (run34-39). ## Measurements **Verdict:** nothing to measure in the game for this PR: docs only. The files pass their format check, and the game-free suite gives the same result as on main (CONTRIBUTING.md, "Measurements": `docs`). | | Before: main @ 868f6be8 | After: this PR @ 3aa8225b | |---|---|---| | `tests/test_tested_configurations.py` | OK | OK, with the nineteen new files; again OK at 5da79b44 on main 0de25c3e with all thirty-nine | | `mise run test` (`tests/run_all.py --strict`) | 695 passed, 139 failed, 17 skipped | 695 passed, 139 failed, 17 skipped: the same 139 | Baseline: these are diagnostics, one run a switch, on one build (`longshot abc-probes (source 166c4da4)`: main 2f1c0c9c with #1667 a871f3ff and #1666 4ec23e96), not a main-against-PR comparison; the two probe runs on #1667 are the same build with no switch. Hardware: Fedora 45 (kernel 7.3.4), KDE Plasma 6.8.3 (Wayland), RX 9070 XT 16 GB, Mesa RADV 26.2.7, Ryzen 5 5600X (12 logical processors), 32 GB Scene: Hyrule Field (child) at 12:00, out of the Castle Town gate; 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), frame generation off, no frame cap Method: a scripted route through the control bridge from the repository save `kokiri-start`: the teleport to the Market, the clock at 12:00, out of the gate, then 120 s of laps between 20 m and 320 m with a 180° camera turn every 10 s (the trigger in #1720); frames from `HOS_PROFILE=1`'s presents trace, the captures from `RX_VK_CAPTURE_SLOW_MS=50`, the waits from `HOS_AWAY_LOG_MS=50`, each thread's user and kernel CPU time and minor faults from `/proc` every 200 ms How to reproduce: the same switches on any build carrying #1667 and #1666 ### What the runs show | From the field entry to the end of the run | Default copy (run04 / run05) | `RX_VK_CAPTURE_THREADS=0` (run01) | `RX_VK_CAPTURE_NT=1` (run02) | `RX_VK_CAP_BLOCKS=1` (run03) | |---|---|---|---|---| | Copy rate of the slow captures (median) | 1.02 / 1.04 GB/s | 0.97 GB/s (one thread) | no capture of 50 ms or more | 1.05 GB/s | | Copy helpers' CPU time in the kernel | 38% / 37% | (no helpers) | 7% | 40% | | Minor faults, copy helpers | 52859 / 73789 | 463605 on the capturing thread | 7285 | 30542 | | Field entry: frames of 100 ms or more, worst | 16 / 13, 454 / 597 ms | 14, 455 ms | 9, 272 ms | 14, 477 ms | | Measured 120 s: 1% low, worst | 15.1 / 17.3 fps, 509 / 277 ms | 15.3 fps, 677 ms | 28.6 fps, 54 ms | 14.7 fps, 1446 ms | - **The copy is bound by page faults.** One thread copies as fast as five (0.97 against 1.02-1.07 GB/s), and with the helpers off the capturing thread takes 0.85 minor faults per copied page: nearly every 4 KB page faults. The capture buffer is cached host memory (`create_buffer_cached`) and the game's memory is imported in place as dma-bufs, so neither end is slow memory on paper. - **Non-temporal stores cut the faults by 7 to 10 times** and the helpers' kernel time from 37-40% to 7%, and that run had no slow capture at all. It's one run, and `RX_VK_CAPTURE_NT` is opt-in because four field runs on an RTX desktop showed no gain and more stutters with it (the comment at `capture_nt()` in `gpu/vk/vk_memory.cpp`): the A/B below repeats it. - **The GPU-sync waits wait for the renderer.** Each of run04's eight waits of 100 ms or more (125 to 461 ms) is on the frame before's fence, already handed over, with the replay 4 submissions behind it (6 once). In the probe runs the render thread sat idle at its whole-frame gate through such a wait and the Presentation Thread waited for the CPU's signal in `present()`: where the work is held is still open. #1121 (merged after these runs) turns that whole-frame gate off by default (`RX_VK_FRAME_GATE=1` brings it back), so these waits were run on main again: they stay (below). ``` run01, RX_VK_CAPTURE_THREADS=0: [hos sead::MainThread] vk: slow capture 264.7 ms: lock 0.1, scan 0.2, late pages 0.0, buffer 18.0 (made on this thread), copy 246.1; 59224 pages (231.3 MB), 48588 minor faults, 0 late run04, HOS_NVN_SYNC_LOG=1: [hos sead::MainThread] nvn: SyncWait 0x1010073b9020 fence 10172 timeout 18446744073709551615 (lr 0x8042cde0); latest 10175, done 10168 [hos sead::MainThread] away: sead::MainThread 460.9 ms (on a CPU 0.9 ms) from longshot+0x1f73a4b8, ended at 25725400343988 ns ``` ### The A/B of `RX_VK_CAPTURE_NT` Six runs on the same build, after a warm-up run that isn't counted, in the order off, on, on, off, off, on (run06 to run11); each figure is a run's, in that order within a side. | | `RX_VK_CAPTURE_NT=0`, the default (run06, run09, run10) | `RX_VK_CAPTURE_NT=1` (run07, run08, run11) | |---|---|---| | Slow captures (50 ms or more), whole run | 16, 21, 16 | 5, 6, 2 | | Copy rate of the slow captures (median) | 1.05, 0.94, 1.03 GB/s | 10.15, 10.16, 7.92 GB/s | | Copy helpers: minor faults, kernel time | 81597, 164061, 148658; 33%, 39%, 37% | 21350, 23172, 25739; 8%, 12%, 8% | | Measured 120 s: 1% low | 19.2, 10.5, 16.9 fps | 20.3, 28.1, 27.4 fps | | Measured 120 s: worst frame | 267, 2612, 797 ms | 302, 56, 113 ms | | Measured 120 s: fps, p99 | 54.0, 52.4, 54.3; 31.0, 32.2, 33.0 ms | 54.7, 54.7, 54.3; 30.0, 29.8, 29.6 ms | | Field entry: worst frame, frames of 100 ms or more | 1185, 570, 479 ms; 15, 17, 14 | 255, 885, 287 ms; 13, 14, 13 | - **Separated on every run:** fewer slow captures, 3.5 to 7 times fewer helper faults, and a higher 1% low with it on. The worst frame and the field entry overlap between the sides (medians 797 against 113 ms and 570 against 287 ms), and the average fps hardly moves. - **What is left with it on:** the copy is 1 to 5 ms; most remaining slow captures are a capture buffer made on the capturing thread, 150 to 300 ms even for 8 to 40 MB. Two copies still took 156 and 217 ms for 28 and 43 MB with few faults, not explained yet. - **Limits:** 3 runs a side, one PC (AMD on Linux), on a build before #1121. The field entry's spikes stay with it on, which fits the GPU-sync waits being a separate cause. ``` run06, RX_VK_CAPTURE_NT=0: [hos sead::MainThread] vk: slow capture 1061.6 ms: lock 0.1, scan 0.3, late pages 0.0, buffer 972.8 (made on this thread), copy 88.2; 64261 pages (251.0 MB), 59 minor faults, 0 late run08, RX_VK_CAPTURE_NT=1: [hos sead::MainThread] vk: slow capture 306.7 ms: lock 0.0, scan 0.2, late pages 0.0, buffer 301.5 (made on this thread), copy 4.9; 10315 pages (40.3 MB), 39 minor faults, 0 late ``` ### On main after #1121: `RX_VK_FRAME_GATE` x `RX_VK_CAPTURE_NT` A second test build, main 819a5967 (after #1121) with the same two probes (`longshot abc-probes-main (source 80057f9b)`); a warm-up run, then 2 runs a cell in the order A B C D D C B A (run12 to run19). | Each cell's two runs | gate off, NT off (main) | gate off, NT on | gate on, NT off | gate on, NT on | |---|---|---|---|---| | Slow captures (50 ms or more), whole run | 20, 18 | 2, 0 | 21, 23 | 2, 0 | | Measured 120 s: 1% low | 12.4, 19.2 fps | 29.6, 30.1 fps | 17.2, 9.6 fps | 27.7, 28.2 fps | | Measured 120 s: worst frame | 942, 313 ms | 50, 68 ms | 727, 1444 ms | 53, 67 ms | | Measured 120 s: frames of 100 ms or more | 5, 7 | 0, 0 | 4, 9 | 0, 0 | | Measured 120 s: fps | 54.4, 56.9 | 57.3, 57.4 | 54.1, 52.0 | 55.6, 55.5 | | Field entry: worst frame | 465, 282 ms | 787, 333 ms | 594, 328 ms | 558, 275 ms | | GPU-sync waits of 100 ms or more (longest) | 10, 8 (496, 457 ms) | 12, 10 (466, 457 ms) | 10, 10 (444, 463 ms) | 11, 9 (462, 466 ms) | - **NT holds on main:** with it on, no run had a frame of 100 ms or more in the measured part, and the 1% low was 27.7-30.1 fps against 9.6-19.2. - **#1121's gate makes no clear difference** on this route, either way. - **The field entry's spikes are the GPU-sync waits,** in every cell; neither switch touches them. - **Limits:** 2 runs a cell, one PC (AMD on Linux). ### The field entry's longest stalls: likely the GPU's ASTC work Twenty runs on main 819a5967 with the two probes, `RX_VK_CAPTURE_NT=1` (the capture copy out of the way) and `HOS_NVN_SYNC_LOG=1` (run20 to run39). - **The CPU side is fine; the GPU is behind.** In the worst wait (~460 ms) the render thread waits 250 ms for a submission's write-back and times out. A local probe at that timeout showed the submission recorded and sent, nothing held in a batch, and the GPU 3 recordings short of it. `RX_VK_MIRROR_SPARSE=0`, `RX_VK_BATCH=0` and `RX_VK_RECORDER=off` (1 run: the second was ended by the kernel's OOM killer, a memory jump not looked into) changed nothing. - **The GPU is at 100% through those waits** (amdgpu `gpu_busy_percent` every 50 ms), against a median of 34-42% through play; the log's GPU time line (longest submission about 20 ms) doesn't show it. - **The ASTC work is the likely part.** This card doesn't sample ASTC: it is decoded on the GPU, and the big textures are re-encoded to BC3 on the GPU too. | Each side's two runs | default (run34, run39) | `RX_VK_ASTC_RGBA8=1`: no BC3 re-encode (run36, run37) | `RX_VK_ASTC_CPU=1` (run35, run38) | |---|---|---|---| | Main-thread GPU-sync waits of 100 ms or more | 8, 7 | 5, 6 | 74, 73 | | Longest | 460, 463 ms | 311, 326 ms | 20.6, 20.7 s (the GPU idle) | | A capture's write-back wait timed out at 250 ms | 1, 1 | 0, 0 | 0, 0 | | Measured 120 s: fps, 1% low (median of 2) | 57.8, 28.9 | 57.2, 29.7 | 39.9, 3.4 | - **Measured: skipping the BC3 re-encode takes about 150 ms off the worst stall** and removes the timeouts. **Not measured: the remaining ~300 ms,** the GPU still at 100%: the decode is the likely candidate, but no run took it off the GPU without moving it to the CPU, which is far worse here (that path is meant for integrated GPUs). - **The upload budget doesn't spread it** (from reading `upload_defer` on main, not tested): it bounds the CPU time in `upload()`, 12 ms a frame, not the decode and encode's GPU time, and never defers terrain textures (layer arrays, or over 1024x1024). - **The waits with the GPU idle are slow captures.** About 2.5 s before the measured part, stepping out of the gate, the main thread waits 205-784 ms on a capture copy (36-85 MB at about 110-180 MB/s with `RX_VK_CAPTURE_NT=1`), or on the Presentation Thread's. With 21-102 minor faults and half or more of the time off the CPU, they aren't the fault cost NT removes; pages read back in from swap (major faults) is a guess, not yet measured. - **Limits:** 2 runs a side (1 for the recorder off), one PC (AMD on Linux). The runs from run30 on were made in a systemd unit capped at 20 GB with no swap (the earlier runs used none); the cap never acted. ### Flags and settings None ## Testing **Tested:** the thirty-nine runs on this PC (above). On main 868f6be8 (the base then): `python -I -B tests/test_tested_configurations.py`: OK; `mise run ci:smoke`: OK (6 tests); `mise run test` (`tests/run_all.py --strict`): 695 passed, 139 failed, 17 skipped on this branch and on main 868f6be8, the same 139 checks on both. Rebased onto main 8b0ea786 (08756302): `tests/test_tested_configurations.py` and `mise run ci:smoke` OK again; the five newer commits on main touch no `docs/tested/` file. Rebased onto main 819a5967 with the A/B's six files (d3ab4f66): `tests/test_tested_configurations.py` OK; onto main ad9fa48e with the main runs' eight (3aa8225b): `tests/test_tested_configurations.py` OK; onto main 0de25c3e with the twenty newer runs (5da79b44): `tests/test_tested_configurations.py` OK. Most of them are tools this PC doesn't have, which `--strict` fails instead of skipping (the Vulkan headers, clang, the SPIR-V headers); none reads `docs/tested/` except `tests/test_tested_configurations.py`, which passes. **Not tested:** nothing in the game: the PR changes no game files. **Must be tested:** - None ## Risks **What could break:** nothing in the game; the catalogue page gains thirty-nine rows. **Turning it off:** revert the commit. **Compatibility:** Nothing changes. ## Issues See #1720 See #1667 See #1666 See #1599 See #1696 See #1735 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`). - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
docs(tested): capture and GPU-sync diagnostics in Hyrule Field on an RX 9070 XT (RADV)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
08756302f0
Five diagnostic runs of the probes from #1667 and #1666 on the field route with camera turns: the copy
helpers off, non-temporal copies, shared capture blocks, and the sync log twice.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
GrimVisage force-pushed docs/tested-capture-diagnostics-rdna4 from 08756302f0
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to d3ab4f66a9
Some checks failed
PR description / Description follows the template (pull_request_target) Has been cancelled
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-19 16:53:12 +00:00
Compare
GrimVisage force-pushed docs/tested-capture-diagnostics-rdna4 from d3ab4f66a9
Some checks failed
PR description / Description follows the template (pull_request_target) Has been cancelled
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 3aa8225bb0
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 18s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-19 17:57:32 +00:00
Compare
GrimVisage force-pushed docs/tested-capture-diagnostics-rdna4 from 3aa8225bb0
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 18s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 5da79b4432
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 16s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-19 22:05:37 +00:00
Compare
Author
Contributor

Closing: #1802 removed the tested-configurations catalogue (docs/tested/), so these run files have nowhere to go. The runs stay on record in this PR's description and comments, and the files in its diff.

Closing: #1802 removed the tested-configurations catalogue (docs/tested/), so these run files have nowhere to go. The runs stay on record in this PR's description and comments, and the files in its diff.
GrimVisage closed this pull request 2026-11-20 02:22:27 +00:00
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 16s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1749
No description provided.