WIP: perf(vk): the host's question how far the replay is answered from the timeline as last read (RX_VK_REPLAYED_CACHE=0 to disable) #1994

Draft
siantm wants to merge 3 commits from siantm/Longshot:perf/vk-replayed-cache into main
Contributor

Summary

The host asks how far the replay is (get_replayed) under its recordings' lock, held exclusively, at every
nvnCommandBufferAddControlMemory (free_dead_copies in host/nvn_native.c), to let go of the copies the GPU is past,
and every answer was a vkGetSemaphoreCounterValue.

With this, completed() keeps the highest value it has read successfully (it only goes up), and get_replayed answers
from it, reading the driver at the first call and every 64th call after, so that it moves when nothing replays. While
the bench holds the answer (replayed_hold) the driver is read at every call, and those calls are not counted. A read
that fails keeps nothing and answers 0 (nothing taken as done); one that comes back VK_ERROR_DEVICE_LOST goes to
device_lost(), as the GPU hang watch's read does. The result expected: far fewer driver calls under the host's lock.

The cost: an answer lower than the GPU keeps a copy the GPU is past until a later answer passes it, up to 63 calls when
nothing else reads the timeline, however long those calls take to come; the copies kept meanwhile go together, in one
pass under the lock. So the host's report every 50,000 recordings (nvn: in the log) now also says the bytes of the
copies let go and waiting for the replay, their peak, and the passes that let them go with their time under the copies'
lock, since the last report: what a run needs to tell a shorter lock from larger, rarer passes.

The switch: RX_VK_REPLAYED_CACHE (new, a kill switch, bool, its row in gpu/vk/switches.def), on by default as the
branch shipped it, read and logged once by replay_start, before the timeline is made and the renderer is given to the
host; empty, 0, off, false or no read the driver at every call and keep no last-read value, as main does.

From bd3ea307 on exp/x86-asm-sync (its replay-parts work).

Read first: replayed_cache_on, completed and get_replayed in gpu/vk/vk_replay.cpp, free_dead_copies and the
report in host/nvn_native.c, then tests/test_replayed_cache.py.

Testing

Tested: Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head d8a583a6 on main f1f2dc5b. Build
jobs of this PC's job queue (fb-1994-1, fb-1995-1, fb-1995-2), the checks in a hidden console: tests/test_replayed_cache.py
fails at the test commit 4b3e94e7 (the failed read kept, the lost device not reported, the off switch keeping the
value) and passes at the head; tests/run_all.py --strict --changed origin/main at the head (the CPM cache's Vulkan headers): 169 checks, 135 passed, 29 failed, 5 skipped (Linux only or a device); the 29 fail on main @ f1f2dc5b too, run from a main worktree in the same job: tests/test_capture_readback_visibility.py, tests/test_crash_dialog.py, tests/test_crash_record.py, tests/test_dev_ab.py, tests/test_dev_packs.py, tests/test_device_lost_exit.py, tests/test_draw_stream_visibility.py, tests/test_flush_nothing_held.py, tests/test_frame_overlap.py, tests/test_fsr_image_retirement.py, tests/test_fsr_uniform_ownership.py, tests/test_gpu_wait_stats.py, tests/test_host_guest_writes.py, tests/test_linux_pressure.py, tests/test_mesh_conv.py, tests/test_perf_report.py, tests/test_present_queue_default.py, tests/test_present_source_crop.py, tests/test_profiling_mode.py, tests/test_recorder_own_idle.py, tests/test_renderer_new_handler.py, tests/test_replay_write_order.py, tests/test_slow_frames.py, tests/test_sparse_mirror.py, tests/test_target_reuse_memory.py, tests/test_ub_update_batch.py, tests/test_upscaler_rows.py, tests/test_vk_occlusion_queries.py, tests/test_walk_key.py; gpu/vk/vk_replay.cpp and host/nvn_native.c compiled (syntax) with the game
build's own commands.

It has NOT run in the game on this PC yet.

Still to test:

  • Windows 11, i7-9700, RTX 4060 (this PC): Hyrule Field, play three minutes running and turning the camera, and a
    scene change that recycles recordings (into and out of a shrine): main @ f1f2dc5b, this branch with
    RX_VK_REPLAYED_CACHE=0 and this branch with it unset (on), each warmed up a minute in the scene first, interleaved
    (ABBA), at least two runs each, expecting the same picture, no FATAL, device lost or replay error; report average FPS,
    1% lows, median and p99 frame times, the nvn: report's copies waiting (MB, peak) and let-go passes (count, time,
    longest) and the process's peak RAM; a gain is median and p99 frame time unset lower than with
    RX_VK_REPLAYED_CACHE=0 and than main, with no rise in the peak of copies waiting or in the longest pass,
    and RX_VK_REPLAYED_CACHE=0 the same as main (by hand)
  • Linux NVIDIA and AMD, Windows AMD and Intel, Steam Deck: the same scene and scene change, three minutes: main @
    f1f2dc5b, this branch with RX_VK_REPLAYED_CACHE=0 and with it unset (on), warmed up, ABBA, expecting the same picture
    and no FATAL, device lost or replay error, and the same figures as above (by hand or on the farm, not run yet)
  • Linux: build the renderer and the host and run python3 -I -B tests/test_replayed_cache.py, expecting PASS (by hand)

Measurements

Verdict: not measured: no micro-benchmark of this port and no run in the game on this PC yet; the baseline is main
@ f1f2dc5b, and the in-game A/B is in Still to test. RX_VK_REPLAYED_CACHE ships on, as the branch shipped it; no
measurement chose that default yet, the in-game A/B in Still to test does.

Risks

What could break: an answer above what the GPU reached would free a copy it still reads (a wrong draw or a device
lost); the answer is never above a value the driver returned from a read that succeeded. At worst it is lower: a copy
the GPU is past stays up to 63 calls longer when nothing else reads the timeline (longer in wall time when the calls are
sparse), so more memory is held between passes and a pass under the copies' lock lets more go at once. The test runs
the real completed(), get_replayed() and the host's free_dead_copies() against a stand-in driver. Turn it off with
RX_VK_REPLAYED_CACHE=0. Renderer and host code: the runtime is rebuilt; nothing changes for players' saves, options or
caches.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A switch added or changed: it ships at the default the measurements chose, with its row in gpu/vk/switches.def.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary The host asks how far the replay is (`get_replayed`) under its recordings' lock, held exclusively, at every `nvnCommandBufferAddControlMemory` (`free_dead_copies` in `host/nvn_native.c`), to let go of the copies the GPU is past, and every answer was a `vkGetSemaphoreCounterValue`. With this, `completed()` keeps the highest value it has read successfully (it only goes up), and `get_replayed` answers from it, reading the driver at the first call and every 64th call after, so that it moves when nothing replays. While the bench holds the answer (`replayed_hold`) the driver is read at every call, and those calls are not counted. A read that fails keeps nothing and answers 0 (nothing taken as done); one that comes back `VK_ERROR_DEVICE_LOST` goes to `device_lost()`, as the GPU hang watch's read does. The result expected: far fewer driver calls under the host's lock. The cost: an answer lower than the GPU keeps a copy the GPU is past until a later answer passes it, up to 63 calls when nothing else reads the timeline, however long those calls take to come; the copies kept meanwhile go together, in one pass under the lock. So the host's report every 50,000 recordings (`nvn:` in the log) now also says the bytes of the copies let go and waiting for the replay, their peak, and the passes that let them go with their time under the copies' lock, since the last report: what a run needs to tell a shorter lock from larger, rarer passes. The switch: `RX_VK_REPLAYED_CACHE` (new, a kill switch, bool, its row in `gpu/vk/switches.def`), on by default as the branch shipped it, read and logged once by `replay_start`, before the timeline is made and the renderer is given to the host; empty, `0`, `off`, `false` or `no` read the driver at every call and keep no last-read value, as main does. From bd3ea307 on `exp/x86-asm-sync` (its replay-parts work). Read first: `replayed_cache_on`, `completed` and `get_replayed` in `gpu/vk/vk_replay.cpp`, `free_dead_copies` and the report in `host/nvn_native.c`, then `tests/test_replayed_cache.py`. ## Testing **Tested:** Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head `d8a583a6` on main `f1f2dc5b`. Build jobs of this PC's job queue (`fb-1994-1`, `fb-1995-1`, `fb-1995-2`), the checks in a hidden console: `tests/test_replayed_cache.py` fails at the test commit `4b3e94e7` (the failed read kept, the lost device not reported, the off switch keeping the value) and passes at the head; `tests/run_all.py --strict --changed origin/main` at the head (the CPM cache's Vulkan headers): 169 checks, 135 passed, 29 failed, 5 skipped (Linux only or a device); the 29 fail on main @ f1f2dc5b too, run from a main worktree in the same job: `tests/test_capture_readback_visibility.py`, `tests/test_crash_dialog.py`, `tests/test_crash_record.py`, `tests/test_dev_ab.py`, `tests/test_dev_packs.py`, `tests/test_device_lost_exit.py`, `tests/test_draw_stream_visibility.py`, `tests/test_flush_nothing_held.py`, `tests/test_frame_overlap.py`, `tests/test_fsr_image_retirement.py`, `tests/test_fsr_uniform_ownership.py`, `tests/test_gpu_wait_stats.py`, `tests/test_host_guest_writes.py`, `tests/test_linux_pressure.py`, `tests/test_mesh_conv.py`, `tests/test_perf_report.py`, `tests/test_present_queue_default.py`, `tests/test_present_source_crop.py`, `tests/test_profiling_mode.py`, `tests/test_recorder_own_idle.py`, `tests/test_renderer_new_handler.py`, `tests/test_replay_write_order.py`, `tests/test_slow_frames.py`, `tests/test_sparse_mirror.py`, `tests/test_target_reuse_memory.py`, `tests/test_ub_update_batch.py`, `tests/test_upscaler_rows.py`, `tests/test_vk_occlusion_queries.py`, `tests/test_walk_key.py`; `gpu/vk/vk_replay.cpp` and `host/nvn_native.c` compiled (syntax) with the game build's own commands. **It has NOT run in the game on this PC yet.** **Still to test:** - Windows 11, i7-9700, RTX 4060 (this PC): Hyrule Field, play three minutes running and turning the camera, and a scene change that recycles recordings (into and out of a shrine): main @ f1f2dc5b, this branch with `RX_VK_REPLAYED_CACHE=0` and this branch with it unset (on), each warmed up a minute in the scene first, interleaved (ABBA), at least two runs each, expecting the same picture, no FATAL, device lost or replay error; report average FPS, 1% lows, median and p99 frame times, the `nvn:` report's copies waiting (MB, peak) and let-go passes (count, time, longest) and the process's peak RAM; a gain is median and p99 frame time unset lower than with `RX_VK_REPLAYED_CACHE=0` and than main, with no rise in the peak of copies waiting or in the longest pass, and `RX_VK_REPLAYED_CACHE=0` the same as main (by hand) - Linux NVIDIA and AMD, Windows AMD and Intel, Steam Deck: the same scene and scene change, three minutes: main @ f1f2dc5b, this branch with `RX_VK_REPLAYED_CACHE=0` and with it unset (on), warmed up, ABBA, expecting the same picture and no FATAL, device lost or replay error, and the same figures as above (by hand or on the farm, not run yet) - Linux: build the renderer and the host and run `python3 -I -B tests/test_replayed_cache.py`, expecting PASS (by hand) ## Measurements **Verdict:** not measured: no micro-benchmark of this port and no run in the game on this PC yet; the baseline is main @ f1f2dc5b, and the in-game A/B is in Still to test. `RX_VK_REPLAYED_CACHE` ships on, as the branch shipped it; no measurement chose that default yet, the in-game A/B in Still to test does. ## Risks What could break: an answer above what the GPU reached would free a copy it still reads (a wrong draw or a device lost); the answer is never above a value the driver returned from a read that succeeded. At worst it is lower: a copy the GPU is past stays up to 63 calls longer when nothing else reads the timeline (longer in wall time when the calls are sparse), so more memory is held between passes and a pass under the copies' lock lets more go at once. The test runs the real `completed()`, `get_replayed()` and the host's `free_dead_copies()` against a stand-in driver. Turn it off with `RX_VK_REPLAYED_CACHE=0`. Renderer and host code: the runtime is rebuilt; nothing changes for players' saves, options or caches. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. - [ ] `mise run test` passes (any failure that also fails on `main` is named). - [ ] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [ ] A switch added or changed: it ships at the default the measurements chose, with its row in `gpu/vk/switches.def`. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [ ] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
perf(vk): the host's question how far the replay is answered from the timeline as last read (RX_VK_REPLAYED_CACHE)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 8s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
ab3a2544a1
The host asks get_replayed under its recordings' lock, held exclusively (host/nvn_native.c free_dead_copies, at every
nvnCommandBufferAddControlMemory: 117 a frame in a CPU profile of Hyrule Field on the branch), to let go of the copies the
GPU is past, and every answer was a vkGetSemaphoreCounterValue there. completed() now keeps the highest value it has read
(completed_seen, it only goes up), and get_replayed answers from it, reading the driver at every 64th call so that it moves
when nothing replays. A lower answer only keeps a copy until a later call. Not while the bench holds the answer
(replayed_hold): the driver is read as before.

From the branch's replay-parts work (bd3ea307). Not measured: neither on the branch nor in play.

Test: tests/test_replayed_cache.py cuts completed(), replayed_cache_on() and get_replayed() from vk_replay.cpp and runs
them against a stand-in driver (63 of 64 answers without a driver call, the bench's hold and =0 read every time).

Switch: RX_VK_REPLAYED_CACHE (kill), on; =0 reads the driver at every call, as before.

Switch reads: the test also runs RX_VK_REPLAYED_CACHE as the source reads it, in fresh processes (unset, 0, 1, 2, off, each twice, the variable flipped after the first read: read once).

Assisted-by: Claude Code
Member

Possibly related, for context:

  • PR #1995 WIP: perf(vk): fewer driver calls at a recording's end and start, and a submit thread where nothing else submits (RX_VK_SUBMIT_BATCH=0 to disable) — Open Vulkan replay optimization depends on this PR and includes its changes
Possibly related, for context: - PR #1995 WIP: perf(vk): fewer driver calls at a recording's end and start, and a submit thread where nothing else submits (RX_VK_SUBMIT_BATCH=0 to disable) — Open Vulkan replay optimization depends on this PR and includes its changes
BeanBot left a comment

Recommendation: Request changes. Publish only successful timeline queries and make the kill switch disable cache updates as well as cache reads. The fixture shows fewer driver calls, but the player benefit and the cost of delayed reclamation are still unmeasured.

The host usually reuses a recent GPU completion value while holding its recording-copy lock. That can shorten the lock hold, but it can also delay returning dead copies to the pool and concentrate reclamation into fewer passes. The new cache also adds atomic work to completion queries elsewhere in the renderer.

@fix will do (11)

  1. L1 gpu/vk/vk_replay.cpp:284: The new cache publishes the output without checking VkResult. (more on the line)
  2. L2 gpu/vk/vk_replay.cpp:285: RX_VK_REPLAYED_CACHE=0 restores the driver's query frequency but leaves cache maintenance active in every completed() caller. (more on the line)
  3. L3 (minor) gpu/vk/vk_replay.cpp:3053: The first reclamation query can log while the host holds copy_lock exclusively. (more on the line)
  4. L4 tests/test_replayed_cache.py:27: The test's compiler fallback can be bypassed in both directions. (more on the line)
  5. L5 (minor) tests/test_replayed_cache.py:128: The read-once check ignores failure from _putenv_s and setenv. (more on the line)
  6. L6 (minor) tests/test_replayed_cache.py:138: ATTEMPT_OFF and the stubs parameter are unused, and the test compiles separate binaries to drive one switch through a stub and through the real helper. (more on the line)
  7. L7 (minor): Risks says a copy can live only a call longer. If the GPU advances just after a refresh and no other caller queries it, the next 63 eligible polls can reuse the older value. Sparse polls can stretch that delay in wall time. Describe the actual refresh interval and the potential for delayed pool reuse and batched reclamation.
  8. L8 (minor) gpu/vk/switches.def:640: The row omits the empty value, which disables the cache through env_on_or, and should describe the cadence as the first eligible call and every 64 eligible calls thereafter. (more on the line)
  9. L9 (minor): The description checks off being on current main, but local main includes #1948 after the stated base. Keep that box open until the branch is current and identify the updated baseline in the measurements.
  10. L10 (its part an edit can do) gpu/vk/vk_replay.cpp:3070: Record peak pending dead-copy bytes and reclamation time under copy_lock so the run can distinguish faster polling from larger reclamation bursts.
  11. L11 (its part an edit can do): Add a game-free host reclamation case that checks a completed copy stays retained on a stale answer and is released after the refresh. Driver and rendering coverage still require the game.

For the author (2) (a run, a measurement or a decision)

  • L10 (minor) gpu/vk/vk_replay.cpp:3070: The planned median and live-copy count cannot establish the gain or show its cost. (more on the line)
  • L11 (minor): The renderer change was opened without the author's in-game run required by the repository. The stand-in driver cannot check actual recording-copy reuse or rendering across a scene transition. Record an own-PC run on the reviewed head with the cache on and off, including a transition that recycles recordings, and report the picture and any device-loss or replay errors.

Checks: Still running: PR description / Description follows the template (pull_request_target), CI / Conventional Commit title (pull_request), CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request), CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request), CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request), CI / Workflow lint (actionlint) (pull_request), CI / Website build and checks (pull_request), Guide screenshots / Guide images and local UI source review (pull_request), Third-party parts / As their notes say (pull_request), Third-party parts / The same as upstream (pull_request).

More: does it help, other systems, switches, what was read

Does it help: Unproven. The author reports that the Windows fixture passed. It checks that 63 of 64 eligible calls skip the driver when no other path refreshes the cache. The commit cites 117 calls per frame in a Hyrule Field profile from the development branch. There are no matched in-game results for main, head with the switch off, and head with it on.

Other systems: The cache applies across Vulkan drivers and recorder modes. Batch flush polling still runs on every host query, including on RADV and Steam Deck, and a sampled benchmark hold still caps the answer. Sync-wait debug logs can now print stale progress. Waits, readbacks, screenshots, and renderer resource retirement still query the driver directly, but their queries can update the shared atomic. Delayed host-copy reuse can raise RAM use and bunch reclamation work under the lock. The diff touches no save, audio, input, shader-cache format, or renderer ABI code.

Switches: RX_VK_REPLAYED_CACHE is a new developer kill switch with a sorted table row. It defaults on when unset and is read once. Empty, 0, off, false, and no disable cached answers. Cache updates currently remain active with it off. The on default awaits matched results on Tier 1 hardware.

Reviewed: ab3a2544a1 against 75317e1f31: Read tests/test_replayed_cache.py in full, the complete diff and commit, and every changed section of gpu/vk/vk_replay.cpp and gpu/vk/switches.def. Traced the host callers, copy pool, recorder dispatch, batch flush, benchmark hold, logging, timeline lifetime, and test runner. Read AGENTS.md, CONTRIBUTING.md, the PR template, the standards, and the relevant architecture and switch rules. Skimmed unrelated recorded runs and source-check references. The rest of the replay file and switch table were outside the changed sections. There is no local review-checklist file or sibling Tooling checkout. No translations or generated files changed.; nothing run (a code review).

Checked and fine:

  • The monotonic compare-and-exchange is atomic, and the unsigned poll counter wraps without changing the cadence.
  • The poll still flushes eligible held submissions before taking the cached path. The supported startup path installs the timeline before exposing the renderer to the host.
  • The existing fixture would catch removal of the cache fast path, periodic refresh, flush poll, or benchmark cap. Its switch cases clear inherited values.
  • The diff stays on one topic, carries an Assisted-by trailer, and contains no game-derived artifacts, generated-file edits, third-party edits, or API layout changes.

Testing: Read-only review. No builds or tests were run. The author reports 131 of 158 Windows checks passed, including the new fixture, and all 27 failures reproduced on the base. Linux renderer and fixture results, in-game results, and measurements are pending. The supplied CI checks were still running and listed no head-only failures.

Overlaps and merge order: No overlapping pull request was identified from the supplied record or local history. The local main change for #1948 touches none of this PR's three files.


Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** Publish only successful timeline queries and make the kill switch disable cache updates as well as cache reads. The fixture shows fewer driver calls, but the player benefit and the cost of delayed reclamation are still unmeasured. The host usually reuses a recent GPU completion value while holding its recording-copy lock. That can shorten the lock hold, but it can also delay returning dead copies to the pool and concentrate reclamation into fewer passes. The new cache also adds atomic work to completion queries elsewhere in the renderer. **`@fix` will do (11)** 1. **L1** `gpu/vk/vk_replay.cpp:284`: The new cache publishes the output without checking VkResult. (more on the line) 2. **L2** `gpu/vk/vk_replay.cpp:285`: RX_VK_REPLAYED_CACHE=0 restores the driver's query frequency but leaves cache maintenance active in every completed() caller. (more on the line) 3. **L3** (minor) `gpu/vk/vk_replay.cpp:3053`: The first reclamation query can log while the host holds copy_lock exclusively. (more on the line) 4. **L4** `tests/test_replayed_cache.py:27`: The test's compiler fallback can be bypassed in both directions. (more on the line) 5. **L5** (minor) `tests/test_replayed_cache.py:128`: The read-once check ignores failure from _putenv_s and setenv. (more on the line) 6. **L6** (minor) `tests/test_replayed_cache.py:138`: ATTEMPT_OFF and the stubs parameter are unused, and the test compiles separate binaries to drive one switch through a stub and through the real helper. (more on the line) 7. **L7** (minor): Risks says a copy can live only a call longer. If the GPU advances just after a refresh and no other caller queries it, the next 63 eligible polls can reuse the older value. Sparse polls can stretch that delay in wall time. Describe the actual refresh interval and the potential for delayed pool reuse and batched reclamation. 8. **L8** (minor) `gpu/vk/switches.def:640`: The row omits the empty value, which disables the cache through env_on_or, and should describe the cadence as the first eligible call and every 64 eligible calls thereafter. (more on the line) 9. **L9** (minor): The description checks off being on current main, but local main includes #1948 after the stated base. Keep that box open until the branch is current and identify the updated baseline in the measurements. 10. **L10** (its part an edit can do) `gpu/vk/vk_replay.cpp:3070`: Record peak pending dead-copy bytes and reclamation time under copy_lock so the run can distinguish faster polling from larger reclamation bursts. 11. **L11** (its part an edit can do): Add a game-free host reclamation case that checks a completed copy stays retained on a stale answer and is released after the refresh. Driver and rendering coverage still require the game. **For the author (2)** (a run, a measurement or a decision) - **L10** (minor) `gpu/vk/vk_replay.cpp:3070`: The planned median and live-copy count cannot establish the gain or show its cost. (more on the line) - **L11** (minor): The renderer change was opened without the author's in-game run required by the repository. The stand-in driver cannot check actual recording-copy reuse or rendering across a scene transition. Record an own-PC run on the reviewed head with the cache on and off, including a transition that recycles recordings, and report the picture and any device-loss or replay errors. **Checks:** Still running: `PR description / Description follows the template (pull_request_target)`, `CI / Conventional Commit title (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request)`, `CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request)`, `CI / Workflow lint (actionlint) (pull_request)`, `CI / Website build and checks (pull_request)`, `Guide screenshots / Guide images and local UI source review (pull_request)`, `Third-party parts / As their notes say (pull_request)`, `Third-party parts / The same as upstream (pull_request)`. <details> <summary>More: does it help, other systems, switches, what was read</summary> **Does it help:** Unproven. The author reports that the Windows fixture passed. It checks that 63 of 64 eligible calls skip the driver when no other path refreshes the cache. The commit cites 117 calls per frame in a Hyrule Field profile from the development branch. There are no matched in-game results for main, head with the switch off, and head with it on. **Other systems:** The cache applies across Vulkan drivers and recorder modes. Batch flush polling still runs on every host query, including on RADV and Steam Deck, and a sampled benchmark hold still caps the answer. Sync-wait debug logs can now print stale progress. Waits, readbacks, screenshots, and renderer resource retirement still query the driver directly, but their queries can update the shared atomic. Delayed host-copy reuse can raise RAM use and bunch reclamation work under the lock. The diff touches no save, audio, input, shader-cache format, or renderer ABI code. **Switches:** RX_VK_REPLAYED_CACHE is a new developer kill switch with a sorted table row. It defaults on when unset and is read once. Empty, 0, off, false, and no disable cached answers. Cache updates currently remain active with it off. The on default awaits matched results on Tier 1 hardware. **Reviewed:** ab3a2544a1b1 against 75317e1f31ff: Read tests/test_replayed_cache.py in full, the complete diff and commit, and every changed section of gpu/vk/vk_replay.cpp and gpu/vk/switches.def. Traced the host callers, copy pool, recorder dispatch, batch flush, benchmark hold, logging, timeline lifetime, and test runner. Read AGENTS.md, CONTRIBUTING.md, the PR template, the standards, and the relevant architecture and switch rules. Skimmed unrelated recorded runs and source-check references. The rest of the replay file and switch table were outside the changed sections. There is no local review-checklist file or sibling Tooling checkout. No translations or generated files changed.; nothing run (a code review). **Checked and fine:** - The monotonic compare-and-exchange is atomic, and the unsigned poll counter wraps without changing the cadence. - The poll still flushes eligible held submissions before taking the cached path. The supported startup path installs the timeline before exposing the renderer to the host. - The existing fixture would catch removal of the cache fast path, periodic refresh, flush poll, or benchmark cap. Its switch cases clear inherited values. - The diff stays on one topic, carries an Assisted-by trailer, and contains no game-derived artifacts, generated-file edits, third-party edits, or API layout changes. **Testing:** Read-only review. No builds or tests were run. The author reports 131 of 158 Windows checks passed, including the new fixture, and all 27 failures reproduced on the base. Linux renderer and fixture results, in-game results, and measurements are pending. The supplied CI checks were still running and listed no head-only failures. **Overlaps and merge order:** No overlapping pull request was identified from the supplied record or local history. The local main change for #1948 touches none of this PR's three files. </details> --- <sub>Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -637,6 +637,7 @@ SW(RX_VK_REFLEX, player, enum, "the menu's NVIDIA Reflex row", "Sets NVIDIA Refl
SW(RX_VK_REFLEX_LOG, diag, bool, "off", "Logs Reflex's latency timings every 300 frames.")
SW(RX_VK_REJECT_KEYS, diag, string, "empty: no program rejected", "Comma list of program key prefixes taken as untranslatable, for bisecting a bad translation.")
SW(RX_VK_REJECT_STAGES, diag, string, "empty: no stage rejected", "Comma list of NVN stage numbers whose programs are taken as untranslatable, for bisecting a bad translation.")
SW(RX_VK_REPLAYED_CACHE, kill, bool, "on (unset): the host's question how far the replay is answered from the timeline as last read, read again every 64th call", "0, off, false or no: the timeline read from the driver at every call, as before. Read once, at first use (the log says which).")
Member

L8 (minor): The row omits the empty value, which disables the cache through env_on_or, and should describe the cadence as the first eligible call and every 64 eligible calls thereafter. Calls during a benchmark hold always query the driver and do not advance that cadence.

**L8** (minor): The row omits the empty value, which disables the cache through env_on_or, and should describe the cadence as the first eligible call and every 64 eligible calls thereafter. Calls during a benchmark hold always query the driver and do not advance that cadence.
@ -279,3 +281,4 @@
std::atomic<uint64_t> completed_seen{0};
uint64_t completed() {
uint64_t v = 0;
vkGetSemaphoreCounterValue(dev.device, timeline, &v);
Member

L1: The new cache publishes the output without checking VkResult. If a driver writes to the output and returns an error, that value can become the permanent high-water mark and a later cache hit can let free_dead_copies recycle a recording without confirmed completion. Publish only after VK_SUCCESS, follow the renderer's device-loss handling for VK_ERROR_DEVICE_LOST, and add a fixture case where a failed query writes a larger value.

**L1**: The new cache publishes the output without checking VkResult. If a driver writes to the output and returns an error, that value can become the permanent high-water mark and a later cache hit can let free_dead_copies recycle a recording without confirmed completion. Publish only after VK_SUCCESS, follow the renderer's device-loss handling for VK_ERROR_DEVICE_LOST, and add a fixture case where a failed query writes a larger value.
@ -279,3 +282,4 @@
uint64_t completed() {
uint64_t v = 0;
vkGetSemaphoreCounterValue(dev.device, timeline, &v);
uint64_t was = completed_seen.load(std::memory_order_relaxed);
Member

L2: RX_VK_REPLAYED_CACHE=0 restores the driver's query frequency but leaves cache maintenance active in every completed() caller. A readback or present on the off build can still write the shared cache line as the GPU advances, so off retains the work the switch is meant to bypass. Initialize the switch before replay starts, gate publication on it, and assert that the off fixture leaves the cache untouched.

**L2**: RX_VK_REPLAYED_CACHE=0 restores the driver's query frequency but leaves cache maintenance active in every completed() caller. A readback or present on the off build can still write the shared cache line as the GPU advances, so off retains the work the switch is meant to bypass. Initialize the switch before replay starts, gate publication on it, and assert that the off fixture leaves the cache untouched.
@ -3044,0 +3050,4 @@
bool replayed_cache_on() {
static const bool v = [] {
const bool on = env_on_or("RX_VK_REPLAYED_CACHE", 1);
logf("vk: the host's question how far the replay is answered from the timeline as last read: %s (RX_VK_REPLAYED_CACHE=0: "
Member

L3 (minor): The first reclamation query can log while the host holds copy_lock exclusively. Linux writes to stderr synchronously, as does Windows with HOS_LOG_SYNC=1. A slow log sink can then stall other recording threads. Read and log the switch during renderer startup.

**L3** (minor): The first reclamation query can log while the host holds copy_lock exclusively. Linux writes to stderr synchronously, as does Windows with HOS_LOG_SYNC=1. A slow log sink can then stall other recording threads. Read and log the switch during renderer startup.
@ -3047,0 +3067,4 @@
// the bench holds it. The saving is not measured yet. RX_VK_REPLAYED_CACHE=0: the driver asked at every call, as before
if (!hold && replayed_cache_on()) {
static std::atomic<uint32_t> asked{0};
if (asked.fetch_add(1, std::memory_order_relaxed) & 63) return completed_seen.load(std::memory_order_relaxed);
Member

L10 (minor): The planned median and live-copy count cannot establish the gain or show its cost. That count excludes dead copies waiting for reuse. Compare main with head off and on in repeated, interleaved runs at matched settings. Report driver calls and time under copy_lock, average FPS, 1% lows, median and p99 frame times, and peak dead-copy bytes and process RAM. Include a heavy scene and the affected Tier 1 Windows, Linux, and Steam Deck configurations before settling the default.

**L10** (minor): The planned median and live-copy count cannot establish the gain or show its cost. That count excludes dead copies waiting for reuse. Compare main with head off and on in repeated, interleaved runs at matched settings. Report driver calls and time under copy_lock, average FPS, 1% lows, median and p99 frame times, and peak dead-copy bytes and process RAM. Include a heavy scene and the affected Tier 1 Windows, Linux, and Steam Deck configurations before settling the default.
@ -0,0 +24,4 @@
from srccut import block # noqa: E402
import srccut # noqa: E402
RUN_ALL_ARGS = ["--cxx", "{cxx}"]
Member

L4: The test's compiler fallback can be bypassed in both directions. On Windows with llvm-mingw configured but off PATH, run_all resolves {cxx} before starting the script and can skip or fail the check before default_cxx runs. On Linux, a direct run with LONGSHOT_MINGW set for Windows cross-builds can pick that compiler and try to execute a Windows binary. Use native-platform compiler selection for both entry points, honoring an explicit --cxx or CXX.

**L4**: The test's compiler fallback can be bypassed in both directions. On Windows with llvm-mingw configured but off PATH, run_all resolves {cxx} before starting the script and can skip or fail the check before default_cxx runs. On Linux, a direct run with LONGSHOT_MINGW set for Windows cross-builds can pick that compiler and try to execute a Windows binary. Use native-platform compiler selection for both entry points, honoring an explicit --cxx or CXX.
@ -0,0 +125,4 @@
int main(int, char** argv) {
const long long first = (long long)(@CALL@);
#ifdef _WIN32
_putenv_s(argv[1], argv[2]);
Member

L5 (minor): The read-once check ignores failure from _putenv_s and setenv. If the update fails, an implementation that rereads the environment can pass because it sees the original value twice. Check the update result and verify that getenv sees the new value before the second call.

**L5** (minor): The read-once check ignores failure from _putenv_s and setenv. If the update fails, an implementation that rereads the environment can pass because it sees the original value twice. Check the update result and verify that getenv sees the new value before the second call.
@ -0,0 +135,4 @@
}
"""
KILL_ON = ((None, 1), ("0", 0), ("1", 1), ("2", 1), ("off", 0)) # (unset: on; 0 or off: off; any other value: on)
ATTEMPT_OFF = ((None, 0), ("0", 0), ("1", 1), ("2", 1), ("off", 0)) # (unset: off)
Member

L6 (minor): ATTEMPT_OFF and the stubs parameter are unused, and the test compiles separate binaries to drive one switch through a stub and through the real helper. Use the real helper in a single fixture with fresh subprocesses for the on and off cases, and remove the unused scaffolding.

**L6** (minor): ATTEMPT_OFF and the stubs parameter are unused, and the test compiles separate binaries to drive one switch through a stub and through the real helper. Use the real helper in a single fixture with fresh subprocesses for the on and off cases, and remove the unused scaffolding.
siantm force-pushed perf/vk-replayed-cache from ab3a2544a1
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 8s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to d8a583a60c
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
beanstalk/approval waiting for a BigBean's 👍 on the request
Website / Website on Cloudflare (pull_request_target) Has been skipped
2026-11-21 04:43:40 +00:00
Compare
Author
Contributor

Thanks for the review (#issuecomment-62985). Pushed: the branch is rebased onto main f1f2dc5b (the PR's own commit unchanged, becb9c7b), with a test commit 4b3e94e7 and a fix d8a583a6. tests/test_replayed_cache.py fails at 4b3e94e7 and passes at d8a583a6 (this PC's job queue, jobs fb-1994-1, fb-1995-1 and fb-1995-2); the renderer's vk_replay.cpp and the host's nvn_native.c compile with the game build's own commands. Not run in the game yet.

  • L1 (62977) fixed in d8a583a6: completed() keeps only a read that returned VK_SUCCESS; a failed read keeps nothing and answers 0 (nothing taken as done), and VK_ERROR_DEVICE_LOST goes to device_lost(), as the GPU hang watch's read does. The fixture has a read that fails after writing 1000 (nothing kept, no answer above the last good read, at completed() and at get_replayed's own read) and a lost device.
  • L2 (62978) fixed in d8a583a6: the switch is read by replay_start into an atomic before the timeline is made; with it off completed() keeps nothing. The off cases assert completed_seen stays 0 after 128 calls.
  • L3 (62979) fixed in d8a583a6: read and logged once at the renderer's start, not under the host's copy lock.
  • L4 (62980) fixed in 4b3e94e7: --cxx, then CXX, then LONGSHOT_MINGW's clang++ on Windows only, then clang++, g++ or c++ on PATH; RUN_ALL_ARGS is gone, so run_all no longer resolves a compiler for it.
  • L5 (62981) fixed in 4b3e94e7: the flip's _putenv_s/setenv result is checked and getenv must return the new value.
  • L6 (62982) fixed in 4b3e94e7: one fixture with the real env_on_or, each case (unset, empty, 0, 1, 2, off, no, twice) in a fresh process; the unused scaffolding is gone.
  • L7 fixed in the description and the comment in get_replayed (d8a583a6): up to 63 calls when nothing else reads the timeline, longer in wall time when calls are sparse, and the copies kept meanwhile let go together.
  • L8 (62983) fixed in d8a583a6: the row names empty as off, the cadence (the first call and every 64th after, calls under the bench's hold not counted) and the delayed let-go.
  • L9: rebased onto f1f2dc5b; Measurements names that baseline.
  • L10 (62984): the edit part is in d8a583a6: the host's report every 50,000 recordings also logs the bytes of copies waiting for the replay, their peak, and the let-go passes with their time under the copies' lock (the fixture checks the counters). The in-game A/B with those figures, FPS, 1% lows, median/p99 frame times and RAM on Tier 1 hardware is pending, in Still to test.
  • L11: the game-free case is in 4b3e94e7 (the host's real free_dead_copies: a copy the GPU is past stays on a stale answer and goes at the next read). The own-PC run with the cache on and off through a scene change is pending, in Still to test.
Thanks for the review (#issuecomment-62985). Pushed: the branch is rebased onto main `f1f2dc5b` (the PR's own commit unchanged, `becb9c7b`), with a test commit `4b3e94e7` and a fix `d8a583a6`. `tests/test_replayed_cache.py` fails at `4b3e94e7` and passes at `d8a583a6` (this PC's job queue, jobs `fb-1994-1`, `fb-1995-1` and `fb-1995-2`); the renderer's `vk_replay.cpp` and the host's `nvn_native.c` compile with the game build's own commands. Not run in the game yet. - **L1** ([62977](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1994#issuecomment-62977)) fixed in `d8a583a6`: `completed()` keeps only a read that returned `VK_SUCCESS`; a failed read keeps nothing and answers 0 (nothing taken as done), and `VK_ERROR_DEVICE_LOST` goes to `device_lost()`, as the GPU hang watch's read does. The fixture has a read that fails after writing 1000 (nothing kept, no answer above the last good read, at `completed()` and at `get_replayed`'s own read) and a lost device. - **L2** ([62978](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1994#issuecomment-62978)) fixed in `d8a583a6`: the switch is read by `replay_start` into an atomic before the timeline is made; with it off `completed()` keeps nothing. The off cases assert `completed_seen` stays 0 after 128 calls. - **L3** ([62979](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1994#issuecomment-62979)) fixed in `d8a583a6`: read and logged once at the renderer's start, not under the host's copy lock. - **L4** ([62980](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1994#issuecomment-62980)) fixed in `4b3e94e7`: `--cxx`, then `CXX`, then `LONGSHOT_MINGW`'s clang++ on Windows only, then clang++, g++ or c++ on PATH; `RUN_ALL_ARGS` is gone, so run_all no longer resolves a compiler for it. - **L5** ([62981](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1994#issuecomment-62981)) fixed in `4b3e94e7`: the flip's `_putenv_s`/`setenv` result is checked and `getenv` must return the new value. - **L6** ([62982](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1994#issuecomment-62982)) fixed in `4b3e94e7`: one fixture with the real `env_on_or`, each case (unset, empty, 0, 1, 2, off, no, twice) in a fresh process; the unused scaffolding is gone. - **L7** fixed in the description and the comment in `get_replayed` (`d8a583a6`): up to 63 calls when nothing else reads the timeline, longer in wall time when calls are sparse, and the copies kept meanwhile let go together. - **L8** ([62983](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1994#issuecomment-62983)) fixed in `d8a583a6`: the row names empty as off, the cadence (the first call and every 64th after, calls under the bench's hold not counted) and the delayed let-go. - **L9**: rebased onto `f1f2dc5b`; Measurements names that baseline. - **L10** ([62984](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1994#issuecomment-62984)): the edit part is in `d8a583a6`: the host's report every 50,000 recordings also logs the bytes of copies waiting for the replay, their peak, and the let-go passes with their time under the copies' lock (the fixture checks the counters). The in-game A/B with those figures, FPS, 1% lows, median/p99 frame times and RAM on Tier 1 hardware is pending, in Still to test. - **L11**: the game-free case is in `4b3e94e7` (the host's real `free_dead_copies`: a copy the GPU is past stays on a stale answer and goes at the next read). The own-PC run with the cache on and off through a scene change is pending, in Still to test.
Author
Contributor

@Beanstalk This adds RX_VK_REPLAYED_CACHE, a kill switch, on by default: the host's question how far the replay is (asked under the recordings' lock at every control memory add) is answered from the timeline value last read, reading the driver at every 64th call (gpu/vk, host/nvn_native.c); RX_VK_REPLAYED_CACHE=0 reads the driver at every call, as main does. It has not run in the game on these PCs yet. Please play Hyrule Field for three minutes, running and turning the camera, after a minute's warm-up, the PR against main on the same PC, on the PR's side with the switch unset and with RX_VK_REPLAYED_CACHE=0, on Linux with NVIDIA, Linux with AMD (RADV), Windows with Intel and the Steam Deck. The same picture, no crash, no device lost and no FATAL line; unset, frame time p99 equal to main's or better and to the switch off; with it off, as main. Please report the peak copies waiting in the log's nvn: report line on each side.

@Beanstalk This adds RX_VK_REPLAYED_CACHE, a kill switch, on by default: the host's question how far the replay is (asked under the recordings' lock at every control memory add) is answered from the timeline value last read, reading the driver at every 64th call (gpu/vk, host/nvn_native.c); RX_VK_REPLAYED_CACHE=0 reads the driver at every call, as main does. It has not run in the game on these PCs yet. Please play Hyrule Field for three minutes, running and turning the camera, after a minute's warm-up, the PR against main on the same PC, on the PR's side with the switch unset and with RX_VK_REPLAYED_CACHE=0, on Linux with NVIDIA, Linux with AMD (RADV), Windows with Intel and the Steam Deck. The same picture, no crash, no device lost and no FATAL line; unset, frame time p99 equal to main's or better and to the switch off; with it off, as main. Please report the peak copies waiting in the log's nvn: report line on each side.
Member

@siantm only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@siantm only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
beanstalk/approval waiting for a BigBean's 👍 on the request
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/vk-replayed-cache:siantm-perf/vk-replayed-cache'
git switch 'siantm-perf/vk-replayed-cache'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1994
No description provided.