WIP: perf(vk): fewer driver calls at a recording's end and start: slots freed by the recordings' timeline, timestamps read once a present (RX_VK_SUBMIT_BATCH=0 to disable) #1995

Draft
siantm wants to merge 7 commits from siantm/Longshot:perf/vk-submit-batch into main
Contributor

Summary

Each recording's submit carried a fence beside the recordings' timeline value it already signals, and the slot's
reuse waited for that fence and reset it; and the GPU timestamps were read slot by slot as each came round.

With this:

  • a slot is free again by its recording's number on the recordings' timeline (slot_free_by_seq), read again only for
    a number past the value last read; the submits carry no fence. A lost device at that read or wait goes to
    device_lost(), and a wait that fails otherwise stops the renderer (replay_fatal): a slot is never reused without
    its recording's completion;
  • the timestamps of every recording that is done are read once a present (gpu_time_collect_done, one
    vkGetQueryPoolResults a run of neighbouring slots), counted as before; slot by slot as before while the frame
    overlap's tracking or RX_VK_GPU_TIMELINE is on; a lost device at either read goes to device_lost().

The result expected: fewer driver calls a recording.

The submit thread this PR first had (mkw_vk submit, held submits sent by a thread of their own where nothing else
batches them) was taken out to be reviewed on its own once fixed and measured in play, off by default; so was a later
part's progress taken as last read (split_part), which could change what a draw reads.

The switches: RX_VK_SUBMIT_BATCH (new, a kill switch, bool, its row in gpu/vk/switches.def), on by default as the
branch shipped it, read once; empty, 0, off, false or no give back main's fences and slot-by-slot timestamps.
RX_VK_REPLAYED_CACHE comes with #1994 and is a switch of its own: main's behaviour is both at 0.

Depends on #1994 (the timeline as last read): its commits show in this diff until it merges.

From bd3ea307 on exp/x86-asm-sync (its replay-parts work).

Read first: rec_done, slot_free_by_seq, recording_wait, submit_recording, part_submit and
gpu_time_collect_done in gpu/vk/vk_replay.cpp, then tests/test_submit_batch.py and
tests/test_fsr_uniform_ownership.py.

Testing

Tested: Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head 56d4ac6b on d8a583a6 (#1994's head,
on main f1f2dc5b). Build jobs of this PC's job queue (fb-1995-1, fb-1995-2), the checks in a hidden console:
tests/test_submit_batch.py fails at the test commit 5e004d35 (a failed or lost wait frees the slot, lost devices
not reported, timestamps collected with RX_VK_GPU_TIMELINE on) and passes at the head; tests/run_all.py --strict --changed origin/main at the head (the CPM cache's Vulkan headers): 170 checks, 135 passed, 30 failed, 5 skipped (Linux only or a device); 29 of them fail on main @ f1f2dc5b too, run from a main worktree in the same job: tests/test_capture_readback_visibility.py, tests/test_crash_dialog.py, tests/test_crash_record.py, tests/test_dev_ab.py, tests/test_dev_packs.py, tests/test_device_lost_exit.py, tests/test_draw_stream_visibility.py, tests/test_flush_nothing_held.py, tests/test_frame_overlap.py, tests/test_fsr_image_retirement.py, tests/test_fsr_uniform_ownership.py, tests/test_gpu_wait_stats.py, tests/test_host_guest_writes.py, tests/test_linux_pressure.py, tests/test_mesh_conv.py, tests/test_perf_report.py, tests/test_present_queue_default.py, tests/test_present_source_crop.py, tests/test_profiling_mode.py, tests/test_recorder_own_idle.py, tests/test_renderer_new_handler.py, tests/test_replay_write_order.py, tests/test_slow_frames.py, tests/test_sparse_mirror.py, tests/test_target_reuse_memory.py, tests/test_ub_update_batch.py, tests/test_upscaler_rows.py, tests/test_vk_occlusion_queries.py, tests/test_walk_key.py; the 30th, tests/test_present_semaphores.py, failed once with no output in that parallel run, passes on main, and passed three times alone at the head (job fb-1995-3); gpu/vk/vk_replay.cpp compiled (syntax) with the game build's
own command. This PR's additions to tests/test_fsr_uniform_ownership.py (its fixture's distinct timeline handle, exact
wait value and failed/lost slot cases), tests/test_fsr_image_retirement.py and tests/test_draw_stream_visibility.py
(their runs without RX_VK_BATCH_MAX) did NOT run: those three fixtures do not build on main @ f1f2dc5b (pre-existing:
their stand-ins and cut markers are behind main), and fail the same way at the head.

It has NOT run in the game on this PC yet.

Still to test:

  • Windows 11, i7-9700, RTX 4060 (this PC): Kokiri Forest and Hyrule Field, play three minutes each with Threaded
    rendering off, on (the recorder thread) and Auto: #1994's head d8a583a6 (this PR's base), this branch with
    RX_VK_SUBMIT_BATCH=0 and this branch with it unset (on), all with RX_VK_REPLAYED_CACHE unset, and main @ f1f2dc5b
    against this branch with both switches =0; each warmed up a minute in the scene first, interleaved (ABBA), at least
    two runs each, expecting the same picture, no FATAL or device lost, the GPU times in the FPS counter's breakdown alike;
    report average FPS, 1% lows, median and p99 frame times and peak RAM; a gain is median and p99 frame time unset lower
    than with RX_VK_SUBMIT_BATCH=0 and than the base, and both =0 the same as main (by hand)
  • Linux NVIDIA and AMD, Windows AMD and Intel, Steam Deck: Kokiri Forest and Hyrule Field, play three minutes each: the
    same sides, warmed up, ABBA, expecting no FATAL or device lost and the same picture, and the same figures as above (by
    hand or on the farm, not run yet)
  • Linux: build the renderer and run python3 -I -B tests/test_submit_batch.py, expecting PASS (by hand)
  • Windows and Linux: once the three fixtures are repaired (a separate fix), run tests/test_fsr_uniform_ownership.py,
    tests/test_fsr_image_retirement.py and tests/test_draw_stream_visibility.py on this branch, the last two also with
    RX_VK_BATCH_MAX=1 set, expecting PASS (by hand)

Measurements

Verdict: not measured: no micro-benchmark of this port and no run in the game on this PC yet; the in-game A/B is
in Still to test. The branch's CPU profiles (its whole replay-parts work) put the slot-by-slot timestamp reads at 1.0 %
of a core in Hyrule Field and submit_recording at 3.1 % in Kokiri Forest; the second was the submit thread's case and
left with it; neither is a measurement of this change. RX_VK_SUBMIT_BATCH ships on, as the branch shipped it; no
measurement chose that default yet, the in-game A/B in Still to test does. RX_VK_REPLAYED_CACHE comes with #1994,
which says its default.

Risks

What could break: a slot reused before the GPU is done with its recording (a wrong draw or a device lost), timestamps
counted twice or lost (only the FPS counter's GPU times). A wait for a slot that fails now ends the game with a FATAL line
instead of reusing the slot. The tests run the real rec_done and slot_free_by_seq
(a done slot, a held one sent first, a failed or lost wait) and the real timestamp collection against a stand-in driver,
and read the submit paths for the slot's number. Turn it off with RX_VK_SUBMIT_BATCH=0. Renderer code: the runtime is
rebuilt; nothing changes for players' saves, options or caches.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A switch added or changed: it ships at the default the measurements chose, with its row in gpu/vk/switches.def.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary Each recording's submit carried a fence beside the recordings' timeline value it already signals, and the slot's reuse waited for that fence and reset it; and the GPU timestamps were read slot by slot as each came round. With this: - a slot is free again by its recording's number on the recordings' timeline (`slot_free_by_seq`), read again only for a number past the value last read; the submits carry no fence. A lost device at that read or wait goes to `device_lost()`, and a wait that fails otherwise stops the renderer (`replay_fatal`): a slot is never reused without its recording's completion; - the timestamps of every recording that is done are read once a present (`gpu_time_collect_done`, one `vkGetQueryPoolResults` a run of neighbouring slots), counted as before; slot by slot as before while the frame overlap's tracking or `RX_VK_GPU_TIMELINE` is on; a lost device at either read goes to `device_lost()`. The result expected: fewer driver calls a recording. The submit thread this PR first had (`mkw_vk submit`, held submits sent by a thread of their own where nothing else batches them) was taken out to be reviewed on its own once fixed and measured in play, off by default; so was a later part's progress taken as last read (`split_part`), which could change what a draw reads. The switches: `RX_VK_SUBMIT_BATCH` (new, a kill switch, bool, its row in `gpu/vk/switches.def`), on by default as the branch shipped it, read once; empty, `0`, `off`, `false` or `no` give back main's fences and slot-by-slot timestamps. `RX_VK_REPLAYED_CACHE` comes with #1994 and is a switch of its own: main's behaviour is both at `0`. Depends on #1994 (the timeline as last read): its commits show in this diff until it merges. From bd3ea307 on `exp/x86-asm-sync` (its replay-parts work). Read first: `rec_done`, `slot_free_by_seq`, `recording_wait`, `submit_recording`, `part_submit` and `gpu_time_collect_done` in `gpu/vk/vk_replay.cpp`, then `tests/test_submit_batch.py` and `tests/test_fsr_uniform_ownership.py`. ## Testing **Tested:** Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head `56d4ac6b` on `d8a583a6` (#1994's head, on main `f1f2dc5b`). Build jobs of this PC's job queue (`fb-1995-1`, `fb-1995-2`), the checks in a hidden console: `tests/test_submit_batch.py` fails at the test commit `5e004d35` (a failed or lost wait frees the slot, lost devices not reported, timestamps collected with `RX_VK_GPU_TIMELINE` on) and passes at the head; `tests/run_all.py --strict --changed origin/main` at the head (the CPM cache's Vulkan headers): 170 checks, 135 passed, 30 failed, 5 skipped (Linux only or a device); 29 of them fail on main @ f1f2dc5b too, run from a main worktree in the same job: `tests/test_capture_readback_visibility.py`, `tests/test_crash_dialog.py`, `tests/test_crash_record.py`, `tests/test_dev_ab.py`, `tests/test_dev_packs.py`, `tests/test_device_lost_exit.py`, `tests/test_draw_stream_visibility.py`, `tests/test_flush_nothing_held.py`, `tests/test_frame_overlap.py`, `tests/test_fsr_image_retirement.py`, `tests/test_fsr_uniform_ownership.py`, `tests/test_gpu_wait_stats.py`, `tests/test_host_guest_writes.py`, `tests/test_linux_pressure.py`, `tests/test_mesh_conv.py`, `tests/test_perf_report.py`, `tests/test_present_queue_default.py`, `tests/test_present_source_crop.py`, `tests/test_profiling_mode.py`, `tests/test_recorder_own_idle.py`, `tests/test_renderer_new_handler.py`, `tests/test_replay_write_order.py`, `tests/test_slow_frames.py`, `tests/test_sparse_mirror.py`, `tests/test_target_reuse_memory.py`, `tests/test_ub_update_batch.py`, `tests/test_upscaler_rows.py`, `tests/test_vk_occlusion_queries.py`, `tests/test_walk_key.py`; the 30th, `tests/test_present_semaphores.py`, failed once with no output in that parallel run, passes on main, and passed three times alone at the head (job `fb-1995-3`); `gpu/vk/vk_replay.cpp` compiled (syntax) with the game build's own command. This PR's additions to `tests/test_fsr_uniform_ownership.py` (its fixture's distinct timeline handle, exact wait value and failed/lost slot cases), `tests/test_fsr_image_retirement.py` and `tests/test_draw_stream_visibility.py` (their runs without `RX_VK_BATCH_MAX`) did NOT run: those three fixtures do not build on main @ f1f2dc5b (pre-existing: their stand-ins and cut markers are behind main), and fail the same way at the head. **It has NOT run in the game on this PC yet.** **Still to test:** - Windows 11, i7-9700, RTX 4060 (this PC): Kokiri Forest and Hyrule Field, play three minutes each with Threaded rendering off, on (the recorder thread) and Auto: #1994's head `d8a583a6` (this PR's base), this branch with `RX_VK_SUBMIT_BATCH=0` and this branch with it unset (on), all with `RX_VK_REPLAYED_CACHE` unset, and main @ f1f2dc5b against this branch with both switches `=0`; each warmed up a minute in the scene first, interleaved (ABBA), at least two runs each, expecting the same picture, no FATAL or device lost, the GPU times in the FPS counter's breakdown alike; report average FPS, 1% lows, median and p99 frame times and peak RAM; a gain is median and p99 frame time unset lower than with `RX_VK_SUBMIT_BATCH=0` and than the base, and both `=0` the same as main (by hand) - Linux NVIDIA and AMD, Windows AMD and Intel, Steam Deck: Kokiri Forest and Hyrule Field, play three minutes each: the same sides, warmed up, ABBA, expecting no FATAL or device lost and the same picture, and the same figures as above (by hand or on the farm, not run yet) - Linux: build the renderer and run `python3 -I -B tests/test_submit_batch.py`, expecting PASS (by hand) - Windows and Linux: once the three fixtures are repaired (a separate fix), run `tests/test_fsr_uniform_ownership.py`, `tests/test_fsr_image_retirement.py` and `tests/test_draw_stream_visibility.py` on this branch, the last two also with `RX_VK_BATCH_MAX=1` set, expecting PASS (by hand) ## Measurements **Verdict:** not measured: no micro-benchmark of this port and no run in the game on this PC yet; the in-game A/B is in Still to test. The branch's CPU profiles (its whole replay-parts work) put the slot-by-slot timestamp reads at 1.0 % of a core in Hyrule Field and `submit_recording` at 3.1 % in Kokiri Forest; the second was the submit thread's case and left with it; neither is a measurement of this change. `RX_VK_SUBMIT_BATCH` ships on, as the branch shipped it; no measurement chose that default yet, the in-game A/B in Still to test does. `RX_VK_REPLAYED_CACHE` comes with #1994, which says its default. ## Risks What could break: a slot reused before the GPU is done with its recording (a wrong draw or a device lost), timestamps counted twice or lost (only the FPS counter's GPU times). A wait for a slot that fails now ends the game with a FATAL line instead of reusing the slot. The tests run the real `rec_done` and `slot_free_by_seq` (a done slot, a held one sent first, a failed or lost wait) and the real timestamp collection against a stand-in driver, and read the submit paths for the slot's number. Turn it off with `RX_VK_SUBMIT_BATCH=0`. Renderer code: the runtime is rebuilt; nothing changes for players' saves, options or caches. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. - [ ] `mise run test` passes (any failure that also fails on `main` is named). - [ ] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [ ] A switch added or changed: it ships at the default the measurements chose, with its row in `gpu/vk/switches.def`. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [ ] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
perf(vk): the host's question how far the replay is answered from the timeline as last read (RX_VK_REPLAYED_CACHE)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 8s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
ab3a2544a1
The host asks get_replayed under its recordings' lock, held exclusively (host/nvn_native.c free_dead_copies, at every
nvnCommandBufferAddControlMemory: 117 a frame in a CPU profile of Hyrule Field on the branch), to let go of the copies the
GPU is past, and every answer was a vkGetSemaphoreCounterValue there. completed() now keeps the highest value it has read
(completed_seen, it only goes up), and get_replayed answers from it, reading the driver at every 64th call so that it moves
when nothing replays. A lower answer only keeps a copy until a later call. Not while the bench holds the answer
(replayed_hold): the driver is read as before.

From the branch's replay-parts work (bd3ea307). Not measured: neither on the branch nor in play.

Test: tests/test_replayed_cache.py cuts completed(), replayed_cache_on() and get_replayed() from vk_replay.cpp and runs
them against a stand-in driver (63 of 64 answers without a driver call, the bench's hold and =0 read every time).

Switch: RX_VK_REPLAYED_CACHE (kill), on; =0 reads the driver at every call, as before.

Switch reads: the test also runs RX_VK_REPLAYED_CACHE as the source reads it, in fresh processes (unset, 0, 1, 2, off, each twice, the variable flipped after the first read: read once).

Assisted-by: Claude Code
perf(vk): fewer driver calls at a recording's end and start, and a submit thread where nothing else submits (RX_VK_SUBMIT_BATCH)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 8s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
402a2ad765
Needs "the host's question how far the replay is answered from the timeline as last read" (RX_VK_REPLAYED_CACHE) first:
it uses that commit's completed_seen.

- A slot is free again by its recording's number on the recordings' timeline (slot_free_by_seq): every recording's
  submit already signals that number, so the fence beside it said the same thing twice: a fence in each submit, and a
  wait for it and a reset at each reuse, 48 recordings later. The timeline's value is kept as last read and read again
  only for a number past it, about once a frame. Both submit_recording's own submit and the recorder thread's
  (part_submit, RX_VK_RECORDER_PARTS) carry no fence then; a held submit is sent first, as recording_wait does for one.
- The GPU timestamps of every recording that is done are read once a present (gpu_time_collect_done, one
  vkGetQueryPoolResults a run of neighbouring slots), not slot by slot as each comes round: 1.0 % of a core in a CPU
  profile of Hyrule Field on the branch. Each is counted as gpu_time_collect counts it (its time, its
  RX_VK_GPU_TIMELINE line, the A/B's side); with the frame overlap's tracking on, every slot is read as it comes round,
  as before, since fo_collect takes them in recording order.
- A later part of a submission (split_part) takes the GPU's progress as last read for the memory work's bookkeeping,
  instead of a driver call: a lower value only keeps what could go a part longer.
- Where submits are neither batched (RX_VK_BATCH unset, not RADV) nor the recorder thread's, a recording's submit is held
  as a batched one is and a submit thread sends it at once (mkw_vk submit), while the replay goes on with the next part:
  submit_recording was 3.1 % of a core on the replaying threads on the branch (i7-9700, Kokiri Forest). With the recorder
  thread main already does this in its own form (RX_VK_RECORDER_PARTS: the recorder ends and submits each part), so the
  submit thread is not started there; it serves the render thread alone (Auto with 4 or 5 cores) and the submitting
  thread's replay. Not with the bench.

From the branch's replay-parts work (bd3ea307). Not measured as a whole, neither on the branch nor in play.

Tests: tests/test_submit_batch.py compiles the real gpu_time_note and gpu_time_collect_done with a stand-in driver (a call
a run of done slots, a slot not done or not there left for its own read, the counts, tracking and =0 reading nothing)
and reads submit_recording, part_submit, recording_wait, the submit thread and split_part for their order;
tests/test_fsr_uniform_ownership.py runs the real slot_free_by_seq through recording_wait and begin_recording (a slot by
its number, a held one sent first, a number reached already needing no driver call). The fixtures that cut
submit_recording, begin_recording or split_part (draw_stream_visibility, fsr_image_retirement, early_split_before_draw)
have the new stand-ins, and test_device_policy finds the batch policy in batch_wanted.

Switch: RX_VK_SUBMIT_BATCH (kill), on; =0 a fence a slot, timestamps read slot by slot, the timeline read at every
recording and the submit made by the replaying thread, as before.

Switch reads: the test also runs RX_VK_SUBMIT_BATCH as the source reads it, in fresh processes (unset, 0, 1, 2, off, each twice, the variable flipped after the first read: read once). If the submit thread cannot be started, the replaying thread submits, as with =0.

Assisted-by: Claude Code
BeanBot left a comment

Recommendation: Request changes. A failed worker start can put recordings on the Vulkan queue out of order, and a failed timeline wait can let replay reset command pools the GPU still uses. The default-on path also adds a profiling data race, and its effect on player frame times is unproven.

Reuses recording slots through the existing Vulkan timeline, harvests completed GPU timestamps in groups, caches some progress reads, and hands eligible submits to a worker. Reusing the timeline can remove redundant fence calls. The worker adds a scheduling hop, queue-lock contention, and new failure and shutdown paths.

@fix will do (17)

  1. L1 gpu/vk/vk_replay.cpp:1235: Worker creation can fail after the first recording is already in pend. (more on the line)
  2. L2 gpu/vk/vk_replay.cpp:291: slot_free_by_seq clears ownership and reports success on every wait result. (more on the line)
  3. L3 gpu/vk/vk_replay.cpp:431: The worker makes the inherited batch-drop path the default on additional GPUs. On VK_ERROR_OUT_OF_HOST_MEMORY, queue_flush_locked marks the batch flushed and discards it even though no signals were submitted. A slot or readback waiting for that batch before another submit can hang indefinitely. Keep ownership until a retry succeeds or fail the renderer explicitly. Cover a failed final batch with a waiter.
  4. L4 gpu/vk/vk_replay.cpp:422: The worker now runs PROF_QSUBMIT against the plain prof_ns array. log_frame_stats reads it every 120 presents under replay_mu alone, and the menu reads the same counter. The worker can update it during either read. Keep worker timing in a separate atomic counter and report it separately from replay-thread time.
  5. L5 gpu/vk/vk_replay.cpp:495: The detached worker can wait forever on a condition variable with a static destructor. (more on the line)
  6. L6 gpu/vk/vk_replay.cpp:660: Bulk collection can read gtl_submit_ns without the recorder ticket synchronization that slot reuse provides. (more on the line)
  7. L7 gpu/vk/vk_replay.cpp:334: completed() publishes the output of vkGetSemaphoreCounterValue even when the call fails. (more on the line)
  8. L8 gpu/vk/vk_replay.cpp:1115: Stale progress can change what a draw reads. (more on the line)
  9. L9 gpu/vk/vk_replay.cpp:390: The worker is enabled on RADV GCN when RX_VK_BATCH is unset and the recorder is off. (more on the line)
  10. L10 (its part an edit can do) gpu/vk/vk_replay.cpp:1234: Use a nonblocking lock attempt for polling paths and leave a busy queue to the worker. Track time spent acquiring the lock against finite wait budgets.
  11. L11 (minor) gpu/vk/vk_replay.cpp:1222: A worker batch can span Reflex frame IDs. (more on the line)
  12. L12 (minor) gpu/vk/vk_replay.cpp:698: RX_VK_GPU_TIMELINE records the enqueue time as the submit time on the worker path. (more on the line)
  13. L13 (minor) gpu/vk/vk_replay.cpp:641: Bulk timestamp collection silently ignores device loss from both the semaphore counter read and the query read. (more on the line)
  14. L14 (minor) tests/test_submit_batch.py:274: The submit-worker checks only look for source strings. (more on the line)
  15. L15 (minor) tests/fsr_image_retirement_test.cpp:146: The retained-batch cases here and in draw_stream_visibility_test.cpp inherit RX_VK_BATCH_MAX. A developer's RX_VK_BATCH_MAX=1 sends the first recording immediately and fails the fixture's assertion that the batch stayed held. Sanitize that variable in both Python runners or set it to a known fixture value.
  16. L16 (minor): RX_VK_SUBMIT_BATCH=0 alone does not restore the supplied main baseline because RX_VK_REPLAYED_CACHE remains on. Describe what each switch restores. Compare against the actual base and use the same replay-cache setting on both sides when isolating #1995.
  17. L17 (its part an edit can do): Record enqueue-to-driver-submit time and counts by flush reason so the runs can distinguish driver savings from moved work and added queue delay.

For the author (2) (a run, a measurement or a decision)

  • L10 gpu/vk/vk_replay.cpp:1234: Holding submits by default exposes previously direct setups to blocking progress polls. (more on the line)
  • L17 (minor): The reported check run supplies no gameplay evidence, and the proposed median-only verdict can miss worse frame pacing. Run exact-head and base gameplay before marking ready, including Off, Auto render-only, recorder-thread and Tier 1 paths. Report average FPS, 1% lows, median and p99 frame time, submit or query call counts, worker CPU time, and peak memory for the deferred-copy path. Include the actual driver, settings, build type and interleaved run results.

Checks: Still running: PR description / Description follows the template (pull_request_target), CI / Conventional Commit title (pull_request), CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request), CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request), CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request), CI / Workflow lint (actionlint) (pull_request), CI / Website build and checks (pull_request), Guide screenshots / Guide images and local UI source review (pull_request), Third-party parts / As their notes say (pull_request), Third-party parts / The same as upstream (pull_request).

More: does it help, other systems, switches, what was read

Does it help: Unproven. The commit cites branch-only CPU samples of 3.1% of one core in submit_recording in Kokiri Forest and 1.0% in timestamp reads in Hyrule Field. Those show where time was spent. There are zero head-versus-base gameplay runs and no measured savings, average FPS, 1% lows, median or p99 frame time, or submit-to-GPU delay.

Other systems: The worker can affect NVIDIA, AMD, Intel and software Vulkan paths with the recorder thread off, including four- and five-core Auto setups. RADV RDNA and Steam Deck keep legacy batching, while recording-slot waits and split-part progress still change there. Frame-overlap tracking keeps slot-by-slot timestamp collection whenever async replay is active. Sparse binds, upload chunks and the present copy keep their queue locks and flushes. FSR and readbacks depend on successful completion before reuse. Reflex, GPU timing diagnostics and host recording-copy reclamation have separate effects called out below. The diff changes no save format, audio or input code, public renderer interface, or shader translation inputs.

Switches: RX_VK_SUBMIT_BATCH is new, read once, and defaults on. Its default has no gameplay evidence and its worker currently bypasses the RADV GCN exclusion. RX_VK_REPLAYED_CACHE comes from #1994, is read once, defaults on, and also lacks gameplay measurements. Both have switch-table rows. RX_VK_BATCH keeps its device-policy default, and an explicit on or off value suppresses the new worker. RX_VK_BATCH_MAX still defaults to 32 command buffers and RX_VK_BATCH_POLL_US to 2000 microseconds. Both now reach worker-held submits on additional drivers. RX_VK_BENCH keeps its existing syntax and suppresses the worker when set.

Reviewed: 402a2ad765 against 75317e1f31: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, developer standards, both commits, the complete diff, and all nine changed test files. Read all changed renderer functions and traced their recorder, queue-lock, progress, host-copy, uniform, readback, Reflex, presentation and exit callers. Read the switch rows and their policy context. Skimmed unrelated portions of vk_replay.cpp, switches.def and supporting renderer files. No review-checklist file, sibling Tooling checkout, raw run logs, translations or generated files were present in the review material.; nothing run (a code review).

Checked and fine:

  • Both commits carry Assisted-by trailers. The diff contains source and authored fixtures, with no game-derived assets, build artifacts, generated shaders or third-party changes. git diff --check was clean.
  • On successful, ordered submits, the recording timeline signals at ALL_COMMANDS. Slot reuse waits for the recorder ticket before reading the slot's submission state.
  • The bulk query indices, stride and buffer size fit the 48-slot ring. The fixture exercises ring wrap, unavailable results, consumption once and the frame-overlap bypass.
  • The new switch fixtures use the shared parser in fresh child processes and isolate the switches they test. They respect an explicit C++ compiler and CXX. The added tests compile local fixtures with standard-library tooling and require no network or external downloads.
  • The stated dependency on #1994 matches the use of completed_seen. The renderer API version and shader generation rules do not require changes for this diff.

Testing: Read-only review. The author reports Windows 11, i7-9700, RTX 4060 and llvm-mingw clang 23, with 132 of 159 checks passing and 27 failures also reproduced on the base. The new submit-batch fixture reportedly passed. Gameplay, Linux build evidence and Tier 1 hardware runs are still outstanding. The supplied CI snapshot has checks in progress and lists no head-only failures.

Overlaps and merge order: #1994 is included in the supplied diff and must merge first if kept as a separate pull request. Resolve the cache finding there, then rebase #1995 and measure against the resulting base. No other overlap was established from the checkout.


Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** A failed worker start can put recordings on the Vulkan queue out of order, and a failed timeline wait can let replay reset command pools the GPU still uses. The default-on path also adds a profiling data race, and its effect on player frame times is unproven. Reuses recording slots through the existing Vulkan timeline, harvests completed GPU timestamps in groups, caches some progress reads, and hands eligible submits to a worker. Reusing the timeline can remove redundant fence calls. The worker adds a scheduling hop, queue-lock contention, and new failure and shutdown paths. **`@fix` will do (17)** 1. **L1** `gpu/vk/vk_replay.cpp:1235`: Worker creation can fail after the first recording is already in pend. (more on the line) 2. **L2** `gpu/vk/vk_replay.cpp:291`: slot_free_by_seq clears ownership and reports success on every wait result. (more on the line) 3. **L3** `gpu/vk/vk_replay.cpp:431`: The worker makes the inherited batch-drop path the default on additional GPUs. On VK_ERROR_OUT_OF_HOST_MEMORY, queue_flush_locked marks the batch flushed and discards it even though no signals were submitted. A slot or readback waiting for that batch before another submit can hang indefinitely. Keep ownership until a retry succeeds or fail the renderer explicitly. Cover a failed final batch with a waiter. 4. **L4** `gpu/vk/vk_replay.cpp:422`: The worker now runs PROF_QSUBMIT against the plain prof_ns array. log_frame_stats reads it every 120 presents under replay_mu alone, and the menu reads the same counter. The worker can update it during either read. Keep worker timing in a separate atomic counter and report it separately from replay-thread time. 5. **L5** `gpu/vk/vk_replay.cpp:495`: The detached worker can wait forever on a condition variable with a static destructor. (more on the line) 6. **L6** `gpu/vk/vk_replay.cpp:660`: Bulk collection can read gtl_submit_ns without the recorder ticket synchronization that slot reuse provides. (more on the line) 7. **L7** `gpu/vk/vk_replay.cpp:334`: completed() publishes the output of vkGetSemaphoreCounterValue even when the call fails. (more on the line) 8. **L8** `gpu/vk/vk_replay.cpp:1115`: Stale progress can change what a draw reads. (more on the line) 9. **L9** `gpu/vk/vk_replay.cpp:390`: The worker is enabled on RADV GCN when RX_VK_BATCH is unset and the recorder is off. (more on the line) 10. **L10** (its part an edit can do) `gpu/vk/vk_replay.cpp:1234`: Use a nonblocking lock attempt for polling paths and leave a busy queue to the worker. Track time spent acquiring the lock against finite wait budgets. 11. **L11** (minor) `gpu/vk/vk_replay.cpp:1222`: A worker batch can span Reflex frame IDs. (more on the line) 12. **L12** (minor) `gpu/vk/vk_replay.cpp:698`: RX_VK_GPU_TIMELINE records the enqueue time as the submit time on the worker path. (more on the line) 13. **L13** (minor) `gpu/vk/vk_replay.cpp:641`: Bulk timestamp collection silently ignores device loss from both the semaphore counter read and the query read. (more on the line) 14. **L14** (minor) `tests/test_submit_batch.py:274`: The submit-worker checks only look for source strings. (more on the line) 15. **L15** (minor) `tests/fsr_image_retirement_test.cpp:146`: The retained-batch cases here and in draw_stream_visibility_test.cpp inherit RX_VK_BATCH_MAX. A developer's RX_VK_BATCH_MAX=1 sends the first recording immediately and fails the fixture's assertion that the batch stayed held. Sanitize that variable in both Python runners or set it to a known fixture value. 16. **L16** (minor): RX_VK_SUBMIT_BATCH=0 alone does not restore the supplied main baseline because RX_VK_REPLAYED_CACHE remains on. Describe what each switch restores. Compare against the actual base and use the same replay-cache setting on both sides when isolating #1995. 17. **L17** (its part an edit can do): Record enqueue-to-driver-submit time and counts by flush reason so the runs can distinguish driver savings from moved work and added queue delay. **For the author (2)** (a run, a measurement or a decision) - **L10** `gpu/vk/vk_replay.cpp:1234`: Holding submits by default exposes previously direct setups to blocking progress polls. (more on the line) - **L17** (minor): The reported check run supplies no gameplay evidence, and the proposed median-only verdict can miss worse frame pacing. Run exact-head and base gameplay before marking ready, including Off, Auto render-only, recorder-thread and Tier 1 paths. Report average FPS, 1% lows, median and p99 frame time, submit or query call counts, worker CPU time, and peak memory for the deferred-copy path. Include the actual driver, settings, build type and interleaved run results. **Checks:** Still running: `PR description / Description follows the template (pull_request_target)`, `CI / Conventional Commit title (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request)`, `CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request)`, `CI / Workflow lint (actionlint) (pull_request)`, `CI / Website build and checks (pull_request)`, `Guide screenshots / Guide images and local UI source review (pull_request)`, `Third-party parts / As their notes say (pull_request)`, `Third-party parts / The same as upstream (pull_request)`. <details> <summary>More: does it help, other systems, switches, what was read</summary> **Does it help:** Unproven. The commit cites branch-only CPU samples of 3.1% of one core in submit_recording in Kokiri Forest and 1.0% in timestamp reads in Hyrule Field. Those show where time was spent. There are zero head-versus-base gameplay runs and no measured savings, average FPS, 1% lows, median or p99 frame time, or submit-to-GPU delay. **Other systems:** The worker can affect NVIDIA, AMD, Intel and software Vulkan paths with the recorder thread off, including four- and five-core Auto setups. RADV RDNA and Steam Deck keep legacy batching, while recording-slot waits and split-part progress still change there. Frame-overlap tracking keeps slot-by-slot timestamp collection whenever async replay is active. Sparse binds, upload chunks and the present copy keep their queue locks and flushes. FSR and readbacks depend on successful completion before reuse. Reflex, GPU timing diagnostics and host recording-copy reclamation have separate effects called out below. The diff changes no save format, audio or input code, public renderer interface, or shader translation inputs. **Switches:** RX_VK_SUBMIT_BATCH is new, read once, and defaults on. Its default has no gameplay evidence and its worker currently bypasses the RADV GCN exclusion. RX_VK_REPLAYED_CACHE comes from #1994, is read once, defaults on, and also lacks gameplay measurements. Both have switch-table rows. RX_VK_BATCH keeps its device-policy default, and an explicit on or off value suppresses the new worker. RX_VK_BATCH_MAX still defaults to 32 command buffers and RX_VK_BATCH_POLL_US to 2000 microseconds. Both now reach worker-held submits on additional drivers. RX_VK_BENCH keeps its existing syntax and suppresses the worker when set. **Reviewed:** 402a2ad76530 against 75317e1f31ff: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, developer standards, both commits, the complete diff, and all nine changed test files. Read all changed renderer functions and traced their recorder, queue-lock, progress, host-copy, uniform, readback, Reflex, presentation and exit callers. Read the switch rows and their policy context. Skimmed unrelated portions of vk_replay.cpp, switches.def and supporting renderer files. No review-checklist file, sibling Tooling checkout, raw run logs, translations or generated files were present in the review material.; nothing run (a code review). **Checked and fine:** - Both commits carry Assisted-by trailers. The diff contains source and authored fixtures, with no game-derived assets, build artifacts, generated shaders or third-party changes. git diff --check was clean. - On successful, ordered submits, the recording timeline signals at ALL_COMMANDS. Slot reuse waits for the recorder ticket before reading the slot's submission state. - The bulk query indices, stride and buffer size fit the 48-slot ring. The fixture exercises ring wrap, unavailable results, consumption once and the frame-overlap bypass. - The new switch fixtures use the shared parser in fresh child processes and isolate the switches they test. They respect an explicit C++ compiler and CXX. The added tests compile local fixtures with standard-library tooling and require no network or external downloads. - The stated dependency on #1994 matches the use of completed_seen. The renderer API version and shader generation rules do not require changes for this diff. **Testing:** Read-only review. The author reports Windows 11, i7-9700, RTX 4060 and llvm-mingw clang 23, with 132 of 159 checks passing and 27 failures also reproduced on the base. The new submit-batch fixture reportedly passed. Gameplay, Linux build evidence and Tier 1 hardware runs are still outstanding. The supplied CI snapshot has checks in progress and lists no head-only failures. **Overlaps and merge order:** #1994 is included in the supplied diff and must merge first if kept as a separate pull request. Resolve the cache finding there, then rebase #1995 and measure against the resulting base. No other overlap was established from the checkout. </details> --- <sub>Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -279,3 +331,4 @@
std::atomic<uint64_t> completed_seen{0};
uint64_t completed() {
uint64_t v = 0;
vkGetSemaphoreCounterValue(dev.device, timeline, &v);
Member

L7: completed() publishes the output of vkGetSemaphoreCounterValue even when the call fails. A failed read does not establish a valid completion value. If it leaves an invalid high value, the monotonic cache keeps it and can let the host recycle recording copies early. Update the cache only on VK_SUCCESS, route device loss through device_lost(), and add a failed-read case to #1994's fixture.

**L7**: completed() publishes the output of vkGetSemaphoreCounterValue even when the call fails. A failed read does not establish a valid completion value. If it leaves an invalid high value, the monotonic cache keeps it and can let the host recycle recording copies early. Update the cache only on VK_SUCCESS, route device loss through device_lost(), and add a failed-read case to #1994's fixture.
@ -248,0 +288,4 @@
wi.semaphoreCount = 1;
wi.pSemaphores = &rec_timeline;
wi.pValues = &r.seq;
if (vkWaitSemaphores(dev.device, &wi, UINT64_MAX) == VK_SUCCESS && r.seq > rec_done_seen) rec_done_seen = r.seq;
Member

L2: slot_free_by_seq clears ownership and reports success on every wait result. If vkWaitSemaphores returns an allocation error with the recording still in flight, begin_recording resets its pools and reuses its streams. Handle device loss immediately and only release the slot after completion is established. A fatal error is safer than reuse when completion cannot be established.

**L2**: slot_free_by_seq clears ownership and reports success on every wait result. If vkWaitSemaphores returns an allocation error with the recording still in flight, begin_recording resets its pools and reuses its streams. Handle device loss immediately and only release the slot after completion is established. A fatal error is safer than reuse when completion cannot be established.
@ -308,0 +387,4 @@
std::atomic<bool> submit_thread_failed{false}; // (its thread could not be started: the replaying thread submits, as with =0)
bool submit_thread_on() {
static const bool on = [] {
const bool v = submit_batch_on() && !dev.policy.set(Sw::batch_submits) && !batch_wanted() && !getenv("RX_VK_BENCH") &&
Member

L9: The worker is enabled on RADV GCN when RX_VK_BATCH is unset and the recorder is off. batch_wanted() is false there specifically to keep the RX 580 on direct submits. The existing Kokiri result was 24 FPS direct versus 20 with legacy batching. The new worker can also coalesce recordings, so preserve the GCN exclusion in device policy and test the effective submission mode.

**L9**: The worker is enabled on RADV GCN when RX_VK_BATCH is unset and the recorder is off. batch_wanted() is false there specifically to keep the RX 580 on direct submits. The existing Kokiri result was 24 FPS direct versus 20 with legacy batching. The new worker can also coalesce recordings, so preserve the GCN exclusion in device policy and test the effective submission mode.
@ -1043,2 +1220,3 @@
// untagged. Batching is on by default only with RADV, so on NVIDIA this matters only with RX_VK_BATCH=1)
// untagged. Batching is on by default only with RADV, so on NVIDIA this matters only with RX_VK_BATCH=1; with the submit
// thread (submit_thread_on) a batch is nearly always one recording, tagged as its own direct submit would be)
if (pend.cbs.empty()) batch_next++, pend_since = std::chrono::steady_clock::now(), pend.reflex = reflex_frame();
Member

L11 (minor): A worker batch can span Reflex frame IDs. If the worker is delayed while frame generation hands off one game frame and replay queues the next, the new frame's commands inherit the first frame's ID or go untagged. Split worker batches at Reflex ID changes and verify the frame-generation marker and submit ordering on NVIDIA.

**L11** (minor): A worker batch can span Reflex frame IDs. If the worker is delayed while frame generation hands off one game frame and replay queues the next, the new frame's commands inherit the first frame's ID or go untagged. Split worker batches at Reflex ID changes and verify the frame-generation marker and submit ordering on NVIDIA.
@ -521,2 +695,4 @@
ts_pending[r.slot] = true;
// (longshot, RX_VK_SUBMIT_BATCH: the number submit_recording, the only caller, gives this recording next; replay_mu)
ts_seq[r.slot] = rec_timeline ? rec_seq + 1 : 0;
if (gtl_on()) gtl_submit_ns[r.slot] = gtl_now(); // (the submit follows at once: submit_recording)
Member

L12 (minor): RX_VK_GPU_TIMELINE records the enqueue time as the submit time on the worker path. A worker delayed behind the queue lock makes the trace attribute that CPU delay to the GPU. Record the actual submit time when flushing, with synchronized per-recording metadata, or expose enqueue and submit as separate events.

**L12** (minor): RX_VK_GPU_TIMELINE records the enqueue time as the submit time on the worker path. A worker delayed behind the queue lock makes the trace attribute that CPU delay to the GPU. Record the actual submit time when flushing, with synchronized per-recording metadata, or expose enqueue and submit as separate events.
@ -396,0 +492,4 @@
std::mutex mu; // (taken with dev.present_mu held, by the kick; never the other way round)
std::condition_variable cv;
bool started = false;
} submit_thread;
Member

L5: The detached worker can wait forever on a condition variable with a static destructor. Windows ExitProcess can kill it while it is in the wait or using renderer state, then run the DLL's destructors. Follow the recorder's process-lifetime ownership pattern for forced process exit and provide an explicit stop before any device or module teardown. Cover quitting with the worker asleep and with work pending.

**L5**: The detached worker can wait forever on a condition variable with a static destructor. Windows ExitProcess can kill it while it is in the wait or using renderer state, then run the DLL's destructors. Follow the recorder's process-lifetime ownership pattern for forced process exit and provide an explicit stop before any device or module teardown. Cover quitting with the worker asleep and with work pending.
@ -491,0 +638,4 @@
for (int s = 0; s < kRecordings && !any; s++) any = ts_pending[s] && ts_seq[s];
if (!any) return;
uint64_t done = 0;
if (vkGetSemaphoreCounterValue(dev.device, rec_timeline, &done) != VK_SUCCESS) return;
Member

L13 (minor): Bulk timestamp collection silently ignores device loss from both the semaphore counter read and the query read. If the driver reports loss there, rendering continues until some later call detects it. Route VK_ERROR_DEVICE_LOST from both calls through device_lost() as required by the renderer's resource rules.

**L13** (minor): Bulk timestamp collection silently ignores device loss from both the semaphore counter read and the query read. If the driver reports loss there, rendering continues until some later call detects it. Route VK_ERROR_DEVICE_LOST from both calls through device_lost() as required by the renderer's resource rules.
@ -491,0 +657,4 @@
const uint64_t* t1 = t[2 * (k - s) + 1];
if (!t0[1] || !t1[1]) continue; // (not there after all: the slot's own read, when it comes round)
ts_pending[k] = false;
if (t1[0] > t0[0]) gpu_time_note(recordings[k], t0[0], t1[0]);
Member

L6: Bulk collection can read gtl_submit_ns without the recorder ticket synchronization that slot reuse provides. With RX_VK_ASYNC=0, RX_VK_RECORDER=thread and RX_VK_GPU_TIMELINE=1, part_submit writes the plain slot timestamp on the recorder while bulk collection reads it under replay_mu alone. Observing GPU timeline progress does not publish that CPU write. Acquire the matching recorder completion before consuming its metadata, or publish the timestamp with suitable synchronization.

**L6**: Bulk collection can read gtl_submit_ns without the recorder ticket synchronization that slot reuse provides. With RX_VK_ASYNC=0, RX_VK_RECORDER=thread and RX_VK_GPU_TIMELINE=1, part_submit writes the plain slot timestamp on the recorder while bulk collection reads it under replay_mu alone. Observing GPU timeline progress does not publish that CPU write. Acquire the matching recorder completion before consuming its metadata, or publish the timestamp with suitable synchronization.
@ -937,3 +1113,3 @@
r.part = ++rec_part_next;
memory_recording_part(r.part);
const uint64_t done = completed();
const uint64_t done = completed_for_recording(); // (longshot, RX_VK_SUBMIT_BATCH: a later part's, as last read)
Member

L8: Stale progress can change what a draw reads. If an earlier GPU write finishes between parts, an Intel draw that overlaps its uniform range can still be classified as having a pending writer. uniform_range then binds the range in place where a fresh read would have selected the captured bytes. The game can overwrite that binding ahead of GPU execution. Keep a fresh read for this decision, or refresh when an earlier writer would suppress the snapshot. Test completion advancing between parts.

**L8**: Stale progress can change what a draw reads. If an earlier GPU write finishes between parts, an Intel draw that overlaps its uniform range can still be classified as having a pending writer. uniform_range then binds the range in place where a fresh read would have selected the captured bytes. The game can overwrite that binding ahead of GPU execution. Keep a fresh read for this decision, or refresh when an earlier writer would suppress the snapshot. Test completion advancing between parts.
@ -1052,3 +1231,4 @@
// (RX_VK_BATCH_MAX=<command buffers>: held at most this many, default 32)
static const size_t cap = getenv("RX_VK_BATCH_MAX") ? (size_t)std::max(1, atoi(getenv("RX_VK_BATCH_MAX"))) : 32;
if (pend.cbs.size() >= cap) flush_reason = FR_CAP, queue_flush_locked();
if (submit_thread_on()) { // (longshot, RX_VK_SUBMIT_BATCH: sent by the submit thread, now, whatever the GPU is at)
Member

L10: Holding submits by default exposes previously direct setups to blocking progress polls. A zero-timeout nvnSyncWait can enter queue_flush_for and block on present_mu before it gets to Vulkan's zero-timeout wait. get_replayed can also block on that mutex while holding the host's exclusive copy lock. Measure poll duration and frame-time tails on a worker-enabled setup with a shared present queue.

**L10**: Holding submits by default exposes previously direct setups to blocking progress polls. A zero-timeout nvnSyncWait can enter queue_flush_for and block on present_mu before it gets to Vulkan's zero-timeout wait. get_replayed can also block on that mutex while holding the host's exclusive copy lock. Measure poll duration and frame-time tails on a worker-enabled setup with a shared present queue.
@ -1053,2 +1232,4 @@
static const size_t cap = getenv("RX_VK_BATCH_MAX") ? (size_t)std::max(1, atoi(getenv("RX_VK_BATCH_MAX"))) : 32;
if (pend.cbs.size() >= cap) flush_reason = FR_CAP, queue_flush_locked();
if (submit_thread_on()) { // (longshot, RX_VK_SUBMIT_BATCH: sent by the submit thread, now, whatever the GPU is at)
if (!pend.cbs.empty()) submit_thread_kick();
Member

L1: Worker creation can fail after the first recording is already in pend. The next recording takes the direct path and can submit sequence 2 before held sequence 1, which may later try to signal the timeline backwards. Select direct submission before publishing any held work, or flush the first recording synchronously under present_mu before switching to direct. Inject thread creation failure and assert actual queue order.

**L1**: Worker creation can fail after the first recording is already in pend. The next recording takes the direct path and can submit sequence 2 before held sequence 1, which may later try to signal the timeline backwards. Select direct submission before publishing any held work, or flush the first recording synchronously under present_mu before switching to direct. Inject thread creation failure and assert actual queue order.
@ -0,0 +271,4 @@
"a slot's wait by its recording's number no longer sends a held submit first, or takes a slot that has no number")
# ---- the submit thread ----
thread = block(replay, "void submit_thread_main() {")
Member

L14 (minor): The submit-worker checks only look for source strings. Removing notify_one() or disabling worker creation leaves them green. Run the handoff against a fake queue and assert that an idle worker wakes, sends the work once and preserves queue order. The slot fixture should also use distinct fake semaphore handles and check the exact target value before releasing pending GPU reads. Its current stub accepts a wait for 6 when the slot requires 7.

**L14** (minor): The submit-worker checks only look for source strings. Removing notify_one() or disabling worker creation leaves them green. Run the handoff against a fake queue and assert that an idle worker wakes, sends the work once and preserves queue order. The slot fixture should also use distinct fake semaphore handles and check the exact target value before releasing pending GPU reads. Its current stub accepts a wait for 6 when the slot requires 7.
siantm force-pushed perf/vk-submit-batch from 402a2ad765
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 8s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to 56d4ac6b5e
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 7s
2026-11-21 04:48:37 +00:00
Compare
siantm changed title from WIP: perf(vk): fewer driver calls at a recording's end and start, and a submit thread where nothing else submits (RX_VK_SUBMIT_BATCH=0 to disable) to WIP: perf(vk): fewer driver calls at a recording's end and start: slots freed by the recordings' timeline, timestamps read once a present (RX_VK_SUBMIT_BATCH=0 to disable) 2026-11-21 04:48:46 +00:00
Author
Contributor

Thanks for the review (#issuecomment-63010). Pushed: rebased onto #1994's new head d8a583a6 (on main f1f2dc5b; the PR's own commit unchanged, bd592530), then 3a4dd2d3 (the submit thread and a later part's progress as last read taken out), a test commit 5e004d35 and a fix 56d4ac6b. The tests fail at 5e004d35 and pass at the head (this PC's job queue, jobs fb-1995-1 and fb-1995-2); vk_replay.cpp compiles with the game build's own command. Not run in the game yet.

The submit thread is taken out of this PR (3a4dd2d3): it returns in a PR of its own, stacked on this one and off by default until measured in play, with the fixes asked here. So:

  • L1 (62997), L3, L4, L5 (62999), L9 (63004), L10 (63005), L11 (63006), L12 (63007), L17 and the submit-thread part of L14 (63009): removed in 3a4dd2d3; it returns in its own PR, off by default, with these fixes (the thread started before any work is held, an explicit stop and state never destroyed at exit, its own atomic profiling counter, RADV excluded, try-lock polls, batches split at Reflex frame changes, off under RX_VK_GPU_TIMELINE or real submit times, a fake-queue handoff test, enqueue-to-submit times and counts by flush reason, a failed held batch failing explicitly). Without it, submits are held only where main holds them (RADV by default, RX_VK_BATCH=1), as on main.
  • L2 (62998) fixed in 56d4ac6b: a lost device at the slot's read or wait goes to device_lost(); a wait that fails otherwise stops the renderer with a FATAL line (replay_fatal) instead of freeing the slot. tests/test_submit_batch.py (5e004d35) runs the real rec_done and slot_free_by_seq through a reached number, a wait, a held batch, a slot with no number, a failed wait, a lost wait and a lost read; the failing three fail at 5e004d35 and pass at the head.
  • L6 (63000) fixed in 56d4ac6b: with RX_VK_GPU_TIMELINE on nothing is collected once a present; each slot is read when it comes round, after the recorder's ticket (rec::Settled) that orders the submit time's write.
  • L7 (63001): #1994's, fixed there in d8a583a6 (only a successful read kept, a lost device to device_lost(), a failed-read case in its fixture); this branch is on it.
  • L8 (63003): removed in 3a4dd2d3: begin_recording reads the timeline at every recording again, as main, so the uniform decision has a fresh read.
  • L13 (63008) fixed in 56d4ac6b: VK_ERROR_DEVICE_LOST from the timeline's read and from the queries' read go to device_lost(); the fixture covers both.
  • L14, the slot fixture: in 5e004d35 the recordings' timeline is a handle of its own and the wait must be for the slot's own number (7) before anything is released, both in tests/test_submit_batch.py's new slot fixture and in fsr_uniform_ownership_test.cpp. That second fixture does not build on main f1f2dc5b (its stand-ins are behind main's begin_recording: pre-existing, it fails there the same way), so the slot cases were added where they run.
  • L15 done in 5e004d35: test_fsr_image_retirement.py and test_draw_stream_visibility.py run their fixtures without RX_VK_BATCH_MAX. Both fail on main f1f2dc5b before reaching their cases (a stand-in missing, a cut marker gone: pre-existing), so the effect could not be run here.
  • L16: the description and the switch row say what each switch gives back: RX_VK_SUBMIT_BATCH=0 the fences and slot-by-slot timestamps; main's behaviour is both it and RX_VK_REPLAYED_CACHE at 0. Still to test isolates this PR against #1994's head with the replay cache at the same setting on every side, and compares main with both off.
  • L17 (the run): pending, in Still to test (Threaded rendering off, on and Auto, Tier 1 hardware, average FPS, 1% lows, median and p99 frame times, peak RAM).
Thanks for the review (#issuecomment-63010). Pushed: rebased onto #1994's new head `d8a583a6` (on main `f1f2dc5b`; the PR's own commit unchanged, `bd592530`), then `3a4dd2d3` (the submit thread and a later part's progress as last read taken out), a test commit `5e004d35` and a fix `56d4ac6b`. The tests fail at `5e004d35` and pass at the head (this PC's job queue, jobs `fb-1995-1` and `fb-1995-2`); `vk_replay.cpp` compiles with the game build's own command. Not run in the game yet. The submit thread is taken out of this PR (`3a4dd2d3`): it returns in a PR of its own, stacked on this one and off by default until measured in play, with the fixes asked here. So: - **L1** ([62997](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-62997)), **L3**, **L4**, **L5** ([62999](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-62999)), **L9** ([63004](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63004)), **L10** ([63005](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63005)), **L11** ([63006](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63006)), **L12** ([63007](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63007)), **L17** and the submit-thread part of **L14** ([63009](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63009)): removed in `3a4dd2d3`; it returns in its own PR, off by default, with these fixes (the thread started before any work is held, an explicit stop and state never destroyed at exit, its own atomic profiling counter, RADV excluded, try-lock polls, batches split at Reflex frame changes, off under `RX_VK_GPU_TIMELINE` or real submit times, a fake-queue handoff test, enqueue-to-submit times and counts by flush reason, a failed held batch failing explicitly). Without it, submits are held only where main holds them (RADV by default, `RX_VK_BATCH=1`), as on main. - **L2** ([62998](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-62998)) fixed in `56d4ac6b`: a lost device at the slot's read or wait goes to `device_lost()`; a wait that fails otherwise stops the renderer with a FATAL line (`replay_fatal`) instead of freeing the slot. `tests/test_submit_batch.py` (`5e004d35`) runs the real `rec_done` and `slot_free_by_seq` through a reached number, a wait, a held batch, a slot with no number, a failed wait, a lost wait and a lost read; the failing three fail at `5e004d35` and pass at the head. - **L6** ([63000](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63000)) fixed in `56d4ac6b`: with `RX_VK_GPU_TIMELINE` on nothing is collected once a present; each slot is read when it comes round, after the recorder's ticket (`rec::Settled`) that orders the submit time's write. - **L7** ([63001](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63001)): #1994's, fixed there in `d8a583a6` (only a successful read kept, a lost device to `device_lost()`, a failed-read case in its fixture); this branch is on it. - **L8** ([63003](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63003)): removed in `3a4dd2d3`: `begin_recording` reads the timeline at every recording again, as main, so the uniform decision has a fresh read. - **L13** ([63008](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1995#issuecomment-63008)) fixed in `56d4ac6b`: `VK_ERROR_DEVICE_LOST` from the timeline's read and from the queries' read go to `device_lost()`; the fixture covers both. - **L14**, the slot fixture: in `5e004d35` the recordings' timeline is a handle of its own and the wait must be for the slot's own number (7) before anything is released, both in `tests/test_submit_batch.py`'s new slot fixture and in `fsr_uniform_ownership_test.cpp`. That second fixture does not build on main `f1f2dc5b` (its stand-ins are behind main's `begin_recording`: pre-existing, it fails there the same way), so the slot cases were added where they run. - **L15** done in `5e004d35`: `test_fsr_image_retirement.py` and `test_draw_stream_visibility.py` run their fixtures without `RX_VK_BATCH_MAX`. Both fail on main `f1f2dc5b` before reaching their cases (a stand-in missing, a cut marker gone: pre-existing), so the effect could not be run here. - **L16**: the description and the switch row say what each switch gives back: `RX_VK_SUBMIT_BATCH=0` the fences and slot-by-slot timestamps; main's behaviour is both it and `RX_VK_REPLAYED_CACHE` at 0. Still to test isolates this PR against #1994's head with the replay cache at the same setting on every side, and compares main with both off. - **L17** (the run): pending, in Still to test (Threaded rendering off, on and Auto, Tier 1 hardware, average FPS, 1% lows, median and p99 frame times, peak RAM).
Author
Contributor

@Beanstalk This adds RX_VK_SUBMIT_BATCH, a kill switch, on by default: a recording's slot is free again by its number on the recordings' timeline instead of a fence, and the GPU timestamps of finished recordings are read once a present (gpu/vk); RX_VK_SUBMIT_BATCH=0 is main's way for this part. It is on #1994, whose commit is in its diff, so the switch set against unset within this PR's build shows this PR's own part. It has not run in the game on these PCs yet. Please play Hyrule Field for three minutes after a minute's warm-up, the PR against main on the same PC, on the PR's side with the switch unset and with RX_VK_SUBMIT_BATCH=0, on Linux with NVIDIA, Linux with AMD (RADV), Windows with Intel and the Steam Deck. The same picture, no crash, hang, device lost or FATAL line, and GPU times in the FPS counter alike; unset, frame time p99 equal to the switch off or better, and to main's or better.

@Beanstalk This adds RX_VK_SUBMIT_BATCH, a kill switch, on by default: a recording's slot is free again by its number on the recordings' timeline instead of a fence, and the GPU timestamps of finished recordings are read once a present (gpu/vk); RX_VK_SUBMIT_BATCH=0 is main's way for this part. It is on #1994, whose commit is in its diff, so the switch set against unset within this PR's build shows this PR's own part. It has not run in the game on these PCs yet. Please play Hyrule Field for three minutes after a minute's warm-up, the PR against main on the same PC, on the PR's side with the switch unset and with RX_VK_SUBMIT_BATCH=0, on Linux with NVIDIA, Linux with AMD (RADV), Windows with Intel and the Steam Deck. The same picture, no crash, hang, device lost or FATAL line, and GPU times in the FPS counter alike; unset, frame time p99 equal to the switch off or better, and to main's or better.
Member

@siantm only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@siantm only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Author
Contributor

Description: Tested now names the three fixtures whose additions did not run (they do not build on main f1f2dc5b), and Still to test lists running them once they are repaired.

Description: Tested now names the three fixtures whose additions did not run (they do not build on main f1f2dc5b), and Still to test lists running them once they are repaired.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
PR description / Description follows the template (pull_request_target) Successful in 7s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/vk-submit-batch:siantm-perf/vk-submit-batch'
git switch 'siantm-perf/vk-submit-batch'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1995
No description provided.