WIP: perf(vk): a draw leaves out a block of push constants its command buffer already has (RX_VK_BIND_CHANGED, on) #2047

Draft
siantm wants to merge 1 commit from siantm/Longshot:perf/vk-push-changed into main
Contributor

Summary

push_constants (gpu/vk/vk_draw.cpp) pushed the program's whole block (page table, task first, render area, fallback addresses) at every draw whose program reads it, though the draws of a pass push the same block one after another. It now leaves the push out when the last block this thread recorded was the same bytes, for the same stages, into the same command buffer (push_standing). gpu/vk/vk_work.h counts every vkCmdPushConstants on the thread (push_epoch), so a push of anyone's in between makes the next draw push again. The block is forgotten where the command buffer's may be something else: at each recording's begin (binds_begin_recording), a masked clear, the temporal upscalers' stand-ins, each dispatch's end, and descriptors_disturbed, which every upscaler and library pass on main calls after recording its own pipelines. A direct mesh draw no longer pushes entry 0's first a second time. The result expected is fewer push calls a frame.

The branch's index-buffer part of the same switch is main's RX_VK_INDEX_REBASE already and is not here.

Switch: RX_VK_BIND_CHANGED (kind kill, on by default, the branch's default; its row in gpu/vk/switches.def). RX_VK_BIND_CHANGED=0 pushes at every draw, main's code. The switch is read once, at its first use (so HOS_AB cannot alternate it); 0, false, off and no turn it off, any other value leaves it on; its first read logs what it chose.

From bd3ea307 on exp/x86-asm-sync.

Read first: push_standing, PushSent, push_forget and push_constants in gpu/vk/vk_draw.cpp; push_epoch in gpu/vk/vk_work.h.

Testing

Tested: Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head aebcf746 on main @ 75317e1f. Every check ran through tests/run_all.py --strict --exact in build jobs of the PC's job queue, in a hidden console (log t-dc3-vk-push-changed):

  • Its own: tests/test_push_changed.py (new: runs push_standing cut from the source and reads the wiring: vk_work.h's count, the check before the push, the block kept after it, every place it is forgotten, the mesh first; and the switch's reader, cut from the source, runs in fresh processes: unset, 0, off, false, 1, on and another value, twice each, and set to the other answer after its first read); the three fixtures that compile draw_begin_recording (tests/draw_stream_visibility_test.cpp, tests/fsr_uniform_ownership_test.cpp, tests/recording_admission_test.cpp) stand in binds_begin_recording: pass.
  • The 184 checks that read a file this change touches (--changed 75317e1f; its own are among them): 161 passed, 23 failed. Failed: tests/test_recorder_sync.py, tests/test_fsr_uniform_ownership.py, tests/test_sampler_admission.py, tests/test_pipeline_play_cache.py, tests/test_draw_stream_visibility.py, tests/test_device_lost_exit.py, tests/test_dev_packs.py, tests/test_dev_ab.py, tests/test_draw_binds.py, tests/test_crash_dialog.py, tests/test_frame_overlap.py, tests/test_gpu_wait_stats.py, tests/test_linux_pressure.py, tests/test_mesh_conv.py, tests/test_pass_barrier.py, tests/test_prepare_cache_save.py, tests/test_present_queue_default.py, tests/test_recorder_own_idle.py, tests/test_target_reuse_memory.py, tests/test_ub_update_batch.py, tests/test_upscaler_rows.py, tests/test_vk_occlusion_queries.py: each fails on main @ 75317e1f too (the same checks on main in the same queue batch, log t-dc3-base). tests/test_recording_admission.py also failed in that run (its parallel case, "worker slot ownership") and passed when run again on this head (log t-dc4-vk-push-changed); main passed it in that batch and in five runs of the next (logs t-dc4-base-ra1 to t-dc4-base-ra5).

It has NOT run in the game on this PC yet.

Still to test:

  • Windows 11, i7-9700, RTX 4060 (this PC): start in Kokiri Forest from the scene's save, play a minute (walking, turning the camera), then with DLSS or FSR as the Upscaler: main @ 75317e1f, this PR with RX_VK_BIND_CHANGED=0 and this PR with it unset (on), each warmed a minute in the scene first, then interleaved (main, =0, unset, unset, =0, main), two runs each, expecting the same picture all three ways (no flicker, no wrong-sized or misplaced geometry), =0 as main, and unset the replay's time a frame the same as or lower than =0's and main's (by hand)
  • Linux NVIDIA and AMD, Windows AMD and Intel, Steam Deck: start in Kokiri Forest from the scene's save, play a minute (walking, turning the camera), main against this PR with RX_VK_BIND_CHANGED=0 and unset, warmed, in ABBA order, expecting the same picture every way and no crash or validation error (@Beanstalk linux+nvidia, linux+amd, windows+amd, windows+intel, steamdeck: Kokiri Forest a minute, main against this PR with RX_VK_BIND_CHANGED=0 and unset, ABBA, the same picture)
  • Linux (any GPU): build the renderer and run tests/test_push_changed.py with --cxx g++ and --cxx clang++, expecting PASS (by hand)

Measurements

Verdict: not measured: no micro-benchmark of this port and no run in the game on this PC yet; the in-game A/B is in Still to test. RX_VK_BIND_CHANGED ships on (kind kill), the branch's default; it is not measured on main.

Risks

A push left out where the command buffer held another block (one recorded outside vk_work.h's count and not followed by descriptors_disturbed): draws would read another pass's page table or render area: wrong or missing geometry, or a device loss. NGX and FidelityFX pushes made outside a dispatch are covered only through descriptors_disturbed. Turned off by RX_VK_BIND_CHANGED=0. The renderer is compiled again after updating.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says. Not yet: it has not run in the game.
  • mise run test passes (any failure that also fails on main is named). Not yet: only the checks in Testing ran.
  • Measured against main, or Measurements says why there is nothing to measure. Not yet (Measurements).
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A switch added or changed: it ships at the default the measurements chose, with its row in gpu/vk/switches.def. Its row is in; not measured yet.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. Not yet: only the checks' cuts compiled (Testing).
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
## Summary `push_constants` (`gpu/vk/vk_draw.cpp`) pushed the program's whole block (page table, task first, render area, fallback addresses) at every draw whose program reads it, though the draws of a pass push the same block one after another. It now leaves the push out when the last block this thread recorded was the same bytes, for the same stages, into the same command buffer (`push_standing`). `gpu/vk/vk_work.h` counts every `vkCmdPushConstants` on the thread (`push_epoch`), so a push of anyone's in between makes the next draw push again. The block is forgotten where the command buffer's may be something else: at each recording's begin (`binds_begin_recording`), a masked clear, the temporal upscalers' stand-ins, each dispatch's end, and `descriptors_disturbed`, which every upscaler and library pass on main calls after recording its own pipelines. A direct mesh draw no longer pushes entry 0's first a second time. The result expected is fewer push calls a frame. The branch's index-buffer part of the same switch is main's `RX_VK_INDEX_REBASE` already and is not here. Switch: `RX_VK_BIND_CHANGED` (kind kill, on by default, the branch's default; its row in `gpu/vk/switches.def`). `RX_VK_BIND_CHANGED=0` pushes at every draw, main's code. The switch is read once, at its first use (so `HOS_AB` cannot alternate it); `0`, `false`, `off` and `no` turn it off, any other value leaves it on; its first read logs what it chose. From bd3ea307 on `exp/x86-asm-sync`. Read first: `push_standing`, `PushSent`, `push_forget` and `push_constants` in `gpu/vk/vk_draw.cpp`; `push_epoch` in `gpu/vk/vk_work.h`. ## Testing **Tested:** Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head aebcf746 on main @ 75317e1f. Every check ran through `tests/run_all.py --strict --exact` in build jobs of the PC's job queue, in a hidden console (log `t-dc3-vk-push-changed`): - Its own: `tests/test_push_changed.py` (new: runs `push_standing` cut from the source and reads the wiring: `vk_work.h`'s count, the check before the push, the block kept after it, every place it is forgotten, the mesh first; and the switch's reader, cut from the source, runs in fresh processes: unset, `0`, `off`, `false`, `1`, `on` and another value, twice each, and set to the other answer after its first read); the three fixtures that compile `draw_begin_recording` (`tests/draw_stream_visibility_test.cpp`, `tests/fsr_uniform_ownership_test.cpp`, `tests/recording_admission_test.cpp`) stand in `binds_begin_recording`: pass. - The 184 checks that read a file this change touches (`--changed 75317e1f`; its own are among them): 161 passed, 23 failed. Failed: `tests/test_recorder_sync.py`, `tests/test_fsr_uniform_ownership.py`, `tests/test_sampler_admission.py`, `tests/test_pipeline_play_cache.py`, `tests/test_draw_stream_visibility.py`, `tests/test_device_lost_exit.py`, `tests/test_dev_packs.py`, `tests/test_dev_ab.py`, `tests/test_draw_binds.py`, `tests/test_crash_dialog.py`, `tests/test_frame_overlap.py`, `tests/test_gpu_wait_stats.py`, `tests/test_linux_pressure.py`, `tests/test_mesh_conv.py`, `tests/test_pass_barrier.py`, `tests/test_prepare_cache_save.py`, `tests/test_present_queue_default.py`, `tests/test_recorder_own_idle.py`, `tests/test_target_reuse_memory.py`, `tests/test_ub_update_batch.py`, `tests/test_upscaler_rows.py`, `tests/test_vk_occlusion_queries.py`: each fails on main @ 75317e1f too (the same checks on main in the same queue batch, log `t-dc3-base`). `tests/test_recording_admission.py` also failed in that run (its `parallel` case, "worker slot ownership") and passed when run again on this head (log `t-dc4-vk-push-changed`); main passed it in that batch and in five runs of the next (logs `t-dc4-base-ra1` to `t-dc4-base-ra5`). It has NOT run in the game on this PC yet. **Still to test:** - Windows 11, i7-9700, RTX 4060 (this PC): start in Kokiri Forest from the scene's save, play a minute (walking, turning the camera), then with DLSS or FSR as the Upscaler: main @ 75317e1f, this PR with `RX_VK_BIND_CHANGED=0` and this PR with it unset (on), each warmed a minute in the scene first, then interleaved (main, =0, unset, unset, =0, main), two runs each, expecting the same picture all three ways (no flicker, no wrong-sized or misplaced geometry), =0 as main, and unset the replay's time a frame the same as or lower than =0's and main's (by hand) - Linux NVIDIA and AMD, Windows AMD and Intel, Steam Deck: start in Kokiri Forest from the scene's save, play a minute (walking, turning the camera), main against this PR with `RX_VK_BIND_CHANGED=0` and unset, warmed, in ABBA order, expecting the same picture every way and no crash or validation error (@Beanstalk linux+nvidia, linux+amd, windows+amd, windows+intel, steamdeck: Kokiri Forest a minute, main against this PR with RX_VK_BIND_CHANGED=0 and unset, ABBA, the same picture) - Linux (any GPU): build the renderer and run `tests/test_push_changed.py` with `--cxx g++` and `--cxx clang++`, expecting PASS (by hand) ## Measurements **Verdict:** not measured: no micro-benchmark of this port and no run in the game on this PC yet; the in-game A/B is in Still to test. `RX_VK_BIND_CHANGED` ships on (kind kill), the branch's default; it is not measured on main. ## Risks A push left out where the command buffer held another block (one recorded outside `vk_work.h`'s count and not followed by `descriptors_disturbed`): draws would read another pass's page table or render area: wrong or missing geometry, or a device loss. NGX and FidelityFX pushes made outside a dispatch are covered only through `descriptors_disturbed`. Turned off by `RX_VK_BIND_CHANGED=0`. The renderer is compiled again after updating. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. Not yet: it has not run in the game. - [ ] `mise run test` passes (any failure that also fails on `main` is named). Not yet: only the checks in Testing ran. - [ ] Measured against `main`, or Measurements says why there is nothing to measure. Not yet (Measurements). - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [ ] A switch added or changed: it ships at the default the measurements chose, with its row in `gpu/vk/switches.def`. Its row is in; not measured yet. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [ ] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. Not yet: only the checks' cuts compiled (Testing). - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers.
perf(vk): a draw leaves out a block of push constants its command buffer already has (RX_VK_BIND_CHANGED, on)
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 5s
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
aebcf74638
push_constants pushed the program's whole block (page table, task first, render area, fallback addresses) at every
draw whose program reads it, though the draws of a pass push the same block one after another. It now leaves the
push out when the last block this thread recorded was the same bytes, for the same stages, into the same command
buffer (push_standing). vk_work.h counts every vkCmdPushConstants on the thread (push_epoch), so a push of anyone's
in between makes the next draw push again. The block is forgotten where the command buffer's may be something else:
at each recording's begin (binds_begin_recording), a masked clear, the temporal upscalers' stand-ins, each
dispatch's end (its stand-ins call NGX / FidelityFX, which push past vk_work.h), and (new here, beside the branch's
sites) in descriptors_disturbed, which every upscaler and library pass on main calls after recording its own
pipelines. A direct mesh draw no longer pushes entry 0's first a second time: it is in the block just sent or left
standing.

The first of RX_VK_BIND_CHANGED's parts on the branch (bd3ea307, "draw-binds"); its other parts follow as their own
commits. Its index-buffer part (bind at the pool buffer's start, draw from the first index) is main's
RX_VK_INDEX_REBASE already and is not ported.

Switch: RX_VK_BIND_CHANGED, kill, on by default (the branch's default); =0 (or false, off, no) pushes at every draw as
before. Read once, at its first use. Row in gpu/vk/switches.def. Not measured on its own on the branch ("the saving is
not measured yet"); not measured in play.

Test: tests/test_push_changed.py runs push_standing (cut from the source) and reads the wiring: vk_work.h's count, the
check before the push, the block kept after it, every place it is forgotten, the mesh first; and the switch's reader
runs in fresh processes (unset, 0, off, false, 1, on and another value, twice each, set again after its first read).
The three fixtures that compile draw_begin_recording stand in binds_begin_recording.

Assisted-by: Claude Code
Author
Contributor

@Beanstalk This changes the renderer so that the push constants of every draw are pushed only when they changed (gpu/vk), behind RX_VK_BIND_CHANGED, a kill switch on by default; RX_VK_BIND_CHANGED=0 is main's way. It has not run in the game on these PCs yet. Please play Kokiri Forest and Hyrule Field, two minutes each, the PR against main on the same PC, on the PR's side with the switch unset and with RX_VK_BIND_CHANGED=0, with a picture in each compared with main's, on Linux with NVIDIA, Linux with AMD (RADV) and the Steam Deck. The same picture, no crash, device lost, validation error or FATAL line; unset, frame time p99 equal to main's or better and to the switch off.

@Beanstalk This changes the renderer so that the push constants of every draw are pushed only when they changed (gpu/vk), behind RX_VK_BIND_CHANGED, a kill switch on by default; RX_VK_BIND_CHANGED=0 is main's way. It has not run in the game on these PCs yet. Please play Kokiri Forest and Hyrule Field, two minutes each, the PR against main on the same PC, on the PR's side with the switch unset and with RX_VK_BIND_CHANGED=0, with a picture in each compared with main's, on Linux with NVIDIA, Linux with AMD (RADV) and the Steam Deck. The same picture, no crash, device lost, validation error or FATAL line; unset, frame time p99 equal to main's or better and to the switch off.
Member

@siantm only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@siantm only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 5s
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/vk-push-changed:siantm-perf/vk-push-changed'
git switch 'siantm-perf/vk-push-changed'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2047
No description provided.