WIP: perf(vk): the stopped queues' watch sleeps until a queue stops or the CPU signals (RX_VK_SYNC_WATCH_EVENT=0 to disable) #2030
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2030
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/vk-sync-watch-event"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
With the replay on the submitting thread (Threaded rendering off), the watch of the stopped queues (
sync_watch,thread
mkw_vk order) woke every millisecond for the whole run to find, nearly every time, nothing stopped.With this it sleeps on a condition variable of its own until there is something for it: with nothing stopped, until a
queue stops (
update_blockedkicks it); with a queue stopped, until the CPU signals an event (event_signal_queuedkicks it) or until
RX_VK_HOLD_MSafter the first one stopped, where main's hold rule decides as before. A queuestopped at an event whose value is in guest memory, a look that found the order lock taken, or a time past
RX_VK_HOLD_MSstill look every millisecond, as before. No wake-up is lost: a kick takes the watch's lock, under whichthe watch tests before it sleeps. A CPU signal is now looked at at once, not up to a millisecond later. The result
expected: a thread that no longer wakes a thousand times a second for nothing.
The switch:
RX_VK_SYNC_WATCH_EVENT(new, a kill switch, bool, its row ingpu/vk/switches.def), on by default as thebranch shipped it, read once;
=0wakes every millisecond, as main does.From bd3ea307 on
exp/x86-asm-sync(its replay-parts work).Read first:
sync_watch,sync_watch_kick,update_blockedandevent_signal_queuedingpu/vk/vk_replay.cpp,then
tests/test_sync_watch_event.py.Testing
Tested: Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head
492641econ main75317e1f. Build jobs of this PC's job queue (batchCapCut3-b1), the checks in a hidden console:tests/run_all.py --strict --exactontests/test_sync_watch_event.pyand every check--changedpicks for the files this changes: 148 checks, 120 passed, 28 failed, 0 skipped; its own check passes, the switch's read included (fresh processes); the failing ones fail on their base too, run from a base worktree in the same batch:tests/test_capture_readback_visibility.py(fails on main @75317e1ftoo),tests/test_crash_dialog.py(fails on main @75317e1ftoo),tests/test_crash_record.py(fails on main @75317e1ftoo),tests/test_dev_ab.py(fails on main @75317e1ftoo),tests/test_dev_packs.py(fails on main @75317e1ftoo),tests/test_device_lost_exit.py(fails on main @75317e1ftoo),tests/test_draw_stream_visibility.py(fails on main @75317e1ftoo),tests/test_flush_nothing_held.py(fails on main @75317e1ftoo),tests/test_frame_overlap.py(fails on main @75317e1ftoo),tests/test_fsr_image_retirement.py(fails on main @75317e1ftoo),tests/test_fsr_uniform_ownership.py(fails on main @75317e1ftoo),tests/test_gpu_wait_stats.py(fails on main @75317e1ftoo),tests/test_linux_pressure.py(fails on main @75317e1ftoo),tests/test_mesh_conv.py(fails on main @75317e1ftoo),tests/test_perf_report.py(fails on main @75317e1ftoo),tests/test_present_queue_default.py(fails on main @75317e1ftoo),tests/test_present_source_crop.py(fails on main @75317e1ftoo),tests/test_profiling_mode.py(fails on main @75317e1ftoo),tests/test_recorder_own_idle.py(fails on main @75317e1ftoo),tests/test_renderer_new_handler.py(fails on main @75317e1ftoo),tests/test_replay_write_order.py(fails on main @75317e1ftoo),tests/test_slow_frames.py(fails on main @75317e1ftoo),tests/test_sparse_mirror.py(fails on main @75317e1ftoo),tests/test_target_reuse_memory.py(fails on main @75317e1ftoo),tests/test_ub_update_batch.py(fails on main @75317e1ftoo),tests/test_upscaler_rows.py(fails on main @75317e1ftoo),tests/test_vk_occlusion_queries.py(fails on main @75317e1ftoo);tests/test_frame_wait_bounds.pytimed out at 60 s in that run (it passes on main) and passed when run again alone in batchCapCut3-b3.It has NOT run in the game on this PC yet.
Still to test:
75317e1f, this branch withRX_VK_SYNC_WATCH_EVENT=0and this branch with it unset (on), each warmed up a minute in the scene first, interleaved (ABBA), at least two runs each, expecting the same picture, no FATAL, no hang or black frame, themkw_vk orderthread near idle in a profile, and a median frame time unset lower than withRX_VK_SYNC_WATCH_EVENT=0and than main, andRX_VK_SYNC_WATCH_EVENT=0the same as main (by hand)75317e1f, this branch withRX_VK_SYNC_WATCH_EVENT=0and this branch with it unset (on), warmed up, ABBA, expecting the same picture, no FATAL and no hang, and a median frame time unset lower than withRX_VK_SYNC_WATCH_EVENT=0and than main, andRX_VK_SYNC_WATCH_EVENT=0the same as main (@Beanstalk linux+nvidia, linux+amd, windows+amd, windows+intel, steamdeck: Kokiri Forest three minutes, Threaded rendering off: main @75317e1f, this branch with RX_VK_SYNC_WATCH_EVENT=0 and this branch with it unset (on), warmed up, ABBA, the log, frame times and a screenshot)python3 -I -B tests/test_sync_watch_event.py, expecting PASS (by hand)Measurements
Verdict: not measured: no micro-benchmark of this port and no run in the game on this PC yet; the in-game A/B is
in Still to test.
RX_VK_SYNC_WATCH_EVENTships on, as the branch shipped it; no measurement chose that default yet, the in-game A/B in Still to test does.Risks
What could break: a wake-up lost would leave a stopped queue waiting (a hang, or a black frame until the next kick
or
RX_VK_HOLD_MS). The test runs the realsync_watch, its kick and the hold rule on a thread of its own againststand-ins (no sleeps while nothing is stopped, one look at
RX_VK_HOLD_MS, an immediate look at a CPU signal, everymillisecond at a wait on guest memory,
=0). Turn it off withRX_VK_SYNC_WATCH_EVENT=0. Renderer code: the runtimeis rebuilt; nothing changes for players' saves, options or caches.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Recommendation: Request changes. The event-driven watcher can miss a GPU wait and force stopped rendering work past its event before the game supplies the frame data. The new deadline also has a signed overflow, and the tests bypass the production code that publishes the wakeups.
With Threaded rendering off, the watcher can sleep through idle periods and react sooner to CPU event signals. The change adds a mutex, condition variable and deadline bookkeeping to shared replay code. The CPU savings and effect on frame pacing are unproven.
@fixwill do (12)gpu/vk/vk_replay.cpp:3057: The grace period depends on the watcher sampling a live GPU waiter. If a queue stops at 0 ms and an older fence wait ends at 48 ms, the new watcher can sleep until 51 ms, see zero waiters, and drain before the main thread signals. The old polling loop could see that wait and extend the hold. Publish the latest wait completion from GpuWaiter, with ordering that lets the watcher see it when the count reaches zero. Test a waiter that starts and finishes before the first deadline.gpu/vk/vk_replay.cpp:3022: RX_VK_HOLD_MS=2147483647 fits the existing int setting, but hold_ms() + 1 overflows before chrono sees it. (more on the line)gpu/vk/vk_replay.cpp:3034: The timed wait captures one deadline and its predicate can swallow the kick for a new stopped interval. (more on the line)gpu/vk/vk_replay.cpp:2665: With Threaded rendering on, the job loop still calls update_blocked. (more on the line)gpu/vk/vk_replay.cpp:2579: The detached thread can stay parked on this global condition variable for the life of the process. (more on the line)tests/test_sync_watch_event.py:69: The fixture replaces both production wakeup producers with copies and checks the originals by matching text. (more on the line)tests/test_sync_watch_event.py:133: The guest-memory case passes if the event path's Sleep(1) is deleted and the watcher spins. (more on the line)tests/test_sync_watch_event.py:33: The suite passes an explicit compiler, bypassing the script's LONGSHOT_MINGW fallback. (more on the line)tests/test_sync_watch_event.py:158: The watcher fixture inherits RX_VK_HOLD_GPU_MS through the real sync_watch getenv call. (more on the line)gpu/vk/switches.def:679: The table says polling only happens on a guest-memory wait or past the hold deadline. (more on the line)e6196f0fand the merge base is75317e1f. Clear that box until the branch is updated, then verify the final head and record the results.For the author (1) (a run, a measurement or a decision)
Checks: Still running:
CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),PR description / Description follows the template (pull_request_target),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven. The commit cites 1.4% of one CPU core for the old watcher in a Hyrule Field profile on the experiment branch. There are no matched in-game runs of
492641ecagainst75317e1f, no wake counts for this port, and no frame-time results.Other systems: The changed watcher affects synchronous replay, including CPU-event capture, guest-memory event polling, per-queue fence waits, and the ordering of work before a present. Those paths are shared across Windows, Linux and Steam Deck, regardless of GPU vendor. Threaded rendering keeps its job loop and hold policy but also calls the changed update_blocked. The diff touches no shader, GPU allocation, save, audio, input, or renderer API code.
Switches: RX_VK_SYNC_WATCH_EVENT is a read-once developer kill switch, default on when unset, with a sorted table entry. Off keeps the polling cadence but still incurs new bookkeeping. The default has no supporting game measurements and currently exposes the GPU-wait regression. RX_VK_HOLD_MS keeps its 50 ms default and now also sets the timed wake deadline. RX_VK_HOLD_GPU_MS keeps its 1000 ms default, but the watcher can now miss the wait history its grace rule uses.
Reviewed:
492641ec96against75317e1f31: Read the full diff, commit, new test, and all changed regions of vk_replay.cpp and switches.def. Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, developer standards, hold_gpu.h, and the relevant architecture and switch rules. Traced replay callers, event publication, fence waits, renderer lifetime, and test discovery. The rest of the replay file and switch table were searched for callers, readers, and related rules. No review checklist, separate run artifacts, generated files, or translations were present in the change.; nothing run (a code review).Checked and fine:
Testing: Read-only review. The author reports that the new fixture passed on Windows with LLVM-MinGW. The changed-check batch reported 120 passes, 27 named failures also seen on base, and one timeout that passed on retry. Linux, the full suite, and in-game results are outstanding. The supplied CI checks were still running.
Overlaps and merge order: Already merged #1977 and #1870 touch other regions of vk_replay.cpp and switches.def on current main. No overlapping open pull request was visible in the checkout or supplied discussion.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -676,6 +676,7 @@ SW(RX_VK_STREAM_MB, tune, int, "1 MB", "Size in MB of each recording's first strSW(RX_VK_SUBMIT_DELAY_US, diag, int, "0: no delay", "Spins for that many microseconds before each submit is replayed, to test for data arriving late.")SW(RX_VK_SYNC_PIPELINES, attempt, bool, "off (a new pipeline is queued for a worker)", "1 builds each new graphics pipeline in the draw that needs it, not on a worker.")SW(RX_VK_SYNC_SHADERS, attempt, bool, "off: graphics programs are translated on a worker, compute where they are", "=1 translates every graphics program at its draw, on the drawing thread.")SW(RX_VK_SYNC_WATCH_EVENT, kill, bool, "on (unset): with the replay on the submitting thread, the stopped queues' watch sleeps until a queue stops, then until the CPU signals an event or RX_VK_HOLD_MS passes (every millisecond only at a wait on guest memory's value or past RX_VK_HOLD_MS)", "0, off, false or no: the stopped-queue watch wakes every millisecond, as before. Read once, at first use (the log says which).")L10 (minor): The table says polling only happens on a guest-memory wait or past the hold deadline. A failed sync_order_mu try-lock also retries after Sleep(1). Describe that fallback and include an empty value among the spellings that disable the switch.
@ -2568,0 +2576,4 @@return v;}std::mutex order_watch_mu;std::condition_variable order_watch_cv;L5 (minor): The detached thread can stay parked on this global condition variable for the life of the process. If renderer globals are torn down through C++ exit while the watcher is parked, destruction can encounter a live waiter. Normal Linux ExitProcess bypasses that teardown, and the fixture does too. Give the wait state process lifetime, following the recorder precedent, or stop and join the watcher before renderer teardown.
@ -2639,1 +2662,4 @@blocked_n = n;// (longshot, RX_VK_SYNC_WATCH_EVENT) the watch is told of what it cannot see by itself: a queue stopped where none was (its// hold_ms starts), or one now stopped at an event whose value is guest memory's (looked at every millisecond from now on)if (n && (!was || (!mem_was && ev_wait_in_memory.load(std::memory_order_acquire)))) sync_watch_kick();L4 (minor): With Threaded rendering on, the job loop still calls update_blocked. A first queue stop takes order_watch_mu and notifies a watcher that was never started. The switch-off path also publishes the extra timestamp. Gate watcher-only bookkeeping and notification on an active event watcher so threaded rendering and the polling baseline avoid that work.
@ -2975,0 +3019,4 @@}std::chrono::steady_clock::time_point watch_hold_until() { // (a millisecond past hold_ms: the rule's `held > hold_ms` is then true)const auto since = std::chrono::duration_cast<std::chrono::steady_clock::duration>(std::chrono::nanoseconds(blocked_since_ns.load()));return std::chrono::steady_clock::time_point(since) + std::chrono::milliseconds(hold_ms() + 1);L2: RX_VK_HOLD_MS=2147483647 fits the existing int setting, but hold_ms() + 1 overflows before chrono sees it. The switch-off loop can also reach this helper at line 3083. Construct the duration before adding the increment, and skip event deadline calculations when the switch is off.
@ -2979,0 +3031,4 @@std::unique_lock wk(order_watch_mu);order_watch_cv.wait(wk, [] { return blocked_n.load() != 0; });if (!poll && !watch_look_due())order_watch_cv.wait_until(wk, watch_hold_until(), [] { return !blocked_n.load() || watch_look_due(); });L3 (minor): The timed wait captures one deadline and its predicate can swallow the kick for a new stopped interval. If A clears at 40 ms and B stops at 41 ms, the watcher can wake and run schedule at A's 51 ms deadline even though B's deadline is 92 ms. Include the stopped interval's generation in the wait predicate and recompute the deadline when it changes.
@ -0,0 +30,4 @@RUN_ALL_ARGS = ["--cxx", "{cxx}"]ROOT = Path(__file__).resolve().parents[1]L8 (minor): The suite passes an explicit compiler, bypassing the script's LONGSHOT_MINGW fallback. A Windows setup with llvm-mingw configured by directory and no C++ compiler on PATH is marked missing by run_all.py before this test starts. The standalone fallback can also pick the Windows cross-compiler on Linux. Use the same native compiler discovery in both entry points, preserving explicit --cxx and CXX overrides.
@ -0,0 +66,4 @@void burst_end() {}void drain(int) { drains++; blocked_n = 0; ev_wait_in_memory = false; }/* KICK */void update_blocked() {} // (the watch's own calls: the stand-ins above set the state)L6: The fixture replaces both production wakeup producers with copies and checks the originals by matching text. For example, guarding the production CPU kick with an always-false condition above the matched lines would still pass. Compile the real update_blocked and synchronous event_signal_queued with minimal stand-ins. Exercise a signal racing the first stop and a transition to a guest-memory wait while another queue is already stopped.
@ -0,0 +130,4 @@const int at = looks;stop(true);std::this_thread::sleep_for(std::chrono::milliseconds(300));CHECK(looks - at >= 5 && drains == 1); // (a millisecond's sleep is up to a timer tick on Windows)L7 (minor): The guest-memory case passes if the event path's Sleep(1) is deleted and the watcher spins. Assert that successive guest-memory looks are separated by the polling wait. Synchronize with the watcher entering its idle and timed waits so a delayed thread cannot make those checks pass without exercising the intended paths.
@ -0,0 +155,4 @@src.write_text(source, encoding="utf-8")subprocess.run([cxx, "-std=c++20", "-O1", "-Wall", "-Wextra", "-Werror", "-Wno-unused-function", "-pthread","-I", str(ROOT / "gpu" / "vk"), str(src), "-o", str(exe)], check=True)r = subprocess.run([str(exe)], text=True, capture_output=True, timeout=60)L9 (minor): The watcher fixture inherits RX_VK_HOLD_GPU_MS through the real sync_watch getenv call. A developer's override changes the force bound the fixture exercises. Give the fixture an explicit environment with that switch cleared or pinned for each case.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.