WIP: perf(vk): RX_VK_CAPTURE_SLOW_MS=<ms> logs a slow capture's parts (probe, off by default) #1667
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1667
Loading…
Reference in a new issue
No description provided.
Delete branch "Molier/Longshot:perf/capture-ready-probe"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: diag probe · Risk: low (off by default; additions only) · Tested on: serv (Linux renderer build,
RX_VK_ONLY=1, 0 failed;tests/test_capture_spare_reuse.pyfixtures) · Needs: a play session withRX_VK_CAPTURE_SLOW_MSset, against a live capture path, to confirm the log format and that the parts it reports add up sensiblyWhy
We measured (on our own stack, not this commit) that in heavy loads the submit-time Vulkan capture (
capture_make, copying every written guest page into a capture buffer) can run 50-290 ms on the game's own thread, and that most of that time is the memcpy itself. Without a probe we could not tell which part of the copy path (lock, scan, late-page wait, buffer, copy) a slow capture actually spent its time in, or whether its buffer was made ahead of time or on the game thread itself. We are contributing the probe (not a fix) first.Narrowed since the first push
This PR originally also added
RX_VK_CAP_READY=1, which touches a capture buffer's pages on the worker thread that made it ahead of time, to move first-touch page faults off the game thread. Our own later data (not from this commit) found that 112 of 116 slow captures in a measured battery made their buffer on the capturing (game) thread itself, not ahead of time on a worker — the pathRX_VK_CAP_READY's pretouch targets. Pretouching a worker-made buffer cannot help a buffer that was never made ahead, so shippingRX_VK_CAP_READYas a fix candidate would have been an unsupported claim. We dropped it and narrowed this PR to theRX_VK_CAPTURE_SLOW_MSprobe alone, which is what the next investigation actually needs.How to reproduce
Seen on: main (
2f1c0c9c), any GPU; this specific commit's correctness is a host-side unit test, not a live capture.Settings and switches: unset (default): no new behaviour.
RX_VK_CAPTURE_SLOW_MS=<ms>: log a capture that takes<ms>ms or more.Steps:
python -I -B tests/test_capture_spare_reuse.py.Expected / actual: see Measurements.
What changed
gpu/vk/vk_memory.cpp:capture_slow_ms()readsRX_VK_CAPTURE_SLOW_MSonce (throughlongshot_env.h'senv_int, clamped to> 0); a capture whose wall time reaches that threshold gets one log line (parts: lock, scan, late-page wait, buffer, copy; page count; the thread's minor-fault delta fromgetrusage(RUSAGE_THREAD, ...)on Linux; whether its buffer was made on this thread).gpu/vk/switches.defanddocs/development.md: one row each forRX_VK_CAPTURE_SLOW_MS.tests/test_capture_spare_reuse.py: the existing fixtures' stub forcap_made_on_threadis nowthread_local, matching the real counter.Review fixes (this push)
cap_made_on_threadwas a process-globalstd::atomic, so when two threads capture at once (main and Presentation Thread both do) one thread's buffer could be credited to the other's slow-capture log line. It is nowthread_local.thread_minflt()guarded itsgetrusage(RUSAGE_THREAD, ...)call with#ifndef _WIN32, a Windows-only exclusion, for a Linux-only constant (RUSAGE_THREADdoes not exist on every non-Windows platform). Changed to#ifdef RUSAGE_THREAD.capture_slow_ms()readRX_VK_CAPTURE_SLOW_MSwith a rawgetenv+atoi(not throughlongshot_env.h's one rule) and did not clamp a negative value, which would have logged every capture. Now reads throughenv_intand clamps to> 0.RX_VK_CAP_READY,cap_ready_on(),cap_pretouch()andtests/test_capture_ready.pyentirely (see "Narrowed since the first push" above), and the switches.def/docs rows forRX_VK_CAP_READY.Measurements
Verdict: the probe is a diagnostic only; it is not presented as a fix. The counter and fault-delta fixes above are shown correct by inspection (thread_local scoping, the RUSAGE_THREAD guard, the env_int clamp) and by the renderer build (0 failed) and the existing fixture passing; no frame-time or capture-time number from this exact commit is claimed, since the probe's job is to produce numbers for the next PR, not to change behaviour itself.
tests/test_capture_spare_reuse.pypasses (no output on success; its asserts covercap_buf_get,cap_spare_makeand the capture-buffer worker path with the new thread_local counter in place).Not measured
RX_VK_CAPTURE_SLOW_MSprobe's own log format has not been exercised against a live capture outside our local (non-upstream) stack.RX_VK_CAP_READYas an actual fix for the 112-of-116-on-the-game-thread pattern: not part of this PR.RUSAGE_THREAD(documented, not a crash).Possibly related, for context:
765f921623a871f3ff73WIP: perf(vk): RX_VK_CAP_READY pretouch + RX_VK_CAPTURE_SLOW_MS probe for the submit-time capture copyto WIP: perf(vk): RX_VK_CAPTURE_SLOW_MS=<ms> logs a slow capture's parts (probe, off by default)Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. This PR's head
a871f3ffand #1666's4ec23e96, both cherry-picked onto their common base2f1c0c9c, in one test build (longshot abc-probes (source 166c4da4)), withRX_VK_CAPTURE_SLOW_MS=50. No errors, hangs or device losses in any run.The probe (this PR's Needs item): it works against a live capture path, and its parts add up: lock, scan, late pages, buffer and copy are within 0.3 ms of the total on all 24 lines. Lock, scan and late pages were 0.5 ms or less on every line; the time is the copy and the buffer.
copy_split, 4 from 12 CPU threads), so that's about 200 MB/s a thread. Neither end is slow memory on paper: the capture buffer is cached host memory (create_buffer_cached:VMA_MEMORY_USAGE_AUTO_PREFER_HOST, random access), and the game's memory is imported in place as dma-bufs. Minor faults on the capturing thread are low (11, 4, 28), but the helpers do most of the copying and their faults and kernel time aren't in the line. Counting those (or the helpers' share of the pages and their time) would show where the rest goes.cap_prewarm_indexis -1 above 192 MB), and the field's two biggest captures needed 243 and 252 MB, so their buffer is always made on the capturing thread (527 ms for the 243 MB one). Those numbers are on #1599, which works on the spare classes.We'd like to look into the copy rate next, unless you're on it already: say if so, and we'll send you runs instead.
Not compared against main: the probe changes nothing while it's off, so these are its readings, not a measurement of this PR.
Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on).
HOS_AWAY_LOG_MS=50(#1666) andHOS_PROFILE=1in every run.Method: a scripted route through the control bridge from the repository save
kokiri-start: the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps along Link's facing between 20 m and 320 m out with a 180° camera turn every 10 s (the trigger in #1720). A warm-up run first, then 2 runs; Link's hearts protected (HOS_OOT_PROTECT_HEARTS=1).Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. The same test build as above (this PR's
a871f3ffand #1666's4ec23e96on2f1c0c9c,longshot abc-probes (source 166c4da4)), withRX_VK_CAPTURE_SLOW_MS=50. No errors, hangs or device losses in any run.A follow-up on the copy rate, with your probe. The copy is bound by page faults, and non-temporal stores (
RX_VK_CAPTURE_NT=1) take most of them away. In five diagnostic runs, one capture thread copied as fast as five (0.97 against 1.02-1.07 GB/s), with 0.85 minor faults per copied 4 KB page. Then an interleaved A/B ofRX_VK_CAPTURE_NT, 3 runs a side, in the order off, on, on, off, off, on after a warm-up:RX_VK_CAPTURE_NT=0, the defaultRX_VK_CAPTURE_NT=1RX_VK_CAPTURE_NTis opt-in because of the RTX desktop runs in the comment atcapture_nt(), so it says nothing yet about a default elsewhere.2f1c0c9c, before #1121 (merged today), which replays jobs as they come instead of whole frames; the copy itself shouldn't change with it, but the frame times may.Every run is on record in #1749 (
docs/tested/, run01 to run11).Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on).
HOS_AWAY_LOG_MS=50(#1666) andHOS_PROFILE=1in every run; the copy helpers' faults and CPU time from/procevery 200 ms.Method: the same scripted route as above: from the repository save
kokiri-start, the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps between 20 m and 320 m out with a 180° camera turn every 10 s (the trigger in #1720). A warm-up run first, then 3 runs a side, interleaved; Link's hearts protected (HOS_OOT_PROTECT_HEARTS=1).Tested in game on Linux: Fedora 45 (kernel 7.3.4), AMD Radeon RX 9070 XT (RADV GFX1201, Mesa 26.2.7), Ryzen 5 5600X (12 logical processors), 32 GB RAM. A new test build on current main:
819a5967(after #1121) with this PR'sa871f3ffand #1666's4ec23e96cherry-picked (longshot abc-probes-main (source 80057f9b)), withRX_VK_CAPTURE_SLOW_MS=50. No errors, hangs or device losses in any run.RX_VK_CAPTURE_NT=1holds on main after #1121, and #1121's gate makes no clear difference. A 2x2 ofRX_VK_FRAME_GATE(0 is main's default since #1121) andRX_VK_CAPTURE_NT, 2 runs a cell, in the order A B C D D C B A after a warm-up:Every run is on record in #1749 (
docs/tested/, run12 to run19).Settings: the player's own (Custom preset), 2560x1440 fullscreen, HDR10, AMD FSR 4 (1920x1080 to 2560x1440), V-Sync Auto (mailbox), no frame cap, frame generation off, FPS counter off; one kept options file for every run. Threaded rendering Auto (on).
HOS_AWAY_LOG_MS=50(#1666) andHOS_PROFILE=1in every run.Method: the same scripted route as above: from the repository save
kokiri-start, the debug teleport to the Market, the clock at 12:00, out of the gate into Hyrule Field, then 120 s of laps between 20 m and 320 m out with a 180° camera turn every 10 s. A warm-up run first, then 2 runs a cell, interleaved; Link's hearts protected (HOS_OOT_PROTECT_HEARTS=1).a871f3ff7368c28bf83cRebased onto
main@89c3d21e. Only the docs row moved (#1802);switches.def,vk_memory.cppand the test merged cleanly with #1763/#1848.tests/test_capture_spare_reuse.pypasses and the renderer builds.Staying WIP: I have a broader capture diagnostic (per-part CPU time and page faults, buffer memory type, per-capture wall time) that answers the same question, and I'd rather fold this threshold log into it than add two capture probes to
switches.def. I'll update this PR when that is ready, or close it if #1599's shape makes it moot.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.