feat(dev): an A/B within one run: HOS_AB alternates a switch every few seconds, scripts/ab_summary.py compares the sides #1661
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1661
Loading…
Reference in a new issue
No description provided.
Delete branch "GummiNaomi/Longshot:perf/in-run-ab"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Windows 11, RTX 4070 SUPER, Ryzen 9 9950X3D (SMT off) · Needs: a build of this head; Linux, any GPU
TL;DR
Two runs of the same build on the same PC can differ by a few fps, more than the 2 to 5 % a CPU change is often worth, so a start-and-compare A/B often can't tell. This PR adds an A/B inside one run: a switch flips every few seconds and each frame is counted for the side it ran on, so whatever drifts slowly falls on both sides alike. A small script gives each side's frame times and the difference with a 95 % interval. It measured every 18 Nov change I'm sending as a PR, resolving 0.3 to 0.5 ms that two-run comparisons could not.
Breakdown
HOS_AB=<switch>=<value>[,<seconds>[,<warm-up s>]]alternates a wired switch within one run; the frame trace gets anab_sidecolumn;scripts/ab_summary.pycompares the sides.HOS_ABa wired switch's reader costs one relaxed load and reads its value as before.HOS_ABis set; a value it refuses is logged and the run goes on without an A/B.Why
Measuring a CPU-side change on one PC means comparing runs, and on a PC of 4 cores the same build, the same switches and the same spot gave 58.7 to 64.5 fps from one start to the next. Two or four runs a side cannot resolve the 0.3 to 1 ms (2 to 6 % of a frame) a typical CPU change is worth, so a real gain looks like noise and noise looks like a gain. Measuring both sides inside one run removes most of that spread: the PC's other work, its clocks and the scene drift slower than a few seconds, so they land on both sides alike.
The idea follows the A/B blocks of dkb-pc-recomp's runtime (
perf.c, TOTK_AB); the code here is Longshot's own.What changed
A switch that is read at each use, instead of once at the start, can be flipped during a run.
HOS_ABnames one such switch and the value to try; after the warm-up the switch alternates every period between its normal value (side 0, first) and the A/B's value (side 1). The host counts each present for its side and logs the sums at the close; the frame trace gets each frame's side; a script turns a trace into a comparison.host/hos_ab.c(new): the plan's parser (HOS_AB=<switch>=<value>[,<seconds>[,<warm-up seconds>]], 3 s and 20 s by default; a bad value is refused with a log line), a thread that flips the side on the host's clock,hos_ab_value(name, normal)inhost/hos.h(one relaxed load with no A/B; during one, a small table of the names seen, looked up alike on both sides),hos_ab_frame(a frame's side, or a transition: a flip inside it or less than 0.3 s before its end), the presents' sums by side and theirab:log line athos_close_finish.hos_ab_startinhos_main.cbefore the game's threads;hos_ab_presentinrx_nvn_present; the old stubhos_abinmkw_glue.cis gone (nothing called it).MkwVkHost.ab_frame(MKWVK_API_VERSION 33 -> 34) and the frame trace's last columnab_side(gpu/vk/vk_replay.cppstatus_tick; -1 from an older host);scripts/profile_report.pyleaves the column out of its averages.scripts/ab_summary.py(new): per side the frames, mean, median, 99th percentile, the share of frames of 16.7 ms or more and fps; side 1 less side 0 with a 95 % interval from a bootstrap of whole periods. It leaves out transitions, the warm-up and a last period the run's end cut short, and trims to whole pairs of periods so a steady drift moves both sides alike. Several traces give each, then all together.HOS_COLLECT_WIDE(host/rx_backend.ccollect_in): the write marks' walk 256 marks a step or 32. The pages a walk finds, their order and the marks it clears are the same either way, so it may change between two walks. The switches PRs wire later add their own rows to the test'sWIRED.docs/development.md, "An A/B within one run" under the profiling section, and the frame trace's column; theHOS_ABrow ingpu/vk/switches.def.tests/test_hos_ab.py(new): the realhos_ab.cover a stubhos.handwindows.hwith a fake clock: the parser (good and 21 bad plans), the schedule every 10 ms over a minute,hos_ab_frame's transitions, the cheap path (no name looked at outside an A/B), 40 names sharing the table's slots, the sums and their log line, the wired switch flipping with the side both ways round;tests/test_collect_wide.py's 400 rounds of random marks with the switch flipping at every walk;scripts/ab_summary.pyon made-up traces (known means, transitions, warm-up, a cut-short period, two files, a steady drift); 7 broken variants ofhos_ab.cand 5 of the script, each caught.tests/test_collect_wide.pygets ahos_ab_valuestand-in;tests/test_frame_trace_waits.pythe new column;tests/crash_record_test.candtests/test_windows_stop.py(its close chain) a stand-in forhos_ab_end.Measurements
Verdict: nothing to measure for the tool itself: without
HOS_ABa wired reader adds one relaxed load, and it changes nothing a player sees. What it measured is in the PRs that use it; two real runs are below as an example of its output.Hardware: Windows 11, RTX 4070 SUPER (NVIDIA 617.42), Ryzen 9 9950X3D with SMT off, 32 GB
Scene: Castle Town's market (child), standing after a debug teleport from the
kokiri-startsave; output 2560x1440, uncapped (HOS_FPS_MAX=1000), no window (HOS_HEADLESS=1), the game kept to 4 cores (affinity 0xF00,HOS_CACHE_CORES=0)Method:
HOS_AB=HOS_OS_YIELD=0,3,130(3 s periods after 130 s),RX_VK_FRAME_TRACE=1, about 160 s standing;scripts/ab_summary.pyon the traceHow to reproduce: the same, with any wired switch; the other environment of these runs was
RX_VK_ASTC_BC_ALL=1 RX_VK_IMAGE_BLOCK_MB=32 RX_VK_LOAD=create RX_VK_BC_LAYERS=8 RX_VK_LATE_FIX=0 HOS_THREAD_TIMES=5Baseline: both sides come from one run of one build: main @
54a42715(93 commits before this PR's base) with this tool and the 18 Nov changes, before they were rebased onto main @10de79fbas separate PRs. Side 0 is the switch's normal value, side 1 the A/B's.HOS_OS_YIELDnormal): build @54a42715HOS_OS_YIELD=0): the same run @54a42715Flags and settings
One switch added, for testing only; one existing switch may now be flipped by it.
HOS_AB<switch>=<value>[,<seconds>[,<warm-up s>]]alternates a wired switchHOS_COLLECT_WIDEHOS_ABmay now flip itTesting
Tested:
python -I -B tests/test_hos_ab.py --cc clangpasses on this head (Windows 11, emsdk's clang). Every check undertests/that reads a changed file (229 of them, on this head) passes or fails as on main on this PC (its MSVC headers under-Werror, a missing Tooling checkout, linker setup): 57 failed with the same messages on main @da91e770before the rebase, and the 3 new since main's last commits compile only files this PR does not change. In the game: this tool, at 93 commits behind this base, ran the A/Bs ofHOS_OOT_BARRIER_WAIT,HOS_OS_YIELD,HOS_CA_NATIVE,HOS_PHYS_NATIVE,HOS_CLOTH_NATIVEandHOS_FASTFP_SUBNORMALon 4 and 16 cores in Castle Town's market (18 Nov): each logged itsab:line at the close, andab_summary.pyread each trace.Not tested: a build of this exact head (the game build was not run for this PR: the PC was busy with timed runs); Linux (the host code is plain C with a thread and a sleep, as the others).
Must be tested:
HOS_AB=HOS_COLLECT_WIDE=0,3,30 RX_VK_FRAME_TRACE=1for two minutes, expecting the log'sab: HOS_COLLECT_WIDE=0 against its normal value ...line at the close andpython -I scripts/ab_summary.py <game>/logs/longshot_frames.csvto print both sides (by hand: me)Risks
What could break: nothing in a run without
HOS_AB. With it, a switch wired without care could misbehave when it changes between two reads; each wired switch's reader says why a flip is safe, and the test lists them.Turning it off: leave
HOS_ABunset (the default); reverting the commit removes it.Compatibility: MKWVK_API_VERSION 34: the renderer and the host must come from the same build, as for every API bump; the frame trace has one more column at its end. Saves, options and caches: nothing changes.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). (The changed files' checks on this PC; failures are main's own here.)third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.MKWVK_API_VERSIONbumped (gpu/vk/vk_api.h'sMkwVkHostorMkwVkRendererchanged).tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Possibly related, for context:
This landed on main with #1663, which was stacked on it and squash-merged both commits (main's
d10b62b3carries host/hos_ab.c, scripts/ab_summary.py and tests/test_hos_ab.py). Rebased onto main, this branch has no changes left, so closing it as merged that way. Thank you!Pull request closed