feat(dev): an A/B within one run: HOS_AB alternates a switch every few seconds, scripts/ab_summary.py compares the sides #1661

Closed
GummiNaomi wants to merge 2 commits from GummiNaomi/Longshot:perf/in-run-ab into main
Member

Type: feat · Risk: low · Tested on: Windows 11, RTX 4070 SUPER, Ryzen 9 9950X3D (SMT off) · Needs: a build of this head; Linux, any GPU

TL;DR

Two runs of the same build on the same PC can differ by a few fps, more than the 2 to 5 % a CPU change is often worth, so a start-and-compare A/B often can't tell. This PR adds an A/B inside one run: a switch flips every few seconds and each frame is counted for the side it ran on, so whatever drifts slowly falls on both sides alike. A small script gives each side's frame times and the difference with a 95 % interval. It measured every 18 Nov change I'm sending as a PR, resolving 0.3 to 0.5 ms that two-run comparisons could not.

Breakdown

  • Who it helps: developers measuring small CPU changes, and reviewers who want to repeat a PR's numbers.
  • What was wrong: run-to-run spread (58.7 to 64.5 fps for the same build on 4 cores) hid differences of 2 to 6 % of a frame.
  • What changes: HOS_AB=<switch>=<value>[,<seconds>[,<warm-up s>]] alternates a wired switch within one run; the frame trace gets an ab_side column; scripts/ab_summary.py compares the sides.
  • What does not change: nothing for a player: without HOS_AB a wired switch's reader costs one relaxed load and reads its value as before.
  • How it was checked: a game-free test with broken variants of both the C and the script; dozens of in-run A/Bs in the game on a tree 93 commits behind this main.
  • What is still open: a build and one A/B run of this exact head; Linux.
  • Risk: low: off unless HOS_AB is set; a value it refuses is logged and the run goes on without an A/B.

Why

Measuring a CPU-side change on one PC means comparing runs, and on a PC of 4 cores the same build, the same switches and the same spot gave 58.7 to 64.5 fps from one start to the next. Two or four runs a side cannot resolve the 0.3 to 1 ms (2 to 6 % of a frame) a typical CPU change is worth, so a real gain looks like noise and noise looks like a gain. Measuring both sides inside one run removes most of that spread: the PC's other work, its clocks and the scene drift slower than a few seconds, so they land on both sides alike.

The idea follows the A/B blocks of dkb-pc-recomp's runtime (perf.c, TOTK_AB); the code here is Longshot's own.

What changed

A switch that is read at each use, instead of once at the start, can be flipped during a run. HOS_AB names one such switch and the value to try; after the warm-up the switch alternates every period between its normal value (side 0, first) and the A/B's value (side 1). The host counts each present for its side and logs the sums at the close; the frame trace gets each frame's side; a script turns a trace into a comparison.

  • host: host/hos_ab.c (new): the plan's parser (HOS_AB=<switch>=<value>[,<seconds>[,<warm-up seconds>]], 3 s and 20 s by default; a bad value is refused with a log line), a thread that flips the side on the host's clock, hos_ab_value(name, normal) in host/hos.h (one relaxed load with no A/B; during one, a small table of the names seen, looked up alike on both sides), hos_ab_frame (a frame's side, or a transition: a flip inside it or less than 0.3 s before its end), the presents' sums by side and their ab: log line at hos_close_finish. hos_ab_start in hos_main.c before the game's threads; hos_ab_present in rx_nvn_present; the old stub hos_ab in mkw_glue.c is gone (nothing called it).
  • renderer: MkwVkHost.ab_frame (MKWVK_API_VERSION 33 -> 34) and the frame trace's last column ab_side (gpu/vk/vk_replay.cpp status_tick; -1 from an older host); scripts/profile_report.py leaves the column out of its averages.
  • script: scripts/ab_summary.py (new): per side the frames, mean, median, 99th percentile, the share of frames of 16.7 ms or more and fps; side 1 less side 0 with a 95 % interval from a bootstrap of whole periods. It leaves out transitions, the warm-up and a last period the run's end cut short, and trims to whole pairs of periods so a steady drift moves both sides alike. Several traces give each, then all together.
  • the first wired switch: HOS_COLLECT_WIDE (host/rx_backend.c collect_in): the write marks' walk 256 marks a step or 32. The pages a walk finds, their order and the marks it clears are the same either way, so it may change between two walks. The switches PRs wire later add their own rows to the test's WIRED.
  • docs: docs/development.md, "An A/B within one run" under the profiling section, and the frame trace's column; the HOS_AB row in gpu/vk/switches.def.
  • tests: tests/test_hos_ab.py (new): the real hos_ab.c over a stub hos.h and windows.h with a fake clock: the parser (good and 21 bad plans), the schedule every 10 ms over a minute, hos_ab_frame's transitions, the cheap path (no name looked at outside an A/B), 40 names sharing the table's slots, the sums and their log line, the wired switch flipping with the side both ways round; tests/test_collect_wide.py's 400 rounds of random marks with the switch flipping at every walk; scripts/ab_summary.py on made-up traces (known means, transitions, warm-up, a cut-short period, two files, a steady drift); 7 broken variants of hos_ab.c and 5 of the script, each caught. tests/test_collect_wide.py gets a hos_ab_value stand-in; tests/test_frame_trace_waits.py the new column; tests/crash_record_test.c and tests/test_windows_stop.py (its close chain) a stand-in for hos_ab_end.

Measurements

Verdict: nothing to measure for the tool itself: without HOS_AB a wired reader adds one relaxed load, and it changes nothing a player sees. What it measured is in the PRs that use it; two real runs are below as an example of its output.

Hardware: Windows 11, RTX 4070 SUPER (NVIDIA 617.42), Ryzen 9 9950X3D with SMT off, 32 GB
Scene: Castle Town's market (child), standing after a debug teleport from the kokiri-start save; output 2560x1440, uncapped (HOS_FPS_MAX=1000), no window (HOS_HEADLESS=1), the game kept to 4 cores (affinity 0xF00, HOS_CACHE_CORES=0)
Method: HOS_AB=HOS_OS_YIELD=0,3,130 (3 s periods after 130 s), RX_VK_FRAME_TRACE=1, about 160 s standing; scripts/ab_summary.py on the trace
How to reproduce: the same, with any wired switch; the other environment of these runs was RX_VK_ASTC_BC_ALL=1 RX_VK_IMAGE_BLOCK_MB=32 RX_VK_LOAD=create RX_VK_BC_LAYERS=8 RX_VK_LATE_FIX=0 HOS_THREAD_TIMES=5

Baseline: both sides come from one run of one build: main @ 54a42715 (93 commits before this PR's base) with this tool and the 18 Nov changes, before they were rebased onto main @ 10de79fb as separate PRs. Side 0 is the switch's normal value, side 1 the A/B's.

Before (side 0, HOS_OS_YIELD normal): build @ 54a42715 After (side 1, HOS_OS_YIELD=0): the same run @ 54a42715
mean frame time (run a / run b) 15.704 / 16.005 ms 15.367 / 15.741 ms
95 % interval of the difference -0.471 to -0.205 / -0.462 to -0.077 ms
ab: HOS_OS_YIELD=0 against its normal value, 3.0 s periods after 130 s: side 0 (normal) 2922 frames, 15.704 ms a frame (63.7 fps); side 1 (the A/B's value) 2813 frames, 15.367 ms a frame (65.1 fps); side 1 -2.14 % frame time; 636 frames in transitions left out (the host's presents; scripts/ab_summary.py on the frame trace for medians and a confidence interval)

frames.csv: 33 periods (17 of side 0, 16 of side 1); left out: 0 period(s) at the end, 636 transition frames, 13095 frames outside the A/B
  side       frames  mean ms   median      p99  >=16.7ms     fps
  0 normal     2922   15.704   15.400   21.508     22.0%    63.7
  1 A/B        2813   15.367   15.080   20.983     16.5%    65.1
  side 1 - side 0: mean -0.337 ms (-2.14 %, fps +2.19 %), median -0.320 ms, p99 -0.525 ms, >=16.7 ms -5.5 points
  95 % interval of the mean's difference: -0.471 to -0.205 ms (-3.00 to -1.31 %): side 1 is faster

Flags and settings

One switch added, for testing only; one existing switch may now be flipped by it.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
HOS_AB an A/B within one run none unset: no A/B <switch>=<value>[,<seconds>[,<warm-up s>]] alternates a wired switch helps measuring; nothing in play off: for measuring only
HOS_COLLECT_WIDE the write marks' walk, 256 marks a step or 32 read once read at each walk; the same default (on) none: HOS_AB may now flip it the pages found are the same either way unchanged: on

Testing

Tested: python -I -B tests/test_hos_ab.py --cc clang passes on this head (Windows 11, emsdk's clang). Every check under tests/ that reads a changed file (229 of them, on this head) passes or fails as on main on this PC (its MSVC headers under -Werror, a missing Tooling checkout, linker setup): 57 failed with the same messages on main @ da91e770 before the rebase, and the 3 new since main's last commits compile only files this PR does not change. In the game: this tool, at 93 commits behind this base, ran the A/Bs of HOS_OOT_BARRIER_WAIT, HOS_OS_YIELD, HOS_CA_NATIVE, HOS_PHYS_NATIVE, HOS_CLOTH_NATIVE and HOS_FASTFP_SUBNORMAL on 4 and 16 cores in Castle Town's market (18 Nov): each logged its ab: line at the close, and ab_summary.py read each trace.

Not tested: a build of this exact head (the game build was not run for this PR: the PC was busy with timed runs); Linux (the host code is plain C with a thread and a sleep, as the others).

Must be tested:

  • Windows, any GPU: build this head, run with HOS_AB=HOS_COLLECT_WIDE=0,3,30 RX_VK_FRAME_TRACE=1 for two minutes, expecting the log's ab: HOS_COLLECT_WIDE=0 against its normal value ... line at the close and python -I scripts/ab_summary.py <game>/logs/longshot_frames.csv to print both sides (by hand: me)
  • Linux, any GPU: the same run, expecting the same line (Test Farm: @Beanstalk linux, a 2-minute run with HOS_AB=HOS_COLLECT_WIDE=0,3,30 and RX_VK_FRAME_TRACE=1, then the log's ab: line)

Risks

What could break: nothing in a run without HOS_AB. With it, a switch wired without care could misbehave when it changes between two reads; each wired switch's reader says why a flip is safe, and the test lists them.

Turning it off: leave HOS_AB unset (the default); reverting the commit removes it.

Compatibility: MKWVK_API_VERSION 34: the renderer and the host must come from the same build, as for every API bump; the frame trace has one more column at its end. Saves, options and caches: nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. (Not yet on this head: it ran in the game 93 commits back; a build of this head first.)
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). (The changed files' checks on this PC; failures are main's own here.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed).
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Not built on Linux yet: Must be tested.)
  • Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** low · **Tested on:** Windows 11, RTX 4070 SUPER, Ryzen 9 9950X3D (SMT off) · **Needs:** a build of this head; Linux, any GPU ## TL;DR Two runs of the same build on the same PC can differ by a few fps, more than the 2 to 5 % a CPU change is often worth, so a start-and-compare A/B often can't tell. This PR adds an A/B inside one run: a switch flips every few seconds and each frame is counted for the side it ran on, so whatever drifts slowly falls on both sides alike. A small script gives each side's frame times and the difference with a 95 % interval. It measured every 18 Nov change I'm sending as a PR, resolving 0.3 to 0.5 ms that two-run comparisons could not. ## Breakdown - **Who it helps:** developers measuring small CPU changes, and reviewers who want to repeat a PR's numbers. - **What was wrong:** run-to-run spread (58.7 to 64.5 fps for the same build on 4 cores) hid differences of 2 to 6 % of a frame. - **What changes:** `HOS_AB=<switch>=<value>[,<seconds>[,<warm-up s>]]` alternates a wired switch within one run; the frame trace gets an `ab_side` column; `scripts/ab_summary.py` compares the sides. - **What does not change:** nothing for a player: without `HOS_AB` a wired switch's reader costs one relaxed load and reads its value as before. - **How it was checked:** a game-free test with broken variants of both the C and the script; dozens of in-run A/Bs in the game on a tree 93 commits behind this main. - **What is still open:** a build and one A/B run of this exact head; Linux. - **Risk:** low: off unless `HOS_AB` is set; a value it refuses is logged and the run goes on without an A/B. ## Why Measuring a CPU-side change on one PC means comparing runs, and on a PC of 4 cores the same build, the same switches and the same spot gave 58.7 to 64.5 fps from one start to the next. Two or four runs a side cannot resolve the 0.3 to 1 ms (2 to 6 % of a frame) a typical CPU change is worth, so a real gain looks like noise and noise looks like a gain. Measuring both sides inside one run removes most of that spread: the PC's other work, its clocks and the scene drift slower than a few seconds, so they land on both sides alike. The idea follows the A/B blocks of dkb-pc-recomp's runtime (`perf.c`, TOTK_AB); the code here is Longshot's own. ## What changed A switch that is read at each use, instead of once at the start, can be flipped during a run. `HOS_AB` names one such switch and the value to try; after the warm-up the switch alternates every period between its normal value (side 0, first) and the A/B's value (side 1). The host counts each present for its side and logs the sums at the close; the frame trace gets each frame's side; a script turns a trace into a comparison. - host: `host/hos_ab.c` (new): the plan's parser (`HOS_AB=<switch>=<value>[,<seconds>[,<warm-up seconds>]]`, 3 s and 20 s by default; a bad value is refused with a log line), a thread that flips the side on the host's clock, `hos_ab_value(name, normal)` in `host/hos.h` (one relaxed load with no A/B; during one, a small table of the names seen, looked up alike on both sides), `hos_ab_frame` (a frame's side, or a transition: a flip inside it or less than 0.3 s before its end), the presents' sums by side and their `ab:` log line at `hos_close_finish`. `hos_ab_start` in `hos_main.c` before the game's threads; `hos_ab_present` in `rx_nvn_present`; the old stub `hos_ab` in `mkw_glue.c` is gone (nothing called it). - renderer: `MkwVkHost.ab_frame` (MKWVK_API_VERSION 33 -> 34) and the frame trace's last column `ab_side` (`gpu/vk/vk_replay.cpp` `status_tick`; -1 from an older host); `scripts/profile_report.py` leaves the column out of its averages. - script: `scripts/ab_summary.py` (new): per side the frames, mean, median, 99th percentile, the share of frames of 16.7 ms or more and fps; side 1 less side 0 with a 95 % interval from a bootstrap of whole periods. It leaves out transitions, the warm-up and a last period the run's end cut short, and trims to whole pairs of periods so a steady drift moves both sides alike. Several traces give each, then all together. - the first wired switch: `HOS_COLLECT_WIDE` (`host/rx_backend.c` `collect_in`): the write marks' walk 256 marks a step or 32. The pages a walk finds, their order and the marks it clears are the same either way, so it may change between two walks. The switches PRs wire later add their own rows to the test's `WIRED`. - docs: `docs/development.md`, "An A/B within one run" under the profiling section, and the frame trace's column; the `HOS_AB` row in `gpu/vk/switches.def`. - tests: `tests/test_hos_ab.py` (new): the real `hos_ab.c` over a stub `hos.h` and `windows.h` with a fake clock: the parser (good and 21 bad plans), the schedule every 10 ms over a minute, `hos_ab_frame`'s transitions, the cheap path (no name looked at outside an A/B), 40 names sharing the table's slots, the sums and their log line, the wired switch flipping with the side both ways round; `tests/test_collect_wide.py`'s 400 rounds of random marks with the switch flipping at every walk; `scripts/ab_summary.py` on made-up traces (known means, transitions, warm-up, a cut-short period, two files, a steady drift); 7 broken variants of `hos_ab.c` and 5 of the script, each caught. `tests/test_collect_wide.py` gets a `hos_ab_value` stand-in; `tests/test_frame_trace_waits.py` the new column; `tests/crash_record_test.c` and `tests/test_windows_stop.py` (its close chain) a stand-in for `hos_ab_end`. ## Measurements **Verdict:** nothing to measure for the tool itself: without `HOS_AB` a wired reader adds one relaxed load, and it changes nothing a player sees. What it measured is in the PRs that use it; two real runs are below as an example of its output. Hardware: Windows 11, RTX 4070 SUPER (NVIDIA 617.42), Ryzen 9 9950X3D with SMT off, 32 GB Scene: Castle Town's market (child), standing after a debug teleport from the `kokiri-start` save; output 2560x1440, uncapped (`HOS_FPS_MAX=1000`), no window (`HOS_HEADLESS=1`), the game kept to 4 cores (affinity 0xF00, `HOS_CACHE_CORES=0`) Method: `HOS_AB=HOS_OS_YIELD=0,3,130` (3 s periods after 130 s), `RX_VK_FRAME_TRACE=1`, about 160 s standing; `scripts/ab_summary.py` on the trace How to reproduce: the same, with any wired switch; the other environment of these runs was `RX_VK_ASTC_BC_ALL=1 RX_VK_IMAGE_BLOCK_MB=32 RX_VK_LOAD=create RX_VK_BC_LAYERS=8 RX_VK_LATE_FIX=0 HOS_THREAD_TIMES=5` Baseline: both sides come from one run of one build: main @ 54a42715 (93 commits before this PR's base) with this tool and the 18 Nov changes, before they were rebased onto main @ 10de79fb as separate PRs. Side 0 is the switch's normal value, side 1 the A/B's. | | Before (side 0, `HOS_OS_YIELD` normal): build @ 54a42715 | After (side 1, `HOS_OS_YIELD=0`): the same run @ 54a42715 | |---|---|---| | mean frame time (run a / run b) | 15.704 / 16.005 ms | 15.367 / 15.741 ms | | 95 % interval of the difference | | -0.471 to -0.205 / -0.462 to -0.077 ms | ``` ab: HOS_OS_YIELD=0 against its normal value, 3.0 s periods after 130 s: side 0 (normal) 2922 frames, 15.704 ms a frame (63.7 fps); side 1 (the A/B's value) 2813 frames, 15.367 ms a frame (65.1 fps); side 1 -2.14 % frame time; 636 frames in transitions left out (the host's presents; scripts/ab_summary.py on the frame trace for medians and a confidence interval) frames.csv: 33 periods (17 of side 0, 16 of side 1); left out: 0 period(s) at the end, 636 transition frames, 13095 frames outside the A/B side frames mean ms median p99 >=16.7ms fps 0 normal 2922 15.704 15.400 21.508 22.0% 63.7 1 A/B 2813 15.367 15.080 20.983 16.5% 65.1 side 1 - side 0: mean -0.337 ms (-2.14 %, fps +2.19 %), median -0.320 ms, p99 -0.525 ms, >=16.7 ms -5.5 points 95 % interval of the mean's difference: -0.471 to -0.205 ms (-3.00 to -1.31 %): side 1 is faster ``` ### Flags and settings One switch added, for testing only; one existing switch may now be flipped by it. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `HOS_AB` | an A/B within one run | none | unset: no A/B | `<switch>=<value>[,<seconds>[,<warm-up s>]]` alternates a wired switch | helps measuring; nothing in play | off: for measuring only | | `HOS_COLLECT_WIDE` | the write marks' walk, 256 marks a step or 32 | read once | read at each walk; the same default (on) | none: `HOS_AB` may now flip it | the pages found are the same either way | unchanged: on | ## Testing **Tested:** `python -I -B tests/test_hos_ab.py --cc clang` passes on this head (Windows 11, emsdk's clang). Every check under `tests/` that reads a changed file (229 of them, on this head) passes or fails as on main on this PC (its MSVC headers under `-Werror`, a missing Tooling checkout, linker setup): 57 failed with the same messages on main @ da91e770 before the rebase, and the 3 new since main's last commits compile only files this PR does not change. In the game: this tool, at 93 commits behind this base, ran the A/Bs of `HOS_OOT_BARRIER_WAIT`, `HOS_OS_YIELD`, `HOS_CA_NATIVE`, `HOS_PHYS_NATIVE`, `HOS_CLOTH_NATIVE` and `HOS_FASTFP_SUBNORMAL` on 4 and 16 cores in Castle Town's market (18 Nov): each logged its `ab:` line at the close, and `ab_summary.py` read each trace. **Not tested:** a build of this exact head (the game build was not run for this PR: the PC was busy with timed runs); Linux (the host code is plain C with a thread and a sleep, as the others). **Must be tested:** - Windows, any GPU: build this head, run with `HOS_AB=HOS_COLLECT_WIDE=0,3,30 RX_VK_FRAME_TRACE=1` for two minutes, expecting the log's `ab: HOS_COLLECT_WIDE=0 against its normal value ...` line at the close and `python -I scripts/ab_summary.py <game>/logs/longshot_frames.csv` to print both sides (by hand: me) - Linux, any GPU: the same run, expecting the same line (Test Farm: @Beanstalk linux, a 2-minute run with HOS_AB=HOS_COLLECT_WIDE=0,3,30 and RX_VK_FRAME_TRACE=1, then the log's ab: line) ## Risks **What could break:** nothing in a run without `HOS_AB`. With it, a switch wired without care could misbehave when it changes between two reads; each wired switch's reader says why a flip is safe, and the test lists them. **Turning it off:** leave `HOS_AB` unset (the default); reverting the commit removes it. **Compatibility:** MKWVK_API_VERSION 34: the renderer and the host must come from the same build, as for every API bump; the frame trace has one more column at its end. Saves, options and caches: nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [ ] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. (Not yet on this head: it ran in the game 93 commits back; a build of this head first.) - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (The changed files' checks on this PC; failures are main's own here.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Not built on Linux yet: Must be tested.) - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Two runs of the same build differ by a few fps, more than the 2-5 % a change is often
worth. HOS_AB=<switch>=<value>[,<seconds>[,<warm-up seconds>]] measures both sides in
one run: after the warm-up the switch alternates every period between its normal value
and <value>, and each frame is counted for its side.

- host/hos_ab.c: the plan's parser, the flip thread, hos_ab_value (one relaxed load
  with no A/B), the host's sums by side logged at the close.
- MkwVkHost.ab_frame (MKWVK_API_VERSION 34): the frame trace's ab_side column.
- scripts/ab_summary.py: per side, the difference and a 95 % interval by a bootstrap of
  whole periods; transitions, the warm-up and a cut-short last period left out.
- HOS_COLLECT_WIDE is the first wired switch (the pages a walk finds are the same
  either way).
- tests/test_hos_ab.py: parser, schedule, frames, the cheap path, the sums, the wired
  switch flipping at every walk, ab_summary.py, broken variants of both.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
test: the Windows stop test stands in for hos_ab_end in the close chain
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Failing after 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 13s
Website / Website on Cloudflare (pull_request_target) Has been skipped
1d168e1ae5
hos_close_finish logs the A/B sums (hos_ab_end) after the close work and before
the log flush; the stop test compiles that chain over its own stand-ins.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • issue #1517 The renderer rewrite's frame-time gates need a noise floor per machine, and the test farm's gate has none — Noise-floor discussion for frame-time measurement; this adds paired in-run measurements.
  • PR #936 perf(host): the write marks' collection reads the marks in vector loads from any page, on every platform — Open overlapping work changes the same HOS_COLLECT_WIDE collector this PR wires for A/B testing.
  • PR #63 perf(host): the write marks' collection steps over 256 clear marks at a time — Merged PR introduced HOS_COLLECT_WIDE, the first switch measured by this tool.
Possibly related, for context: - issue #1517 The renderer rewrite's frame-time gates need a noise floor per machine, and the test farm's gate has none — Noise-floor discussion for frame-time measurement; this adds paired in-run measurements. - PR #936 perf(host): the write marks' collection reads the marks in vector loads from any page, on every platform — Open overlapping work changes the same HOS_COLLECT_WIDE collector this PR wires for A/B testing. - PR #63 perf(host): the write marks' collection steps over 256 clear marks at a time — Merged PR introduced HOS_COLLECT_WIDE, the first switch measured by this tool.
Owner

This landed on main with #1663, which was stacked on it and squash-merged both commits (main's d10b62b3 carries host/hos_ab.c, scripts/ab_summary.py and tests/test_hos_ab.py). Rebased onto main, this branch has no changes left, so closing it as merged that way. Thank you!

This landed on main with #1663, which was stacked on it and squash-merged both commits (main's d10b62b3 carries host/hos_ab.c, scripts/ab_summary.py and tests/test_hos_ab.py). Rebased onto main, this branch has no changes left, so closing it as merged that way. Thank you!
HumanBean closed this pull request 2026-11-18 22:06:17 +00:00
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Failing after 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 13s
Website / Website on Cloudflare (pull_request_target) Has been skipped

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1661
No description provided.