feat(dev): dev.py ab adds a hitch row, a Spread line, --threads and where the time went #1998

Open
zeldaootr wants to merge 7 commits from zeldaootr/Longshot:dev-ab-table into main
Contributor

Summary

dev.py ab tells you whether a change made the game faster, but its write-up leaves some questions open: were there
hitches, could the difference be noise, which thread got busier, and why did the job take so long. This PR answers
them:

  • The table gains a row of frames of 50 ms or more, the hitches a player sees.
  • A Spread: line under the table gives the fps difference these runs would show most times, from their own spread.
    A reader can see whether a small gain could be noise without working it out.
  • --threads adds the four busiest threads' CPU to the table (HOS_THREAD_TIMES=5 on both sides).
  • After the runs, one line says where the job's time went: each phase's median (start, title and load, teleport,
    warm-up, measured, stop), and each run's phases are kept in results.json.

Two fixes come with it. With --runs 1 the verdict called any difference "faster" or "slower", because one run is
always above or below the other: it now says one run a side cannot tell, as the Spread line does. And the Setup line
said "(window 1920x1080)" for a --headless run: it now says "(headless)". The first commit fixes
tests/test_dev_ab.py, which fails on main because the menu rows it checks moved into gpu/vk/menu/.

Read first: table, verdict, spread_line and run_one in scripts/devtools/ab.py, then the Timing and
WriteUp tests in tests/test_dev_ab.py.

Testing

Tested:

  • Steam Deck OLED, in the game (SteamOS 3.9.5, Mesa RADV, AMD Custom GPU 0932; GPU clock fixed at 1600 MHz, on
    power), at this head: dev.py ab --env HOS_OCB_HOLD_MS=0 --scene kokiri-house --runs 2 --threads --headless with
    the Steam Deck developers' options, on one build of main 2210931cf. The switch changes nothing in the frame, so
    the job tests only what dev.py ab prints. All four starts ran and everything new printed:
    • the 50 ms row: | frames of 50 ms or more (median) | 0.5 | 0 | -0.5 |
    • the Spread line: **Spread:** with 2 runs a side, an fps difference of about 11.5% or more would show most times (2.8 x the runs' spread x sqrt(1/nA + 1/nB)); a smaller one can go unseen.
    • the --threads rows: | Presentation Thread CPU (% of a core, median) | 53 | 54 | +1 |, and three more threads
    • the time line: dev.py ab: the job took 8.1 min for 4 runs; a run's median: start 9 s, title and load 38 s, teleport 0 s, warm-up 10 s, measured 63 s, stop 2 s
    • the Setup line says (headless); on main the same run says "(window 1920x1080)"
    • the verdict, as on main: no difference shown: +0.4% fps (median 45.9 against 45.7), within the runs' spread
  • The one-run rule, seen on main on the same Deck: a --runs 1 job on that switch called the result a loss
    ("slower: -3.7% fps (median 26.0 against 27.0), every B run below every A run"). With this PR the same numbers give
    "one run a side cannot tell a difference from noise" (game-free check below).
  • Linux, game-free: tests/test_dev_ab.py passes: new checks for the 50 ms count, the Spread figure and line
    (with one run a side too), the one-run verdict, the thread lines and their shares, the time summary and its line,
    and the headless Setup line. The write-up still passes check_pr_description.py's Measurements check. Every check
    that reads the changed files (--changed: test_dev_ab, test_dev_packs, test_driver_cache_env, test_hos_ab,
    test_oot_state_tap, test_vk_pgo, plus test_dev_build, test_dev_update, test_job_queue, test_dev_migrate,
    test_dev_vcs and byte_compile_tracked) passes here except test_dev_packs, which fails the same way on main
    (longshot_fsr4_pipelines.bin is not classified). On main, test_dev_ab fails too; this PR fixes it.
  • The whole tests/run_all.py, on this branch and on main @ 6e1521a70: this branch 788 passed, 89 failed, 28 skipped; main 786 passed, 91 failed, 28 skipped (877 checks, through the job queue, without --strict). Every check that fails here also fails on main. main fails two more: test_dev_ab, which this PR fixes, and test_perf_report, which failed there under the full run's load and passes on its own on main (13 s). The 89 that fail on both: test_appimage_player_build, test_bug_report_defaults, test_capture_readback_visibility, test_capture_read_once, test_crash_dialog, test_crash_record, test_crt_options, test_damage_debug_menu, test_device_lost_exit, test_dev_packs, test_discover, test_display_startup_failure, test_dlss_feature_size, test_draw_binds, test_draw_stream_visibility, test_failed_compile_keeps_objects, test_flush_nothing_held, test_fpx_sticky, test_frame_overlap, test_fsr3fi_hud_layer, test_fsr_image_retirement, test_fsr_uniform_ownership, test_gpu_wait_stats, test_guest_imported, test_high_fps_fixes, test_host_guest_writes, test_image_reaper_threads, test_libc_hooks, test_lift_cache, test_lift_lookups, test_linux_preflight, test_linux_pressure, test_linux_tools, test_linux_update, test_log_open_failure, test_low_vram_texture_mips, test_make_game_cache, test_make_game_ngx, test_menu_module_slots, test_menu_modules, test_menu_sections, test_mesh_conv, test_mise_build, test_missing_nup, test_mods_list, test_mods_size_table, test_ngx_sdk, test_nonuniform_image_arrays, test_oot_shadow_fit, test_pads_after_menu, test_pass_barrier, test_pending_guest_import, test_pipeline_cache_save, test_pipeline_play_cache, test_pipelines_outlive_cache_format, test_prepare_cache_save, test_prepare_stall, test_present_queue_default, test_present_source_crop, test_profiling_mode, test_program_cache_cleanup, test_program_cache_one_variant, test_psychov35_fsr3_fg, test_readback_timeout_source, test_recorder_own_idle, test_recorder_sync, test_reflex_swapchain_lifetime, test_renderer_new_handler, test_renhdrum, test_replay_write_order, test_sampler_admission, test_settings_table, test_slow_frames, test_song_note_cue, test_sparse_mirror, test_spirv_pack, test_sprint, test_staging_burst, test_target_reuse_memory, test_texel_buffer_ownership, test_thinlto_tls, test_ub_update_batch, test_ui_record, test_upscaler_rows, test_vbytes, test_vcore_escape, test_vcore_mask0, test_vk_occlusion_queries, test_walk. test_fpx_sticky and test_vbytes among them ran out of time (600 s) on both sides. Of them, only test_dev_packs reads a file this PR changes, and it fails on a renderer file it does not know (longshot_fsr4_pipelines.bin), on both sides.

Still to test:

  • A Linux desktop or a Windows PC: one dev.py ab --env <any switch> --runs 2 --threads, expecting the 50 ms row, the
    Spread line, the thread rows and the time line, and a Setup line that names the window (by hand)

What the time line shows today. For whoever wants to make dev.py ab faster, this is what the new time line shows about one start of a 60 s scene on
the Steam Deck above (Link's house, 4 starts, 8.1 min): start 9 s, title and load 38 s, warm-up 10 s, measured 63 s
(60 s, then 3 s for the frame trace to be written), stop 2 s. So about half of each start is outside the measured
minute, and the title and load is the biggest part of that. A scene reached by teleport adds the teleport and a longer
warm-up, about 10 s more on this Deck.

Measurements

Nothing to measure: this changes what dev.py ab prints, not the game.

Risks

Only dev.py ab changes; the game is untouched. The write-up gains a row, a Spread line and, with --threads, thread
rows: a script that reads measurements.md line by line could notice. A --runs 1 job's verdict no longer says
"faster" or "slower". --threads turns on HOS_THREAD_TIMES on both sides, which wakes a thread every 5 s; it is off
unless asked. Reverting is enough.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says. (a Steam Deck OLED, at this head)
  • mise run test passes (any failure that also fails on main is named). (the whole tests/run_all.py here and on main 6e1521a70: every failure here also fails on main, named under Tested)
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested. (a Linux desktop or Windows PC to come)
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). (the Steam Deck only)
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary `dev.py ab` tells you whether a change made the game faster, but its write-up leaves some questions open: were there hitches, could the difference be noise, which thread got busier, and why did the job take so long. This PR answers them: - The table gains a row of **frames of 50 ms or more**, the hitches a player sees. - A **Spread:** line under the table gives the fps difference these runs would show most times, from their own spread. A reader can see whether a small gain could be noise without working it out. - **`--threads`** adds the four busiest threads' CPU to the table (`HOS_THREAD_TIMES=5` on both sides). - After the runs, one line says **where the job's time went**: each phase's median (start, title and load, teleport, warm-up, measured, stop), and each run's phases are kept in `results.json`. Two fixes come with it. With `--runs 1` the verdict called any difference "faster" or "slower", because one run is always above or below the other: it now says one run a side cannot tell, as the Spread line does. And the Setup line said "(window 1920x1080)" for a `--headless` run: it now says "(headless)". The first commit fixes `tests/test_dev_ab.py`, which fails on `main` because the menu rows it checks moved into `gpu/vk/menu/`. Read first: `table`, `verdict`, `spread_line` and `run_one` in `scripts/devtools/ab.py`, then the `Timing` and `WriteUp` tests in `tests/test_dev_ab.py`. ## Testing **Tested:** - **Steam Deck OLED, in the game** (SteamOS 3.9.5, Mesa RADV, AMD Custom GPU 0932; GPU clock fixed at 1600 MHz, on power), at this head: `dev.py ab --env HOS_OCB_HOLD_MS=0 --scene kokiri-house --runs 2 --threads --headless` with the Steam Deck developers' options, on one build of `main` 2210931cf. The switch changes nothing in the frame, so the job tests only what `dev.py ab` prints. All four starts ran and everything new printed: - the 50 ms row: `| frames of 50 ms or more (median) | 0.5 | 0 | -0.5 |` - the Spread line: `**Spread:** with 2 runs a side, an fps difference of about 11.5% or more would show most times (2.8 x the runs' spread x sqrt(1/nA + 1/nB)); a smaller one can go unseen.` - the `--threads` rows: `| Presentation Thread CPU (% of a core, median) | 53 | 54 | +1 |`, and three more threads - the time line: `dev.py ab: the job took 8.1 min for 4 runs; a run's median: start 9 s, title and load 38 s, teleport 0 s, warm-up 10 s, measured 63 s, stop 2 s` - the Setup line says `(headless)`; on `main` the same run says "(window 1920x1080)" - the verdict, as on `main`: `no difference shown: +0.4% fps (median 45.9 against 45.7), within the runs' spread` - **The one-run rule**, seen on `main` on the same Deck: a `--runs 1` job on that switch called the result a loss ("slower: -3.7% fps (median 26.0 against 27.0), every B run below every A run"). With this PR the same numbers give "one run a side cannot tell a difference from noise" (game-free check below). - **Linux, game-free:** `tests/test_dev_ab.py` passes: new checks for the 50 ms count, the Spread figure and line (with one run a side too), the one-run verdict, the thread lines and their shares, the time summary and its line, and the headless Setup line. The write-up still passes `check_pr_description.py`'s Measurements check. Every check that reads the changed files (`--changed`: `test_dev_ab`, `test_dev_packs`, `test_driver_cache_env`, `test_hos_ab`, `test_oot_state_tap`, `test_vk_pgo`, plus `test_dev_build`, `test_dev_update`, `test_job_queue`, `test_dev_migrate`, `test_dev_vcs` and `byte_compile_tracked`) passes here except `test_dev_packs`, which fails the same way on `main` (`longshot_fsr4_pipelines.bin` is not classified). On `main`, `test_dev_ab` fails too; this PR fixes it. - **The whole `tests/run_all.py`**, on this branch and on `main` @ 6e1521a70: this branch 788 passed, 89 failed, 28 skipped; `main` 786 passed, 91 failed, 28 skipped (877 checks, through the job queue, without `--strict`). Every check that fails here also fails on `main`. `main` fails two more: `test_dev_ab`, which this PR fixes, and `test_perf_report`, which failed there under the full run's load and passes on its own on `main` (13 s). The 89 that fail on both: `test_appimage_player_build`, `test_bug_report_defaults`, `test_capture_readback_visibility`, `test_capture_read_once`, `test_crash_dialog`, `test_crash_record`, `test_crt_options`, `test_damage_debug_menu`, `test_device_lost_exit`, `test_dev_packs`, `test_discover`, `test_display_startup_failure`, `test_dlss_feature_size`, `test_draw_binds`, `test_draw_stream_visibility`, `test_failed_compile_keeps_objects`, `test_flush_nothing_held`, `test_fpx_sticky`, `test_frame_overlap`, `test_fsr3fi_hud_layer`, `test_fsr_image_retirement`, `test_fsr_uniform_ownership`, `test_gpu_wait_stats`, `test_guest_imported`, `test_high_fps_fixes`, `test_host_guest_writes`, `test_image_reaper_threads`, `test_libc_hooks`, `test_lift_cache`, `test_lift_lookups`, `test_linux_preflight`, `test_linux_pressure`, `test_linux_tools`, `test_linux_update`, `test_log_open_failure`, `test_low_vram_texture_mips`, `test_make_game_cache`, `test_make_game_ngx`, `test_menu_module_slots`, `test_menu_modules`, `test_menu_sections`, `test_mesh_conv`, `test_mise_build`, `test_missing_nup`, `test_mods_list`, `test_mods_size_table`, `test_ngx_sdk`, `test_nonuniform_image_arrays`, `test_oot_shadow_fit`, `test_pads_after_menu`, `test_pass_barrier`, `test_pending_guest_import`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_pipelines_outlive_cache_format`, `test_prepare_cache_save`, `test_prepare_stall`, `test_present_queue_default`, `test_present_source_crop`, `test_profiling_mode`, `test_program_cache_cleanup`, `test_program_cache_one_variant`, `test_psychov35_fsr3_fg`, `test_readback_timeout_source`, `test_recorder_own_idle`, `test_recorder_sync`, `test_reflex_swapchain_lifetime`, `test_renderer_new_handler`, `test_renhdrum`, `test_replay_write_order`, `test_sampler_admission`, `test_settings_table`, `test_slow_frames`, `test_song_note_cue`, `test_sparse_mirror`, `test_spirv_pack`, `test_sprint`, `test_staging_burst`, `test_target_reuse_memory`, `test_texel_buffer_ownership`, `test_thinlto_tls`, `test_ub_update_batch`, `test_ui_record`, `test_upscaler_rows`, `test_vbytes`, `test_vcore_escape`, `test_vcore_mask0`, `test_vk_occlusion_queries`, `test_walk`. `test_fpx_sticky` and `test_vbytes` among them ran out of time (600 s) on both sides. Of them, only `test_dev_packs` reads a file this PR changes, and it fails on a renderer file it does not know (`longshot_fsr4_pipelines.bin`), on both sides. **Still to test:** - A Linux desktop or a Windows PC: one `dev.py ab --env <any switch> --runs 2 --threads`, expecting the 50 ms row, the Spread line, the thread rows and the time line, and a Setup line that names the window (by hand) **What the time line shows today.** For whoever wants to make `dev.py ab` faster, this is what the new time line shows about one start of a 60 s scene on the Steam Deck above (Link's house, 4 starts, 8.1 min): start 9 s, title and load 38 s, warm-up 10 s, measured 63 s (60 s, then 3 s for the frame trace to be written), stop 2 s. So about half of each start is outside the measured minute, and the title and load is the biggest part of that. A scene reached by teleport adds the teleport and a longer warm-up, about 10 s more on this Deck. ## Measurements Nothing to measure: this changes what `dev.py ab` prints, not the game. ## Risks Only `dev.py ab` changes; the game is untouched. The write-up gains a row, a Spread line and, with `--threads`, thread rows: a script that reads `measurements.md` line by line could notice. A `--runs 1` job's verdict no longer says "faster" or "slower". `--threads` turns on `HOS_THREAD_TIMES` on both sides, which wakes a thread every 5 s; it is off unless asked. Reverting is enough. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. (a Steam Deck OLED, at this head) - [x] `mise run test` passes (any failure that also fails on `main` is named). (the whole `tests/run_all.py` here and on `main` 6e1521a70: every failure here also fails on `main`, named under Tested) - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. (a Linux desktop or Windows PC to come) - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). (the Steam Deck only) - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
The check looked for the Skip title screen, V-Sync and Frame generation rows in
`gpu/vk/vk_menu.cpp`, which no longer holds them: every row moved into a module
under `gpu/vk/menu/`. It now finds each row by its key there and checks its label
and the value it sets.

Assisted-by: Claude
A frame of 50 ms or more is a hitch a player sees, and the fps and p99 rows can
hide a few of them. Each run now counts them, and the table gains a row with
each side's median count and the difference.

Assisted-by: Claude
The verdict calls a difference a gain or a loss when every B run is above, or
below, every A run. With `--runs 1` that is always true unless the two runs
tie, so any noise came out as "faster" or "slower". One run a side has no
spread to compare with: the verdict now says it cannot tell, and asks for
`--runs 2` or more.

Assisted-by: Claude
The Measurements section gains a Spread line: the fps difference these runs
would show most times, from their own spread (2.8 x the spread x
sqrt(1/nA + 1/nB)). A reader can then tell whether a small gain could be noise
without working it out. With one run a side there is no spread, and the line
says so, as the verdict does.

Assisted-by: Claude
`--threads` sets `HOS_THREAD_TIMES=5` on both sides. Each run reads the
thread lines its game logged during the measured stretch, and the table gains a
row for each of the four busiest threads: its median share of a core on each
side and the difference. The lines are read from the game's log file, because
the bridge's log events cut a long line short.

Assisted-by: Claude
Each run keeps how long each phase took in `results.json`: the start, the title
and load, the teleport, the warm-up, the measured time and the stop. The job
then prints one line with how long it took, how long the builds and checks
before the runs took, and each phase's median. The builds are named only when
they took 0.1 min or more, so a switch A/B, which builds nothing, leaves them
out. The line shows where a job's minutes go, and a change to `dev.py ab`
itself can be timed against the job before it.

Assisted-by: Claude
fix(dev): dev.py ab's Setup line says headless for a --headless run
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
06e96a8b5c
With `HOS_HEADLESS=1` the game's status still reports a window size, so the
Setup line said "(window 1920x1080)" for a run that had no window. It now says
"(headless)" whenever the runs set `HOS_HEADLESS=1`.

Assisted-by: Claude
zeldaootr changed title from WIP: feat(dev): dev.py ab adds a hitch row, a Spread line, --threads and where the time went to feat(dev): dev.py ab adds a hitch row, a Spread line, --threads and where the time went 2026-11-21 03:45:04 +00:00
Member

Possibly related, for context:

  • issue #1517 The renderer rewrite's frame-time gates need a noise floor per machine, and the test farm's gate has none — Noise-floor context for benchmark spread
  • PR #1661 feat(dev): an A/B within one run: HOS_AB alternates a switch every few seconds, scripts/ab_summary.py compares the sides — Earlier A/B measurement tooling; distinct in-run method
Possibly related, for context: - issue #1517 The renderer rewrite's frame-time gates need a noise floor per machine, and the test farm's gate has none — Noise-floor context for benchmark spread - PR #1661 feat(dev): an A/B within one run: HOS_AB alternates a switch every few seconds, scripts/ab_summary.py compares the sides — Earlier A/B measurement tooling; distinct in-run method
BeanBot left a comment

Recommendation: Request changes. The Deck run shows the new output and the one-run verdict fix. The Spread line overstates what two runs can establish, and the thread report can silently mix measurement windows, omit samples, or activate an unsafe Linux sampler path.

Developers get a 50 ms frame count, a variability estimate, optional thread CPU rows, and phase timings. These can make A/B reports easier to interpret. The current implementation adds live log reads to ordinary benchmarks and gives some incomplete or approximate measurements more certainty than they warrant.

@fix will do (17)

  1. L1 scripts/devtools/ab.py:408: The Spread line promises a detection probability the command does not calculate. (more on the line)
  2. L2 scripts/devtools/ab.py:452: The in-run table returns before adding the hitch row. (more on the line)
  3. L3 scripts/devtools/ab.py:807: --threads accepts automatic HOS_AB, starts the sampler, and discards its output through the side == "AB" branch. A five-second CPU interval can also cross multiple three-second switch periods. Reject this combination before launch with instructions to use --separate, or implement side-aware sampling.
  4. L4 scripts/devtools/ab.py:802: --threads exposes an unsafe Linux sampler path. (more on the line)
  5. L5 scripts/devtools/ab.py:700: Every benchmark now grabs the session manifest lock and scans a growing log twice during play, then parses it again after stop. (more on the line)
  6. L6 scripts/devtools/ab.py:703: Log line counts do not identify the frame-trace window. (more on the line)
  7. L7 scripts/devtools/ab.py:467: Missing thread samples silently change the population used for the median. (more on the line)
  8. L8 scripts/devtools/ab.py:278: Reading the file bypasses the bridge event limit, but both host producers still call hos_log, whose formatter truncates the message at 1,023 bytes. (more on the line)
  9. L9 scripts/devtools/ab.py:802: --threads --set HOS_THREAD_TIMES=0 is accepted and disables the sampler, although the help and docs promise five-second samples on both sides. (more on the line)
  10. L10 (minor) scripts/devtools/ab.py:479: The Setup label checks only common_env and only the literal 1. (more on the line)
  11. L11 (minor) scripts/devtools/ab.py:628: The summary discards the failed bucket, which can contain most of the job. (more on the line)
  12. L12 (minor) scripts/devtools/ab.py:844: job_s starts after preparation, yet the report calls it the job's duration. (more on the line)
  13. L13 (minor) docs/dev/developing.md:386: The write-up description reads as if Spread and the one-run rule apply to all A/Bs. (more on the line)
  14. L14 (minor) tests/test_dev_ab.py:236: The lap test passes if every phase is recorded as zero, and the thread table test injects finished stats without exercising CLI wiring. (more on the line)
  15. L15 (minor) tests/test_dev_ab.py:123: The migrated source check unpacks the pinned value and never checks it. (more on the line)
  16. L16 (minor): The checklist marks mise run test complete, while the evidence is a non-strict run with 28 skips. The documented command uses --strict. Record strict results for head and base or leave the box open and name what remains. Include the required lint result and qualify the no-cost claim until profiling overhead is measured.
  17. L17 (its part an edit can do) scripts/devtools/ab.py:891: Keep the sampler opt-in and remove live log reads from ordinary runs. Its remaining opt-in cost needs a run.

For the author (1) (a run, a measurement or a decision)

  • L17 (minor) scripts/devtools/ab.py:891: The reported A/B enables thread profiling on both sides, so it cannot show what profiling costs. (more on the line)

Checks: Still running: CI / Conventional Commit title (pull_request), CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request), CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request), CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request), CI / Workflow lint (actionlint) (pull_request), CI / Website build and checks (pull_request), Guide screenshots / Guide images and local UI source review (pull_request), Third-party parts / As their notes say (pull_request), Third-party parts / The same as upstream (pull_request), Website / Website on Cloudflare (pull_request).

More: does it help, other systems, switches, what was read

Does it help: Yes for the two demonstrated reporting fixes. The author reports that a single pair with a −3.7% FPS difference used to say “slower” and now says it cannot distinguish the difference from noise. The four-start Deck run changed the Setup label from a 1920×1080 window to headless and printed the new rows. The +0.4% FPS difference, 45.9 versus 45.7, compares switch settings on one runtime. The benchmark overhead and the Spread line's claimed detection rate remain unproven.

Other systems: The reporting code is shared across GPU vendors. The sampler is platform-specific. Linux enumerates /proc and guest threads, while Windows enumerates OS threads and batches log output. The Deck run exercises one Linux RADV headless path. Windowed operation, Windows logging, and automatic HOS_AB have no reported end-to-end run. The diff leaves player builds, renderer interfaces, saves, audio, and input paths untouched. Local results.json and measurements.md gain fields and rows. The checkout search found no in-tree consumer of those artifacts.

Switches: --threads defaults off, which is appropriate for a benchmark tool. It normally sets the existing diagnostic HOS_THREAD_TIMES to 5 seconds on both sides. --set can currently override or disable that value, and log collection runs even with the flag off. The host switch retains its existing off default and its row in gpu/vk/switches.def. --headless retains its launch behavior and changes the Setup label.

Reviewed: 06e96a8b5c against 6e1521a70a: Read AGENTS.md, CONTRIBUTING.md, docs/dev/standards.md, the Forgejo template, the full changed Python files, and the changed developer docs section with its surrounding guidance. Read the three diffs and all seven commit messages. Traced the relevant session, queue, frame trace, host samplers, log writer, headless status, menu rows, and PR description checker. Skimmed the reported runs and adjacent developer docs. No repository review checklist or sibling Tooling checkout was present. The underlying run logs and results files were unavailable.; nothing run (a code review).

Checked and fine:

  • All seven commits carry Assisted-by trailers. The diff contains no game-derived fixtures, build outputs, generated files, or third-party edits.
  • The 50 ms calculation includes the boundary on the separate-run path, and the fixture exercises 49.9, 50.0, and 70.0 ms.
  • The single-pair test exercises a difference that previously produced a gain. The ordinary window fixture still checks the window size.
  • Changed Python uses the standard library, Unicode paths and explicit UTF-8 log decoding. The tests use the current Python interpreter and introduce no compiler dependency.
  • The title after the WIP prefix follows the Conventional Commit format. No shader, renderer API, switch table, or player settings table update is required.

Testing: Static review only. No repository code or tests were run. The author reports a successful four-start Deck smoke test and named shared suite failures. CI was still running, and the supplied CI status listed no head-only failures. There were no tester comments or previous reviews.

Overlaps and merge order: No open overlap could be verified from the checkout. #1870 is already on local main and changes HOS_OCB_HOLD_MS, the switch used in the Deck smoke test. It has no file overlap with this PR. Refresh the smoke test on current main before merge.


Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The Deck run shows the new output and the one-run verdict fix. The Spread line overstates what two runs can establish, and the thread report can silently mix measurement windows, omit samples, or activate an unsafe Linux sampler path. Developers get a 50 ms frame count, a variability estimate, optional thread CPU rows, and phase timings. These can make A/B reports easier to interpret. The current implementation adds live log reads to ordinary benchmarks and gives some incomplete or approximate measurements more certainty than they warrant. **`@fix` will do (17)** 1. **L1** `scripts/devtools/ab.py:408`: The Spread line promises a detection probability the command does not calculate. (more on the line) 2. **L2** `scripts/devtools/ab.py:452`: The in-run table returns before adding the hitch row. (more on the line) 3. **L3** `scripts/devtools/ab.py:807`: `--threads` accepts automatic HOS_AB, starts the sampler, and discards its output through the `side == "AB"` branch. A five-second CPU interval can also cross multiple three-second switch periods. Reject this combination before launch with instructions to use `--separate`, or implement side-aware sampling. 4. **L4** `scripts/devtools/ab.py:802`: `--threads` exposes an unsafe Linux sampler path. (more on the line) 5. **L5** `scripts/devtools/ab.py:700`: Every benchmark now grabs the session manifest lock and scans a growing log twice during play, then parses it again after stop. (more on the line) 6. **L6** `scripts/devtools/ab.py:703`: Log line counts do not identify the frame-trace window. (more on the line) 7. **L7** `scripts/devtools/ab.py:467`: Missing thread samples silently change the population used for the median. (more on the line) 8. **L8** `scripts/devtools/ab.py:278`: Reading the file bypasses the bridge event limit, but both host producers still call `hos_log`, whose formatter truncates the message at 1,023 bytes. (more on the line) 9. **L9** `scripts/devtools/ab.py:802`: `--threads --set HOS_THREAD_TIMES=0` is accepted and disables the sampler, although the help and docs promise five-second samples on both sides. (more on the line) 10. **L10** (minor) `scripts/devtools/ab.py:479`: The Setup label checks only `common_env` and only the literal `1`. (more on the line) 11. **L11** (minor) `scripts/devtools/ab.py:628`: The summary discards the `failed` bucket, which can contain most of the job. (more on the line) 12. **L12** (minor) `scripts/devtools/ab.py:844`: `job_s` starts after preparation, yet the report calls it the job's duration. (more on the line) 13. **L13** (minor) `docs/dev/developing.md:386`: The write-up description reads as if Spread and the one-run rule apply to all A/Bs. (more on the line) 14. **L14** (minor) `tests/test_dev_ab.py:236`: The lap test passes if every phase is recorded as zero, and the thread table test injects finished stats without exercising CLI wiring. (more on the line) 15. **L15** (minor) `tests/test_dev_ab.py:123`: The migrated source check unpacks the pinned value and never checks it. (more on the line) 16. **L16** (minor): The checklist marks `mise run test` complete, while the evidence is a non-strict run with 28 skips. The documented command uses `--strict`. Record strict results for head and base or leave the box open and name what remains. Include the required lint result and qualify the no-cost claim until profiling overhead is measured. 17. **L17** (its part an edit can do) `scripts/devtools/ab.py:891`: Keep the sampler opt-in and remove live log reads from ordinary runs. Its remaining opt-in cost needs a run. **For the author (1)** (a run, a measurement or a decision) - **L17** (minor) `scripts/devtools/ab.py:891`: The reported A/B enables thread profiling on both sides, so it cannot show what profiling costs. (more on the line) **Checks:** Still running: `CI / Conventional Commit title (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request)`, `CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request)`, `CI / Workflow lint (actionlint) (pull_request)`, `CI / Website build and checks (pull_request)`, `Guide screenshots / Guide images and local UI source review (pull_request)`, `Third-party parts / As their notes say (pull_request)`, `Third-party parts / The same as upstream (pull_request)`, `Website / Website on Cloudflare (pull_request)`. <details> <summary>More: does it help, other systems, switches, what was read</summary> **Does it help:** Yes for the two demonstrated reporting fixes. The author reports that a single pair with a −3.7% FPS difference used to say “slower” and now says it cannot distinguish the difference from noise. The four-start Deck run changed the Setup label from a 1920×1080 window to headless and printed the new rows. The +0.4% FPS difference, 45.9 versus 45.7, compares switch settings on one runtime. The benchmark overhead and the Spread line's claimed detection rate remain unproven. **Other systems:** The reporting code is shared across GPU vendors. The sampler is platform-specific. Linux enumerates /proc and guest threads, while Windows enumerates OS threads and batches log output. The Deck run exercises one Linux RADV headless path. Windowed operation, Windows logging, and automatic HOS_AB have no reported end-to-end run. The diff leaves player builds, renderer interfaces, saves, audio, and input paths untouched. Local results.json and measurements.md gain fields and rows. The checkout search found no in-tree consumer of those artifacts. **Switches:** `--threads` defaults off, which is appropriate for a benchmark tool. It normally sets the existing diagnostic `HOS_THREAD_TIMES` to 5 seconds on both sides. `--set` can currently override or disable that value, and log collection runs even with the flag off. The host switch retains its existing off default and its row in gpu/vk/switches.def. `--headless` retains its launch behavior and changes the Setup label. **Reviewed:** 06e96a8b5ce4 against 6e1521a70a32: Read AGENTS.md, CONTRIBUTING.md, docs/dev/standards.md, the Forgejo template, the full changed Python files, and the changed developer docs section with its surrounding guidance. Read the three diffs and all seven commit messages. Traced the relevant session, queue, frame trace, host samplers, log writer, headless status, menu rows, and PR description checker. Skimmed the reported runs and adjacent developer docs. No repository review checklist or sibling Tooling checkout was present. The underlying run logs and results files were unavailable.; nothing run (a code review). **Checked and fine:** - All seven commits carry Assisted-by trailers. The diff contains no game-derived fixtures, build outputs, generated files, or third-party edits. - The 50 ms calculation includes the boundary on the separate-run path, and the fixture exercises 49.9, 50.0, and 70.0 ms. - The single-pair test exercises a difference that previously produced a gain. The ordinary window fixture still checks the window size. - Changed Python uses the standard library, Unicode paths and explicit UTF-8 log decoding. The tests use the current Python interpreter and introduce no compiler dependency. - The title after the WIP prefix follows the Conventional Commit format. No shader, renderer API, switch table, or player settings table update is required. **Testing:** Static review only. No repository code or tests were run. The author reports a successful four-start Deck smoke test and named shared suite failures. CI was still running, and the supplied CI status listed no head-only failures. There were no tester comments or previous reviews. **Overlaps and merge order:** No open overlap could be verified from the checkout. #1870 is already on local main and changes HOS_OCB_HOLD_MS, the switch used in the Deck smoke test. It has no file overlap with this PR. Refresh the smoke test on current main before merge. </details> --- <sub>Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -384,3 +384,3 @@
| Measure | `--seconds` (60) from the frame trace, `--runs` (3) pairs interleaved A, B, A, B. A switch whose reader takes `HOS_AB` is flipped within each run instead ([below](#an-ab-within-one-run); `--separate` to not) |
| Check | Refuses to write up runs that differ in a menu setting, a switch, the renderer or the window, apart from what is compared (`--allow KEY` for a row the change adds) |
| Write up | Each run's numbers, the medians and spread, a verdict (gain, loss, within the spread) and the Measurements section, in `runs/ab/` of the development folder (`dev.py paths runs`) and printed. Exit status 1 when a run failed, 3 when the sides differed |
| Write up | Each run's numbers (fps, 1% low, median and p99 frame time, frames of 50 ms or more), the medians and spread, a verdict (gain, loss, within the spread; one run a side cannot tell), a **Spread:** line (the fps difference these runs' own spread would show most times) and the Measurements section, in `runs/ab/` of the development folder (`dev.py paths runs`) and printed. `--threads` adds the busiest threads' CPU (`HOS_THREAD_TIMES=5` on both sides, read from the game's log file). Then where the time went: each run's phases (the start, the title and load, the teleport, the warm-up, the measured time, the stop) in `results.json`, and their medians in a line printed after the runs. Exit status 1 when a run failed, 3 when the sides differed |
Member

L13 (minor): The write-up description reads as if Spread and the one-run rule apply to all A/Bs. An automatic HOS_AB run uses a period-based interval and can produce a verdict from one launch. Document the separate-run and in-run outputs and explain that CPU estimates omit sub-5% samples and combine equal thread names.

**L13** (minor): The write-up description reads as if Spread and the one-run rule apply to all A/Bs. An automatic HOS_AB run uses a period-based interval and can produce a verdict from one launch. Document the separate-run and in-run outputs and explain that CPU estimates omit sub-5% samples and combine equal thread names.
@ -262,0 +275,4 @@
out = []
for line in text.splitlines():
m = THREADS_LINE.search(line)
if m:
Member

L8: Reading the file bypasses the bridge event limit, but both host producers still call hos_log, whose formatter truncates the message at 1,023 bytes. A busy thread late in a long snapshot can disappear and then count as zero. Emit bounded records with a way to detect an incomplete snapshot, or mark truncated snapshots unavailable. Test with a long snapshot whose busiest entry comes last.

**L8**: Reading the file bypasses the bridge event limit, but both host producers still call `hos_log`, whose formatter truncates the message at 1,023 bytes. A busy thread late in a long snapshot can disappear and then count as zero. Emit bounded records with a way to detect an incomplete snapshot, or mark truncated snapshots unavailable. Test with a long snapshot whose busiest entry comes last.
@ -331,0 +405,4 @@
v = spread_pct(fps["A"], fps["B"])
if v is None:
return "**Spread:** with %d run a side, the runs have no spread to measure: --runs 2 or more." % n
return ("**Spread:** with %d runs a side, an fps difference of about %.1f%% or more would show most times (2.8 x the "
Member

L1: The Spread line promises a detection probability the command does not calculate. The 2.8 factor is a large-sample normal approximation for a mean-based test with known variance, while the verdict uses non-overlapping ranges and reports medians. With two identical FPS values on each side the line even promises that a 0.0% difference would show most times. Report the observed variability as a descriptive statistic, or use a small-sample method tied to the actual verdict and handle zero observed variance explicitly.

**L1**: The Spread line promises a detection probability the command does not calculate. The 2.8 factor is a large-sample normal approximation for a mean-based test with known variance, while the verdict uses non-overlapping ranges and reports medians. With two identical FPS values on each side the line even promises that a 0.0% difference would show most times. Report the observed variability as a descriptive statistic, or use a small-sample method tied to the actual verdict and handle zero observed variance explicitly.
@ -370,18 +451,36 @@ def table(plan, results):
rows.append("| %s | %s | %s | %s |" % (label, fmt % s0.get(k, 0), fmt % s1.get(k, 0), delta))
return "\n".join(rows)
Member

L2: The in-run table returns before adding the hitch row. A normal --env HOS_COLLECT_WIDE=0 run therefore produces no 50 ms result. Add a per-side hitch metric to the retained HOS_AB periods and test both sides, accounting for unequal time spent on each side.

**L2**: The in-run table returns before adding the hitch row. A normal `--env HOS_COLLECT_WIDE=0` run therefore produces no 50 ms result. Add a per-side hitch metric to the retained HOS_AB periods and test both sides, accounting for unequal time spent on each side.
@ -379,3 +464,4 @@
rows.append("| %s | %s (median %.1f) | %s (median %.1f) | %s |" % (label, cells(va, fmt), ma, cells(vb, fmt), mb, delta))
else:
rows.append("| %s | %s | %s | %s |" % (label, fmt % ma, fmt % mb, delta))
for name in top_threads(by["A"] + by["B"]):
Member

L7: Missing thread samples silently change the population used for the median. A valid --seconds 10 run can have no usable sample after the boundary interval is dropped, and a live read error becomes a count of zero. One missing run then disappears from its side's median, or the whole CPU table disappears while the command succeeds. Preserve missing data separately from zero, report sample coverage, and explicitly report unavailable comparisons.

**L7**: Missing thread samples silently change the population used for the median. A valid `--seconds 10` run can have no usable sample after the boundary interval is dropped, and a live read error becomes a count of zero. One missing run then disappears from its side's median, or the whole CPU table disappears while the command succeeds. Preserve missing data separately from zero, report sample coverage, and explicitly report unavailable comparisons.
@ -385,0 +476,4 @@
def window_text(plan, ran):
"""The window a run had; "headless" with HOS_HEADLESS=1, though the game's status still reports a window size"""
w = ran.get("window")
if not w or plan["common_env"].get("HOS_HEADLESS") == "1":
Member

L10 (minor): The Setup label checks only common_env and only the literal 1. --set HOS_HEADLESS=on launches headless under the host's boolean rules and can still print window dimensions. Side overrides are also ignored. Record the effective mode for each run and render it from the results, including mixed modes if they are allowed.

**L10** (minor): The Setup label checks only `common_env` and only the literal `1`. `--set HOS_HEADLESS=on` launches headless under the host's boolean rules and can still print window dimensions. Side overrides are also ignored. Record the effective mode for each run and render it from the results, including mixed modes if they are allowed.
@ -501,0 +625,4 @@
for k, v in (r.get("timing") or {}).items():
phases.setdefault(k, []).append(v)
return {"job_s": round(job_s, 1), "runs": len(runs),
"median_s": {k: statistics.median(phases[k]) for k in TIMING_ORDER if k in phases}}
Member

L11 (minor): The summary discards the failed bucket, which can contain most of the job. A 300-second title timeout followed by cleanup can leave a line showing only the start median. Attribute elapsed time to the active phase on failure, report cleanup separately, and cover a timeout in the timing test.

**L11** (minor): The summary discards the `failed` bucket, which can contain most of the job. A 300-second title timeout followed by cleanup can leave a line showing only the start median. Attribute elapsed time to the active phase on failure, report cleanup separately, and cover a timeout in the timing test.
@ -556,3 +698,3 @@
raise drive.Failed("no frame trace at %s (RX_VK_FRAME_TRACE)" % trace)
start = num(rows[-1], "at_ns")
time.sleep(plan["seconds"] + 3) # (3 s more: the trace is flushed every 60 frames)
live_log = session.info()["log"]
Member

L5: Every benchmark now grabs the session manifest lock and scans a growing log twice during play, then parses it again after stop. This includes runs without --threads and in-run comparisons that discard the data. A verbose run can compete with the game for CPU or disk at the measurement boundaries, and thread rows can appear through --set alone. Gate collection and reporting on the requested, supported mode and use bounded reads outside the measured interval.

**L5**: Every benchmark now grabs the session manifest lock and scans a growing log twice during play, then parses it again after stop. This includes runs without `--threads` and in-run comparisons that discard the data. A verbose run can compete with the game for CPU or disk at the measurement boundaries, and thread rows can appear through `--set` alone. Gate collection and reporting on the requested, supported mode and use bounded reads outside the measured interval.
@ -559,0 +700,4 @@
live_log = session.info()["log"]
threads_at = count_thread_lines(live_log)
time.sleep(plan["seconds"])
threads_at = (threads_at, count_thread_lines(live_log))
Member

L6: Log line counts do not identify the frame-trace window. The trace starts at the last visible CSV row, while the live snapshots bracket a later wall-clock sleep. Windows queues log writes for 250 ms by default, so a sample taken just before the end can appear only after the second count. Dropping the first line cannot reconcile those clocks. Timestamp complete CPU sample intervals on the trace clock and select them after the run.

**L6**: Log line counts do not identify the frame-trace window. The trace starts at the last visible CSV row, while the live snapshots bracket a later wall-clock sleep. Windows queues log writes for 250 ms by default, so a sample taken just before the end can appear only after the second count. Dropping the first line cannot reconcile those clocks. Timestamp complete CPU sample intervals on the trace clock and select them after the run.
@ -644,0 +799,4 @@
if a.threads:
if "HOS_THREAD_TIMES" in set(a_env) | set(b_env):
die("--threads sets HOS_THREAD_TIMES, which is compared here")
common_env.setdefault("HOS_THREAD_TIMES", str(THREAD_EVERY_S))
Member

L4: --threads exposes an unsafe Linux sampler path. tasks() in host/posix/prof.c uses borrowed handles and name pointers after hos_thread_handles() releases the registry lock. A guest thread can be destroyed before GetThreadId() or the name copy, and start and rename write the fields without that lock. A simultaneous --set HOS_PROFILE_EVERY=5 also makes both samplers write the same static scratch arrays. Use an owned, synchronized snapshot of IDs and names and per-call scratch storage before enabling this path from the tool.

**L4**: `--threads` exposes an unsafe Linux sampler path. `tasks()` in host/posix/prof.c uses borrowed handles and name pointers after `hos_thread_handles()` releases the registry lock. A guest thread can be destroyed before `GetThreadId()` or the name copy, and start and rename write the fields without that lock. A simultaneous `--set HOS_PROFILE_EVERY=5` also makes both samplers write the same static scratch arrays. Use an owned, synchronized snapshot of IDs and names and per-call scratch storage before enabling this path from the tool.
Member

L9: --threads --set HOS_THREAD_TIMES=0 is accepted and disables the sampler, although the help and docs promise five-second samples on both sides. Reject the conflicting --set, or define and validate interval overrides and report the effective interval.

**L9**: `--threads --set HOS_THREAD_TIMES=0` is accepted and disables the sampler, although the help and docs promise five-second samples on both sides. Reject the conflicting `--set`, or define and validate interval overrides and report the effective interval.
@ -726,5 +888,7 @@ def add_commands(sub):
p.add_argument("--allow", action="append", metavar="KEY", help="a menu row (its options-file key) or switch the runs "
"may differ in (a row the change adds)")
p.add_argument("--dirty", action="store_true", help="measure a change with uncommitted files")
p.add_argument("--threads", action="store_true", help="each thread's CPU too (HOS_THREAD_TIMES=%d on both sides): "
Member

L17 (minor): The reported A/B enables thread profiling on both sides, so it cannot show what profiling costs. The Windows sampler takes a system thread snapshot every five seconds. Run interleaved on/off pairs on the same build with a windowed Windows PC and a Linux PC, and report FPS, p99 or 1% lows, and 50 ms frame counts. Verify the output on a current runtime.

**L17** (minor): The reported A/B enables thread profiling on both sides, so it cannot show what profiling costs. The Windows sampler takes a system thread snapshot every five seconds. Run interleaved on/off pairs on the same build with a windowed Windows PC and a Linux PC, and report FPS, p99 or 1% lows, and 50 ms frame counts. Verify the output on a current runtime.
@ -682,2 +841,4 @@
began = time.monotonic()
runs = [run_one(plan, side, k, pinned) for side, k in plan["order"]]
res = summarise(plan, runs)
res["timing"] = dict(timing_summary(runs, time.monotonic() - began), before_runs_s=round(began - called, 1))
Member

L12 (minor): job_s starts after preparation, yet the report calls it the job's duration. Four minutes of preparation followed by one minute of runs becomes “the job took 1.0 min.” Record and label the run batch and elapsed total separately, and account for the post-stop trace copy and analysis when explaining where the minutes went.

**L12** (minor): `job_s` starts after preparation, yet the report calls it the job's duration. Four minutes of preparation followed by one minute of runs becomes “the job took 1.0 min.” Record and label the run batch and elapsed total separately, and account for the post-stop trace copy and analysis when explaining where the minutes went.
@ -120,0 +120,4 @@
with open(os.path.join(menu, name), encoding="utf-8") as f:
for m in re.finditer(r'\.row\(\{\.key = "(\w+)",\s*\.label = "([^"]+)"(.*?)\}', f.read(), re.S):
rows[m.group(1)] = (m.group(2), m.group(3), name)
for (key, value), label, field in ((ab.SKIP_TITLE, "Skip title screen", "&opts.skip_title"),
Member

L15 (minor): The migrated source check unpacks the pinned value and never checks it. It also accepts a pointer with a matching prefix and finds the V-Sync labels anywhere in the module. Moving the row to &opts.vsync_backup or leaving an unused label array could pass. Check the complete field and tie the actual pinned value to the row's active Off mapping.

**L15** (minor): The migrated source check unpacks the pinned value and never checks it. It also accepts a pointer with a matching prefix and finds the V-Sync labels anywhere in the module. Moving the row to `&opts.vsync_backup` or leaving an unused label array could pass. Check the complete field and tie the actual pinned value to the row's active Off mapping.
@ -222,1 +233,4 @@
class Timing(unittest.TestCase):
"""Where a job's time goes: each run's phases, the job's medians"""
def test_lap(self):
Member

L14 (minor): The lap test passes if every phase is recorded as zero, and the thread table test injects finished stats without exercising CLI wiring. Use a fake clock with distinct durations and add a game-free command or session test that fails if --threads stops setting the environment or if disabled collection still reads the log.

**L14** (minor): The lap test passes if every phase is recorded as zero, and the thread table test injects finished stats without exercising CLI wiring. Use a fake clock with distinct durations and add a game-free command or session test that fails if `--threads` stops setting the environment or if disabled collection still reads the log.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'dev-ab-table:zeldaootr-dev-ab-table'
git switch 'zeldaootr-dev-ab-table'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1998
No description provided.