feat(dev): dev.py ab adds a hitch row, a Spread line, --threads and where the time went #1998
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1998
Loading…
Reference in a new issue
No description provided.
Delete branch "zeldaootr/Longshot:dev-ab-table"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
dev.py abtells you whether a change made the game faster, but its write-up leaves some questions open: were therehitches, could the difference be noise, which thread got busier, and why did the job take so long. This PR answers
them:
A reader can see whether a small gain could be noise without working it out.
--threadsadds the four busiest threads' CPU to the table (HOS_THREAD_TIMES=5on both sides).warm-up, measured, stop), and each run's phases are kept in
results.json.Two fixes come with it. With
--runs 1the verdict called any difference "faster" or "slower", because one run isalways above or below the other: it now says one run a side cannot tell, as the Spread line does. And the Setup line
said "(window 1920x1080)" for a
--headlessrun: it now says "(headless)". The first commit fixestests/test_dev_ab.py, which fails onmainbecause the menu rows it checks moved intogpu/vk/menu/.Read first:
table,verdict,spread_lineandrun_oneinscripts/devtools/ab.py, then theTimingandWriteUptests intests/test_dev_ab.py.Testing
Tested:
power), at this head:
dev.py ab --env HOS_OCB_HOLD_MS=0 --scene kokiri-house --runs 2 --threads --headlesswiththe Steam Deck developers' options, on one build of
main2210931cf. The switch changes nothing in the frame, sothe job tests only what
dev.py abprints. All four starts ran and everything new printed:| frames of 50 ms or more (median) | 0.5 | 0 | -0.5 |**Spread:** with 2 runs a side, an fps difference of about 11.5% or more would show most times (2.8 x the runs' spread x sqrt(1/nA + 1/nB)); a smaller one can go unseen.--threadsrows:| Presentation Thread CPU (% of a core, median) | 53 | 54 | +1 |, and three more threadsdev.py ab: the job took 8.1 min for 4 runs; a run's median: start 9 s, title and load 38 s, teleport 0 s, warm-up 10 s, measured 63 s, stop 2 s(headless); onmainthe same run says "(window 1920x1080)"main:no difference shown: +0.4% fps (median 45.9 against 45.7), within the runs' spreadmainon the same Deck: a--runs 1job on that switch called the result a loss("slower: -3.7% fps (median 26.0 against 27.0), every B run below every A run"). With this PR the same numbers give
"one run a side cannot tell a difference from noise" (game-free check below).
tests/test_dev_ab.pypasses: new checks for the 50 ms count, the Spread figure and line(with one run a side too), the one-run verdict, the thread lines and their shares, the time summary and its line,
and the headless Setup line. The write-up still passes
check_pr_description.py's Measurements check. Every checkthat reads the changed files (
--changed:test_dev_ab,test_dev_packs,test_driver_cache_env,test_hos_ab,test_oot_state_tap,test_vk_pgo, plustest_dev_build,test_dev_update,test_job_queue,test_dev_migrate,test_dev_vcsandbyte_compile_tracked) passes here excepttest_dev_packs, which fails the same way onmain(
longshot_fsr4_pipelines.binis not classified). Onmain,test_dev_abfails too; this PR fixes it.tests/run_all.py, on this branch and onmain@6e1521a70: this branch 788 passed, 89 failed, 28 skipped;main786 passed, 91 failed, 28 skipped (877 checks, through the job queue, without--strict). Every check that fails here also fails onmain.mainfails two more:test_dev_ab, which this PR fixes, andtest_perf_report, which failed there under the full run's load and passes on its own onmain(13 s). The 89 that fail on both:test_appimage_player_build,test_bug_report_defaults,test_capture_readback_visibility,test_capture_read_once,test_crash_dialog,test_crash_record,test_crt_options,test_damage_debug_menu,test_device_lost_exit,test_dev_packs,test_discover,test_display_startup_failure,test_dlss_feature_size,test_draw_binds,test_draw_stream_visibility,test_failed_compile_keeps_objects,test_flush_nothing_held,test_fpx_sticky,test_frame_overlap,test_fsr3fi_hud_layer,test_fsr_image_retirement,test_fsr_uniform_ownership,test_gpu_wait_stats,test_guest_imported,test_high_fps_fixes,test_host_guest_writes,test_image_reaper_threads,test_libc_hooks,test_lift_cache,test_lift_lookups,test_linux_preflight,test_linux_pressure,test_linux_tools,test_linux_update,test_log_open_failure,test_low_vram_texture_mips,test_make_game_cache,test_make_game_ngx,test_menu_module_slots,test_menu_modules,test_menu_sections,test_mesh_conv,test_mise_build,test_missing_nup,test_mods_list,test_mods_size_table,test_ngx_sdk,test_nonuniform_image_arrays,test_oot_shadow_fit,test_pads_after_menu,test_pass_barrier,test_pending_guest_import,test_pipeline_cache_save,test_pipeline_play_cache,test_pipelines_outlive_cache_format,test_prepare_cache_save,test_prepare_stall,test_present_queue_default,test_present_source_crop,test_profiling_mode,test_program_cache_cleanup,test_program_cache_one_variant,test_psychov35_fsr3_fg,test_readback_timeout_source,test_recorder_own_idle,test_recorder_sync,test_reflex_swapchain_lifetime,test_renderer_new_handler,test_renhdrum,test_replay_write_order,test_sampler_admission,test_settings_table,test_slow_frames,test_song_note_cue,test_sparse_mirror,test_spirv_pack,test_sprint,test_staging_burst,test_target_reuse_memory,test_texel_buffer_ownership,test_thinlto_tls,test_ub_update_batch,test_ui_record,test_upscaler_rows,test_vbytes,test_vcore_escape,test_vcore_mask0,test_vk_occlusion_queries,test_walk.test_fpx_stickyandtest_vbytesamong them ran out of time (600 s) on both sides. Of them, onlytest_dev_packsreads a file this PR changes, and it fails on a renderer file it does not know (longshot_fsr4_pipelines.bin), on both sides.Still to test:
dev.py ab --env <any switch> --runs 2 --threads, expecting the 50 ms row, theSpread line, the thread rows and the time line, and a Setup line that names the window (by hand)
What the time line shows today. For whoever wants to make
dev.py abfaster, this is what the new time line shows about one start of a 60 s scene onthe Steam Deck above (Link's house, 4 starts, 8.1 min): start 9 s, title and load 38 s, warm-up 10 s, measured 63 s
(60 s, then 3 s for the frame trace to be written), stop 2 s. So about half of each start is outside the measured
minute, and the title and load is the biggest part of that. A scene reached by teleport adds the teleport and a longer
warm-up, about 10 s more on this Deck.
Measurements
Nothing to measure: this changes what
dev.py abprints, not the game.Risks
Only
dev.py abchanges; the game is untouched. The write-up gains a row, a Spread line and, with--threads, threadrows: a script that reads
measurements.mdline by line could notice. A--runs 1job's verdict no longer says"faster" or "slower".
--threadsturns onHOS_THREAD_TIMESon both sides, which wakes a thread every 5 s; it is offunless asked. Reverting is enough.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named). (the wholetests/run_all.pyhere and onmain6e1521a70: every failure here also fails onmain, named under Tested)main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.dev.py abcounts the frames of 50 ms or more e8d8492989dev.py abdoesn't call a gain or a loss from one run a side 138fe821dfdev.py absays how small a difference its runs can show ce2ad0f75edev.py ab --threadsadds each busy thread's CPU to the table 70e109dbe3dev.py absays where each run's time went 0f65ae2221dev.py ab's Setup line says headless for a--headlessrunWIP: feat(dev): dev.py ab adds a hitch row, a Spread line, --threads and where the time wentto feat(dev): dev.py ab adds a hitch row, a Spread line, --threads and where the time wentPossibly related, for context:
Recommendation: Request changes. The Deck run shows the new output and the one-run verdict fix. The Spread line overstates what two runs can establish, and the thread report can silently mix measurement windows, omit samples, or activate an unsafe Linux sampler path.
Developers get a 50 ms frame count, a variability estimate, optional thread CPU rows, and phase timings. These can make A/B reports easier to interpret. The current implementation adds live log reads to ordinary benchmarks and gives some incomplete or approximate measurements more certainty than they warrant.
@fixwill do (17)scripts/devtools/ab.py:408: The Spread line promises a detection probability the command does not calculate. (more on the line)scripts/devtools/ab.py:452: The in-run table returns before adding the hitch row. (more on the line)scripts/devtools/ab.py:807:--threadsaccepts automatic HOS_AB, starts the sampler, and discards its output through theside == "AB"branch. A five-second CPU interval can also cross multiple three-second switch periods. Reject this combination before launch with instructions to use--separate, or implement side-aware sampling.scripts/devtools/ab.py:802:--threadsexposes an unsafe Linux sampler path. (more on the line)scripts/devtools/ab.py:700: Every benchmark now grabs the session manifest lock and scans a growing log twice during play, then parses it again after stop. (more on the line)scripts/devtools/ab.py:703: Log line counts do not identify the frame-trace window. (more on the line)scripts/devtools/ab.py:467: Missing thread samples silently change the population used for the median. (more on the line)scripts/devtools/ab.py:278: Reading the file bypasses the bridge event limit, but both host producers still callhos_log, whose formatter truncates the message at 1,023 bytes. (more on the line)scripts/devtools/ab.py:802:--threads --set HOS_THREAD_TIMES=0is accepted and disables the sampler, although the help and docs promise five-second samples on both sides. (more on the line)scripts/devtools/ab.py:479: The Setup label checks onlycommon_envand only the literal1. (more on the line)scripts/devtools/ab.py:628: The summary discards thefailedbucket, which can contain most of the job. (more on the line)scripts/devtools/ab.py:844:job_sstarts after preparation, yet the report calls it the job's duration. (more on the line)docs/dev/developing.md:386: The write-up description reads as if Spread and the one-run rule apply to all A/Bs. (more on the line)tests/test_dev_ab.py:236: The lap test passes if every phase is recorded as zero, and the thread table test injects finished stats without exercising CLI wiring. (more on the line)tests/test_dev_ab.py:123: The migrated source check unpacks the pinned value and never checks it. (more on the line)mise run testcomplete, while the evidence is a non-strict run with 28 skips. The documented command uses--strict. Record strict results for head and base or leave the box open and name what remains. Include the required lint result and qualify the no-cost claim until profiling overhead is measured.scripts/devtools/ab.py:891: Keep the sampler opt-in and remove live log reads from ordinary runs. Its remaining opt-in cost needs a run.For the author (1) (a run, a measurement or a decision)
scripts/devtools/ab.py:891: The reported A/B enables thread profiling on both sides, so it cannot show what profiling costs. (more on the line)Checks: Still running:
CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request),Website / Website on Cloudflare (pull_request).More: does it help, other systems, switches, what was read
Does it help: Yes for the two demonstrated reporting fixes. The author reports that a single pair with a −3.7% FPS difference used to say “slower” and now says it cannot distinguish the difference from noise. The four-start Deck run changed the Setup label from a 1920×1080 window to headless and printed the new rows. The +0.4% FPS difference, 45.9 versus 45.7, compares switch settings on one runtime. The benchmark overhead and the Spread line's claimed detection rate remain unproven.
Other systems: The reporting code is shared across GPU vendors. The sampler is platform-specific. Linux enumerates /proc and guest threads, while Windows enumerates OS threads and batches log output. The Deck run exercises one Linux RADV headless path. Windowed operation, Windows logging, and automatic HOS_AB have no reported end-to-end run. The diff leaves player builds, renderer interfaces, saves, audio, and input paths untouched. Local results.json and measurements.md gain fields and rows. The checkout search found no in-tree consumer of those artifacts.
Switches:
--threadsdefaults off, which is appropriate for a benchmark tool. It normally sets the existing diagnosticHOS_THREAD_TIMESto 5 seconds on both sides.--setcan currently override or disable that value, and log collection runs even with the flag off. The host switch retains its existing off default and its row in gpu/vk/switches.def.--headlessretains its launch behavior and changes the Setup label.Reviewed:
06e96a8b5cagainst6e1521a70a: Read AGENTS.md, CONTRIBUTING.md, docs/dev/standards.md, the Forgejo template, the full changed Python files, and the changed developer docs section with its surrounding guidance. Read the three diffs and all seven commit messages. Traced the relevant session, queue, frame trace, host samplers, log writer, headless status, menu rows, and PR description checker. Skimmed the reported runs and adjacent developer docs. No repository review checklist or sibling Tooling checkout was present. The underlying run logs and results files were unavailable.; nothing run (a code review).Checked and fine:
Testing: Static review only. No repository code or tests were run. The author reports a successful four-start Deck smoke test and named shared suite failures. CI was still running, and the supplied CI status listed no head-only failures. There were no tester comments or previous reviews.
Overlaps and merge order: No open overlap could be verified from the checkout. #1870 is already on local main and changes HOS_OCB_HOLD_MS, the switch used in the Deck smoke test. It has no file overlap with this PR. Refresh the smoke test on current main before merge.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -384,3 +384,3 @@| Measure | `--seconds` (60) from the frame trace, `--runs` (3) pairs interleaved A, B, A, B. A switch whose reader takes `HOS_AB` is flipped within each run instead ([below](#an-ab-within-one-run); `--separate` to not) || Check | Refuses to write up runs that differ in a menu setting, a switch, the renderer or the window, apart from what is compared (`--allow KEY` for a row the change adds) || Write up | Each run's numbers, the medians and spread, a verdict (gain, loss, within the spread) and the Measurements section, in `runs/ab/` of the development folder (`dev.py paths runs`) and printed. Exit status 1 when a run failed, 3 when the sides differed || Write up | Each run's numbers (fps, 1% low, median and p99 frame time, frames of 50 ms or more), the medians and spread, a verdict (gain, loss, within the spread; one run a side cannot tell), a **Spread:** line (the fps difference these runs' own spread would show most times) and the Measurements section, in `runs/ab/` of the development folder (`dev.py paths runs`) and printed. `--threads` adds the busiest threads' CPU (`HOS_THREAD_TIMES=5` on both sides, read from the game's log file). Then where the time went: each run's phases (the start, the title and load, the teleport, the warm-up, the measured time, the stop) in `results.json`, and their medians in a line printed after the runs. Exit status 1 when a run failed, 3 when the sides differed |L13 (minor): The write-up description reads as if Spread and the one-run rule apply to all A/Bs. An automatic HOS_AB run uses a period-based interval and can produce a verdict from one launch. Document the separate-run and in-run outputs and explain that CPU estimates omit sub-5% samples and combine equal thread names.
@ -262,0 +275,4 @@out = []for line in text.splitlines():m = THREADS_LINE.search(line)if m:L8: Reading the file bypasses the bridge event limit, but both host producers still call
hos_log, whose formatter truncates the message at 1,023 bytes. A busy thread late in a long snapshot can disappear and then count as zero. Emit bounded records with a way to detect an incomplete snapshot, or mark truncated snapshots unavailable. Test with a long snapshot whose busiest entry comes last.@ -331,0 +405,4 @@v = spread_pct(fps["A"], fps["B"])if v is None:return "**Spread:** with %d run a side, the runs have no spread to measure: --runs 2 or more." % nreturn ("**Spread:** with %d runs a side, an fps difference of about %.1f%% or more would show most times (2.8 x the "L1: The Spread line promises a detection probability the command does not calculate. The 2.8 factor is a large-sample normal approximation for a mean-based test with known variance, while the verdict uses non-overlapping ranges and reports medians. With two identical FPS values on each side the line even promises that a 0.0% difference would show most times. Report the observed variability as a descriptive statistic, or use a small-sample method tied to the actual verdict and handle zero observed variance explicitly.
@ -370,18 +451,36 @@ def table(plan, results):rows.append("| %s | %s | %s | %s |" % (label, fmt % s0.get(k, 0), fmt % s1.get(k, 0), delta))return "\n".join(rows)L2: The in-run table returns before adding the hitch row. A normal
--env HOS_COLLECT_WIDE=0run therefore produces no 50 ms result. Add a per-side hitch metric to the retained HOS_AB periods and test both sides, accounting for unequal time spent on each side.@ -379,3 +464,4 @@rows.append("| %s | %s (median %.1f) | %s (median %.1f) | %s |" % (label, cells(va, fmt), ma, cells(vb, fmt), mb, delta))else:rows.append("| %s | %s | %s | %s |" % (label, fmt % ma, fmt % mb, delta))for name in top_threads(by["A"] + by["B"]):L7: Missing thread samples silently change the population used for the median. A valid
--seconds 10run can have no usable sample after the boundary interval is dropped, and a live read error becomes a count of zero. One missing run then disappears from its side's median, or the whole CPU table disappears while the command succeeds. Preserve missing data separately from zero, report sample coverage, and explicitly report unavailable comparisons.@ -385,0 +476,4 @@def window_text(plan, ran):"""The window a run had; "headless" with HOS_HEADLESS=1, though the game's status still reports a window size"""w = ran.get("window")if not w or plan["common_env"].get("HOS_HEADLESS") == "1":L10 (minor): The Setup label checks only
common_envand only the literal1.--set HOS_HEADLESS=onlaunches headless under the host's boolean rules and can still print window dimensions. Side overrides are also ignored. Record the effective mode for each run and render it from the results, including mixed modes if they are allowed.@ -501,0 +625,4 @@for k, v in (r.get("timing") or {}).items():phases.setdefault(k, []).append(v)return {"job_s": round(job_s, 1), "runs": len(runs),"median_s": {k: statistics.median(phases[k]) for k in TIMING_ORDER if k in phases}}L11 (minor): The summary discards the
failedbucket, which can contain most of the job. A 300-second title timeout followed by cleanup can leave a line showing only the start median. Attribute elapsed time to the active phase on failure, report cleanup separately, and cover a timeout in the timing test.@ -556,3 +698,3 @@raise drive.Failed("no frame trace at %s (RX_VK_FRAME_TRACE)" % trace)start = num(rows[-1], "at_ns")time.sleep(plan["seconds"] + 3) # (3 s more: the trace is flushed every 60 frames)live_log = session.info()["log"]L5: Every benchmark now grabs the session manifest lock and scans a growing log twice during play, then parses it again after stop. This includes runs without
--threadsand in-run comparisons that discard the data. A verbose run can compete with the game for CPU or disk at the measurement boundaries, and thread rows can appear through--setalone. Gate collection and reporting on the requested, supported mode and use bounded reads outside the measured interval.@ -559,0 +700,4 @@live_log = session.info()["log"]threads_at = count_thread_lines(live_log)time.sleep(plan["seconds"])threads_at = (threads_at, count_thread_lines(live_log))L6: Log line counts do not identify the frame-trace window. The trace starts at the last visible CSV row, while the live snapshots bracket a later wall-clock sleep. Windows queues log writes for 250 ms by default, so a sample taken just before the end can appear only after the second count. Dropping the first line cannot reconcile those clocks. Timestamp complete CPU sample intervals on the trace clock and select them after the run.
@ -644,0 +799,4 @@if a.threads:if "HOS_THREAD_TIMES" in set(a_env) | set(b_env):die("--threads sets HOS_THREAD_TIMES, which is compared here")common_env.setdefault("HOS_THREAD_TIMES", str(THREAD_EVERY_S))L4:
--threadsexposes an unsafe Linux sampler path.tasks()in host/posix/prof.c uses borrowed handles and name pointers afterhos_thread_handles()releases the registry lock. A guest thread can be destroyed beforeGetThreadId()or the name copy, and start and rename write the fields without that lock. A simultaneous--set HOS_PROFILE_EVERY=5also makes both samplers write the same static scratch arrays. Use an owned, synchronized snapshot of IDs and names and per-call scratch storage before enabling this path from the tool.L9:
--threads --set HOS_THREAD_TIMES=0is accepted and disables the sampler, although the help and docs promise five-second samples on both sides. Reject the conflicting--set, or define and validate interval overrides and report the effective interval.@ -726,5 +888,7 @@ def add_commands(sub):p.add_argument("--allow", action="append", metavar="KEY", help="a menu row (its options-file key) or switch the runs ""may differ in (a row the change adds)")p.add_argument("--dirty", action="store_true", help="measure a change with uncommitted files")p.add_argument("--threads", action="store_true", help="each thread's CPU too (HOS_THREAD_TIMES=%d on both sides): "L17 (minor): The reported A/B enables thread profiling on both sides, so it cannot show what profiling costs. The Windows sampler takes a system thread snapshot every five seconds. Run interleaved on/off pairs on the same build with a windowed Windows PC and a Linux PC, and report FPS, p99 or 1% lows, and 50 ms frame counts. Verify the output on a current runtime.
@ -682,2 +841,4 @@began = time.monotonic()runs = [run_one(plan, side, k, pinned) for side, k in plan["order"]]res = summarise(plan, runs)res["timing"] = dict(timing_summary(runs, time.monotonic() - began), before_runs_s=round(began - called, 1))L12 (minor):
job_sstarts after preparation, yet the report calls it the job's duration. Four minutes of preparation followed by one minute of runs becomes “the job took 1.0 min.” Record and label the run batch and elapsed total separately, and account for the post-stop trace copy and analysis when explaining where the minutes went.@ -120,0 +120,4 @@with open(os.path.join(menu, name), encoding="utf-8") as f:for m in re.finditer(r'\.row\(\{\.key = "(\w+)",\s*\.label = "([^"]+)"(.*?)\}', f.read(), re.S):rows[m.group(1)] = (m.group(2), m.group(3), name)for (key, value), label, field in ((ab.SKIP_TITLE, "Skip title screen", "&opts.skip_title"),L15 (minor): The migrated source check unpacks the pinned value and never checks it. It also accepts a pointer with a matching prefix and finds the V-Sync labels anywhere in the module. Moving the row to
&opts.vsync_backupor leaving an unused label array could pass. Check the complete field and tie the actual pinned value to the row's active Off mapping.@ -222,1 +233,4 @@class Timing(unittest.TestCase):"""Where a job's time goes: each run's phases, the job's medians"""def test_lap(self):L14 (minor): The lap test passes if every phase is recorded as zero, and the thread table test injects finished stats without exercising CLI wiring. Use a fake clock with distinct durations and add a game-free command or session test that fails if
--threadsstops setting the environment or if disabled collection still reads the log.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.