[Tracking] Six-hour main rebuilds and in-game performance benchmarks #2595
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
2 participants
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot#2595
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
[Tracking] Six-hour main rebuilds and in-game performance benchmarks
This issue tracks a planned six-hour full-rebuild and gameplay record for official
main, alongside the Linux companion measurements in the comments. Each Windows cycle will rebuild even when the commit has not changed and identify the exact tested revision. Windows scheduling and publication of run results are currently paused pending review of the completed local baseline and runner.Fixed starting baseline
The fixed starting-main baseline is
55b976c0e15f38fc10dcac757e332800d09b7c24. This replaces the originally proposed v2.1.0 release baseline. It is a starting reference, not a claim that this revision is regression-free.Correction — November 12, 2026: this SHA is immediately before the readback fix in #2602. The fix's merge commit
76bbe4aahas55b976c0as its sole parent. The fixed reference stays unchanged; it must not be described as already including that fix.Later comparable runs will be compared against both this fixed baseline and the previous comparable successful run, with absolute and percentage changes. The baseline will not slide forward with main. Three initial gameplay passes per phase have been collected locally to establish an average and observed variation; routine cycles may use one pass as a screening observation. A historical-reference difference alone is not a confirmed regression. Build repeat count and unmeasured build-time variance will be explicit.
Build measurements
Gameplay protocol
Hardware: Windows 11, Intel i7-13700F, RTX 4070 12 GB, 32 GB RAM. The current GPU check reports NVIDIA driver 621.01; each run records the driver again.
Fixed graphics: a non-activating borderless Windows window with verified 2560×1440 client/source/output, 16:9, 100% render resolution, dynamic resolution and handheld profile off, the game's own upscaler. DLSS, DLAA, FSR, frame generation, HDR, Reflex, Anti-Lag and V-Sync are off; game FPS is uncapped with real-time game speed. Quality, FOV, sensitivity and presentation mode remain fixed for comparable runs.
Four separate 60-second wall-clock phases, with gameplay running:
Saved checkpoints restore the approximate location and camera view. Field uses the preserved daytime checkpoint. Each checkpoint is copied into a disposable user folder; original saves/settings are preserved. The accepted motion uses 60% forward analog deflection with full right-stick rotation and camera speed 200%. Inputs are exclusive during stationary and motion phases and released at a monotonic deadline. Frame-based bridge holds are only an upper bound, not the measurement clock.
Startup, loading, a stationary warm-up and the complete camera-circle preconditioning are outside timing. Measurements use fresh checkpoint loads after preconditioning. Per-build game caches start fresh and are warmed; the local driver cache persists. First-visit compilation is excluded from steady-state numbers and is not yet a separately quantified metric.
The runner verifies effective options, actual client/output dimensions, expected map, non-null gameplay state, event/loading flags, stalls, exclusive inputs and competing CPU load, rejecting invalid phases. Screenshots for route review occur outside measured intervals; no injected overlays, screenshot readbacks or sampled profiling run inside them.
Mean FPS comes from aligned per-frame game-present return intervals and frame count/time. Median, p95 and p99 intervals come from those samples, never from once-per-second bridge FPS summaries. These are game-present return timings, not direct GPU duration or monitor scanout. All four workloads stay separate; repeats and observed spread accompany the averages. Reports will also show each measured pass, sample count/span, 1% low, frames strictly over 20/25 ms, longest interval, and competing CPU mean/peak with eligibility. Here, 1% low means 1000 divided by the mean of the slowest ceil(N/100) intervals, matching the workbench nightly reader; percentiles use nearest rank.
Related work and credit
Credit to mooki000’s hookshot-workbench for its public harness, agent guidance, comparison methods and reusable tools. We reviewed its repository description, skills, agent brief, publication checks and relevant measurement/rolling code at revision f45e4a3f. Our local runner reuses its MIT-licensed frame-statistics and empirical-band helpers, preserving the license and attribution.
This project adds fixed-cadence Windows measurements to that Linux testing. The reports use the same game-present timing boundary, explicit cache/warm-up policy, quiet-load principle, exact build identities, per-pass evidence and honest missing/failed results. Windows CPU accounting uses system/process counters; quiet means other work below three cores on average with no approximately five-second sample above six cores.
The results should be read alongside each other as within-machine trends, not pooled or compared as raw FPS/build seconds. CPU/GPU, OS/driver, compiler, worker count, scene/save, actual render/output size and windowed versus headless presentation can differ. Our Windows builds include the locally accepted NGX SDK, with runtime DLSS/DLAA off. The Linux companion reports also include a separate DLAA test. SDK/build configuration and runtime settings must be recorded per tested build, rather than inferred from an older recipe. Per-thread CPU, GPU duration, peak memory and Linux kernel-driver diagnostics are not currently equivalent measurements in this runner.
The workbench’s paired new/previous/new/previous checks are stronger evidence for a regression than a single scheduled sample. Its first companion report distinguishes paired main comparisons from dated release samples; its next cycle judges changes against the spread between repeats. These provide complementary evidence, not Windows results.
Different platforms, cards/VRAM budgets and scenes broaden coverage. Agreement in within-machine trends is useful corroboration. A disagreement alone cannot identify the OS as the cause when hardware, scene, resolution, driver and presentation also differ. Neither raw FPS nor build seconds will be pooled across the two PCs.
Methodology changes before Windows scheduling resumes
These are agreed directions for the next revision of the protocol; they are not claims that the runner already implements or has measured them.
A particularly useful proposed next Windows comparison is the retained
55b976c0build against its direct child76bbe4aa(#2602), alternating on this PC. That would test the fix on Windows without including the many later merges in the sampled range. This experiment has not been run. No automatic bisect or second scheduled campaign is being started.Current status and reporting
The fixed starting-baseline full build and three measured passes for each of the four Windows phases have completed locally. Client/source/output dimensions, frame traces, saved-checkpoint manifests and clean process exits were checked. The full game-free suite still has remaining failures documented locally; successful build/gameplay validation does not mean the suite is green. Detailed Windows result tables remain unpublished pending review. Calibration FPS is not benchmark evidence.
Individual run reports will be concise, deduplicated comments here, including failures, UTC time, SHA/commit link, build stages, each valid gameplay phase, comparisons, repeat variation, configuration and candidate intervening commit range. A range between samples is not proof of causation, and build time alone is not evidence of an in-game regression.
Game files, generated code, binaries, caches, saves, maps and raw traces stay on the local PC. Public reports contain reviewed measurements and permitted log excerpts only.
A Linux companion to the Windows cycles, from this PC's rolling rebuilds (the workbench tool the issue credits): the v2.1.0 release, main before the readback fix, and main as of 2026-11-12 08:14 UTC, each built fresh from source and measured in the same three scenes.
Machine: Arch Linux (kernel 7.2.15), i9-13900K, 94 GB, RTX 4090 24 GB, NVIDIA 615.78.08; no window (frames are the game's offscreen-present intervals, not scanout); 2048x1152.
Builds:
8ef84611(v2.1.0),ef81aee7(main of 2026-11-11, before #2602),46780503(main, with #2602 and #2698). Clang 23.1.3, LLD, 16 compile and 8 lift workers, release settings with the recorded profile.46780503What it says for this machine:
How it was measured, and where it differs from this issue's protocol: each scene is one hold on the card with the two main builds alternating (new, before, new, before), a prepared cache and an unmeasured warm-up first, 30 s measured from 15 s after the load; all runs counted as quiet (other work under 3 cores on average, repeats within 5 %). The same opt-in readback logging runs on all three builds; its cost is not subtracted. The release's runs come from separate holds earlier the same night (2026-11-12 05:29 to 08:14 UTC), so the last column is arithmetic between dated samples, not a paired verdict. The route is this PC's timed input sweep, not the four standard phases, with a 240 cap on the uncapped arms (the measured rates stay well below it) and the camera not pixel-locked. So: a trend within this machine, not numbers to pool with the Windows ones.
Build wall time: 649 s, 753 s, 568 s for the three, under different machine loads; not a compile-speed comparison.
The tool and its report format:
github.com/mooki000/hookshot-workbench@0febe1c99b/nightlyNext Linux cycle, same machine and method as above: main
51da0464(2026-11-12 11:19 UTC) against the previous cycle's main46780503, 6 commits apart, the two builds alternating in one hold per scene.4678050351da0464No change: all three differences are inside the spread between repeats (the tool's verdict for each is "within repeat spread"). All twelve runs counted as quiet. Build wall time 571 s (568 s last cycle).
Windows A/B results for three perf PRs, posted here to consolidate alongside the rolling cycles (requested in #dev-work).
Machine: Windows 11 build 26200, Intel i5-14600KF, NVIDIA RTX 5070 12 GB, driver 32.0.16.2114, Vulkan back end, warm caches. Control = the v2.1.0 release, not the fixed baseline
55b976c0. Different hardware, scenes and settings from this issue's four-phase protocol and from mooki000's Linux cycles: these are within-machine paired PR checks only, not poolable and not main-regression tracking.Method: each PR is a paired control/patch A/B on the same route — rainy Kokiri Forest clearing, 80 s window after frame 3000, 2 clean runs per build, avg / 1% low / p99 from the game-frame trace. 2560x1440 output, frame generation on, 240 cap, default quality — except #2180 which runs a 3840x2160 output. #2156 and #2238 run under a synthetic 4700 MB allowance (
RX_VK_TEST_ALLOWED_MB+RX_VK_VRAM_MB); the real driver allowance is 11175 MB, so the pressure path is exercised without real paging.not squeezed12 -> 0 over the same harness = the fix itself134b3abrender targetsmade new2441 -> 1339 mean, let-go +38% over control, pool reuse +44%, max frame 35.9 -> 23.0 ms(#2242 + #2218 has no in-game surface — headless prepare path, no numbers to post.)
Limits, stated the way this issue asks: a 12 GB card cannot replay #2156's 8 GB paging repro (the honest evidence here is the squeeze state lines, not fps); the synthetic allowance never pages on this driver; #2180's original 16 GB AMD repro was not hit (graveyard save with headroom); none of these runs uses the four phases or the fixed baseline. Commit causation is not claimed from paired PR diffs alone.
Full run reports live in the fork PRs: https://github.com/Revens2/Hookshot/pull/6 (#2238), https://github.com/Revens2/Hookshot/pull/8 (#2156), https://github.com/Revens2/Hookshot/pull/9 (#2180), all heading upstream. Happy to keep dropping PR-level A/Bs here when they help alongside the rolling cycles.
PASS — 2026-11-12T14:03:22.277990+00:00
Commit: 51da0464b4cf
Protocol:
windows-checkpoints-v2; Windows 11, i7-13700F, RTX 4070; NVIDIA GeForce RTX 4070, 621.01.Version:
v2.1.0-258-g51da0464 (source 18ed87b5).Compiler:
clang version 23.1.5 (https://github.com/llvm/llvm-project.git 392dfff97d29b59d77e5cac6d4b6ef2585fe7117).Full build: 1231.55 s (20.53 min), one repeat; six compiler workers; release cold O1 / hot O2 + shipped PGO.
Stages (wall seconds): read 106.67, lift 108.67, finish 133.22, compilation 879.89, linking 1.83, startup 1.26.
Fetch/setup, tests and shader preconditioning excluded. Build repeat variance is not measured.
Lift process observations: lift: emit and write (16 processes) in 23.8s (63s); lift: patches (16 processes) in 25.7s (90s).
Build versus fixed starting baseline: -71.75 (-5.51%) s.
Build versus previous comparable success: -71.75 (-5.51%) s.
Game-present return intervals, frame-count/time mean; neither GPU duration nor monitor scanout. Percentiles shown are per-pass percentiles averaged across repeats.
2560×1440 borderless client/output, 16:9, fixed 100%, dynamic resolution off, game’s own upscaler; DLSS/DLAA/FSR/FG/HDR/Reflex/Anti-Lag/V-Sync off, uncapped, real-time speed. Fresh checkpoint loads; stationary and complete camera-circle preconditioning outside timing.
Driver cache is persistent; per-build game caches start fresh and are warmed. First-visit compilation is excluded. Single-pass changes are candidates only; repeat suspicious results before calling a regression.
1% low = 1000 / mean of the slowest ceil(N/100) frame intervals, matching the workbench nightly reader. Slow-frame thresholds are strictly greater than 20/25 ms. No peak-memory, GPU-duration or per-thread CPU claim is made.
Validation and observations
All expected phases completed with valid trace coverage/alignment, checkpoint/settings checks and background CPU eligibility.
game_language=0andmenu_language=0, which were absent from the baseline configuration snapshot. These additions are not expected to affect the checkpoint performance tests; baseline comparisons are included under this disclosed assumption. Other configuration/instrumentation checks remain strict.Intervening commits; candidate range, not proof of causation.
No visual regression claim: measured intervals have no screenshot readbacks. Build time is separate from in-game speed.
Method and metric-reader credit: mooki000’s hookshot-workbench, reviewed at
f45e4a3f; reused helpers retain the MIT license. This Windows route complements its Linux testing. Different hardware/scenes/presentation modes make absolute FPS and build seconds unsuitable for direct comparison; compare within-machine trends on exact commits. A stronger regression conclusion needs clean alternating old/new runs.Third Linux cycle, same PC and method as my two above (RTX 4090, NVIDIA 615.78.08, i9-13900K; no window; two alternated runs a scene in one exclusive hold). Main had not moved since the last cycle (
51da0464), so this one is a fresh rebuild of the same commit measured against the previous cycle's build of it: a check of how far two builds of identical source differ here.51da046451da0464All three are inside the spread between repeats, as they should be. So on this PC a difference under about 1% between two cycles says nothing; the 20% and 35% steps reported earlier in this thread are far outside that.
Peak resident memory was again 16.7 to 17.5 GiB in every run (v2.1.0: 13.3 to 13.7 GiB in the first cycle). The fresh build took 1,323 s, more than twice the last one, because the PC was busy with other builds: not a build-speed signal.
The next cycle will have a new main to compare (several pull requests have merged since).