The frame-time rule fails and passes PRs by chance, and the farm cannot see whether frame generation ran #88
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk#88
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: measurement · Area: frame-time rules and probes (agent, report) · Part of: MagicBeanDev/Longshot#1493
TL;DR
The farm's frame-time rule compares p99 over one 20 second window; at a 60 cap that is the 13th slowest of 1,200 frames, so one more slow frame moves it by 43%. In one report main run against main differed by 11% in p99 and 29% in the 1% low, and a pull request failed at +26.9% on one run a side while its two other reports passed. We propose judging frame-time rules against main's own spread, as #87 did for pictures, with longer windows, every run's numbers shown, and probes that say whether frame generation ran. This matters now because the renderer rewrite (Longshot #1493) will lean on these gates.
Breakdown
max_delta_pctrules, which keep working when the baseline is absent.Evidence
Beanstalk read at main
ede3cdf(17 Nov 2026). Numbers are from the public threads and reports named; none was re-run for this issue except the simulation.deku, while the rule that failed was onfield. "The median of 2 reports" is their mean.screenshot.max_diff(0.0506 against 0.05), which #87 fixed.vi: present intervalslines, frames drawn independently, so a floor under real noise): a 10% p99 gate fails with no change 4.0% (9070 XT), 0.0% (4070) and 13.9% (3070) of the time; the 1% low fails 0.3%, 27.6% and 6.4%. Real windows are less steady than independent frames.frame interpolation:line and says the render thread's jobs take 7.66 ms of a 16.67 ms frame with 9.01 ms idle. A running 2x or 4x writes two to four frame-log rows per game frame (made frames end in present_finish, gpu/vk/vk_replay.cpp:6019-6090 and 6100 on Longshot main 310e7c9df), so 16.67 ms medians point to the game's frames alone. That is an inference from code and numbers; the report does not say which engine ran.Proposed change
inconclusive, notfail(as #1 asked under "Per-PR test plan, pass rules").frame interpolation:lines and thevk: swapchainline from the log, and report the engine, the frames made and shown on time, and the present mode. A cell that asked for frame generation and made none becomes amismatch, as a setting read back differently already does (docs/test-plan.md:493).process,drawsandfps_meanprobes), so renderer cost is not hidden by the cap.How we would test it
frametime_p99_ms.max_delta_pct = 10with main's runs 19.39 and 21.59 and the PR's 22.0 and 23.2 fails today (+10.3% by the medians) and must pass with the spread of 11.3% times 1.5; one start a side must giveinconclusive. A snippet that reproduces the cliff:frame interpolation:line and frame generation asked givesmismatch.Related
New on this forge; not opened on GitHub.
Update from the telemetry side, relevant to proposals 2 and 4: Longshot PR #1530 (MagicBeanDev/Longshot#1530) now records the missing presentation facts in the game itself.
present_mode,vsyncandframegenlabels, so a report can say which present mode and frame-generation setting a cell actually used instead of inferring it from the p50 (proposal 4'smismatchcase).fine_ms): extra bucket edges at 15.5/16/16.7/17.5/18.5 ms, which stop a perfectly paced 60 fps map reading 49-50 fps from bucket boundaries.RX_VK_FRAME_TRACE) now hasshown_ms(present-to-shown fromVK_KHR_present_wait) andshown_at_ns(the absolute host clock of that confirmation), so a probe could read the actual presentation moment from the log rather than guessing from frame-log rows.The farm's sandbox has no network by design, so none of this reaches the relay from an agent — the probe would read the local log/summary, as the existing probes do. The labels and fine histogram are in the relay's schema (already deployed); the game build is still in PR #1530.
I have not changed any farm rule here: the noise baseline, 60 s windows and per-run numbers in proposals 1-3 are still the fix for the false fails the issue describes.
I reproduced the one-run p99 failure with synthetic frame times. Both 1,200-frame windows had a median of 17.4 ms. With 12 frames at 25 ms, p99 was 17.476 ms. With 13, it was 25.000 ms, or +43.1%, which trips the 20% limit. This shows how a same-build comparison can fail when it picks up one extra slow frame. Estimating how often that happens on real hardware still needs same-build runs.
The smallest change I’d start with is to require two usable starts per arm for the same timing metric, segment and settings. If either arm has fewer, report the rule as inconclusive and show the available values and counts. Carry that outcome through the overall report and have the coordinator accept and display it. A crash or another hard failure should still fail the report. Keep the existing percentage limit for comparisons with enough evidence and calibrate it separately against same-build runs on each PC and scene.
A fixture should check that one usable run per arm is inconclusive even at +43.1%, that two usable runs per arm still fail at that delta, and that a crash still fails a report whose timing result is inconclusive.
The timing probes and rule and report validation checked for this review are unchanged on main at
e4316243. Longshot’s measurement proposal covers the calibration work.Picking up the smaller verdict change from the synthetic reproduction above. I’ll cover fewer than two usable starts per arm for a required frame-time comparison, with a game-free fixture that verifies inconclusive propagates to the report and hard failures still fail. Same-build threshold calibration follows after the verdict path is in place.
The local implementation now reports a required frame-time percentage comparison as inconclusive when either arm has fewer than two usable starts for the same metric, segment, cell and variant. It shows the usable counts and available values, and the report carries an unjudged outcome. Hard failures still take priority. Sufficiently sampled comparisons keep their existing thresholds.
The synthetic +43.1% case now reports inconclusive with one start per arm and still fails at the 20% limit with two. The local validation passed 1,921 agent core checks, six Windows helper checks and 385 coordinator tests. Windows symlink permissions prevented the setup section from running in the disposable copy. The coordinator run excluded 182 Postgres-dependent checks and had one skip. Roll out coordinator support for the new rule status before updated agents emit it.
The same-build hardware pilot is prepared. No hardware noise measurements yet. Calibration follow-up