The frame-time rule fails and passes PRs by chance, and the farm cannot see whether frame generation ran #88

Open
opened 2026-11-18 01:53:46 +00:00 by mooki · 4 comments
Member

Type: measurement · Area: frame-time rules and probes (agent, report) · Part of: MagicBeanDev/Longshot#1493

TL;DR

The farm's frame-time rule compares p99 over one 20 second window; at a 60 cap that is the 13th slowest of 1,200 frames, so one more slow frame moves it by 43%. In one report main run against main differed by 11% in p99 and 29% in the 1% low, and a pull request failed at +26.9% on one run a side while its two other reports passed. We propose judging frame-time rules against main's own spread, as #87 did for pictures, with longer windows, every run's numbers shown, and probes that say whether frame generation ran. This matters now because the renderer rewrite (Longshot #1493) will lean on these gates.

Breakdown

  • Who it affects: pull request authors (a false fail costs a day, a false pass hides a regression) and the renderer rewrite, whose gates are frame-time rules on this farm.
  • What is wrong: p99 is a cliff statistic at this window size; frame-time rules have no noise baseline (pictures have had one since #87); one game start a side cannot support a 10 to 20% gate; the capped 60 cells hide renderer cost, and the frame generation cells probably never ran frame generation (inferred from the code and the numbers below, not confirmed).
  • What we propose: a noise baseline for numeric rules, 60 second windows, several statistics together, per-run numbers in the report, a probe for frame generation and present mode, and an uncapped cost cell.
  • What does not change: ABBA order, per-report judgement, the screenshot rule, and the existing max_delta_pct rules, which keep working when the baseline is absent.
  • How we would prove it: game-free checks on the rule and the probe, and a main-against-main measurement (10 pairs on three PCs, 20 s and 60 s windows) that counts false fails before and after.

Evidence

Beanstalk read at main ede3cdf (17 Nov 2026). Numbers are from the public threads and reports named; none was re-run for this issue except the simulation.

  • p99 is the 13th slowest frame. The probe takes numpy's interpolated percentile (agent/Core/Probes.cs:20-46). A 20 s window at 60 is 1,200 frames, so p99 sits at index 1187.01: with 12 frames slower than the steady ones it still reads the steady value, and the 13th moves it. A simulation (steady 17.4 ms, timer lateness up to 0.8 ms): 12 frames of 25 ms leave p99 at 17.53 ms and a 13th makes it 25.00 (+43%). With 33.3 ms frames: 17.61, then 33.30. The mean of the slowest 1% moves smoothly (17.40, 21.21, 24.37, 25.00 for 0, 6, 11 and 13 slow frames).
  • Longshot #1462's report (limit 20%, one start a side, field 20 s after 5 s warm-up, 60 cap, Windows): RTX 4070 field p99 18.27 to 23.19 ms (+26.9%, fail). RX 9070 XT 19.93 to 20.63 (+3.5%, pass). RTX 3070 19.88 to 19.89 (+0%, pass). The review's reading (quoted in Longshot #1489) is that main's side of the 4070 run was unusually smooth (field max 20.7 ms against 47 to 88 ms on main for the other PCs) and p50 did not move (16.67 to 16.62). The comment's headline row said +2%: it shows each report's first measure segment (coordinator/beanstalk/reporter.py:194-234), deku, while the rule that failed was on field. "The median of 2 reports" is their mean.
  • Longshot #1458's report (RX 9070 XT, two starts a side, field 30 s after 10 s warm-up, limit 15%): main's two runs of one cell differed by up to +11.3% in p99 (19.39 and 21.59 ms, V-Sync 1, frame generation 4), +12.2% in the 1% low there, and at V-Sync 0, frame generation 4: +28.8% in the 1% low (19.66 and 25.32), +98% in max (20.54 and 40.73), stutters 0 and 5. The first cell's apparent improvement of 5% in p99 (20.5 to 19.5 in the report's medians) came from main's two runs being far apart. The same report failed on screenshot.max_diff (0.0506 against 0.05), which #87 fixed.
  • A simulation from the reports' own histograms (the vi: present intervals lines, frames drawn independently, so a floor under real noise): a 10% p99 gate fails with no change 4.0% (9070 XT), 0.0% (4070) and 13.9% (3070) of the time; the 1% low fails 0.3%, 27.6% and 6.4%. Real windows are less steady than independent frames.
  • The capped cells hide cost and maybe frame generation. All four #1458 cells have p50 16.63 to 16.72 ms. The 12 s of log the report keeps has no frame interpolation: line and says the render thread's jobs take 7.66 ms of a 16.67 ms frame with 9.01 ms idle. A running 2x or 4x writes two to four frame-log rows per game frame (made frames end in present_finish, gpu/vk/vk_replay.cpp:6019-6090 and 6100 on Longshot main 310e7c9df), so 16.67 ms medians point to the game's frames alone. That is an inference from code and numbers; the report does not say which engine ran.

Proposed change

  1. Judge frame-time rules against main's own spread (extends #87, agent/Core/Rules.cs:208-210 and 263-270). With two or more starts a side, the spread is main's runs against each other. A rule fails only when the delta is over its limit and over 1.5 times that spread; the report shows the spread as the rule's baseline, with a note when it saved the PR. With one start a side the result for a frame-time rule is inconclusive, not fail (as #1 asked under "Per-PR test plan, pass rules").
  2. Longer windows, several statistics. Measure 60 s (3,600 frames at 60, so p99 is the 36th slowest). Report p99, the mean of the slowest 1% and the count of frames over twice the median together, and never gate on p99 alone. The planner aims at 20 minutes a plan (the limit is 60): a 60 s window adds 40 s to each measure step, per run.
  3. Show every run, not a median of medians. List each run's value for both sides and the same-side spread. The headline row names the segment that failed.
  4. A frame generation probe. Read the frame interpolation: lines and the vk: swapchain line from the log, and report the engine, the frames made and shown on time, and the present mode. A cell that asked for frame generation and made none becomes a mismatch, as a setting read back differently already does (docs/test-plan.md:493).
  5. An uncapped cost cell (cap 1000 with the existing process, draws and fps_mean probes), so renderer cost is not hidden by the cap.
  6. A per-PC null pair. The ABBA order already runs main twice. Keep each PC's main-against-main spread per scenario and print it in every report.

How we would test it

  • Game-free, red today: a synthetic frame log of 1,200 rows with 12, then 13, frames of 25 ms (the probe's p99 jumps +43%); a rule frametime_p99_ms.max_delta_pct = 10 with main's runs 19.39 and 21.59 and the PR's 22.0 and 23.2 fails today (+10.3% by the medians) and must pass with the spread of 11.3% times 1.5; one start a side must give inconclusive. A snippet that reproduces the cliff:
import random
random.seed(1)
def p99(xs):
    xs = sorted(xs); pos = (len(xs) - 1) * 0.99; lo = int(pos)
    return xs[lo] + (xs[min(lo + 1, len(xs) - 1)] - xs[lo]) * (pos - lo)
base = [16.7 + random.random() * 0.8 for _ in range(1200)]
for k in (0, 12, 13): print(k, round(p99(base[:1200 - k] + [25.0] * k), 2))   # 17.49, 17.57, 25.0
  • Game-free: a log with no frame interpolation: line and frame generation asked gives mismatch.
  • Farm run: 10 pairs of the same build (a null pair) in the #1462 field and Deku cells on an RTX 4070, an RX 9070 XT and an RTX 3070, at 20 s and 60 s, counting false fails of today's rule, then of the new one (target: at most 1%). Then a PR arm with a known cost added (the host has a diagnostic frame sleep, HOS_FRAME_SLEEP_MS) to show a real loss still fails.
  • Beanstalk #87 (pictures judged against main's own noise: the pattern to extend) and #1 (item 5, "same machine and same session isn't enough for timing", asks for an inconclusive result and a minimum number of runs); #82 (measure around steps).
  • Longshot #1458, #1462 and #1489 (the numbers above), and #1493 (the renderer rewrite: Step 0, item 4, and its rule of paired timing after a main-against-main null pair).
  • Searched all 8 Beanstalk issues and its pull requests, and 1,495 Longshot issues and pull requests, open and closed, for p99, null pair, same-arm, noise, false fail and farm. No item proposes a noise baseline for frame times; #1 item 5 names the problem in one paragraph.

New on this forge; not opened on GitHub.

**Type:** measurement · **Area:** frame-time rules and probes (agent, report) · **Part of:** MagicBeanDev/Longshot#1493 ## TL;DR The farm's frame-time rule compares p99 over one 20 second window; at a 60 cap that is the 13th slowest of 1,200 frames, so one more slow frame moves it by 43%. In one report main run against main differed by 11% in p99 and 29% in the 1% low, and a pull request failed at +26.9% on one run a side while its two other reports passed. We propose judging frame-time rules against main's own spread, as #87 did for pictures, with longer windows, every run's numbers shown, and probes that say whether frame generation ran. This matters now because the renderer rewrite (Longshot #1493) will lean on these gates. ## Breakdown - **Who it affects:** pull request authors (a false fail costs a day, a false pass hides a regression) and the renderer rewrite, whose gates are frame-time rules on this farm. - **What is wrong:** p99 is a cliff statistic at this window size; frame-time rules have no noise baseline (pictures have had one since #87); one game start a side cannot support a 10 to 20% gate; the capped 60 cells hide renderer cost, and the frame generation cells probably never ran frame generation (inferred from the code and the numbers below, not confirmed). - **What we propose:** a noise baseline for numeric rules, 60 second windows, several statistics together, per-run numbers in the report, a probe for frame generation and present mode, and an uncapped cost cell. - **What does not change:** ABBA order, per-report judgement, the screenshot rule, and the existing `max_delta_pct` rules, which keep working when the baseline is absent. - **How we would prove it:** game-free checks on the rule and the probe, and a main-against-main measurement (10 pairs on three PCs, 20 s and 60 s windows) that counts false fails before and after. ## Evidence Beanstalk read at main ede3cdf (17 Nov 2026). Numbers are from the public threads and reports named; none was re-run for this issue except the simulation. - **p99 is the 13th slowest frame.** The probe takes numpy's interpolated percentile (agent/Core/Probes.cs:20-46). A 20 s window at 60 is 1,200 frames, so p99 sits at index 1187.01: with 12 frames slower than the steady ones it still reads the steady value, and the 13th moves it. A simulation (steady 17.4 ms, timer lateness up to 0.8 ms): 12 frames of 25 ms leave p99 at 17.53 ms and a 13th makes it 25.00 (+43%). With 33.3 ms frames: 17.61, then 33.30. The mean of the slowest 1% moves smoothly (17.40, 21.21, 24.37, 25.00 for 0, 6, 11 and 13 slow frames). - **Longshot #1462's report (limit 20%, one start a side, field 20 s after 5 s warm-up, 60 cap, Windows):** RTX 4070 field p99 18.27 to 23.19 ms (+26.9%, fail). RX 9070 XT 19.93 to 20.63 (+3.5%, pass). RTX 3070 19.88 to 19.89 (+0%, pass). The review's reading (quoted in Longshot #1489) is that main's side of the 4070 run was unusually smooth (field max 20.7 ms against 47 to 88 ms on main for the other PCs) and p50 did not move (16.67 to 16.62). The comment's headline row said +2%: it shows each report's first measure segment (coordinator/beanstalk/reporter.py:194-234), `deku`, while the rule that failed was on `field`. "The median of 2 reports" is their mean. - **Longshot #1458's report (RX 9070 XT, two starts a side, field 30 s after 10 s warm-up, limit 15%):** main's two runs of one cell differed by up to +11.3% in p99 (19.39 and 21.59 ms, V-Sync 1, frame generation 4), +12.2% in the 1% low there, and at V-Sync 0, frame generation 4: +28.8% in the 1% low (19.66 and 25.32), +98% in max (20.54 and 40.73), stutters 0 and 5. The first cell's apparent improvement of 5% in p99 (20.5 to 19.5 in the report's medians) came from main's two runs being far apart. The same report failed on `screenshot.max_diff` (0.0506 against 0.05), which #87 fixed. - **A simulation from the reports' own histograms** (the `vi: present intervals` lines, frames drawn independently, so a floor under real noise): a 10% p99 gate fails with no change 4.0% (9070 XT), 0.0% (4070) and 13.9% (3070) of the time; the 1% low fails 0.3%, 27.6% and 6.4%. Real windows are less steady than independent frames. - **The capped cells hide cost and maybe frame generation.** All four #1458 cells have p50 16.63 to 16.72 ms. The 12 s of log the report keeps has no `frame interpolation:` line and says the render thread's jobs take 7.66 ms of a 16.67 ms frame with 9.01 ms idle. A running 2x or 4x writes two to four frame-log rows per game frame (made frames end in present_finish, gpu/vk/vk_replay.cpp:6019-6090 and 6100 on Longshot main 310e7c9df), so 16.67 ms medians point to the game's frames alone. That is an inference from code and numbers; the report does not say which engine ran. ## Proposed change 1. **Judge frame-time rules against main's own spread** (extends #87, agent/Core/Rules.cs:208-210 and 263-270). With two or more starts a side, the spread is main's runs against each other. A rule fails only when the delta is over its limit and over 1.5 times that spread; the report shows the spread as the rule's baseline, with a note when it saved the PR. With one start a side the result for a frame-time rule is `inconclusive`, not `fail` (as #1 asked under "Per-PR test plan, pass rules"). 2. **Longer windows, several statistics.** Measure 60 s (3,600 frames at 60, so p99 is the 36th slowest). Report p99, the mean of the slowest 1% and the count of frames over twice the median together, and never gate on p99 alone. The planner aims at 20 minutes a plan (the limit is 60): a 60 s window adds 40 s to each measure step, per run. 3. **Show every run, not a median of medians.** List each run's value for both sides and the same-side spread. The headline row names the segment that failed. 4. **A frame generation probe.** Read the `frame interpolation:` lines and the `vk: swapchain` line from the log, and report the engine, the frames made and shown on time, and the present mode. A cell that asked for frame generation and made none becomes a `mismatch`, as a setting read back differently already does (docs/test-plan.md:493). 5. **An uncapped cost cell** (cap 1000 with the existing `process`, `draws` and `fps_mean` probes), so renderer cost is not hidden by the cap. 6. **A per-PC null pair.** The ABBA order already runs main twice. Keep each PC's main-against-main spread per scenario and print it in every report. ## How we would test it - **Game-free, red today:** a synthetic frame log of 1,200 rows with 12, then 13, frames of 25 ms (the probe's p99 jumps +43%); a rule `frametime_p99_ms.max_delta_pct = 10` with main's runs 19.39 and 21.59 and the PR's 22.0 and 23.2 fails today (+10.3% by the medians) and must pass with the spread of 11.3% times 1.5; one start a side must give `inconclusive`. A snippet that reproduces the cliff: ```python import random random.seed(1) def p99(xs): xs = sorted(xs); pos = (len(xs) - 1) * 0.99; lo = int(pos) return xs[lo] + (xs[min(lo + 1, len(xs) - 1)] - xs[lo]) * (pos - lo) base = [16.7 + random.random() * 0.8 for _ in range(1200)] for k in (0, 12, 13): print(k, round(p99(base[:1200 - k] + [25.0] * k), 2)) # 17.49, 17.57, 25.0 ``` - **Game-free:** a log with no `frame interpolation:` line and frame generation asked gives `mismatch`. - **Farm run:** 10 pairs of the same build (a null pair) in the #1462 field and Deku cells on an RTX 4070, an RX 9070 XT and an RTX 3070, at 20 s and 60 s, counting false fails of today's rule, then of the new one (target: at most 1%). Then a PR arm with a known cost added (the host has a diagnostic frame sleep, HOS_FRAME_SLEEP_MS) to show a real loss still fails. ## Related - Beanstalk #87 (pictures judged against main's own noise: the pattern to extend) and #1 (item 5, "same machine and same session isn't enough for timing", asks for an inconclusive result and a minimum number of runs); #82 (measure around steps). - Longshot #1458, #1462 and #1489 (the numbers above), and #1493 (the renderer rewrite: Step 0, item 4, and its rule of paired timing after a main-against-main null pair). - Searched all 8 Beanstalk issues and its pull requests, and 1,495 Longshot issues and pull requests, open and closed, for p99, null pair, same-arm, noise, false fail and farm. No item proposes a noise baseline for frame times; #1 item 5 names the problem in one paragraph. New on this forge; not opened on GitHub.
Member

Update from the telemetry side, relevant to proposals 2 and 4: Longshot PR #1530 (MagicBeanDev/Longshot#1530) now records the missing presentation facts in the game itself.

  • v2 summaries carry present_mode, vsync and framegen labels, so a report can say which present mode and frame-generation setting a cell actually used instead of inferring it from the p50 (proposal 4's mismatch case).
  • v2 also carries a fine 60 Hz frame-time histogram (fine_ms): extra bucket edges at 15.5/16/16.7/17.5/18.5 ms, which stop a perfectly paced 60 fps map reading 49-50 fps from bucket boundaries.
  • The per-frame trace (RX_VK_FRAME_TRACE) now has shown_ms (present-to-shown from VK_KHR_present_wait) and shown_at_ns (the absolute host clock of that confirmation), so a probe could read the actual presentation moment from the log rather than guessing from frame-log rows.

The farm's sandbox has no network by design, so none of this reaches the relay from an agent — the probe would read the local log/summary, as the existing probes do. The labels and fine histogram are in the relay's schema (already deployed); the game build is still in PR #1530.

I have not changed any farm rule here: the noise baseline, 60 s windows and per-run numbers in proposals 1-3 are still the fix for the false fails the issue describes.

Update from the telemetry side, relevant to proposals 2 and 4: Longshot PR #1530 (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1530) now records the missing presentation facts in the game itself. - v2 summaries carry `present_mode`, `vsync` and `framegen` labels, so a report can say which present mode and frame-generation setting a cell actually used instead of inferring it from the p50 (proposal 4's `mismatch` case). - v2 also carries a fine 60 Hz frame-time histogram (`fine_ms`): extra bucket edges at 15.5/16/16.7/17.5/18.5 ms, which stop a perfectly paced 60 fps map reading 49-50 fps from bucket boundaries. - The per-frame trace (`RX_VK_FRAME_TRACE`) now has `shown_ms` (present-to-shown from `VK_KHR_present_wait`) and `shown_at_ns` (the absolute host clock of that confirmation), so a probe could read the actual presentation moment from the log rather than guessing from frame-log rows. The farm's sandbox has no network by design, so none of this reaches the relay from an agent — the probe would read the local log/summary, as the existing probes do. The labels and fine histogram are in the relay's schema (already deployed); the game build is still in PR #1530. I have not changed any farm rule here: the noise baseline, 60 s windows and per-run numbers in proposals 1-3 are still the fix for the false fails the issue describes.
Member

I reproduced the one-run p99 failure with synthetic frame times. Both 1,200-frame windows had a median of 17.4 ms. With 12 frames at 25 ms, p99 was 17.476 ms. With 13, it was 25.000 ms, or +43.1%, which trips the 20% limit. This shows how a same-build comparison can fail when it picks up one extra slow frame. Estimating how often that happens on real hardware still needs same-build runs.

The smallest change I’d start with is to require two usable starts per arm for the same timing metric, segment and settings. If either arm has fewer, report the rule as inconclusive and show the available values and counts. Carry that outcome through the overall report and have the coordinator accept and display it. A crash or another hard failure should still fail the report. Keep the existing percentage limit for comparisons with enough evidence and calibrate it separately against same-build runs on each PC and scene.

A fixture should check that one usable run per arm is inconclusive even at +43.1%, that two usable runs per arm still fail at that delta, and that a crash still fails a report whose timing result is inconclusive.

The timing probes and rule and report validation checked for this review are unchanged on main at e4316243. Longshot’s measurement proposal covers the calibration work.

I reproduced the one-run p99 failure with synthetic frame times. Both 1,200-frame windows had a median of 17.4 ms. With 12 frames at 25 ms, p99 was 17.476 ms. With 13, it was 25.000 ms, or +43.1%, which trips the 20% limit. This shows how a same-build comparison can fail when it picks up one extra slow frame. Estimating how often that happens on real hardware still needs same-build runs. The smallest change I’d start with is to require two usable starts per arm for the same timing metric, segment and settings. If either arm has fewer, report the rule as **inconclusive** and show the available values and counts. Carry that outcome through the overall report and have the coordinator accept and display it. A crash or another hard failure should still fail the report. Keep the existing percentage limit for comparisons with enough evidence and calibrate it separately against same-build runs on each PC and scene. A fixture should check that one usable run per arm is inconclusive even at +43.1%, that two usable runs per arm still fail at that delta, and that a crash still fails a report whose timing result is inconclusive. The [timing probes and rule](https://git.magicbean.dev/MagicBeanDev/Beanstalk/src/commit/ede3cdf099f7a5bec8f709a4ba4b59353fb6719f/agent/Core/Rules.cs#L273) and [report validation](https://git.magicbean.dev/MagicBeanDev/Beanstalk/src/commit/ede3cdf099f7a5bec8f709a4ba4b59353fb6719f/coordinator/beanstalk/reports.py#L135) checked for this review are unchanged on main at `e4316243`. [Longshot’s measurement proposal](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1517) covers the calibration work.
Member

Picking up the smaller verdict change from the synthetic reproduction above. I’ll cover fewer than two usable starts per arm for a required frame-time comparison, with a game-free fixture that verifies inconclusive propagates to the report and hard failures still fail. Same-build threshold calibration follows after the verdict path is in place.

Picking up the smaller verdict change from the synthetic reproduction above. I’ll cover fewer than two usable starts per arm for a required frame-time comparison, with a game-free fixture that verifies inconclusive propagates to the report and hard failures still fail. Same-build threshold calibration follows after the verdict path is in place.
Member

The local implementation now reports a required frame-time percentage comparison as inconclusive when either arm has fewer than two usable starts for the same metric, segment, cell and variant. It shows the usable counts and available values, and the report carries an unjudged outcome. Hard failures still take priority. Sufficiently sampled comparisons keep their existing thresholds.

The synthetic +43.1% case now reports inconclusive with one start per arm and still fails at the 20% limit with two. The local validation passed 1,921 agent core checks, six Windows helper checks and 385 coordinator tests. Windows symlink permissions prevented the setup section from running in the disposable copy. The coordinator run excluded 182 Postgres-dependent checks and had one skip. Roll out coordinator support for the new rule status before updated agents emit it.

The same-build hardware pilot is prepared. No hardware noise measurements yet. Calibration follow-up

The local implementation now reports a required frame-time percentage comparison as **inconclusive** when either arm has fewer than two usable starts for the same metric, segment, cell and variant. It shows the usable counts and available values, and the report carries an unjudged outcome. Hard failures still take priority. Sufficiently sampled comparisons keep their existing thresholds. The synthetic +43.1% case now reports inconclusive with one start per arm and still fails at the 20% limit with two. The local validation passed 1,921 agent core checks, six Windows helper checks and 385 coordinator tests. Windows symlink permissions prevented the setup section from running in the disposable copy. The coordinator run excluded 182 Postgres-dependent checks and had one skip. Roll out coordinator support for the new rule status before updated agents emit it. The same-build hardware pilot is prepared. No hardware noise measurements yet. [Calibration follow-up](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1517)
Sign in to join this conversation.
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk#88
No description provided.