The renderer rewrite's frame-time gates need a noise floor per machine, and the test farm's gate has none #1517

Open
opened 2026-11-18 01:53:46 +00:00 by mooki · 5 comments
Member

Type: measurement · Area: frame-time gates, on the test farm and in the local lane · Part of: #1493

TL;DR

The test farm's frame-time gate fails and passes pull requests by chance: at a 60 cap its p99 over 20 seconds is the 13th slowest of 1,200 frames, main run against main differed by 11% in p99 and 29% in the 1% low in one report, and #1462 failed at +26.9% on one run a side while its two other reports passed. The farm's code lives in MagicBeanDev/Beanstalk, where the change list is filed. Here we track what the renderer rewrite needs from it: every gate set from a main-against-main pair on each machine, and cells that can see frame generation and renderer cost.

Breakdown

  • Who it affects: pull request authors and reviewers today, and the renderer rewrite (#1493), whose gates are frame-time rules that the farm will run on volunteers' PCs.
  • What is wrong: the rule has no noise baseline (pictures got one in Beanstalk #87), p99 is a cliff statistic at this window, one start a side cannot carry a 10 to 20% gate, and the capped 60 cells hide renderer cost. The frame generation cells probably never ran frame generation (inferred from code and numbers, not confirmed).
  • What we propose: no gate is used until a null pair (the same build in both arms) has been run on that rig and scene, and its threshold is the larger of the nominal limit and 1.5 times that spread, printed with every result. Add an uncapped cost cell and a verified frame generation cell. Until then a frame-time failure on one start a side is read as "look at it", not as a verdict.
  • What does not change: the farm's ABBA order, its per-report judgement and the rule syntax. Local paired runs keep their own rule (quiet machine, same-arm spread at most 5%).
  • How we would prove it: 10 pairs of the same build on three PCs at 20 s and 60 s windows, counting false fails of today's rule and then of the new one, plus one arm with a known added cost to show a real loss still fails.

Evidence

All numbers are from the public threads named, except the simulation. Farm code read in Beanstalk at main ede3cdf; Longshot at 310e7c9df.

  • The cliff. The farm's probe is numpy's interpolated p99 (Beanstalk agent/Core/Probes.cs:20-46). In a simulated 1,200-frame window (steady 17.4 ms with up to 0.8 ms of timer lateness), 12 frames of 25 ms leave p99 at 17.53 ms and a 13th makes it 25.00 (+43%). The mean of the slowest 1% moves smoothly: 17.40, 21.21, 24.37, 25.00 for 0, 6, 11 and 13 slow frames.
  • #1462 (limit 20%, one start a side, 20 s after 5 s warm-up, Windows): RTX 4070 field p99 18.27 to 23.19 ms (+26.9%, fail); RX 9070 XT 19.93 to 20.63 (+3.5%, pass); RTX 3070 19.88 to 19.89 (+0%, pass). The review's reading, quoted in #1489, is that main's side of the 4070 run was unusually smooth and p50 did not move (16.67 to 16.62).
  • #1458 (RX 9070 XT, two starts a side, 30 s after 10 s warm-up, limit 15%): main's two runs of one cell differed by +11.3% in p99 (19.39 and 21.59 ms) and +12.2% in the 1% low; in another cell by +28.8% in the 1% low (19.66 and 25.32), +98% in max, and 0 against 5 stutters.
  • A simulation from those reports' histograms (frames drawn independently, so a floor under real noise): a 10% p99 gate fails with no change 4.0%, 0.0% and 13.9% of the time on the 9070 XT, 4070 and 3070 logs; the 1% low fails 0.3%, 27.6% and 6.4%.
  • Capped cells hide cost. All four #1458 cells have p50 16.63 to 16.72 ms and the render thread idle 9.01 ms of 16.67. A cheaper renderer would move none of those cells' numbers on that PC. The local lane avoids this by measuring cost uncapped on a quiet PC with a same-arm spread of at most 5% (the convention in #60).

Proposed change

  1. Null pair first. For each rig and scene, run the same build in both arms (10 pairs), record the spread of p99, the mean of the slowest 1% and the count of frames over twice the median, and set the gate from it. Re-measure when the farm's PCs or the scenes change.
  2. The cells the farm lacks. An uncapped cell (cap 1000, Per frame, HOS_FPS_STEP=0) for renderer cost; a frame generation cell checked by a probe that reads the frame interpolation: and vk: swapchain lines; later a PresentMon probe on Windows for the shown cadence.
  3. The farm's change list (noise baseline for numeric rules, 60 s windows, every run shown, inconclusive for one start a side) is filed in MagicBeanDev/Beanstalk, linked in the first comment.
  4. Order. The null pair comes before any rewrite gate is written into a plan, and the pacing gates named in #1493 stay advisory until it exists.

How we would test it

  • Farm run (the measurement itself): 10 pairs of one build on an RTX 4070, an RX 9070 XT and an RTX 3070 (the three PCs of #1462), in the field and Deku cells, at 20 s and 60 s. Today's rule is counted first, so the false-fail rate is on record. Then the same with the new rule; the target is at most 1% false fails.
  • Real loss still fails: one pair where the second arm has a known cost (the host has a diagnostic frame sleep, HOS_FRAME_SLEEP_MS), at the cell where the cost cannot hide behind the cap.
  • Game-free: the checks for the rule and the probes live with the farm's code and are listed in the Beanstalk issue (the 12 and 13 frame cliff, a main spread of 11.3% saving a +10.3% delta, one start a side giving inconclusive).
  • Every claim about speed in the rewrite keeps the standing rule: paired timing after a null main-against-main pair.
  • MagicBeanDev/Beanstalk: #87 (pictures judged against main's own noise), #1 (item 5, "same machine and same session isn't enough for timing"), #82 (measure around steps), and the issue filed with this one.
  • #1458, #1462 and #1489 (the numbers), #1235 (its gain is judged against a 3-against-3 run noise band), #60 (the local lane's quiet-PC rule), #69 (the rolling benchmarks, another reader of frame times with pass ranges).
  • #1493, the renderer rewrite (Step 0, item 4: frame metrics that mean what they say; its Metrics paragraph already names the farm's p99).
  • Searched all 570 issues and 925 pull requests, open and closed, by title and body (p99, null pair, same-arm, noise, false fail, flaky, farm, performance gate). Nothing proposes a noise floor for the farm's frame-time rule.

New on this forge; not opened on GitHub.

**Type:** measurement · **Area:** frame-time gates, on the test farm and in the local lane · **Part of:** #1493 ## TL;DR The test farm's frame-time gate fails and passes pull requests by chance: at a 60 cap its p99 over 20 seconds is the 13th slowest of 1,200 frames, main run against main differed by 11% in p99 and 29% in the 1% low in one report, and #1462 failed at +26.9% on one run a side while its two other reports passed. The farm's code lives in MagicBeanDev/Beanstalk, where the change list is filed. Here we track what the renderer rewrite needs from it: every gate set from a main-against-main pair on each machine, and cells that can see frame generation and renderer cost. ## Breakdown - **Who it affects:** pull request authors and reviewers today, and the renderer rewrite (#1493), whose gates are frame-time rules that the farm will run on volunteers' PCs. - **What is wrong:** the rule has no noise baseline (pictures got one in Beanstalk #87), p99 is a cliff statistic at this window, one start a side cannot carry a 10 to 20% gate, and the capped 60 cells hide renderer cost. The frame generation cells probably never ran frame generation (inferred from code and numbers, not confirmed). - **What we propose:** no gate is used until a null pair (the same build in both arms) has been run on that rig and scene, and its threshold is the larger of the nominal limit and 1.5 times that spread, printed with every result. Add an uncapped cost cell and a verified frame generation cell. Until then a frame-time failure on one start a side is read as "look at it", not as a verdict. - **What does not change:** the farm's ABBA order, its per-report judgement and the rule syntax. Local paired runs keep their own rule (quiet machine, same-arm spread at most 5%). - **How we would prove it:** 10 pairs of the same build on three PCs at 20 s and 60 s windows, counting false fails of today's rule and then of the new one, plus one arm with a known added cost to show a real loss still fails. ## Evidence All numbers are from the public threads named, except the simulation. Farm code read in Beanstalk at main ede3cdf; Longshot at 310e7c9df. - **The cliff.** The farm's probe is numpy's interpolated p99 (Beanstalk agent/Core/Probes.cs:20-46). In a simulated 1,200-frame window (steady 17.4 ms with up to 0.8 ms of timer lateness), 12 frames of 25 ms leave p99 at 17.53 ms and a 13th makes it 25.00 (+43%). The mean of the slowest 1% moves smoothly: 17.40, 21.21, 24.37, 25.00 for 0, 6, 11 and 13 slow frames. - **#1462 (limit 20%, one start a side, 20 s after 5 s warm-up, Windows):** RTX 4070 field p99 18.27 to 23.19 ms (+26.9%, fail); RX 9070 XT 19.93 to 20.63 (+3.5%, pass); RTX 3070 19.88 to 19.89 (+0%, pass). The review's reading, quoted in #1489, is that main's side of the 4070 run was unusually smooth and p50 did not move (16.67 to 16.62). - **#1458 (RX 9070 XT, two starts a side, 30 s after 10 s warm-up, limit 15%):** main's two runs of one cell differed by +11.3% in p99 (19.39 and 21.59 ms) and +12.2% in the 1% low; in another cell by +28.8% in the 1% low (19.66 and 25.32), +98% in max, and 0 against 5 stutters. - **A simulation from those reports' histograms** (frames drawn independently, so a floor under real noise): a 10% p99 gate fails with no change 4.0%, 0.0% and 13.9% of the time on the 9070 XT, 4070 and 3070 logs; the 1% low fails 0.3%, 27.6% and 6.4%. - **Capped cells hide cost.** All four #1458 cells have p50 16.63 to 16.72 ms and the render thread idle 9.01 ms of 16.67. A cheaper renderer would move none of those cells' numbers on that PC. The local lane avoids this by measuring cost uncapped on a quiet PC with a same-arm spread of at most 5% (the convention in #60). ## Proposed change 1. **Null pair first.** For each rig and scene, run the same build in both arms (10 pairs), record the spread of p99, the mean of the slowest 1% and the count of frames over twice the median, and set the gate from it. Re-measure when the farm's PCs or the scenes change. 2. **The cells the farm lacks.** An uncapped cell (cap 1000, Per frame, HOS_FPS_STEP=0) for renderer cost; a frame generation cell checked by a probe that reads the `frame interpolation:` and `vk: swapchain` lines; later a PresentMon probe on Windows for the shown cadence. 3. **The farm's change list** (noise baseline for numeric rules, 60 s windows, every run shown, `inconclusive` for one start a side) is filed in MagicBeanDev/Beanstalk, linked in the first comment. 4. **Order.** The null pair comes before any rewrite gate is written into a plan, and the pacing gates named in #1493 stay advisory until it exists. ## How we would test it - **Farm run (the measurement itself):** 10 pairs of one build on an RTX 4070, an RX 9070 XT and an RTX 3070 (the three PCs of #1462), in the field and Deku cells, at 20 s and 60 s. Today's rule is counted first, so the false-fail rate is on record. Then the same with the new rule; the target is at most 1% false fails. - **Real loss still fails:** one pair where the second arm has a known cost (the host has a diagnostic frame sleep, HOS_FRAME_SLEEP_MS), at the cell where the cost cannot hide behind the cap. - **Game-free:** the checks for the rule and the probes live with the farm's code and are listed in the Beanstalk issue (the 12 and 13 frame cliff, a main spread of 11.3% saving a +10.3% delta, one start a side giving `inconclusive`). - Every claim about speed in the rewrite keeps the standing rule: paired timing after a null main-against-main pair. ## Related - MagicBeanDev/Beanstalk: #87 (pictures judged against main's own noise), #1 (item 5, "same machine and same session isn't enough for timing"), #82 (measure around steps), and the issue filed with this one. - #1458, #1462 and #1489 (the numbers), #1235 (its gain is judged against a 3-against-3 run noise band), #60 (the local lane's quiet-PC rule), #69 (the rolling benchmarks, another reader of frame times with pass ranges). - #1493, the renderer rewrite (Step 0, item 4: frame metrics that mean what they say; its Metrics paragraph already names the farm's p99). - Searched all 570 issues and 925 pull requests, open and closed, by title and body (p99, null pair, same-arm, noise, false fail, flaky, farm, performance gate). Nothing proposes a noise floor for the farm's frame-time rule. New on this forge; not opened on GitHub.
Author
Member

The farm's side of this is filed in Beanstalk: MagicBeanDev/Beanstalk#88

The farm's side of this is filed in Beanstalk: https://git.magicbean.dev/MagicBeanDev/Beanstalk/issues/88
Collaborator

Possibly related, for context:

  • issue #1493 Renderer rewrite — Renderer rewrite whose frame-metric gates this informs
  • issue #1503 The virtual-core gate takes one global lock on every block and wake, and a low-priority thread can wait up to a second for a core — Related measurement issue discussing null-pair benchmarking
  • PR #1235 perf(vi): the vsync ticker waits on a high-resolution timer instead of yielding — Merged performance change judged against repeated-run noise band
Possibly related, for context: - issue #1493 Renderer rewrite — Renderer rewrite whose frame-metric gates this informs - issue #1503 The virtual-core gate takes one global lock on every block and wake, and a low-priority thread can wait up to a second for a core — Related measurement issue discussing null-pair benchmarking - PR #1235 perf(vi): the vsync ticker waits on a high-resolution timer instead of yielding — Merged performance change judged against repeated-run noise band
Member

Picking up the calibration protocol alongside Beanstalk #88. First I’ll establish a repeatable same-build pilot with exact rig, scene and settings, per-run values and usable-run counts. The synthetic result establishes the failure mechanism. Hardware measurements will determine the size of the noise margin.

Picking up the calibration protocol alongside Beanstalk #88. First I’ll establish a repeatable same-build pilot with exact rig, scene and settings, per-run values and usable-run counts. The synthetic result establishes the failure mechanism. Hardware measurements will determine the size of the noise margin.
Member

The first same-build pilot is ready to run on the Windows RTX 4070 in one fixed Hyrule Field scene. It uses four fresh starts in ABBA order, with the identical executable and renderer verified for both arms, warmed caches and the same settings. We will retain per-start p50, p99, slowest-1% mean and hitch counts for both 20 s and 60 s windows.

The farm's manual endpoint rejects identical resolved commits, so this pilot will use the local exclusive timing queue. The pilot checks the measurement procedure. A larger same-build series is still needed to set any threshold. No hardware measurements have been collected.

The first same-build pilot is ready to run on the Windows RTX 4070 in one fixed Hyrule Field scene. It uses four fresh starts in ABBA order, with the identical executable and renderer verified for both arms, warmed caches and the same settings. We will retain per-start p50, p99, slowest-1% mean and hitch counts for both 20 s and 60 s windows. The farm's manual endpoint rejects identical resolved commits, so this pilot will use the local exclusive timing queue. The pilot checks the measurement procedure. A larger same-build series is still needed to set any threshold. No hardware measurements have been collected.
Member

Same-build pilot on one Windows RTX 4070. Four fresh starts completed in ABBA order, with two usable starts per arm. Both arms used the same hash-verified runtime from retained main 93bc8a7fb8d53463ad5086603b10ea860cccfdb0, with embedded source fingerprint f6d97276. Windows 11 build 26300, NVIDIA 621.01, fixed 144 Hz display. Stationary child Link at the forest-bridge entrance in Hyrule Field, 60 FPS cap, windowed 1280×720 FIFO, V-Sync On, frame generation and upscaling Off. The renderer's 100% setting read back as 1720×720 in every start.

I warmed the runtime before the block and let each start settle in the scene for 10 seconds. The 20-second view is the first 20 seconds of the same 60-second window. Each short window had 1,201 presents. The long windows had 3,601–3,602.

Window Baseline p50 Candidate p50 Baseline p99 Candidate p99 p99 delta Baseline / candidate slowest 1% mean
20 s 16.674 ms 16.688 ms 18.002 ms 18.196 ms +1.08% 22.327 / 19.580 ms
60 s 16.672 ms 16.682 ms 18.071 ms 18.165 ms +0.52% 20.248 / 19.191 ms

Per-start p99 in ABBA order was 17.997, 18.380, 18.012, and 18.006 ms at 20 s, and 18.090, 18.214, 18.116, and 18.052 ms at 60 s. The longer window narrowed the candidate repeat spread from 2.04% to 0.54%, while the baseline repeat spread rose from 0.05% to 0.21%. The baseline and candidate had three and one frames above twice the per-start median in each view, respectively. No frame exceeded 50 ms.

Both windows would pass the illustrative 20% p99 rule. This one block validates the collection procedure and shows how the window length changed these samples. A larger same-build series is needed to estimate the false-fail rate or set a threshold. These are the farm probe's presentation intervals. Raw local traces stayed on the PC.

**Same-build pilot on one Windows RTX 4070.** Four fresh starts completed in ABBA order, with two usable starts per arm. Both arms used the same hash-verified runtime from retained main `93bc8a7fb8d53463ad5086603b10ea860cccfdb0`, with embedded source fingerprint `f6d97276`. Windows 11 build 26300, NVIDIA 621.01, fixed 144 Hz display. Stationary child Link at the forest-bridge entrance in Hyrule Field, 60 FPS cap, windowed 1280×720 FIFO, V-Sync On, frame generation and upscaling Off. The renderer's 100% setting read back as 1720×720 in every start. I warmed the runtime before the block and let each start settle in the scene for 10 seconds. The 20-second view is the first 20 seconds of the same 60-second window. Each short window had 1,201 presents. The long windows had 3,601–3,602. | Window | Baseline p50 | Candidate p50 | Baseline p99 | Candidate p99 | p99 delta | Baseline / candidate slowest 1% mean | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | 20 s | 16.674 ms | 16.688 ms | 18.002 ms | 18.196 ms | +1.08% | 22.327 / 19.580 ms | | 60 s | 16.672 ms | 16.682 ms | 18.071 ms | 18.165 ms | +0.52% | 20.248 / 19.191 ms | Per-start p99 in ABBA order was 17.997, 18.380, 18.012, and 18.006 ms at 20 s, and 18.090, 18.214, 18.116, and 18.052 ms at 60 s. The longer window narrowed the candidate repeat spread from 2.04% to 0.54%, while the baseline repeat spread rose from 0.05% to 0.21%. The baseline and candidate had three and one frames above twice the per-start median in each view, respectively. No frame exceeded 50 ms. Both windows would pass the illustrative 20% p99 rule. This one block validates the collection procedure and shows how the window length changed these samples. A larger same-build series is needed to estimate the false-fail rate or set a threshold. These are the farm probe's presentation intervals. Raw local traces stayed on the PC.
Sign in to join this conversation.
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1517
No description provided.