The renderer rewrite's frame-time gates need a noise floor per machine, and the test farm's gate has none #1517
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#1517
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: measurement · Area: frame-time gates, on the test farm and in the local lane · Part of: #1493
TL;DR
The test farm's frame-time gate fails and passes pull requests by chance: at a 60 cap its p99 over 20 seconds is the 13th slowest of 1,200 frames, main run against main differed by 11% in p99 and 29% in the 1% low in one report, and #1462 failed at +26.9% on one run a side while its two other reports passed. The farm's code lives in MagicBeanDev/Beanstalk, where the change list is filed. Here we track what the renderer rewrite needs from it: every gate set from a main-against-main pair on each machine, and cells that can see frame generation and renderer cost.
Breakdown
Evidence
All numbers are from the public threads named, except the simulation. Farm code read in Beanstalk at main ede3cdf; Longshot at
310e7c9df.Proposed change
frame interpolation:andvk: swapchainlines; later a PresentMon probe on Windows for the shown cadence.inconclusivefor one start a side) is filed in MagicBeanDev/Beanstalk, linked in the first comment.How we would test it
inconclusive).Related
New on this forge; not opened on GitHub.
The farm's side of this is filed in Beanstalk: MagicBeanDev/Beanstalk#88
Possibly related, for context:
Picking up the calibration protocol alongside Beanstalk #88. First I’ll establish a repeatable same-build pilot with exact rig, scene and settings, per-run values and usable-run counts. The synthetic result establishes the failure mechanism. Hardware measurements will determine the size of the noise margin.
The first same-build pilot is ready to run on the Windows RTX 4070 in one fixed Hyrule Field scene. It uses four fresh starts in ABBA order, with the identical executable and renderer verified for both arms, warmed caches and the same settings. We will retain per-start p50, p99, slowest-1% mean and hitch counts for both 20 s and 60 s windows.
The farm's manual endpoint rejects identical resolved commits, so this pilot will use the local exclusive timing queue. The pilot checks the measurement procedure. A larger same-build series is still needed to set any threshold. No hardware measurements have been collected.
Same-build pilot on one Windows RTX 4070. Four fresh starts completed in ABBA order, with two usable starts per arm. Both arms used the same hash-verified runtime from retained main
93bc8a7fb8d53463ad5086603b10ea860cccfdb0, with embedded source fingerprintf6d97276. Windows 11 build 26300, NVIDIA 621.01, fixed 144 Hz display. Stationary child Link at the forest-bridge entrance in Hyrule Field, 60 FPS cap, windowed 1280×720 FIFO, V-Sync On, frame generation and upscaling Off. The renderer's 100% setting read back as 1720×720 in every start.I warmed the runtime before the block and let each start settle in the scene for 10 seconds. The 20-second view is the first 20 seconds of the same 60-second window. Each short window had 1,201 presents. The long windows had 3,601–3,602.
Per-start p99 in ABBA order was 17.997, 18.380, 18.012, and 18.006 ms at 20 s, and 18.090, 18.214, 18.116, and 18.052 ms at 60 s. The longer window narrowed the candidate repeat spread from 2.04% to 0.54%, while the baseline repeat spread rose from 0.05% to 0.21%. The baseline and candidate had three and one frames above twice the per-start median in each view, respectively. No frame exceeded 50 ms.
Both windows would pass the illustrative 20% p99 rule. This one block validates the collection procedure and shows how the window length changed these samples. A larger same-build series is needed to estimate the false-fail rate or set a threshold. These are the farm probe's presentation intervals. Raw local traces stayed on the PC.