WIP: fix(rules): mark under-sampled frame-time comparisons inconclusive #94
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk!94
Loading…
Reference in a new issue
No description provided.
Delete branch "futondestroyer/Beanstalk-contributions:fix/issue-88-timing-evidence"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
In a short capped run, a single slow frame can swing p99 enough to make the farm report a regression. A percentage frame-time comparison now needs two usable starts per side and reports insufficient evidence explicitly while retaining the observed values. See #88.
What changed
The agent evaluates frame-time percentage rules using usable starts for the measured metric. The coordinator accepts and reports the new
inconclusiveresult, and the report page renders it as a warning. Deploy the coordinator parser before rolling out agents that can emit the new status.Evidence
Validation on the reviewed head
263aa15ccovered 1,921 agent core checks, six Windows helper checks, and 385 coordinator checks. The Windows symlink setup group and 182 PostgreSQL-dependent checks were excluded, and one check was skipped. No farm or production deployment was used. This stays WIP pending the excluded coverage and maintainer review.Report screenshots
Synthetic report data from the dashboard dev server.
PostgreSQL coverage at
263aa15cagainste4316243. On disposable PostgreSQL 17.12, all 68 focused report, timing, and public-pulls tests passed. The paired full suites had 550 passed, 14 failed, and 1 skipped on base, and 553 passed, 14 failed, and 1 skipped on head. The same 14 test IDs failed on both revisions, including 12 dashboard cases where Windows rejects the date format. One additional public-pulls failure on the initial head run cleared in the paired and focused reruns.Native Windows coverage remains open. The unmodified strict agent suite builds on both revisions and stops at the setup fixture's tar symlink because the test token lacks symbolic-link privilege. The remaining check needs a Windows runner that can create symlinks. This PR stays WIP while the outstanding checks are covered. Roll out the coordinator parser before any agent build sends the new
inconclusivestatus.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.