WIP: fix(rules): mark under-sampled frame-time comparisons inconclusive #94

Draft
futondestroyer wants to merge 1 commit from futondestroyer/Beanstalk-contributions:fix/issue-88-timing-evidence into main
Member

Why

In a short capped run, a single slow frame can swing p99 enough to make the farm report a regression. A percentage frame-time comparison now needs two usable starts per side and reports insufficient evidence explicitly while retaining the observed values. See #88.

What changed

The agent evaluates frame-time percentage rules using usable starts for the measured metric. The coordinator accepts and reports the new inconclusive result, and the report page renders it as a warning. Deploy the coordinator parser before rolling out agents that can emit the new status.

Evidence

Synthetic result Before After
One usable start per side with a 43.1% p99 increase Fail Inconclusive, with the observed delta preserved
Two usable starts per side at the same delta Fail Fail

Validation on the reviewed head 263aa15c covered 1,921 agent core checks, six Windows helper checks, and 385 coordinator checks. The Windows symlink setup group and 182 PostgreSQL-dependent checks were excluded, and one check was skipped. No farm or production deployment was used. This stays WIP pending the excluded coverage and maintainer review.

Report screenshots

Synthetic report data from the dashboard dev server.

Desktop synthetic report

390 px synthetic report, Rules section

## Why In a short capped run, a single slow frame can swing p99 enough to make the farm report a regression. A percentage frame-time comparison now needs two usable starts per side and reports insufficient evidence explicitly while retaining the observed values. See #88. ## What changed The agent evaluates frame-time percentage rules using usable starts for the measured metric. The coordinator accepts and reports the new `inconclusive` result, and the report page renders it as a warning. Deploy the coordinator parser before rolling out agents that can emit the new status. ## Evidence | Synthetic result | Before | After | |---|---|---| | One usable start per side with a 43.1% p99 increase | Fail | Inconclusive, with the observed delta preserved | | Two usable starts per side at the same delta | Fail | Fail | Validation on the reviewed head `263aa15c` covered 1,921 agent core checks, six Windows helper checks, and 385 coordinator checks. The Windows symlink setup group and 182 PostgreSQL-dependent checks were excluded, and one check was skipped. No farm or production deployment was used. This stays WIP pending the excluded coverage and maintainer review. ## Report screenshots Synthetic report data from the dashboard dev server. ![Desktop synthetic report](https://git.magicbean.dev/attachments/11933e58-f287-4db9-b689-667e7710abac) ![390 px synthetic report, Rules section](https://git.magicbean.dev/attachments/273c564f-40d2-4c57-a470-19cde5f66f3f)
fix(rules): mark under-sampled frame-time comparisons inconclusive
Some checks are pending
CI / Coordinator tests (pull_request) Waiting to run
CI / Agent build and checks (pull_request) Waiting to run
CI / Coordinator image (pull_request) Waiting to run
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
263aa15c52
Assisted-by: OpenAI Codex

Co-Authored-By: OpenAI Codex <[email protected]>
Author
Member

PostgreSQL coverage at 263aa15c against e4316243. On disposable PostgreSQL 17.12, all 68 focused report, timing, and public-pulls tests passed. The paired full suites had 550 passed, 14 failed, and 1 skipped on base, and 553 passed, 14 failed, and 1 skipped on head. The same 14 test IDs failed on both revisions, including 12 dashboard cases where Windows rejects the date format. One additional public-pulls failure on the initial head run cleared in the paired and focused reruns.

Native Windows coverage remains open. The unmodified strict agent suite builds on both revisions and stops at the setup fixture's tar symlink because the test token lacks symbolic-link privilege. The remaining check needs a Windows runner that can create symlinks. This PR stays WIP while the outstanding checks are covered. Roll out the coordinator parser before any agent build sends the new inconclusive status.

**PostgreSQL coverage at `263aa15c` against `e4316243`.** On disposable PostgreSQL 17.12, all 68 focused report, timing, and public-pulls tests passed. The paired full suites had 550 passed, 14 failed, and 1 skipped on base, and 553 passed, 14 failed, and 1 skipped on head. The same 14 test IDs failed on both revisions, including 12 dashboard cases where Windows rejects the date format. One additional public-pulls failure on the initial head run cleared in the paired and focused reruns. Native Windows coverage remains open. The unmodified strict agent suite builds on both revisions and stops at the setup fixture's tar symlink because the test token lacks symbolic-link privilege. The remaining check needs a Windows runner that can create symlinks. This PR stays WIP while the outstanding checks are covered. Roll out the coordinator parser before any agent build sends the new `inconclusive` status.
Some checks are pending
CI / Coordinator tests (pull_request) Waiting to run
CI / Agent build and checks (pull_request) Waiting to run
CI / Coordinator image (pull_request) Waiting to run
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
This pull request is marked as a work in progress.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/issue-88-timing-evidence:futondestroyer-fix/issue-88-timing-evidence'
git switch 'futondestroyer-fix/issue-88-timing-evidence'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!94
No description provided.