feat: plan schema 8, [[pass.compare]] judges a matrix's cells against each other #98
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk!98
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Beanstalk:feat/compare-cells"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A Longshot PR that adds a switch or setting often needs two judgements: the PR against main, and the switch on against off. The farm could do one or the other, never both, so plans for Longshot #994, #1478 and #1367 said the two configurations couldn't be compared.
What changed
[[pass.compare]](plan schema 8, up to 8 rules): each rule names a metric (optionally one segment's), a matrix axis, the judgedvalue, the reference valueagainst, aside(prby default, orbaseline), and one ofmax_delta_pctormin_delta_pct.idle.frametime_p99_ms: RX_VK_ZERO_GAP=1 vs RX_VK_ZERO_GAP=0 (PR); its baseline column holds the reference cell's number. A skipped or mismatched cell in the pair makes the rule skipped or mismatched.compare..maxand.minjudge the PR side only, so a log line that only the PR prints is alog_matchexpected pattern withlog_expected_missing.max = 0. The plans for #475 and #1464 thought this couldn't be done.[[pass.compare]]normalizes to schema 8 and goes only to agents that say 8, which the agent now does. The agent and the coordinator check it alike. The docs (test-plan.md,report-schema.md,protocol.md) and the plan's own wording cover it.Nothing on screen changes beyond rule names in the Rules table, so there are no screenshots.
Testing
python -I agent/tests/test_agent_core.py --strict: 2115 checks passed. The new fixtures (compare-cells.tomland 12bad-compare-*) are cross-checked against the coordinator's reader. New rule checks cover pairs across a second axis, the reference and judged columns, a baseline-side rule, a failing rule failing the report, and a skipped reference cell.pytest -qincoordinator/: 588 passed, 1 skipped.🤖 Generated with Claude Code
Pull request closed