feat: plan schema 8, [[pass.compare]] judges a matrix's cells against each other #98

Closed
HumanBean wants to merge 0 commits from HumanBean/Beanstalk:feat/compare-cells into main
Owner

A Longshot PR that adds a switch or setting often needs two judgements: the PR against main, and the switch on against off. The farm could do one or the other, never both, so plans for Longshot #994, #1478 and #1367 said the two configurations couldn't be compared.

What changed

  • [[pass.compare]] (plan schema 8, up to 8 rules): each rule names a metric (optionally one segment's), a matrix axis, the judged value, the reference value against, a side (pr by default, or baseline), and one of max_delta_pct or min_delta_pct.
  • How it's judged: for each scenario on the matrix and each build variant, every pair of cells that differ only in that axis is judged on that side's medians. The result belongs to the judged cell, named idle.frametime_p99_ms: RX_VK_ZERO_GAP=1 vs RX_VK_ZERO_GAP=0 (PR); its baseline column holds the reference cell's number. A skipped or mismatched cell in the pair makes the rule skipped or mismatched.
  • Refused: a PR-only axis on the baseline side, a value the axis doesn't have, a cell against itself, an absolute limit, or a plan with no matrix. A measure step can no longer be named compare.
[[pass.compare]]
metric = "frametime_p99_ms"
segment = "idle"
axis = "RX_VK_ZERO_GAP"
value = "1"
against = "0"
max_delta_pct = 5
  • Planner prompt: the planner is now also told something that was already true. .max and .min judge the PR side only, so a log line that only the PR prints is a log_match expected pattern with log_expected_missing.max = 0. The plans for #475 and #1464 thought this couldn't be done.
  • Who gets these plans: a plan with [[pass.compare]] normalizes to schema 8 and goes only to agents that say 8, which the agent now does. The agent and the coordinator check it alike. The docs (test-plan.md, report-schema.md, protocol.md) and the plan's own wording cover it.

Nothing on screen changes beyond rule names in the Rules table, so there are no screenshots.

Testing

  • python -I agent/tests/test_agent_core.py --strict: 2115 checks passed. The new fixtures (compare-cells.toml and 12 bad-compare-*) are cross-checked against the coordinator's reader. New rule checks cover pairs across a second axis, the reference and judged columns, a baseline-side rule, a failing rule failing the report, and a skipped reference cell.
  • pytest -q in coordinator/: 588 passed, 1 skipped.
  • Not tested: a real farm job.

🤖 Generated with Claude Code

A Longshot PR that adds a switch or setting often needs two judgements: the PR against main, and the switch on against off. The farm could do one or the other, never both, so plans for [Longshot #994](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/994), [#1478](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1478) and [#1367](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1367) said the two configurations couldn't be compared. ## What changed - **`[[pass.compare]]` (plan schema 8, up to 8 rules):** each rule names a metric (optionally one segment's), a matrix axis, the judged `value`, the reference value `against`, a `side` (`pr` by default, or `baseline`), and one of `max_delta_pct` or `min_delta_pct`. - **How it's judged:** for each scenario on the matrix and each build variant, every pair of cells that differ only in that axis is judged on that side's medians. The result belongs to the judged cell, named `idle.frametime_p99_ms: RX_VK_ZERO_GAP=1 vs RX_VK_ZERO_GAP=0 (PR)`; its baseline column holds the reference cell's number. A skipped or mismatched cell in the pair makes the rule skipped or mismatched. - **Refused:** a PR-only axis on the baseline side, a value the axis doesn't have, a cell against itself, an absolute limit, or a plan with no matrix. A measure step can no longer be named `compare`. ```toml [[pass.compare]] metric = "frametime_p99_ms" segment = "idle" axis = "RX_VK_ZERO_GAP" value = "1" against = "0" max_delta_pct = 5 ``` - **Planner prompt:** the planner is now also told something that was already true. `.max` and `.min` judge the PR side only, so a log line that only the PR prints is a `log_match` expected pattern with `log_expected_missing.max = 0`. The plans for #475 and #1464 thought this couldn't be done. - **Who gets these plans:** a plan with `[[pass.compare]]` normalizes to schema 8 and goes only to agents that say 8, which the agent now does. The agent and the coordinator check it alike. The docs (`test-plan.md`, `report-schema.md`, `protocol.md`) and the plan's own wording cover it. Nothing on screen changes beyond rule names in the Rules table, so there are no screenshots. ## Testing - `python -I agent/tests/test_agent_core.py --strict`: 2115 checks passed. The new fixtures (`compare-cells.toml` and 12 `bad-compare-*`) are cross-checked against the coordinator's reader. New rule checks cover pairs across a second axis, the reference and judged columns, a baseline-side rule, a failing rule failing the report, and a skipped reference cell. - `pytest -q` in `coordinator/`: 588 passed, 1 skipped. - **Not tested:** a real farm job. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat: plan schema 8, [[pass.compare]] judges a matrix's cells against each other
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
e62fbc6f85
A Longshot PR that adds a switch or a setting often needs two
judgements: the PR against main, and the switch on against off. The
farm's rules could only do the first (or only the second, with
baseline.ref = "pr"), so plans for #994, #1478 and #1367 said the two
configurations could not be compared.

[[pass.compare]] (schema 8, up to 8 rules) takes a metric (optionally
one segment's), a matrix axis, the judged value and the reference value,
a side (pr by default, or baseline) and one of max_delta_pct or
min_delta_pct. For each scenario on the matrix and build variant, each
pair of cells that differ only in that axis is judged on that side's
medians; the result is the judged cell's, its baseline column the
reference cell's number. A skipped or mismatched cell of the pair skips
or mismatches the rule. A measure step can't be named compare now.

The planner is also told what was already true: .max and .min judge the
PR side only, so a log line only the PR prints is a log_match expected
pattern with log_expected_missing.max = 0 (plans for #475 and #1464
thought it could not be done).

Agent and coordinator check it alike (new fixtures, cross-checked); the
agent says plan schema 8.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean closed this pull request 2026-11-19 07:25:44 +00:00
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!98
No description provided.