feat(coordinator): an A/B/C plan shows each level side by side on the PR's comment #121

Merged
HumanBean merged 1 commit from HumanBean/Beanstalk:feat/abc-tables into main 2026-11-20 19:29:42 +00:00
Owner

Summary

Longshot #1681 compares a switch's levels 0, 1 and 2 (RX_VK_TARGET_REUSE_MEMORY, PR side only), but the farm's comment showed one "main → PR" frame-time column (one cell's numbers, whichever came first) and an 18-row table of every cell and target. For a plan that compares one matrix axis's levels (reporter.compared_axis: its pass.compare rules' axis and reference value, else the matrix's one axis with two or more values and its first value):

  • under the results table, per scenario, one row per target: main once (for a PR-only switch), then frame time p99 at each level, every level but the reference with its change against it. For a switch on both sides, each level is main → PR;
  • the results table drops its main → PR frame-time column (it was one cell's);
  • the folded "Every scenario and matrix cell" table is left out (the new table says it).

Plans without levels to compare, and plans with several multi-valued axes, are unchanged.

What #1681's comment will show once reports come:

**`field-turn-and-stand`** · frame time p99 (ms)

| Target | main | `RX_VK_TARGET_REUSE_MEMORY=0` (reference) | `RX_VK_TARGET_REUSE_MEMORY=1` | `RX_VK_TARGET_REUSE_MEMORY=2` |
| --- | --- | --- | --- | --- |
| `linux+amd` | 12.5 | 12.0 | 11.4 (−5%) | 13.8 (+15%) |

Testing

Tested: the coordinator suite in a python:3.13-alpine container with Postgres 17 on the farm's server: 594 passed, 3 failed (test_regression.py, failing the same way on main), 1 skipped. New test_an_abc_plan_shows_each_level_side_by_side: a 3-level PR-only plan with compare rules gives the per-level table (main once, the reference plain, the others with their change, a target with no report as dashes) and a results table without the frame-time column; the default plan gives none.

Still to test: #1681's comment once its reports come.

🤖 Generated with Claude Code

## Summary Longshot #1681 compares a switch's levels 0, 1 and 2 (`RX_VK_TARGET_REUSE_MEMORY`, PR side only), but the farm's comment showed one "main → PR" frame-time column (one cell's numbers, whichever came first) and an 18-row table of every cell and target. For a plan that compares one matrix axis's levels (`reporter.compared_axis`: its `pass.compare` rules' axis and reference value, else the matrix's one axis with two or more values and its first value): - under the results table, per scenario, one row per target: main once (for a PR-only switch), then frame time p99 at each level, every level but the reference with its change against it. For a switch on both sides, each level is main → PR; - the results table drops its main → PR frame-time column (it was one cell's); - the folded "Every scenario and matrix cell" table is left out (the new table says it). Plans without levels to compare, and plans with several multi-valued axes, are unchanged. What #1681's comment will show once reports come: ``` **`field-turn-and-stand`** · frame time p99 (ms) | Target | main | `RX_VK_TARGET_REUSE_MEMORY=0` (reference) | `RX_VK_TARGET_REUSE_MEMORY=1` | `RX_VK_TARGET_REUSE_MEMORY=2` | | --- | --- | --- | --- | --- | | `linux+amd` | 12.5 | 12.0 | 11.4 (−5%) | 13.8 (+15%) | ``` ## Testing **Tested:** the coordinator suite in a python:3.13-alpine container with Postgres 17 on the farm's server: 594 passed, 3 failed (`test_regression.py`, failing the same way on main), 1 skipped. New `test_an_abc_plan_shows_each_level_side_by_side`: a 3-level PR-only plan with compare rules gives the per-level table (main once, the reference plain, the others with their change, a target with no report as dashes) and a results table without the frame-time column; the default plan gives none. **Still to test:** #1681's comment once its reports come. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(coordinator): an A/B/C plan shows each level side by side on the PR's comment
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 17s
CI / Workflow lint (actionlint) (pull_request) Successful in 18s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
9e497ddb78
Longshot #1681 compares a switch's levels 0, 1 and 2 (PR side only), but the
comment had one "main → PR" frame-time column (one cell's numbers, whichever
came first) and an 18-row table of every cell and target. For a plan that
compares one matrix axis's levels (reporter.compared_axis: its pass.compare
rules' axis and reference, else the matrix's one axis with 2+ values):

- under the results table, per scenario, a row per target with main once (a
  PR-only switch) and frame time p99 at each level, every level but the
  reference with its change against it (a switch on both sides: main → PR at
  each level);
- the results table drops its main → PR frame-time column;
- the folded every-cell table is left out (the new table says it).

Other plans are unchanged.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • PR #98 feat: plan schema 8, pass.compare judges a matrix's cells against each other — Introduced compare rules this report displays
Possibly related, for context: - PR #98 feat: plan schema 8, [[pass.compare]] judges a matrix's cells against each other — Introduced compare rules this report displays
HumanBean force-pushed feat/abc-tables from 9e497ddb78
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 17s
CI / Workflow lint (actionlint) (pull_request) Successful in 18s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
to 23d4905f3e
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 6s
CI / Workflow lint (actionlint) (pull_request) Successful in 7s
CI / Coordinator tests (pull_request) Failing after 1m26s
CI / Agent build and checks (pull_request) Successful in 52s
CI / Coordinator image (pull_request) Successful in 15s
2026-11-20 19:27:48 +00:00
Compare
HumanBean changed title from WIP: feat(coordinator): an A/B/C plan shows each level side by side on the PR's comment to feat(coordinator): an A/B/C plan shows each level side by side on the PR's comment 2026-11-20 19:29:38 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!121
No description provided.