feat(coordinator): a clearer PR comment, answer first, and an AI summary when a farm run ends #117

Merged
HumanBean merged 1 commit from HumanBean/Beanstalk:feat/clearer-comment-summary into main 2026-11-20 16:03:51 +00:00
Owner

The farm's PR comment was one long block. On #1574, for example:

  • the plan was described twice, once as a paragraph and once as bullets;
  • the answer was buried halfway down;
  • "2/1 report" was cryptic;
  • rule names were cut off mid-word (RX_VK_ACQUI);
  • the per-target and per-cell tables seemed to disagree;
  • every row carried a line of hardware tags.

This PR puts the answer first and folds the rest, and adds an AI summary once a run has finished.

The comment

### Beanstalk test farm · ❌ failed on linux+nvidia
Commit `2f24a49` · approved by HumanBean, futondestroyer (double) · [dashboard]

> **Summary**, written by AI (`gpt-6.1-terra`) from the numbers below:
> <2–4 sentences>

Done (started 04:45 UTC, last report 05:52 UTC). Comment `@Beanstalk` to run it again.

| Target | Result | Reports | Frame time p99 (ms), switch off → switch on | VRAM peak (MB) | Crashes |
| `linux+nvidia` | ❌ fail | 2 [1] [2] | 13.2 → 13.4 (+2%) | — | none |
| `intel` | ⏳ queued | 0 of 1 | — | — | — |

**Why it failed**
- `linux+nvidia`: `hyrule-field · RX_VK_ACQUIRE_AHEAD=1 (PR only)` `field.fps_mean: RX_VK_ACQUIRE_AHEAD=1 vs RX_VK_ACQUIRE_AHEAD=0`: `60.0 → 42.6 (−29.0%); limit 5.0`

▸ What was tested   ▸ Every scenario and matrix cell   ▸ Measurements on `linux+nvidia`, for the PR description
▸ The testers' hardware   ▸ Run by hand on PCs that don't match their target
  • The title says the result: ✅ passed, ❌ failed on the targets that failed, ⚠️ couldn't be judged, 🔄 testing (with the reports so far), waiting for approval, planning, cancelled, or ⛔ when the plan was refused or held.
  • The results table: one row per target. It shows the result, the reports (the count once there are enough, "0 of 1" while short, each linked), frame time and VRAM from the baseline to the PR, and crashes. The hardware moved to its own folded table.
  • Why it failed: each failed rule once per target, with its full name, where it ran, the numbers and the limit.
  • Couldn't be judged: each such report, linked, with the first line of its error. It also says plainly that this is the tester's PC failing, not the PR.
  • Folded: what was tested, the per-cell table, the Measurements block with its Markdown copy for the PR description, the hardware and the off-target runs.
  • The per-cell table now uses the same "baseline → PR" order as the main table.
  • The Measurements verdict keeps rule names whole (up to 140 characters instead of 60).
  • Earlier runs are still folded when a new run starts. folded_body drops the whole new title line.

The AI summary (results_summary.py)

  • When: once a round, when every target has finished (pass, fail, couldn't be judged or cancelled, with at least one report). It's claimed in the database before the model is asked (result_summary_round), so it's asked only once. A new round (@Beanstalk again) gets a new summary.

  • Which model: the planner's, through a new farm.summary_llm. It's set only when the app builds that model itself, so tests that pass a fake planner model don't get summaries by accident. If AI planning isn't set up, there's no summary.

  • What it reads: only the farm's own judgement, made small (at most 24 KB):

    • each target's result and how many reports it needs;
    • each report's result, and the PC's OS, GPU and driver;
    • the rules that didn't pass, with their numbers;
    • each scenario's status, note and medians;
    • the first line of an error.

    It never reads a log, a picture or the PR's description, so nothing made from the game leaves.

  • Safety: the prompt says the JSON is data and never instructions, and the answer must fit a JSON schema. What's shown is cleaned: one paragraph, no @mention, link or HTML, at most 700 characters.

  • Where it shows:

    • at the top of the PR comment;
    • on the PR page on the dashboard;
    • in a manual run's reply;
    • on a manual run's page.
  • Storage: migration 0022_result_summaries.sql adds result_summary, result_summary_round and result_summary_model to commits and runs.

Tests

  • Updated assertions: the e2e flow, the off-target case, and the any/linux+amd comment, for the new layout. Also the cell table's "baseline → PR" order and the folded Measurements.
  • e2e: a fake summary model. The summary is asked once, with the results and no log_tail, shown before the table, cleaned (no link, HTML or live @mention) and on the PR page. Updating the PR again doesn't ask again.
  • Unit tests:
    • a failed rule's wording (whole names, numbers, limit, a note's backticks kept inert);
    • folding an earlier comment;
    • the facts leaving out logs, passed rules and non-numbers;
    • the facts capped in size;
    • finished;
    • clean;
    • write with good, broken and empty answers.

Not done here

  • No screenshots: nothing of Beanstalk runs on this PC, the dev server included, so this has none, although AGENTS.md asks for them.
  • Not built or run: Python passes ast.parse only. CI runs pytest.
  • The scheduled runs' Discord post is left as it was.

🤖 Generated with Claude Code

The farm's PR comment was one long block. On #1574, for example: - the plan was described twice, once as a paragraph and once as bullets; - the answer was buried halfway down; - "2/1 report" was cryptic; - rule names were cut off mid-word (`RX_VK_ACQUI`); - the per-target and per-cell tables seemed to disagree; - every row carried a line of hardware tags. This PR puts the answer first and folds the rest, and adds an AI summary once a run has finished. ## The comment ``` ### Beanstalk test farm · ❌ failed on linux+nvidia Commit `2f24a49` · approved by HumanBean, futondestroyer (double) · [dashboard] > **Summary**, written by AI (`gpt-6.1-terra`) from the numbers below: > <2–4 sentences> Done (started 04:45 UTC, last report 05:52 UTC). Comment `@Beanstalk` to run it again. | Target | Result | Reports | Frame time p99 (ms), switch off → switch on | VRAM peak (MB) | Crashes | | `linux+nvidia` | ❌ fail | 2 [1] [2] | 13.2 → 13.4 (+2%) | — | none | | `intel` | ⏳ queued | 0 of 1 | — | — | — | **Why it failed** - `linux+nvidia`: `hyrule-field · RX_VK_ACQUIRE_AHEAD=1 (PR only)` `field.fps_mean: RX_VK_ACQUIRE_AHEAD=1 vs RX_VK_ACQUIRE_AHEAD=0`: `60.0 → 42.6 (−29.0%); limit 5.0` ▸ What was tested ▸ Every scenario and matrix cell ▸ Measurements on `linux+nvidia`, for the PR description ▸ The testers' hardware ▸ Run by hand on PCs that don't match their target ``` - **The title says the result:** ✅ passed, ❌ failed on the targets that failed, ⚠️ couldn't be judged, 🔄 testing (with the reports so far), waiting for approval, planning, cancelled, or ⛔ when the plan was refused or held. - **The results table:** one row per target. It shows the result, the reports (the count once there are enough, "0 of 1" while short, each linked), frame time and VRAM from the baseline to the PR, and crashes. The hardware moved to its own folded table. - **Why it failed:** each failed rule once per target, with its full name, where it ran, the numbers and the limit. - **Couldn't be judged:** each such report, linked, with the first line of its error. It also says plainly that this is the tester's PC failing, not the PR. - **Folded:** what was tested, the per-cell table, the Measurements block with its Markdown copy for the PR description, the hardware and the off-target runs. - **The per-cell table** now uses the same "baseline → PR" order as the main table. - **The Measurements verdict** keeps rule names whole (up to 140 characters instead of 60). - **Earlier runs** are still folded when a new run starts. `folded_body` drops the whole new title line. ## The AI summary (`results_summary.py`) - **When:** once a round, when every target has finished (pass, fail, couldn't be judged or cancelled, with at least one report). It's claimed in the database before the model is asked (`result_summary_round`), so it's asked only once. A new round (`@Beanstalk` again) gets a new summary. - **Which model:** the planner's, through a new `farm.summary_llm`. It's set only when the app builds that model itself, so tests that pass a fake planner model don't get summaries by accident. If AI planning isn't set up, there's no summary. - **What it reads:** only the farm's own judgement, made small (at most 24 KB): - each target's result and how many reports it needs; - each report's result, and the PC's OS, GPU and driver; - the rules that didn't pass, with their numbers; - each scenario's status, note and medians; - the first line of an error. It never reads a log, a picture or the PR's description, so nothing made from the game leaves. - **Safety:** the prompt says the JSON is data and never instructions, and the answer must fit a JSON schema. What's shown is cleaned: one paragraph, no @mention, link or HTML, at most 700 characters. - **Where it shows:** - at the top of the PR comment; - on the PR page on the dashboard; - in a manual run's reply; - on a manual run's page. - **Storage:** migration `0022_result_summaries.sql` adds `result_summary`, `result_summary_round` and `result_summary_model` to `commits` and `runs`. ## Tests - **Updated assertions:** the e2e flow, the off-target case, and the any/linux+amd comment, for the new layout. Also the cell table's "baseline → PR" order and the folded Measurements. - **e2e:** a fake summary model. The summary is asked once, with the results and no `log_tail`, shown before the table, cleaned (no link, HTML or live @mention) and on the PR page. Updating the PR again doesn't ask again. - **Unit tests:** - a failed rule's wording (whole names, numbers, limit, a note's backticks kept inert); - folding an earlier comment; - the facts leaving out logs, passed rules and non-numbers; - the facts capped in size; - `finished`; - `clean`; - `write` with good, broken and empty answers. ## Not done here - **No screenshots:** nothing of Beanstalk runs on this PC, the dev server included, so this has none, although AGENTS.md asks for them. - **Not built or run:** Python passes `ast.parse` only. CI runs pytest. - **The scheduled runs' Discord post** is left as it was. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(coordinator): a clearer PR comment, answer first, and an AI summary when a farm run ends
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
CI / Coordinator tests (pull_request) Failing after 1m26s
CI / Agent build and checks (pull_request) Successful in 42s
CI / Coordinator image (pull_request) Successful in 11s
a4989c7ec7
The comment's title now says the result (passed, failed on which targets,
couldn't be judged, testing, waiting...). Under it: the AI's summary once
every target has finished, where the run is, one row per target (result,
reports, frame time and VRAM from the baseline to the PR, crashes), and
why it failed or couldn't be judged in plain words, rule names whole.
What was tested, every cell's numbers, the Measurements block for the PR
description, the hardware and off-target runs are folded.

results_summary.py: once a round, when every target is done, the
planner's model sums the results up from the farm's judgement only
(results, rules, medians, the PC, an error's first line; no log, picture
or PR text), cleaned (no @mention, link or HTML, 700 characters). Shown
on the PR comment and page, and a manual run's reply and page.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean changed title from WIP: feat(coordinator): a clearer PR comment, answer first, and an AI summary when a farm run ends to feat(coordinator): a clearer PR comment, answer first, and an AI summary when a farm run ends 2026-11-20 16:03:25 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!117
No description provided.