feat(runs): the AI planner plans manual runs (phase 3) #49

Closed
HumanBean wants to merge 2 commits from HumanBean:feat/runs-planner into main
Owner

Manual runs (#48) all run the default plan, so a BigBean's words about what to test are kept but never used. This lets the AI planner plan a run the way it plans a PR, and lets a BigBean plan a run again with new words.

Builds on #48 and should merge after it. This branch is stacked on #48's commit, so until #48 merges the diff here shows both: the change of this PR is its last commit, feat(runs): the AI planner plans manual runs, ....

What changed

  • The planner reads a run (planner.gather_run). A run has no PR, so the model reads:

    • the compare between the baseline and the candidate: the commit messages (newest 60, 12 KB together), the changed files, and the diff, capped like a PR's (MAX_DIFF);
    • the run's title;
    • the saves at the baseline;
    • .farm/test.toml at the candidate, as a hint;
    • the live tags;
    • the issues that the words or the commit messages name (gather_issues, as for a PR).

    Each of these goes in its own nonce-fenced data block under the same system rules, plus one short paragraph that says this is a manual run ("PR" means the candidate, "main" the baseline). The requester's instructions take the place of the maintainer's instructions.

  • New Forge calls: compare(base, head) (the commits oldest first, the changed paths including a rename's old path, and whether the list is complete) and compare_diff(base, head, limit), which reads the web route /{repo}/compare/{a}...{b}.diff because the API has no compare diff. I checked both against git.magicbean.dev with read-only GETs. The fake Forgejo has them too.

  • Submitting (runs.submit) stores the run as planning with no plan and starts planner.spawn_run in the background. Its tasks are keyed ("plan-run", id), so they can't collide with a commit's. With AI planning off it uses the default plan at once, as before. A failed or refused AI plan, or any unexpected error, falls back to the default plan with the targets asked for, so a run never stays planning. resume_pending restarts runs that a restart left planning. A planning run counts toward the requester's 5 live runs, because it has no jobs yet.

  • Targets asked for are enforced, not just suggested. The model is told they are the maintainer's choice, and the plan's [[target]] list is then replaced with them (planner.fixed_targets, which keeps the model's min_reports for a target with the same name). I chose this over a hint because the client and the API show the targets that were asked for. If the model could widen or narrow them, the run would quietly test hardware the BigBean didn't choose, and a hint has no way to catch that.

  • POST /api/v1/runs/{id}/plan {"instructions": "..."} (any BigBean, like cancel) plans the run again, as @Beanstalk plan does for a PR. It starts a new round (runs.round), and the old round's queued and running jobs are cancelled with the reason re-planned by <login>. With no body, the run is planned again with its own words. A cancelled run is opened again. An old round's planning that is still running finds itself superseded, because the round is claimed as planner.claim claims a commit's.

  • API (additive only): the run gains round and plan: {state: planning|ready, source, summary, model}. No new run or job states. While the run is being planned, its state is queued with jobs: [] (it isn't finished, so farm_run.py keeps waiting). targets are the plan's, or the ones asked for while planning. Errors are unchanged: always JSON {"error"}, with 401/403/404/400.

  • The run page shows the plan like a PR's page: the model (or "default"), a planning… chip, the plan's summary, "In plain words", "Asked for", the source and round, and any error. The /runs cards show planning.

  • Nothing about a run goes to Forgejo. The planner only reads from it.

  • devserver plans its second seeded run with a canned stand-in model, so the page can be seen with a plan.

  • Docs: protocol.md (the endpoint and the new fields), using.md and architecture.md.

Tests

  • New tests/test_runs_planner.py, using the fake model from the planner's tests:
    • what the model reads (system text, words, title, commit, file, diff, hint, saves, linked issue), each in its data block;
    • the targets that were asked for replace the model's;
    • a model error and a refused answer each fall back to the default plan;
    • re-planning: 401/403/404/400, any BigBean may do it, the old jobs are cancelled, round 2 gets the new words, no body keeps the old words, a cancelled run is opened again, and the old round's planning is superseded;
    • a planning run counts toward the cap;
    • the page while planning and after;
    • resume_pending picks up a run that was left planning;
    • commit messages are capped;
    • compare_doc parsing.
  • The whole coordinator suite passes: 456 passed, 1 skipped (Postgres 17, local).
  • Not tested: a real model planning a real run, and the real compare_diff web route inside the coordinator (I only checked it by hand with curl).

Screenshots

From the dev server (made-up data; the plan is the canned stand-in's). Before: the run page in #48, which showed "Plan (default)" and "Kept for the planner".

After, desktop: After: a planned run's page, desktop

After, 390 px: After: a planned run's page, phone

Manual runs (#48) all run the default plan, so a BigBean's words about what to test are kept but never used. This lets the AI planner plan a run the way it plans a PR, and lets a BigBean plan a run again with new words. **Builds on #48 and should merge after it.** This branch is stacked on #48's commit, so until #48 merges the diff here shows both: the change of this PR is its last commit, `feat(runs): the AI planner plans manual runs, ...`. ## What changed - **The planner reads a run** (`planner.gather_run`). A run has no PR, so the model reads: - the compare between the baseline and the candidate: the commit messages (newest 60, 12 KB together), the changed files, and the diff, capped like a PR's (`MAX_DIFF`); - the run's title; - the saves at the baseline; - `.farm/test.toml` at the candidate, as a hint; - the live tags; - the issues that the words or the commit messages name (`gather_issues`, as for a PR). Each of these goes in its own nonce-fenced data block under the same system rules, plus one short paragraph that says this is a manual run ("PR" means the candidate, "main" the baseline). The requester's `instructions` take the place of the maintainer's instructions. - **New Forge calls**: `compare(base, head)` (the commits oldest first, the changed paths including a rename's old path, and whether the list is complete) and `compare_diff(base, head, limit)`, which reads the web route `/{repo}/compare/{a}...{b}.diff` because the API has no compare diff. I checked both against git.magicbean.dev with read-only GETs. The fake Forgejo has them too. - **Submitting** (`runs.submit`) stores the run as `planning` with no plan and starts `planner.spawn_run` in the background. Its tasks are keyed `("plan-run", id)`, so they can't collide with a commit's. With AI planning off it uses the default plan at once, as before. A failed or refused AI plan, or any unexpected error, falls back to the default plan with the targets asked for, so a run never stays `planning`. `resume_pending` restarts runs that a restart left `planning`. A planning run counts toward the requester's 5 live runs, because it has no jobs yet. - **Targets asked for are enforced, not just suggested.** The model is told they are the maintainer's choice, and the plan's `[[target]]` list is then replaced with them (`planner.fixed_targets`, which keeps the model's `min_reports` for a target with the same name). I chose this over a hint because the client and the API show the targets that were asked for. If the model could widen or narrow them, the run would quietly test hardware the BigBean didn't choose, and a hint has no way to catch that. - **`POST /api/v1/runs/{id}/plan`** `{"instructions": "..."}` (any BigBean, like cancel) plans the run again, as `@Beanstalk plan` does for a PR. It starts a new round (`runs.round`), and the old round's queued and running jobs are cancelled with the reason `re-planned by <login>`. With no body, the run is planned again with its own words. A cancelled run is opened again. An old round's planning that is still running finds itself superseded, because the round is claimed as `planner.claim` claims a commit's. - **API (additive only)**: the run gains `round` and `plan: {state: planning|ready, source, summary, model}`. **No new run or job states.** While the run is being planned, its `state` is `queued` with `jobs: []` (it isn't finished, so `farm_run.py` keeps waiting). `targets` are the plan's, or the ones asked for while planning. Errors are unchanged: always JSON `{"error"}`, with 401/403/404/400. - **The run page** shows the plan like a PR's page: the model (or "default"), a `planning…` chip, the plan's summary, "In plain words", "Asked for", the source and round, and any error. The `/runs` cards show `planning`. - **Nothing about a run goes to Forgejo.** The planner only reads from it. - `devserver` plans its second seeded run with a canned stand-in model, so the page can be seen with a plan. - Docs: protocol.md (the endpoint and the new fields), using.md and architecture.md. ## Tests - New `tests/test_runs_planner.py`, using the fake model from the planner's tests: - what the model reads (system text, words, title, commit, file, diff, hint, saves, linked issue), each in its data block; - the targets that were asked for replace the model's; - a model error and a refused answer each fall back to the default plan; - re-planning: 401/403/404/400, any BigBean may do it, the old jobs are cancelled, round 2 gets the new words, no body keeps the old words, a cancelled run is opened again, and the old round's planning is superseded; - a planning run counts toward the cap; - the page while planning and after; - `resume_pending` picks up a run that was left planning; - commit messages are capped; - `compare_doc` parsing. - The whole coordinator suite passes: 456 passed, 1 skipped (Postgres 17, local). - **Not tested:** a real model planning a real run, and the real `compare_diff` web route inside the coordinator (I only checked it by hand with curl). ## Screenshots From the dev server (made-up data; the plan is the canned stand-in's). Before: the run page in #48, which showed "Plan (default)" and "Kept for the planner". After, desktop: ![After: a planned run's page, desktop](https://git.magicbean.dev/attachments/be06e57b-89e6-4313-8cfa-dd4ea5e96048) After, 390 px: ![After: a planned run's page, phone](https://git.magicbean.dev/attachments/aca5dd46-3366-4858-a4e8-f923b933bea8)
feat: manual runs, phase 1: a BigBean's baseline-against-candidate test
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
54ac1cb927
POST /api/v1/runs lets a BigBean queue a test that is not a pull request: a
baseline commit against a candidate (two nightlies, main against a tag). Both
refs are pinned to commits on main or a tag, the candidate must descend from
the baseline and tools.lock must match. The plan is the default plan narrowed
to the asked targets. Jobs and reports name the run instead of a PR
(migration 0012); nothing about a run goes to Forgejo.

The agent takes a ticket of kind run (no pr, run_id, fetch_refs limited to
refs/heads and refs/tags, strict_base) and says so with the runs capability;
only such agents are offered run jobs.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
feat(runs): the AI planner plans manual runs, and a run can be planned again
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 3s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
a5dbb20585
A run is planned as a PR's head is, from what it has instead of a PR: the
compare between its baseline and its candidate (commit messages, changed
files, the diff), the requester's words as the maintainer's instructions,
the saves at the baseline, .farm/test.toml at the candidate and the issues
the words or commits name, in the same untrusted data blocks. The targets
asked for replace the plan's. A failed plan, or AI planning off, gives the
default plan with those targets.

POST /api/v1/runs/{id}/plan (BigBeans) plans a run again with new words or
its own: a new round, the old jobs cancelled. The API's run gains round and
plan {state, source, summary, model}; a run being planned is queued. The
run's page shows the plan as a PR's page does.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Owner

Folded into #48: manual runs land as one pull request, and this phase is now a commit on its branch.

Folded into #48: manual runs land as one pull request, and this phase is now a commit on its branch.
HumanBean closed this pull request 2026-11-16 07:01:34 +00:00
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 3s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!49
No description provided.