feat: manual runs (coordinator, agent, planner, PR heads and @Beanstalk run) #48

Merged
HumanBean merged 3 commits from HumanBean/Beanstalk:feat/manual-runs into main 2026-11-16 07:55:05 +00:00
Owner

Why

The lead wants BigBeans to be able to queue a test that isn't a pull request, such as comparing two nightlies, two commits, or main against a PR's head. Until now everything in the farm was keyed by a Longshot PR. This pull request covers all of manual runs:

  1. the coordinator and the agent (phase 1);
  2. the AI planner for runs (phase 3);
  3. PR heads as commits, a second BigBean for sensitive paths, nightly and release tags only, @Beanstalk run in comments, and #50's security check for runs (phase 4).

It is three commits, one per phase, on top of main as of #50.

Security notes (CONTRIBUTING.md, "Security comes first")

This loosens what the farm runs, so it needs two maintainers' review.

  • Who can ask. Only a BigBean (a member of FARM_MAINTAINER_TEAMS) can ask for a run, and this is checked with Forgejo on every call. A run's tier is reviewed: a BigBean asking for it counts the same as their @Beanstalk on a PR.
  • PR heads. A run may now test a pull request's head that is not on main. This is the same code that a BigBean could already start with @Beanstalk on that PR, so it opens no new way to run code. The rule for sensitive paths is kept: when the changes the candidate has that main lacks (compare main...candidate, which covers the baseline as well) touch trust.sensitive_paths, or the list of changed files can't be read in full, the run gets no jobs until a different BigBean confirms it (never its requester). Its jobs are then double-reviewed, as a PR's would be.
  • The agent fetches one new kind of ref: refs/pull/<n>/head (n from 1, no leading zero). It is allowed only with capability runs: 2, and only for a run whose min_protocol is 2. Merge refs, other namespaces and refspecs are still refused.
  • Tags. Anyone with write access can push a tag, so runs now take only nightly-* and v<digit>* tags, the nightlies and releases that only the team's workflows and release managers push. I read "v*" as v followed by a digit, so that a tag like vulkan-test is refused. Any other tag is refused with that reason.
  • The security check from #50 screens a run's commits and diff before the plan, as it does a PR's commit.

What changed

Phase 1: coordinator and agent

API (docs/protocol.md, "Manual runs")

  • POST /api/v1/runs takes Authorization: Bearer <Forgejo token> and {"baseline", "candidate", "targets"?, "instructions"?, "title"?}. It answers 201 with the run.
  • GET /api/v1/runs/{id} is public. POST /api/v1/runs/{id}/cancel takes a BigBean's token.
  • Who may ask is checked on every call, the same way agent registration is: GET /user, then membership of FARM_MAINTAINER_TEAMS. Nothing is cached, and the token is never stored or logged.
  • Errors are always JSON {"error": ...}: 401 for a missing or refused token, 403 for someone who isn't a BigBean, 404 for an unknown run, 400 for a refused run, 429 at 5 live runs per person, 503 when Forgejo doesn't answer.
  • Job states are queued, running, done, error and cancelled, with result and report_url kept separate. A run's state is queued, running, done, error or cancelled, plus awaiting_approval (phase 4, below).

Resolving refs (runs.py)

  • A ref can be a full sha, main or a branch, a tag, or nightly-N. nightly-N must match exactly one nightly-N-<sha> tag (found through the git refs API); otherwise the run is refused as ambiguous or missing.
  • Each ref is pinned to a commit when the run is asked for. It also gets the named ref an agent will fetch: refs/heads/main for anything on main, refs/tags/<tag> for a tag.
  • A sha or a branch must already be on main, which is checked with Forgejo's compare API. This Forgejo (17.0.1) has the compare API, and the fake has it too. (Tags and PR heads: see phase 4.)
  • These runs are refused: a candidate that doesn't descend from the baseline, two refs that resolve to the same commit, and a pair whose tools.lock differs (read through the contents API). Each refusal says why.

Plan and jobs

  • Jobs are made the way ensure_jobs makes them. They get tier reviewed, and the requester stands in as author and approver. (The plan: phase 3.)

Database (migration 0012)

  • New runs table.
  • jobs.pr and reports.pr become nullable, and new run_id columns are added. A CHECK requires exactly one of pr and run_id.
  • The jobs foreign key to commits now goes through a generated column, commit_sha (the sha for a PR job, NULL for a run's). So PR jobs still need their commit row, and run jobs don't need one.
  • The slot uniqueness becomes two partial unique indexes: per commit for PR jobs (as before) and per run for run jobs.

Scheduler

  • Run jobs are offered and accepted only by agents whose capabilities say "runs": 1. The rule sits in run_refusal, so it also covers the app's Run on this PC.
  • The "same commit and target" checks are now scoped per PR or per run.
  • The ticket stays v1. For a run it has "kind": "run", run_id, fetch_refs and "strict_base": true, and leaves out pr.
  • GET /api/v1/jobs lists a run's jobs only to apps that say runs.

PR side

  • Every query over a commit's jobs or reports now adds run_id IS NULL. That covers pr_view, outliers, resume, new_round, start again, the planning check and refresh_base.
  • The reporter, webhooks, cancel_jobs and running_jobs work by PR number, so they never touch a run. sweep() returns no PR for a run job.
  • No statuses or labels are written for a run. The only thing on Forgejo is the farm's reply under an @Beanstalk run comment (phase 4).

Agent

  • TicketVerifier.Parse accepts kind pr or run. It refuses mixed tickets: a PR ticket with run fields, or a run ticket with pr.
  • fetch_refs is checked against an allow-list: 1 to 8 refs, each refs/heads/... or refs/tags/... using [A-Za-z0-9._/-], without git's forbidden forms.
  • Workspace.FetchRefsAsync fetches them into refs/beanstalk/run/..., using NetworkRetry and the same git safety settings. It only fetches when one of the two commits is missing.
  • With strict_base, the job is refused with "the baseline is not an ancestor of the candidate". It never falls back to the merge base.
  • Hardware.Capabilities says "runs": 2 (1 in phase 1; 2 since phase 4, below).
  • The window shows a run's title instead of "#0", and progress words say "the baseline" and "the candidate".
  • Older agents refuse run tickets (no pr), and they are never offered run jobs.

Dashboard

  • New pages: /runs (the list) and /runs/{id} (built like the PR page, with "Baseline" and "Candidate").
  • The queue gets a "Manual runs" section, and run jobs are kept out of the PR cards.
  • The report, tester and home pages link to the run, and the report page says baseline and candidate for runs.

Docs: protocol.md (endpoints, the capability, the run ticket), architecture.md (the runs module and tables), using.md (a "Manual runs" section).

Phase 3: the AI planner for runs

  • What the planner reads (planner.gather_run). A run has no PR, so the model gets:

    • the compare between the baseline and the candidate: the commit messages (the newest 60, 12 KB in total), the changed files, and the diff, capped like a PR's (MAX_DIFF);
    • the run's title;
    • the saves at the baseline;
    • .farm/test.toml at the candidate, as a hint;
    • the live tags;
    • the issues that the words or the commit messages name.

    Each goes in its own nonce-fenced data block under the same system rules, plus a short paragraph that says this is a manual run. The requester's instructions act as the maintainer's instructions.

  • New Forge calls: compare(base, head) (the commits oldest first, the changed paths including a rename's old path, and whether the list is complete) and compare_diff. compare_diff reads the web route /{repo}/compare/{a}...{b}.diff, because the API has no compare diff. I checked both against git.magicbean.dev with read-only GETs.

  • Planning in the background. runs.submit stores the run as planning and starts planner.spawn_run, keyed ("plan-run", id) so it can't collide with a PR commit's task. With AI planning off, the run gets the default plan at once, as in phase 1. If the AI plan fails or is refused, or anything unexpected goes wrong, the run falls back to the default plan with the targets that were asked for. resume_pending picks up runs that a restart left planning. A planning run counts toward the requester's 5.

  • The targets asked for are enforced. The model is told they are the requester's choice, and then its [[target]] list is replaced by them (planner.fixed_targets). I chose this over only telling the model: the client and the API show the targets that were asked for, and a model that widened or narrowed them would quietly test other hardware.

  • POST /api/v1/runs/{id}/plan {"instructions"?} (any BigBean) plans a run again, as @Beanstalk plan does for a PR. It starts a new round and cancels the old round's queued and running jobs. With no body, the run is planned again with its own words. A cancelled run is opened again, and an older round's planning finds itself superseded.

  • The run's page shows the plan the way a PR's page does: the model, a planning… chip, "In plain words", "Asked for", the source and round, and any error.

  • devserver plans one seeded run with a canned stand-in model.

Phase 4: PR heads, a second BigBean, tags, @Beanstalk run, the security check

  1. PR heads. #123, pr:123 and refs/pull/123/head resolve through get_pr to the head sha, pinned when the run is asked for. A head that is on main (a merged PR) is fetched through refs/heads/main by any runs 1 app. Otherwise it is fetched through refs/pull/123/head, and the run's min_protocol becomes 2: the scheduler offers its jobs, and accepts them (including Run on this PC), only from agents that say runs: 2. Other agents see the reason, "this run tests a pull request's head: it needs a newer Beanstalk app". On the agent, RunRefs.Allowed takes refs/pull/<n>/head and RunRefs.Protocol is 2. A PR head works on either side. A push after the run was asked for keeps the old head fetchable; a force-push loses it, and testers then fail the job.
  2. A second BigBean. See the security notes. The new state is awaiting_approval. The new endpoint is POST /api/v1/runs/{id}/approve: any BigBean but the requester (403 for the requester), 404 for an unknown run, 409 for a cancelled one, and it is idempotent once confirmed. The run's page shows the sensitive paths and the confirmation, and /runs shows the state.
  3. Tags: only nightly-* and v<digit>*; others are refused with the reason. A branch with the same name as a refused tag is still taken as the branch.
  4. @Beanstalk run <baseline> <candidate> [on <targets>] [words…] works on any Longshot issue or PR (also /beanstalk run …).
    • Routing. Comments on issues now reach commands.handle_issue_comment, which acts only on run and help; anything else on an issue, such as a passing mention, is left alone.
    • Parsing (commands.parse_run). It accepts A B, A vs/against/to/-> B, or A..B; then on t1, t2 (comma-separated); then words, which may continue on the next lines. It is a run only when both tokens look like refs (a sha, #n, pr:n, main, nightly-…, v<digit>…, refs/…, or anything with a /). So on a PR, @Beanstalk run it on AMD is still words for that PR's planner, as before.
    • Who. Only BigBeans; anyone else gets "only members of … can ask for manual runs." A refused run gets "that run was not started: ", and unreadable words get the usage.
    • The reply. The farm reacts 👀 and replies once with the run's link, its state and its plan. runs.update_reply edits that one comment, only when its text changes, after planning, confirmation, cancel, re-plan and every job event. When the run is done, it holds a row per target with links to the reports (example below). @Beanstalk help lists the command.
  5. #50's security check for runs. planner.screen runs on the run's commits and compare diff before the plan. A held run gets no plan and no jobs. Its state becomes error, plan.error starts with security check: , and the page and the issue reply say why. /plan checks it again.
  6. Fix: cancelling a run that has already finished no longer turns it into cancelled. The phase-1 docs already said it was left as it is.

Example of the edited reply when a run is done (made-up data):

@bigbean started a manual run: [nightly-325-e4d035e9 vs nightly-326-bbe421f5](https://beanstalk.magicbean.dev/runs/rn_0123456789abcdef)

Candidate `nightly-326-bbe421f5` (`bbbbbbbbbb`) against baseline `nightly-325-e4d035e9` (`eeeeeeeeee`). **Done.**

> Runs Hyrule Field for a minute after a fresh start, twice per side; fails if the 99th-percentile frame time rises over 10%.

| Target | Result | Reports |
| --- | --- | --- |
| `amd` | 1/1 pass | [1](https://beanstalk.magicbean.dev/reports/rp_1) |
| `windows+nvidia` | 1/1 fail | [1](https://beanstalk.magicbean.dev/reports/rp_2) |

API contract (for MagicBeanDev/Tooling#13, farm_run.py)

Every change is additive. Errors are always JSON {"error"} with 401/403/404 as before; cancel answers the run; job states are unchanged.

  • One new run state, awaiting_approval. It is not finished, so the client keeps waiting (it stops only at done, error or cancelled).
  • While a run is planned its state is queued with jobs: [].
  • A run held by the security check ends as error.
  • New fields:
    • round;
    • plan: {state: planning|ready, source, summary, model, error};
    • approval: {needed, sensitive_paths, approved_by}.
  • New endpoints: POST /api/v1/runs/{id}/plan and POST /api/v1/runs/{id}/approve.

Database: everything is in migration 0012, which this PR adds. On runs it also adds min_protocol, sensitive_paths, needs_approval, approved_by/approved_at, and issue/reply_comment_id/reply_hash. Docs: protocol.md, using.md and architecture.md cover all of it.

Screenshots

All from the dev server's made-up data. Phase 1's (taken earlier, still current):

After: /runs, desktop

After: /runs/{id}, desktop

After: /queue with the Manual runs section, desktop

After: /runs, 390 px

After: /runs/{id}, 390 px

The new states:

After: a run planned by the AI (the dev server's canned model), desktop

After: a planned run, 390 px

After: a run being planned

After: a run held by the security check (its row set by hand in the dev database)

After: a run of a PR's head waiting for a second BigBean, desktop

After: waiting for a second BigBean, 390 px

After: /runs with an awaiting-approval run and a held one

Testing

Coordinator: 486 passed, 1 skipped (Postgres 17, local), on top of current main.

  • tests/test_runs.py: phase 1. Its tests are listed in the commit; the tag cases are updated for phase 4.
  • tests/test_runs_planner.py (fake model):
    • what the model reads, in data blocks;
    • the targets asked for replace the model's;
    • fallbacks to the default plan;
    • re-planning: auth, any BigBean, new and kept words, a cancelled run opened again, an older round superseded;
    • the cap;
    • the page while planning;
    • resume_pending;
    • commit-message caps;
    • compare_doc;
    • the security check holding a run and passing it when planned again.
  • tests/test_runs_pr_heads.py:
    • resolving #n, pr:n and refs/pull/n/head, and a merged PR's head going through main;
    • the runs: 2 gate on offers, accept, Run on this PC and the listing;
    • the ticket's fetch_refs;
    • tags limited to nightly and release;
    • sensitive paths waiting for a second BigBean: the requester's 403, the confirmation, double-reviewed jobs, the page; an incomplete file list counted as sensitive; main's commits needing no confirmation;
    • waiting runs counting toward the cap, and confirming a cancelled run giving 409;
    • parse_run forms;
    • @Beanstalk run on an issue: the reply, 👀, and the one reply edited to Done with the report link; no edit when nothing changed;
    • refusals: a non-BigBean, a refused pair, unreadable words, a refused tag, help; a passing mention is ignored;
    • @Beanstalk run on a PR leaves the PR alone, while @Beanstalk run it on AMD is still planner words.

Agent: agent/tests/test_agent_core.py passes 1822 checks, including refs/pull/<n>/head allowed and its refspec, merge refs and malformed pull refs refused, and capabilities saying runs: 2. dotnet build agent/App -c Release succeeds.

Not tested

  • No real agent has run a run job end to end, and none has fetched refs/pull/<n>/head.
  • No real model has planned a run.
  • HttpForge.compare and compare_diff have only been checked by hand against git.magicbean.dev (read-only).
  • Nothing was posted on a real issue: the comment command was only tested against the fake Forgejo.
  • The Windows app wasn't run.

Deploy notes

  • Migration 0012 runs at start (see phase 1 for what it does to jobs and reports). It doesn't change existing rows.
  • Comments on issues now reach the coordinator. The webhook must send issue-comment events for issues as well as PRs, which the repo-wide webhook already does.
  • The bot token needs to read pulls (already used) and post and edit comments on issues (the same rights as on PRs).
  • Testers take run jobs once their app updates: runs: 1 for main's commits and tags, and this PR's agent (runs: 2) for PR heads. Until then those runs stay queued.

Left to do

  • farm_run.py (Tooling#13) could show awaiting_approval, plan.state and approval, and add approve and plan subcommands.
  • The page has no buttons for re-plan or confirm, because the dashboard has no sign-in. Both are API-only for now.
## Why The lead wants BigBeans to be able to queue a test that isn't a pull request, such as comparing two nightlies, two commits, or main against a PR's head. Until now everything in the farm was keyed by a Longshot PR. This pull request covers all of manual runs: 1. the coordinator and the agent (phase 1); 2. the AI planner for runs (phase 3); 3. PR heads as commits, a second BigBean for sensitive paths, nightly and release tags only, `@Beanstalk run` in comments, and #50's security check for runs (phase 4). It is three commits, one per phase, on top of main as of #50. ## Security notes (CONTRIBUTING.md, "Security comes first") This loosens what the farm runs, so it needs two maintainers' review. - **Who can ask.** Only a BigBean (a member of `FARM_MAINTAINER_TEAMS`) can ask for a run, and this is checked with Forgejo on every call. A run's tier is `reviewed`: a BigBean asking for it counts the same as their `@Beanstalk` on a PR. - **PR heads.** A run may now test a pull request's head that is not on main. This is the same code that a BigBean could already start with `@Beanstalk` on that PR, so it opens no new way to run code. The rule for sensitive paths is kept: when the changes the candidate has that main lacks (`compare main...candidate`, which covers the baseline as well) touch `trust.sensitive_paths`, or the list of changed files can't be read in full, the run gets no jobs until a different BigBean confirms it (never its requester). Its jobs are then `double-reviewed`, as a PR's would be. - **The agent fetches one new kind of ref:** `refs/pull/<n>/head` (n from 1, no leading zero). It is allowed only with capability `runs: 2`, and only for a run whose `min_protocol` is 2. Merge refs, other namespaces and refspecs are still refused. - **Tags.** Anyone with write access can push a tag, so runs now take only `nightly-*` and `v<digit>*` tags, the nightlies and releases that only the team's workflows and release managers push. I read "v*" as `v` followed by a digit, so that a tag like `vulkan-test` is refused. Any other tag is refused with that reason. - **The security check from #50** screens a run's commits and diff before the plan, as it does a PR's commit. ## What changed ### Phase 1: coordinator and agent **API** (docs/protocol.md, "Manual runs") - `POST /api/v1/runs` takes `Authorization: Bearer <Forgejo token>` and `{"baseline", "candidate", "targets"?, "instructions"?, "title"?}`. It answers `201` with the run. - `GET /api/v1/runs/{id}` is public. `POST /api/v1/runs/{id}/cancel` takes a BigBean's token. - Who may ask is checked on every call, the same way agent registration is: `GET /user`, then membership of `FARM_MAINTAINER_TEAMS`. Nothing is cached, and the token is never stored or logged. - Errors are always JSON `{"error": ...}`: 401 for a missing or refused token, 403 for someone who isn't a BigBean, 404 for an unknown run, 400 for a refused run, 429 at 5 live runs per person, 503 when Forgejo doesn't answer. - Job states are `queued`, `running`, `done`, `error` and `cancelled`, with `result` and `report_url` kept separate. A run's state is `queued`, `running`, `done`, `error` or `cancelled`, plus `awaiting_approval` (phase 4, below). **Resolving refs** (`runs.py`) - A ref can be a full sha, `main` or a branch, a tag, or `nightly-N`. `nightly-N` must match exactly one `nightly-N-<sha>` tag (found through the git refs API); otherwise the run is refused as ambiguous or missing. - Each ref is pinned to a commit when the run is asked for. It also gets the named ref an agent will fetch: `refs/heads/main` for anything on main, `refs/tags/<tag>` for a tag. - A sha or a branch must already be on main, which is checked with Forgejo's compare API. This Forgejo (17.0.1) has the compare API, and the fake has it too. (Tags and PR heads: see phase 4.) - These runs are refused: a candidate that doesn't descend from the baseline, two refs that resolve to the same commit, and a pair whose `tools.lock` differs (read through the contents API). Each refusal says why. **Plan and jobs** - Jobs are made the way `ensure_jobs` makes them. They get tier `reviewed`, and the requester stands in as author and approver. (The plan: phase 3.) **Database** (migration 0012) - New `runs` table. - `jobs.pr` and `reports.pr` become nullable, and new `run_id` columns are added. A CHECK requires exactly one of `pr` and `run_id`. - The jobs foreign key to `commits` now goes through a generated column, `commit_sha` (the sha for a PR job, NULL for a run's). So PR jobs still need their commit row, and run jobs don't need one. - The slot uniqueness becomes two partial unique indexes: per commit for PR jobs (as before) and per run for run jobs. **Scheduler** - Run jobs are offered and accepted only by agents whose capabilities say `"runs": 1`. The rule sits in `run_refusal`, so it also covers the app's Run on this PC. - The "same commit and target" checks are now scoped per PR or per run. - The ticket stays v1. For a run it has `"kind": "run"`, `run_id`, `fetch_refs` and `"strict_base": true`, and leaves out `pr`. - `GET /api/v1/jobs` lists a run's jobs only to apps that say `runs`. **PR side** - Every query over a commit's jobs or reports now adds `run_id IS NULL`. That covers `pr_view`, outliers, `resume`, `new_round`, start again, the planning check and `refresh_base`. - The reporter, webhooks, `cancel_jobs` and `running_jobs` work by PR number, so they never touch a run. `sweep()` returns no PR for a run job. - No statuses or labels are written for a run. The only thing on Forgejo is the farm's reply under an `@Beanstalk run` comment (phase 4). **Agent** - `TicketVerifier.Parse` accepts `kind` `pr` or `run`. It refuses mixed tickets: a PR ticket with run fields, or a run ticket with `pr`. - `fetch_refs` is checked against an allow-list: 1 to 8 refs, each `refs/heads/...` or `refs/tags/...` using `[A-Za-z0-9._/-]`, without git's forbidden forms. - `Workspace.FetchRefsAsync` fetches them into `refs/beanstalk/run/...`, using `NetworkRetry` and the same git safety settings. It only fetches when one of the two commits is missing. - With `strict_base`, the job is refused with "the baseline is not an ancestor of the candidate". It never falls back to the merge base. - `Hardware.Capabilities` says `"runs": 2` (1 in phase 1; 2 since phase 4, below). - The window shows a run's title instead of "#0", and progress words say "the baseline" and "the candidate". - Older agents refuse run tickets (no `pr`), and they are never offered run jobs. **Dashboard** - New pages: `/runs` (the list) and `/runs/{id}` (built like the PR page, with "Baseline" and "Candidate"). - The queue gets a "Manual runs" section, and run jobs are kept out of the PR cards. - The report, tester and home pages link to the run, and the report page says baseline and candidate for runs. **Docs**: protocol.md (endpoints, the capability, the run ticket), architecture.md (the runs module and tables), using.md (a "Manual runs" section). ### Phase 3: the AI planner for runs - **What the planner reads** (`planner.gather_run`). A run has no PR, so the model gets: - the compare between the baseline and the candidate: the commit messages (the newest 60, 12 KB in total), the changed files, and the diff, capped like a PR's (`MAX_DIFF`); - the run's title; - the saves at the baseline; - `.farm/test.toml` at the candidate, as a hint; - the live tags; - the issues that the words or the commit messages name. Each goes in its own nonce-fenced data block under the same system rules, plus a short paragraph that says this is a manual run. The requester's `instructions` act as the maintainer's instructions. - **New Forge calls:** `compare(base, head)` (the commits oldest first, the changed paths including a rename's old path, and whether the list is complete) and `compare_diff`. `compare_diff` reads the web route `/{repo}/compare/{a}...{b}.diff`, because the API has no compare diff. I checked both against git.magicbean.dev with read-only GETs. - **Planning in the background.** `runs.submit` stores the run as `planning` and starts `planner.spawn_run`, keyed `("plan-run", id)` so it can't collide with a PR commit's task. With AI planning off, the run gets the default plan at once, as in phase 1. If the AI plan fails or is refused, or anything unexpected goes wrong, the run falls back to the default plan with the targets that were asked for. `resume_pending` picks up runs that a restart left planning. A planning run counts toward the requester's 5. - **The targets asked for are enforced.** The model is told they are the requester's choice, and then its `[[target]]` list is replaced by them (`planner.fixed_targets`). I chose this over only telling the model: the client and the API show the targets that were asked for, and a model that widened or narrowed them would quietly test other hardware. - **`POST /api/v1/runs/{id}/plan`** `{"instructions"?}` (any BigBean) plans a run again, as `@Beanstalk plan` does for a PR. It starts a new round and cancels the old round's queued and running jobs. With no body, the run is planned again with its own words. A cancelled run is opened again, and an older round's planning finds itself superseded. - **The run's page** shows the plan the way a PR's page does: the model, a `planning…` chip, "In plain words", "Asked for", the source and round, and any error. - **`devserver`** plans one seeded run with a canned stand-in model. ### Phase 4: PR heads, a second BigBean, tags, `@Beanstalk run`, the security check 1. **PR heads.** `#123`, `pr:123` and `refs/pull/123/head` resolve through `get_pr` to the head sha, pinned when the run is asked for. A head that is on main (a merged PR) is fetched through `refs/heads/main` by any `runs 1` app. Otherwise it is fetched through `refs/pull/123/head`, and the run's `min_protocol` becomes 2: the scheduler offers its jobs, and accepts them (including Run on this PC), only from agents that say `runs: 2`. Other agents see the reason, "this run tests a pull request's head: it needs a newer Beanstalk app". On the agent, `RunRefs.Allowed` takes `refs/pull/<n>/head` and `RunRefs.Protocol` is 2. A PR head works on either side. A push after the run was asked for keeps the old head fetchable; a force-push loses it, and testers then fail the job. 2. **A second BigBean.** See the security notes. The new state is **`awaiting_approval`**. The new endpoint is **`POST /api/v1/runs/{id}/approve`**: any BigBean but the requester (403 for the requester), 404 for an unknown run, 409 for a cancelled one, and it is idempotent once confirmed. The run's page shows the sensitive paths and the confirmation, and `/runs` shows the state. 3. **Tags**: only `nightly-*` and `v<digit>*`; others are refused with the reason. A branch with the same name as a refused tag is still taken as the branch. 4. **`@Beanstalk run <baseline> <candidate> [on <targets>] [words…]`** works on any Longshot issue or PR (also `/beanstalk run …`). - **Routing.** Comments on issues now reach `commands.handle_issue_comment`, which acts only on `run` and `help`; anything else on an issue, such as a passing mention, is left alone. - **Parsing** (`commands.parse_run`). It accepts `A B`, `A vs/against/to/-> B`, or `A..B`; then `on t1, t2` (comma-separated); then words, which may continue on the next lines. It is a run only when both tokens look like refs (a sha, `#n`, `pr:n`, `main`, `nightly-…`, `v<digit>…`, `refs/…`, or anything with a `/`). So on a PR, `@Beanstalk run it on AMD` is still words for that PR's planner, as before. - **Who.** Only BigBeans; anyone else gets "only members of … can ask for manual runs." A refused run gets "that run was not started: <reason>", and unreadable words get the usage. - **The reply.** The farm reacts 👀 and replies once with the run's link, its state and its plan. `runs.update_reply` edits that one comment, only when its text changes, after planning, confirmation, cancel, re-plan and every job event. When the run is done, it holds a row per target with links to the reports (example below). `@Beanstalk help` lists the command. 5. **#50's security check for runs.** `planner.screen` runs on the run's commits and compare diff before the plan. A held run gets no plan and no jobs. Its state becomes `error`, `plan.error` starts with `security check: `, and the page and the issue reply say why. `/plan` checks it again. 6. **Fix:** cancelling a run that has already finished no longer turns it into `cancelled`. The phase-1 docs already said it was left as it is. Example of the edited reply when a run is done (made-up data): ``` @bigbean started a manual run: [nightly-325-e4d035e9 vs nightly-326-bbe421f5](https://beanstalk.magicbean.dev/runs/rn_0123456789abcdef) Candidate `nightly-326-bbe421f5` (`bbbbbbbbbb`) against baseline `nightly-325-e4d035e9` (`eeeeeeeeee`). **Done.** > Runs Hyrule Field for a minute after a fresh start, twice per side; fails if the 99th-percentile frame time rises over 10%. | Target | Result | Reports | | --- | --- | --- | | `amd` | 1/1 pass | [1](https://beanstalk.magicbean.dev/reports/rp_1) | | `windows+nvidia` | 1/1 fail | [1](https://beanstalk.magicbean.dev/reports/rp_2) | ``` ### API contract (for MagicBeanDev/Tooling#13, `farm_run.py`) Every change is additive. Errors are always JSON `{"error"}` with 401/403/404 as before; cancel answers the run; job states are unchanged. - **One new run state, `awaiting_approval`.** It is not finished, so the client keeps waiting (it stops only at `done`, `error` or `cancelled`). - **While a run is planned** its state is `queued` with `jobs: []`. - **A run held by the security check** ends as `error`. - **New fields:** - `round`; - `plan: {state: planning|ready, source, summary, model, error}`; - `approval: {needed, sensitive_paths, approved_by}`. - **New endpoints:** `POST /api/v1/runs/{id}/plan` and `POST /api/v1/runs/{id}/approve`. **Database:** everything is in migration 0012, which this PR adds. On `runs` it also adds `min_protocol`, `sensitive_paths`, `needs_approval`, `approved_by`/`approved_at`, and `issue`/`reply_comment_id`/`reply_hash`. **Docs:** protocol.md, using.md and architecture.md cover all of it. ## Screenshots All from the dev server's made-up data. Phase 1's (taken earlier, still current): ![After: /runs, desktop](https://git.magicbean.dev/attachments/a6e5b8ce-69d4-4bdd-8025-a5f98bab2ff5) ![After: /runs/{id}, desktop](https://git.magicbean.dev/attachments/9e75bde7-b510-482f-bc8a-23f7d6277f1b) ![After: /queue with the Manual runs section, desktop](https://git.magicbean.dev/attachments/1b7a242b-8c3c-408f-aaae-722e80520e3b) ![After: /runs, 390 px](https://git.magicbean.dev/attachments/c580869e-1759-459c-bde5-db2e3cbe56d5) ![After: /runs/{id}, 390 px](https://git.magicbean.dev/attachments/592f9fc4-878c-4c2e-acf8-eb6bbd1175d1) The new states: ![After: a run planned by the AI (the dev server's canned model), desktop](https://git.magicbean.dev/attachments/92f5a782-0987-4053-9d98-e7add56a566c) ![After: a planned run, 390 px](https://git.magicbean.dev/attachments/d07e8b80-da54-41b9-8034-9929df4c2adc) ![After: a run being planned](https://git.magicbean.dev/attachments/e392a168-5b65-4dad-ae40-97c13cab396f) ![After: a run held by the security check (its row set by hand in the dev database)](https://git.magicbean.dev/attachments/ada5c673-b782-40b7-b41f-f7c9c1c78e57) ![After: a run of a PR's head waiting for a second BigBean, desktop](https://git.magicbean.dev/attachments/48135c81-2641-4c64-b5f8-43fd46bf9bd6) ![After: waiting for a second BigBean, 390 px](https://git.magicbean.dev/attachments/2b3c453f-028a-4f5d-bfe4-3c6df5915b1b) ![After: /runs with an awaiting-approval run and a held one](https://git.magicbean.dev/attachments/5b7a2243-95f9-4f07-9256-b9a7b31887a2) ## Testing **Coordinator:** 486 passed, 1 skipped (Postgres 17, local), on top of current main. - `tests/test_runs.py`: phase 1. Its tests are listed in the commit; the tag cases are updated for phase 4. - `tests/test_runs_planner.py` (fake model): - what the model reads, in data blocks; - the targets asked for replace the model's; - fallbacks to the default plan; - re-planning: auth, any BigBean, new and kept words, a cancelled run opened again, an older round superseded; - the cap; - the page while planning; - `resume_pending`; - commit-message caps; - `compare_doc`; - the security check holding a run and passing it when planned again. - `tests/test_runs_pr_heads.py`: - resolving `#n`, `pr:n` and `refs/pull/n/head`, and a merged PR's head going through main; - the `runs: 2` gate on offers, accept, Run on this PC and the listing; - the ticket's `fetch_refs`; - tags limited to nightly and release; - sensitive paths waiting for a second BigBean: the requester's 403, the confirmation, `double-reviewed` jobs, the page; an incomplete file list counted as sensitive; main's commits needing no confirmation; - waiting runs counting toward the cap, and confirming a cancelled run giving 409; - `parse_run` forms; - `@Beanstalk run` on an issue: the reply, 👀, and the one reply edited to Done with the report link; no edit when nothing changed; - refusals: a non-BigBean, a refused pair, unreadable words, a refused tag, `help`; a passing mention is ignored; - `@Beanstalk run` on a PR leaves the PR alone, while `@Beanstalk run it on AMD` is still planner words. **Agent:** `agent/tests/test_agent_core.py` passes 1822 checks, including `refs/pull/<n>/head` allowed and its refspec, merge refs and malformed pull refs refused, and capabilities saying `runs: 2`. `dotnet build agent/App -c Release` succeeds. **Not tested** - No real agent has run a run job end to end, and none has fetched `refs/pull/<n>/head`. - No real model has planned a run. - `HttpForge.compare` and `compare_diff` have only been checked by hand against git.magicbean.dev (read-only). - Nothing was posted on a real issue: the comment command was only tested against the fake Forgejo. - The Windows app wasn't run. ## Deploy notes - **Migration 0012** runs at start (see phase 1 for what it does to `jobs` and `reports`). It doesn't change existing rows. - **Comments on issues now reach the coordinator.** The webhook must send issue-comment events for issues as well as PRs, which the repo-wide webhook already does. - The bot token needs to read pulls (already used) and post and edit comments on issues (the same rights as on PRs). - Testers take run jobs once their app updates: `runs: 1` for main's commits and tags, and this PR's agent (`runs: 2`) for PR heads. Until then those runs stay queued. ## Left to do - `farm_run.py` (Tooling#13) could show `awaiting_approval`, `plan.state` and `approval`, and add `approve` and `plan` subcommands. - The page has no buttons for re-plan or confirm, because the dashboard has no sign-in. Both are API-only for now.
feat: manual runs, phase 1: a BigBean's baseline-against-candidate test
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
54ac1cb927
POST /api/v1/runs lets a BigBean queue a test that is not a pull request: a
baseline commit against a candidate (two nightlies, main against a tag). Both
refs are pinned to commits on main or a tag, the candidate must descend from
the baseline and tools.lock must match. The plan is the default plan narrowed
to the asked targets. Jobs and reports name the run instead of a PR
(migration 0012); nothing about a run goes to Forgejo.

The agent takes a ticket of kind run (no pr, run_id, fetch_refs limited to
refs/heads and refs/tags, strict_base) and says so with the runs capability;
only such agents are offered run jobs.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
feat(runs): the AI planner plans manual runs, and a run can be planned again
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 3s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
a5dbb20585
A run is planned as a PR's head is, from what it has instead of a PR: the
compare between its baseline and its candidate (commit messages, changed
files, the diff), the requester's words as the maintainer's instructions,
the saves at the baseline, .farm/test.toml at the candidate and the issues
the words or commits name, in the same untrusted data blocks. The targets
asked for replace the plan's. A failed plan, or AI planning off, gives the
default plan with those targets.

POST /api/v1/runs/{id}/plan (BigBeans) plans a run again with new words or
its own: a new round, the old jobs cancelled. The API's run gains round and
plan {state, source, summary, model}; a run being planned is queued. The
run's page shows the plan as a PR's page does.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
feat(runs): pull requests' heads, a second BigBean for sensitive paths, and @Beanstalk run
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
6ca6cedce1
A run's baseline or candidate may be a pull request's head (#123, pr:123):
pinned through get_pr and fetched as refs/pull/123/head when it is not on
main. Agents fetch that ref only from runs 2 (RunRefs.Allowed, the
capability bumped to 2), and the scheduler offers such runs to runs 2 alone.

When the candidate is not on main and compare main...candidate touches
trust.sensitive_paths (or its files can't all be listed), the run waits in
awaiting_approval until another BigBean, never the requester, confirms it
with POST /api/v1/runs/{id}/approve; its jobs are then double-reviewed.

Tags are kept to nightly-* and v<digit>*, the team's own; others are
refused with the reason.

`@Beanstalk run <baseline> <candidate> [on <targets>] [words]` on any
issue or PR asks for a run (BigBeans only): the farm replies there with the
run and edits that one reply as it goes. Comments on issues now reach the
farm, which acts only on run and help there. A finished run is no longer
turned into a cancelled one by cancel.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean force-pushed feat/manual-runs from 6ca6cedce1
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
to 56944abfd2
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 6s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
2026-11-16 07:14:20 +00:00
Compare
HumanBean changed title from feat: manual runs, phase 1 (coordinator and agent) to feat: manual runs (coordinator, agent, planner, PR heads and @Beanstalk run) 2026-11-16 07:16:20 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!48
No description provided.