feat: manual runs (coordinator, agent, planner, PR heads and @Beanstalk run) #48
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk!48
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Beanstalk:feat/manual-runs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
The lead wants BigBeans to be able to queue a test that isn't a pull request, such as comparing two nightlies, two commits, or main against a PR's head. Until now everything in the farm was keyed by a Longshot PR. This pull request covers all of manual runs:
@Beanstalk runin comments, and #50's security check for runs (phase 4).It is three commits, one per phase, on top of main as of #50.
Security notes (CONTRIBUTING.md, "Security comes first")
This loosens what the farm runs, so it needs two maintainers' review.
FARM_MAINTAINER_TEAMS) can ask for a run, and this is checked with Forgejo on every call. A run's tier isreviewed: a BigBean asking for it counts the same as their@Beanstalkon a PR.@Beanstalkon that PR, so it opens no new way to run code. The rule for sensitive paths is kept: when the changes the candidate has that main lacks (compare main...candidate, which covers the baseline as well) touchtrust.sensitive_paths, or the list of changed files can't be read in full, the run gets no jobs until a different BigBean confirms it (never its requester). Its jobs are thendouble-reviewed, as a PR's would be.refs/pull/<n>/head(n from 1, no leading zero). It is allowed only with capabilityruns: 2, and only for a run whosemin_protocolis 2. Merge refs, other namespaces and refspecs are still refused.nightly-*andv<digit>*tags, the nightlies and releases that only the team's workflows and release managers push. I read "v*" asvfollowed by a digit, so that a tag likevulkan-testis refused. Any other tag is refused with that reason.What changed
Phase 1: coordinator and agent
API (docs/protocol.md, "Manual runs")
POST /api/v1/runstakesAuthorization: Bearer <Forgejo token>and{"baseline", "candidate", "targets"?, "instructions"?, "title"?}. It answers201with the run.GET /api/v1/runs/{id}is public.POST /api/v1/runs/{id}/canceltakes a BigBean's token.GET /user, then membership ofFARM_MAINTAINER_TEAMS. Nothing is cached, and the token is never stored or logged.{"error": ...}: 401 for a missing or refused token, 403 for someone who isn't a BigBean, 404 for an unknown run, 400 for a refused run, 429 at 5 live runs per person, 503 when Forgejo doesn't answer.queued,running,done,errorandcancelled, withresultandreport_urlkept separate. A run's state isqueued,running,done,errororcancelled, plusawaiting_approval(phase 4, below).Resolving refs (
runs.py)mainor a branch, a tag, ornightly-N.nightly-Nmust match exactly onenightly-N-<sha>tag (found through the git refs API); otherwise the run is refused as ambiguous or missing.refs/heads/mainfor anything on main,refs/tags/<tag>for a tag.tools.lockdiffers (read through the contents API). Each refusal says why.Plan and jobs
ensure_jobsmakes them. They get tierreviewed, and the requester stands in as author and approver. (The plan: phase 3.)Database (migration 0012)
runstable.jobs.prandreports.prbecome nullable, and newrun_idcolumns are added. A CHECK requires exactly one ofprandrun_id.commitsnow goes through a generated column,commit_sha(the sha for a PR job, NULL for a run's). So PR jobs still need their commit row, and run jobs don't need one.Scheduler
"runs": 1. The rule sits inrun_refusal, so it also covers the app's Run on this PC."kind": "run",run_id,fetch_refsand"strict_base": true, and leaves outpr.GET /api/v1/jobslists a run's jobs only to apps that sayruns.PR side
run_id IS NULL. That coverspr_view, outliers,resume,new_round, start again, the planning check andrefresh_base.cancel_jobsandrunning_jobswork by PR number, so they never touch a run.sweep()returns no PR for a run job.@Beanstalk runcomment (phase 4).Agent
TicketVerifier.Parseacceptskindprorrun. It refuses mixed tickets: a PR ticket with run fields, or a run ticket withpr.fetch_refsis checked against an allow-list: 1 to 8 refs, eachrefs/heads/...orrefs/tags/...using[A-Za-z0-9._/-], without git's forbidden forms.Workspace.FetchRefsAsyncfetches them intorefs/beanstalk/run/..., usingNetworkRetryand the same git safety settings. It only fetches when one of the two commits is missing.strict_base, the job is refused with "the baseline is not an ancestor of the candidate". It never falls back to the merge base.Hardware.Capabilitiessays"runs": 2(1 in phase 1; 2 since phase 4, below).pr), and they are never offered run jobs.Dashboard
/runs(the list) and/runs/{id}(built like the PR page, with "Baseline" and "Candidate").Docs: protocol.md (endpoints, the capability, the run ticket), architecture.md (the runs module and tables), using.md (a "Manual runs" section).
Phase 3: the AI planner for runs
What the planner reads (
planner.gather_run). A run has no PR, so the model gets:MAX_DIFF);.farm/test.tomlat the candidate, as a hint;Each goes in its own nonce-fenced data block under the same system rules, plus a short paragraph that says this is a manual run. The requester's
instructionsact as the maintainer's instructions.New Forge calls:
compare(base, head)(the commits oldest first, the changed paths including a rename's old path, and whether the list is complete) andcompare_diff.compare_diffreads the web route/{repo}/compare/{a}...{b}.diff, because the API has no compare diff. I checked both against git.magicbean.dev with read-only GETs.Planning in the background.
runs.submitstores the run asplanningand startsplanner.spawn_run, keyed("plan-run", id)so it can't collide with a PR commit's task. With AI planning off, the run gets the default plan at once, as in phase 1. If the AI plan fails or is refused, or anything unexpected goes wrong, the run falls back to the default plan with the targets that were asked for.resume_pendingpicks up runs that a restart left planning. A planning run counts toward the requester's 5.The targets asked for are enforced. The model is told they are the requester's choice, and then its
[[target]]list is replaced by them (planner.fixed_targets). I chose this over only telling the model: the client and the API show the targets that were asked for, and a model that widened or narrowed them would quietly test other hardware.POST /api/v1/runs/{id}/plan{"instructions"?}(any BigBean) plans a run again, as@Beanstalk plandoes for a PR. It starts a new round and cancels the old round's queued and running jobs. With no body, the run is planned again with its own words. A cancelled run is opened again, and an older round's planning finds itself superseded.The run's page shows the plan the way a PR's page does: the model, a
planning…chip, "In plain words", "Asked for", the source and round, and any error.devserverplans one seeded run with a canned stand-in model.Phase 4: PR heads, a second BigBean, tags,
@Beanstalk run, the security check#123,pr:123andrefs/pull/123/headresolve throughget_prto the head sha, pinned when the run is asked for. A head that is on main (a merged PR) is fetched throughrefs/heads/mainby anyruns 1app. Otherwise it is fetched throughrefs/pull/123/head, and the run'smin_protocolbecomes 2: the scheduler offers its jobs, and accepts them (including Run on this PC), only from agents that sayruns: 2. Other agents see the reason, "this run tests a pull request's head: it needs a newer Beanstalk app". On the agent,RunRefs.Allowedtakesrefs/pull/<n>/headandRunRefs.Protocolis 2. A PR head works on either side. A push after the run was asked for keeps the old head fetchable; a force-push loses it, and testers then fail the job.awaiting_approval. The new endpoint isPOST /api/v1/runs/{id}/approve: any BigBean but the requester (403 for the requester), 404 for an unknown run, 409 for a cancelled one, and it is idempotent once confirmed. The run's page shows the sensitive paths and the confirmation, and/runsshows the state.nightly-*andv<digit>*; others are refused with the reason. A branch with the same name as a refused tag is still taken as the branch.@Beanstalk run <baseline> <candidate> [on <targets>] [words…]works on any Longshot issue or PR (also/beanstalk run …).commands.handle_issue_comment, which acts only onrunandhelp; anything else on an issue, such as a passing mention, is left alone.commands.parse_run). It acceptsA B,A vs/against/to/-> B, orA..B; thenon t1, t2(comma-separated); then words, which may continue on the next lines. It is a run only when both tokens look like refs (a sha,#n,pr:n,main,nightly-…,v<digit>…,refs/…, or anything with a/). So on a PR,@Beanstalk run it on AMDis still words for that PR's planner, as before.runs.update_replyedits that one comment, only when its text changes, after planning, confirmation, cancel, re-plan and every job event. When the run is done, it holds a row per target with links to the reports (example below).@Beanstalk helplists the command.planner.screenruns on the run's commits and compare diff before the plan. A held run gets no plan and no jobs. Its state becomeserror,plan.errorstarts withsecurity check:, and the page and the issue reply say why./planchecks it again.cancelled. The phase-1 docs already said it was left as it is.Example of the edited reply when a run is done (made-up data):
API contract (for MagicBeanDev/Tooling#13,
farm_run.py)Every change is additive. Errors are always JSON
{"error"}with 401/403/404 as before; cancel answers the run; job states are unchanged.awaiting_approval. It is not finished, so the client keeps waiting (it stops only atdone,errororcancelled).queuedwithjobs: [].error.round;plan: {state: planning|ready, source, summary, model, error};approval: {needed, sensitive_paths, approved_by}.POST /api/v1/runs/{id}/planandPOST /api/v1/runs/{id}/approve.Database: everything is in migration 0012, which this PR adds. On
runsit also addsmin_protocol,sensitive_paths,needs_approval,approved_by/approved_at, andissue/reply_comment_id/reply_hash. Docs: protocol.md, using.md and architecture.md cover all of it.Screenshots
All from the dev server's made-up data. Phase 1's (taken earlier, still current):
The new states:
Testing
Coordinator: 486 passed, 1 skipped (Postgres 17, local), on top of current main.
tests/test_runs.py: phase 1. Its tests are listed in the commit; the tag cases are updated for phase 4.tests/test_runs_planner.py(fake model):resume_pending;compare_doc;tests/test_runs_pr_heads.py:#n,pr:nandrefs/pull/n/head, and a merged PR's head going through main;runs: 2gate on offers, accept, Run on this PC and the listing;fetch_refs;double-reviewedjobs, the page; an incomplete file list counted as sensitive; main's commits needing no confirmation;parse_runforms;@Beanstalk runon an issue: the reply, 👀, and the one reply edited to Done with the report link; no edit when nothing changed;help; a passing mention is ignored;@Beanstalk runon a PR leaves the PR alone, while@Beanstalk run it on AMDis still planner words.Agent:
agent/tests/test_agent_core.pypasses 1822 checks, includingrefs/pull/<n>/headallowed and its refspec, merge refs and malformed pull refs refused, and capabilities sayingruns: 2.dotnet build agent/App -c Releasesucceeds.Not tested
refs/pull/<n>/head.HttpForge.compareandcompare_diffhave only been checked by hand against git.magicbean.dev (read-only).Deploy notes
jobsandreports). It doesn't change existing rows.runs: 1for main's commits and tags, and this PR's agent (runs: 2) for PR heads. Until then those runs stay queued.Left to do
farm_run.py(Tooling#13) could showawaiting_approval,plan.stateandapproval, and addapproveandplansubcommands.A run is planned as a PR's head is, from what it has instead of a PR: the compare between its baseline and its candidate (commit messages, changed files, the diff), the requester's words as the maintainer's instructions, the saves at the baseline, .farm/test.toml at the candidate and the issues the words or commits name, in the same untrusted data blocks. The targets asked for replace the plan's. A failed plan, or AI planning off, gives the default plan with those targets. POST /api/v1/runs/{id}/plan (BigBeans) plans a run again with new words or its own: a new round, the old jobs cancelled. The API's run gains round and plan {state, source, summary, model}; a run being planned is queued. The run's page shows the plan as a PR's page does. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]>6ca6cedce156944abfd2feat: manual runs, phase 1 (coordinator and agent)to feat: manual runs (coordinator, agent, planner, PR heads and @Beanstalk run)