feat(skills): farm-run, a manual Test Farm run of two nightlies or commits, BigBeans only #13

Merged
HumanBean merged 1 commit from HumanBean/Tooling:feat/farm-run into main 2026-11-16 07:42:10 +00:00
Owner

Type: feat · Risk: low · Tested on: Linux, against a local stand-in for the farm (the farm's manual runs aren't deployed yet) · Needs: Beanstalk's manual runs deployed before it is useful

Why

The Test Farm only tests a pull request today (@Beanstalk on it). Beanstalk is adding manual runs: a BigBean queues
a comparison of two commits or nightlies that isn't a pull request. This adds the client and a skill, so a BigBean can
say "/magicbean:farm-run nightly-70 nightly-71 on AMD" and get the run queued, its link, and its results when it ends.

What changed

  • scripts/farm_run.py, standard library only: submit <baseline> <candidate> [--target T ...] [--title] [--instructions],
    status <run_id>, wait <run_id> (checks every 60 s and prints only what changed, until every job is done, error or
    cancelled; stops after 6 hours by default with exit code 2, and the run goes on), and cancel <run_id>. It prints each
    ref and the commit it resolved to (nightly-70 -> 1a2b3c4d5e), the dashboard link, and each target's state, result
    and report link. The farm's {"error"} is printed as the farm gave it. A 404 or 405 with no error, on the route
    itself, says the farm doesn't have manual runs yet. A target is checked before any call (tags joined by +,
    lowercased, any dropped). BEANSTALK_URL names another farm for testing.
  • The token: submit and cancel send Authorization: Bearer with the git.magicbean.dev sign-in that
    magicbean_login.py keeps (forge_session(), the same one forge.py and git use, renewed by itself). When the farm
    answers 401, the script renews the token once (forge_expire()) and tries again. It is never printed, logged or saved
    anywhere new, and status and wait send no token.
  • skills/farm-run/SKILL.md: when to use it; how to turn the user's words into the call (baseline is the older build;
    "on AMD", "on Windows NVIDIA", "on the Steam Deck" become amd, windows+nvidia, steamdeck, from Beanstalk's tag
    list; the other words become instructions); BigBeans only; run submit, give the link, offer wait in the
    background; the farm's rules people will hit (the candidate is newer than the baseline, the same tools.lock, unknown
    refs, no manual runs yet).
  • tests/test_farm_run.py: a stand-in for the farm's three routes on 127.0.0.1. It checks the request body, the Bearer
    token and the User-Agent, that status sends no token, the farm's error text, the missing route, the renewal on 401, a
    bad target, that wait prints only what changed and stops at its limit, that cancel is signed, and that no token is
    ever printed.
  • README.md, AGENTS.md: the script and the skill in their lists. The plugin finds the skill in skills/ by itself,
    as it does the others.

The API it is written against

From Beanstalk's manual-run work, not deployed yet:

  • POST /api/v1/runs with a Bearer token, body {"baseline", "candidate", "targets": [..], "instructions", "title"}
    (only baseline and candidate are required). It answers 201 with {"run_id", "url", "state", "baseline": {"ref", "sha"}, "candidate": {"ref", "sha"}, "jobs": [{"target", "state", "result", "report_url"}]}, or 4xx with
    {"error"}. The script also takes a 200.
  • GET /api/v1/runs/{id}, public: the same plus title and requester.
  • POST /api/v1/runs/{id}/cancel with the same Bearer token. The script sends {}, takes 200, 202 or 204, and prints
    the run when the answer is one.
  • A job is finished at done, error or cancelled. With no jobs listed, the run's own state decides.

Measurements

Nothing to measure: a command-line client.

Testing

Tested: python -I -B tests/run_all.py: 5 of 5 pass, the new test among them. I also ran submit, wait and the
no-runs answer by hand against the same stand-in and read what they print.

Not tested: the real farm. Its manual runs aren't deployed, and the script was never pointed at
beanstalk.magicbean.dev's /api/v1/runs.

Must be tested: one submit, wait and cancel against the farm once manual runs are deployed, by a BigBean.

Risks

What could break: nothing that exists now. The script and the skill are new, and nothing else calls them.

Turning it off: revert the commit.

Compatibility: Nothing changes. It reads the sign-in saved by magicbean_login.py as it is.

**Type:** feat · **Risk:** low · **Tested on:** Linux, against a local stand-in for the farm (the farm's manual runs aren't deployed yet) · **Needs:** Beanstalk's manual runs deployed before it is useful ## Why The Test Farm only tests a pull request today (`@Beanstalk` on it). Beanstalk is adding manual runs: a BigBean queues a comparison of two commits or nightlies that isn't a pull request. This adds the client and a skill, so a BigBean can say "/magicbean:farm-run nightly-70 nightly-71 on AMD" and get the run queued, its link, and its results when it ends. ## What changed - `scripts/farm_run.py`, standard library only: `submit <baseline> <candidate> [--target T ...] [--title] [--instructions]`, `status <run_id>`, `wait <run_id>` (checks every 60 s and prints only what changed, until every job is done, error or cancelled; stops after 6 hours by default with exit code 2, and the run goes on), and `cancel <run_id>`. It prints each ref and the commit it resolved to (`nightly-70 -> 1a2b3c4d5e`), the dashboard link, and each target's state, result and report link. The farm's `{"error"}` is printed as the farm gave it. A 404 or 405 with no `error`, on the route itself, says the farm doesn't have manual runs yet. A target is checked before any call (tags joined by `+`, lowercased, `any` dropped). `BEANSTALK_URL` names another farm for testing. - **The token**: `submit` and `cancel` send `Authorization: Bearer` with the git.magicbean.dev sign-in that `magicbean_login.py` keeps (`forge_session()`, the same one forge.py and git use, renewed by itself). When the farm answers 401, the script renews the token once (`forge_expire()`) and tries again. It is never printed, logged or saved anywhere new, and `status` and `wait` send no token. - `skills/farm-run/SKILL.md`: when to use it; how to turn the user's words into the call (baseline is the older build; "on AMD", "on Windows NVIDIA", "on the Steam Deck" become `amd`, `windows+nvidia`, `steamdeck`, from Beanstalk's tag list; the other words become instructions); BigBeans only; run `submit`, give the link, offer `wait` in the background; the farm's rules people will hit (the candidate is newer than the baseline, the same tools.lock, unknown refs, no manual runs yet). - `tests/test_farm_run.py`: a stand-in for the farm's three routes on 127.0.0.1. It checks the request body, the Bearer token and the User-Agent, that status sends no token, the farm's error text, the missing route, the renewal on 401, a bad target, that wait prints only what changed and stops at its limit, that cancel is signed, and that no token is ever printed. - `README.md`, `AGENTS.md`: the script and the skill in their lists. The plugin finds the skill in `skills/` by itself, as it does the others. ## The API it is written against From Beanstalk's manual-run work, not deployed yet: - `POST /api/v1/runs` with a Bearer token, body `{"baseline", "candidate", "targets": [..], "instructions", "title"}` (only `baseline` and `candidate` are required). It answers 201 with `{"run_id", "url", "state", "baseline": {"ref", "sha"}, "candidate": {"ref", "sha"}, "jobs": [{"target", "state", "result", "report_url"}]}`, or 4xx with `{"error"}`. The script also takes a 200. - `GET /api/v1/runs/{id}`, public: the same plus `title` and `requester`. - `POST /api/v1/runs/{id}/cancel` with the same Bearer token. The script sends `{}`, takes 200, 202 or 204, and prints the run when the answer is one. - A job is finished at `done`, `error` or `cancelled`. With no jobs listed, the run's own `state` decides. ## Measurements Nothing to measure: a command-line client. ## Testing **Tested:** `python -I -B tests/run_all.py`: 5 of 5 pass, the new test among them. I also ran submit, wait and the no-runs answer by hand against the same stand-in and read what they print. **Not tested:** the real farm. Its manual runs aren't deployed, and the script was never pointed at beanstalk.magicbean.dev's `/api/v1/runs`. **Must be tested:** one `submit`, `wait` and `cancel` against the farm once manual runs are deployed, by a BigBean. ## Risks **What could break:** nothing that exists now. The script and the skill are new, and nothing else calls them. **Turning it off:** revert the commit. **Compatibility:** Nothing changes. It reads the sign-in saved by `magicbean_login.py` as it is.
scripts/farm_run.py queues a Beanstalk manual run (submit), follows it
(status, wait) and cancels it, against the farm's /api/v1/runs. submit and
cancel are signed with the git.magicbean.dev sign-in magicbean_login.py
keeps (renewed once when the farm refuses it); status and wait are public.
The skill turns the user's words into the call: baseline and candidate,
targets from the hardware they name, the rest as instructions.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean deleted branch feat/farm-run 2026-11-16 07:42:10 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Tooling!13
No description provided.