feat(skills): farm-run, a manual Test Farm run of two nightlies or commits, BigBeans only #13
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Tooling!13
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Tooling:feat/farm-run"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Linux, against a local stand-in for the farm (the farm's manual runs aren't deployed yet) · Needs: Beanstalk's manual runs deployed before it is useful
Why
The Test Farm only tests a pull request today (
@Beanstalkon it). Beanstalk is adding manual runs: a BigBean queuesa comparison of two commits or nightlies that isn't a pull request. This adds the client and a skill, so a BigBean can
say "/magicbean:farm-run nightly-70 nightly-71 on AMD" and get the run queued, its link, and its results when it ends.
What changed
scripts/farm_run.py, standard library only:submit <baseline> <candidate> [--target T ...] [--title] [--instructions],status <run_id>,wait <run_id>(checks every 60 s and prints only what changed, until every job is done, error orcancelled; stops after 6 hours by default with exit code 2, and the run goes on), and
cancel <run_id>. It prints eachref and the commit it resolved to (
nightly-70 -> 1a2b3c4d5e), the dashboard link, and each target's state, resultand report link. The farm's
{"error"}is printed as the farm gave it. A 404 or 405 with noerror, on the routeitself, says the farm doesn't have manual runs yet. A target is checked before any call (tags joined by
+,lowercased,
anydropped).BEANSTALK_URLnames another farm for testing.submitandcancelsendAuthorization: Bearerwith the git.magicbean.dev sign-in thatmagicbean_login.pykeeps (forge_session(), the same one forge.py and git use, renewed by itself). When the farmanswers 401, the script renews the token once (
forge_expire()) and tries again. It is never printed, logged or savedanywhere new, and
statusandwaitsend no token.skills/farm-run/SKILL.md: when to use it; how to turn the user's words into the call (baseline is the older build;"on AMD", "on Windows NVIDIA", "on the Steam Deck" become
amd,windows+nvidia,steamdeck, from Beanstalk's taglist; the other words become instructions); BigBeans only; run
submit, give the link, offerwaitin thebackground; the farm's rules people will hit (the candidate is newer than the baseline, the same tools.lock, unknown
refs, no manual runs yet).
tests/test_farm_run.py: a stand-in for the farm's three routes on 127.0.0.1. It checks the request body, the Bearertoken and the User-Agent, that status sends no token, the farm's error text, the missing route, the renewal on 401, a
bad target, that wait prints only what changed and stops at its limit, that cancel is signed, and that no token is
ever printed.
README.md,AGENTS.md: the script and the skill in their lists. The plugin finds the skill inskills/by itself,as it does the others.
The API it is written against
From Beanstalk's manual-run work, not deployed yet:
POST /api/v1/runswith a Bearer token, body{"baseline", "candidate", "targets": [..], "instructions", "title"}(only
baselineandcandidateare required). It answers 201 with{"run_id", "url", "state", "baseline": {"ref", "sha"}, "candidate": {"ref", "sha"}, "jobs": [{"target", "state", "result", "report_url"}]}, or 4xx with{"error"}. The script also takes a 200.GET /api/v1/runs/{id}, public: the same plustitleandrequester.POST /api/v1/runs/{id}/cancelwith the same Bearer token. The script sends{}, takes 200, 202 or 204, and printsthe run when the answer is one.
done,errororcancelled. With no jobs listed, the run's ownstatedecides.Measurements
Nothing to measure: a command-line client.
Testing
Tested:
python -I -B tests/run_all.py: 5 of 5 pass, the new test among them. I also ran submit, wait and theno-runs answer by hand against the same stand-in and read what they print.
Not tested: the real farm. Its manual runs aren't deployed, and the script was never pointed at
beanstalk.magicbean.dev's
/api/v1/runs.Must be tested: one
submit,waitandcancelagainst the farm once manual runs are deployed, by a BigBean.Risks
What could break: nothing that exists now. The script and the skill are new, and nothing else calls them.
Turning it off: revert the commit.
Compatibility: Nothing changes. It reads the sign-in saved by
magicbean_login.pyas it is.ec2008150btod132b8b620