feat: test farm coordinator, dashboard and desktop agent #3

Merged

HumanBean merged 1 commit from HumanBean/Beanstalk:feat/test-farm into main 2026-11-15 17:18:04 +00:00

Conversation 3 Commits 1 Files changed 295 +40384 -23

HumanBean commented 2026-11-14 18:48:00 +00:00

Owner

• Copy link
• Copy Markdown

Longshot PRs need testing on hardware the core team doesn't own, and a full build and A/B run takes too long to babysit. This adds Beanstalk, the Longshot Test Farm: an approver comments @Beanstalk on a PR, an AI model writes a test plan for that exact commit, and volunteer PCs build it next to main in a sandbox, run both and post the numbers back.

What changed

Coordinator (coordinator/beanstalk/, Python 3.13, Starlette, Postgres)

• web.py: the agent API (docs/protocol.md), the HMAC-checked /webhook, and the read-only dashboard (templates/, static/).
• webhooks.py: PR opened, pushed, labelled, closed. A new push records the commit with its sensitive paths, cancels the old commit's jobs and makes its 👍 requests stale. Label events the bot itself caused are ignored, so the farm's approval label can't echo back as a PR change.
• commands.py: @Beanstalk and /beanstalk (start, run again, <words>, plan, cancel (queued jobs; running ones finish and count), cancel force and revoke (running ones stop within about 5 s), help), 👀 for approvers, and 👍 requests from everyone else, polled since Forgejo sends no webhook for reactions. A stale request's reply is edited, not answered again.
• trust.py: the tiers (untrusted, reviewed, double-reviewed; no auto tier) and the sensitive paths that need two approvers.
• planner.py, llm.py: the AI planner. It fences the PR's text in nonce-marked data blocks, checks the answer as a plan, retries once, and falls back to default_plan.toml. llm.py calls OpenAI's Codex backend with a ChatGPT sign-in whose rotating tokens live in Postgres.
• plan.py, menu_options.py, destinations.py, default_plan.toml: plan schema 4 (targets by tags, plus VRAM and RAM sizes (vram_gb = 8, "> 12", "4-8", rounded to whole GB as people say them) and GPU/CPU model patterns (gpu = "gtx 1080", "gtx 10xx", cpu = "5800x3d"), build variants from an allowlist of build options (march, hot_opt, split_giants), start states, settings, switches, matrix, steps, probes, rules; a plan without variants stays schema 3 for older agents), the Insert menu's rows (generated by scripts/gen_menu_options.py) and the teleport destinations.
• scheduler.py: jobs per commit, round, target and slot; offers (rarest hardware first, counting tags, sizes and models; author share cap), leases, heartbeats every few seconds (next_s) so a stop arrives at once, release, a tester's skip and Run it again (unskip), the sweep. Every try is recorded in job_attempts (opened on accept, closed by the report, a fail, a stop or skip, an expired lease or a cancel).
• tickets.py, agents.py, reports.py: signed tickets, agent registration with a Forgejo token, report checks (report v3: what was in use per cell, mismatch; reports are plain JSON authenticated by the agent token and the lease, an old signed envelope still unwrapped) and outlier flags.
• reporter.py: the statuses (none until the farm is engaged on a PR), the one PR comment with a Measurements block in Longshot's PR template format and a Flags and settings table, and the label (changed only when it is wrong).
• Dashboard: the landing page with downloads, the queue, each PR (stacked, with the raw plan), the Testers page with each tester's status (testing, ready, ready but no job fits, waiting on the busy check with its reason, paused, a problem) and a page per tester (hardware, which queued jobs fit and why not, every try and how it ended).
• migrations/0001–0009 (0006 turns stored os/gpu targets, jobs and agents into tags; 0007 drops the agents' report keys and adds jobs.skipped_agents; 0008 the testers' status; 0009 job_attempts, filled from past reports, failures and skips), tests/, Dockerfile, hash-pinned requirements*.lock.

Agent (agent/, .NET 10, Avalonia 12)

• Core/Envelope.cs, Policy.cs, JobRunner.cs: ticket checks against the pinned coordinator key, the volunteer's policy (when to test and the disk it may use; older keys ignored), the job loop with heartbeats at the coordinator's pace, Stop (release), Skip this test and Run it again, which takes jobs only while testing is on, and only while the PC has room (CPU under 50 %, and 4 GB of memory free or a quarter of it if that is less; Core/Load.cs). It tells the coordinator its status (POST /api/v1/agents/me/status), and reports that could not be sent are kept and sent again (PendingReport.cs). The agent key is gone: an old agent_key.pem is deleted at start.
• Core/AgentConfig.cs, DataFolder.cs: the setup, testing on/off and the close prompts' answers; one data folder for everything big, with its minimum (100 GB on Linux, 150 GB on Windows) and the free-space warning.
• Core/Workspace.cs, JobPipeline.cs: tools fetched outside the sandbox (and Longshot's ZBIC reader, which reading some .nup files needs), the warm main base and the PR's pack with scripts/dev_packs.py, both runtimes, per build variant (each variant its own cached base, keyed by commit and options). The game's romfs is kept once and shown read-only to the base, the pack and both runtimes; lifted code is kept between main builds and read-only for PRs; the shader caches carry over when main moves. Each runtime gets its own folder, written only by its own side, and the baseline its own base.json, so a PR's code can't change what main's side runs.
• Core/Sandbox.cs: bubblewrap on Linux. Core/WindowsSandbox.cs, WindowsSelfTest.cs, WindowsPlan.cs, WindowsTools.cs: the AppContainer mode, the beanstalk-runner fallback and the self-test that chooses between them.
• Core/Tags.cs: the PC's tags (OS, GPU maker, Vulkan driver, igpu/dgpu, Steam Deck from DMI), the volunteer's extra_tags, and the same size and model matching as the coordinator.
• Core/Plan.cs, ScenarioRunner.cs, Probes.cs, Report.cs, Hardware.cs: the same plan checks as the coordinator, the game driven through its control bridge, the probes, what was in use (each setting read back, each switch's "in use" log line, each build's CPU level) and the scrubbed report.
• Core/AgentUpdate.cs, App/ViewModels/UpdatesViewModel.cs: self-update in the window, as the Longshot Launcher does it: a check at start and every 6 hours, a banner on the main screen (Update Beanstalk, with progress and Cancel), and an Updates section in Settings (Automatic, Nightly, Beta, Release; Check now). While a test runs it offers "Update after this test": the runner is held so it takes no new job, the update waits for the test to end, then the agent restarts into the new version with testing still on. Development builds don't update themselves.
• App/: the window (sign-in, then the setup with the launcher's game picker, then the main screen: Start testing / Stop, the running test (its live page, with Skip this test), recent tests and each test's page; Settings and Log out), the tray and the beanstalk-agent command line (setup --data-dir, start, stop). tests/: checks, including every plan fixture read by both sides. scripts/: packaging and releases.

Deployment: deploy/compose.example.yml, deploy/openai-login.sh.

CI: .forgejo/workflows/ci.yml (PR title; coordinator tests and agent checks on main), release-agent.yml: the agent's channels, nightly (each merge to main that changes the agent), beta (a vX.Y.Z-beta.N tag, the first v0.1.0-beta.1) and release (a vX.Y.Z tag, the first v0.1.0), signed with Beanstalk's own release key.

Docs: README.md (the front door), docs/using.md (new: for BigBeans and PR authors), docs/architecture.md (new), docs/test-plan.md, docs/protocol.md, docs/report-schema.md, docs/deploy.md, agent/README.md, CONTRIBUTING.md, AGENTS.md. The wiki Spec is updated to match.

How to review

A suggested reading order, from the overview to the riskiest parts:

1. docs/architecture.md, then docs/using.md for what users see.
2. Trust and commands: coordinator/beanstalk/trust.py, commands.py, webhooks.py. Check that nothing runs without an approver's approval of the exact head, and that sensitive paths need two.
3. Tickets and reports: docs/protocol.md, coordinator/beanstalk/tickets.py, scheduler.py (accept, heartbeat, release), agent/Core/Envelope.cs, coordinator/beanstalk/reports.py (plain reports, the transition for old envelopes).
4. The sandbox: agent/Core/Sandbox.cs, Workspace.cs (what runs outside it), then WindowsSandbox.cs and WindowsSelfTest.cs.
5. The planner: coordinator/beanstalk/planner.py (fencing, checks, fallback), llm.py, and plan.py with docs/test-plan.md.

Testing

Tested

• Coordinator test suite: 388 passed, 1 skipped (a throwaway local Postgres). Agent checks: 1519 passed.
• The coordinator is live at https://beanstalk.magicbean.dev (image v21) and receives Longshot's webhooks. Volunteers are testing with dev builds: at the time of writing about ten testers on Linux and Windows have registered, and Linux testers have built and run PRs.
• Targets are tags: a tester matches a target when it has every tag it lists (tags = [] is any PC; radv, linux+amd, steamdeck). Tag matching, names, the validator's refusals (including old os/gpu targets), reserved extra_tags, the scheduler's order, migration 0006 and Steam Deck detection from made-up DMI files are covered by those tests. Deployed (v11 onward; migration 0006 ran on the live database); no real job has used one yet.
• AI planner dry run on Longshot #307: a valid plan in 13 s.
• Real runs on a Linux PC with an RTX 3090: agent setup and hardware probe; a cold main base build in bwrap; the PR's pack and both runtimes; the game booting in the sandbox with the control bridge, recording frame times and VRAM.
• Bugs those runs found, now fixed:
  • bwrap's --die-with-parent killed builds when a .NET thread-pool thread retired. Processes now start from one long-lived thread.
  • The control socket's path was over the 107-byte Unix limit. Sockets now go in ~/.cache/beanstalk-sock/<random>, never /run/user.
  • Git's protocol.file.allow blocked the job's local clone. It is allowed for that one clone only.
• Each fresh game start spends about 5 minutes creating pipelines, so a small A/B plan takes about 40 minutes.

Not tested

• No complete A/B report from a real job yet. The furthest run (#718, on a volunteer's RTX 4070) built both sides and started the game, then stopped with "the game did not read the controller": being looked into.
• The smaller data folder on a real job (the shared read-only romfs, the kept caches), and on Windows at all.
• Build variants, the in-use records and mismatch on a real game and build: only made-up logs and fake bridges so far. Switch "in use" detection relies on Longshot naming each switch in its own log line; a switch that logs nothing is recorded as not_logged, not a mismatch.
• Cancel force's stop latency and Skip this test against the live coordinator.
• The new-save routine (start = "new"), live setting changes (option_set and read-back), load-time polling, the RX_VK_PASS_TIMING and shader-preparation parsers on real logs.
• Repository saves (start = "save:<name>") on a real job (Longshot #543 and the first save, #610, are merged).
• The agent's release and self-update against a real release: no release has been made yet (the first nightly comes with the merge).

Needs other testers

• Windows: testers have registered with the AppContainer sandbox, but no Windows job has run yet (Longshot #544, the control pipe for sandboxed games, is merged).
• A Linux AMD PC, and a second agent of the same kind to check min_reports and outlier flags.

Deployment

Live at https://beanstalk.magicbean.dev on the Unraid magicbean stack: magicbean-beanstalk (the coordinator, image v21) and magicbean-beanstalk-db (Postgres 18), behind the Cloudflare tunnel. The Longshot webhook goes to the public /webhook, checked with HMAC. The bot account is @Beanstalk (team FarmBot with write on Longshot, and in BigBeans to read its members). Approvers are BigBeans only (FARM_MAINTAINER_TEAMS=BigBeans), and BigBeans may start their own PRs (FARM_SELF_APPROVE_TEAMS=BigBeans). The planner uses gpt-6.1-terra at medium effort through a ChatGPT sign-in made by deploy/openai-login.sh. docs/deploy.md has every step.

Follow-ups

• Longshot #542: the Test Farm page for volunteers (ready).
• After this merges: the first nightly, then redeploying the coordinator from main, and testers moving from the dev builds to the nightly (dev builds don't update themselves).
• Longshot: a --fsr pass-through in scripts/dev_packs.py would let FSR 4 become a build variant (left out for now).
• The first beta (v0.1.0-beta.1) and release (v0.1.0).
• The first Windows job.
• A complete A/B report from a real job.

Longshot PRs need testing on hardware the core team doesn't own, and a full build and A/B run takes too long to babysit. This adds Beanstalk, the Longshot Test Farm: an approver comments `@Beanstalk` on a PR, an AI model writes a test plan for that exact commit, and volunteer PCs build it next to `main` in a sandbox, run both and post the numbers back. ## What changed **Coordinator** (`coordinator/beanstalk/`, Python 3.13, Starlette, Postgres) - `web.py`: the agent API (`docs/protocol.md`), the HMAC-checked `/webhook`, and the read-only dashboard (`templates/`, `static/`). - `webhooks.py`: PR opened, pushed, labelled, closed. A new push records the commit with its sensitive paths, cancels the old commit's jobs and makes its 👍 requests stale. Label events the bot itself caused are ignored, so the farm's approval label can't echo back as a PR change. - `commands.py`: `@Beanstalk` and `/beanstalk` (start, run again, `<words>`, `plan`, `cancel` (queued jobs; running ones finish and count), `cancel force` and `revoke` (running ones stop within about 5 s), `help`), 👀 for approvers, and 👍 requests from everyone else, polled since Forgejo sends no webhook for reactions. A stale request's reply is edited, not answered again. - `trust.py`: the tiers (untrusted, reviewed, double-reviewed; no auto tier) and the sensitive paths that need two approvers. - `planner.py`, `llm.py`: the AI planner. It fences the PR's text in nonce-marked data blocks, checks the answer as a plan, retries once, and falls back to `default_plan.toml`. `llm.py` calls OpenAI's Codex backend with a ChatGPT sign-in whose rotating tokens live in Postgres. - `plan.py`, `menu_options.py`, `destinations.py`, `default_plan.toml`: plan schema 4 (targets by tags, plus VRAM and RAM sizes (`vram_gb = 8`, `"> 12"`, `"4-8"`, rounded to whole GB as people say them) and GPU/CPU model patterns (`gpu = "gtx 1080"`, `"gtx 10xx"`, `cpu = "5800x3d"`), build variants from an allowlist of build options (`march`, `hot_opt`, `split_giants`), start states, settings, switches, matrix, steps, probes, rules; a plan without variants stays schema 3 for older agents), the Insert menu's rows (generated by `scripts/gen_menu_options.py`) and the teleport destinations. - `scheduler.py`: jobs per commit, round, target and slot; offers (rarest hardware first, counting tags, sizes and models; author share cap), leases, heartbeats every few seconds (`next_s`) so a stop arrives at once, release, a tester's skip and Run it again (`unskip`), the sweep. Every try is recorded in `job_attempts` (opened on accept, closed by the report, a fail, a stop or skip, an expired lease or a cancel). - `tickets.py`, `agents.py`, `reports.py`: signed tickets, agent registration with a Forgejo token, report checks (report v3: what was in use per cell, `mismatch`; reports are plain JSON authenticated by the agent token and the lease, an old signed envelope still unwrapped) and outlier flags. - `reporter.py`: the statuses (none until the farm is engaged on a PR), the one PR comment with a Measurements block in Longshot's PR template format and a Flags and settings table, and the label (changed only when it is wrong). - Dashboard: the landing page with downloads, the queue, each PR (stacked, with the raw plan), the Testers page with each tester's status (testing, ready, ready but no job fits, waiting on the busy check with its reason, paused, a problem) and a page per tester (hardware, which queued jobs fit and why not, every try and how it ended). - `migrations/0001`–`0009` (0006 turns stored `os`/`gpu` targets, jobs and agents into tags; 0007 drops the agents' report keys and adds `jobs.skipped_agents`; 0008 the testers' status; 0009 `job_attempts`, filled from past reports, failures and skips), `tests/`, `Dockerfile`, hash-pinned `requirements*.lock`. **Agent** (`agent/`, .NET 10, Avalonia 12) - `Core/Envelope.cs`, `Policy.cs`, `JobRunner.cs`: ticket checks against the pinned coordinator key, the volunteer's policy (when to test and the disk it may use; older keys ignored), the job loop with heartbeats at the coordinator's pace, Stop (release), Skip this test and Run it again, which takes jobs only while testing is on, and only while the PC has room (CPU under 50 %, and 4 GB of memory free or a quarter of it if that is less; `Core/Load.cs`). It tells the coordinator its status (`POST /api/v1/agents/me/status`), and reports that could not be sent are kept and sent again (`PendingReport.cs`). The agent key is gone: an old `agent_key.pem` is deleted at start. - `Core/AgentConfig.cs`, `DataFolder.cs`: the setup, testing on/off and the close prompts' answers; one data folder for everything big, with its minimum (100 GB on Linux, 150 GB on Windows) and the free-space warning. - `Core/Workspace.cs`, `JobPipeline.cs`: tools fetched outside the sandbox (and Longshot's ZBIC reader, which reading some `.nup` files needs), the warm `main` base and the PR's pack with `scripts/dev_packs.py`, both runtimes, per build variant (each variant its own cached base, keyed by commit and options). The game's `romfs` is kept once and shown read-only to the base, the pack and both runtimes; lifted code is kept between `main` builds and read-only for PRs; the shader caches carry over when `main` moves. Each runtime gets its own folder, written only by its own side, and the baseline its own `base.json`, so a PR's code can't change what `main`'s side runs. - `Core/Sandbox.cs`: bubblewrap on Linux. `Core/WindowsSandbox.cs`, `WindowsSelfTest.cs`, `WindowsPlan.cs`, `WindowsTools.cs`: the AppContainer mode, the `beanstalk-runner` fallback and the self-test that chooses between them. - `Core/Tags.cs`: the PC's tags (OS, GPU maker, Vulkan driver, `igpu`/`dgpu`, Steam Deck from DMI), the volunteer's `extra_tags`, and the same size and model matching as the coordinator. - `Core/Plan.cs`, `ScenarioRunner.cs`, `Probes.cs`, `Report.cs`, `Hardware.cs`: the same plan checks as the coordinator, the game driven through its control bridge, the probes, what was in use (each setting read back, each switch's "in use" log line, each build's CPU level) and the scrubbed report. - `Core/AgentUpdate.cs`, `App/ViewModels/UpdatesViewModel.cs`: self-update in the window, as the Longshot Launcher does it: a check at start and every 6 hours, a banner on the main screen (Update Beanstalk, with progress and Cancel), and an Updates section in Settings (Automatic, Nightly, Beta, Release; Check now). While a test runs it offers "Update after this test": the runner is held so it takes no new job, the update waits for the test to end, then the agent restarts into the new version with testing still on. Development builds don't update themselves. - `App/`: the window (sign-in, then the setup with the launcher's game picker, then the main screen: Start testing / Stop, the running test (its live page, with Skip this test), recent tests and each test's page; Settings and Log out), the tray and the `beanstalk-agent` command line (`setup --data-dir`, `start`, `stop`). `tests/`: checks, including every plan fixture read by both sides. `scripts/`: packaging and releases. **Deployment:** `deploy/compose.example.yml`, `deploy/openai-login.sh`. **CI:** `.forgejo/workflows/ci.yml` (PR title; coordinator tests and agent checks on `main`), `release-agent.yml`: the agent's channels, nightly (each merge to `main` that changes the agent), beta (a `vX.Y.Z-beta.N` tag, the first `v0.1.0-beta.1`) and release (a `vX.Y.Z` tag, the first `v0.1.0`), signed with Beanstalk's own release key. **Docs:** `README.md` (the front door), `docs/using.md` (new: for BigBeans and PR authors), `docs/architecture.md` (new), `docs/test-plan.md`, `docs/protocol.md`, `docs/report-schema.md`, `docs/deploy.md`, `agent/README.md`, `CONTRIBUTING.md`, `AGENTS.md`. The wiki [Spec](https://git.magicbean.dev/MagicBeanDev/Beanstalk/wiki/Spec) is updated to match. ## How to review A suggested reading order, from the overview to the riskiest parts: 1. `docs/architecture.md`, then `docs/using.md` for what users see. 2. Trust and commands: `coordinator/beanstalk/trust.py`, `commands.py`, `webhooks.py`. Check that nothing runs without an approver's approval of the exact head, and that sensitive paths need two. 3. Tickets and reports: `docs/protocol.md`, `coordinator/beanstalk/tickets.py`, `scheduler.py` (`accept`, `heartbeat`, `release`), `agent/Core/Envelope.cs`, `coordinator/beanstalk/reports.py` (plain reports, the transition for old envelopes). 4. The sandbox: `agent/Core/Sandbox.cs`, `Workspace.cs` (what runs outside it), then `WindowsSandbox.cs` and `WindowsSelfTest.cs`. 5. The planner: `coordinator/beanstalk/planner.py` (fencing, checks, fallback), `llm.py`, and `plan.py` with `docs/test-plan.md`. ## Testing **Tested** - Coordinator test suite: 388 passed, 1 skipped (a throwaway local Postgres). Agent checks: 1519 passed. - The coordinator is live at https://beanstalk.magicbean.dev (image v21) and receives Longshot's webhooks. Volunteers are testing with dev builds: at the time of writing about ten testers on Linux and Windows have registered, and Linux testers have built and run PRs. - Targets are tags: a tester matches a target when it has every tag it lists (`tags = []` is any PC; `radv`, `linux+amd`, `steamdeck`). Tag matching, names, the validator's refusals (including old `os`/`gpu` targets), reserved `extra_tags`, the scheduler's order, migration 0006 and Steam Deck detection from made-up DMI files are covered by those tests. Deployed (v11 onward; migration 0006 ran on the live database); no real job has used one yet. - AI planner dry run on Longshot #307: a valid plan in 13 s. - Real runs on a Linux PC with an RTX 3090: agent setup and hardware probe; a cold `main` base build in bwrap; the PR's pack and both runtimes; the game booting in the sandbox with the control bridge, recording frame times and VRAM. - Bugs those runs found, now fixed: - bwrap's `--die-with-parent` killed builds when a .NET thread-pool thread retired. Processes now start from one long-lived thread. - The control socket's path was over the 107-byte Unix limit. Sockets now go in `~/.cache/beanstalk-sock/<random>`, never `/run/user`. - Git's `protocol.file.allow` blocked the job's local clone. It is allowed for that one clone only. - Each fresh game start spends about 5 minutes creating pipelines, so a small A/B plan takes about 40 minutes. **Not tested** - No complete A/B report from a real job yet. The furthest run (#718, on a volunteer's RTX 4070) built both sides and started the game, then stopped with "the game did not read the controller": being looked into. - The smaller data folder on a real job (the shared read-only `romfs`, the kept caches), and on Windows at all. - Build variants, the in-use records and `mismatch` on a real game and build: only made-up logs and fake bridges so far. Switch "in use" detection relies on Longshot naming each switch in its own log line; a switch that logs nothing is recorded as `not_logged`, not a mismatch. - Cancel force's stop latency and Skip this test against the live coordinator. - The new-save routine (`start = "new"`), live setting changes (`option_set` and read-back), load-time polling, the `RX_VK_PASS_TIMING` and shader-preparation parsers on real logs. - Repository saves (`start = "save:<name>"`) on a real job (Longshot #543 and the first save, #610, are merged). - The agent's release and self-update against a real release: no release has been made yet (the first nightly comes with the merge). **Needs other testers** - Windows: testers have registered with the AppContainer sandbox, but no Windows job has run yet (Longshot #544, the control pipe for sandboxed games, is merged). - A Linux AMD PC, and a second agent of the same kind to check `min_reports` and outlier flags. ## Deployment Live at https://beanstalk.magicbean.dev on the Unraid magicbean stack: `magicbean-beanstalk` (the coordinator, image v21) and `magicbean-beanstalk-db` (Postgres 18), behind the Cloudflare tunnel. The Longshot webhook goes to the public `/webhook`, checked with HMAC. The bot account is `@Beanstalk` (team FarmBot with write on Longshot, and in BigBeans to read its members). Approvers are BigBeans only (`FARM_MAINTAINER_TEAMS=BigBeans`), and BigBeans may start their own PRs (`FARM_SELF_APPROVE_TEAMS=BigBeans`). The planner uses `gpt-6.1-terra` at medium effort through a ChatGPT sign-in made by `deploy/openai-login.sh`. `docs/deploy.md` has every step. ## Follow-ups - Longshot #542: the Test Farm page for volunteers (ready). - After this merges: the first nightly, then redeploying the coordinator from `main`, and testers moving from the dev builds to the nightly (dev builds don't update themselves). - Longshot: a `--fsr` pass-through in `scripts/dev_packs.py` would let FSR 4 become a build variant (left out for now). - The first beta (`v0.1.0-beta.1`) and release (`v0.1.0`). - The first Windows job. - A complete A/B report from a real job.

HumanBean added 1 commit 2026-11-14 18:48:00 +00:00

feat: test farm coordinator, dashboard and desktop agent ... c5b50b8463

Beanstalk, the Longshot Test Farm (wiki: Spec): maintainers comment
/beanstalk on a pull request, an AI planner writes the test plan, and
volunteer agents build that exact commit next to main in a sandbox, run
the plan on both and post signed reports back to the pull request.

- coordinator/: Python 3.13, Starlette, Postgres. Forgejo webhooks, the
  /beanstalk commands, SHA-pinned approvals (two maintainers for build
  paths), the job queue with leases, signed job tickets, signed reports,
  commit statuses and one results comment per PR, the dashboard, and the
  AI planner (OpenAI through a ChatGPT sign-in, gpt-6.1-terra).
- agent/: .NET 10 + Avalonia desktop app and CLI (beanstalk-agent), built
  like the LongshotLauncher: sign-in, hardware, policy, toolchain prefetch,
  dev_packs builds, the bwrap sandbox on Linux and AppContainer or a
  low-rights user on Windows, the control-bridge scenario runner, probes,
  rules and signed reports.
- docs/: the protocol, the test plan schema 2, the report schema, deploy.
- CI: PR titles; coordinator tests on Postgres and the agent build and
  checks on pushes to main.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>

HumanBean force-pushed feat/test-farm from c5b50b8463 to 361bb491f9 2026-11-14 18:51:39 +00:00 Compare

HumanBean commented 2026-11-14 18:51:47 +00:00

Author

Owner

• Copy link
• Copy Markdown

Pushed: the bot account is now @Beanstalk, and a mention works as the trigger (@Beanstalk, @Beanstalk <what to test, in words>, @Beanstalk cancel and so on; quoted lines and code blocks are ignored). The farm reacts 👀 to every comment it takes as a command, before planning. Repo saves copy only slot_00, never account.sav or option.sav. 276 coordinator tests pass, plus 611 agent checks. Live on the coordinator as v6.

Pushed: the bot account is now `@Beanstalk`, and a mention works as the trigger (`@Beanstalk`, `@Beanstalk <what to test, in words>`, `@Beanstalk cancel` and so on; quoted lines and code blocks are ignored). The farm reacts 👀 to every comment it takes as a command, before planning. Repo saves copy only `slot_00`, never `account.sav` or `option.sav`. 276 coordinator tests pass, plus 611 agent checks. Live on the coordinator as v6.

HumanBean referenced this pull request from MagicBeanDev/Longshot 2026-11-14 18:54:08 +00:00

docs: the Test Farm page, which replaces test requests #542

HumanBean referenced this pull request from MagicBeanDev/Longshot 2026-11-14 18:55:05 +00:00

feat(saves): scrubbed saves in the repository, and oot_saves.py is now longshot_saves.py #543

HumanBean referenced this pull request from MagicBeanDev/Longshot 2026-11-14 18:57:09 +00:00

fix(control): the bridge's pipe lets a sandboxed game's agent in (AppContainer, a farm user) #544

HumanBean force-pushed feat/test-farm from 361bb491f9 to 88178d71db 2026-11-14 19:01:32 +00:00 Compare

HumanBean force-pushed feat/test-farm from 88178d71db to 49caafa2ed 2026-11-14 19:27:46 +00:00 Compare

HumanBean commented 2026-11-14 19:28:03 +00:00

Author

Owner

• Copy link
• Copy Markdown

Pushed (live as v8):

• Approvers are the BigBeans team only (FARM_MAINTAINER_TEAMS), and BigBeans may start the farm on their own PRs (FARM_SELF_APPROVE_TEAMS).
• Anyone can ask with @Beanstalk: the farm replies, and an approver approves with 👍 on the request. Reactions are polled every 30 s, since Forgejo sends no webhook for them. The 👍 is refused if the PR changed since the request, and requests expire after 7 days.
• run is folded into the trigger: @Beanstalk on a finished or cancelled commit runs it again with the same plan. targets is removed.
• The dashboard lists only PRs being tested, with their state.
• 291 coordinator tests pass.

Pushed (live as v8): - Approvers are the `BigBeans` team only (`FARM_MAINTAINER_TEAMS`), and BigBeans may start the farm on their own PRs (`FARM_SELF_APPROVE_TEAMS`). - Anyone can ask with `@Beanstalk`: the farm replies, and an approver approves with 👍 on the request. Reactions are polled every 30 s, since Forgejo sends no webhook for them. The 👍 is refused if the PR changed since the request, and requests expire after 7 days. - `run` is folded into the trigger: `@Beanstalk` on a finished or cancelled commit runs it again with the same plan. `targets` is removed. - The dashboard lists only PRs being tested, with their state. - 291 coordinator tests pass.

HumanBean force-pushed feat/test-farm from 49caafa2ed to 61165d5bea 2026-11-14 19:54:22 +00:00 Compare

HumanBean force-pushed feat/test-farm from 61165d5bea to ffdc1044d5 2026-11-14 20:17:23 +00:00 Compare

HumanBean force-pushed feat/test-farm from ffdc1044d5 to 0522114500 2026-11-14 21:45:30 +00:00 Compare

HumanBean force-pushed feat/test-farm from 0522114500 to 591998faf8 2026-11-14 22:04:21 +00:00 Compare

HumanBean force-pushed feat/test-farm from 591998faf8 to 0278608bb2 2026-11-14 22:23:07 +00:00 Compare

HumanBean force-pushed feat/test-farm from 0278608bb2 to dce17f50de 2026-11-14 22:26:27 +00:00 Compare

HumanBean force-pushed feat/test-farm from dce17f50de to 9bffbefd07 2026-11-14 22:32:44 +00:00 Compare

HumanBean referenced this pull request from a commit 2026-11-14 22:56:11 +00:00

feat(saves): scrubbed saves in the repository, and oot_saves.py is now longshot_saves.py (#543)

HumanBean force-pushed feat/test-farm from 9bffbefd07 to 97d91374c5 2026-11-14 23:15:58 +00:00 Compare

HumanBean referenced this pull request from a commit 2026-11-14 23:31:55 +00:00

fix(control): the bridge's pipe lets a sandboxed game's agent in (AppContainer, a farm user) (#544)

HumanBean commented 2026-11-15 04:37:16 +00:00

Author

Owner

• Copy link
• Copy Markdown

Test builds of the agent (dev-ee5fd24), updated 2026-11-15. They are not signed and not a release.

• Windows x64: beanstalk-agent-dev-ee5fd24-windows-x64.zip holds beanstalk-agent-dev-ee5fd24-windows-x64-setup.exe, a single self-contained file; no .NET install is needed. It is unsigned, so SmartScreen will warn: choose More info › Run anyway.
• Linux x86_64: beanstalk-agent-dev-ee5fd24-linux-x86_64.zip holds beanstalk-agent-dev-ee5fd24-x86_64.AppImage. Run chmod +x on it, then run it.

They are built from this branch plus local changes that are not pushed here yet:

• VRAM, RAM, GPU and CPU targets;
• the smaller data folder, now 100 GB on Linux and 150 GB on Windows;
• Run it again for skipped tests;
• the .nup reading fix;
• the busy check: it needs 4 GB of memory free, or a quarter of the PC's memory if that is less (it was 8 GB);
• the tester's status on the Testers page;
• upload retries;
• the setup and main-screen fixes.

SHA-256 of the files inside the zips:

c410a36c5337cbd4a63f23af37d05e2df646395722692b81c046414913131cb3  beanstalk-agent-dev-ee5fd24-windows-x64-setup.exe
ffdbf19bb5b27ac7a5f289cf66cbaf07f990100124cc507b7f0a78388540cfcc  beanstalk-agent-dev-ee5fd24-x86_64.AppImage

Test builds of the agent (`dev-ee5fd24`), updated 2026-11-15. They are not signed and not a release. - **Windows x64:** [beanstalk-agent-dev-ee5fd24-windows-x64.zip](https://git.magicbean.dev/attachments/211b6ce8-836d-4104-87d7-b4fd87b5556b) holds `beanstalk-agent-dev-ee5fd24-windows-x64-setup.exe`, a single self-contained file; no .NET install is needed. It is unsigned, so SmartScreen will warn: choose More info › Run anyway. - **Linux x86_64:** [beanstalk-agent-dev-ee5fd24-linux-x86_64.zip](https://git.magicbean.dev/attachments/2f13738b-4e3b-4ca3-88d1-7d50990ec990) holds `beanstalk-agent-dev-ee5fd24-x86_64.AppImage`. Run `chmod +x` on it, then run it. They are built from this branch plus local changes that are not pushed here yet: - VRAM, RAM, GPU and CPU targets; - the smaller data folder, now 100 GB on Linux and 150 GB on Windows; - Run it again for skipped tests; - the .nup reading fix; - the busy check: it needs 4 GB of memory free, or a quarter of the PC's memory if that is less (it was 8 GB); - the tester's status on the Testers page; - upload retries; - the setup and main-screen fixes. SHA-256 of the files inside the zips: ``` c410a36c5337cbd4a63f23af37d05e2df646395722692b81c046414913131cb3 beanstalk-agent-dev-ee5fd24-windows-x64-setup.exe ffdbf19bb5b27ac7a5f289cf66cbaf07f990100124cc507b7f0a78388540cfcc beanstalk-agent-dev-ee5fd24-x86_64.AppImage ```

beanstalk-agent-dev-ee5fd24-windows-x64.zip

43 MiB

beanstalk-agent-dev-ee5fd24-linux-x86_64.zip

37 MiB

HumanBean force-pushed feat/test-farm from 97d91374c5 to 3f841c9cd1 2026-11-15 09:48:27 +00:00 Compare

HumanBean force-pushed feat/test-farm from 3f841c9cd1 to 2fe5b86527 2026-11-15 09:48:44 +00:00 Compare

HumanBean force-pushed feat/test-farm from 2fe5b86527 to fa856b7233 2026-11-15 09:59:41 +00:00 Compare

HumanBean changed title from WIP: feat: test farm coordinator, dashboard and desktop agent to feat: test farm coordinator, dashboard and desktop agent 2026-11-15 17:17:59 +00:00

HumanBean merged commit 447d6c256f into main 2026-11-15 17:18:04 +00:00

HumanBean referenced this pull request from a commit 2026-11-15 17:18:05 +00:00

Merge pull request 'feat: test farm coordinator, dashboard and desktop agent' (#3) from HumanBean/Beanstalk:feat/test-farm into main

HumanBean referenced this pull request 2026-11-15 17:33:24 +00:00

fix(agent): isolate gameplay saves, with Windows gameplay opt-in #7

jabbaxipil referenced this pull request 2026-11-15 17:48:32 +00:00

agent: loading a save gives up after one 30 s press while the game is still building its pipelines #10

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

risk

high

Broad or core change: the renderer, defaults on many systems, saves, installs, security

  

risk

low

Small, contained change: little can break

  

risk

medium

Changes behaviour in one area, or a default on some systems

  

ci: approved

No labels

risk

high

risk

low

risk

medium

ci: approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Beanstalk!3

No description provided.