MagicBeanDev/Beanstalk
Watch
12
0
Fork
You've already forked Beanstalk
 1
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity Actions

Spec: field notes from agent-led testing #1

New issue
Open
opened 2026-11-14 01:14:19 +00:00 by tonesi · 0 comments
tonesi commented 2026-11-14 01:14:19 +00:00

The Spec is sound. Running agent-led tests over the last few days turned up seven things it should plan for. Those runs were the Linux tests posted on PR 95 and PR 162 (T-GTM9), and earlier on GitHub PR 2871.

These notes add to the spec's Review notes (Nov 13), which check the design against the code. These check it against practice. All tests ran on one PC: RTX 3070 Ti 8 GB (it also drives the desktop), i7-12700K, 16 GB RAM, Linux. Some runs were attended; some ran overnight with nobody at the PC. Queue counts are from test_requests.py on 2026-11-13.

Most of Phase 0–1 already runs

Today's tools already do most of what the spec asks of Phase 0 and Phase 1. That answers review note 8: the farm can grow out of the test-request system rather than replace it.

Spec part What does it today What it lacks against the spec
Coordinator: queue, leases, approval gate, results on the PR scripts/test_requests.py over Matrix and Discord Approval per PR, not per SHA; needs are free text; requests stay open after a merge
Tester procedure The /testing skill in Longshot Written for one attended test at a time
Farm agent: local limits, headless runs, evidence Mooki's hookshot-workbench harness: gates for one game, one build and one checks run at a time, a GPU guard, a local save catalogue, windowless runs No sandbox; Linux only; per-machine setup
Unattended agent An overnight loop built on /testing and the harness: a worker builds, runs and drafts; the main session checks the draft before it posts Picks only what is answerable from logs and pictures

What the field showed

1. Only 16 of 125 open requests name what they need

Of 125 open requests, 118 are on a Longshot PR, 42 of those are approved, 32 of those are on a PR still open, and 16 of those name the hardware they need. The other 109 can't be claimed, are on a merged PR, or need the PR read to know who can test them.

  • 76 (61%) wait for the testing wanted label. In practice the label is the only approval, since few authors have push rights.
  • 34 are on PRs already merged. Only requests the label made close by themselves.
  • 7 are from the GitHub era and can never be marked done.
  • Free-text needs can't be matched by a program. On the first overnight run, 2 PRs out of about 40 requests fit this PC.

The spec's [[target]] fields and "a new push cancels queued jobs" fix real problems, not hypothetical ones.

2. Approval doesn't name a commit

test_requests.py checks the label or the author's push rights when someone claims. It records no SHA, neither then nor with the result. So the bait-and-switch in the threat model is open today. Our loop only avoids it by checking the PR head before posting. This is the largest security gap between today's system and the spec.

3. Many tests can't be probes

Of the live requests, many need a person or hardware few have: a real ultrawide or 4:3 screen, a gamepad's feel, HDR colour, the Windows crash window, a save next to an NPC. The spec assumes every test fits a declarative probe. A queue with two job kinds, unattended (probes) and attended (a person, or an AI agent with a person), would cover both. The spec doesn't mention AI agents, yet most testing today goes through them.

4. Saves were the main blocker

Review note 3 is right that scenarios can't ship saves. In practice, a missing save is what stopped tests: one overnight candidate was skipped over "a save nobody had". The debug teleport helps for places, not for game state. A shared catalogue of place names (kokiri-village-day, hyrule-field-sign) stays within the rule: names travel, each tester keeps their own saves under those names. The harness already keeps a local catalogue like this.

5. Same machine and same session isn't enough for timing

On PR 162, Hyrule Field runs on this PC fell into two GPU states, about 11 and 15.5 ms of GPU time a frame, with the same draws and VRAM. Both showed in every arm, cause unknown. A rule like frametime_p99_ms.max_delta_pct = 10 would have passed or failed that PR at random. Other traps from the same runs:

  • The Auto (this PC) setting rewrites the options file on a copy's first play run. Freeze it and checksum it, or the arms run different settings.
  • HOS_INPUT_FRAME_CLOCK=1 breaks timed menu presses at HOS_FPS_MAX=240: the script runs about three times too fast.
  • Ubuntu's validation layer 1.3.275 crashes on the recorder thread where SDK 1.4.341 on Windows doesn't. The tester's tool versions change the result.
  • Headless fps comes from submission intervals, not display timing.

6. Limits must not break the volunteer's own PC

A game run takes about 10 GB of this PC's RAM and the card that drives the desktop; a build takes every core. Gates that allow one game, one build and one checks run at a time, plus a wait for free memory, worked well. But the harness's GPU guard, which kills programs that open the card outside a gate, silently killed every start of the owner's own launcher, since AppImages run from /tmp. It took a patch to fix. run_when = "idle" and the sandbox need the same care: they have to fail visibly and leave the owner's own programs alone.

7. Agent-written reports need a check before they post

In our overnight loop, a worker agent builds, runs and drafts, and the main session checks each draft before anything is posted. The check asks four things:

  • Is the PR head still the commit that was built?
  • Do the pictures show what the words say?
  • Is anything local or personal in the text?
  • Is a single run labelled as not a measurement?

This supports the spec's structured JSON report: a table built from probe output can't overstate. Free-text notes can ride on top.

Suggested changes to the spec

Each change below is small. Together they let today's agent testers and the future farm agent work from the same plan and the same queue.

Spec section Suggested change From finding
Rollout Name test_requests.py as the seed of the coordinator, and Phase 0 as the existing harness plus a plan format Mapping
Trust and approval Pin approval to a SHA in the existing system first: record the head at label time, at claim and with each result 2
Per-PR test plan Add kind = "attended" jobs with a needs_person line (what a person must look at or do), beside probe jobs 3
Per-PR test plan Scenarios start from a named place in a shared catalogue; each tester supplies their own save for it, or the agent uses the debug teleport 4
Per-PR test plan, pass rules "Inconclusive" as a result of its own; a minimum number of alternated runs per arm; settings frozen and checksummed before timing 5
Farm agent, capabilities Report GPU clock state during runs, the Vulkan validation layer or SDK version, and whether the run was headless 5
Farm agent, local policy The agent's limits fail visibly and never touch programs it didn't start 6
Reports and PR feedback The coordinator posts the probe table; tester notes are optional, under it 7
Scheduling rules Close or hide a request when its PR merges or closes, however the request was made 1

The capability fields add to what the agent uploads, so they should stay at model and version level, as the threat model's privacy row asks.

CircleCI and the desktop app

No cloud CI can run the game, CircleCI included: that would mean uploading a dump, which the rule with no exceptions forbids. So cloud CI fits the game-free half only.

  • Cloud CI: the builder is already prebuilt and signed (review note 1), so building it per SHA saves little. The cheap win is running tests/run_all.py on every PR. Today it is workflow_dispatch only, and every tester runs it by hand. The project's own Forgejo runners may be enough for that.
  • The desktop app: Beanstock's README already names agent/ as the desktop app volunteers run, and its lock-file rules point to .NET, the launcher's stack. The launcher already has signed releases (Core/ReleaseTrust.cs) and a Vulkan hardware probe (Core/SystemRequirements.cs), so the agent can share them, or be a mode of the launcher. Players already run it, so it is the shortest path to new volunteers.

What we can do next

We can measure two of the spec's open questions on this PC, and draft the first steps toward Phase 1, if the architects want them.

  1. A warm (incremental) build time, which the spec lists as still open. Our PR builds so far were full source builds: 18 min 37 s to 25 min 10 s. We would time scripts/dev_packs.py (a warm main base plus the PR on top) on the same PRs.
  2. Headless against windowed frame times on the same scene, alternating runs, to answer "Can the runtime render fully offscreen without skewing frame times?"
  3. test_requests.py changes, each a small Longshot PR: record the PR head SHA at claim and with each result, and show when a PR has moved since it was tested; close or hide requests on merged or closed PRs. Only if nobody else has claimed this, and only if the architects want the existing system to be the seed.

🤖 Written with Claude Code from the notes of these test runs.

The [Spec](https://git.magicbean.dev/MagicBeanDev/Beanstock/wiki/Spec) is sound. Running agent-led tests over the last few days turned up seven things it should plan for. Those runs were the Linux tests posted on [PR 95](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/95#issuecomment-13030) and [PR 162](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/162) (T-GTM9), and earlier on [GitHub PR 2871](https://github.com/NotRexed/Hookshot/pull/2871). These notes add to the spec's Review notes (Nov 13), which check the design against the code. These check it against practice. All tests ran on one PC: RTX 3070 Ti 8 GB (it also drives the desktop), i7-12700K, 16 GB RAM, Linux. Some runs were attended; some ran overnight with nobody at the PC. Queue counts are from `test_requests.py` on 2026-11-13. ## Most of Phase 0–1 already runs Today's tools already do most of what the spec asks of Phase 0 and Phase 1. That answers review note 8: the farm can grow out of the test-request system rather than replace it. | Spec part | What does it today | What it lacks against the spec | | --- | --- | --- | | Coordinator: queue, leases, approval gate, results on the PR | `scripts/test_requests.py` over Matrix and Discord | Approval per PR, not per SHA; needs are free text; requests stay open after a merge | | Tester procedure | The `/testing` skill in Longshot | Written for one attended test at a time | | Farm agent: local limits, headless runs, evidence | Mooki's hookshot-workbench harness: gates for one game, one build and one checks run at a time, a GPU guard, a local save catalogue, windowless runs | No sandbox; Linux only; per-machine setup | | Unattended agent | An overnight loop built on `/testing` and the harness: a worker builds, runs and drafts; the main session checks the draft before it posts | Picks only what is answerable from logs and pictures | ## What the field showed ### 1. Only 16 of 125 open requests name what they need Of 125 open requests, 118 are on a Longshot PR, 42 of those are approved, 32 of those are on a PR still open, and 16 of those name the hardware they need. The other 109 can't be claimed, are on a merged PR, or need the PR read to know who can test them. - 76 (61%) wait for the `testing wanted` label. In practice the label is the only approval, since few authors have push rights. - 34 are on PRs already merged. Only requests the label made close by themselves. - 7 are from the GitHub era and can never be marked done. - Free-text needs can't be matched by a program. On the first overnight run, 2 PRs out of about 40 requests fit this PC. The spec's `[[target]]` fields and "a new push cancels queued jobs" fix real problems, not hypothetical ones. ### 2. Approval doesn't name a commit `test_requests.py` checks the label or the author's push rights when someone claims. It records no SHA, neither then nor with the result. So the bait-and-switch in the threat model is open today. Our loop only avoids it by checking the PR head before posting. This is the largest security gap between today's system and the spec. ### 3. Many tests can't be probes Of the live requests, many need a person or hardware few have: a real ultrawide or 4:3 screen, a gamepad's feel, HDR colour, the Windows crash window, a save next to an NPC. The spec assumes every test fits a declarative probe. A queue with two job kinds, unattended (probes) and attended (a person, or an AI agent with a person), would cover both. The spec doesn't mention AI agents, yet most testing today goes through them. ### 4. Saves were the main blocker Review note 3 is right that scenarios can't ship saves. In practice, a missing save is what stopped tests: one overnight candidate was skipped over "a save nobody had". The debug teleport helps for places, not for game state. A shared catalogue of place names (`kokiri-village-day`, `hyrule-field-sign`) stays within the rule: names travel, each tester keeps their own saves under those names. The harness already keeps a local catalogue like this. ### 5. Same machine and same session isn't enough for timing On PR 162, Hyrule Field runs on this PC fell into two GPU states, about 11 and 15.5 ms of GPU time a frame, with the same draws and VRAM. Both showed in every arm, cause unknown. A rule like `frametime_p99_ms.max_delta_pct = 10` would have passed or failed that PR at random. Other traps from the same runs: - The Auto (this PC) setting rewrites the options file on a copy's first play run. Freeze it and checksum it, or the arms run different settings. - `HOS_INPUT_FRAME_CLOCK=1` breaks timed menu presses at `HOS_FPS_MAX=240`: the script runs about three times too fast. - Ubuntu's validation layer 1.3.275 crashes on the recorder thread where SDK 1.4.341 on Windows doesn't. The tester's tool versions change the result. - Headless fps comes from submission intervals, not display timing. ### 6. Limits must not break the volunteer's own PC A game run takes about 10 GB of this PC's RAM and the card that drives the desktop; a build takes every core. Gates that allow one game, one build and one checks run at a time, plus a wait for free memory, worked well. But the harness's GPU guard, which kills programs that open the card outside a gate, silently killed every start of the owner's own launcher, since AppImages run from `/tmp`. It took a patch to fix. `run_when = "idle"` and the sandbox need the same care: they have to fail visibly and leave the owner's own programs alone. ### 7. Agent-written reports need a check before they post In our overnight loop, a worker agent builds, runs and drafts, and the main session checks each draft before anything is posted. The check asks four things: - Is the PR head still the commit that was built? - Do the pictures show what the words say? - Is anything local or personal in the text? - Is a single run labelled as not a measurement? This supports the spec's structured JSON report: a table built from probe output can't overstate. Free-text notes can ride on top. ## Suggested changes to the spec Each change below is small. Together they let today's agent testers and the future farm agent work from the same plan and the same queue. | Spec section | Suggested change | From finding | | --- | --- | --- | | Rollout | Name `test_requests.py` as the seed of the coordinator, and Phase 0 as the existing harness plus a plan format | Mapping | | Trust and approval | Pin approval to a SHA in the existing system first: record the head at label time, at claim and with each result | 2 | | Per-PR test plan | Add `kind = "attended"` jobs with a `needs_person` line (what a person must look at or do), beside probe jobs | 3 | | Per-PR test plan | Scenarios start from a named place in a shared catalogue; each tester supplies their own save for it, or the agent uses the debug teleport | 4 | | Per-PR test plan, pass rules | "Inconclusive" as a result of its own; a minimum number of alternated runs per arm; settings frozen and checksummed before timing | 5 | | Farm agent, capabilities | Report GPU clock state during runs, the Vulkan validation layer or SDK version, and whether the run was headless | 5 | | Farm agent, local policy | The agent's limits fail visibly and never touch programs it didn't start | 6 | | Reports and PR feedback | The coordinator posts the probe table; tester notes are optional, under it | 7 | | Scheduling rules | Close or hide a request when its PR merges or closes, however the request was made | 1 | The capability fields add to what the agent uploads, so they should stay at model and version level, as the threat model's privacy row asks. ## CircleCI and the desktop app No cloud CI can run the game, CircleCI included: that would mean uploading a dump, which the rule with no exceptions forbids. So cloud CI fits the game-free half only. - **Cloud CI:** the builder is already prebuilt and signed (review note 1), so building it per SHA saves little. The cheap win is running `tests/run_all.py` on every PR. Today it is `workflow_dispatch` only, and every tester runs it by hand. The project's own Forgejo runners may be enough for that. - **The desktop app:** Beanstock's README already names `agent/` as the desktop app volunteers run, and its lock-file rules point to .NET, the launcher's stack. The launcher already has signed releases (`Core/ReleaseTrust.cs`) and a Vulkan hardware probe (`Core/SystemRequirements.cs`), so the agent can share them, or be a mode of the launcher. Players already run it, so it is the shortest path to new volunteers. ## What we can do next We can measure two of the spec's open questions on this PC, and draft the first steps toward Phase 1, if the architects want them. 1. **A warm (incremental) build time**, which the spec lists as still open. Our PR builds so far were full source builds: 18 min 37 s to 25 min 10 s. We would time `scripts/dev_packs.py` (a warm `main` base plus the PR on top) on the same PRs. 2. **Headless against windowed frame times** on the same scene, alternating runs, to answer "Can the runtime render fully offscreen without skewing frame times?" 3. **`test_requests.py` changes**, each a small Longshot PR: record the PR head SHA at claim and with each result, and show when a PR has moved since it was tested; close or hide requests on merged or closed PRs. Only if nobody else has claimed this, and only if the architects want the existing system to be the seed. 🤖 Written with [Claude Code](https://claude.com/claude-code) from the notes of these test runs.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk#1
No description provided.