Idea: targeted gameplay scenarios from Longshot's test player, run on the farm #82

Open
opened 2026-11-16 22:40:49 +00:00 by mooki · 2 comments
Member

Why

Of the 245 Longshot issues labelled kind/bug, 153 of the 168 with a platform label are Windows and 84 of the 166 with a GPU label are AMD or Intel. Of the 36 whose title names a crash, freeze, hang, black screen, device loss or driver timeout, 20 of the 24 with a GPU label are AMD or Intel.

Our automated test player (Longshot #1198) can place Link and play a fixed list of actions, but only on one Linux PC with an RTX 4090. The farm has the other hardware: 27 registered testers, 17 on Windows, 10 on AMD, 1 on Intel (no working sandbox yet).

What a scenario is

A place, a fixed list of actions and what to watch.

  • Place: the game's debug teleport and age switch.
  • Actions: fight until it is over, open and close the map, cross a loading zone. A scripted driver or a recorded input playback plays them: no model, no key.
  • Watch: crash, hang, stall, and visual faults (a black or frozen picture, or one that flickers in a burst of eight pictures).
  • Result: a few numbers and flags (crashed, hang, longest stall, black, frozen and flickering pictures, the step it ended on). No model text.

How it could fit Beanstalk

Your [[scenario]] is already this unit. Plans can teleport, switch age, press, run a button script and expect, and the crash probe is always on (test-plan.md). Two ways to ship the rest:

  1. Built into the agent. Fits "Plans are data" (README), but every scenario is an agent release and game details are copied into Beanstalk, as destinations.py already warns.
  2. A file in Longshot, named by the plan. The coordinator already lists assets/saves/ on main with each README's first paragraph for the planner and checks the names (gather_saves, ai_checks). A scenario would be a [[scenario]] fragment in assets/scenarios/<name>/, reviewed like a save.

I think 2 fits better, on two conditions. Read it from main, not the PR's commit, or the author writes the test for their own change (saves already come from main; .farm/ needs two approvers). And let the coordinator expand the fragment into the plan before signing, so the ticket carries the exact steps (architecture.md, "Tickets") and the agent reads no repository file for steps. What the agent cannot do yet stays a built-in step (the Spec's script probe is not built), offered only to agents with the new plan_schema (protocol.md). Four small ones, no game logic:

  • repeat with an until on a status field (the battle music stopped), with a timeout.
  • measure around steps. Today it is a fixed window that runs no steps, so frame times around a fight cannot be taken.
  • A visual probe on the pictures: black, frozen, alternating.
  • A position check in expect. Recordings drift (Spec, review note 2), so playback should stop where it left the route.

What the game side needs

Already in main's control bridge (control.md): teleport (31 destinations), age, status (map, position, life, prompt, music), press, stick, sequence, waits, queued pictures. Missing: a time-of-day command (today only a launch-time input file, which plans may not set), enemy spawning (in progress elsewhere, see this map), a coordinate teleport. Items and flags need guest memory writes, which plans should keep refusing; the first set does not need them.

Privacy

No game file leaves any PC. A scenario holds teleport names, button scripts and settings, and starts from a new game or a repository save (today kokiri-start), never a tester's own. Pictures stay local unless a plan says upload. Our maps and routes stay private.

A first small set

  • #809 (Windows, AMD iGPU): crash after opening and closing the map in Hyrule Field. Teleport there, open and close the map ten times. Needs nothing new.
  • #817 (Windows, RX 6600): crashes at one spot on Death Mountain. Teleport to the trail and walk it from a recorded input.
  • #1215 (Windows, RTX 3070): slow Hyrule Field, corrupt Castle Town. Field then Market by teleports, with frame time, VRAM and a picture at each stop. Your shot against main catches corruption main lacks; our detectors would not.
  • #1213 (Linux, RTX 5080): frame spikes when enemies die in Hyrule Field. The stretch case: needs until, measure around steps, time of day and a repository save where Link has a weapon.

Each scenario is two game starts at about 5 minutes (test-plan.md, "Time"). Four with stand-in steps come to 52 of the 60 minutes a plan may take, at runs = 1.

Questions

  1. Is assets/scenarios/ in Longshot, reviewed like saves, acceptable, or would you keep scenarios in Beanstalk?
  2. Button scripts and a few positions are not game files, but a route is nearer the line than a teleport name. Is that fine under the no-game-files rule?
  3. A scenario that reproduces an open bug crashes main too; only a PR's crash fails a report today. Should these run on every renderer PR, or only when asked?
  4. Fights need more repository saves (a sword, later story states). Which are worth two maintainers' review?
  5. Does start = "new" run on testers' PCs? The docs say it was written without running the game.

Read from your docs and code at commit a89e366; nothing here has run on the farm.

## Why Of the 245 Longshot issues labelled `kind/bug`, 153 of the 168 with a platform label are Windows and 84 of the 166 with a GPU label are AMD or Intel. Of the 36 whose title names a crash, freeze, hang, black screen, device loss or driver timeout, 20 of the 24 with a GPU label are AMD or Intel. Our automated test player ([Longshot #1198](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1198)) can place Link and play a fixed list of actions, but only on one Linux PC with an RTX 4090. The farm has the other hardware: [27 registered testers](https://beanstalk.magicbean.dev/agents), 17 on Windows, 10 on AMD, 1 on Intel (no working sandbox yet). ## What a scenario is A place, a fixed list of actions and what to watch. - **Place:** the game's debug teleport and age switch. - **Actions:** fight until it is over, open and close the map, cross a loading zone. A scripted driver or a recorded input playback plays them: no model, no key. - **Watch:** crash, hang, stall, and visual faults (a black or frozen picture, or one that flickers in a burst of eight pictures). - **Result:** a few numbers and flags (crashed, hang, longest stall, black, frozen and flickering pictures, the step it ended on). No model text. ## How it could fit Beanstalk Your `[[scenario]]` is already this unit. Plans can teleport, switch age, press, run a button script and `expect`, and the `crash` probe is always on ([test-plan.md](https://git.magicbean.dev/MagicBeanDev/Beanstalk/src/branch/main/docs/test-plan.md)). Two ways to ship the rest: 1. **Built into the agent.** Fits "Plans are data" (README), but every scenario is an agent release and game details are copied into Beanstalk, as `destinations.py` already warns. 2. **A file in Longshot, named by the plan.** The coordinator already lists `assets/saves/` on `main` with each README's first paragraph for the planner and checks the names (`gather_saves`, `ai_checks`). A scenario would be a `[[scenario]]` fragment in `assets/scenarios/<name>/`, reviewed like a save. I think 2 fits better, on two conditions. Read it from `main`, not the PR's commit, or the author writes the test for their own change (saves already come from `main`; `.farm/` needs two approvers). And let the coordinator expand the fragment into the plan before signing, so the ticket carries the exact steps ([architecture.md](https://git.magicbean.dev/MagicBeanDev/Beanstalk/src/branch/main/docs/architecture.md), "Tickets") and the agent reads no repository file for steps. What the agent cannot do yet stays a built-in step (the Spec's `script` probe is not built), offered only to agents with the new `plan_schema` ([protocol.md](https://git.magicbean.dev/MagicBeanDev/Beanstalk/src/branch/main/docs/protocol.md)). Four small ones, no game logic: - `repeat` with an `until` on a status field (the battle music stopped), with a timeout. - `measure` around steps. Today it is a fixed window that runs no steps, so frame times around a fight cannot be taken. - A `visual` probe on the pictures: black, frozen, alternating. - A position check in `expect`. Recordings drift ([Spec](https://git.magicbean.dev/MagicBeanDev/Beanstalk/wiki/Spec), review note 2), so playback should stop where it left the route. ## What the game side needs Already in `main`'s control bridge ([control.md](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/docs/control.md)): teleport (31 destinations), age, status (map, position, life, prompt, music), press, stick, sequence, waits, queued pictures. Missing: a time-of-day command (today only a launch-time input file, which plans may not set), enemy spawning (in progress elsewhere, see [this map](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1198#issuecomment-37391)), a coordinate teleport. Items and flags need guest memory writes, which plans should keep refusing; the first set does not need them. ## Privacy No game file leaves any PC. A scenario holds teleport names, button scripts and settings, and starts from a new game or a repository save (today `kokiri-start`), never a tester's own. Pictures stay local unless a plan says `upload`. Our maps and routes stay private. ## A first small set - [#809](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/809) (Windows, AMD iGPU): crash after opening and closing the map in Hyrule Field. Teleport there, open and close the map ten times. Needs nothing new. - [#817](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/817) (Windows, RX 6600): crashes at one spot on Death Mountain. Teleport to the trail and walk it from a recorded input. - [#1215](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1215) (Windows, RTX 3070): slow Hyrule Field, corrupt Castle Town. Field then Market by teleports, with frame time, VRAM and a picture at each stop. Your `shot` against `main` catches corruption `main` lacks; our detectors would not. - [#1213](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1213) (Linux, RTX 5080): frame spikes when enemies die in Hyrule Field. The stretch case: needs `until`, `measure` around steps, time of day and a repository save where Link has a weapon. Each scenario is two game starts at about 5 minutes (test-plan.md, "Time"). Four with stand-in steps come to 52 of the 60 minutes a plan may take, at `runs = 1`. ## Questions 1. Is `assets/scenarios/` in Longshot, reviewed like saves, acceptable, or would you keep scenarios in Beanstalk? 2. Button scripts and a few positions are not game files, but a route is nearer the line than a teleport name. Is that fine under the no-game-files rule? 3. A scenario that reproduces an open bug crashes `main` too; only a PR's crash fails a report today. Should these run on every renderer PR, or only when asked? 4. Fights need more repository saves (a sword, later story states). Which are worth two maintainers' review? 5. Does `start = "new"` run on testers' PCs? The docs say it was written without running the game. Read from your docs and code at commit `a89e366`; nothing here has run on the farm.
Collaborator

Possibly related, for context:

  • #1 Spec: field notes from agent-led testing — Discusses test scenarios, saves, and limitations of declarative farm probes.
  • #2 Spec: commit pins for test requests exist as Longshot pull request 71; two risks in the planned /beanstalk flow — Related farm-plan security, commit pinning and coordinator workflow concerns.
  • MagicBeanDev/Longshot#43 Idea: opt-in hardware tests on self-hosted Forgejo runners — Related proposal for opt-in hardware tests running scripted scenes.
Possibly related, for context: - #1 Spec: field notes from agent-led testing — Discusses test scenarios, saves, and limitations of declarative farm probes. - #2 Spec: commit pins for test requests exist as Longshot pull request 71; two risks in the planned /beanstalk flow — Related farm-plan security, commit pinning and coordinator workflow concerns. - MagicBeanDev/Longshot#43 Idea: opt-in hardware tests on self-hosted Forgejo runners — Related proposal for opt-in hardware tests running scripted scenes.
Author
Member

Co-op scenes, as a follow-up to the first set above. Nothing here has run on the farm.

Why co-op needs the farm. Longshot's multiplayer (MagicBeanDev/Longshot#68) adds a thread and a UDP socket, a menu tab and extra world actors for the other players. It has run on one Linux PC with an RTX 4090 and one Windows laptop with Intel Arc. Two unlike machines (say Windows with AMD and Linux with NVIDIA) are exactly what has never run, and exactly what the farm has.

What blocks it today (README, docs/architecture.md and docs/test-plan.md):

  1. The game runs with no network (bubblewrap with every namespace unshared; an AppContainer with no capabilities; the runner account behind a firewall rule), so two games on two testers cannot talk.
  2. A job is one tester. A plan cannot say "two testers, one room": a scenario key role is refused as unknown.
  3. The baseline is main, which has no co-op. A rule against the baseline means "what co-op costs"; whether a pair worked must be an absolute rule (.max, log_forbidden_hits.max = 0, crash = false).

A way in, smallest first.

  • A. One machine, co-op alone. Fits the schema as it is. The game hosts the relay itself on loopback (HOS_COOP_HOST), makes a room at start (HOS_COOP_AUTO=create) and plays alone in it. It checks that the co-op thread starts and ends without a crash or a hang on every GPU and driver, and what co-op costs a frame, with the three switches set on the PR side only ([matrix.pr_only]). It needs loopback to be up inside the sandbox on Linux; whether it is on Windows we do not know.
  • B. Two games on one tester, both in the one sandbox, on loopback. Needs the agent to start a second game (its own control socket and --user folder) under one lease. No network rule changes.
  • C. Two testers. Needs a relay run by the farm, UDP from the sandbox to that one address, and two jobs of one commit that start together with the same room secret in both signed tickets (the coordinator makes it; no new call from the agent, which keeps "agents only call out"). The switches already pass the env rules: HOS_COOP, HOS_COOP_AUTO, HOS_COOP_ROOM, HOS_COOP_SERVER, HOS_COOP_ACCEPT (a test-only shortcut for consent that works only from the environment).

The scenes, and what each machine checks. A is the maker, B the joiner. Every check is a line the game already writes (the real lines are quoted in log_match regexes), a probe the farm already has, or the bridge's status. None needs a model.

Scene A (maker) checks B (joiner) checks Level
coop-alone created (member 1) logged; no relay lost, no room: or socket error; no crash or hang; p99 frame time against main (cost of co-op on) n/a A (the plan below)
pair-join room made; sync: room accepted; the other player's body logged as made (puppet: member 2 is shown as CoopPuppet_) within 60 s of B's join; no relay lost joined room ... as member 2; first answer from the relay (the relay answered in N ms) logged; body of member 1 logged as made; no uses an older co-op version B or C
pair-walk both walk a scripted loop in Hyrule Field for 60 s; frametime and hitch_count; the once-a-minute coop: stats: line shows events given up 0 and outbox full 0 the same B or C
pair-map-change A teleports to Kakariko while B stays: B's body logged hidden within 10 s; B then follows: shown again within 30 s; load_time for each teleport the mirror: A's body hidden, then shown B or C
pair-rejoin B leaves and joins again (the bridge's relaunch): A logs the leave and the join, accepts again, the body returns within 60 s, events given up unchanged joined room twice; accepted again B or C
pair-loss HOS_COOP_DROP=30 (the game's own receive-side loss): bodies still logged as shown; reliable retries above 0 and events given up 0 the same B or C

What the farm cannot judge, and a person must (as docs/test-plan.md asks us to say): whether the other body moves smoothly, whether a name or arrow reads well, and chat (its text is never written to the log). Go to friend is left out until there is a step that presses the tab's row; nothing in the bridge does.

Pairing hardware. A pair target would name two tag sets, for example windows+amd with linux+nvidia, and windows+intel with windows+nvidia: the Windows pairs also check that the game's UDP socket works at all under the sandbox.

Questions. (1) Would the maintainers consider one farm relay and a UDP allowance for the sandbox limited to it, or should co-op stay on one tester (levels A and B)? (2) Is loopback up inside the Linux sandbox and the two Windows modes? (3) Is a second game process on one tester acceptable for memory on the smallest GPUs?

Level A as a plan (parses with plan.parse_plan at 4ec688b, estimate 23 minutes of the 60 allowed):

schema = 3
summary = "Co-op on, alone in a room the game hosts itself: no crash, no hang, and the frame-time cost of the co-op thread"

[[target]]
tags = []

[baseline]
ref = "main"

[matrix.pr_only]
HOS_COOP = ["1"]
HOS_COOP_HOST = ["1"]
HOS_COOP_AUTO = ["create"]

[[scenario]]
name = "coop-alone-hosted"
start = "save:kokiri-start"
runs = 2
step = [
  { do = "wait", event = "in_play", timeout_s = 120 },
  { do = "expect", log = "coop: room .* created \\(member 1\\)" },
  { do = "teleport", to = "HyruleField_Child" },
  { do = "wait", event = "map", map = "HyruleField_Child", timeout_s = 60 },
  { do = "wait", event = "in_play", timeout_s = 60 },
  { do = "measure", name = "field", duration_s = 30, warmup_s = 10 },
  { do = "expect", state = "play" },
]

[[probe]]
kind = "frametime"

[[probe]]
kind = "log_match"
forbidden = ["relay lost", "no room:", "cannot open a UDP socket", "cannot start the co-op thread"]

[pass]
crash = false
log_forbidden_hits.max = 0
frametime_p99_ms.max_delta_pct = 10

Not tried: that HOS_COOP_AUTO=create hosts the relay when HOS_COOP_HOST is set (read from the game's code, not run), that loopback works in the sandbox, and whether the 10 percent rule is the right size for a thread that sleeps.

Co-op scenes, as a follow-up to the first set above. Nothing here has run on the farm. **Why co-op needs the farm.** Longshot's multiplayer (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/68) adds a thread and a UDP socket, a menu tab and extra world actors for the other players. It has run on one Linux PC with an RTX 4090 and one Windows laptop with Intel Arc. Two unlike machines (say Windows with AMD and Linux with NVIDIA) are exactly what has never run, and exactly what the farm has. **What blocks it today** (README, `docs/architecture.md` and `docs/test-plan.md`): 1. The game runs with no network (bubblewrap with every namespace unshared; an AppContainer with no capabilities; the runner account behind a firewall rule), so two games on two testers cannot talk. 2. A job is one tester. A plan cannot say "two testers, one room": a scenario key `role` is refused as unknown. 3. The baseline is `main`, which has no co-op. A rule against the baseline means "what co-op costs"; whether a pair worked must be an absolute rule (`.max`, `log_forbidden_hits.max = 0`, `crash = false`). **A way in, smallest first.** * **A. One machine, co-op alone.** Fits the schema as it is. The game hosts the relay itself on loopback (`HOS_COOP_HOST`), makes a room at start (`HOS_COOP_AUTO=create`) and plays alone in it. It checks that the co-op thread starts and ends without a crash or a hang on every GPU and driver, and what co-op costs a frame, with the three switches set on the PR side only (`[matrix.pr_only]`). It needs loopback to be up inside the sandbox on Linux; whether it is on Windows we do not know. * **B. Two games on one tester**, both in the one sandbox, on loopback. Needs the agent to start a second game (its own control socket and `--user` folder) under one lease. No network rule changes. * **C. Two testers.** Needs a relay run by the farm, UDP from the sandbox to that one address, and two jobs of one commit that start together with the same room secret in both signed tickets (the coordinator makes it; no new call from the agent, which keeps "agents only call out"). The switches already pass the env rules: `HOS_COOP`, `HOS_COOP_AUTO`, `HOS_COOP_ROOM`, `HOS_COOP_SERVER`, `HOS_COOP_ACCEPT` (a test-only shortcut for consent that works only from the environment). **The scenes, and what each machine checks.** A is the maker, B the joiner. Every check is a line the game already writes (the real lines are quoted in `log_match` regexes), a probe the farm already has, or the bridge's `status`. None needs a model. | Scene | A (maker) checks | B (joiner) checks | Level | |---|---|---|---| | `coop-alone` | `created (member 1)` logged; no `relay lost`, `no room:` or socket error; no crash or hang; p99 frame time against `main` (cost of co-op on) | n/a | A (the plan below) | | `pair-join` | room made; `sync: room accepted`; the other player's body logged as made (`puppet: member 2 is shown as CoopPuppet_`) within 60 s of B's join; no `relay lost` | `joined room ... as member 2`; first answer from the relay (`the relay answered in N ms`) logged; body of member 1 logged as made; no `uses an older co-op version` | B or C | | `pair-walk` | both walk a scripted loop in Hyrule Field for 60 s; `frametime` and `hitch_count`; the once-a-minute `coop: stats:` line shows `events given up 0` and `outbox full 0` | the same | B or C | | `pair-map-change` | A teleports to Kakariko while B stays: B's body logged hidden within 10 s; B then follows: shown again within 30 s; `load_time` for each teleport | the mirror: A's body hidden, then shown | B or C | | `pair-rejoin` | B leaves and joins again (the bridge's `relaunch`): A logs the leave and the join, accepts again, the body returns within 60 s, `events given up` unchanged | `joined room` twice; accepted again | B or C | | `pair-loss` | `HOS_COOP_DROP=30` (the game's own receive-side loss): bodies still logged as shown; `reliable retries` above 0 and `events given up 0` | the same | B or C | **What the farm cannot judge**, and a person must (as `docs/test-plan.md` asks us to say): whether the other body moves smoothly, whether a name or arrow reads well, and chat (its text is never written to the log). Go to friend is left out until there is a step that presses the tab's row; nothing in the bridge does. **Pairing hardware.** A pair target would name two tag sets, for example `windows+amd` with `linux+nvidia`, and `windows+intel` with `windows+nvidia`: the Windows pairs also check that the game's UDP socket works at all under the sandbox. **Questions.** (1) Would the maintainers consider one farm relay and a UDP allowance for the sandbox limited to it, or should co-op stay on one tester (levels A and B)? (2) Is loopback up inside the Linux sandbox and the two Windows modes? (3) Is a second game process on one tester acceptable for memory on the smallest GPUs? Level A as a plan (parses with `plan.parse_plan` at `4ec688b`, estimate 23 minutes of the 60 allowed): ```toml schema = 3 summary = "Co-op on, alone in a room the game hosts itself: no crash, no hang, and the frame-time cost of the co-op thread" [[target]] tags = [] [baseline] ref = "main" [matrix.pr_only] HOS_COOP = ["1"] HOS_COOP_HOST = ["1"] HOS_COOP_AUTO = ["create"] [[scenario]] name = "coop-alone-hosted" start = "save:kokiri-start" runs = 2 step = [ { do = "wait", event = "in_play", timeout_s = 120 }, { do = "expect", log = "coop: room .* created \\(member 1\\)" }, { do = "teleport", to = "HyruleField_Child" }, { do = "wait", event = "map", map = "HyruleField_Child", timeout_s = 60 }, { do = "wait", event = "in_play", timeout_s = 60 }, { do = "measure", name = "field", duration_s = 30, warmup_s = 10 }, { do = "expect", state = "play" }, ] [[probe]] kind = "frametime" [[probe]] kind = "log_match" forbidden = ["relay lost", "no room:", "cannot open a UDP socket", "cannot start the co-op thread"] [pass] crash = false log_forbidden_hits.max = 0 frametime_p99_ms.max_delta_pct = 10 ``` Not tried: that `HOS_COOP_AUTO=create` hosts the relay when `HOS_COOP_HOST` is set (read from the game's code, not run), that loopback works in the sandbox, and whether the 10 percent rule is the right size for a thread that sleeps.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk#82
No description provided.