Idea: an automated test player that plays sections of the game and hunts for bugs #1198

Open
opened 2026-11-16 17:33:48 +00:00 by mooki · 7 comments
Member

The idea

An automated test player that plays Longshot by itself, so that sections of the game can be played again on every build and bugs are found by playing, not only by tests that never start the game. Long term it becomes a bug hunter: it plays while detectors watch for crashes, hangs, softlocks, falling out of the map, visual glitches and co-op desyncs, and every finding comes with the exact inputs to replay it.

How it works today

  • It drives a normal local build through the game's local control channel (host/control.c): it reads the sampled game state (position, map, age, health, the on-screen prompt) and sends controller input. Nothing made from the game leaves the PC.
  • At each step a small language model picks one action from a short list of candidates that make sense right now (walk to a point, open the chest in front, talk, advance text, fight, play a song, use an item, follow the other player). A second model plans routes. When it is stuck or something looks odd it can send one screenshot to a vision model and gets hints back ("stuck, blocked ahead", "door on the left").
  • Without a model it runs a fixed rules policy, which is the baseline every model run is compared with.
  • Every action counts as done only when the game state proves it: a flag, a count, a map change or a picture. "Pressed the button" is not enough (one of the first lessons: the song was "played" and the windmill never changed).

Results so far (Linux, RTX 4090, builds of #68 and main)

What Result
Two test players in one co-op room, Hyrule Field by day, opposite loops then a race 16.5 minutes, 511 decisions, 0.12 USD; 20 route steps ended stuck at the same few spots
Eight interaction tasks, first try 4 solved: open a chest, take a load zone, talk (a 42 second conversation with a choice), fight with lock-on. Song pressed but no effect; picking up rupees, free play and a house door failed
Why those failed Our own map data: the rupees were under water and on a 2 m pillar, the door was approached 0.5 m off and from the wrong side, and an item was used while Link still held the ocarina. All fixed, being rerun
Second round with fixes (six games) None solved yet, each for a new reason found in the game: the door showed its prompt but the step walked on past it, the chosen rupees cannot be reached, the song's button order is still unconfirmed, and the item button was the wrong one. Fixed or probed; the third round is queued
Short benchmark, door and talk course Plain rules: 4 of 4 goals in 77 s. Model with vision: 3 of 4 in 119 s
Vision on real frames Saw Link stuck against a closed gate (checked against the frame); agreed with the largest model on 11 of 11 frames after one question was reworded

Pictures with the decision overlay (the goal, the choices and the one taken, at a glance) follow in a comment, together with each new round's results.

Cost. About 0.0002 USD per decision, and 0.0003 to 0.025 USD per screenshot look depending on the model. The first 2,600 model requests cost about half a dollar together.

Next

  1. Whole sections: "from a new save, get the Kokiri Sword and the Deku Shield, then clear the Deku Tree to Gohma", "find and open every chest in Kakariko as an adult", "cross Hyrule Field to Lake Hylia". A planning model breaks the aim into steps and re-plans when one fails.
  2. More actions: jumping, climbing, swimming, pushing blocks, shops, signs, the slingshot, boomerang, bow, bombs and hookshot, shield, Epona, warp songs.
  3. Bug hunting: detectors for crashes and aborts, hangs, stalls, softlocks, out of bounds, visual anomalies, new log errors and co-op desync, each finding saved with its build, save, inputs, state trace, frames and logs, plus a replay that re-drives the inputs.
  4. Debug tools from the command line: teleport, age, time of day, items and flags, so a test starts anywhere in seconds instead of from a save.
  5. An overlay with the test player's decisions at a glance.

Anything else

The code is being published in the workbench repository once it is proven in the game; the world data, saves and routes stay local to each PC, because they are made from the game. Questions for the maintainers:

  1. Would a nightly bug hunt on main be useful, and where should its findings go (an issue per finding, one report issue, the team chat)?
  2. A live overlay inside the game would need a small control-channel command that shows a line of text on screen. Is that welcome as a pull request?
  3. Which sections or known bugs should it try first?
### The idea An automated test player that plays Longshot by itself, so that sections of the game can be played again on every build and bugs are found by playing, not only by tests that never start the game. Long term it becomes a bug hunter: it plays while detectors watch for crashes, hangs, softlocks, falling out of the map, visual glitches and co-op desyncs, and every finding comes with the exact inputs to replay it. **How it works today** - It drives a normal local build through the game's local control channel (`host/control.c`): it reads the sampled game state (position, map, age, health, the on-screen prompt) and sends controller input. Nothing made from the game leaves the PC. - At each step a small language model picks one action from a short list of candidates that make sense right now (walk to a point, open the chest in front, talk, advance text, fight, play a song, use an item, follow the other player). A second model plans routes. When it is stuck or something looks odd it can send one screenshot to a vision model and gets hints back ("stuck, blocked ahead", "door on the left"). - Without a model it runs a fixed rules policy, which is the baseline every model run is compared with. - Every action counts as done only when the game state proves it: a flag, a count, a map change or a picture. "Pressed the button" is not enough (one of the first lessons: the song was "played" and the windmill never changed). **Results so far** (Linux, RTX 4090, builds of https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/68 and main) | What | Result | |---|---| | Two test players in one co-op room, Hyrule Field by day, opposite loops then a race | 16.5 minutes, 511 decisions, 0.12 USD; 20 route steps ended stuck at the same few spots | | Eight interaction tasks, first try | 4 solved: open a chest, take a load zone, talk (a 42 second conversation with a choice), fight with lock-on. Song pressed but no effect; picking up rupees, free play and a house door failed | | Why those failed | Our own map data: the rupees were under water and on a 2 m pillar, the door was approached 0.5 m off and from the wrong side, and an item was used while Link still held the ocarina. All fixed, being rerun | | Second round with fixes (six games) | None solved yet, each for a new reason found in the game: the door showed its prompt but the step walked on past it, the chosen rupees cannot be reached, the song's button order is still unconfirmed, and the item button was the wrong one. Fixed or probed; the third round is queued | | Short benchmark, door and talk course | Plain rules: 4 of 4 goals in 77 s. Model with vision: 3 of 4 in 119 s | | Vision on real frames | Saw Link stuck against a closed gate (checked against the frame); agreed with the largest model on 11 of 11 frames after one question was reworded | Pictures with the decision overlay (the goal, the choices and the one taken, at a glance) follow in a comment, together with each new round's results. **Cost.** About 0.0002 USD per decision, and 0.0003 to 0.025 USD per screenshot look depending on the model. The first 2,600 model requests cost about half a dollar together. **Next** 1. Whole sections: "from a new save, get the Kokiri Sword and the Deku Shield, then clear the Deku Tree to Gohma", "find and open every chest in Kakariko as an adult", "cross Hyrule Field to Lake Hylia". A planning model breaks the aim into steps and re-plans when one fails. 2. More actions: jumping, climbing, swimming, pushing blocks, shops, signs, the slingshot, boomerang, bow, bombs and hookshot, shield, Epona, warp songs. 3. Bug hunting: detectors for crashes and aborts, hangs, stalls, softlocks, out of bounds, visual anomalies, new log errors and co-op desync, each finding saved with its build, save, inputs, state trace, frames and logs, plus a replay that re-drives the inputs. 4. Debug tools from the command line: teleport, age, time of day, items and flags, so a test starts anywhere in seconds instead of from a save. 5. An overlay with the test player's decisions at a glance. ### Anything else The code is being published in the workbench repository once it is proven in the game; the world data, saves and routes stay local to each PC, because they are made from the game. Questions for the maintainers: 1. Would a nightly bug hunt on main be useful, and where should its findings go (an issue per finding, one report issue, the team chat)? 2. A live overlay inside the game would need a small control-channel command that shows a line of text on screen. Is that welcome as a pull request? 3. Which sections or known bugs should it try first?
Collaborator

Possibly related, for context:

  • #1113 Idea: Record and replay inputs for automated testing — Input recording and replay is a component of reproducible automated gameplay testing, but this proposal adds an autonomous player and bug-detection system.
  • #43 Idea: opt-in hardware tests on self-hosted Forgejo runners — Related interest in automated testing infrastructure, but that issue concerns opt-in hardware runner tests rather than gameplay automation.
Possibly related, for context: - #1113 Idea: Record and replay inputs for automated testing — Input recording and replay is a component of reproducible automated gameplay testing, but this proposal adds an autonomous player and bug-detection system. - #43 Idea: opt-in hardware tests on self-hosted Forgejo runners — Related interest in automated testing infrastructure, but that issue concerns opt-in hardware runner tests rather than gameplay automation.
Author
Member

Pictures with the decision overlay (16 November)

Each picture is the game's own frame with a panel below it, drawn afterwards from the test player's log (the game's pixels are not covered). How to read the panel: GOAL is the current goal and whether it is met; CANDIDATES lists the actions that made sense at that moment, with the chosen one highlighted (and the model's percentages when a model chose); STEP is what it is doing and its result word; LAST is the previous step's result; MODEL says which model answered, or "rules" when no model was asked; COST is the spend so far; STATE shows moving or recovering; VISION is the latest screenshot look in a few words; AT is the map, position, age, life and rupees.

  1. Opening a chest in the Deku Tree: walk up, press A, Link holds up the Compass, A through the message, goal met by the chest's flag going from 0 to 1. Plain rules, no model call.
    Opening a chest, with the decision panel

  2. A screenshot look while stuck. At 11.6 s the camera does not show Link and the model sees open ground (stuck 3%); at 30.9 s it reports a wall ahead (stuck 94%) with Link at the closed gate, which is right.
    Two screenshot looks while stuck

  3. A conversation with a Kokiri, including a two-choice question where the default answer is taken.
    A conversation

  4. A fight: lock-on (the triangles), three slashes, the Deku Baba's death burst, goal met. It kept striking for three more decisions after the kill, which is now fixed (it stops when the battle music ends).
    A fight with lock-on

  5. A failure, explained: both rupee targets were wrong in our own map data (one on a pond floor, one on a pillar in the water). The step even reported "picked" while the rupee count stayed 0, a false positive that is now fixed: a pickup counts only when the count goes up.
    A failure explained: rupee targets in the water

A 33-second clip of the chest run at real speed with the panel: https://git.magicbean.dev/attachments/ac338a9d-43d4-44d7-ba36-c5ecdaa50901

Where it stands. The second round with the first fixes solved none of six runs, each for a new reason found in the game (the door step walked past its prompt and then sidestepped 24 times at the door frame; the chosen rupees cannot be reached; the song's button order and the item button are unconfirmed). All are fixed or probed; the third round (10 runs, including probes that read the game's own note symbols and find the item button) is queued, then the full benchmark with cost per minute, then two-player runs. Results will follow here.

**Pictures with the decision overlay (16 November)** Each picture is the game's own frame with a panel below it, drawn afterwards from the test player's log (the game's pixels are not covered). How to read the panel: GOAL is the current goal and whether it is met; CANDIDATES lists the actions that made sense at that moment, with the chosen one highlighted (and the model's percentages when a model chose); STEP is what it is doing and its result word; LAST is the previous step's result; MODEL says which model answered, or "rules" when no model was asked; COST is the spend so far; STATE shows moving or recovering; VISION is the latest screenshot look in a few words; AT is the map, position, age, life and rupees. 1. Opening a chest in the Deku Tree: walk up, press A, Link holds up the Compass, A through the message, goal met by the chest's flag going from 0 to 1. Plain rules, no model call. ![Opening a chest, with the decision panel](https://git.magicbean.dev/attachments/7d6ae391-8a9a-48db-b8a7-e50460c8e6a1) 2. A screenshot look while stuck. At 11.6 s the camera does not show Link and the model sees open ground (stuck 3%); at 30.9 s it reports a wall ahead (stuck 94%) with Link at the closed gate, which is right. ![Two screenshot looks while stuck](https://git.magicbean.dev/attachments/7799e831-cc58-4a9a-9b10-ea6c6958046a) 3. A conversation with a Kokiri, including a two-choice question where the default answer is taken. ![A conversation](https://git.magicbean.dev/attachments/51447ed5-5180-4984-b121-ed31dd33d73e) 4. A fight: lock-on (the triangles), three slashes, the Deku Baba's death burst, goal met. It kept striking for three more decisions after the kill, which is now fixed (it stops when the battle music ends). ![A fight with lock-on](https://git.magicbean.dev/attachments/504b29b5-15b1-4984-b6eb-7e4415c79396) 5. A failure, explained: both rupee targets were wrong in our own map data (one on a pond floor, one on a pillar in the water). The step even reported "picked" while the rupee count stayed 0, a false positive that is now fixed: a pickup counts only when the count goes up. ![A failure explained: rupee targets in the water](https://git.magicbean.dev/attachments/9dde0043-17f9-4298-a48b-327af5684010) A 33-second clip of the chest run at real speed with the panel: https://git.magicbean.dev/attachments/ac338a9d-43d4-44d7-ba36-c5ecdaa50901 **Where it stands.** The second round with the first fixes solved none of six runs, each for a new reason found in the game (the door step walked past its prompt and then sidestepped 24 times at the door frame; the chosen rupees cannot be reached; the song's button order and the item button are unconfirmed). All are fixed or probed; the third round (10 runs, including probes that read the game's own note symbols and find the item button) is queued, then the full benchmark with cost per minute, then two-player runs. Results will follow here.
Author
Member

Bug hunting: first run (16 November)

The bug-hunting mode is built and had its first run: one pair session and four solo sessions, about 17 minutes in all, on a build of #68.

What it watches for: crashes and aborts, hangs, stalls, softlocks (input going in, nothing changing), falling out of the map, visual problems (black or frozen pictures, flicker, and a short list of questions to the vision model), log lines that are new to a build, co-op desync (shared values that disagree, a body far from its player) and items given twice.

A known bug, on purpose, to see that it is caught: both test players opened the same Forest Temple small-key chest at the same instant, with 300 ms of extra delay on one player's link. Each game's key count went from 3 to 5 for one chest, the double give that 68 lists as a known limit. The detector catches it on both sides and writes a packet: build, save, seed, the exact inputs, the state around it, frames before and after, and both logs. The first version missed it, because the game raises the key count while the item message is still open and sets the chest's flag a second later; that is fixed.

The shared-chest double give, caught by the detector

No new real bug in this first run (no crash, hang, softlock or out of bounds). The first pass also produced noise that is now filtered: the vision model called every load screen and fade "blank" (25 of 25 looks), so it now only looks once the player has control and skips black pictures.

Replay: every finding comes with a recording of its inputs in a small frame-stamped format that converts to the game's own input script, so it lines up with #1113. Replaying a packet in the game has not been tried yet; the next run replays the double give and roams four more places (inside the Deku Tree, Kakariko, the field by the lake, Dodongo's Cavern) looking for a real finding.

**Bug hunting: first run (16 November)** The bug-hunting mode is built and had its first run: one pair session and four solo sessions, about 17 minutes in all, on a build of https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/68. **What it watches for:** crashes and aborts, hangs, stalls, softlocks (input going in, nothing changing), falling out of the map, visual problems (black or frozen pictures, flicker, and a short list of questions to the vision model), log lines that are new to a build, co-op desync (shared values that disagree, a body far from its player) and items given twice. **A known bug, on purpose, to see that it is caught:** both test players opened the same Forest Temple small-key chest at the same instant, with 300 ms of extra delay on one player's link. Each game's key count went from 3 to 5 for one chest, the double give that 68 lists as a known limit. The detector catches it on both sides and writes a packet: build, save, seed, the exact inputs, the state around it, frames before and after, and both logs. The first version missed it, because the game raises the key count while the item message is still open and sets the chest's flag a second later; that is fixed. ![The shared-chest double give, caught by the detector](https://git.magicbean.dev/attachments/25bd4dbf-1253-4892-bef2-dcb0b78109da) **No new real bug in this first run** (no crash, hang, softlock or out of bounds). The first pass also produced noise that is now filtered: the vision model called every load screen and fade "blank" (25 of 25 looks), so it now only looks once the player has control and skips black pictures. **Replay:** every finding comes with a recording of its inputs in a small frame-stamped format that converts to the game's own input script, so it lines up with https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1113. Replaying a packet in the game has not been tried yet; the next run replays the double give and roams four more places (inside the Deku Tree, Kakariko, the field by the lake, Dodongo's Cavern) looking for a real finding.
Author
Member

Automated testing: who is doing what, and how the pieces fit (16 November)

Several people are working on parts of the same thing: automated in-game testing. This is a map of the pieces so nobody builds the same part twice, with the gaps between them. It covers the record and replay idea in #1113 as well as this issue's test player.

Piece Who Where it is State
Recording inputs and playing them back issue 1113 asks for it in the game's debug menu (including the menu's teleports). Phishinabowl already replays inputs with scripts for performance measurements. This issue's test player writes a frame-stamped recording of every input it sends the test player's recorder works today from outside the game, through the control bridge; nothing in the game yet recording works; replaying those recordings in the game is being tested now
Setting up a scene: place, time of day, age, items, flags the debug tools: a command-line tool and a Python interface over what the game's control bridge and debug menu already offer (teleport, age, items, flags, health, rupees, magic, snapshot/restore), each reading the game's state back to confirm it took effect built and tested without a game; the first run in the game is queued see also #1046 (a debug teleport the game dropped is sent again)
Spawning enemies and presets for them Zirem not on the forge yet in progress
Performance captures for a scenario Phishinabowl's measurement scripts; the frame-time and log tools the test player already collects separate today
Playing the game: walking, interacting, whole sections, two players this issue's test player dev branches, publishing to the workbench repository see the comments above
Finding bugs while playing this issue's bug hunter (crash, hang, softlock, out of bounds, visual, new log lines, co-op desync) dev branch first run caught the planted double give
Running it on other PCs the Test Farm (its coordinator and desktop agent already run game tests on testers' PCs) not connected to any of this yet
Splitting the audio: dialogue, cutscenes, effects, music Phishinabowl starting a player feature first; for testing it would let a check hear whether dialogue or effects play

Gaps, and what would close them

  1. One recording format. If the in-game recorder from 1113 and the outside recorder write the same file, a recording made by hand in the menu can be replayed by the test harness, and a recording the test player or bug hunter makes (for example the inputs that led to a crash) can be replayed in the game's own menu. Proposal: a small JSON lines file, one line per frame that changes input, plus the setup at the start (map, position, age, time, items) and any teleport as its own line. The outside recorder's format is a starting point, not a fixed choice.

  2. One scenario file. A scenario is a setup plus what to do: a recording, a test-player goal, or a measurement. If the debug-menu presets, the enemy presets and the debug tools read the same scenario file, a performance scenario and a bug-hunting scenario are the same thing with a different check at the end.

  3. Enemy spawning through the control bridge. If spawning is also a control-bridge command, scenarios can include it and the test player can use it for fights and stress tests without a save that happens to have the right enemies.

  4. A text line on screen. A control-bridge command that shows one line of text would let any of these tools show what they are doing while they play (the test player's overlay is drawn afterwards today).

  5. Where the code lives. Game-side parts (the in-game recorder, bridge commands, enemy spawning, the audio split) belong in Longshot pull requests; the outside harness (test player, bug hunter, debug tools) is published in the workbench repository so anyone can run it.

  6. Options for who and what drives it, with fallbacks. Not everyone has the same model account, and some tests need no model at all. The plan is several drivers behind one interface, each falling back to the next:

    • the decisions model (what the test player uses today: cheap and fast, one choice from a short list);
    • other models through a provider router such as OpenRouter, or any chat model, for people with other accounts or who want to compare models;
    • a general model or agent driving the game through the workbench's MCP tools (the debug tools, input and screenshots) for exploratory play and playback;
    • plain recorded playback, no model involved;
    • scripted, hard-coded tests (the rules policy the test player already has, and fixed input scripts) for checks that must give the same result every time.

    A scenario names the driver it wants and the fallbacks it accepts, so the same scenario runs for someone with no model account at all, just with less exploring.

Happy to adjust the test player and the debug tools to whatever format the in-game recorder settles on. Who wants which piece?

**Automated testing: who is doing what, and how the pieces fit (16 November)** Several people are working on parts of the same thing: automated in-game testing. This is a map of the pieces so nobody builds the same part twice, with the gaps between them. It covers the record and replay idea in https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1113 as well as this issue's test player. | Piece | Who | Where it is | State | |---|---|---|---| | Recording inputs and playing them back | issue 1113 asks for it in the game's debug menu (including the menu's teleports). Phishinabowl already replays inputs with scripts for performance measurements. This issue's test player writes a frame-stamped recording of every input it sends | the test player's recorder works today from outside the game, through the control bridge; nothing in the game yet | recording works; replaying those recordings in the game is being tested now | | Setting up a scene: place, time of day, age, items, flags | the debug tools: a command-line tool and a Python interface over what the game's control bridge and debug menu already offer (teleport, age, items, flags, health, rupees, magic, snapshot/restore), each reading the game's state back to confirm it took effect | built and tested without a game; the first run in the game is queued | see also https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1046 (a debug teleport the game dropped is sent again) | | Spawning enemies and presets for them | Zirem | not on the forge yet | in progress | | Performance captures for a scenario | Phishinabowl's measurement scripts; the frame-time and log tools the test player already collects | separate today | | | Playing the game: walking, interacting, whole sections, two players | this issue's test player | dev branches, publishing to the workbench repository | see the comments above | | Finding bugs while playing | this issue's bug hunter (crash, hang, softlock, out of bounds, visual, new log lines, co-op desync) | dev branch | first run caught the planted double give | | Running it on other PCs | the Test Farm (its coordinator and desktop agent already run game tests on testers' PCs) | | not connected to any of this yet | | Splitting the audio: dialogue, cutscenes, effects, music | Phishinabowl | starting | a player feature first; for testing it would let a check hear whether dialogue or effects play | **Gaps, and what would close them** 1. **One recording format.** If the in-game recorder from 1113 and the outside recorder write the same file, a recording made by hand in the menu can be replayed by the test harness, and a recording the test player or bug hunter makes (for example the inputs that led to a crash) can be replayed in the game's own menu. Proposal: a small JSON lines file, one line per frame that changes input, plus the setup at the start (map, position, age, time, items) and any teleport as its own line. The outside recorder's format is a starting point, not a fixed choice. 2. **One scenario file.** A scenario is a setup plus what to do: a recording, a test-player goal, or a measurement. If the debug-menu presets, the enemy presets and the debug tools read the same scenario file, a performance scenario and a bug-hunting scenario are the same thing with a different check at the end. 3. **Enemy spawning through the control bridge.** If spawning is also a control-bridge command, scenarios can include it and the test player can use it for fights and stress tests without a save that happens to have the right enemies. 4. **A text line on screen.** A control-bridge command that shows one line of text would let any of these tools show what they are doing while they play (the test player's overlay is drawn afterwards today). 5. **Where the code lives.** Game-side parts (the in-game recorder, bridge commands, enemy spawning, the audio split) belong in Longshot pull requests; the outside harness (test player, bug hunter, debug tools) is published in the workbench repository so anyone can run it. 6. **Options for who and what drives it, with fallbacks.** Not everyone has the same model account, and some tests need no model at all. The plan is several drivers behind one interface, each falling back to the next: - the decisions model (what the test player uses today: cheap and fast, one choice from a short list); - other models through a provider router such as OpenRouter, or any chat model, for people with other accounts or who want to compare models; - a general model or agent driving the game through the workbench's MCP tools (the debug tools, input and screenshots) for exploratory play and playback; - plain recorded playback, no model involved; - scripted, hard-coded tests (the rules policy the test player already has, and fixed input scripts) for checks that must give the same result every time. A scenario names the driver it wants and the fallbacks it accepts, so the same scenario runs for someone with no model account at all, just with less exploring. Happy to adjust the test player and the debug tools to whatever format the in-game recorder settles on. Who wants which piece?
Author
Member

Bug hunting: the replay worked, and fewer false alarms (16 November)

Update on the bug hunter from the first run, on a build of #68. This round asked three things: does a finding replay from its own packet, can the false alarms be cut without hiding anything, and can a hunt start in many places quickly.

The replay reproduced the double give. The first run's packet for the shared-chest double give (both players open the same Forest Temple small-key chest at the same instant, one player's link delayed by 300 ms each way) was played again in a new session from the packet's own case and inputs. Both games went from 3 keys to 5 for one chest again, and the detector fired on both sides with the same key as the first time, so the packet is enough to see the bug a second time. It is the known limit that 68 lists (two players taking the same chest within the network delay both get its item). The pictures are from the first run's packet: the moment of the give, and a second later.

The first player has opened the small-key chest and the key is raised above Link's head. This is the moment both games raise their key count.
About a second later: the body labelled ChestB (the second test player) stands at the chest, and the hearts and buttons are back on screen.

Four more roams found nothing: inside the Deku Tree, Kakariko, the field by the lake and Dodongo's Cavern (as a child), about 10 minutes of play in all. No crash, hang, softlock, out of bounds, stall, new log line or visual problem. Three of the four ended early because every generated checkpoint had been walked, so the next runs use more checkpoints.

Fewer false alarms, nothing dropped. Over the evidence kept so far (seven sessions: three pair sessions and four solo ones), the findings that need a person went from 48 to 30, and 21 were marked as explained with their reason instead of being listed as findings. The rules:

  • a stall that the machine's own pressure explains (the kernel's pressure counters for CPU, disk and memory are now recorded once a second next to the game's clock);
  • two games stalling at the same moment are one finding, not two;
  • the co-op lines about the other player going away are explained when they come at the other game's end, or just after a stall of either game;
  • the identity line before the room is accepted, and a send rate that only touches its limit (under 1.25 times), are explained;
  • the body snap that follows a stall is explained.
    For the first run's pair session alone the findings went from 10 to 3, the three being the double give on both games and one "both games stalled together". The two 6 second stalls of the first run did not come back when the same case was played again, which fits a busy machine rather than the game.

A flicker check. A flicker (the picture alternating A, B, A while nothing moves) cannot be seen in pictures taken a second and a half apart. The hunt now asks the game for a burst of eight consecutive pictures through its own control bridge when the live check suspects something (a black picture while playing, the same picture while Link moved, a large change while nothing moved), and also at quiet moments now and then so the rate of false alarms of the check itself can be measured. It is tested against a stand-in bridge; it has not seen a real game yet.

Starting in many places. A roam can now begin where the game's debug tools put Link: any of the game's 31 destinations, at a given age and hour (a dungeon, a town, the field at night), instead of needing a save made there. It uses the debug tools from the map of pieces earlier on this issue (teleport, age and time of day, each read back from the game to confirm it worked). This round's placed hunt (two roams and one co-op pair, in the field at night, Jabu-Jabu's Belly and Kakariko at night) is queued and has not run yet.

**Bug hunting: the replay worked, and fewer false alarms (16 November)** Update on the bug hunter from the first run, on a build of https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/68. This round asked three things: does a finding replay from its own packet, can the false alarms be cut without hiding anything, and can a hunt start in many places quickly. **The replay reproduced the double give.** The first run's packet for the shared-chest double give (both players open the same Forest Temple small-key chest at the same instant, one player's link delayed by 300 ms each way) was played again in a new session from the packet's own case and inputs. Both games went from 3 keys to 5 for one chest again, and the detector fired on both sides with the same key as the first time, so the packet is enough to see the bug a second time. It is the known limit that 68 lists (two players taking the same chest within the network delay both get its item). The pictures are from the first run's packet: the moment of the give, and a second later. ![The first player has opened the small-key chest and the key is raised above Link's head. This is the moment both games raise their key count.](https://git.magicbean.dev/attachments/b6109436-f750-42fa-871e-c4d0c7c0c1f9) ![About a second later: the body labelled ChestB (the second test player) stands at the chest, and the hearts and buttons are back on screen.](https://git.magicbean.dev/attachments/83118f8f-2878-4da0-98d9-42c476803fcc) **Four more roams found nothing:** inside the Deku Tree, Kakariko, the field by the lake and Dodongo's Cavern (as a child), about 10 minutes of play in all. No crash, hang, softlock, out of bounds, stall, new log line or visual problem. Three of the four ended early because every generated checkpoint had been walked, so the next runs use more checkpoints. **Fewer false alarms, nothing dropped.** Over the evidence kept so far (seven sessions: three pair sessions and four solo ones), the findings that need a person went from 48 to 30, and 21 were marked as explained with their reason instead of being listed as findings. The rules: - a stall that the machine's own pressure explains (the kernel's pressure counters for CPU, disk and memory are now recorded once a second next to the game's clock); - two games stalling at the same moment are one finding, not two; - the co-op lines about the other player going away are explained when they come at the other game's end, or just after a stall of either game; - the identity line before the room is accepted, and a send rate that only touches its limit (under 1.25 times), are explained; - the body snap that follows a stall is explained. For the first run's pair session alone the findings went from 10 to 3, the three being the double give on both games and one "both games stalled together". The two 6 second stalls of the first run did not come back when the same case was played again, which fits a busy machine rather than the game. **A flicker check.** A flicker (the picture alternating A, B, A while nothing moves) cannot be seen in pictures taken a second and a half apart. The hunt now asks the game for a burst of eight consecutive pictures through its own control bridge when the live check suspects something (a black picture while playing, the same picture while Link moved, a large change while nothing moved), and also at quiet moments now and then so the rate of false alarms of the check itself can be measured. It is tested against a stand-in bridge; it has not seen a real game yet. **Starting in many places.** A roam can now begin where the game's debug tools put Link: any of the game's 31 destinations, at a given age and hour (a dungeon, a town, the field at night), instead of needing a save made there. It uses the debug tools from the map of pieces earlier on this issue (teleport, age and time of day, each read back from the game to confirm it worked). This round's placed hunt (two roams and one co-op pair, in the field at night, Jabu-Jabu's Belly and Kakariko at night) is queued and has not run yet.
Author
Member

The pilot plays whole stretches of the game: first chapter runs (16 November)

Follow-up on #1198. A chapter is an aim in plain words ("reach Lake Hylia"), a few progress markers the game's own state can prove (a map, a place, a flag, rupees), and budgets. A planning model (gpt-6-astra here) reads the aim, the game state, a summary of what the map holds around Link and one picture, and answers with two or three objectives (go to a place, open a chest, read a sign, enter a door). The pilot plays each objective with its usual step chooser (gpt-6-luna) and asks the planner again when one fails, telling it what the steps reported. This is the first time it ran in the game, on a build of #68. About ten minutes of play in all.

A lost hold first. The first slot we got ran for a quarter of a second: the movement and combat steps added since the last run were imported in a way that only fails when the script is started the way the pilot starts it. All five runs of that hold ended at once, no game was started and nothing was spent. There is now a test that starts every script the way a hold does, and the hold tool refuses a plan whose scripts cannot start. The second slot ran fully.

Field to Lake Hylia: solved. Link starts as a child in the middle of Hyrule Field by day. The planner made two objectives in 12 seconds for 0.08 USD: read a signpost on the way south, then follow the walkable route to the lake's north shore. The pilot walked about 320 m in 101 seconds of game time to 27 m from the shore, where the chapter's marker (within 30 m) held. Link was then stopped by a scene: Kaepora Gaebora speaks ("Hoo-hooo!"), which the run reports as an event holding him. Total cost 0.0825 USD, of which the plan is 0.0816 USD and the step choices 0.0009 USD.

39 seconds in, on the way to the sign objective's point (106 m away along the route). Under the picture: the goal, the four things it could do with the chosen one in yellow, the model that chose (gpt-6-luna, 0.18 s), what vision last saw, the cost so far and where Link is.
105 seconds in, at the lake. The owl's scene holds Link, the step ends with the word "held", the goal is marked met and the run solved.

One thing it got wrong, and a fix. The sign objective is marked solved, but the sign was not read: the planner tied that objective to the marker "Link is more than 100 m from where he began", and a rule in the chapter engine ended an objective as soon as the marker it serves held, which happened on the way. That rule was added the day before after a test against a simulated game, where an objective could never end because the map gave a height 9 m off. It now applies only to plain walking objectives (and to the whole chapter being solved), so a chest, a talk or a sign still has to be done.

Warp song (Minuet of Forest): not solved. The pilot took the ocarina out and played the six notes of both guesses for the button order (one after the other, then the first again), the game accepted them and closed the note wait each time, and Link stayed in the field. The vision model's look at the screen only said the ocarina controls were still showing. The planner was asked three times (0.24 USD in all, 7 to 16 seconds each), first for the song, then to "release the ocarina", then for a fresh attempt, and in the last objective the run hit its wall time. The note buttons and the note length are still unconfirmed (150 ms notes), so no warp is expected until those are settled by the separate probe of the songbook. A second fault showed here: while the ocarina waits for a note, the "advance the text" step presses A every 1.2 seconds, which is a note, not a way out.

The song step: "take the ocarina out and play minuet as L A X Y X Y" while the hint bar shows "Check songbook, Play a note, Cancel". The panel shows the last attempt done with six notes, no model asked because nothing had changed, and Link still in the field.

The new steps, in the game (rules decider, no model, three short routes):

  • Swimming and diving work. In the Kokiri pond, A held for 2 seconds took Link from y 108.31 to 106.97 and back up, and the swim steps moved him across the water.
  • A roll moves 2.2 to 2.3 m. Holding ZL with the stick sideways and no target strafes 4.2 m in 1.2 seconds with the facing unchanged, in both directions. The shield step keeps life unchanged.
  • A jump lifts Link 0.7 to 0.8 m and carries him 3 to 5 m. Both jumps in the field probe ended "short" of a point 7 to 14 m away, as expected for a probe that asked more than a jump gives.
  • The item step reports "used", but only because the camera moved, so the item button is still not found. The ride step reports "walked": there is no horse in that field save.
  • In the Deku Tree's first room the walk steps reached their points, the first jump ended "landed" while Link had barely moved (his start was already inside the step's 1.2 m radius, a probe error that the step should not accept), and then a Deku Baba took Link from 12 to 0 quarter hearts in 30 seconds while the route kept walking. The ledge and the ivy were never reached. A route with no chooser does not fight, so that is the route's fault, not the steps'; the Deku Tree chapter has fights in its plan.

Next: fix the two faults above plus the jump radius, then the Deku Tree chapter (the three chests, then the web floor down toward Gohma), the Kokiri forest chapter (Saria and Mido, the rupees, the shop's shield, the sword chest, the Deku Tree's mouth), the Kakariko chests and a free roam of the field, one hold each, with a timeline and a cost for each.

**The pilot plays whole stretches of the game: first chapter runs (16 November)** Follow-up on https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1198. A chapter is an aim in plain words ("reach Lake Hylia"), a few progress markers the game's own state can prove (a map, a place, a flag, rupees), and budgets. A planning model (gpt-6-astra here) reads the aim, the game state, a summary of what the map holds around Link and one picture, and answers with two or three objectives (go to a place, open a chest, read a sign, enter a door). The pilot plays each objective with its usual step chooser (gpt-6-luna) and asks the planner again when one fails, telling it what the steps reported. This is the first time it ran in the game, on a build of https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/68. About ten minutes of play in all. **A lost hold first.** The first slot we got ran for a quarter of a second: the movement and combat steps added since the last run were imported in a way that only fails when the script is started the way the pilot starts it. All five runs of that hold ended at once, no game was started and nothing was spent. There is now a test that starts every script the way a hold does, and the hold tool refuses a plan whose scripts cannot start. The second slot ran fully. **Field to Lake Hylia: solved.** Link starts as a child in the middle of Hyrule Field by day. The planner made two objectives in 12 seconds for 0.08 USD: read a signpost on the way south, then follow the walkable route to the lake's north shore. The pilot walked about 320 m in 101 seconds of game time to 27 m from the shore, where the chapter's marker (within 30 m) held. Link was then stopped by a scene: Kaepora Gaebora speaks ("Hoo-hooo!"), which the run reports as an event holding him. Total cost 0.0825 USD, of which the plan is 0.0816 USD and the step choices 0.0009 USD. ![39 seconds in, on the way to the sign objective's point (106 m away along the route). Under the picture: the goal, the four things it could do with the chosen one in yellow, the model that chose (gpt-6-luna, 0.18 s), what vision last saw, the cost so far and where Link is.](https://git.magicbean.dev/attachments/03e28363-3d25-4778-8437-537f947ccb7a) ![105 seconds in, at the lake. The owl's scene holds Link, the step ends with the word "held", the goal is marked met and the run solved.](https://git.magicbean.dev/attachments/d3997eca-29f6-422c-a681-617ec1c7d02a) One thing it got wrong, and a fix. The sign objective is marked solved, but the sign was not read: the planner tied that objective to the marker "Link is more than 100 m from where he began", and a rule in the chapter engine ended an objective as soon as the marker it serves held, which happened on the way. That rule was added the day before after a test against a simulated game, where an objective could never end because the map gave a height 9 m off. It now applies only to plain walking objectives (and to the whole chapter being solved), so a chest, a talk or a sign still has to be done. **Warp song (Minuet of Forest): not solved.** The pilot took the ocarina out and played the six notes of both guesses for the button order (one after the other, then the first again), the game accepted them and closed the note wait each time, and Link stayed in the field. The vision model's look at the screen only said the ocarina controls were still showing. The planner was asked three times (0.24 USD in all, 7 to 16 seconds each), first for the song, then to "release the ocarina", then for a fresh attempt, and in the last objective the run hit its wall time. The note buttons and the note length are still unconfirmed (150 ms notes), so no warp is expected until those are settled by the separate probe of the songbook. A second fault showed here: while the ocarina waits for a note, the "advance the text" step presses A every 1.2 seconds, which is a note, not a way out. ![The song step: "take the ocarina out and play minuet as L A X Y X Y" while the hint bar shows "Check songbook, Play a note, Cancel". The panel shows the last attempt done with six notes, no model asked because nothing had changed, and Link still in the field.](https://git.magicbean.dev/attachments/f772d964-1239-4e15-85dc-cf8008cedb37) **The new steps, in the game (rules decider, no model, three short routes):** - Swimming and diving work. In the Kokiri pond, A held for 2 seconds took Link from y 108.31 to 106.97 and back up, and the swim steps moved him across the water. - A roll moves 2.2 to 2.3 m. Holding ZL with the stick sideways and no target strafes 4.2 m in 1.2 seconds with the facing unchanged, in both directions. The shield step keeps life unchanged. - A jump lifts Link 0.7 to 0.8 m and carries him 3 to 5 m. Both jumps in the field probe ended "short" of a point 7 to 14 m away, as expected for a probe that asked more than a jump gives. - The item step reports "used", but only because the camera moved, so the item button is still not found. The ride step reports "walked": there is no horse in that field save. - In the Deku Tree's first room the walk steps reached their points, the first jump ended "landed" while Link had barely moved (his start was already inside the step's 1.2 m radius, a probe error that the step should not accept), and then a Deku Baba took Link from 12 to 0 quarter hearts in 30 seconds while the route kept walking. The ledge and the ivy were never reached. A route with no chooser does not fight, so that is the route's fault, not the steps'; the Deku Tree chapter has fights in its plan. **Next:** fix the two faults above plus the jump radius, then the Deku Tree chapter (the three chests, then the web floor down toward Gohma), the Kokiri forest chapter (Saria and Mido, the rupees, the shop's shield, the sword chest, the Deku Tree's mouth), the Kakariko chests and a free roam of the field, one hold each, with a timeline and a cost for each.
Author
Member

A bug-hunting roam you can watch live: Kakariko Village, 4 minutes 24 seconds (16 November)

Follow-up on #1198, on a build of #68. Until now the pilot played with no window at all. This was the first run where a person watched it as it happened: the game in a window on the left two thirds of the monitor, never focused, so keys and mouse never reached it, and a small terminal panel on the right third showing each decision as it was made. The panel lists the candidate steps with the chosen one marked and its probability, the step and its result word, the stuck and recovery state, map, position and life, the latest answer of the picture look, any finding as it is raised, and the cost so far. The game was a Kakariko Village roam as an adult with 15 checkpoints, the steps chosen by gpt-6-luna and the pictures looked at by the vision model.

What it did. 36 decisions in 215 seconds of play (4 minutes 24 seconds from start to the game's end): 15 of them asked the model, 14 were made by the decision guard without a question (for instance moving on once a checkpoint is reached), 7 were stuck-recovery rungs. 21 steps ended: 16 reached their point, 5 ended stuck. Three times the walk was blocked, and all three recovered (once by turning the approach angle, twice by stepping aside). The vision model looked 5 times. The whole run cost 0.0043 USD (20 requests). The window stayed where the rules put it (1024 by 864 on a 1536 by 864 desktop, floating, same workspace), was never the active window, and the active window and the other windows were the same after the run as before it.

It ended early, by the roam's own rule and not by a fault. We asked for 12 to 14 minutes. The village gave 15 reachable checkpoints (30 were asked for), all of them were done or given up on after 215 seconds of play, and the roam goal then closes as failed ("every target done or failed"), which ends the run. The same happened in earlier roams. The fix for the next run is a longer list: the field is large enough to hand out 60 checkpoints.

Placement was partial, and why. The roam was meant to start at 22:00. The debug tools' placement did two of its three steps and read both back as verified (the age, the teleport to Kakariko Village) and failed the third, the time of day: that step needs the file listing the game's debug actions, which the pilot did not pass. The pilot then did what it is built to do and played on from where its save already stood (Kakariko by day), saying so on the panel. About two minutes in we set the clock to 22:00 by hand with the same tools (the game acknowledged the request), so the roam has a day half and a night half. The two stills show the same stretch of the village before and after: the dark picture is the clock change taking effect, not a different place. The next fix is for the placement to be given that file.

Kakariko Village by day: Link with his fairy beside a rock face, the village map in the corner.
The same stretch after the clock was set to 22:00: the same rocks and the same fairy, now in blue night light, with a torch burning on the wall beyond.
Twelve decisions from the run drawn on the game's own pictures, from the first checkpoint by day to the last at night: each shows the candidates with the chosen one highlighted, the step, the model and its latency, the cost so far, the recovery state and the latest picture look.

What the hunter found. Nothing that is a game bug.

  • Visual, one finding at error level: black pictures while the state said play, in the first seconds of play right after the placement reloaded the map (the first pictures were black for under two seconds, the next ones normal). That is the scene fading in after the reload, which the placement tool's own notes describe, and the detector does not know about it yet. We will have it skip a short settle window after a placement.
  • Log, one warning: a line new to the baseline, a start-up notice that the crash watcher is on ("crash window: watching this run"). Harmless, and it will go into the baseline.
  • Stall, information only: one slow present of 623 ms in the first second of play, while programs were being prepared (explained).
  • Nothing from the crash, hang, soft-lock, out-of-bounds or audio detectors (the double-give detector had no flag to judge in this area); the game's log has no FATAL, CRASH or device-lost lines and the kernel log was clean.
  • Looking by eye at the twelve decision pictures of the contact sheet and the two stills: no missing or stretched texture and nothing drawn wrong. In one picture a villager stands beside Link with the Talk prompt showing; the roam walked on. The 5 stuck steps came with the vision model reporting a wall or obstacle ahead, which is the recovery ladder's normal work.

Not a speed test. The machine was busy with other runs (load average 12 to 15), pictures were taken every 20 presents, and the window was being recorded, so the frame times from this run say nothing about performance.

**A bug-hunting roam you can watch live: Kakariko Village, 4 minutes 24 seconds (16 November)** Follow-up on https://git.magicbean.dev/MagicBeanDev/Longshot/issues/1198, on a build of https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/68. Until now the pilot played with no window at all. This was the first run where a person watched it as it happened: the game in a window on the left two thirds of the monitor, never focused, so keys and mouse never reached it, and a small terminal panel on the right third showing each decision as it was made. The panel lists the candidate steps with the chosen one marked and its probability, the step and its result word, the stuck and recovery state, map, position and life, the latest answer of the picture look, any finding as it is raised, and the cost so far. The game was a Kakariko Village roam as an adult with 15 checkpoints, the steps chosen by gpt-6-luna and the pictures looked at by the vision model. **What it did.** 36 decisions in 215 seconds of play (4 minutes 24 seconds from start to the game's end): 15 of them asked the model, 14 were made by the decision guard without a question (for instance moving on once a checkpoint is reached), 7 were stuck-recovery rungs. 21 steps ended: 16 reached their point, 5 ended stuck. Three times the walk was blocked, and all three recovered (once by turning the approach angle, twice by stepping aside). The vision model looked 5 times. The whole run cost 0.0043 USD (20 requests). The window stayed where the rules put it (1024 by 864 on a 1536 by 864 desktop, floating, same workspace), was never the active window, and the active window and the other windows were the same after the run as before it. **It ended early, by the roam's own rule and not by a fault.** We asked for 12 to 14 minutes. The village gave 15 reachable checkpoints (30 were asked for), all of them were done or given up on after 215 seconds of play, and the roam goal then closes as failed ("every target done or failed"), which ends the run. The same happened in earlier roams. The fix for the next run is a longer list: the field is large enough to hand out 60 checkpoints. **Placement was partial, and why.** The roam was meant to start at 22:00. The debug tools' placement did two of its three steps and read both back as verified (the age, the teleport to Kakariko Village) and failed the third, the time of day: that step needs the file listing the game's debug actions, which the pilot did not pass. The pilot then did what it is built to do and played on from where its save already stood (Kakariko by day), saying so on the panel. About two minutes in we set the clock to 22:00 by hand with the same tools (the game acknowledged the request), so the roam has a day half and a night half. The two stills show the same stretch of the village before and after: the dark picture is the clock change taking effect, not a different place. The next fix is for the placement to be given that file. ![Kakariko Village by day: Link with his fairy beside a rock face, the village map in the corner.](https://git.magicbean.dev/attachments/c6eb3225-e790-43a3-84fe-ab8d9bbc2bd5) ![The same stretch after the clock was set to 22:00: the same rocks and the same fairy, now in blue night light, with a torch burning on the wall beyond.](https://git.magicbean.dev/attachments/2385f53e-fe69-4380-96cf-59f3c0ed2985) ![Twelve decisions from the run drawn on the game's own pictures, from the first checkpoint by day to the last at night: each shows the candidates with the chosen one highlighted, the step, the model and its latency, the cost so far, the recovery state and the latest picture look.](https://git.magicbean.dev/attachments/a3c5bb33-d987-46b8-9d9f-5065ab3f646c) **What the hunter found.** Nothing that is a game bug. - Visual, one finding at error level: black pictures while the state said play, in the first seconds of play right after the placement reloaded the map (the first pictures were black for under two seconds, the next ones normal). That is the scene fading in after the reload, which the placement tool's own notes describe, and the detector does not know about it yet. We will have it skip a short settle window after a placement. - Log, one warning: a line new to the baseline, a start-up notice that the crash watcher is on ("crash window: watching this run"). Harmless, and it will go into the baseline. - Stall, information only: one slow present of 623 ms in the first second of play, while programs were being prepared (explained). - Nothing from the crash, hang, soft-lock, out-of-bounds or audio detectors (the double-give detector had no flag to judge in this area); the game's log has no FATAL, CRASH or device-lost lines and the kernel log was clean. - Looking by eye at the twelve decision pictures of the contact sheet and the two stills: no missing or stretched texture and nothing drawn wrong. In one picture a villager stands beside Link with the Talk prompt showing; the roam walked on. The 5 stuck steps came with the vision model reporting a wall or obstacle ahead, which is the recovery ladder's normal work. **Not a speed test.** The machine was busy with other runs (load average 12 to 15), pictures were taken every 20 presents, and the window was being recorded, so the frame times from this run say nothing about performance.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1198
No description provided.