Idea: an automated test player that plays sections of the game and hunts for bugs #1198
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#1198
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The idea
An automated test player that plays Longshot by itself, so that sections of the game can be played again on every build and bugs are found by playing, not only by tests that never start the game. Long term it becomes a bug hunter: it plays while detectors watch for crashes, hangs, softlocks, falling out of the map, visual glitches and co-op desyncs, and every finding comes with the exact inputs to replay it.
How it works today
host/control.c): it reads the sampled game state (position, map, age, health, the on-screen prompt) and sends controller input. Nothing made from the game leaves the PC.Results so far (Linux, RTX 4090, builds of #68 and main)
Pictures with the decision overlay (the goal, the choices and the one taken, at a glance) follow in a comment, together with each new round's results.
Cost. About 0.0002 USD per decision, and 0.0003 to 0.025 USD per screenshot look depending on the model. The first 2,600 model requests cost about half a dollar together.
Next
Anything else
The code is being published in the workbench repository once it is proven in the game; the world data, saves and routes stay local to each PC, because they are made from the game. Questions for the maintainers:
Possibly related, for context:
Pictures with the decision overlay (16 November)
Each picture is the game's own frame with a panel below it, drawn afterwards from the test player's log (the game's pixels are not covered). How to read the panel: GOAL is the current goal and whether it is met; CANDIDATES lists the actions that made sense at that moment, with the chosen one highlighted (and the model's percentages when a model chose); STEP is what it is doing and its result word; LAST is the previous step's result; MODEL says which model answered, or "rules" when no model was asked; COST is the spend so far; STATE shows moving or recovering; VISION is the latest screenshot look in a few words; AT is the map, position, age, life and rupees.
Opening a chest in the Deku Tree: walk up, press A, Link holds up the Compass, A through the message, goal met by the chest's flag going from 0 to 1. Plain rules, no model call.

A screenshot look while stuck. At 11.6 s the camera does not show Link and the model sees open ground (stuck 3%); at 30.9 s it reports a wall ahead (stuck 94%) with Link at the closed gate, which is right.

A conversation with a Kokiri, including a two-choice question where the default answer is taken.

A fight: lock-on (the triangles), three slashes, the Deku Baba's death burst, goal met. It kept striking for three more decisions after the kill, which is now fixed (it stops when the battle music ends).

A failure, explained: both rupee targets were wrong in our own map data (one on a pond floor, one on a pillar in the water). The step even reported "picked" while the rupee count stayed 0, a false positive that is now fixed: a pickup counts only when the count goes up.

A 33-second clip of the chest run at real speed with the panel: https://git.magicbean.dev/attachments/ac338a9d-43d4-44d7-ba36-c5ecdaa50901
Where it stands. The second round with the first fixes solved none of six runs, each for a new reason found in the game (the door step walked past its prompt and then sidestepped 24 times at the door frame; the chosen rupees cannot be reached; the song's button order and the item button are unconfirmed). All are fixed or probed; the third round (10 runs, including probes that read the game's own note symbols and find the item button) is queued, then the full benchmark with cost per minute, then two-player runs. Results will follow here.
Bug hunting: first run (16 November)
The bug-hunting mode is built and had its first run: one pair session and four solo sessions, about 17 minutes in all, on a build of #68.
What it watches for: crashes and aborts, hangs, stalls, softlocks (input going in, nothing changing), falling out of the map, visual problems (black or frozen pictures, flicker, and a short list of questions to the vision model), log lines that are new to a build, co-op desync (shared values that disagree, a body far from its player) and items given twice.
A known bug, on purpose, to see that it is caught: both test players opened the same Forest Temple small-key chest at the same instant, with 300 ms of extra delay on one player's link. Each game's key count went from 3 to 5 for one chest, the double give that 68 lists as a known limit. The detector catches it on both sides and writes a packet: build, save, seed, the exact inputs, the state around it, frames before and after, and both logs. The first version missed it, because the game raises the key count while the item message is still open and sets the chest's flag a second later; that is fixed.
No new real bug in this first run (no crash, hang, softlock or out of bounds). The first pass also produced noise that is now filtered: the vision model called every load screen and fade "blank" (25 of 25 looks), so it now only looks once the player has control and skips black pictures.
Replay: every finding comes with a recording of its inputs in a small frame-stamped format that converts to the game's own input script, so it lines up with #1113. Replaying a packet in the game has not been tried yet; the next run replays the double give and roams four more places (inside the Deku Tree, Kakariko, the field by the lake, Dodongo's Cavern) looking for a real finding.
Automated testing: who is doing what, and how the pieces fit (16 November)
Several people are working on parts of the same thing: automated in-game testing. This is a map of the pieces so nobody builds the same part twice, with the gaps between them. It covers the record and replay idea in #1113 as well as this issue's test player.
Gaps, and what would close them
One recording format. If the in-game recorder from 1113 and the outside recorder write the same file, a recording made by hand in the menu can be replayed by the test harness, and a recording the test player or bug hunter makes (for example the inputs that led to a crash) can be replayed in the game's own menu. Proposal: a small JSON lines file, one line per frame that changes input, plus the setup at the start (map, position, age, time, items) and any teleport as its own line. The outside recorder's format is a starting point, not a fixed choice.
One scenario file. A scenario is a setup plus what to do: a recording, a test-player goal, or a measurement. If the debug-menu presets, the enemy presets and the debug tools read the same scenario file, a performance scenario and a bug-hunting scenario are the same thing with a different check at the end.
Enemy spawning through the control bridge. If spawning is also a control-bridge command, scenarios can include it and the test player can use it for fights and stress tests without a save that happens to have the right enemies.
A text line on screen. A control-bridge command that shows one line of text would let any of these tools show what they are doing while they play (the test player's overlay is drawn afterwards today).
Where the code lives. Game-side parts (the in-game recorder, bridge commands, enemy spawning, the audio split) belong in Longshot pull requests; the outside harness (test player, bug hunter, debug tools) is published in the workbench repository so anyone can run it.
Options for who and what drives it, with fallbacks. Not everyone has the same model account, and some tests need no model at all. The plan is several drivers behind one interface, each falling back to the next:
A scenario names the driver it wants and the fallbacks it accepts, so the same scenario runs for someone with no model account at all, just with less exploring.
Happy to adjust the test player and the debug tools to whatever format the in-game recorder settles on. Who wants which piece?
Bug hunting: the replay worked, and fewer false alarms (16 November)
Update on the bug hunter from the first run, on a build of #68. This round asked three things: does a finding replay from its own packet, can the false alarms be cut without hiding anything, and can a hunt start in many places quickly.
The replay reproduced the double give. The first run's packet for the shared-chest double give (both players open the same Forest Temple small-key chest at the same instant, one player's link delayed by 300 ms each way) was played again in a new session from the packet's own case and inputs. Both games went from 3 keys to 5 for one chest again, and the detector fired on both sides with the same key as the first time, so the packet is enough to see the bug a second time. It is the known limit that 68 lists (two players taking the same chest within the network delay both get its item). The pictures are from the first run's packet: the moment of the give, and a second later.
Four more roams found nothing: inside the Deku Tree, Kakariko, the field by the lake and Dodongo's Cavern (as a child), about 10 minutes of play in all. No crash, hang, softlock, out of bounds, stall, new log line or visual problem. Three of the four ended early because every generated checkpoint had been walked, so the next runs use more checkpoints.
Fewer false alarms, nothing dropped. Over the evidence kept so far (seven sessions: three pair sessions and four solo ones), the findings that need a person went from 48 to 30, and 21 were marked as explained with their reason instead of being listed as findings. The rules:
For the first run's pair session alone the findings went from 10 to 3, the three being the double give on both games and one "both games stalled together". The two 6 second stalls of the first run did not come back when the same case was played again, which fits a busy machine rather than the game.
A flicker check. A flicker (the picture alternating A, B, A while nothing moves) cannot be seen in pictures taken a second and a half apart. The hunt now asks the game for a burst of eight consecutive pictures through its own control bridge when the live check suspects something (a black picture while playing, the same picture while Link moved, a large change while nothing moved), and also at quiet moments now and then so the rate of false alarms of the check itself can be measured. It is tested against a stand-in bridge; it has not seen a real game yet.
Starting in many places. A roam can now begin where the game's debug tools put Link: any of the game's 31 destinations, at a given age and hour (a dungeon, a town, the field at night), instead of needing a save made there. It uses the debug tools from the map of pieces earlier on this issue (teleport, age and time of day, each read back from the game to confirm it worked). This round's placed hunt (two roams and one co-op pair, in the field at night, Jabu-Jabu's Belly and Kakariko at night) is queued and has not run yet.
The pilot plays whole stretches of the game: first chapter runs (16 November)
Follow-up on #1198. A chapter is an aim in plain words ("reach Lake Hylia"), a few progress markers the game's own state can prove (a map, a place, a flag, rupees), and budgets. A planning model (gpt-6-astra here) reads the aim, the game state, a summary of what the map holds around Link and one picture, and answers with two or three objectives (go to a place, open a chest, read a sign, enter a door). The pilot plays each objective with its usual step chooser (gpt-6-luna) and asks the planner again when one fails, telling it what the steps reported. This is the first time it ran in the game, on a build of #68. About ten minutes of play in all.
A lost hold first. The first slot we got ran for a quarter of a second: the movement and combat steps added since the last run were imported in a way that only fails when the script is started the way the pilot starts it. All five runs of that hold ended at once, no game was started and nothing was spent. There is now a test that starts every script the way a hold does, and the hold tool refuses a plan whose scripts cannot start. The second slot ran fully.
Field to Lake Hylia: solved. Link starts as a child in the middle of Hyrule Field by day. The planner made two objectives in 12 seconds for 0.08 USD: read a signpost on the way south, then follow the walkable route to the lake's north shore. The pilot walked about 320 m in 101 seconds of game time to 27 m from the shore, where the chapter's marker (within 30 m) held. Link was then stopped by a scene: Kaepora Gaebora speaks ("Hoo-hooo!"), which the run reports as an event holding him. Total cost 0.0825 USD, of which the plan is 0.0816 USD and the step choices 0.0009 USD.
One thing it got wrong, and a fix. The sign objective is marked solved, but the sign was not read: the planner tied that objective to the marker "Link is more than 100 m from where he began", and a rule in the chapter engine ended an objective as soon as the marker it serves held, which happened on the way. That rule was added the day before after a test against a simulated game, where an objective could never end because the map gave a height 9 m off. It now applies only to plain walking objectives (and to the whole chapter being solved), so a chest, a talk or a sign still has to be done.
Warp song (Minuet of Forest): not solved. The pilot took the ocarina out and played the six notes of both guesses for the button order (one after the other, then the first again), the game accepted them and closed the note wait each time, and Link stayed in the field. The vision model's look at the screen only said the ocarina controls were still showing. The planner was asked three times (0.24 USD in all, 7 to 16 seconds each), first for the song, then to "release the ocarina", then for a fresh attempt, and in the last objective the run hit its wall time. The note buttons and the note length are still unconfirmed (150 ms notes), so no warp is expected until those are settled by the separate probe of the songbook. A second fault showed here: while the ocarina waits for a note, the "advance the text" step presses A every 1.2 seconds, which is a note, not a way out.
The new steps, in the game (rules decider, no model, three short routes):
Next: fix the two faults above plus the jump radius, then the Deku Tree chapter (the three chests, then the web floor down toward Gohma), the Kokiri forest chapter (Saria and Mido, the rupees, the shop's shield, the sword chest, the Deku Tree's mouth), the Kakariko chests and a free roam of the field, one hold each, with a timeline and a cost for each.
A bug-hunting roam you can watch live: Kakariko Village, 4 minutes 24 seconds (16 November)
Follow-up on #1198, on a build of #68. Until now the pilot played with no window at all. This was the first run where a person watched it as it happened: the game in a window on the left two thirds of the monitor, never focused, so keys and mouse never reached it, and a small terminal panel on the right third showing each decision as it was made. The panel lists the candidate steps with the chosen one marked and its probability, the step and its result word, the stuck and recovery state, map, position and life, the latest answer of the picture look, any finding as it is raised, and the cost so far. The game was a Kakariko Village roam as an adult with 15 checkpoints, the steps chosen by gpt-6-luna and the pictures looked at by the vision model.
What it did. 36 decisions in 215 seconds of play (4 minutes 24 seconds from start to the game's end): 15 of them asked the model, 14 were made by the decision guard without a question (for instance moving on once a checkpoint is reached), 7 were stuck-recovery rungs. 21 steps ended: 16 reached their point, 5 ended stuck. Three times the walk was blocked, and all three recovered (once by turning the approach angle, twice by stepping aside). The vision model looked 5 times. The whole run cost 0.0043 USD (20 requests). The window stayed where the rules put it (1024 by 864 on a 1536 by 864 desktop, floating, same workspace), was never the active window, and the active window and the other windows were the same after the run as before it.
It ended early, by the roam's own rule and not by a fault. We asked for 12 to 14 minutes. The village gave 15 reachable checkpoints (30 were asked for), all of them were done or given up on after 215 seconds of play, and the roam goal then closes as failed ("every target done or failed"), which ends the run. The same happened in earlier roams. The fix for the next run is a longer list: the field is large enough to hand out 60 checkpoints.
Placement was partial, and why. The roam was meant to start at 22:00. The debug tools' placement did two of its three steps and read both back as verified (the age, the teleport to Kakariko Village) and failed the third, the time of day: that step needs the file listing the game's debug actions, which the pilot did not pass. The pilot then did what it is built to do and played on from where its save already stood (Kakariko by day), saying so on the panel. About two minutes in we set the clock to 22:00 by hand with the same tools (the game acknowledged the request), so the roam has a day half and a night half. The two stills show the same stretch of the village before and after: the dark picture is the clock change taking effect, not a different place. The next fix is for the placement to be given that file.
What the hunter found. Nothing that is a game bug.
Not a speed test. The machine was busy with other runs (load average 12 to 15), pictures were taken every 20 presents, and the window was being recorded, so the frame times from this run say nothing about performance.