Idea: targeted gameplay scenarios from Longshot's test player, run on the farm #82
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk#82
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
Of the 245 Longshot issues labelled
kind/bug, 153 of the 168 with a platform label are Windows and 84 of the 166 with a GPU label are AMD or Intel. Of the 36 whose title names a crash, freeze, hang, black screen, device loss or driver timeout, 20 of the 24 with a GPU label are AMD or Intel.Our automated test player (Longshot #1198) can place Link and play a fixed list of actions, but only on one Linux PC with an RTX 4090. The farm has the other hardware: 27 registered testers, 17 on Windows, 10 on AMD, 1 on Intel (no working sandbox yet).
What a scenario is
A place, a fixed list of actions and what to watch.
How it could fit Beanstalk
Your
[[scenario]]is already this unit. Plans can teleport, switch age, press, run a button script andexpect, and thecrashprobe is always on (test-plan.md). Two ways to ship the rest:destinations.pyalready warns.assets/saves/onmainwith each README's first paragraph for the planner and checks the names (gather_saves,ai_checks). A scenario would be a[[scenario]]fragment inassets/scenarios/<name>/, reviewed like a save.I think 2 fits better, on two conditions. Read it from
main, not the PR's commit, or the author writes the test for their own change (saves already come frommain;.farm/needs two approvers). And let the coordinator expand the fragment into the plan before signing, so the ticket carries the exact steps (architecture.md, "Tickets") and the agent reads no repository file for steps. What the agent cannot do yet stays a built-in step (the Spec'sscriptprobe is not built), offered only to agents with the newplan_schema(protocol.md). Four small ones, no game logic:repeatwith anuntilon a status field (the battle music stopped), with a timeout.measurearound steps. Today it is a fixed window that runs no steps, so frame times around a fight cannot be taken.visualprobe on the pictures: black, frozen, alternating.expect. Recordings drift (Spec, review note 2), so playback should stop where it left the route.What the game side needs
Already in
main's control bridge (control.md): teleport (31 destinations), age, status (map, position, life, prompt, music), press, stick, sequence, waits, queued pictures. Missing: a time-of-day command (today only a launch-time input file, which plans may not set), enemy spawning (in progress elsewhere, see this map), a coordinate teleport. Items and flags need guest memory writes, which plans should keep refusing; the first set does not need them.Privacy
No game file leaves any PC. A scenario holds teleport names, button scripts and settings, and starts from a new game or a repository save (today
kokiri-start), never a tester's own. Pictures stay local unless a plan saysupload. Our maps and routes stay private.A first small set
shotagainstmaincatches corruptionmainlacks; our detectors would not.until,measurearound steps, time of day and a repository save where Link has a weapon.Each scenario is two game starts at about 5 minutes (test-plan.md, "Time"). Four with stand-in steps come to 52 of the 60 minutes a plan may take, at
runs = 1.Questions
assets/scenarios/in Longshot, reviewed like saves, acceptable, or would you keep scenarios in Beanstalk?maintoo; only a PR's crash fails a report today. Should these run on every renderer PR, or only when asked?start = "new"run on testers' PCs? The docs say it was written without running the game.Read from your docs and code at commit
a89e366; nothing here has run on the farm.Possibly related, for context:
Co-op scenes, as a follow-up to the first set above. Nothing here has run on the farm.
Why co-op needs the farm. Longshot's multiplayer (MagicBeanDev/Longshot#68) adds a thread and a UDP socket, a menu tab and extra world actors for the other players. It has run on one Linux PC with an RTX 4090 and one Windows laptop with Intel Arc. Two unlike machines (say Windows with AMD and Linux with NVIDIA) are exactly what has never run, and exactly what the farm has.
What blocks it today (README,
docs/architecture.mdanddocs/test-plan.md):roleis refused as unknown.main, which has no co-op. A rule against the baseline means "what co-op costs"; whether a pair worked must be an absolute rule (.max,log_forbidden_hits.max = 0,crash = false).A way in, smallest first.
HOS_COOP_HOST), makes a room at start (HOS_COOP_AUTO=create) and plays alone in it. It checks that the co-op thread starts and ends without a crash or a hang on every GPU and driver, and what co-op costs a frame, with the three switches set on the PR side only ([matrix.pr_only]). It needs loopback to be up inside the sandbox on Linux; whether it is on Windows we do not know.--userfolder) under one lease. No network rule changes.HOS_COOP,HOS_COOP_AUTO,HOS_COOP_ROOM,HOS_COOP_SERVER,HOS_COOP_ACCEPT(a test-only shortcut for consent that works only from the environment).The scenes, and what each machine checks. A is the maker, B the joiner. Every check is a line the game already writes (the real lines are quoted in
log_matchregexes), a probe the farm already has, or the bridge'sstatus. None needs a model.coop-alonecreated (member 1)logged; norelay lost,no room:or socket error; no crash or hang; p99 frame time againstmain(cost of co-op on)pair-joinsync: room accepted; the other player's body logged as made (puppet: member 2 is shown as CoopPuppet_) within 60 s of B's join; norelay lostjoined room ... as member 2; first answer from the relay (the relay answered in N ms) logged; body of member 1 logged as made; nouses an older co-op versionpair-walkframetimeandhitch_count; the once-a-minutecoop: stats:line showsevents given up 0andoutbox full 0pair-map-changeload_timefor each teleportpair-rejoinrelaunch): A logs the leave and the join, accepts again, the body returns within 60 s,events given upunchangedjoined roomtwice; accepted againpair-lossHOS_COOP_DROP=30(the game's own receive-side loss): bodies still logged as shown;reliable retriesabove 0 andevents given up 0What the farm cannot judge, and a person must (as
docs/test-plan.mdasks us to say): whether the other body moves smoothly, whether a name or arrow reads well, and chat (its text is never written to the log). Go to friend is left out until there is a step that presses the tab's row; nothing in the bridge does.Pairing hardware. A pair target would name two tag sets, for example
windows+amdwithlinux+nvidia, andwindows+intelwithwindows+nvidia: the Windows pairs also check that the game's UDP socket works at all under the sandbox.Questions. (1) Would the maintainers consider one farm relay and a UDP allowance for the sandbox limited to it, or should co-op stay on one tester (levels A and B)? (2) Is loopback up inside the Linux sandbox and the two Windows modes? (3) Is a second game process on one tester acceptable for memory on the smallest GPUs?
Level A as a plan (parses with
plan.parse_planat4ec688b, estimate 23 minutes of the 60 allowed):Not tried: that
HOS_COOP_AUTO=createhosts the relay whenHOS_COOP_HOSTis set (read from the game's code, not run), that loopback works in the sandbox, and whether the 10 percent rule is the right size for a thread that sleeps.