This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Longshot Test Farm — Distributed PR Testing Spec

Nov 12, 2026 · @Someone · updated Nov 14, 2026 with what was built

Built as Beanstalk and running at beanstalk.magicbean.dev. Implementation (Nov 14, 2026) lists where the build differs from this spec, and what is left.

Summary

The Test Farm lets contributors donate their machines to test Longshot PRs: a central coordinator watches Forgejo, a maintainer approves a specific commit, and volunteer agents pull matching jobs, build and run that commit in a sandbox, and post a report back to the PR. The core security rule is that no code reaches a volunteer's machine unless a maintainer approved that exact commit SHA, and a PR's test plan is data the agent interprets, never a script it executes.

Goals

• Turn a 20–40 minute build-and-test cycle into something a PR author can queue and walk away from.
• Cover hardware nobody on the core team owns (e.g. AMD on Linux, Intel Arc on Windows) using volunteers' machines.
• Let each PR say what to measure (VRAM, frame times, crashes, visual diffs) and compare it against main on the same machine.
• Keep volunteers safe: approval gates, signed jobs, sandboxing, and a local policy they control.
• Stay outside Forgejo's code, talking to it only through webhooks and the API, so it survives Forgejo upgrades.

Non-goals

• Distributing game dumps or recompiled game output. Every tester supplies their own dump; only the builder (our code) is ever shared.
• Replacing normal CI (lint, unit tests, builder compile checks). Those stay in Forgejo Actions.
• Running PRs from unknown authors automatically, ever.

Threat model

Testing a PR means running its native code on someone's GPU, so the design assumes PR code is hostile until a maintainer says otherwise, and limits the damage if that judgement is wrong. The builder build, the recompile step and the game binary all run PR code, so all three are sandboxed, not just the test step.

Threat	Example	Mitigation
Unknown contributor submits malware	New account opens a PR whose CMake step exfiltrates ~/.ssh	Nothing runs until a maintainer approves the commit SHA; untrusted PRs never auto-run
Bait and switch	PR is approved, then the author force-pushes a malicious commit	Approval is pinned to one SHA; any new push resets it to "awaiting approval"
Compromised trusted account	A trusted contributor's token is stolen and used to push	Trusted-author auto-runs go only to agents that opted in; sensitive paths need two approvers; sandbox limits blast radius
Compromised coordinator	Attacker gets the coordinator server and tries to dispatch jobs directly	Agents verify maintainer signatures on each approval using keys pinned in the agent release (phase 3)
Malicious test plan	PR's test file tries rm -rf or opens a socket	Test plans are declarative TOML; custom logic runs in a capability-less WASM/Luau VM
Sandbox escape	Exploit in the GPU driver from inside the container	Residual risk, stated plainly to volunteers; offer "reviewed-only" policy and dedicated OS user
Fake results	A tester uploads fabricated "pass" reports	Signed reports tied to a Forgejo account, multiple reports per target, outlier detection, output-hash cross-checks
Game data leaks	Recompiled output or dumps shared via the farm	Only builder binaries are distributed; dumps are mounted read-only and never uploaded; screenshots are opt-in
Tester privacy	Logs reveal usernames, paths, serial numbers	Agent scrubs paths and hostnames; hardware is reported at model/driver level only

Architecture

Five parts, with one rule for the network: agents only make outbound HTTPS calls, so nothing can push to a volunteer's machine and nobody needs to open a port.

[embedded content: Test Farm components · 6 parts, every agent arrow points outward]

Every arrow leaving the volunteer machine is a request the agent starts; the coordinator can only answer, never push.

Component	Runs on	Job
Forgejo (git.magicbean.dev)	Project server	Source of truth for PRs, teams, labels; sends webhooks; receives commit statuses and comments
Coordinator	Project server, separate service	Webhook receiver, approval engine, job queue (Postgres), scheduler, report store, PR reporter, small web dashboard
Trusted builder	Project-owned Forgejo Actions runner, fresh VM per job	Builds the builder for each approved SHA, hashes and signs it, uploads to the artifact store
Artifact store	Forgejo package registry or S3/MinIO	Holds signed builder binaries and input-replay files; never game output
Farm agent	Volunteer's machine	Single binary (Rust suggested) with CLI and optional tray UI; polls for jobs, verifies signatures, runs recompile and tests in a sandbox, uploads a signed report

Why not just register volunteers as Forgejo Actions runners: a workflow file is code the PR can edit, so anyone who gets a workflow to run gets a shell on the volunteer's machine. Actions stays for trusted, project-owned work only.

Why build the builder centrally: it is CPU-only and identical for every tester, so building it once per SHA saves every volunteer that part of the 20–40 minutes. Each agent then only does the recompile and shader compile, which depend on its own dump, GPU and driver. If the project has no spare build server, the agent falls back to building the builder itself inside the sandbox.

Trust and approval

Every job carries a trust tier, set by the coordinator from who wrote the PR and who approved which SHA. Volunteers choose the lowest tier their agent will accept; the default is Reviewed.

Tier	How a commit gets it	Default agent policy
Untrusted	Any PR from an author outside the farm-trusted team, before approval	Never runs, on any agent
Auto	Author is in farm-trusted; no human looked at this SHA	Off (volunteer must opt in)
Reviewed	A member of farm-maintainers approved this exact SHA	On
Double-reviewed	Two maintainers approved; required when the diff touches build scripts, CMake, .farm/, toolchain files or the agent itself	On

How approval works

1. PR opens or gets a new push. Coordinator sets commit status farm/approval to pending with the text "awaiting maintainer approval".
2. A maintainer reviews the diff and comments /farm approve (optionally /farm approve linux-nvidia to limit targets). The coordinator checks the commenter's team membership through the Forgejo API, then records approver + SHA + time.
3. The coordinator labels the PR farm: approved @ abc1234 for visibility, but the database record is the source of truth: labels can be added by anyone with triage rights.
4. Any new push creates a new SHA with no approval. Queued jobs for the old SHA are cancelled.
5. /farm revoke cancels queued and running jobs for that PR; agents abort within one heartbeat.

[embedded content: approval flow · 4 tiers, 2 decisions]

Every push starts at the left again; an untrusted author's PR sits in "Awaiting approval" until a maintainer approves that exact SHA.

Other commands: /farm run (re-run), /farm targets (list which hardware has reported), /farm cancel.

Signed approvals (phase 3). Today the agent trusts the coordinator's signature on a job. Later, the maintainer's own key signs (repo, PR, SHA, tier, targets, expiry) — for example with ssh-keygen -Y sign from a CLI helper — and agents check it against maintainer public keys pinned in the agent release. Then even a fully compromised coordinator can't send unapproved code to volunteers.

Per-PR test plan

Each PR can ship .farm/test.toml, a declarative test plan. It picks hardware targets, scenarios and built-in probes; it can't run shell commands. A PR without one gets the default plan from main (boot, run the standard scenario, check crashes, frame times and VRAM).

schema = 1
summary = "Texture cache rewrite: check VRAM and frame pacing"

# Which hardware should report, and how many independent reports each
[[target]]
os = "linux"
gpu = "nvidia"
min_reports = 2

[[target]]
os = "windows"
gpu = "amd"
min_reports = 1

# Run main's build in the same session for an A/B comparison
[baseline]
ref = "main"

[[scenario]]
name = "title-to-hub"
input = "inputs/title_to_hub.replay"   # recorded controller input, data only
duration_s = 120
warmup_s = 10

[[probe]]
kind = "frametime"          # p50 / p99 / max, stutter count

[[probe]]
kind = "vram"
sample_ms = 500

[[probe]]
kind = "screenshot"
at_s = [30, 90]
compare = "baseline"        # perceptual diff vs main
upload = false              # stays local unless the tester opts in

# Pass/fail rules, relative to baseline on the same machine
[pass]
crash = false
frametime_p99_ms.max_delta_pct = 10
vram_peak_mb.max_delta_pct = 5
screenshot.max_diff = 0.02

Built-in probes (shipped in the agent, not the PR): crash (exit code, fatal log lines, hangs), frametime, vram, cpu (per-thread time), shader_compile (count and time, cold vs cached), screenshot, log_match (regex expected/forbidden), boot_time.

Custom logic. When a probe isn't enough, a PR can add kind = "script" pointing at a WASM module or Luau script. It runs in a VM with no filesystem or network access; it can only call agent APIs such as read_metric, send_input, screenshot and assert. New probes are added to the agent in normal PRs, so they get reviewed once rather than per test.

Runtime hooks needed in Longshot. The probes work best if the runtime exposes a small metrics channel (frame start/end timestamps, VRAM allocations, shader compile events) and an input-replay mode. Adding these to the recomp runtime is a prerequisite for phase 1.

Farm agent

The agent is the only trusted code on a volunteer's machine. It's released from main as signed binaries, updates only to releases signed with the project's pinned key, and never runs anything from a PR outside its sandbox.

Setup. farm-agent login signs in through Forgejo OAuth2 and gets an agent token scoped to that account. farm-agent setup finds the game dump (hash only is reported), detects hardware, and writes a local policy file.

Capabilities reported: OS and version, GPU vendor/model/VRAM, driver version, Vulkan/D3D12 versions, CPU model and core count, RAM, free disk, game dump hash. The coordinator matches these against each plan's [[target]] entries.

Local policy (volunteer controls it):

min_tier = "reviewed"        # untrusted is never allowed
allowed_authors = []          # optional extra allowlist
run_when = "idle"             # idle | always | schedule
schedule = "Mon-Fri 22:00-07:00"
max_jobs_per_day = 4
disk_budget_gb = 60
upload_screenshots = false

Job lifecycle on the agent

1. Long-poll the coordinator for offers matching its capabilities.
2. Drop any offer that fails local policy; accept one and take a lease (heartbeat every 60 s, lease lost after a missed window so the job can be reassigned).
3. Verify the job ticket signature, then fetch source at the exact SHA and the builder artifact, checking both hashes against the ticket.
4. Start the sandbox and run recompile + shader compile. Caches are keyed by builder hash, GPU and driver so re-runs and baselines are fast.
5. Run the baseline build and the PR build back to back, same scenarios, same session, alternating order across runs to cancel thermal drift.
6. Tear down the sandbox, collect metrics, sign the report with the agent key and upload it.

Sandbox, Linux. Rootless Podman or bubblewrap: no network namespace, read-only mounts for the dump and builder, one writable work directory, GPU via /dev/dri (AMD/Intel) or NVIDIA CDI devices. Don't pass the host X11 socket — it lets any client log keystrokes and read the screen. Render headless instead: either an offscreen mode in the Longshot runtime, or nested gamescope --backend headless.

Sandbox, Windows. Harder. Baseline: run under a dedicated low-privilege local user with an AppContainer (no capabilities, so no network) and a Job Object for resource limits, with ACLs that only allow the work directory. Windows Sandbox with vGPU is an optional stricter mode for Pro/Enterprise, at some performance cost. Whether Vulkan and D3D12 drivers behave inside an AppContainer needs a spike before committing.

What the sandbox can't fix. GPU drivers are a large attack surface and a driver exploit can escape any of these. The agent's onboarding says this plainly, and suggests running under a separate OS user and keeping min_tier = "reviewed".

Reports and PR feedback

Results land on the PR in two places: one commit status per target (e.g. farm/linux-nvidia: 2/2 pass), so branch protection can require them, and one sticky comment that the coordinator edits as reports arrive instead of posting new ones.

Example sticky comment:

Target	Reports	Crash	Frame p99 (ms) vs main	VRAM peak (MB) vs main	Result
linux-nvidia RTX 3070, 565.x	2/2	none	14.1 vs 14.6 (−3%)	3,880 vs 4,410 (−12%)	pass
windows-amd RX 6700 XT, 24.x	0/1	—	—	—	queued

(Illustrative numbers.) Each row links to the full report on the dashboard.

Report contents (JSON, versioned schema): job id, SHA, builder hash, recompiled-output hash, agent version, coarse hardware/driver, scenario metrics for PR and baseline, probe results, pass/fail per rule, scrubbed logs, optional screenshots, agent signature.

Trusting results.

• Comparisons are always against main on the same machine in the same session, which removes most hardware noise.
• min_reports per target gives independent confirmation; the coordinator flags a report that disagrees sharply with its peers.
• If two testers with the same builder hash and dump hash get different recompiled-output hashes, the build is non-deterministic or someone is lying — either way it's flagged.
• Each tester builds a reputation from reports that agree with peers; low-reputation reports are shown but don't count toward min_reports.

Caching and scheduling

Most of the 20–40 minutes is rebuilding things that didn't change, so caching is the biggest speed win after parallelism.

The farm reuses Longshot's own caches instead of adding new ones (source: Longshot README). Each tester keeps a warm main build; a PR builds in a copy of it, so only files the PR changed compile.

Cache	Size	Farm rule
Translated C and object files, kept with LONGSHOT_KEEP_BUILD_FILES=1	about 4 GB	main tree is the base; each PR builds in a copy (reflink or hard link where the filesystem allows)
Shader program cache longshot_programs.bin	about 6.9 GB, 1.83 GB with RX_VK_PROGRAM_CACHE_ZSTD=1	PR run starts from a copy of main's; warmed with --prepare-shaders before timing starts
Builder tool cache (~/.cache/longshot-builder, %LOCALAPPDATA%\LongshotBuilder)	—	Shared; written only by main builds
CPM cache (RX_CPM: Vulkan headers, FFmpeg)	—	Shared, read-only to PR builds
Baseline results for main	small	Reused until main moves

Cache safety rules.

• Only a main build writes to a shared cache. A PR build writes to its own copy, which is deleted after the report, so one PR can't poison the next run.
• A PR that touches nx2/ (lifter, build scripts) or shader translation gets a cold build and a cold program cache; the report says so.
• Every report records whether the build and the program cache were warm or cold, so first-run stutter isn't read as a regression.
• Answered: the incremental build decides staleness by content hashes (sha1 of source, flags and headers in build.py), not timestamps. The program cache has a version key (translator stamp plus format, vk_progcache.cpp) but no driver identity. The lift key holds absolute paths, so a copied tree re-lifts. See Review notes, item 4.

Scheduling rules.

• A new push supersedes and cancels queued jobs for the same PR.
• Jobs are offered to agents whose hardware is rarest first, so a lone AMD-on-Linux machine isn't busy with jobs an NVIDIA box could take.
• A PR label farm: priority jumps the queue; maintainers only.
• Fairness: no single PR author holds more than a set share of the farm at once.
• An agent's disk budget triggers LRU eviction of old recompiled outputs and shader caches.

Rollout

Ship something useful at each phase; don't open the farm to outside volunteers until signed approvals and both sandboxes exist.

1. Phase 0 — Local runner. Runtime metrics channel and input replay in Longshot; .farm/test.toml schema; farm-agent run --pr 123 runs a plan locally against a PR the user chose. No coordinator. Exit: maintainers use it for real PRs and trust the numbers.
2. Phase 1 — Coordinator, maintainers only. Webhooks, /farm approve, SHA pinning, queue and leases, commit statuses, Linux sandbox. Agents run by the core team. Exit: a PR gets results from two machines without anyone babysitting it.
3. Phase 2 — Wider hardware. Windows sandbox, central builder artifacts, baseline A/B, sticky comment, dashboard, caching. Invite known community testers. Exit: Linux and Windows reports agree with manual testing on a set of past PRs.
4. Phase 3 — Open volunteering. Maintainer-signed approvals with pinned keys, reputation, outlier detection, double review on sensitive paths, public onboarding. Exit: a security review of the agent and sandbox by someone outside the core team.

Implementation (Nov 14, 2026)

The farm is called Beanstalk. Its code is in MagicBeanDev/Beanstalk: the coordinator in coordinator/, the agent in agent/. The contract between them is in docs/:

• protocol.md: the API, the signed envelopes and the job ticket
• test-plan.md: the .farm/test.toml schema
• report-schema.md: what a report holds
• deploy.md: deployment

Where it differs from the spec above

Spec	Built	Why
Agent in Rust	A .NET 10 + Avalonia desktop app with a CLI (beanstalk-agent), built like the LongshotLauncher. It reuses the launcher's signed-release check, its Vulkan probe (extended with the GPU model, the driver version and the driver properties), its atomic files and its self-update.	It shares code, packaging and the release flow with the launcher.
Coordinator with no language set	Python 3.13 (Starlette, uvicorn, Jinja2, psycopg 3) with Postgres, hash-pinned. It is one container plus a private Postgres on the magicbean stack.	Postgres as the spec says. It is hardened like the other magicbean services.
Central trusted builder and artifact store	Not built. The agent builds with Longshot's scripts/dev_packs.py: a warm main base, the PR as a pack, a runtime for each side.	Review note 1: almost all the time depends on the tester's own dump, CPU and GPU.
Input replay files	Scenarios are control-bridge steps: wait, press, stick, sequence, teleport, age, option, measure. start = "title" boots to the title screen. start = "save:<name>" uses a scrubbed repo save from Longshot's assets/saves/; start = "new" makes a fresh save. A tester's own save (start = "save") still works but is never uploaded and the planner doesn't pick it.	Review notes 2 and 3 (and the saves exception).
No network inside the sandbox	The agent fetches the toolchains outside the sandbox first, with builder/linux/linux_tools.py env --install from main checked against tools.lock. Then everything runs offline. A PR that changes the Linux lines of tools.lock gets error. The build's Python is a venv built from main's requirements.txt, outside the sandbox.	Review note 5.
disk_budget_gb = 60	The default is 120.	Review note 6.
Output-hash cross-checks	Dropped.	The open question below: builds aren't reproducible across machines.
farm-maintainers, farm-trusted	Configurable lists of teams. Production: maintainers = Maintainers, BigBeans, auto tier = BigBeans.	The teams already exist.
Signatures	ECDSA P-256 envelopes, signed over the exact payload bytes, so no JSON canonicalization is needed. The coordinator's ticket key (id 69ad0351614821cb) is pinned in the agent build. Each agent makes its own key and signs its reports with it.	The same scheme as the launcher's release signatures.
Windows sandbox	Not yet. A Windows agent reports sandbox: "none" and is never offered farm jobs. It can only test a PR its owner picks with run --pr N.	The AppContainer spike is still open.
cpu, shader_compile and script probes	cpu and shader_compile report unsupported. The coordinator refuses script with "not supported yet".	Left for later PRs to the agent.

Behaviour worth knowing

• The farm stays quiet on PRs that don't use it. Every push gets the farm/approval status. The per-target statuses and the sticky comment only appear once there is an approval, a job, a report or a refused plan.
• Approvals: authors can't approve their own PR. /farm approve can take a SHA prefix, which must be the current head, and a comment written before the latest push doesn't count. A sensitive diff stays at one approval ("needs a second maintainer") until a second maintainer approves.
• Scheduling: an agent never fills two report slots for the same commit and target. Authors at their share cap are offered last, but the cap never leaves an agent idle.
• Leases and retries: a lease is lost after 3 missed heartbeats. A job goes to error after 3 failed attempts on different agents.
• Outliers: a report is flagged when its PR-versus-main change is more than 25 points from its peers' median, or more than 3× MAD once there are 3 or more peers.

Deployment

• The coordinator runs on the Unraid stack as magicbean-beanstalk and magicbean-beanstalk-db, reached through the Cloudflare tunnel.
• The Forgejo account farm-bot posts the statuses, the comments and the labels. Its team, FarmBot, has write access on Longshot. It is also in Maintainers and BigBeans, because Forgejo hides team membership from non-members. The coordinator ignores /farm commands that farm-bot writes itself.
• The Longshot webhook goes to https://beanstalk.magicbean.dev/webhook and is checked with HMAC. Sending it over the internal network instead would need magicbean-beanstalk in Forgejo's webhook.ALLOWED_HOST_LIST, which takes a Forgejo restart.

Found while testing it for real

• Unix socket paths are limited to 107 bytes. A run folder's control socket path was longer, so the game switched its bridge off. Sockets now go in a short $XDG_RUNTIME_DIR/beanstalk-<random>/.
• bwrap's --die-with-parent follows the thread that started bwrap. Started from a .NET thread-pool thread, a build was killed when .NET retired that thread. The agent now starts every process from one long-lived thread.
• Each game start in a fresh runtime spends about 5 minutes creating pipelines before the first frame, so one A/B run with 2 scenarios × 2 runs per side takes about 40 minutes.

Status by phase

• Phase 0: beanstalk-agent run --pr N works locally. The first full run on real hardware is in progress.
• Phase 1: done (webhooks, /farm commands, SHA pinning, queue and leases, statuses, the Linux sandbox). No agents are registered yet.
• Phase 2: done except the Windows sandbox and central builds (A/B baseline, sticky comment, dashboard, caching, rarest-hardware-first scheduling).
• Phase 3: not started (maintainer-signed approvals, reputation), apart from outlier flags and double review on sensitive paths.

Open questions

Answers below come from reading the Longshot and LongshotLauncher code on Nov 13, 2026. File references are to Longshot unless they say otherwise.

• How does the build time split between translating (lift), compiling, linking and first-start shader preparation, cold and warm? A cold player build of nightly-56 (Ryzen 9 7900X, RTX 3090, Linux) took about 14 minutes: read 64–89 s, lift ~57 s, compile and link ~5 min (274 files, -O1), finish ~66 s, shader preparation (Most) ~5 min (docs/development.md, "Build times"). Shader preparation took 982 s on an RX 6700 XT (docs/tested-configurations.md).

  • The launcher's "about 45 / 37 minutes" was a hard-coded guess from the 2.x Builder, not a measurement. LongshotLauncher PR 28 replaced it with per-stage forecasts based on the figures above.
  • make_game already records this. It prints @@stage / @@times lines and a "where the time went" table, and it keeps the last 20 builds' per-file and per-phase times in <work>/build_times.json. The farm can read stage times from that file.
  • Still open: a measured warm (incremental) build time.

• Does the project have a server that can build the builder for each approved SHA? Not needed: the farm builds on each agent from source (see Implementation). The existing build-linux.yml / build-windows.yml workflows already build the AppImage and the Windows builder on the project's Linux Docker runners, with no game data, and can be called from another workflow. The runners have no Windows host and no GPU. See Review notes: a per-SHA builder saves less time than the spec assumes.

• Do Vulkan/D3D12 drivers work inside a Windows AppContainer, and how much does Windows Sandbox vGPU cost? Not answered by the code: nothing in either repo sandboxes make_game or the game today. Still needs the spike.

• Can the runtime render fully offscreen without skewing frame times? Half answered. HOS_HEADLESS=1 already runs with no window and draws to an offscreen image (host/display.c, gpu/vk/vk_main.cpp), and RX_VK_FRAME_LOG works headless. Whether headless frame times match windowed ones is unmeasured. There is no swapchain or present when headless, so pacing likely differs.

• Is the build deterministic enough for output-hash cross-checks between testers? Not as it stands. Several things make outputs differ between machines:

  • nx2/version.py embeds a source fingerprint and git describe (including -dirty).
  • RX_MARCH defaults to the host CPU's level.
  • The hot-piece split depends on the work folder's history (make_game.py: "two work folders with other histories may lift the same game into other pieces").
  • A play profile puts an absolute path into the hot pieces' flags.

  Cross-checks would need all of these pinned. Proposal: drop output-hash cross-checks, or limit them to a pinned "reference" build mode.

• Which game versions must a tester have; does the plan need a game_version filter? Only one version is supported: program 0400486024DEC000, update v131072, build ID e0f3ccfa (README). make_game.py --check already enforces it (check_supported_build compares the NSO module ID). No filter is needed yet. The agent should report the build ID rather than hash the whole dump.

• Who sits in farm-maintainers and farm-trusted? Maintainers and BigBeans approve; BigBeans' PRs get the auto tier (Nov 14, 2026). Earlier note: not a code question. Precedent: scripts/test_requests.py only lets a request be claimed when the PR author may push to Longshot or a maintainer added the testing wanted label.

• Are screenshots of game content OK to store on the dashboard? Existing policy allows it: CONTRIBUTING.md says "A screenshot, a video and longshot_log.txt are fine" to share. Keep upload = false as the default so testers opt in.

Review notes (Nov 13, 2026)

These concerns come from checking the spec against the Longshot and LongshotLauncher code.

1. The central builder saves little. The builder is already downloaded prebuilt and signed (SHA256SUMS + SHA256SUMS.sig). Almost all of a build is the lift, compile and shader preparation, which depend on the tester's dump, CPU and GPU. An agent also runs make_game from the PR's own source anyway. The per-SHA builder and artifact store can probably leave phases 1–2.

2. Most of the phase 0 runtime work already exists.

   • Control bridge: HOS_CONTROL (docs/control.md, scripts/longshot_ctl.py) can press buttons, wait for a state or map, teleport, take shots, read status (fps, draws), and launch headless with a save slot.
   • Timed input: HOS_INPUT_SCRIPT, combined with HOS_INPUT_FRAME_CLOCK, replays timed input against game frames.
   • Metrics: RX_VK_FRAME_LOG and HOS_PRESENT_TRACE write frame times. The periodic vk: VRAM: and vk: N presents, F fps log lines report memory and fps. RX_VK_PASS_TIMING gives GPU time per pass.
   • Frame cap: HOS_FPS_MAX=1000 and RX_VK_IMMEDIATE=1 uncap the frame rate.

   Scenarios should be control-bridge scripts (wait for state, then teleport, then measure) rather than raw input recordings, which drift.

3. Scenarios can't ship save files. Saves are made from the game, so they fall under the rule in CONTRIBUTING.md. A scenario must start from the title screen or use the debug teleport.

   • Update, Nov 14, 2026: the team proposes a narrow exception. Scrubbed saves would live in Longshot's assets/saves/<name>/, one slot each with only progress.sav and caption.sav.
     • scripts/longshot_saves.py scrub replaces the picture with plain grey and resets the save date to 2026-01-01 00:00.
     • check-repo-saves enforces it.
     • Each save is made fresh for the farm and added through a pull request two maintainers approve.
   • These facts were checked with a real game on the AMD lab PC:
     • the game shows no player name;
     • a slot loads with no account.sav or option.sav, which the game then makes itself, so those are never shipped;
     • the Load screen's date comes from caption.sav's header, not the file times.
   • Plans use these saves with start = "save:<name>", and longshot_ctl.py launch --repo-save <name> loads them by hand. start = "new" (a fresh blank save the agent makes) covers the rest.

4. Use scripts/dev_packs.py rather than copying build trees. It already does "warm main plus a PR on top" (base, pack, rt) and gives A/B runtimes (docs/development.md, "Testing a change against a base").

   • Copying a tree is worse than the spec expects. The lift key holds absolute paths (work, out and dump folders), so a copy re-lifts, re-runs the shader census and recompiles the hot pieces when a profile is used.
   • Object staleness uses content hashes (build.py: sha1 of source, flags and headers), so the object cache is safe to share. A PR that touches nx2/ doesn't need a forced cold build. Only the "PR builds never write to shared caches" rule is needed against poisoning.
   • The program cache is keyed by a translator stamp (vk_progcache.cpp, translator_stamp()) with no driver identity. The Vulkan pipeline cache does check vendor, device and pipelineCacheUUID.

5. "No network namespace" conflicts with the build. make_game's first stage downloads toolchains, and DLSS/FSR fetch SDKs. The agent has to pre-fetch everything outside the sandbox, checked against tools.lock hashes, before starting the offline sandbox.

6. The disk budget is too small. The launcher reserves 45 GB for one build. A base, a PR pack, two runtimes and program caches of up to 6.9 GB each won't fit in the default disk_budget_gb = 60.

7. The CI wording is off. Only the PR-title check runs automatically. tests/run_all.py is workflow_dispatch only ("off for now"), and there is no Windows or GPU runner (docs/forgejo-runners.md). Saying "lint and unit tests stay in Forgejo Actions" assumes CI that doesn't run yet.

8. It overlaps the test-request system. scripts/test_requests.py (docs/test-requests.md) already has approval gating, claims with leases, and results posted as PR comments, through Matrix and Discord. The spec should say whether the farm is phase 1 of that system or replaces it.

9. Reusable parts:

   • Signed updates: the launcher's signed-release check (ECDSA P-256 over SHA256SUMS, pinned key, Core/ReleaseTrust.cs) fits the agent's own updates.
   • Sign-in: scripts/forge.py login already does Forgejo OAuth.
   • Hardware detection: the launcher's Vulkan probe (Core/SystemRequirements.cs) reads VRAM and API version but not the GPU model or driver version, so it needs extending.

10. "20–40 minutes" is too high for fast machines. It is about 14 minutes cold on a high-end PC. Slower machines and shader preparation on some GPUs still take much longer.