feat: plan schema 9, a scenario's cache: driver cold, driver warm or cold starts #99

Merged
HumanBean merged 2 commits from HumanBean/Beanstalk:feat/cache-states into main 2026-11-19 07:25:30 +00:00
Owner

A PR about a first start needs to know what the start's caches hold, and a plan couldn't say. Plans for Longshot #1701 (the ASTC converters on a first start), #1450 (a second start reusing the first's cache) and #426 (an empty shader cache) each said so.

Stacked on #98 (schema 8). Its commit is in this branch until #98 merges; merge #98 first.

What changed

A scenario's cache (plan schema 9):

cache The driver's shader cache The game's cache/
as_is (default) as before as before
driver_cold empty kept
driver_warm the side's own, kept across its starts; one unmeasured start of each side fills it first kept
cold empty moved aside for the start and put back after
  • No sharing between sides: the driver's cache follows the run's home (XDG_CACHE_HOME), so main and the PR never share one. driver_warm keeps one home per scenario, launch group, side and build variant (runs/homes/...).
  • Checking the cache really was empty: runs report driver_cache_start_mb, read from the game's vk: driver shader cache at the start: N MB line. A driver_cold or cold run that found a cache is an error.
  • Linux only for now: on Windows the drivers keep their caches in the sandbox account's profile, outside the run's folder, so a Windows agent reports these scenarios as skipped. The planner is told to target linux for them.
  • Time estimate: it counts driver_warm's extra start per side, and about 10 more minutes for each cold start.
  • Who gets these plans: they normalize to schema 9 and go only to agents of plan schema 9. The agent and the coordinator check them alike. The planner prompt and the docs (test-plan.md, protocol.md) describe it.
[[scenario]]
name = "first-start"
start = "save:kokiri-start"
runs = 1
cache = "driver_cold"
step = [...]

No screenshots: nothing on screen changes beyond the plan's words.

Testing

  • python -I agent/tests/test_agent_core.py --strict: 2157 checks passed. New fixtures (cache-states.toml, cache-cold.toml, cache-as-is.toml, bad-cache-schema.toml, bad-cache-value.toml) are cross-checked against the coordinator's reader. New runner checks use a capturing sandbox:
    • a cold start finds no game cache, and the cache is put back afterwards;
    • driver_warm's first start is the unmeasured one, in the side's kept home;
    • driver_cold gets a new home;
    • a cache left aside by a stopped run is restored;
    • the driver-cache line is parsed.
  • pytest -q in coordinator/: 589 passed, 1 skipped. Two test_public_pulls.py tests failed once on timing and passed when run again; they aren't touched here.
  • Not tested: a real farm job, in particular a full cold start's pipeline preparation time and a driver_warm start actually hitting the kept cache.

🤖 Generated with Claude Code

A PR about a first start needs to know what the start's caches hold, and a plan couldn't say. Plans for [Longshot #1701](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1701) (the ASTC converters on a first start), [#1450](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1450) (a second start reusing the first's cache) and [#426](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/426) (an empty shader cache) each said so. **Stacked on #98** (schema 8). Its commit is in this branch until #98 merges; merge #98 first. ## What changed A scenario's `cache` (plan schema 9): | `cache` | The driver's shader cache | The game's `cache/` | | --- | --- | --- | | `as_is` (default) | as before | as before | | `driver_cold` | empty | kept | | `driver_warm` | the side's own, kept across its starts; one unmeasured start of each side fills it first | kept | | `cold` | empty | moved aside for the start and put back after | - **No sharing between sides:** the driver's cache follows the run's home (`XDG_CACHE_HOME`), so main and the PR never share one. `driver_warm` keeps one home per scenario, launch group, side and build variant (`runs/homes/...`). - **Checking the cache really was empty:** runs report `driver_cache_start_mb`, read from the game's `vk: driver shader cache at the start: N MB` line. A `driver_cold` or `cold` run that found a cache is an error. - **Linux only for now:** on Windows the drivers keep their caches in the sandbox account's profile, outside the run's folder, so a Windows agent reports these scenarios as `skipped`. The planner is told to target `linux` for them. - **Time estimate:** it counts `driver_warm`'s extra start per side, and about 10 more minutes for each `cold` start. - **Who gets these plans:** they normalize to schema 9 and go only to agents of plan schema 9. The agent and the coordinator check them alike. The planner prompt and the docs (`test-plan.md`, `protocol.md`) describe it. ```toml [[scenario]] name = "first-start" start = "save:kokiri-start" runs = 1 cache = "driver_cold" step = [...] ``` No screenshots: nothing on screen changes beyond the plan's words. ## Testing - `python -I agent/tests/test_agent_core.py --strict`: 2157 checks passed. New fixtures (`cache-states.toml`, `cache-cold.toml`, `cache-as-is.toml`, `bad-cache-schema.toml`, `bad-cache-value.toml`) are cross-checked against the coordinator's reader. New runner checks use a capturing sandbox: - a `cold` start finds no game cache, and the cache is put back afterwards; - `driver_warm`'s first start is the unmeasured one, in the side's kept home; - `driver_cold` gets a new home; - a cache left aside by a stopped run is restored; - the driver-cache line is parsed. - `pytest -q` in `coordinator/`: 589 passed, 1 skipped. Two `test_public_pulls.py` tests failed once on timing and passed when run again; they aren't touched here. - **Not tested:** a real farm job, in particular a full `cold` start's pipeline preparation time and a `driver_warm` start actually hitting the kept cache. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat: plan schema 8, [[pass.compare]] judges a matrix's cells against each other
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
e62fbc6f85
A Longshot PR that adds a switch or a setting often needs two
judgements: the PR against main, and the switch on against off. The
farm's rules could only do the first (or only the second, with
baseline.ref = "pr"), so plans for #994, #1478 and #1367 said the two
configurations could not be compared.

[[pass.compare]] (schema 8, up to 8 rules) takes a metric (optionally
one segment's), a matrix axis, the judged value and the reference value,
a side (pr by default, or baseline) and one of max_delta_pct or
min_delta_pct. For each scenario on the matrix and build variant, each
pair of cells that differ only in that axis is judged on that side's
medians; the result is the judged cell's, its baseline column the
reference cell's number. A skipped or mismatched cell of the pair skips
or mismatches the rule. A measure step can't be named compare now.

The planner is also told what was already true: .max and .min judge the
PR side only, so a log line only the PR prints is a log_match expected
pattern with log_expected_missing.max = 0 (plans for #475 and #1464
thought it could not be done).

Agent and coordinator check it alike (new fixtures, cross-checked); the
agent says plan schema 8.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
feat: plan schema 9, a scenario's cache: driver cold, driver warm or cold starts
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
7f5d248b1e
A PR about a first start (Longshot #1701's ASTC converters, #1450's
second start reusing the first's cache, #426's empty shader cache) needs
the start's caches to be known, and a plan could not say what they held.

A scenario's cache (schema 9) says it: as_is (the default, as before),
driver_cold (an empty driver shader cache), driver_warm (each side's own
driver cache, kept across its starts in the scenario and filled first by
one unmeasured start of each side) or cold (the driver's cache empty and
the runtime's cache/ moved aside for the start and put back after). The
driver's cache follows the run's home (XDG_CACHE_HOME), so main and the
PR never share one; driver_warm keeps a home per scenario, launch group,
side and build variant. Runs report driver_cache_start_mb from the
game's "vk: driver shader cache at the start" line, and a driver_cold or
cold run that found a cache is an error.

Linux only for now: on Windows the drivers keep their caches in the
sandbox account's profile, outside the run's folder, so a Windows agent
reports such scenarios skipped. The estimate counts driver_warm's extra
start a side and about 10 more minutes a cold start. Agent and
coordinator check it alike (new fixtures); the planner is told when to
use which; the agent says plan schema 9.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!99
No description provided.