feat: plan schema 7, log_value and whole-run frame metrics #97

Merged
HumanBean merged 1 commit from HumanBean/Beanstalk:feat/log-values into main 2026-11-19 06:41:25 +00:00
Owner

A Longshot PR that removes a one-time hitch, such as Longshot #1701 and the first ASTC texture's 316 ms frame on a start with an empty driver cache, couldn't be measured by the farm. The frametime probe counts only the measure steps' frames, after their warmup, and log_match can count lines but can't read the 316.5 out of vk: slow upload 316.5 ms.

What changed

Plan schema 7 adds two things:

  • Whole-run frame metrics. The frametime probe also reports run_frametime_max_ms and run_hitch_count. They cover every frame of a cell, from the start of its steps to their end (the bridge's frame numbers), loads and warmups included.
  • The log_value probe. It reads numbers the game logs: up to 8 values, each a regex with one capture group, taken as max, min, sum, first, last or count. none = N sets the value for when no line matches. Each value is the metric value_<name>, and [pass] judges it like any run metric. Patterns run without backtracking and with a time limit, as log_match's do.
[[probe]]
kind = "log_value"
  [[probe.value]]
  name = "slow_upload_ms"
  pattern = 'vk: slow upload ([0-9.]+) ms'
  none = 0
[pass]
value_slow_upload_ms.max = 50
run_frametime_max_ms.max = 100

A plan with a log_value probe or a rule on a run_* metric normalizes to schema 7 and goes only to agents of plan schema 7. The agent now reports 7, so these plans wait for testers to update. Any other plan normalizes as before. The agent (Plan.cs, Probes.cs, ScenarioRunner.cs) and the coordinator (plan.py) check plans alike. The planner's prompt, the plan's words and the docs (test-plan.md, report-schema.md, protocol.md, the agent's README) describe both.

Nothing changes on screen apart from new metric names in the plan's words and the Measurements table, so there are no screenshots.

Testing

  • python -I agent/tests/test_agent_core.py --strict: 2073 checks passed. New fixtures (log-value.toml, run-metrics-plain.toml and 10 bad-log-value-* and bad-run-metric-schema refusals) are cross-checked against the coordinator's reader. New probe checks cover every take, none, a line without a number, and a back-reference or catastrophic pattern. New rule checks use #1701's own numbers.
  • pytest -q in coordinator/: 580 passed, 1 skipped.
  • Not tested: a real farm job. The frame window (status at a cell's start and end) has not run against the game; on a cell that ends in a crash or timeout it runs to the end of the frame log.

Follow-up, not in this PR: a plan can't choose the driver cache state yet. A Linux run's sandbox home is new each run, so its driver cache is likely empty every time. On Windows the runner account's cache likely persists. That's the third step of the proposal.

🤖 Generated with Claude Code

A Longshot PR that removes a one-time hitch, such as [Longshot #1701](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1701) and the first ASTC texture's 316 ms frame on a start with an empty driver cache, couldn't be measured by the farm. The `frametime` probe counts only the measure steps' frames, after their warmup, and `log_match` can count lines but can't read the 316.5 out of `vk: slow upload 316.5 ms`. ## What changed Plan schema 7 adds two things: - **Whole-run frame metrics.** The `frametime` probe also reports `run_frametime_max_ms` and `run_hitch_count`. They cover every frame of a cell, from the start of its steps to their end (the bridge's frame numbers), loads and warmups included. - **The `log_value` probe.** It reads numbers the game logs: up to 8 values, each a regex with one capture group, taken as `max`, `min`, `sum`, `first`, `last` or `count`. `none = N` sets the value for when no line matches. Each value is the metric `value_<name>`, and `[pass]` judges it like any run metric. Patterns run without backtracking and with a time limit, as `log_match`'s do. ```toml [[probe]] kind = "log_value" [[probe.value]] name = "slow_upload_ms" pattern = 'vk: slow upload ([0-9.]+) ms' none = 0 [pass] value_slow_upload_ms.max = 50 run_frametime_max_ms.max = 100 ``` A plan with a `log_value` probe or a rule on a `run_*` metric normalizes to schema 7 and goes only to agents of plan schema 7. The agent now reports 7, so these plans wait for testers to update. Any other plan normalizes as before. The agent (`Plan.cs`, `Probes.cs`, `ScenarioRunner.cs`) and the coordinator (`plan.py`) check plans alike. The planner's prompt, the plan's words and the docs (`test-plan.md`, `report-schema.md`, `protocol.md`, the agent's README) describe both. Nothing changes on screen apart from new metric names in the plan's words and the Measurements table, so there are no screenshots. ## Testing - `python -I agent/tests/test_agent_core.py --strict`: 2073 checks passed. New fixtures (`log-value.toml`, `run-metrics-plain.toml` and 10 `bad-log-value-*` and `bad-run-metric-schema` refusals) are cross-checked against the coordinator's reader. New probe checks cover every take, `none`, a line without a number, and a back-reference or catastrophic pattern. New rule checks use #1701's own numbers. - `pytest -q` in `coordinator/`: 580 passed, 1 skipped. - **Not tested:** a real farm job. The frame window (status at a cell's start and end) has not run against the game; on a cell that ends in a crash or timeout it runs to the end of the frame log. Follow-up, not in this PR: a plan can't choose the driver cache state yet. A Linux run's sandbox home is new each run, so its driver cache is likely empty every time. On Windows the runner account's cache likely persists. That's the third step of the proposal. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat: plan schema 7, log_value and whole-run frame metrics
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
503b24be76
A PR like Longshot #1701 removes a one-time hitch: the first ASTC
texture's 316 ms frame on a start with an empty driver cache. The farm
could not see it: the frametime probe counts only the measure steps'
frames, after their warmup, and log_match can only count lines, not
read the 316.5 out of "vk: slow upload 316.5 ms".

Plan schema 7 adds two things. The frametime probe's run metrics
run_frametime_max_ms and run_hitch_count cover every frame of a cell,
from its steps' start to their end (the bridge's frame numbers), loads
and warmups included. The log_value probe reads numbers the game logs:
up to 8 values, each a regex with one capture group, taken as max, min,
sum, first, last or count, with none = N for when no line matches; each
is the metric value_<name>, judged by [pass] like any run metric.
Patterns run without backtracking and with a time limit, as log_match's.

A plan with a log_value probe or a rule on a run_* metric normalizes to
schema 7 and goes only to agents of plan schema 7 (the agent says 7
now). Agent and coordinator check it alike (new fixtures, cross-
checked); the planner, the plan's words and the docs know both.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!97
No description provided.