feat: plan schema 7, log_value and whole-run frame metrics #97
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk!97
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Beanstalk:feat/log-values"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A Longshot PR that removes a one-time hitch, such as Longshot #1701 and the first ASTC texture's 316 ms frame on a start with an empty driver cache, couldn't be measured by the farm. The
frametimeprobe counts only the measure steps' frames, after their warmup, andlog_matchcan count lines but can't read the 316.5 out ofvk: slow upload 316.5 ms.What changed
Plan schema 7 adds two things:
frametimeprobe also reportsrun_frametime_max_msandrun_hitch_count. They cover every frame of a cell, from the start of its steps to their end (the bridge's frame numbers), loads and warmups included.log_valueprobe. It reads numbers the game logs: up to 8 values, each a regex with one capture group, taken asmax,min,sum,first,lastorcount.none = Nsets the value for when no line matches. Each value is the metricvalue_<name>, and[pass]judges it like any run metric. Patterns run without backtracking and with a time limit, aslog_match's do.A plan with a
log_valueprobe or a rule on arun_*metric normalizes to schema 7 and goes only to agents of plan schema 7. The agent now reports 7, so these plans wait for testers to update. Any other plan normalizes as before. The agent (Plan.cs,Probes.cs,ScenarioRunner.cs) and the coordinator (plan.py) check plans alike. The planner's prompt, the plan's words and the docs (test-plan.md,report-schema.md,protocol.md, the agent's README) describe both.Nothing changes on screen apart from new metric names in the plan's words and the Measurements table, so there are no screenshots.
Testing
python -I agent/tests/test_agent_core.py --strict: 2073 checks passed. New fixtures (log-value.toml,run-metrics-plain.tomland 10bad-log-value-*andbad-run-metric-schemarefusals) are cross-checked against the coordinator's reader. New probe checks cover every take,none, a line without a number, and a back-reference or catastrophic pattern. New rule checks use #1701's own numbers.pytest -qincoordinator/: 580 passed, 1 skipped.Follow-up, not in this PR: a plan can't choose the driver cache state yet. A Linux run's sandbox home is new each run, so its driver cache is likely empty every time. On Windows the runner account's cache likely persists. That's the third step of the proposal.
🤖 Generated with Claude Code