feat: a failure's pictures, kept a day #85

Merged
HumanBean merged 1 commit from HumanBean/Beanstalk:feat/failure-pictures into main 2026-11-17 19:02:03 +00:00
Owner

Why

When a farm test fails or gets stuck, nobody can see what the game showed. #1359's RADV runs are an example: "a step ran out of time: not met in time" four times. Pictures of the failure, kept for a day, make that debuggable without keeping game pictures forever.

What changed

  • Agent: what it captures
    • Last frame: when a step runs out of time (and the game isn't hung) or an expect step doesn't hold, it asks the game for a shot before quitting (best effort, 30 s at most).
    • Pictures over screenshot.max_diff: for these, it keeps main's and the PR's files from the worst pair. Before, it kept the first pair's PR file only.
  • Agent: how it sends them
    • These go up whatever the plan's upload says. Each is shrunk to fit 960x540 (Png.Shrink) and written as an RGB PNG (Png.Encode, with a CRC table: no new package).
    • At most 4 per report, 2 MB together, marked "keep": "day".
    • When the report would pass the 4 MB limit with them, the last ones and their mentions are left out, so a failure's report always goes.
    • The plan's own pictures now count apart from these (6 files, 2 MB, as before).
  • Report fields: last_frames on a scenario, and kept: {main, pr} on a screenshot.
  • Coordinator
    • Takes these besides the plan's own pictures.
    • Stores them with expires_at a day on (migration 0019); the sweeper deletes them after.
    • The upload route stops serving them at expiry.
    • The report page shows them under "Pictures of the failure" until then, and says they were deleted after.
  • Docs: report-schema.md, and using.md (including "What leaves your PC").

Rollout

Coordinator first, since an older coordinator would refuse the larger uploads. New agents pick it up from the next nightly.

Tests

  • Coordinator: 554 passed, 1 skipped. New tests cover:
    • the refusals (keep, count, 2 MB);
    • storing with a day's expiry and the sweeper's deletion;
    • the report page before and after expiry, and the upload route's 404.
  • Agent: 1982 checks passed, including:
    • an encode round trip, the shrink's size and box mean;
    • last frames kept a day, at most 4;
    • a noisy frame left out when the report would pass 4 MB.
  • Agent release build: 0 warnings.
  • Unrelated flake: test_runs_planner::test_planning_again_with_new_words failed 2 times in 5 runs while this PC's load average was 30. It races the background planner, and this change doesn't touch it.
  • Not tested: the last frame against a real stuck game. It shows on the next timeout on the farm.
## Why When a farm test fails or gets stuck, nobody can see what the game showed. #1359's RADV runs are an example: "a step ran out of time: not met in time" four times. Pictures of the failure, kept for a day, make that debuggable without keeping game pictures forever. ## What changed - **Agent: what it captures** - **Last frame:** when a step runs out of time (and the game isn't hung) or an expect step doesn't hold, it asks the game for a `shot` before quitting (best effort, 30 s at most). - **Pictures over `screenshot.max_diff`:** for these, it keeps main's and the PR's files from the worst pair. Before, it kept the first pair's PR file only. - **Agent: how it sends them** - These go up whatever the plan's `upload` says. Each is shrunk to fit 960x540 (`Png.Shrink`) and written as an RGB PNG (`Png.Encode`, with a CRC table: no new package). - At most 4 per report, 2 MB together, marked `"keep": "day"`. - When the report would pass the 4 MB limit with them, the last ones and their mentions are left out, so a failure's report always goes. - The plan's own pictures now count apart from these (6 files, 2 MB, as before). - **Report fields:** `last_frames` on a scenario, and `kept: {main, pr}` on a screenshot. - **Coordinator** - Takes these besides the plan's own pictures. - Stores them with `expires_at` a day on (migration 0019); the sweeper deletes them after. - The upload route stops serving them at expiry. - The report page shows them under "Pictures of the failure" until then, and says they were deleted after. - **Docs:** report-schema.md, and using.md (including "What leaves your PC"). ## Rollout Coordinator first, since an older coordinator would refuse the larger uploads. New agents pick it up from the next nightly. ## Tests - **Coordinator:** 554 passed, 1 skipped. New tests cover: - the refusals (`keep`, count, 2 MB); - storing with a day's expiry and the sweeper's deletion; - the report page before and after expiry, and the upload route's 404. - **Agent:** 1982 checks passed, including: - an encode round trip, the shrink's size and box mean; - last frames kept a day, at most 4; - a noisy frame left out when the report would pass 4 MB. - **Agent release build:** 0 warnings. - **Unrelated flake:** `test_runs_planner::test_planning_again_with_new_words` failed 2 times in 5 runs while this PC's load average was 30. It races the background planner, and this change doesn't touch it. - **Not tested:** the last frame against a real stuck game. It shows on the next timeout on the farm.
feat: a failure's pictures, kept a day
Some checks are pending
CI / Coordinator tests (pull_request) Waiting to run
CI / Agent build and checks (pull_request) Waiting to run
CI / Coordinator image (pull_request) Waiting to run
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
CI / Conventional Commit title (pull_request) Successful in 15s
4718e64734
When a step runs out of time or an expect step doesn't hold, the agent
takes the game's last frame before it quits; when a picture differs
from main's more than screenshot.max_diff, it keeps main's and the
PR's from the worst pair. Those go up whatever the plan's upload says
(they show only the game, as the plan's pictures), shrunk to fit
960x540 (Png.Shrink and a small RGB Png.Encode), at most 4 and 2 MB
together, marked keep = "day"; the last ones are left out when the
report would pass 4 MB with them, so a failure's report always goes.

The coordinator takes them beside the plan's own (6, 2 MB), stores them
with expires_at a day on (migration 0019), the sweeper deletes them
after, and the report page shows them under Pictures of the failure
until then, and says they were deleted after. The plan's own pictures
stay as before.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean deleted branch feat/failure-pictures 2026-11-17 19:02:07 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!85
No description provided.