fix(rules): a picture is judged against main's own noise, not the limit alone #87

Merged
HumanBean merged 1 commit from HumanBean/Beanstalk:fix/screenshot-noise into main 2026-11-17 20:04:22 +00:00
Owner

Why

rp_c64831cd35a6deae (Longshot #1458) failed only on screenshot.max_diff: 0.0506 against a 0.05 limit, in a Hyrule Field shot taken after a 40 s measure. It wasn't the PR:

  • The same PR's two runs of one cell differed from main by 0.0098 and 0.0604.
  • The shot is of a live scene: clouds, grass, the time of day, and frame generation's in-between frames.

The farm had no measure of how much main differs from itself, so a fixed limit failed PRs by chance.

What changed

  • Agent (ScenarioRunner.PairScreenshots): diffs main's own pictures of each shot against each other. That's two or more runs a side, which the ABBA order gives. The worst pair is the picture's Noise.
  • Rule (Rules.Evaluate, screenshot.max_diff): a picture fails only when its diff is over the limit and over NoiseMargin (1.5) × that noise.
    • The rule's baseline is now the noise.
    • When the noise is what saved it, a note says so.
    • With one run a side there's no noise, and the limit alone decides, as before.
  • Report: each compared picture carries noise. A failure's kept pictures (#85) follow the same judgement.
  • Report page: shows each rule's note under its status, and each picture's "main against itself" next to its diff.
  • Docs: test-plan.md (screenshot.max_diff) and report-schema.md (noise, the rule's baseline).

Tests

  • Agent: 1988 checks passed. New checks:
    • within the noise: passes, with the noise as the baseline and a note;
    • past 1.5× the noise: fails;
    • no noise known: the limit decides;
    • under the limit: passes.
  • Agent release build: 0 warnings.
  • Coordinator: 557 passed, 1 skipped.
  • Not tested: a real game run. It shows on the next report with pictures, once testers update to the nightly.
## Why [rp_c64831cd35a6deae](https://beanstalk.magicbean.dev/reports/rp_c64831cd35a6deae) (Longshot #1458) failed only on `screenshot.max_diff`: 0.0506 against a 0.05 limit, in a Hyrule Field shot taken after a 40 s measure. It wasn't the PR: - The same PR's two runs of one cell differed from main by 0.0098 and 0.0604. - The shot is of a live scene: clouds, grass, the time of day, and frame generation's in-between frames. The farm had no measure of how much main differs from itself, so a fixed limit failed PRs by chance. ## What changed - **Agent (`ScenarioRunner.PairScreenshots`):** diffs main's own pictures of each shot against each other. That's two or more runs a side, which the ABBA order gives. The worst pair is the picture's `Noise`. - **Rule (`Rules.Evaluate`, `screenshot.max_diff`):** a picture fails only when its diff is over the limit **and** over `NoiseMargin` (1.5) × that noise. - The rule's `baseline` is now the noise. - When the noise is what saved it, a `note` says so. - With one run a side there's no noise, and the limit alone decides, as before. - **Report:** each compared picture carries `noise`. A failure's kept pictures (#85) follow the same judgement. - **Report page:** shows each rule's note under its status, and each picture's "main against itself" next to its diff. - **Docs:** test-plan.md (`screenshot.max_diff`) and report-schema.md (`noise`, the rule's baseline). ## Tests - **Agent:** 1988 checks passed. New checks: - within the noise: passes, with the noise as the baseline and a note; - past 1.5× the noise: fails; - no noise known: the limit decides; - under the limit: passes. - **Agent release build:** 0 warnings. - **Coordinator:** 557 passed, 1 skipped. - **Not tested:** a real game run. It shows on the next report with pictures, once testers update to the nightly.
fix(rules): a picture is judged against main's own noise, not the limit alone
Some checks are pending
CI / Coordinator tests (pull_request) Waiting to run
CI / Agent build and checks (pull_request) Waiting to run
CI / Coordinator image (pull_request) Waiting to run
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 3s
fdf3ecdf27
A live scene moves between game starts (clouds, grass, the time of day,
a generated frame), so a fixed screenshot.max_diff failed PRs by chance:
rp_c64831cd35a6deae (#1458) failed at 0.0506 against 0.05 while the
same PR's two runs of one cell differed from main by 0.0098 and 0.0604.
The agent now diffs main's own pictures of each shot against each
other (two or more runs a side: the ABBA order has them) and a picture
fails only over the limit and over 1.5x that noise. The rule's baseline
is the noise, its note says when it saved a picture, and each picture in
the report carries its noise; the report page shows both. A failure's
kept pictures follow the same judgement. With one run a side the limit
alone decides, as before.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean deleted branch fix/screenshot-noise 2026-11-17 20:04:22 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!87
No description provided.