fix: the farm pauses a PC that fails setup again and again, and Windows tests its sandbox again once it has git #125

Merged
HumanBean merged 1 commit from HumanBean/Beanstalk:fix/pause-broken-pcs into main 2026-11-20 20:38:09 +00:00
Owner

Summary

One Windows PC (Flashman4's agent-90b2bd) failed 35 jobs in six hours before any PR code ran: git in its AppContainer can't open /dev/null (NUL). It did with the PC's git, and since #122 does with the agent's MinGit too (its message is #122's new wording), while every other Windows tester's MinGit runs in the same sandbox (one is past its build already). So that PC's AppContainer is refused NUL. Two fixes:

The farm pauses a PC that fails setup again and again (scheduler.setup_pauses, coordinator):

  • a PC whose last 3 finished tries, at 2 jobs or more, all failed at fetch, prefetch or build-base (main's build, which works on every other PC), not passing, is offered nothing for an hour after the last of them. It was failing job after job, each a try the job then lacked; it also covers the next broken PC (a failing disk, a full one);
  • passing failures (its network, a pruned ref), lost leases and stops don't count, nor a try that got to the PR's own build. A try the app called passing says so even once the job's passing retries are used up (its reason starts passing: ), so a shared outage never pauses healthy PCs;
  • the Testers page, the tester's page and the health page show it as a Problem with the reason; its tester's own Run on this PC still works, to check a fix.

Windows tests its sandbox again once it has git (SandboxChoice.FingerprintOf, agent): whether the sandbox has git (the agent's MinGit) is part of the self-test's fingerprint. A self-test before the first job's MinGit skipped its git check and kept the AppContainer; once MinGit is there it runs again (once on every Windows PC), and git that fails there rules the AppContainer out (ModeChoice.FailWhenFailed): the runner account if it is set up, else no sandbox and no jobs, the app saying why.

Screenshots

The dev server's made-up data (beanstalk.devserver) on a throwaway database, one made-up tester given three failed setups. No page looked like this before: the state is new.

Testers as a BigBean, desktop: a paused PC is a Problem, with the reason

Testers as a BigBean, 390 px

The paused tester's own page

Testers signed out, 390 px: counted as a problem

Testing

Tested: on the farm's server, in containers (not a tester's PC):

  • the coordinator suite (python:3.13-alpine, Postgres 17): 601 passed, 1 skipped, 0 failed (run again after the review's fixes, below). New test_a_pc_that_fails_setup_again_and_again_is_paused: 3 setup failures on 3 jobs pause that PC (another still gets offers), the pause ends after an hour, a passing failure or a try at the PR's own build breaks the run;
  • the agent (dotnet/sdk:10.0): build with 0 warnings and test_agent_core.py --strict: 2261 checks passed (new: MinGit arriving makes the self-test run again).

Still to test: Flashman4's PC after the next nightly: its self-test runs again and leaves the AppContainer.

πŸ€– Generated with Claude Code

## Summary One Windows PC (Flashman4's agent-90b2bd) failed 35 jobs in six hours before any PR code ran: git in its AppContainer can't open `/dev/null` (NUL). It did with the PC's git, and since #122 does with the agent's MinGit too (its message is #122's new wording), while every other Windows tester's MinGit runs in the same sandbox (one is past its build already). So that PC's AppContainer is refused NUL. Two fixes: **The farm pauses a PC that fails setup again and again** (`scheduler.setup_pauses`, coordinator): - a PC whose last 3 finished tries, at 2 jobs or more, all failed at `fetch`, `prefetch` or `build-base` (main's build, which works on every other PC), not passing, is offered nothing for an hour after the last of them. It was failing job after job, each a try the job then lacked; it also covers the next broken PC (a failing disk, a full one); - passing failures (its network, a pruned ref), lost leases and stops don't count, nor a try that got to the PR's own build. A try the app called passing says so even once the job's passing retries are used up (its reason starts `passing: `), so a shared outage never pauses healthy PCs; - the Testers page, the tester's page and the health page show it as a **Problem** with the reason; its tester's own Run on this PC still works, to check a fix. **Windows tests its sandbox again once it has git** (`SandboxChoice.FingerprintOf`, agent): whether the sandbox has git (the agent's MinGit) is part of the self-test's fingerprint. A self-test before the first job's MinGit skipped its git check and kept the AppContainer; once MinGit is there it runs again (once on every Windows PC), and git that fails there rules the AppContainer out (`ModeChoice.FailWhenFailed`): the runner account if it is set up, else no sandbox and no jobs, the app saying why. ## Screenshots The dev server's made-up data (beanstalk.devserver) on a throwaway database, one made-up tester given three failed setups. No page looked like this before: the state is new. ![Testers as a BigBean, desktop: a paused PC is a Problem, with the reason](https://git.magicbean.dev/attachments/47237be4-b39d-4cd3-b207-e1c28dfb5654) ![Testers as a BigBean, 390 px](https://git.magicbean.dev/attachments/aafbd302-6523-49d2-b51d-b4dc4232a52b) ![The paused tester's own page](https://git.magicbean.dev/attachments/939987b4-bf25-4d36-8b08-7e850ff3becf) ![Testers signed out, 390 px: counted as a problem](https://git.magicbean.dev/attachments/122ba003-3fca-44ca-b3f5-76d793f76dda) ## Testing **Tested:** on the farm's server, in containers (not a tester's PC): - the coordinator suite (python:3.13-alpine, Postgres 17): 601 passed, 1 skipped, 0 failed (run again after the review's fixes, below). New `test_a_pc_that_fails_setup_again_and_again_is_paused`: 3 setup failures on 3 jobs pause that PC (another still gets offers), the pause ends after an hour, a passing failure or a try at the PR's own build breaks the run; - the agent (`dotnet/sdk:10.0`): build with 0 warnings and `test_agent_core.py --strict`: 2261 checks passed (new: MinGit arriving makes the self-test run again). **Still to test:** Flashman4's PC after the next nightly: its self-test runs again and leaves the AppContainer. πŸ€– Generated with [Claude Code](https://claude.com/claude-code)
fix: the farm pauses a PC that fails setup again and again, and Windows tests its sandbox again once it has git
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 8s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
563f706cb9
One Windows PC failed 35 jobs in six hours before any PR code ran: git in its
AppContainer can't open /dev/null (NUL), with the PC's git and, since #122,
with the agent's MinGit too. Every other Windows tester's MinGit runs there.

Coordinator (scheduler.setup_pauses):
- A PC whose last 3 finished tries, at 2 jobs or more, all failed at fetch,
  prefetch or build-base (not passing) is offered nothing for an hour after the
  last of them: it was failing job after job, each a try the job then lacked.
  Its tester's own Run on this PC still works.
- The Testers page, a tester's page and the health page show it as a Problem
  with the reason (tester_status's farm_pause).

Agent (SandboxChoice.FingerprintOf):
- Whether the sandbox has git (the agent's MinGit) is part of the self-test's
  fingerprint. A self-test before the first job's MinGit skipped its git check
  and kept the AppContainer; once MinGit is there it runs again, and git that
  fails there rules the AppContainer out (ModeChoice.FailWhenFailed): the
  runner account if it is set up, else no sandbox and no jobs, saying why.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • issue #92 Windows: AppContainer sandbox breaks when git on PATH is a shim (Scoop) β€” Related Windows sandbox git/AppContainer failures.
  • issue #111 Two Windows testers fail every job before building, with the sandbox unable to read the job's working directory β€” Asks the farm to stop repeatedly assigning jobs to failing testers.
  • PR #93 fix(agent): skip git shims on PATH and check git in Windows self-test β€” Merged prior sandbox git self-test; this PR changes when it reruns.
Possibly related, for context: - issue #92 Windows: AppContainer sandbox breaks when git on PATH is a shim (Scoop) β€” Related Windows sandbox git/AppContainer failures. - issue #111 Two Windows testers fail every job before building, with the sandbox unable to read the job's working directory β€” Asks the farm to stop repeatedly assigning jobs to failing testers. - PR #93 fix(agent): skip git shims on PATH and check git in Windows self-test β€” Merged prior sandbox git self-test; this PR changes when it reruns.
BeanBot left a comment

Recommendation: Request changes. Passing failures can incorrectly pause healthy PCs after a shared outage, and the Windows check has not shown that it detects the reported Git failure. The dashboard change also lacks the screenshots required by the repository.

The change limits repeated setup failures and reruns the Windows sandbox self-test after MinGit arrives. It could preserve job attempts, but its failure classification can unnecessarily withhold work from healthy testers, and the Windows fix remains unproven.

Does it help: Unproven. The PR reports 35 failed jobs in six hours before the change and a passing coordinator test, but no run of this commit on the affected Windows PC. The new Windows test compares fingerprints; it does not demonstrate that the sandbox check catches the failing Git operation.
Other systems: The coordinator pause applies to Windows and Linux testers, including manual and scheduled runs. The Windows fingerprint change reruns the sandbox test on existing Windows installations; its effect on the affected PC and other Windows setups has not been demonstrated.
Switches: None.

Reviewed: 563f706cb9 against 59dd514c72: Read AGENTS.md and CONTRIBUTING.md, the complete diff and commit message, and relevant scheduler, dashboard, Git, sandbox and job-runner paths. The changed code files and documentation were not read in full; no review checklist or pull request template was present. No earlier comments or reviews were supplied.; nothing run (a code review).
Checked and fine:

  • The manual acceptance path remains separate from the new offer pause.
  • A completed passing report, PR-build failure, or expired lease breaks the selected sequence of setup failures.
  • The commit records the required AI assistance trailers.

Blocking (to change before merge)

  1. coordinator/beanstalk/scheduler.py:655: Passing failures are identified by a reason prefix that is recorded only while the job is eligible for a passing retry. Once a job has exhausted its three passing retries, another network failure is recorded without that prefix; three such failures across two jobs can pause a healthy PC for an hour during a shared outage. Preserve the agent's passing classification independently of the retry limit and exclude all such failures from the pause, with coverage for exhausted retries.
  2. agent/Core/WindowsSelfTest.cs:61: The rerun still probes only git --version, whereas the reported failure occurs during the sandboxed checkout's rev-parse operation. A successful version command does not establish that the operation needed by a build can open NUL, so the affected PC could retain AppContainer and continue failing jobs. Exercise a representative repository operation in the self-test and verify the failing and fallback cases, including on the affected PC.
  3. coordinator/beanstalk/web.py:859: The new Problem state changes dashboard screens, but the PR description has no desktop or phone screenshots. CONTRIBUTING.md requires before-and-after dashboard screenshots for changes visible on screen; add them using made-up data.

Testing: No code or tests were run, as requested. The PR reports 601 coordinator tests passed and one skipped; its agent build and strict test were still running when the description was written. The affected Windows PC has not yet been tested.
Overlaps and merge order: Not checked; the network was unavailable.


Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** Passing failures can incorrectly pause healthy PCs after a shared outage, and the Windows check has not shown that it detects the reported Git failure. The dashboard change also lacks the screenshots required by the repository. The change limits repeated setup failures and reruns the Windows sandbox self-test after MinGit arrives. It could preserve job attempts, but its failure classification can unnecessarily withhold work from healthy testers, and the Windows fix remains unproven. **Does it help:** Unproven. The PR reports 35 failed jobs in six hours before the change and a passing coordinator test, but no run of this commit on the affected Windows PC. The new Windows test compares fingerprints; it does not demonstrate that the sandbox check catches the failing Git operation. **Other systems:** The coordinator pause applies to Windows and Linux testers, including manual and scheduled runs. The Windows fingerprint change reruns the sandbox test on existing Windows installations; its effect on the affected PC and other Windows setups has not been demonstrated. **Switches:** None. **Reviewed:** 563f706cb9b6 against 59dd514c728d: Read AGENTS.md and CONTRIBUTING.md, the complete diff and commit message, and relevant scheduler, dashboard, Git, sandbox and job-runner paths. The changed code files and documentation were not read in full; no review checklist or pull request template was present. No earlier comments or reviews were supplied.; nothing run (a code review). **Checked and fine:** - The manual acceptance path remains separate from the new offer pause. - A completed passing report, PR-build failure, or expired lease breaks the selected sequence of setup failures. - The commit records the required AI assistance trailers. **Blocking** (to change before merge) 1. coordinator/beanstalk/scheduler.py:655: Passing failures are identified by a reason prefix that is recorded only while the job is eligible for a passing retry. Once a job has exhausted its three passing retries, another network failure is recorded without that prefix; three such failures across two jobs can pause a healthy PC for an hour during a shared outage. Preserve the agent's passing classification independently of the retry limit and exclude all such failures from the pause, with coverage for exhausted retries. 2. agent/Core/WindowsSelfTest.cs:61: The rerun still probes only git --version, whereas the reported failure occurs during the sandboxed checkout's rev-parse operation. A successful version command does not establish that the operation needed by a build can open NUL, so the affected PC could retain AppContainer and continue failing jobs. Exercise a representative repository operation in the self-test and verify the failing and fallback cases, including on the affected PC. 3. coordinator/beanstalk/web.py:859: The new Problem state changes dashboard screens, but the PR description has no desktop or phone screenshots. CONTRIBUTING.md requires before-and-after dashboard screenshots for changes visible on screen; add them using made-up data. **Testing:** No code or tests were run, as requested. The PR reports 601 coordinator tests passed and one skipped; its agent build and strict test were still running when the description was written. The affected Windows PC has not yet been tested. **Overlaps and merge order:** Not checked; the network was unavailable. --- <sub>Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -59,3 +59,3 @@
}
var text = WinPipes.MaskSids(scrubber.Scrub(ModeChoice.Text(header, reports)));
var choice = SandboxChoice.From(reports, SandboxChoice.FingerprintOf(hardware.Gpu, hardware.OsVersion), DateTimeOffset.Now);
var choice = SandboxChoice.From(reports, SandboxChoice.FingerprintOf(hardware.Gpu, hardware.OsVersion, WinHost.Git(home.Data).PathDir != null),
Member

The rerun still tests only git --version, while the reported failure occurs in a checkout's rev-parse operation. Please exercise a representative repository operation and verify that the affected failure rejects AppContainer and selects the intended fallback.

The rerun still tests only git --version, while the reported failure occurs in a checkout's rev-parse operation. Please exercise a representative repository operation and verify that the affected failure rejects AppContainer and selects the intended fallback.
@ -627,0 +652,4 @@
for agent_id, tries in by.items():
if len(tries) < SETUP_FAILS or len({t["job_id"] for t in tries}) < 2:
continue
if any(t["outcome"] != "error" or t["stage"] not in SETUP_STAGES or (t["reason"] or "").startswith("passing")
Member

A passing failure loses this prefix once the job exhausts its passing retries. During a shared network outage, three such failures across two jobs can therefore pause a healthy PC. Preserve the passing classification independently of retry eligibility and test that case.

A passing failure loses this prefix once the job exhausts its passing retries. During a shared network outage, three such failures across two jobs can therefore pause a healthy PC. Preserve the passing classification independently of retry eligibility and test that case.
@ -852,6 +859,8 @@ def tester_status(agent: dict, fit: str | None = None) -> dict:
seen = agent.get("last_seen")
Member

This adds a visible dashboard state. CONTRIBUTING.md requires before-and-after screenshots at desktop and phone widths; please add them to the PR using made-up data.

This adds a visible dashboard state. CONTRIBUTING.md requires before-and-after screenshots at desktop and phone widths; please add them to the PR using made-up data.
HumanBean force-pushed fix/pause-broken-pcs from 563f706cb9
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 8s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
to 8e09e9aeb7
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 7s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
2026-11-20 20:33:34 +00:00
Compare
HumanBean changed title from WIP: fix: the farm pauses a PC that fails setup again and again, and Windows tests its sandbox again once it has git to fix: the farm pauses a PC that fails setup again and again, and Windows tests its sandbox again once it has git 2026-11-20 20:33:38 +00:00
HumanBean force-pushed fix/pause-broken-pcs from 8e09e9aeb7
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 7s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
CI / Coordinator tests (pull_request) Has been cancelled
CI / Agent build and checks (pull_request) Has been cancelled
CI / Coordinator image (pull_request) Has been cancelled
to 4080563096
Some checks failed
CI / Conventional Commit title (pull_request) Successful in 24s
CI / Workflow lint (actionlint) (pull_request) Successful in 25s
CI / Coordinator tests (pull_request) Failing after 1m21s
CI / Agent build and checks (pull_request) Successful in 46s
CI / Coordinator image (pull_request) Successful in 11s
2026-11-20 20:36:57 +00:00
Compare
Author
Owner

Answers to the review of 563f706c:

  1. Passing failures after the retries run out: fixed. fail() now marks every try the app called passing (passing: <why>), whether or not the job still had a passing retry, and setup_pauses leaves all of them out, so a shared outage never pauses healthy PCs. test_a_passing_fail_is_tried_again_by_the_same_pc checks the try after the retries are used up.
  2. git --version vs rev-parse: the self-test's git --version catches this failure. git opens /dev/null in sanitize_stdfds() as it starts, before it parses any command, so --version dies the same way rev-parse does. The affected PCs show it: the job's probe asks for git --version after rev-parse fails, and its version line is empty in every one of their reports (...readable .git\objects True; version).). On every healthy Windows PC on nightly-211 both work: one has already built main and is running the game with MinGit in its AppContainer.
  3. Screenshots: added to the description (desktop and 390 px, as a BigBean and signed out, the dev server's made-up data with one made-up tester paused).
Answers to the review of 563f706c: 1. **Passing failures after the retries run out**: fixed. `fail()` now marks every try the app called passing (`passing: <why>`), whether or not the job still had a passing retry, and `setup_pauses` leaves all of them out, so a shared outage never pauses healthy PCs. `test_a_passing_fail_is_tried_again_by_the_same_pc` checks the try after the retries are used up. 2. **`git --version` vs `rev-parse`**: the self-test's `git --version` catches this failure. git opens `/dev/null` in `sanitize_stdfds()` as it starts, before it parses any command, so `--version` dies the same way `rev-parse` does. The affected PCs show it: the job's probe asks for `git --version` after `rev-parse` fails, and its `version` line is empty in every one of their reports (`...readable .git\objects True; version).`). On every healthy Windows PC on nightly-211 both work: one has already built main and is running the game with MinGit in its AppContainer. 3. **Screenshots**: added to the description (desktop and 390 px, as a BigBean and signed out, the dev server's made-up data with one made-up tester paused).
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!125
No description provided.