fix: the farm pauses a PC that fails setup again and again, and Windows tests its sandbox again once it has git #125
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk!125
Loadingβ¦
Reference in a new issue
No description provided.
Delete branch "HumanBean/Beanstalk:fix/pause-broken-pcs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
One Windows PC (Flashman4's agent-90b2bd) failed 35 jobs in six hours before any PR code ran: git in its AppContainer can't open
/dev/null(NUL). It did with the PC's git, and since #122 does with the agent's MinGit too (its message is #122's new wording), while every other Windows tester's MinGit runs in the same sandbox (one is past its build already). So that PC's AppContainer is refused NUL. Two fixes:The farm pauses a PC that fails setup again and again (
scheduler.setup_pauses, coordinator):fetch,prefetchorbuild-base(main's build, which works on every other PC), not passing, is offered nothing for an hour after the last of them. It was failing job after job, each a try the job then lacked; it also covers the next broken PC (a failing disk, a full one);passing:), so a shared outage never pauses healthy PCs;Windows tests its sandbox again once it has git (
SandboxChoice.FingerprintOf, agent): whether the sandbox has git (the agent's MinGit) is part of the self-test's fingerprint. A self-test before the first job's MinGit skipped its git check and kept the AppContainer; once MinGit is there it runs again (once on every Windows PC), and git that fails there rules the AppContainer out (ModeChoice.FailWhenFailed): the runner account if it is set up, else no sandbox and no jobs, the app saying why.Screenshots
The dev server's made-up data (beanstalk.devserver) on a throwaway database, one made-up tester given three failed setups. No page looked like this before: the state is new.
Testing
Tested: on the farm's server, in containers (not a tester's PC):
test_a_pc_that_fails_setup_again_and_again_is_paused: 3 setup failures on 3 jobs pause that PC (another still gets offers), the pause ends after an hour, a passing failure or a try at the PR's own build breaks the run;dotnet/sdk:10.0): build with 0 warnings andtest_agent_core.py --strict: 2261 checks passed (new: MinGit arriving makes the self-test run again).Still to test: Flashman4's PC after the next nightly: its self-test runs again and leaves the AppContainer.
π€ Generated with Claude Code
Possibly related, for context:
Recommendation: Request changes. Passing failures can incorrectly pause healthy PCs after a shared outage, and the Windows check has not shown that it detects the reported Git failure. The dashboard change also lacks the screenshots required by the repository.
The change limits repeated setup failures and reruns the Windows sandbox self-test after MinGit arrives. It could preserve job attempts, but its failure classification can unnecessarily withhold work from healthy testers, and the Windows fix remains unproven.
Does it help: Unproven. The PR reports 35 failed jobs in six hours before the change and a passing coordinator test, but no run of this commit on the affected Windows PC. The new Windows test compares fingerprints; it does not demonstrate that the sandbox check catches the failing Git operation.
Other systems: The coordinator pause applies to Windows and Linux testers, including manual and scheduled runs. The Windows fingerprint change reruns the sandbox test on existing Windows installations; its effect on the affected PC and other Windows setups has not been demonstrated.
Switches: None.
Reviewed:
563f706cb9against59dd514c72: Read AGENTS.md and CONTRIBUTING.md, the complete diff and commit message, and relevant scheduler, dashboard, Git, sandbox and job-runner paths. The changed code files and documentation were not read in full; no review checklist or pull request template was present. No earlier comments or reviews were supplied.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Testing: No code or tests were run, as requested. The PR reports 601 coordinator tests passed and one skipped; its agent build and strict test were still running when the description was written. The affected Windows PC has not yet been tested.
Overlaps and merge order: Not checked; the network was unavailable.
Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.
@ -59,3 +59,3 @@}var text = WinPipes.MaskSids(scrubber.Scrub(ModeChoice.Text(header, reports)));var choice = SandboxChoice.From(reports, SandboxChoice.FingerprintOf(hardware.Gpu, hardware.OsVersion), DateTimeOffset.Now);var choice = SandboxChoice.From(reports, SandboxChoice.FingerprintOf(hardware.Gpu, hardware.OsVersion, WinHost.Git(home.Data).PathDir != null),The rerun still tests only git --version, while the reported failure occurs in a checkout's rev-parse operation. Please exercise a representative repository operation and verify that the affected failure rejects AppContainer and selects the intended fallback.
@ -627,0 +652,4 @@for agent_id, tries in by.items():if len(tries) < SETUP_FAILS or len({t["job_id"] for t in tries}) < 2:continueif any(t["outcome"] != "error" or t["stage"] not in SETUP_STAGES or (t["reason"] or "").startswith("passing")A passing failure loses this prefix once the job exhausts its passing retries. During a shared network outage, three such failures across two jobs can therefore pause a healthy PC. Preserve the passing classification independently of retry eligibility and test that case.
@ -852,6 +859,8 @@ def tester_status(agent: dict, fit: str | None = None) -> dict:seen = agent.get("last_seen")This adds a visible dashboard state. CONTRIBUTING.md requires before-and-after screenshots at desktop and phone widths; please add them to the PR using made-up data.
563f706cb98e09e9aeb7WIP: fix: the farm pauses a PC that fails setup again and again, and Windows tests its sandbox again once it has gitto fix: the farm pauses a PC that fails setup again and again, and Windows tests its sandbox again once it has git8e09e9aeb74080563096Answers to the review of
563f706c:fail()now marks every try the app called passing (passing: <why>), whether or not the job still had a passing retry, andsetup_pausesleaves all of them out, so a shared outage never pauses healthy PCs.test_a_passing_fail_is_tried_again_by_the_same_pcchecks the try after the retries are used up.git --versionvsrev-parse: the self-test'sgit --versioncatches this failure. git opens/dev/nullinsanitize_stdfds()as it starts, before it parses any command, so--versiondies the same wayrev-parsedoes. The affected PCs show it: the job's probe asks forgit --versionafterrev-parsefails, and itsversionline is empty in every one of their reports (...readable .git\objects True; version).). On every healthy Windows PC on nightly-211 both work: one has already built main and is running the game with MinGit in its AppContainer.