fix(agent): Linux setup without system packages, the farm's clock for tickets, a job's own errors reported #56

Merged
HumanBean merged 1 commit from HumanBean/Beanstalk:fix/linux-setup-clock into main 2026-11-16 10:52:50 +00:00
Owner

Why

In the last three days only 3 of 22 reports were real results. The others stopped in the tester's setup, before any game ran. These are the four causes in the queue now:

  • SteamOS (Steam Deck), #1073: the agent's venv of the system python3 had no pip ("No module named pip").
  • Bazzite, #461: pip built lz4 and zstandard from source and failed, because there's no compiler.
  • SteamOS, #637 and #916: the ZBIC reader (needed to read the .nup) didn't build. nx2/zbic.py compiles with the fetched clang but without the sysroot or lld, so it needs the system's C headers and linker, which SteamOS doesn't have.
  • Windows, #518: the PC's clock is more than 10 minutes behind, so every ticket was refused as "issued in the future".
  • Windows, #910: three times in a row the heartbeats stopped about 8 minutes into the base build while the app went on polling. The lease then ran out, and the farm was never told why.

What changed

  • Linux Python (LinuxPython): when the system's python3 lacks Longshot's modules, the agent fetches the CPython (python-build-standalone) that main's scripts/linux/deps.sh pins for the AppImage (PY_TAG, PY_VER, PY_SHA256).
    • It checks the download's SHA-256, unpacks it into <work>/python in place of the old venv, and installs requirements.txt with --only-binary :all:.
    • It needs nothing from the system: no python3-venv, no pip, no compiler.
    • A stamp file holds the pin and the requirement files' hash; it's made again when either changes.
    • The download is cached in <work>/.cache/python.
  • ZBIC on Linux (Workspace.ZbicLinuxCommand): the same compile as zbic.py, plus --sysroot=<fetched sysroot> -fuse-ld=lld, as nx2/build.py links the game. Off Linux, or without the fetched emsdk and sysroot, zbic.py's own build is used as before.
  • Error messages (Workspace.ErrorLines): a failed compile or pip install now quotes its error lines instead of the end of the traceback, which only showed the command.
  • Clock (CoordinatorClient.ClockOffset): the offset comes from the Date header of the coordinator's answers, counted only when it's over 2 minutes. Tickets are checked against the coordinator's time, and the agent logs that the PC's clock is off.
  • A job's unexpected errors: any exception the runner didn't expect is now sent to the farm as a passing failure with its type, and written to agent.log with its stack trace. Before, it went unreported and the lease simply ran out. The heartbeat loop now treats any error the way it treats a coordinator error, instead of ending, and it never sends a non-finite progress.

Testing

  • Agent checks: 1853 passed. The new setup section (21 checks) covers:
    • the pin parsing, with lines that aren't plain values refused;
    • the stamp;
    • pip from wheels only;
    • unpacking a test archive over an old venv, keeping the bin/python3 link;
    • the cache, and a bad hash refused before unpacking;
    • the ZBIC command;
    • the error lines;
    • the clock offset.
  • dotnet build agent/App -c Release: 0 warnings.
  • On this PC (Arch), by hand: downloaded the pinned archive (its SHA-256 matches deps.sh) and installed main's requirements.txt with --only-binary :all:. All of lz4, numpy, capstone, unicorn and zstandard come as wheels and import.
  • Not tested on a Steam Deck, on Bazzite or on Windows. The ZBIC flags are the ones the game's own Linux link uses, but nobody has run them on SteamOS yet. Tommy's job loss isn't explained by this PR. The next time it happens, the farm and agent.log will show the error.
## Why In the last three days only 3 of 22 reports were real results. The others stopped in the tester's setup, before any game ran. These are the four causes in the queue now: - **SteamOS (Steam Deck), #1073:** the agent's venv of the system `python3` had no pip ("No module named pip"). - **Bazzite, #461:** pip built `lz4` and `zstandard` from source and failed, because there's no compiler. - **SteamOS, #637 and #916:** the ZBIC reader (needed to read the `.nup`) didn't build. `nx2/zbic.py` compiles with the fetched clang but without the sysroot or lld, so it needs the system's C headers and linker, which SteamOS doesn't have. - **Windows, #518:** the PC's clock is more than 10 minutes behind, so every ticket was refused as "issued in the future". - **Windows, #910:** three times in a row the heartbeats stopped about 8 minutes into the base build while the app went on polling. The lease then ran out, and the farm was never told why. ## What changed - **Linux Python** (`LinuxPython`): when the system's `python3` lacks Longshot's modules, the agent fetches the CPython (python-build-standalone) that main's `scripts/linux/deps.sh` pins for the AppImage (`PY_TAG`, `PY_VER`, `PY_SHA256`). - It checks the download's SHA-256, unpacks it into `<work>/python` in place of the old venv, and installs `requirements.txt` with `--only-binary :all:`. - It needs nothing from the system: no python3-venv, no pip, no compiler. - A stamp file holds the pin and the requirement files' hash; it's made again when either changes. - The download is cached in `<work>/.cache/python`. - **ZBIC on Linux** (`Workspace.ZbicLinuxCommand`): the same compile as `zbic.py`, plus `--sysroot=<fetched sysroot> -fuse-ld=lld`, as `nx2/build.py` links the game. Off Linux, or without the fetched emsdk and sysroot, `zbic.py`'s own build is used as before. - **Error messages** (`Workspace.ErrorLines`): a failed compile or pip install now quotes its `error` lines instead of the end of the traceback, which only showed the command. - **Clock** (`CoordinatorClient.ClockOffset`): the offset comes from the Date header of the coordinator's answers, counted only when it's over 2 minutes. Tickets are checked against the coordinator's time, and the agent logs that the PC's clock is off. - **A job's unexpected errors:** any exception the runner didn't expect is now sent to the farm as a passing failure with its type, and written to `agent.log` with its stack trace. Before, it went unreported and the lease simply ran out. The heartbeat loop now treats any error the way it treats a coordinator error, instead of ending, and it never sends a non-finite progress. ## Testing - Agent checks: 1853 passed. The new `setup` section (21 checks) covers: - the pin parsing, with lines that aren't plain values refused; - the stamp; - pip from wheels only; - unpacking a test archive over an old venv, keeping the `bin/python3` link; - the cache, and a bad hash refused before unpacking; - the ZBIC command; - the error lines; - the clock offset. - `dotnet build agent/App -c Release`: 0 warnings. - On this PC (Arch), by hand: downloaded the pinned archive (its SHA-256 matches `deps.sh`) and installed main's `requirements.txt` with `--only-binary :all:`. All of lz4, numpy, capstone, unicorn and zstandard come as wheels and import. - **Not tested on a Steam Deck, on Bazzite or on Windows.** The ZBIC flags are the ones the game's own Linux link uses, but nobody has run them on SteamOS yet. Tommy's job loss isn't explained by this PR. The next time it happens, the farm and `agent.log` will show the error.
fix(agent): Linux setup without system packages, the farm's clock for tickets, a job's own errors reported
Some checks are pending
CI / Coordinator tests (pull_request) Waiting to run
CI / Agent build and checks (pull_request) Waiting to run
CI / Coordinator image (pull_request) Waiting to run
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
9d683f4991
Linux PCs whose python3 lacks Longshot's modules got a venv of it, which
needed python3-venv and pip (SteamOS: no pip) and built lz4/zstandard
from source when no wheel fit (Bazzite: no compiler). The agent now
fetches the CPython that main's scripts/linux/deps.sh pins for the
AppImage, checks its SHA-256, and installs requirements.txt from wheels
only. It replaces the old venv in the same folder.

The ZBIC reader's Linux build gets the fetched sysroot and lld, as the
game's own build does; zbic.py's compile needed the system's C headers
and linker, which SteamOS lacks. Its failure now quotes the compiler's
error lines.

Tickets are checked against the coordinator's clock, from the Date
header of its answers, so a PC whose clock is minutes off no longer
refuses every ticket as issued in the future; the agent says so.

Any error a job hits that the runner did not expect is now told to the
farm (passing, with its type) and written to agent.log in full, and the
heartbeat loop no longer ends on one: a Windows job three times lost its
heartbeat mid-build, unexplained, while the app went on polling.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!56
No description provided.