fix(tests): size run_all's default checks by memory, not CPUs alone #2177
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2177
Loading…
Reference in a new issue
No description provided.
Delete branch "dazemc/Longshot:fix/run-all-memory-aware-jobs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
tests/run_all.pyran--jobschecks at once, and--jobsdefaulted toos.cpu_count(). On this PC that is 32:32 checks at once, each able to compile and link a test binary, on a 15.5 GiB machine.
mise run testfilled memory,froze the desktop and got some checks OOM-killed.
The default now depends on the memory actually free to the runner as well as on the CPUs:
one check per 2 GiB of memory after 4 GiB kept for the OS and the desktop (a compiling check peaks at a few hundred
MiB, often over a gigabyte), never more than eight (past eight the checks contend for the CPU anyway and memory is the
limit), never fewer than two. The budget is the smallest of the tightest cgroup v2
memory.maxon the process (what asystemd-run -p MemoryMax=…scope sets), the RAM available now (/proc/meminfo'sMemAvailable) and the total: inside areal 8 GiB scope this reads 8.0 GiB and picks 2, where seven checks were OOM-killed in that same scope, and 32 GiB with
5 GiB free picks 2, not 8. Windows reads
GlobalMemoryStatusEx'sullTotalPhysandullAvailPhys; any other OS, or aread that fails, falls back to the CPU count capped at eight (a 12-core Mac picks 8). A
--jobs Nthe caller passesstill wins, whatever the memory (
argparsekeeps the defaultNonesomainasks the rule).The same one-check run on this 32-thread, 15.5 GiB PC:
main9e10f6f732 at once1df1d2db,--jobsunset2 at onceas the PC stands (busy: 1.5 GiB available; 4-5 when idle)How to see it on
main:python -I -B tests/run_all.py --exact tests/test_run_all_schedule.pyprints… 32 at once (linux); here it prints… 2 at oncewhile the machine is busy (more when it is idle: the count follows the memoryfree), and the same command inside
systemd-run --scope -p MemoryMax=8Gstill prints2 at once.Read first:
tests/run_all.py(default_jobs,check_jobs,total_memory,parser) andtests/test_run_all_default_jobs.py.#2171 already fixed the three scratch-repo checks (
test_run_all_output,test_run_all_reap,test_run_all_mingwnow copy
tests/compiler_command.pytoo). This PR is only the memory-aware default; it does not touch those checks.Testing
Tested: Linux (no GPU involved), this PC (32 threads, 15.5 GiB).
python -I -B tests/run_all.py --strict test_run_all_default_jobs test_run_all_junit test_run_all_declarations test_run_all_schedule test_run_all_output test_run_all_reap test_run_all_mingw→ 7 passed, 0 failed, 0 skipped in 22 s, in an 8 GiB scope, printed as
2 at once.On
main9e10f6f7, the same run prints32 at onceand the three scratch-repo checks pass (they were red until #2171);the new check fails there with
AttributeError: module 'run_all' has no attribute 'default_jobs', becausemainhas nomemory-aware rule at all. On this branch it passes.
The new check pins the rule from the numbers alone: 32 CPUs/16 GiB → 6 (the reserve), 32/64 GiB → 8 (cap), 4/64 GiB → 4,
8/8 GiB → 2, 32/2 GiB → 2 (floor), 32/10 GiB → 3, an 8 GiB cgroup limit → 2 (with 9 GiB free and 15.5 GiB total), 32 GiB
total with 5 GiB available → 2, unknown memory → the CPU count capped at 8 (12 cores → 8), one CPU → 1, a passed
--jobs(1, 3, 32, 0) winning, and--jobsunset stayingNoneinargparseso a CPU-count default cannot come back.This PC's own case is pinned at 5 for a 15.5 GiB budget, where the rule without the reserve gave 7; as it stands the PC
is busy and
default_jobs()gives the floor of 2.Full suite on
fa21a408/7156b164(pinned headers, one run):934 checks, 18 skipped, 3 at once (linux)→912 passed, 22 failed, 18 skipped in 626 s. The 22 failing names are exactly the main baseline's at9e10f6f7(
911 passed, 22 failed, 18 skipped):set(branch) − set(main) = ∅,set(main) − set(branch) = ∅— no added or missingname, and the one extra pass is this PR's new check. That run does not cover this head: the branch was then rebased
onto
a2dc5385and its rule corrected to read the cgroup limit and available memory, which changedtests/run_all.py(98 insertions / 32 deletions against7156b164) andtests/test_run_all_default_jobs.py(67/18) — the suite ran before boththe rebase and the final rule. The unit set (this check,
junit,declarations,schedule,output,reap,mingw)was re-run on
1df1d2dbin an 8 GiB scope: 7 passed, 0 failed, 0 skipped, at the default's2 at oncein 22 s (and at--jobs 3in 21 s). Full suite on the tree of1df1d2db(the same tree as head5c4051f4, below) —tests/run_all.py --strict --jobs 5insidesystemd-run --user --scope -p MemoryMax=12G -p MemorySwapMax=0with the pinned headers: 912 passed / 23 failed /18 skipped in 440 s (5 at once), the 23 failing names identical to the
a2dc5385baseline's (compared name by name),and this PR's
tests/test_run_all_default_jobs.pypasses. Head5c4051f4is1df1d2dbwith theAssisted-by:trailer added to its message only — tree14636770, byte-identical — so the suite and the numbers describe the head's tree.The one earlier full run without
LONGSHOT_VULKAN_HEADERS/LONGSHOT_SPIRV_HEADERShad 86not found: … (--strict)failures before any test ran; the reported run has the headers and none of that remains.
In-game run: N/A — test runner only. Nothing in the game loads
tests/run_all.py; the change is how the game-freechecks start.
Still to test:
GlobalMemoryStatusExpath, by hand or CI.Measurements
Nothing to measure: a test-runner default, with no game or switch to A/B. The before/after job counts are in Testing.
Risks
Only how many checks run at once changes: every check still runs, in the same order, and
--jobs Noverrides it. Thecount reads the memory actually free to the runner — the tightest cgroup v2
memory.maxon the process, Linux'sMemAvailable, Windows'ullAvailPhys— and the total; the smallest wins, so a loaded desktop or aMemoryMaxscope isrespected. A cgroup v1 limit or a kernel without
MemAvailableis not read, falling back to the total, and the floor oftwo checks applies even inside a scope smaller than 4 GiB. The Windows
GlobalMemoryStatusExpath could not be run onWindows here; if it fails, the CPU count capped at eight applies. Nothing changes for players after updating: no saves,
longshot_options.inikeys or caches.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named). Tree14636770at1df1d2db(head5c4051f4, message-only amend, same tree): 912 passed / 23 failed / 18 skipped in 440 s,--jobs 5in a 12 GiB scope with the pinned headers; the 23 failing names are identical, name by name, to thea2dc5385baseline's 911 / 23 / 18, and the extra pass is this PR'stests/test_run_all_default_jobs.py.main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here —tests/test_run_all_default_jobs.py: it fails onmainwithAttributeError: module 'run_all' has no attribute 'default_jobs'and passes here.Assisted-by:/Co-Authored-By:trailers. The head carriesAssisted-by: DeepSeek V4.1 Flash 1017 (opencode-go), added to1df1d2db's message only (5c4051f4; same tree,14636770), so the suite above still measures the head's tree.Ready to merge
main, measured again or a comment says why the numbers still hold.7156b164ebfa21a40872dazemc referenced this pull request2026-11-22 04:11:19 +00:00
fa21a40872bfabc80f45bfabc80f451df1d2dbbdfix(tests): size run_all's default checks by memory, not CPUs aloneto WIP: fix(tests): size run_all's default checks by memory, not CPUs aloneThe full-suite box was ticked on a run at
fa21a408/7156b164, which predates the rebase ontoa2dc5385and the final rule:tests/run_all.pychanged 98/32 andtests/test_run_all_default_jobs.py67/18 since, so that result does not cover1df1d2db. Unticked it and back to WIP.tests/run_all.py --strict --jobs 5at1df1d2dbunderMemoryMax=12Gwith the pinned headers is queued after the current queue; re-ready when it is green againsta2dc5385's 911/23/18.The suite at the head is in:
tests/run_all.py --strict --jobs 5at1df1d2dbinsidesystemd-run --user --scope -p MemoryMax=12G -p MemorySwapMax=0with the pinned headers — 912 passed / 23 failed / 18 skipped in 440 s. The 23 failing names are identical to thea2dc5385baseline's 911/23/18, compared name by name (the saved baseline list against this run'sFAILlines), and the extra pass is this PR'stests/test_run_all_default_jobs.py. Full-suite box ticked; back to ready.WIP: fix(tests): size run_all's default checks by memory, not CPUs aloneto fix(tests): size run_all's default checks by memory, not CPUs aloneThe
### When it appliessection is missing theAI assistance recordedbox (and theA check under tests/box if it applies; the newtests/test_run_all_default_jobs.pyis theA fix). The five Ready-for-review boxes are settled at1df1d2db; the WindowsGlobalMemoryStatusExpath is honestly in Still to test.Description fix from the quality sweep: the
When it appliessection now names the A fix's check (tests/test_run_all_default_jobs.py, the check that fails onmainfor the missingdefault_jobsand passes here) and adds the missing AI-assistance box, recorded here because a literalAssisted-by:trailer would rewrite1df1d2db, the commit the suite ran. No code change; stays ready.1df1d2dbbda2dc538590AI-assistance corrected to real evidence: head
5c4051f4is1df1d2dbwith a message-onlyAssisted-by: DeepSeek V4.1 Flash 1017 (opencode-go)trailer, force-pushed with lease. Tree14636770is byte-identical (verified before and after the amend:1df1d2db^{tree}=5c4051f4^{tree}), so the suite at1df1d2db(912 / 23 / 18 in 440 s) describes the head's tree; the description now re-cites that tree. The AI box is ticked from the trailer.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.