WIP: fix(builder): account for available memory and compiler children on macOS #455
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!455
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:mac/mem-gate"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is for: anyone building LS on a Mac. The builder's memory gate starts compilers only while there is memory
for them, and on macOS it reads nothing: free memory comes back unknown, and a compiler's children count as 0 MB. This PR
gives the gate native macOS readers, part of the macOS port (#420). What is measured: on an Apple M4 Max with macOS
26.7.1,
tests/test_mem_gate.pyandtests/test_mem_gate_children.pyfail onmainand pass here; the newtests/test_mem_gate_darwin.pyfails againstmain'sbuild.py(2 of 13) and passes here; two interleaved runs each.Also: three controlled 256 MiB allocations against
vm_stat, a four-process tree againstps, the readers' cost, and thewhole suite compared by test name with
main. Not measured: Windows and Linux runs, macOS 14.4, an Intel Mac, areal build's compiler peaks, build speed.
Why
free_mb()andtree_rss()(nx2/build.py) read/procand Windows' memory status. On a Mac/procis missing, sofree_mb()returnsNoneandtree_rss()raises, which the gate counts as 0 MB. The gate then admits compilers by theworker limit alone, whatever memory is left.
What changed
nx2/build.py: on macOS,free_mb()andtree_rss()callnx2/darwin_memory.py(three lines each, and theirdocstrings).
rss_mb(), the reserve, the floor of one compiler and every Windows and Linux line are unchanged.nx2/darwin_memory.py(new), standard libraryctypesonly:sysctlbynamereads the free, file-backed (pageable) and purgeable page counts, scaled byhw.pagesize. Speculative pages are counted once. Ordinary anonymous, wired, compressed and swapped memory is notcounted as available. Each counter's returned length is honoured (8, 4, 4 and 8 bytes here);
proc_pidinfo(PROC_PIDTASKINFO)for each process andproc_listchildpidsforits children, growing the buffer when it is full, each PID counted once. A process that has gone or cannot be read
counts 0, as on Linux.
tests/test_mem_gate_darwin.py(new): 13 cases from canned input, on every OS, with no Apple library loaded:counter lengths and page scaling, the reads being current, the libproc structure layout, child counts, descendants,
duplicates and cycles, and
build.py's own dispatch and process lifetime (--sourceselects anotherbuild.py).docs/development.md: what the macOS estimate counts, and what it does not.test_mem_gate.py,test_mem_gate_children.py,test_rss_liveness.py),every switch (
LONGSHOT_JOBS,LONGSHOT_MEM_GATE,LONGSHOT_MEM_LIMIT_GB),nx2/make_game.py, and everythingoutside the builder.
Flags and settings
None. An unreadable free-memory value still returns
None, and an unreadable process still counts 0.Measurements
main:
5e072fc8c2adadba5ea610148be1c47800ba0cca(2026-11-14)this PR:
mac/mem-gate@5fc528d6c250471b25e520cf0e3e7880cf67bcfaHardware: Apple M4 Max, 16 cores, 64 GiB, 16 KiB pages, macOS 26.7.1 (25G241); Python 3.14.10 arm64. No GPU.
Scene: none. Game-free checks, plus allocation and process-tree probes from the first version (below).
Method: the memory checks through
tests/run_all.pyonmainand on this PR, interleaved, two runs each; the new testagainst
main'sbuild.py(--source) and this PR's, interleaved, two runs each. The whole suite on this PR, compared bytest name with a
mainrun on the same Mac.How to reproduce:
python -I -B tests/run_all.py mem_gate rss_liveness, thenpython -I -B tests/test_mem_gate_darwin.py --source <a checkout of main>/nx2/build.py.5e072fc85fc528d6tests/test_mem_gate.py(run 1 / run 2)the free memory reads (None)tests/test_mem_gate_children.pya driver's child counts (0 MB read, the child holds 300 MB)tests/test_mem_gate_darwin.pyFAILED (failures=2)/ the same, of 13OK, 13 / the sametests/test_mem_gate_leases.py,tests/test_rss_liveness.pyThe readers' own measurements were made on the first version of this PR, on the old base
514591a3, and arecarried here.
nx2/darwin_memory.pyis byte-identical to the version measured atf6dc5135(then2b8c796f), andfree_mb()'s dispatch has not changed since; the process-tree reader was measured one commit earlier and is unchanged(the first version's report, taken as reported):
less available, 251.922 MiB back at once after the unmap (the rest of the busy Mac moves the numbers too; every trial
is kept). In the trial compared step by step, the gate's readings stayed within 5 MiB of
vm_stat's(free + file-backed + purgeable) / 64MiB.The first reader,
host_statistics64(e517f675here), was dropped: after an unmap it kept reading 29,662 MiB whilevm_stathad moved to 29,919 MiB. Apple'shost.ccaches those statistics for one second for callers other than thesystem's.
proc_pidinfoandpsagreed per process (43.53, 123.59, 59.63and 156.38 MiB); the gate counted 383 MiB, the grandchild included, against
ps's 383.125.free_mb()5.1 µs (p95 5.4 µs); one process's RSS 1.8 µs; a four-processtree 141 µs, timed before the dispatch moved behind
tree_rss()(5fc528d6).pstakes 56.5 ms.Testing
Tested: the above, and the whole suite,
tests/run_all.py, 6 at once, on the same Mac:test_mem_gateandtest_mem_gate_childrenFAIL → pass, and the newtest_mem_gate_darwinpasses.test_mem_gate_leasesandtest_rss_livenesspass on both.main:test_appimage_package_retry,test_collect_wide,test_control,test_framegen_resources,test_game_layout,test_low_memory_notice,test_mirror_trim,test_module_ids,test_oot_state_tap,test_pad_layout,test_pcache_header_read,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_prof_symbols,test_program_cache_cleanup,test_program_cache_one_variant. The port's other PRs fix them, apart from three that are stale onmainon everyplatform (
test_framegen_resources,test_module_ids,test_pad_layout).Not tested:
Needs other testers:
mainandthe new test passing (request to open).
sysctlvalues and the memory checks (request to open).Screenshots
Nothing on screen.
Risk and rollback
MemAvailableand not a promise: file-backed pages mayneed writing or reading again, the three counters are not read at one instant, and other programs move them.
count in each process, as on Linux.
checks.
Reviewer notes
build.py, thendarwin_memory.py'sfree_mb().e517f675(the first reader),f6dc5135(thesysctlreader that replaced it) and5fc528d6(the dispatch moved behindtree_rss(), sorss_mb()'s lifetime rule, whichtest_rss_liveness.pychecks, applies on macOS too).
e517f675to5fc528d6: the whole branch is this PR's own (it depends on no other PR).Issues
See #420.
Checklist
For everyone
type(scope): description).main. (Branched from5e072fc8.)python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above). (528 passed,17 skipped, and 17 failures that also fail on
main, named under Testing.)Testing.)
requests are still to open.
fix: the tests fail onmainand pass here, two interleaved runs each.)third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.mainand passing here.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.