WIP: fix(builder): account for available memory and compiler children on macOS #455

Draft
mooki wants to merge 3 commits from mooki/Longshot:mac/mem-gate into main
Member

What this is for: anyone building LS on a Mac. The builder's memory gate starts compilers only while there is memory
for them, and on macOS it reads nothing: free memory comes back unknown, and a compiler's children count as 0 MB. This PR
gives the gate native macOS readers, part of the macOS port (#420). What is measured: on an Apple M4 Max with macOS
26.7.1, tests/test_mem_gate.py and tests/test_mem_gate_children.py fail on main and pass here; the new
tests/test_mem_gate_darwin.py fails against main's build.py (2 of 13) and passes here; two interleaved runs each.
Also: three controlled 256 MiB allocations against vm_stat, a four-process tree against ps, the readers' cost, and the
whole suite compared by test name with main. Not measured: Windows and Linux runs, macOS 14.4, an Intel Mac, a
real build's compiler peaks, build speed.

Why

free_mb() and tree_rss() (nx2/build.py) read /proc and Windows' memory status. On a Mac /proc is missing, so
free_mb() returns None and tree_rss() raises, which the gate counts as 0 MB. The gate then admits compilers by the
worker limit alone, whatever memory is left.

What changed

  • nx2/build.py: on macOS, free_mb() and tree_rss() call nx2/darwin_memory.py (three lines each, and their
    docstrings). rss_mb(), the reserve, the floor of one compiler and every Windows and Linux line are unchanged.
  • nx2/darwin_memory.py (new), standard library ctypes only:
    • available memory: sysctlbyname reads the free, file-backed (pageable) and purgeable page counts, scaled by
      hw.pagesize. Speculative pages are counted once. Ordinary anonymous, wired, compressed and swapped memory is not
      counted as available. Each counter's returned length is honoured (8, 4, 4 and 8 bytes here);
    • a process tree's resident bytes: proc_pidinfo(PROC_PIDTASKINFO) for each process and proc_listchildpids for
      its children, growing the buffer when it is full, each PID counted once. A process that has gone or cannot be read
      counts 0, as on Linux.
  • tests/test_mem_gate_darwin.py (new): 13 cases from canned input, on every OS, with no Apple library loaded:
    counter lengths and page scaling, the reads being current, the libproc structure layout, child counts, descendants,
    duplicates and cycles, and build.py's own dispatch and process lifetime (--source selects another build.py).
  • docs/development.md: what the macOS estimate counts, and what it does not.
  • Unchanged: the existing memory tests (test_mem_gate.py, test_mem_gate_children.py, test_rss_liveness.py),
    every switch (LONGSHOT_JOBS, LONGSHOT_MEM_GATE, LONGSHOT_MEM_LIMIT_GB), nx2/make_game.py, and everything
    outside the builder.

Flags and settings

None. An unreadable free-memory value still returns None, and an unreadable process still counts 0.

Measurements

main: 5e072fc8c2adadba5ea610148be1c47800ba0cca (2026-11-14)
this PR: mac/mem-gate @ 5fc528d6c250471b25e520cf0e3e7880cf67bcfa
Hardware: Apple M4 Max, 16 cores, 64 GiB, 16 KiB pages, macOS 26.7.1 (25G241); Python 3.14.10 arm64. No GPU.
Scene: none. Game-free checks, plus allocation and process-tree probes from the first version (below).
Method: the memory checks through tests/run_all.py on main and on this PR, interleaved, two runs each; the new test
against main's build.py (--source) and this PR's, interleaved, two runs each. The whole suite on this PR, compared by
test name with a main run on the same Mac.
How to reproduce: python -I -B tests/run_all.py mem_gate rss_liveness, then
python -I -B tests/test_mem_gate_darwin.py --source <a checkout of main>/nx2/build.py.

main @ 5e072fc8 this PR @ 5fc528d6
tests/test_mem_gate.py (run 1 / run 2) FAIL / FAIL: the free memory reads (None) pass / pass
tests/test_mem_gate_children.py FAIL / FAIL: a driver's child counts (0 MB read, the child holds 300 MB) pass / pass
tests/test_mem_gate_darwin.py FAILED (failures=2) / the same, of 13 OK, 13 / the same
tests/test_mem_gate_leases.py, tests/test_rss_liveness.py pass / pass pass / pass
main @ 5e072fc8:
FAIL: the free memory reads (None)
FAIL: a driver's child counts (0 MB read, the child holds 300 MB)
2 passed, 2 failed, 0 skipped in 11 s (linux, 6 at once)
FAIL: test_build_free_memory_dispatch (__main__.DarwinMemoryTests.test_build_free_memory_dispatch)
AssertionError: None != 4321
FAIL: test_build_resident_dispatch_and_lifetime (__main__.DarwinMemoryTests.test_build_resident_dispatch_and_lifetime)
AssertionError: 0 != 321

this PR @ 5fc528d6:
5 passed, 0 failed, 0 skipped in 3 s (linux, 6 at once)
Ran 13 tests in 0.020s
OK

The readers' own measurements were made on the first version of this PR, on the old base 514591a3, and are
carried here. nx2/darwin_memory.py is byte-identical to the version measured at f6dc5135 (then 2b8c796f), and
free_mb()'s dispatch has not changed since; the process-tree reader was measured one commit earlier and is unchanged
(the first version's report, taken as reported):

  • Allocation: three 256 MiB allocations, held, then unmapped. Medians: 256.000 MiB resident increase, 266.500 MiB
    less available, 251.922 MiB back at once after the unmap (the rest of the busy Mac moves the numbers too; every trial
    is kept). In the trial compared step by step, the gate's readings stayed within 5 MiB of vm_stat's
    (free + file-backed + purgeable) / 64 MiB.
    The first reader, host_statistics64 (e517f675 here), was dropped: after an unmap it kept reading 29,662 MiB while
    vm_stat had moved to 29,919 MiB. Apple's host.c caches those statistics for one second for callers other than the
    system's.
  • A process tree: four processes holding 272 MiB. proc_pidinfo and ps agreed per process (43.53, 123.59, 59.63
    and 156.38 MiB); the gate counted 383 MiB, the grandchild included, against ps's 383.125.
  • Cost, medians of 1,000 warm reads: free_mb() 5.1 µs (p95 5.4 µs); one process's RSS 1.8 µs; a four-process
    tree 141 µs, timed before the dispatch moved behind tree_rss() (5fc528d6). ps takes 56.5 ms.

Testing

Tested: the above, and the whole suite, tests/run_all.py, 6 at once, on the same Mac:

main @ 5e072fc8:     525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once)
this PR @ 5fc528d6:  528 passed, 17 failed, 17 skipped in 169 s (linux, 6 at once)
  • By test name, three checks changed: test_mem_gate and test_mem_gate_children FAIL → pass, and the new
    test_mem_gate_darwin passes. test_mem_gate_leases and test_rss_liveness pass on both.
  • The 17 remaining failures also fail on main: test_appimage_package_retry, test_collect_wide, test_control,
    test_framegen_resources, test_game_layout, test_low_memory_notice, test_mirror_trim, test_module_ids,
    test_oot_state_tap, test_pad_layout, test_pcache_header_read, test_pipeline_cache_file,
    test_pipeline_cache_save, test_pipeline_play_cache, test_prof_symbols, test_program_cache_cleanup,
    test_program_cache_one_variant. The port's other PRs fix them, apart from three that are stale on main on every
    platform (test_framegen_resources, test_module_ids, test_pad_layout).
  • No crash report was left by this run.

Not tested:

  • Windows and Linux. Their code is unchanged; the new test runs there without any Apple library.
  • macOS 14.4 and an Intel Mac.
  • A real build: the compilers' memory estimates were not measured again on a Mac.

Needs other testers:

  • Windows and Linux: the memory checks and the new test, then the whole suite. Expect the same results as main and
    the new test passing (request to open).
  • macOS 14.4: the four sysctl values and the memory checks (request to open).

Screenshots

Nothing on screen.

Risk and rollback

  • It is an estimate of memory that can be had back, not Linux's MemAvailable and not a promise: file-backed pages may
    need writing or reading again, the three counters are not read at one instant, and other programs move them.
  • A process tree read races exits and PID reuse. A process counted twice is prevented within one read; shared mappings
    count in each process, as on Linux.
  • Apple's SDK marks libproc's header as private and subject to change, so the oldest supported macOS should run the
    checks.
  • Rollback: revert the PR.

Reviewer notes

  • Start with the two dispatches in build.py, then darwin_memory.py's free_mb().
  • The three commits are the history: e517f675 (the first reader), f6dc5135 (the sysctl reader that replaced it) and
    5fc528d6 (the dispatch moved behind tree_rss(), so rss_mb()'s lifetime rule, which test_rss_liveness.py
    checks, applies on macOS too).
  • Review all three commits, e517f675 to 5fc528d6: the whole branch is this PR's own (it depends on no other PR).

Issues

See #420.

Checklist

For everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description).
  • Rebased on current main. (Branched from 5e072fc8.)
  • python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above). (528 passed,
    17 skipped, and 17 failures that also fail on main, named under Testing.)
  • Tested in the game, or what wasn't is named under Testing. (No game run: the builder's memory gate; named under
    Testing.)
  • Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test
    requests are still to open.
  • Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (For a fix: the tests fail on main and pass here, two interleaved runs each.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.
  • Docs updated.
**What this is for:** anyone building LS on a Mac. The builder's memory gate starts compilers only while there is memory for them, and on macOS it reads nothing: free memory comes back unknown, and a compiler's children count as 0 MB. This PR gives the gate native macOS readers, part of the macOS port (#420). **What is measured:** on an Apple M4 Max with macOS 26.7.1, `tests/test_mem_gate.py` and `tests/test_mem_gate_children.py` fail on `main` and pass here; the new `tests/test_mem_gate_darwin.py` fails against `main`'s `build.py` (2 of 13) and passes here; two interleaved runs each. Also: three controlled 256 MiB allocations against `vm_stat`, a four-process tree against `ps`, the readers' cost, and the whole suite compared by test name with `main`. **Not measured:** Windows and Linux runs, macOS 14.4, an Intel Mac, a real build's compiler peaks, build speed. ## Why `free_mb()` and `tree_rss()` (`nx2/build.py`) read `/proc` and Windows' memory status. On a Mac `/proc` is missing, so `free_mb()` returns `None` and `tree_rss()` raises, which the gate counts as 0 MB. The gate then admits compilers by the worker limit alone, whatever memory is left. ## What changed - **`nx2/build.py`:** on macOS, `free_mb()` and `tree_rss()` call `nx2/darwin_memory.py` (three lines each, and their docstrings). `rss_mb()`, the reserve, the floor of one compiler and every Windows and Linux line are unchanged. - **`nx2/darwin_memory.py` (new), standard library `ctypes` only:** - **available memory:** `sysctlbyname` reads the free, file-backed (pageable) and purgeable page counts, scaled by `hw.pagesize`. Speculative pages are counted once. Ordinary anonymous, wired, compressed and swapped memory is not counted as available. Each counter's returned length is honoured (8, 4, 4 and 8 bytes here); - **a process tree's resident bytes:** `proc_pidinfo(PROC_PIDTASKINFO)` for each process and `proc_listchildpids` for its children, growing the buffer when it is full, each PID counted once. A process that has gone or cannot be read counts 0, as on Linux. - **`tests/test_mem_gate_darwin.py` (new):** 13 cases from canned input, on every OS, with no Apple library loaded: counter lengths and page scaling, the reads being current, the libproc structure layout, child counts, descendants, duplicates and cycles, and `build.py`'s own dispatch and process lifetime (`--source` selects another `build.py`). - **`docs/development.md`:** what the macOS estimate counts, and what it does not. - **Unchanged:** the existing memory tests (`test_mem_gate.py`, `test_mem_gate_children.py`, `test_rss_liveness.py`), every switch (`LONGSHOT_JOBS`, `LONGSHOT_MEM_GATE`, `LONGSHOT_MEM_LIMIT_GB`), `nx2/make_game.py`, and everything outside the builder. ## Flags and settings None. An unreadable free-memory value still returns `None`, and an unreadable process still counts 0. ## Measurements main: `5e072fc8c2adadba5ea610148be1c47800ba0cca` (2026-11-14) this PR: `mac/mem-gate` @ `5fc528d6c250471b25e520cf0e3e7880cf67bcfa` Hardware: Apple M4 Max, 16 cores, 64 GiB, 16 KiB pages, macOS 26.7.1 (25G241); Python 3.14.10 arm64. No GPU. Scene: none. Game-free checks, plus allocation and process-tree probes from the first version (below). Method: the memory checks through `tests/run_all.py` on `main` and on this PR, interleaved, two runs each; the new test against `main`'s `build.py` (`--source`) and this PR's, interleaved, two runs each. The whole suite on this PR, compared by test name with a `main` run on the same Mac. How to reproduce: `python -I -B tests/run_all.py mem_gate rss_liveness`, then `python -I -B tests/test_mem_gate_darwin.py --source <a checkout of main>/nx2/build.py`. | | main @ 5e072fc8 | this PR @ 5fc528d6 | |---|---|---| | `tests/test_mem_gate.py` (run 1 / run 2) | FAIL / FAIL: `the free memory reads (None)` | pass / pass | | `tests/test_mem_gate_children.py` | FAIL / FAIL: `a driver's child counts (0 MB read, the child holds 300 MB)` | pass / pass | | `tests/test_mem_gate_darwin.py` | `FAILED (failures=2)` / the same, of 13 | `OK`, 13 / the same | | `tests/test_mem_gate_leases.py`, `tests/test_rss_liveness.py` | pass / pass | pass / pass | ``` main @ 5e072fc8: FAIL: the free memory reads (None) FAIL: a driver's child counts (0 MB read, the child holds 300 MB) 2 passed, 2 failed, 0 skipped in 11 s (linux, 6 at once) FAIL: test_build_free_memory_dispatch (__main__.DarwinMemoryTests.test_build_free_memory_dispatch) AssertionError: None != 4321 FAIL: test_build_resident_dispatch_and_lifetime (__main__.DarwinMemoryTests.test_build_resident_dispatch_and_lifetime) AssertionError: 0 != 321 this PR @ 5fc528d6: 5 passed, 0 failed, 0 skipped in 3 s (linux, 6 at once) Ran 13 tests in 0.020s OK ``` **The readers' own measurements** were made on the first version of this PR, on the old base `514591a3`, and are carried here. `nx2/darwin_memory.py` is byte-identical to the version measured at `f6dc5135` (then `2b8c796f`), and `free_mb()`'s dispatch has not changed since; the process-tree reader was measured one commit earlier and is unchanged (the first version's report, taken as reported): - **Allocation:** three 256 MiB allocations, held, then unmapped. Medians: 256.000 MiB resident increase, 266.500 MiB less available, 251.922 MiB back at once after the unmap (the rest of the busy Mac moves the numbers too; every trial is kept). In the trial compared step by step, the gate's readings stayed within 5 MiB of `vm_stat`'s `(free + file-backed + purgeable) / 64` MiB. The first reader, `host_statistics64` (`e517f675` here), was dropped: after an unmap it kept reading 29,662 MiB while `vm_stat` had moved to 29,919 MiB. Apple's `host.c` caches those statistics for one second for callers other than the system's. - **A process tree:** four processes holding 272 MiB. `proc_pidinfo` and `ps` agreed per process (43.53, 123.59, 59.63 and 156.38 MiB); the gate counted 383 MiB, the grandchild included, against `ps`'s 383.125. - **Cost**, medians of 1,000 warm reads: `free_mb()` 5.1 µs (p95 5.4 µs); one process's RSS 1.8 µs; a four-process tree 141 µs, timed before the dispatch moved behind `tree_rss()` (`5fc528d6`). `ps` takes 56.5 ms. ## Testing **Tested:** the above, and the whole suite, `tests/run_all.py`, 6 at once, on the same Mac: ``` main @ 5e072fc8: 525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once) this PR @ 5fc528d6: 528 passed, 17 failed, 17 skipped in 169 s (linux, 6 at once) ``` - **By test name, three checks changed:** `test_mem_gate` and `test_mem_gate_children` FAIL → pass, and the new `test_mem_gate_darwin` passes. `test_mem_gate_leases` and `test_rss_liveness` pass on both. - **The 17 remaining failures also fail on `main`:** `test_appimage_package_retry`, `test_collect_wide`, `test_control`, `test_framegen_resources`, `test_game_layout`, `test_low_memory_notice`, `test_mirror_trim`, `test_module_ids`, `test_oot_state_tap`, `test_pad_layout`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_prof_symbols`, `test_program_cache_cleanup`, `test_program_cache_one_variant`. The port's other PRs fix them, apart from three that are stale on `main` on every platform (`test_framegen_resources`, `test_module_ids`, `test_pad_layout`). - No crash report was left by this run. **Not tested:** - **Windows and Linux.** Their code is unchanged; the new test runs there without any Apple library. - **macOS 14.4 and an Intel Mac.** - **A real build:** the compilers' memory estimates were not measured again on a Mac. **Needs other testers:** - Windows and Linux: the memory checks and the new test, then the whole suite. Expect the same results as `main` and the new test passing (request to open). - macOS 14.4: the four `sysctl` values and the memory checks (request to open). ## Screenshots Nothing on screen. ## Risk and rollback - It is an estimate of memory that can be had back, not Linux's `MemAvailable` and not a promise: file-backed pages may need writing or reading again, the three counters are not read at one instant, and other programs move them. - A process tree read races exits and PID reuse. A process counted twice is prevented within one read; shared mappings count in each process, as on Linux. - Apple's SDK marks libproc's header as private and subject to change, so the oldest supported macOS should run the checks. - Rollback: revert the PR. ## Reviewer notes - Start with the two dispatches in `build.py`, then `darwin_memory.py`'s `free_mb()`. - The three commits are the history: `e517f675` (the first reader), `f6dc5135` (the `sysctl` reader that replaced it) and `5fc528d6` (the dispatch moved behind `tree_rss()`, so `rss_mb()`'s lifetime rule, which `test_rss_liveness.py` checks, applies on macOS too). - Review all three commits, `e517f675` to `5fc528d6`: the whole branch is this PR's own (it depends on no other PR). ## Issues See #420. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. (Branched from 5e072fc8.) - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). (528 passed, 17 skipped, and 17 failures that also fail on `main`, named under Testing.) - [x] Tested in the game, or what wasn't is named under Testing. (No game run: the builder's memory gate; named under Testing.) - [ ] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test requests are still to open. - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (For a `fix`: the tests fail on `main` and pass here, two interleaved runs each.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] Docs updated.
Read the host VM statistics and compiler process trees through ctypes on
Darwin instead of falling through to /proc. Count speculative cache pages
once, exclude ordinary anonymous and compressed memory from the available
estimate, and preserve the gate's existing reserve and failure behavior.

Keep Windows and Linux reader text unchanged. Add portable canned-layout,
return-count, child-tree and Darwin dispatch tests, and document the estimate.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The controlled allocation probe found host_statistics64 returning the old
available-memory value after 256 MiB had been unmapped. Apple's host.c
caches non-platform queries for a one-second window after a small burst.

Use read-only sysctl VM counters for the same free, file-backed and
purgeable page classes. Cache only the library bindings and page size.
Decode the kernel's returned 32- or 64-bit integer lengths explicitly and
replace the obsolete Mach-layout fixtures with canned sysctl coverage.
The libproc reader, gate policy, and Linux and Windows paths are unchanged.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(builder): keep Darwin RSS behind the shared tree backend
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 9s
Third-party parts / As their notes say (pull_request) Successful in 11s
Website / Website on Cloudflare (pull_request_target) Has been skipped
5fc528d6c2
The full suite exposed the existing liveness contract through tree_rss and
tree_ws: a live root samples that byte-valued backend, read errors become
unknown RSS, and an exited root is never sampled. Dispatching Darwin in
rss_mb bypassed that seam and broke test_rss_liveness.py.

Move only the Darwin dispatch into tree_rss and restore rss_mb unchanged.
Keep the existing liveness fixture unchanged and extend the new canned
fixture's extracted functions to exercise the complete dispatch path.
The syscall readers, memory policy, and Windows/Linux bodies are unchanged.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 9s
Third-party parts / As their notes say (pull_request) Successful in 11s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'mac/mem-gate:mooki-mac/mem-gate'
git switch 'mooki-mac/mem-gate'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!455
No description provided.