fix(builder): account for available memory and compiler children on macOS #455

Closed

mooki wants to merge 3 commits from mooki/Longshot:mac/mem-gate into main

pull from: mooki/Longshot:mac/mem-gate

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 1 Commits 3 Files changed 4 +344 -2

mooki commented 2026-11-14 14:42:09 +00:00

Member

• Copy link
• Copy Markdown

Type: fix · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (game-free checks and memory probes, no game run) · Needs: Windows and Linux, no GPU needed (the memory checks and the suite); macOS 14.4

Why

Anyone building LS on a Mac: the builder's memory gate starts compilers only while there is memory for them, and on
macOS it reads nothing. Free memory comes back unknown, and a compiler's children count as 0 MB, so the gate admits
compilers by the worker limit alone, whatever memory is left. This PR gives the gate native macOS readers, part of the
macOS port (#420).

free_mb() and tree_rss() (nx2/build.py) read /proc and Windows' memory status. On a Mac /proc is missing, so
free_mb() returns None and tree_rss() raises, which the gate counts as 0 MB.

How to reproduce

Seen on: macOS 26.7.1, Apple M4 Max (tests/test_mem_gate.py and tests/test_mem_gate_children.py on main). macOS
only: Windows and Linux read their own memory status.
Settings and switches: defaults (the memory gate on, as it ships).
Steps: on a Mac with main, python -I -B tests/run_all.py mem_gate rss_liveness.
Expected: the gate reads the memory available now and counts a compiler's children.
Actual: FAIL: the free memory reads (None) and FAIL: a driver's child counts (0 MB read, the child holds 300 MB).

What changed

On macOS, the builder's free_mb() and tree_rss() now call native readers in a new nx2/darwin_memory.py:
available memory from the kernel's VM counters, and a compiler tree's resident bytes from libproc. Every Windows and
Linux line is unchanged.

• Read first: the two dispatches in build.py, then darwin_memory.py's free_mb().
• nx2/build.py: on macOS, free_mb() and tree_rss() call nx2/darwin_memory.py (three lines each, and their
  docstrings). rss_mb(), the reserve, the floor of one compiler and every Windows and Linux line are unchanged.
• nx2/darwin_memory.py (new), standard library ctypes only:
  • available memory: sysctlbyname reads the free, file-backed (pageable) and purgeable page counts, scaled by
    hw.pagesize. Speculative pages are counted once. Ordinary anonymous, wired, compressed and swapped memory is not
    counted as available. Each counter's returned length is honoured (8, 4, 4 and 8 bytes here);
  • a process tree's resident bytes: proc_pidinfo(PROC_PIDTASKINFO) for each process and proc_listchildpids for
    its children, growing the buffer when it is full, each PID counted once. A process that has gone or cannot be read
    counts 0, as on Linux.
• tests/test_mem_gate_darwin.py (new): 13 cases from canned input, on every OS, with no Apple library loaded:
  counter lengths and page scaling, the reads being current, the libproc structure layout, child counts, descendants,
  duplicates and cycles, and build.py's own dispatch and process lifetime (--source selects another build.py).
• docs/development.md: what the macOS estimate counts, and what it does not.
• Unchanged: the existing memory tests (test_mem_gate.py, test_mem_gate_children.py, test_rss_liveness.py),
  every switch (LONGSHOT_JOBS, LONGSHOT_MEM_GATE, LONGSHOT_MEM_LIMIT_GB), nx2/make_game.py, and everything
  outside the builder.
• Design: the three commits are the history: e517f675 (the first reader), f6dc5135 (the sysctl reader that
  replaced it) and 5fc528d6 (the dispatch moved behind tree_rss(), so rss_mb()'s lifetime rule, which
  test_rss_liveness.py checks, applies on macOS too). Rebased, they are 0eb1aa01, 0e1505df and 41d17603. Review
  all three commits, to 41d17603: the whole branch is this PR's own (it depends on no other PR).

Measurements

Verdict: fixed on macOS: tests/test_mem_gate.py and tests/test_mem_gate_children.py fail on main and pass here;
the new tests/test_mem_gate_darwin.py fails against main's build.py (2 of 13) and passes here; two interleaved runs
each before the rebase, one each after it with the same results. The suite gains those 3 passes and no failure (534
passed, 30 failed against main's 531 and 32). Also: three controlled 256 MiB allocations against vm_stat, a
four-process tree against ps, and the readers' cost (below, from the first version).

Commits: main 6c9edffe5da1f5b5383977aeab3f544c0de25d4c (2026-11-14); this PR mac/mem-gate @
41d17603309e022d12fab4d2b04afe854baa17f8.
Hardware: Apple M4 Max, 16 cores, 64 GiB, 16 KiB pages, macOS 26.7.1 (25G241); Python 3.14.10 arm64. No GPU.
Scene: none. Game-free checks, plus allocation and process-tree probes from the first version (below).
Method: the memory checks through tests/run_all.py on main and on this PR; the new test against main's build.py
(--source) and this PR's. Before the rebase (on 5e072fc8) each ran twice, interleaved; after it, once each, with the
same results. The whole suite on this PR, compared by test name with a main run on the same Mac.
How to reproduce: python -I -B tests/run_all.py mem_gate rss_liveness, then
python -I -B tests/test_mem_gate_darwin.py --source <a checkout of main>/nx2/build.py.

	Before: main @ 6c9edffe	After: this PR @ 41d17603
tests/test_mem_gate.py (runs 1 and 2 before the rebase; the run after it the same)	FAIL / FAIL: the free memory reads (None)	pass / pass
tests/test_mem_gate_children.py	FAIL / FAIL: a driver's child counts (0 MB read, the child holds 300 MB)	pass / pass
tests/test_mem_gate_darwin.py	FAILED (failures=2) / the same, of 13	OK, 13 / the same
tests/test_mem_gate_leases.py, tests/test_rss_liveness.py	pass / pass	pass / pass

main @ 6c9edffe:
FAIL: the free memory reads (None)
FAIL: a driver's child counts (0 MB read, the child holds 300 MB)
0 passed, 2 failed, 0 skipped in 10 s (linux, 6 at once)   (test_mem_gate and test_mem_gate_children only)
FAIL: test_build_free_memory_dispatch (__main__.DarwinMemoryTests.test_build_free_memory_dispatch)
AssertionError: None != 4321
FAIL: test_build_resident_dispatch_and_lifetime (__main__.DarwinMemoryTests.test_build_resident_dispatch_and_lifetime)
AssertionError: 0 != 321

this PR @ 41d17603:
5 passed, 0 failed, 0 skipped in 2 s (linux, 6 at once)
Ran 13 tests in 0.025s
OK

The readers' own measurements were made on the first version of this PR, on the old base 514591a3, and are
carried here. nx2/darwin_memory.py is byte-identical to the version measured at f6dc5135 (then 2b8c796f), and
free_mb()'s dispatch has not changed since; the process-tree reader was measured one commit earlier and is unchanged
(the first version's report, taken as reported):

• Allocation: three 256 MiB allocations, held, then unmapped. Medians: 256.000 MiB resident increase, 266.500 MiB
  less available, 251.922 MiB back at once after the unmap (the rest of the busy Mac moves the numbers too; every trial
  is kept). In the trial compared step by step, the gate's readings stayed within 5 MiB of vm_stat's
  (free + file-backed + purgeable) / 64 MiB.
  The first reader, host_statistics64 (e517f675 here), was dropped: after an unmap it kept reading 29,662 MiB while
  vm_stat had moved to 29,919 MiB. Apple's host.c caches those statistics for one second for callers other than the
  system's.
• A process tree: four processes holding 272 MiB. proc_pidinfo and ps agreed per process (43.53, 123.59, 59.63
  and 156.38 MiB); the gate counted 383 MiB, the grandchild included, against ps's 383.125.
• Cost, medians of 1,000 warm reads: free_mb() 5.1 µs (p95 5.4 µs); one process's RSS 1.8 µs; a four-process
  tree 141 µs, timed before the dispatch moved behind tree_rss() (5fc528d6). ps takes 56.5 ms.

Screenshots: none; nothing on screen.

Flags and settings

None. An unreadable free-memory value still returns None, and an unreadable process still counts 0.

Testing

Tested: the above, and the whole suite, tests/run_all.py, 6 at once, on the same Mac:

main @ 6c9edffe:     531 passed, 32 failed, 17 skipped in 181 s (linux, 6 at once)
this PR @ 41d17603:  534 passed, 30 failed, 17 skipped in 174 s (linux, 6 at once)

• By test name, against main 6c9edffe's run on the same Mac: 3 statuses changed: test_mem_gate_darwin, absent on main,
  passes here; test_mem_gate: FAIL on main, pass here; test_mem_gate_children: FAIL on main, pass here.
• These 30 checks fail here and on main on this Mac, by name (not triaged here: some need the macOS port's other pull
  requests, some are checks main added or changed since 5e072fc8): test_appimage_package_retry,
  test_appimage_player_build, test_collect_wide, test_control, test_cpm_discovery, test_crash_dialog,
  test_crash_record, test_env_on, test_game_layout, test_lift_cleanup_error, test_linux_preflight,
  test_longshot_ctl, test_low_memory_notice, test_make_game_cache, test_mirror_trim, test_oot_state_tap,
  test_pcache_header_read, test_pipeline_cache_file, test_pipeline_cache_save, test_pipeline_play_cache,
  test_pipeline_tour, test_prof_symbols, test_program_cache_cleanup, test_program_cache_one_variant,
  test_readback_late, test_report_run, test_split_giants, test_vk_pgo, test_vram_budget_row, test_walk_key.
• The same 17 checks skipped as on main.
• Crash reports during the run: none (main's run: fault-arm64-2026-11-14-180704.ips,
  recorder_sync_test.exe-2026-11-14-180744.ips).
• After the rebase onto main aa95d35f, at 82c2e103 (41d17603 before the rebases onto e090cc48 and aa95d35f):
  Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they
  apply) gave the same results as before; python -I -B tests/run_all.py on wave S1's merge 11a5b4e7, which carries
  this head: 568/22/17 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure
  (reports/2026-11-15-rebase-sweep-3.md).

Not tested:

• Windows and Linux runs. Their code is unchanged (the dispatches are behind sys.platform == "darwin"; only the
  two docstrings changed); the new test runs there without any Apple library, but nobody has run it there.
• macOS 14.4 and an Intel Mac.
• A real build: the compilers' memory estimates (a real build's compiler peaks) were not measured again on a Mac,
  nor build speed.

Must be tested:

• Windows and Linux, no GPU needed: on this PR (head 82c2e103) and main aa95d35f, run
  python -I -B tests/run_all.py mem_gate rss_liveness (it includes the new test_mem_gate_darwin.py), then the
  whole suite, expecting the same results as main by test name and the new test passing (13 tests, no Apple library
  loaded), where main has no such test (by hand: test request T-58EG)
• macOS 14.4, an Apple silicon Mac: on this PR, sysctl vm.page_free_count vm.page_pageable_external_count vm.page_purgeable_count hw.pagesize and python -I -B tests/run_all.py mem_gate rss_liveness, expecting the four
  values to read and the memory checks to pass as on macOS 26.7.1 (by hand: anyone with a Mac on macOS 14.4; not part
  of this PR's test request)

Risks

What could break: low. On Windows and Linux nothing runs differently. On macOS:

• It is an estimate of memory that can be had back, not Linux's MemAvailable and not a promise: file-backed pages may
  need writing or reading again, the three counters are not read at one instant, and other programs move them.
• A process tree read races exits and PID reuse. A process counted twice is prevented within one read; shared mappings
  count in each process, as on Linux.
• Apple's SDK marks libproc's header as private and subject to change, so the oldest supported macOS should run the
  checks.
• Landing order: rebased on main 6c9edffe (2026-11-14) without a conflict. Main changed nx2/build.py in seven commits
  since this branch's base (#371, #385, #370, #573, #524, #569, #574) and docs/development.md often; this PR's hunks
  applied as they were, and nx2/darwin_memory.py is byte for byte the measured one. No conflict with the author's
  other open PRs as rebased on 6c9edffe (git merge-tree of the heads, 2026-11-14).

Turning it off: LONGSHOT_MEM_GATE=0 turns the memory check off, as before; or revert the PR.

Compatibility: the builder's tools: on macOS the builder now admits compilers by the memory estimate, as it does on
Windows and Linux. Nothing changes for saves, the options file, the caches or the vk ABI.

Issues

See #420.

Checklist

Everyone

• One topic.
• The title is a Conventional Commit (type(scope): description) of the status line's type.
• Why gives the reason plainly and links the issues that fit (the tracker searched).
• Measurements has a verdict, or one line saying why there is nothing to measure.
• Testing names what was tested, what wasn't, and what must still be tested, where and how.
• Rebased on current main: on aa95d35f (#761, 2026-11-15), head 82c2e103, without a conflict. (The e090cc48
  rebase's one, docs/development.md with #314's reworded prompts paragraph, stays resolved: this PR's macOS memory
  paragraph first; this PR's own commits, old → new: 0eb1aa01 → 8f8e03b7, 0e1505df → 5240e431, 41d17603 →
  82c2e103; the commit names elsewhere in this description are from before the rebase.)
• python -I -B tests/run_all.py passes (any failure that also fails on main is named). On macOS: 534 passed, 17
  skipped, and 30 failures that also fail on main 6c9edffe on this Mac, named under Testing. After the rebase onto main
  aa95d35f (head 82c2e103): wave S1's merge 11a5b4e7, 568/22/17 against main aa95d35f's 543/33/17, no new failure.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• A game-free test added for the fix, failing on main and passing here.
• Docs updated.

**Type:** fix · **Risk:** low · **Tested on:** macOS 26.7.1, Apple M4 Max (game-free checks and memory probes, no game run) · **Needs:** Windows and Linux, no GPU needed (the memory checks and the suite); macOS 14.4 ## Why Anyone building LS on a Mac: the builder's memory gate starts compilers only while there is memory for them, and on macOS it reads nothing. Free memory comes back unknown, and a compiler's children count as 0 MB, so the gate admits compilers by the worker limit alone, whatever memory is left. This PR gives the gate native macOS readers, part of the macOS port (#420). `free_mb()` and `tree_rss()` (`nx2/build.py`) read `/proc` and Windows' memory status. On a Mac `/proc` is missing, so `free_mb()` returns `None` and `tree_rss()` raises, which the gate counts as 0 MB. ## How to reproduce **Seen on:** macOS 26.7.1, Apple M4 Max (`tests/test_mem_gate.py` and `tests/test_mem_gate_children.py` on main). macOS only: Windows and Linux read their own memory status. **Settings and switches:** defaults (the memory gate on, as it ships). **Steps:** on a Mac with main, `python -I -B tests/run_all.py mem_gate rss_liveness`. **Expected:** the gate reads the memory available now and counts a compiler's children. **Actual:** `FAIL: the free memory reads (None)` and `FAIL: a driver's child counts (0 MB read, the child holds 300 MB)`. ## What changed On macOS, the builder's `free_mb()` and `tree_rss()` now call native readers in a new `nx2/darwin_memory.py`: available memory from the kernel's VM counters, and a compiler tree's resident bytes from libproc. Every Windows and Linux line is unchanged. - Read first: the two dispatches in `build.py`, then `darwin_memory.py`'s `free_mb()`. - **`nx2/build.py`:** on macOS, `free_mb()` and `tree_rss()` call `nx2/darwin_memory.py` (three lines each, and their docstrings). `rss_mb()`, the reserve, the floor of one compiler and every Windows and Linux line are unchanged. - **`nx2/darwin_memory.py` (new), standard library `ctypes` only:** - **available memory:** `sysctlbyname` reads the free, file-backed (pageable) and purgeable page counts, scaled by `hw.pagesize`. Speculative pages are counted once. Ordinary anonymous, wired, compressed and swapped memory is not counted as available. Each counter's returned length is honoured (8, 4, 4 and 8 bytes here); - **a process tree's resident bytes:** `proc_pidinfo(PROC_PIDTASKINFO)` for each process and `proc_listchildpids` for its children, growing the buffer when it is full, each PID counted once. A process that has gone or cannot be read counts 0, as on Linux. - **`tests/test_mem_gate_darwin.py` (new):** 13 cases from canned input, on every OS, with no Apple library loaded: counter lengths and page scaling, the reads being current, the libproc structure layout, child counts, descendants, duplicates and cycles, and `build.py`'s own dispatch and process lifetime (`--source` selects another `build.py`). - **`docs/development.md`:** what the macOS estimate counts, and what it does not. - **Unchanged:** the existing memory tests (`test_mem_gate.py`, `test_mem_gate_children.py`, `test_rss_liveness.py`), every switch (`LONGSHOT_JOBS`, `LONGSHOT_MEM_GATE`, `LONGSHOT_MEM_LIMIT_GB`), `nx2/make_game.py`, and everything outside the builder. - Design: the three commits are the history: `e517f675` (the first reader), `f6dc5135` (the `sysctl` reader that replaced it) and `5fc528d6` (the dispatch moved behind `tree_rss()`, so `rss_mb()`'s lifetime rule, which `test_rss_liveness.py` checks, applies on macOS too). Rebased, they are `0eb1aa01`, `0e1505df` and `41d17603`. Review all three commits, to `41d17603`: the whole branch is this PR's own (it depends on no other PR). ## Measurements **Verdict:** fixed on macOS: `tests/test_mem_gate.py` and `tests/test_mem_gate_children.py` fail on main and pass here; the new `tests/test_mem_gate_darwin.py` fails against main's `build.py` (2 of 13) and passes here; two interleaved runs each before the rebase, one each after it with the same results. The suite gains those 3 passes and no failure (534 passed, 30 failed against main's 531 and 32). Also: three controlled 256 MiB allocations against `vm_stat`, a four-process tree against `ps`, and the readers' cost (below, from the first version). Commits: main `6c9edffe5da1f5b5383977aeab3f544c0de25d4c` (2026-11-14); this PR `mac/mem-gate` @ `41d17603309e022d12fab4d2b04afe854baa17f8`. Hardware: Apple M4 Max, 16 cores, 64 GiB, 16 KiB pages, macOS 26.7.1 (25G241); Python 3.14.10 arm64. No GPU. Scene: none. Game-free checks, plus allocation and process-tree probes from the first version (below). Method: the memory checks through `tests/run_all.py` on `main` and on this PR; the new test against `main`'s `build.py` (`--source`) and this PR's. Before the rebase (on 5e072fc8) each ran twice, interleaved; after it, once each, with the same results. The whole suite on this PR, compared by test name with a `main` run on the same Mac. How to reproduce: `python -I -B tests/run_all.py mem_gate rss_liveness`, then `python -I -B tests/test_mem_gate_darwin.py --source <a checkout of main>/nx2/build.py`. | | Before: main @ 6c9edffe | After: this PR @ 41d17603 | |---|---|---| | `tests/test_mem_gate.py` (runs 1 and 2 before the rebase; the run after it the same) | FAIL / FAIL: `the free memory reads (None)` | pass / pass | | `tests/test_mem_gate_children.py` | FAIL / FAIL: `a driver's child counts (0 MB read, the child holds 300 MB)` | pass / pass | | `tests/test_mem_gate_darwin.py` | `FAILED (failures=2)` / the same, of 13 | `OK`, 13 / the same | | `tests/test_mem_gate_leases.py`, `tests/test_rss_liveness.py` | pass / pass | pass / pass | ``` main @ 6c9edffe: FAIL: the free memory reads (None) FAIL: a driver's child counts (0 MB read, the child holds 300 MB) 0 passed, 2 failed, 0 skipped in 10 s (linux, 6 at once) (test_mem_gate and test_mem_gate_children only) FAIL: test_build_free_memory_dispatch (__main__.DarwinMemoryTests.test_build_free_memory_dispatch) AssertionError: None != 4321 FAIL: test_build_resident_dispatch_and_lifetime (__main__.DarwinMemoryTests.test_build_resident_dispatch_and_lifetime) AssertionError: 0 != 321 this PR @ 41d17603: 5 passed, 0 failed, 0 skipped in 2 s (linux, 6 at once) Ran 13 tests in 0.025s OK ``` **The readers' own measurements** were made on the first version of this PR, on the old base `514591a3`, and are carried here. `nx2/darwin_memory.py` is byte-identical to the version measured at `f6dc5135` (then `2b8c796f`), and `free_mb()`'s dispatch has not changed since; the process-tree reader was measured one commit earlier and is unchanged (the first version's report, taken as reported): - **Allocation:** three 256 MiB allocations, held, then unmapped. Medians: 256.000 MiB resident increase, 266.500 MiB less available, 251.922 MiB back at once after the unmap (the rest of the busy Mac moves the numbers too; every trial is kept). In the trial compared step by step, the gate's readings stayed within 5 MiB of `vm_stat`'s `(free + file-backed + purgeable) / 64` MiB. The first reader, `host_statistics64` (`e517f675` here), was dropped: after an unmap it kept reading 29,662 MiB while `vm_stat` had moved to 29,919 MiB. Apple's `host.c` caches those statistics for one second for callers other than the system's. - **A process tree:** four processes holding 272 MiB. `proc_pidinfo` and `ps` agreed per process (43.53, 123.59, 59.63 and 156.38 MiB); the gate counted 383 MiB, the grandchild included, against `ps`'s 383.125. - **Cost**, medians of 1,000 warm reads: `free_mb()` 5.1 µs (p95 5.4 µs); one process's RSS 1.8 µs; a four-process tree 141 µs, timed before the dispatch moved behind `tree_rss()` (`5fc528d6`). `ps` takes 56.5 ms. Screenshots: none; nothing on screen. ### Flags and settings None. An unreadable free-memory value still returns `None`, and an unreadable process still counts 0. ## Testing **Tested:** the above, and the whole suite, `tests/run_all.py`, 6 at once, on the same Mac: ``` main @ 6c9edffe: 531 passed, 32 failed, 17 skipped in 181 s (linux, 6 at once) this PR @ 41d17603: 534 passed, 30 failed, 17 skipped in 174 s (linux, 6 at once) ``` - By test name, against main 6c9edffe's run on the same Mac: 3 statuses changed: `test_mem_gate_darwin`, absent on main, passes here; `test_mem_gate`: FAIL on main, pass here; `test_mem_gate_children`: FAIL on main, pass here. - These 30 checks fail here and on main on this Mac, by name (not triaged here: some need the macOS port's other pull requests, some are checks main added or changed since 5e072fc8): `test_appimage_package_retry`, `test_appimage_player_build`, `test_collect_wide`, `test_control`, `test_cpm_discovery`, `test_crash_dialog`, `test_crash_record`, `test_env_on`, `test_game_layout`, `test_lift_cleanup_error`, `test_linux_preflight`, `test_longshot_ctl`, `test_low_memory_notice`, `test_make_game_cache`, `test_mirror_trim`, `test_oot_state_tap`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_pipeline_tour`, `test_prof_symbols`, `test_program_cache_cleanup`, `test_program_cache_one_variant`, `test_readback_late`, `test_report_run`, `test_split_giants`, `test_vk_pgo`, `test_vram_budget_row`, `test_walk_key`. - The same 17 checks skipped as on main. - Crash reports during the run: none (main's run: fault-arm64-2026-11-14-180704.ips, recorder_sync_test.exe-2026-11-14-180744.ips). - **After the rebase onto main aa95d35f**, at `82c2e103` (`41d17603` before the rebases onto e090cc48 and aa95d35f): Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they apply) gave the same results as before; `python -I -B tests/run_all.py` on wave S1's merge `11a5b4e7`, which carries this head: 568/22/17 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure (`reports/2026-11-15-rebase-sweep-3.md`). **Not tested:** - **Windows and Linux runs.** Their code is unchanged (the dispatches are behind `sys.platform == "darwin"`; only the two docstrings changed); the new test runs there without any Apple library, but nobody has run it there. - **macOS 14.4 and an Intel Mac.** - **A real build:** the compilers' memory estimates (a real build's compiler peaks) were not measured again on a Mac, nor build speed. **Must be tested:** - Windows and Linux, no GPU needed: on this PR (head 82c2e103) and main aa95d35f, run `python -I -B tests/run_all.py mem_gate rss_liveness` (it includes the new `test_mem_gate_darwin.py`), then the whole suite, expecting the same results as main by test name and the new test passing (13 tests, no Apple library loaded), where main has no such test (by hand: test request T-58EG) - macOS 14.4, an Apple silicon Mac: on this PR, `sysctl vm.page_free_count vm.page_pageable_external_count vm.page_purgeable_count hw.pagesize` and `python -I -B tests/run_all.py mem_gate rss_liveness`, expecting the four values to read and the memory checks to pass as on macOS 26.7.1 (by hand: anyone with a Mac on macOS 14.4; not part of this PR's test request) ## Risks **What could break:** low. On Windows and Linux nothing runs differently. On macOS: - It is an estimate of memory that can be had back, not Linux's `MemAvailable` and not a promise: file-backed pages may need writing or reading again, the three counters are not read at one instant, and other programs move them. - A process tree read races exits and PID reuse. A process counted twice is prevented within one read; shared mappings count in each process, as on Linux. - Apple's SDK marks libproc's header as private and subject to change, so the oldest supported macOS should run the checks. - Landing order: rebased on main 6c9edffe (2026-11-14) without a conflict. Main changed `nx2/build.py` in seven commits since this branch's base (#371, #385, #370, #573, #524, #569, #574) and `docs/development.md` often; this PR's hunks applied as they were, and `nx2/darwin_memory.py` is byte for byte the measured one. No conflict with the author's other open PRs as rebased on 6c9edffe (`git merge-tree` of the heads, 2026-11-14). **Turning it off:** `LONGSHOT_MEM_GATE=0` turns the memory check off, as before; or revert the PR. **Compatibility:** the builder's tools: on macOS the builder now admits compilers by the memory estimate, as it does on Windows and Linux. Nothing changes for saves, the options file, the caches or the vk ABI. ## Issues See #420. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`: on aa95d35f (#761, 2026-11-15), head `82c2e103`, without a conflict. (The e090cc48 rebase's one, `docs/development.md` with #314's reworded prompts paragraph, stays resolved: this PR's macOS memory paragraph first; this PR's own commits, old → new: `0eb1aa01` → `8f8e03b7`, `0e1505df` → `5240e431`, `41d17603` → `82c2e103`; the commit names elsewhere in this description are from before the rebase.) - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). On macOS: 534 passed, 17 skipped, and 30 failures that also fail on main 6c9edffe on this Mac, named under Testing. After the rebase onto main aa95d35f (head `82c2e103`): wave S1's merge `11a5b4e7`, 568/22/17 against main aa95d35f's 543/33/17, no new failure. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] Docs updated.

mooki added 3 commits 2026-11-14 14:42:09 +00:00

fix(builder): account for available memory and compiler children on macOS ... e517f675d5

Read the host VM statistics and compiler process trees through ctypes on
Darwin instead of falling through to /proc. Count speculative cache pages
once, exclude ordinary anonymous and compressed memory from the available
estimate, and preserve the gate's existing reserve and failure behavior.

Keep Windows and Linux reader text unchanged. Add portable canned-layout,
return-count, child-tree and Darwin dispatch tests, and document the estimate.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(builder): avoid stale Darwin available-memory snapshots ... f6dc513544

The controlled allocation probe found host_statistics64 returning the old
available-memory value after 256 MiB had been unmapped. Apple's host.c
caches non-platform queries for a one-second window after a small burst.

Use read-only sysctl VM counters for the same free, file-backed and
purgeable page classes. Cache only the library bindings and page size.
Decode the kernel's returned 32- or 64-bit integer lengths explicitly and
replace the obsolete Mach-layout fixtures with canned sysctl coverage.
The libproc reader, gate policy, and Linux and Windows paths are unchanged.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(builder): keep Darwin RSS behind the shared tree backend ... 5fc528d6c2

The full suite exposed the existing liveness contract through tree_rss and
tree_ws: a live root samples that byte-valued backend, read errors become
unknown RSS, and an exited root is never sampled. Dispatching Darwin in
rss_mb bypassed that seam and broke test_rss_liveness.py.

Move only the Darwin dispatch into tree_rss and restore rss_mb unchanged.
Keep the existing liveness fixture unchanged and extend the new canned
fixture's extracted functions to exercise the complete dispatch path.
The syscall readers, memory policy, and Windows/Linux bodies are unchanged.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

mooki added the

kind

bug

priority

high

labels 2026-11-14 14:57:35 +00:00

mooki added the

area/install

label 2026-11-14 16:28:06 +00:00

mooki added the

platform/macos

label 2026-11-14 18:24:10 +00:00

mooki force-pushed mac/mem-gate from 5fc528d6c2 to 41d1760330 2026-11-15 01:52:07 +00:00 Compare

mooki force-pushed mac/mem-gate from 41d1760330 to 3dcbb58005 2026-11-15 07:13:57 +00:00 Compare

mooki force-pushed mac/mem-gate from 3dcbb58005 to 82c2e10341 2026-11-15 13:05:33 +00:00 Compare

mooki changed title from WIP: fix(builder): account for available memory and compiler children on macOS to fix(builder): account for available memory and compiler children on macOS 2026-11-15 13:15:34 +00:00

mooki referenced this pull request from a commit 2026-11-15 17:38:50 +00:00

fix(builder): account for available memory and compiler children on macOS

mooki referenced this pull request 2026-11-15 17:39:32 +00:00

WIP: feat(macos): run the host on macOS: build, runtime, control, crash records and the Vulkan loader #845

MaikelJSQ referenced this pull request 2026-11-15 18:05:02 +00:00

Game: Longshot closed unexpectedly while you were playing #859

mooki commented 2026-11-15 18:07:19 +00:00

Author

Member

• Copy link
• Copy Markdown

Folded into #845 (feat(macos): run the host on macOS: build, runtime, control, crash records and the Vulkan loader)

Folded into #845 (feat(macos): run the host on macOS: build, runtime, control, crash records and the Vulkan loader)

mooki closed this pull request 2026-11-15 18:07:21 +00:00

Some checks failed

Hide checks

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 4s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 7s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s

Details

Third-party parts / As their notes say (pull_request) Successful in 9s

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

beanstalk: approved @ 1697603

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 2c5a62e

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 4fd82b7

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 64d6d09

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 6f91fae

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 70a78f9

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 8d361dc

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 9d8356e

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ abf2227

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ d82f202

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ e5f1e40

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ ee8ab5f

Set by Beanstalk for show; the farm's database is what counts

  

ci: approved

A maintainer approved CircleCI for this fork PR (sticky; removed after 14 quiet days)

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Affects Linux (a change for every system carries both Windows and Linux)

  

platform/macos

macOS (Apple silicon or Intel Macs)

  

platform/steam-deck

Steam Deck

  

platform/windows

Affects Windows (a change for every system carries both Windows and Linux)

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

regression

Worked in an earlier build, broken in a newer one: highest priority, above other bugs

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

  

ci: approved

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

beanstalk: approved @ 1697603

beanstalk: approved @ 2c5a62e

beanstalk: approved @ 4fd82b7

beanstalk: approved @ 64d6d09

beanstalk: approved @ 6f91fae

beanstalk: approved @ 70a78f9

beanstalk: approved @ 8d361dc

beanstalk: approved @ 9d8356e

beanstalk: approved @ abf2227

beanstalk: approved @ d82f202

beanstalk: approved @ e5f1e40

beanstalk: approved @ ee8ab5f

ci: approved

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/macos

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

regression

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

ci: approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!455

No description provided.