WIP: build(macos): classify Darwin helpers and package a relocatable macOS 14.4 dylib set #479

Draft
mooki wants to merge 13 commits from mooki/Longshot:mac/build-complete into main
Member

What this is for: the macOS port (#420). The macOS build-flags PR (#453) forms Mach-O commands, but its source list
takes every host/posix/darwin_*.c into the executable, and the macOS game folder gets the executable, the renderer and
one SDL file, with the libraries still loaded from wherever they were installed (Homebrew's are arm64 only, macOS 26).
This PR says which macOS helper goes into which image, checks every library the game will load against the target
(architecture, macOS 14.4, the whole dependency closure, how dyld will bind it), and packages a game folder that carries
its own copies, relocated and signed. Windows and Linux are unchanged: their compile and link commands are the same 36
arrays, and their game-folder staging is the same code. What is measured: three game-free tests (the macOS
commands, the package, the game folder's staging) fail on main and on its two dependencies alone, and pass here, on
arm64 and on x86_64 under Rosetta, two interleaved rounds. Also the full game-free suite on an Apple M4 Max (macOS
26.7.1), compared by test name with main. Not measured: Windows, Linux, a complete game build or run, macOS 14.4
itself, an Intel Mac.

Depends on #453 (mac/build-flags) and #437 (mac/waits): this branch starts at #453's head 4723ee36 and merges
#437's head 0d62666d unchanged (merge d59e4c32). Review d59e4c32..fff15660, this PR's own four commits.

Why

A Mac game folder has to run on another Mac. With #453 alone:

  • the executable and the renderer both take every darwin_*.c that exists, so a helper that only the executable may own
    (the wait state, the guest memory's owner, the profiler, the control sockets, the Vulkan loader) would get a second
    copy in the renderer, with the host's logger and SDL calls in it;
  • the libraries come from wherever pkg-config found them, checked only by a tiny test link. A link that succeeds with
    warnings is no proof: Homebrew's FFmpeg and SDL are arm64 only and need macOS 26, so the folder would not start on
    x86_64 or on macOS 14.4;
  • the folder refers to those libraries by their installed paths, so it only starts on the Mac that built it.

What changed

  • build: nx2/darwin_build.py.
    • An explicit table of the macOS helpers. Both images: darwin_fault.c, darwin_fiber.c, darwin_mem.c,
      darwin_process.c, darwin_wait.c. The executable only: darwin_control_sockets.c, darwin_cpu.c,
      darwin_guest_mem.c, darwin_prof.c, darwin_vulkan.c, darwin_wait_state.c. A darwin_*.c the table does not
      name stops the selection (unclassified), instead of being taken into both. Only the files present are compiled,
      so the helpers of PRs not merged yet are simply absent.
    • dependencies(): FFmpeg, SDL2, SDL3 and MoltenVK resolved to exact files once, and the same files used for the
      preflight, the link and the package. Every non-system -l becomes a file path before linking.
    • SDL2-compat loads SDL3 by name at run time, not through a load command, so an SDL2 file that names SDL3 anywhere
      requires LONGSHOT_SDL3.
    • The cache key of macOS objects includes the compiler's real path, version and bytes, and the SDK settings' bytes.
  • build: nx2/darwin_package.py (new).
    • A Mach-O reader: thin, universal and ar files; the target slice; the minimum macOS; the recursive dylib closure.
    • The imports dyld will really bind, from the LC_DYLD_INFO bind, weak-bind and lazy-bind opcodes and the
      LC_DYLD_CHAINED_FIXUPS import table as well as the symbol table. Flat-namespace and main-executable imports, and
      library ordinals that name no loaded library, are refused.
    • After both links: copies the admitted closure into owned staging, thins universal files to the target, rewrites
      non-system loads to @loader_path/<name>, removes external rpaths and signs each copy ad hoc (codesign --sign -).
      Only checked copies replace the build output; the original library files are never written.
    • longshot-macos.json lists every output file with its hash. Only alias entries (libSDL3.dylib →
      libSDL3.0.dylib) may be symlinks.
  • build: nx2/darwin_archive.py (new). FFmpeg's NASM objects (x86_64) carry no minimum-macOS command. They are refused
    unless LONGSHOT_DARWIN_ARCHIVE_RECEIPT and LONGSHOT_DARWIN_ARCHIVE_RECEIPT_SHA256 name one exact receipt binding
    each such member's bytes, sources, assembler command, imports and instruction set.
  • build: nx2/build.py on macOS: the dependencies are admitted before any compile of a standalone build; the
    executable link gets -undefined error too, and both links -headerpad_max_install_names (room for the rewritten
    paths); the package step runs after both links. The old one-file SDL copy is gone.
  • build: nx2/make_game.py on macOS: the preflight uses the same dependency plan and inspects its test links' products;
    the game-folder stage copies and checks every file in longshot-macos.json (aliases kept as links), and removes only
    unchanged files an earlier manifest owned. On Windows and Linux every changed line keeps its old branch.
  • build: fff15660 is new: #453 moved architecture() and darwin_sdk() from a64-kit's toolchain.py into
    nx2/darwin_toolchain.py, so the dependency plan and the post-link check ask darwin_toolchain for them. Without it
    both stop with AttributeError on a Mac.
  • tests:
    • tests/test_macos_build_flags.py: the helper table; strict links (no flat namespace, no dynamic_lookup); the
      preflight's commands; and a real link of darwin_wait_state.c with the production host flags, read back from two
      separately linked dylibs (the wait PR's shared state).
    • tests/test_macos_package.py (new): dependency and receipt refusals on authored metadata (every OS), and on a Mac
      real C/C++ products linked with the production commands, packaged, moved and loaded with their original folder
      hidden; real chained-fixup and opcode-stream links. --arch selects the routes, --fixtures-out keeps them.
    • tests/test_finish_output.py: the macOS stage through the production finish statements: the whole manifest set,
      a changed or missing file refused before any swap, a partial copy leaving the installed folder as it was, an
      unmanaged Vulkan loader not overriding the packaged MoltenVK, symlinked non-alias entries refused, a retry after a
      stopped first SDL3 install. The existing fixtures run as Windows/Linux (DARWIN=False) on every OS.
  • docs: docs/development.md: the macOS build section (inputs, packaging, receipts, the helper table), and the waits
    paragraph from #437, rewritten around that table.

Rebase onto 5e072fc8: the three original commits are cherry-picks of ea75ec4a, 127ddcb2 and 5495c3f4. The
first met #453's toolchain move in build.py, darwin_build.py and make_game.py (kept #453's darwin_toolchain
calls with this PR's additions), and docs/development.md, where it had also rewritten the guest-memory, host and
fault-context sections of PRs this one does not carry: those hunks are left to their own PRs.

Flags and settings

Flag What it does Default now Recommendation Helps on Doesn't help on Untested on
LONGSHOT_SDL2 (changed on macOS) An SDL2 or SDL2-compat dylib, packaged as libSDL2-2.0.0.dylib and checked against the target pkg-config SDL2-compat with SDL3, both built for the target and 14.4 macOS Windows and Linux: as before Controllers and audio through the packaged copy
LONGSHOT_SDL3 (new on macOS) The SDL3 dylib SDL2-compat loads, packaged as libSDL3.0.dylib with the libSDL3.dylib alias unset; required when the SDL2 file names SDL3 Set it beside LONGSHOT_SDL2 macOS SDL2-compat really loading the packaged SDL3
LONGSHOT_MOLTENVK (new) The MoltenVK dylib, packaged as libMoltenVK.dylib unset; a macOS build refuses without it Set it macOS Windows and Linux: not read
LONGSHOT_VK_PREBUILT (on macOS) A renderer built already; its target, minimum and closure are checked too unset macOS A Mac renderer
LONGSHOT_DARWIN_ARCHIVE_RECEIPT, ..._SHA256 (new) The receipt for FFmpeg's NASM members without a minimum-macOS command, and its digest unset: such members refuse Set both for an x86_64 FFmpeg with NASM macOS x86_64 arm64 (none needed)
Test: --arch (repeatable), --fixtures-out, --wait-provider, --llvm-prefix Real-link routes; keep the products; a wait-state source; a second compiler The process's architecture, Apple's compiler the tests Not a build switch

Each is in docs/development.md's macOS build section. All are build switches; the game reads none.

Measurements

main: 5e072fc8c2adadba5ea610148be1c47800ba0cca (2026-11-14)
dependencies alone: d59e4c32 (#453 + #437, merged, without this PR's commits)
this PR: mac/build-complete @ fff15660d355e5caca78a61d7734234cc18fdce5
Hardware: Apple M4 Max (16 CPUs, 64 GiB), macOS 26.7.1 (25G241), Apple clang 21.0.0, Python 3.14.10. No GPU, no game.
Scene: none (a build change; its proof is the tests, as CONTRIBUTING's build row allows).
Method: the three tests below, with this PR's test files, against each tree's own nx2/ and host/ (git archive of
main and of the dependencies, with the three test files copied in), in two interleaved rounds (PR, main,
dependencies, then again). --arch arm64 --arch x86_64 for the two that take it (x86_64 runs under Rosetta).
How to reproduce: python -I -B tests/test_macos_build_flags.py --arch arm64 --arch x86_64, the same for
tests/test_macos_package.py, and python -I -B tests/test_finish_output.py.

test (round 1 / round 2) main @ 5e072fc8 dependencies @ d59e4c32 this PR @ fff15660
test_macos_build_flags.py (16 tests) 13 failures, 3 errors / the same 3 failures, 3 errors / the same OK / OK
test_macos_package.py (24 tests) does not start: No module named 'darwin_build' / the same does not start: No module named 'darwin_package' / the same OK / OK
test_finish_output.py (20 tests, 3 skipped) 7 errors (No module named 'darwin_package') / the same 7 errors / the same OK / OK

On the dependencies alone, the failures are this PR's checks: the helper table (darwin_control_sockets.c and the
other executable-only helpers taken into the renderer), the executable link without -undefined error, the preflight
without a dependency plan, and the real link, whose products are checked with darwin_package.py (absent there).

this PR @ fff15660, both rounds, both routes:
MACHO compiler=Apple21 arch=arm64 min=14.4 links=3 load=PASS exports=PASS order=PASS binding=PASS undefined=rejected weak_unbound=rejected
MACHO compiler=Apple21 arch=x86_64 min=14.4 links=3 load=PASS exports=PASS order=PASS binding=PASS undefined=rejected weak_unbound=rejected
canonical_wait_export=PASS words=256 host_flags=production
MACHO_BINDINGS arch=arm64 chained=PASS opcodes=PASS weak_coalesce=accepted lazy_weak=accepted flat=rejected disguised_flat=rejected
MACHO_BINDINGS arch=x86_64 chained=PASS opcodes=PASS weak_coalesce=accepted lazy_weak=accepted flat=rejected disguised_flat=rejected
MACHO_PACKAGE arch=arm64 relocated=PASS private_closure=PASS original_inputs=unchanged device=not-created
MACHO_PACKAGE arch=x86_64 relocated=PASS private_closure=PASS original_inputs=unchanged device=not-created

Earlier, on the old base (127ddcb2, before this rebase; not rerun here): the archive reader read the real macOS
14.4 dependency archives built for the port; all 86 unstamped NASM members hashed to their recorded original object
digests, and the 293 stamped x86_64 and 333 arm64 members kept their digests. The reader is byte-identical here.

Testing

Tested:

  • On an Apple M4 Max with macOS 26.7.1, game-free only, natively and under Rosetta, as above.

  • The full suite, tests/run_all.py, 6 at once, on this PR, compared by test name with main on the same Mac (the
    runner labels macOS linux):

    main @ 5e072fc8:     525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once)
    this PR @ fff15660:  527 passed, 20 failed, 17 skipped in 184 s (linux, 6 at once)
    
    • By test name: tests/test_macos_build_flags.py and tests/test_macos_package.py (new to main) pass, and so
      does #437's tests/test_darwin_waits.py, which this branch carries.
    • tests/test_gpu_hang_watch.py failed once, on time, not on behaviour: the stuck fixture ended with exit 3 and
      the device-lost line as it should, but after 3.7 s, past the check's 3.4 s bound, with the Mac's load average at
      12 to 19. Run alone right after, it passed on this PR (12.3 s) and on main (13.1 s). This PR changes no renderer
      or host code.
    • The other 19 failures are the same on both. They are main's own on a Mac, which the macOS port's other pull
      requests address: test_appimage_package_retry, test_collect_wide, test_control, test_framegen_resources,
      test_game_layout, test_low_memory_notice, test_mem_gate, test_mem_gate_children, test_mirror_trim,
      test_module_ids, test_oot_state_tap, test_pad_layout, test_pcache_header_read, test_pipeline_cache_file,
      test_pipeline_cache_save, test_pipeline_play_cache, test_prof_symbols, test_program_cache_cleanup,
      test_program_cache_one_variant. The same 17 are skipped.
    • No crash report was born during the run.

Not tested:

  • Windows and Linux. Nothing was built or run there. Their command arrays are checked against main's frozen
    copies by tests/test_macos_build_flags.py, and their game-folder staging by tests/test_finish_output.py's
    existing cases, both on this Mac.
  • A complete macOS game build and a game run from the packaged folder. main does not build for macOS until the
    port's other pull requests land (#420); the packaged SDL3, MoltenVK and codec copies have not loaded a game from a
    moved folder. The relocation tests use small authored products.
  • The --llvm-prefix route (LLVM 23.1.5) of the build-flags test; Apple's clang only here.
  • macOS 14.4 and an Intel Mac. Built for 14.4, run on 26.7.1 on Apple silicon only.

Needs other testers:

  • Windows (llvm-mingw) and Linux: tests/run_all.py, expecting the same results as on main, and a normal game build
    (request to open).
  • macOS 14.4 on Apple silicon, and an Intel Mac: python -I -B tests/test_macos_package.py and
    tests/test_macos_build_flags.py, expecting OK (request to open).

Screenshots

Nothing on screen.

Risk and rollback

  • Windows and Linux: the 36 compile and link arrays are compared with main's frozen copies by the build-flags test,
    and pass. Every changed line in build.py and make_game.py keeps its old branch for them. None of the changed files
    is one of the lift's sources (make_game.py's lift_sources), so no platform lifts the game again, and none is
    hashed into LONGSHOT_SHADER_STAMP (nx2/build.py:1176-1182).
  • macOS: a build now refuses inputs it used to accept with warnings (a library for the wrong architecture or a newer
    macOS, an SDL2-compat without its SDL3, an unknown helper, an unbound import). Each refusal names the file and why.
  • codesign --sign - is local and ad hoc: no account, certificate or keychain. The folder is not notarized.
  • Rollback: revert the PR. The build then goes back to #453's commands and its one-file SDL copy.

Reviewer notes

  • Start with posix_sources and dependencies in darwin_build.py, then package in darwin_package.py.
  • The four commits: 2a53a8f8, a888ab4a and 635ccef6 are ea75ec4a, 127ddcb2 and 5495c3f4 from the port's
    branch, with the conflicts above resolved and the Assisted-by / Co-Authored-By trailers of this rebase added;
    127ddcb2's "stays two-level" for ordinal -3 is corrected by 635ccef6. fff15660 is new (see What changed).
  • The macOS guest-memory, host and fault-context sections of docs/development.md (#478, #477, #436, and the darwin-host
    PR) still say each helper must be added to the macOS source list; whichever of them lands after this PR should say
    the table includes it. This PR's table already names all eleven helpers.
  • Against current main (dc390e59, git merge-tree): one conflict, nx2/build.py's renderer compile block, with
    #370 (the shader stamp moved into nx2/translator_files.py) against #453's vflags = renderer_flags(...); it is
    #453's conflict, unchanged by this PR's commits (the same block when #453 merges alone). Textual: keep both lines.
    #371's object store merges without a conflict but is near this PR's object keys; the build-flags test's frozen
    arrays need a fresh look in that rebase. #370 changes no claim here: this PR edits no file the stamp hashes, before or
    after #370.
  • Written from Apple's mach-o/loader.h, mach-o/fixup-chains.h, ar(5), ld(1), install_name_tool(1) and
    codesign(1); nothing copied.

Issues

See #420 (the macOS port).

Checklist

For everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description).
  • Rebased on current main. Branched from 5e072fc8, on #453 and #437, which it depends on; main has moved since
    (the one conflict with dc390e59 is under Reviewer notes).
  • python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
    On macOS: 527 passed, 17 skipped, the same 19 failures as main on this Mac, and test_gpu_hang_watch failing once
    on its time bound under load and passing when rerun; named under Testing.
  • Tested in the game, or what wasn't is named under Testing. (No game build or run; named under Testing.)
  • Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test
    requests are still to open.
  • Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each,
    the median reported. (For a build change: the tests fail on main and pass here, 2 interleaved rounds, both
    routes.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Flags noted, and the switch reference updated (docs/development.md, the README). (Build switches only: no README
    row.)
  • Docs updated.
**What this is for:** the macOS port (#420). The macOS build-flags PR (#453) forms Mach-O commands, but its source list takes every `host/posix/darwin_*.c` into the executable, and the macOS game folder gets the executable, the renderer and one SDL file, with the libraries still loaded from wherever they were installed (Homebrew's are arm64 only, macOS 26). This PR says which macOS helper goes into which image, checks every library the game will load against the target (architecture, macOS 14.4, the whole dependency closure, how dyld will bind it), and packages a game folder that carries its own copies, relocated and signed. Windows and Linux are unchanged: their compile and link commands are the same 36 arrays, and their game-folder staging is the same code. **What is measured:** three game-free tests (the macOS commands, the package, the game folder's staging) fail on `main` and on its two dependencies alone, and pass here, on arm64 and on x86_64 under Rosetta, two interleaved rounds. Also the full game-free suite on an Apple M4 Max (macOS 26.7.1), compared by test name with `main`. **Not measured:** Windows, Linux, a complete game build or run, macOS 14.4 itself, an Intel Mac. Depends on #453 (`mac/build-flags`) and #437 (`mac/waits`): this branch starts at #453's head `4723ee36` and merges #437's head `0d62666d` unchanged (merge `d59e4c32`). Review `d59e4c32..fff15660`, this PR's own four commits. ## Why A Mac game folder has to run on another Mac. With #453 alone: - the executable and the renderer both take every `darwin_*.c` that exists, so a helper that only the executable may own (the wait state, the guest memory's owner, the profiler, the control sockets, the Vulkan loader) would get a second copy in the renderer, with the host's logger and SDL calls in it; - the libraries come from wherever `pkg-config` found them, checked only by a tiny test link. A link that succeeds with warnings is no proof: Homebrew's FFmpeg and SDL are arm64 only and need macOS 26, so the folder would not start on x86_64 or on macOS 14.4; - the folder refers to those libraries by their installed paths, so it only starts on the Mac that built it. ## What changed - build: `nx2/darwin_build.py`. - An explicit table of the macOS helpers. Both images: `darwin_fault.c`, `darwin_fiber.c`, `darwin_mem.c`, `darwin_process.c`, `darwin_wait.c`. The executable only: `darwin_control_sockets.c`, `darwin_cpu.c`, `darwin_guest_mem.c`, `darwin_prof.c`, `darwin_vulkan.c`, `darwin_wait_state.c`. A `darwin_*.c` the table does not name stops the selection (`unclassified`), instead of being taken into both. Only the files present are compiled, so the helpers of PRs not merged yet are simply absent. - `dependencies()`: FFmpeg, SDL2, SDL3 and MoltenVK resolved to exact files once, and the same files used for the preflight, the link and the package. Every non-system `-l` becomes a file path before linking. - SDL2-compat loads SDL3 by name at run time, not through a load command, so an SDL2 file that names SDL3 anywhere requires `LONGSHOT_SDL3`. - The cache key of macOS objects includes the compiler's real path, version and bytes, and the SDK settings' bytes. - build: `nx2/darwin_package.py` (new). - A Mach-O reader: thin, universal and `ar` files; the target slice; the minimum macOS; the recursive dylib closure. - The imports dyld will really bind, from the `LC_DYLD_INFO` bind, weak-bind and lazy-bind opcodes and the `LC_DYLD_CHAINED_FIXUPS` import table as well as the symbol table. Flat-namespace and main-executable imports, and library ordinals that name no loaded library, are refused. - After both links: copies the admitted closure into owned staging, thins universal files to the target, rewrites non-system loads to `@loader_path/<name>`, removes external rpaths and signs each copy ad hoc (`codesign --sign -`). Only checked copies replace the build output; the original library files are never written. - `longshot-macos.json` lists every output file with its hash. Only alias entries (`libSDL3.dylib` → `libSDL3.0.dylib`) may be symlinks. - build: `nx2/darwin_archive.py` (new). FFmpeg's NASM objects (x86_64) carry no minimum-macOS command. They are refused unless `LONGSHOT_DARWIN_ARCHIVE_RECEIPT` and `LONGSHOT_DARWIN_ARCHIVE_RECEIPT_SHA256` name one exact receipt binding each such member's bytes, sources, assembler command, imports and instruction set. - build: `nx2/build.py` on macOS: the dependencies are admitted before any compile of a standalone build; the executable link gets `-undefined error` too, and both links `-headerpad_max_install_names` (room for the rewritten paths); the package step runs after both links. The old one-file SDL copy is gone. - build: `nx2/make_game.py` on macOS: the preflight uses the same dependency plan and inspects its test links' products; the game-folder stage copies and checks every file in `longshot-macos.json` (aliases kept as links), and removes only unchanged files an earlier manifest owned. On Windows and Linux every changed line keeps its old branch. - build: `fff15660` is new: #453 moved `architecture()` and `darwin_sdk()` from a64-kit's `toolchain.py` into `nx2/darwin_toolchain.py`, so the dependency plan and the post-link check ask `darwin_toolchain` for them. Without it both stop with `AttributeError` on a Mac. - tests: - `tests/test_macos_build_flags.py`: the helper table; strict links (no flat namespace, no `dynamic_lookup`); the preflight's commands; and a real link of `darwin_wait_state.c` with the production host flags, read back from two separately linked dylibs (the wait PR's shared state). - `tests/test_macos_package.py` (new): dependency and receipt refusals on authored metadata (every OS), and on a Mac real C/C++ products linked with the production commands, packaged, moved and loaded with their original folder hidden; real chained-fixup and opcode-stream links. `--arch` selects the routes, `--fixtures-out` keeps them. - `tests/test_finish_output.py`: the macOS stage through the production finish statements: the whole manifest set, a changed or missing file refused before any swap, a partial copy leaving the installed folder as it was, an unmanaged Vulkan loader not overriding the packaged MoltenVK, symlinked non-alias entries refused, a retry after a stopped first SDL3 install. The existing fixtures run as Windows/Linux (`DARWIN=False`) on every OS. - docs: `docs/development.md`: the macOS build section (inputs, packaging, receipts, the helper table), and the waits paragraph from #437, rewritten around that table. **Rebase onto `5e072fc8`:** the three original commits are cherry-picks of `ea75ec4a`, `127ddcb2` and `5495c3f4`. The first met #453's toolchain move in `build.py`, `darwin_build.py` and `make_game.py` (kept #453's `darwin_toolchain` calls with this PR's additions), and `docs/development.md`, where it had also rewritten the guest-memory, host and fault-context sections of PRs this one does not carry: those hunks are left to their own PRs. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `LONGSHOT_SDL2` (changed on macOS) | An SDL2 or SDL2-compat dylib, packaged as `libSDL2-2.0.0.dylib` and checked against the target | `pkg-config` | SDL2-compat with SDL3, both built for the target and 14.4 | macOS | Windows and Linux: as before | Controllers and audio through the packaged copy | | `LONGSHOT_SDL3` (new on macOS) | The SDL3 dylib SDL2-compat loads, packaged as `libSDL3.0.dylib` with the `libSDL3.dylib` alias | unset; required when the SDL2 file names SDL3 | Set it beside `LONGSHOT_SDL2` | macOS | | SDL2-compat really loading the packaged SDL3 | | `LONGSHOT_MOLTENVK` (new) | The MoltenVK dylib, packaged as `libMoltenVK.dylib` | unset; a macOS build refuses without it | Set it | macOS | Windows and Linux: not read | | | `LONGSHOT_VK_PREBUILT` (on macOS) | A renderer built already; its target, minimum and closure are checked too | unset | | macOS | | A Mac renderer | | `LONGSHOT_DARWIN_ARCHIVE_RECEIPT`, `..._SHA256` (new) | The receipt for FFmpeg's NASM members without a minimum-macOS command, and its digest | unset: such members refuse | Set both for an x86_64 FFmpeg with NASM | macOS x86_64 | arm64 (none needed) | | | Test: `--arch` (repeatable), `--fixtures-out`, `--wait-provider`, `--llvm-prefix` | Real-link routes; keep the products; a wait-state source; a second compiler | The process's architecture, Apple's compiler | | the tests | Not a build switch | | Each is in `docs/development.md`'s macOS build section. All are build switches; the game reads none. ## Measurements main: `5e072fc8c2adadba5ea610148be1c47800ba0cca` (2026-11-14) dependencies alone: `d59e4c32` (#453 + #437, merged, without this PR's commits) this PR: `mac/build-complete` @ `fff15660d355e5caca78a61d7734234cc18fdce5` Hardware: Apple M4 Max (16 CPUs, 64 GiB), macOS 26.7.1 (25G241), Apple clang 21.0.0, Python 3.14.10. No GPU, no game. Scene: none (a build change; its proof is the tests, as CONTRIBUTING's `build` row allows). Method: the three tests below, with this PR's test files, against each tree's own `nx2/` and `host/` (`git archive` of `main` and of the dependencies, with the three test files copied in), in two interleaved rounds (PR, main, dependencies, then again). `--arch arm64 --arch x86_64` for the two that take it (x86_64 runs under Rosetta). How to reproduce: `python -I -B tests/test_macos_build_flags.py --arch arm64 --arch x86_64`, the same for `tests/test_macos_package.py`, and `python -I -B tests/test_finish_output.py`. | test (round 1 / round 2) | main @ 5e072fc8 | dependencies @ d59e4c32 | this PR @ fff15660 | |---|---|---|---| | `test_macos_build_flags.py` (16 tests) | 13 failures, 3 errors / the same | 3 failures, 3 errors / the same | OK / OK | | `test_macos_package.py` (24 tests) | does not start: `No module named 'darwin_build'` / the same | does not start: `No module named 'darwin_package'` / the same | OK / OK | | `test_finish_output.py` (20 tests, 3 skipped) | 7 errors (`No module named 'darwin_package'`) / the same | 7 errors / the same | OK / OK | On the dependencies alone, the failures are this PR's checks: the helper table (`darwin_control_sockets.c` and the other executable-only helpers taken into the renderer), the executable link without `-undefined error`, the preflight without a dependency plan, and the real link, whose products are checked with `darwin_package.py` (absent there). ``` this PR @ fff15660, both rounds, both routes: MACHO compiler=Apple21 arch=arm64 min=14.4 links=3 load=PASS exports=PASS order=PASS binding=PASS undefined=rejected weak_unbound=rejected MACHO compiler=Apple21 arch=x86_64 min=14.4 links=3 load=PASS exports=PASS order=PASS binding=PASS undefined=rejected weak_unbound=rejected canonical_wait_export=PASS words=256 host_flags=production MACHO_BINDINGS arch=arm64 chained=PASS opcodes=PASS weak_coalesce=accepted lazy_weak=accepted flat=rejected disguised_flat=rejected MACHO_BINDINGS arch=x86_64 chained=PASS opcodes=PASS weak_coalesce=accepted lazy_weak=accepted flat=rejected disguised_flat=rejected MACHO_PACKAGE arch=arm64 relocated=PASS private_closure=PASS original_inputs=unchanged device=not-created MACHO_PACKAGE arch=x86_64 relocated=PASS private_closure=PASS original_inputs=unchanged device=not-created ``` **Earlier, on the old base** (`127ddcb2`, before this rebase; not rerun here): the archive reader read the real macOS 14.4 dependency archives built for the port; all 86 unstamped NASM members hashed to their recorded original object digests, and the 293 stamped x86_64 and 333 arm64 members kept their digests. The reader is byte-identical here. ## Testing **Tested:** - On an Apple M4 Max with macOS 26.7.1, game-free only, natively and under Rosetta, as above. - The full suite, `tests/run_all.py`, 6 at once, on this PR, compared by test name with `main` on the same Mac (the runner labels macOS `linux`): ``` main @ 5e072fc8: 525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once) this PR @ fff15660: 527 passed, 20 failed, 17 skipped in 184 s (linux, 6 at once) ``` - **By test name:** `tests/test_macos_build_flags.py` and `tests/test_macos_package.py` (new to `main`) pass, and so does #437's `tests/test_darwin_waits.py`, which this branch carries. - **`tests/test_gpu_hang_watch.py` failed once, on time, not on behaviour:** the stuck fixture ended with exit 3 and the device-lost line as it should, but after 3.7 s, past the check's 3.4 s bound, with the Mac's load average at 12 to 19. Run alone right after, it passed on this PR (12.3 s) and on `main` (13.1 s). This PR changes no renderer or host code. - **The other 19 failures are the same on both.** They are `main`'s own on a Mac, which the macOS port's other pull requests address: `test_appimage_package_retry`, `test_collect_wide`, `test_control`, `test_framegen_resources`, `test_game_layout`, `test_low_memory_notice`, `test_mem_gate`, `test_mem_gate_children`, `test_mirror_trim`, `test_module_ids`, `test_oot_state_tap`, `test_pad_layout`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_prof_symbols`, `test_program_cache_cleanup`, `test_program_cache_one_variant`. The same 17 are skipped. - No crash report was born during the run. **Not tested:** - **Windows and Linux.** Nothing was built or run there. Their command arrays are checked against `main`'s frozen copies by `tests/test_macos_build_flags.py`, and their game-folder staging by `tests/test_finish_output.py`'s existing cases, both on this Mac. - **A complete macOS game build and a game run from the packaged folder.** `main` does not build for macOS until the port's other pull requests land (#420); the packaged SDL3, MoltenVK and codec copies have not loaded a game from a moved folder. The relocation tests use small authored products. - **The `--llvm-prefix` route** (LLVM 23.1.5) of the build-flags test; Apple's clang only here. - **macOS 14.4** and an Intel Mac. Built for 14.4, run on 26.7.1 on Apple silicon only. **Needs other testers:** - Windows (llvm-mingw) and Linux: `tests/run_all.py`, expecting the same results as on `main`, and a normal game build (request to open). - macOS 14.4 on Apple silicon, and an Intel Mac: `python -I -B tests/test_macos_package.py` and `tests/test_macos_build_flags.py`, expecting OK (request to open). ## Screenshots Nothing on screen. ## Risk and rollback - Windows and Linux: the 36 compile and link arrays are compared with `main`'s frozen copies by the build-flags test, and pass. Every changed line in `build.py` and `make_game.py` keeps its old branch for them. None of the changed files is one of the lift's sources (`make_game.py`'s `lift_sources`), so no platform lifts the game again, and none is hashed into `LONGSHOT_SHADER_STAMP` (`nx2/build.py:1176-1182`). - macOS: a build now refuses inputs it used to accept with warnings (a library for the wrong architecture or a newer macOS, an SDL2-compat without its SDL3, an unknown helper, an unbound import). Each refusal names the file and why. - `codesign --sign -` is local and ad hoc: no account, certificate or keychain. The folder is not notarized. - Rollback: revert the PR. The build then goes back to #453's commands and its one-file SDL copy. ## Reviewer notes - Start with `posix_sources` and `dependencies` in `darwin_build.py`, then `package` in `darwin_package.py`. - The four commits: `2a53a8f8`, `a888ab4a` and `635ccef6` are `ea75ec4a`, `127ddcb2` and `5495c3f4` from the port's branch, with the conflicts above resolved and the `Assisted-by` / `Co-Authored-By` trailers of this rebase added; `127ddcb2`'s "stays two-level" for ordinal -3 is corrected by `635ccef6`. `fff15660` is new (see What changed). - The macOS guest-memory, host and fault-context sections of `docs/development.md` (#478, #477, #436, and the darwin-host PR) still say each helper must be added to the macOS source list; whichever of them lands after this PR should say the table includes it. This PR's table already names all eleven helpers. - Against current `main` (`dc390e59`, `git merge-tree`): one conflict, `nx2/build.py`'s renderer compile block, with #370 (the shader stamp moved into `nx2/translator_files.py`) against #453's `vflags = renderer_flags(...)`; it is #453's conflict, unchanged by this PR's commits (the same block when #453 merges alone). Textual: keep both lines. #371's object store merges without a conflict but is near this PR's object keys; the build-flags test's frozen arrays need a fresh look in that rebase. #370 changes no claim here: this PR edits no file the stamp hashes, before or after #370. - Written from Apple's `mach-o/loader.h`, `mach-o/fixup-chains.h`, `ar(5)`, `ld(1)`, `install_name_tool(1)` and `codesign(1)`; nothing copied. ## Issues See #420 (the macOS port). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. Branched from 5e072fc8, on #453 and #437, which it depends on; `main` has moved since (the one conflict with `dc390e59` is under Reviewer notes). - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). On macOS: 527 passed, 17 skipped, the same 19 failures as `main` on this Mac, and `test_gpu_hang_watch` failing once on its time bound under load and passing when rerun; named under Testing. - [x] Tested in the game, or what wasn't is named under Testing. (No game build or run; named under Testing.) - [ ] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test requests are still to open. - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (For a `build` change: the tests fail on `main` and pass here, 2 interleaved rounds, both routes.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). (Build switches only: no README row.) - [x] Docs updated.
Add macOS 14.4 public address waits, awake-uptime deadlines, relative
monitor waits, absolute timer sleeps, and real initialization for zeroed
SRW locks. Keep the Linux source behind each Apple branch byte-identical.

Exercise the extracted layer with real primitives and C/C++ header clients
on arm64 and x86_64. Document the existing broadcast address wakes, timer
restrictions and non-atomic polling, without claiming a complete host port.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The pointer mailbox test uses the real POSIX SRW wrappers, which now call
into darwin_wait.c. Compile that production C dependency and link it into
the existing native-default fixture on macOS only. Keep the Linux and
Windows compiler invocation and every existing assertion unchanged.

The full-suite failure at 1da4f9b3 is retained. This repair does not resolve
the held cross-image address-wait contract decision or qualify the port.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Move the address sequence words into an executable-only provider and resolve
that canonical service once in each shim image. Fail closed when the host
provider is absent; retain full-width comparisons and broadcast ordering.

Extend the existing wait driver with real host/two-dylib behavior coverage,
including upper-half and cross-word changes, same-image controls, image-local
error state, broadcast wakes, and rejection of a dylib-only provider. Keep
the existing timing campaign intact and independently selectable.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs(runtime): describe the executable-owned macOS address-wait state
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 7s
CI / Conventional Commit title (pull_request) Successful in 6s
CI / Workflow lint (actionlint) (pull_request) Successful in 7s
Third-party parts / As their notes say (pull_request) Successful in 8s
Website / Website on Cloudflare (pull_request_target) Has been skipped
0d62666d45
The wait section still said each image's darwin_wait.c is self-contained.
Since the address sequence words moved into the executable, the renderer's
copy finds them through RTLD_MAIN_ONLY, a missing provider aborts, and the
build has to link darwin_wait_state.c into the host only. Say so, and name
the test's --shared-only mode.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Separate macOS from Linux, select arm64 or x86_64 without changing the
Windows/Linux command arrays, and discover LLVM/Apple tools through the
existing prefix, PATH and xcrun. Use the Apple SDK, a 14.4 deployment
target, Mach-O export/map/order/rpath options and system libc++.

Keep the runtime port outside this change. Pick up Darwin host helpers,
exclude the Linux sampler, and test command construction, prefix quoting,
missing tools and real C/C++ dylib loading on native ARM and Rosetta.
The a64-kit adaptation and refreshed tree digest are recorded in its
PROVENANCE.md. No game or generated game content is included.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Do not require Rosetta in an ordinary Apple Silicon game-free suite or
skip all native x86_64 evidence on Intel. Keep both simulated target
contracts everywhere and add repeatable --arch options for explicit
dual-route qualification. Keep every binding, export, symbol-order,
C++ runtime and undefined-symbol assertion and all production flags.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Use canonical POSIX paths only for the virtual /fixture command inputs.
Keep host paths for real discovery, temporary-directory helpers and real
Mach-O runs, and patch exists on the selected namespace path object.

Exercise the real frozen-array comparison under an ntpath host view for
all six configurations. Leave the baseline JSON and production flags
unchanged.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
build(macos): keep third_party/a64-kit as it is; nx2/darwin_toolchain.py makes its commands Mach-O
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 4s
PR description / Description follows the template (pull_request_target) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 7s
Third-party parts / As their notes say (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
4723ee36a4
CONTRIBUTING keeps third_party/ as its upstream copies. Restore a64-kit's
toolchain.py, build_emit.py and PROVENANCE.md to main's bytes and move
their macOS behaviour into nx2/darwin_toolchain.py, an LS file.

On macOS, kit.py and make_game.py install it: toolchain's find, need,
native_target and native_link become its own (tools from EMSDK, PATH,
then xcrun; -arch, a 14.4 target and Apple's SDK; a Mach-O linker), and
build_emit builds liba64emit-<process arch>.dylib, rebuilt when the
compiler, its version or the flags change, as nx2/insn.py overrides
a64emit.translate. system(), architecture(), darwin_sdk() and version()
move with it; build.py, make_game.py, darwin_build.py, host_cpu.py and
zbic.py call them there. Windows and Linux never import the file.

kit.py's import makes the new file one of the lift's sources
(make_game.py lift_sources), so every platform lifts the game again
once, as the a64-kit edit did; build_emit is imported by name for that
walk (tests/test_lift_cache.py).

The command contracts now load main's own toolchain.py, so the 36
Windows/Linux arrays are compared with a64-kit untouched. A new check
proves kit.py and make_game.py install the functions on a Mac. The
test_cpm_discovery stub no longer needs toolchain.system().

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This PR needs the macOS waits PR (#437); its commits are kept as they are.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Classify Darwin image-local helpers separately from executable-owned wait,
guest-memory, profiler, CPU, control and SDL/Vulkan state. Fail on unknown
helpers instead of broad-globbing host dependencies into the renderer.

Resolve and inspect exact target/minimum-OS dependency inputs before the
standalone build. Relocate and ad-hoc sign only owned copies, preserve the
SDL3 compatibility alias, and deploy the complete hash-bound output manifest.
Qualify otherwise unstamped NASM members only through bounded source/tool,
member/import/ISA evidence and an exact semantic-review receipt.

Keep strict two-level undefined-error links and the frozen Windows/Linux
command arrays. Retain real negative weak-link evidence and verify the real
canonical wait provider with production host flags on arm64 and Rosetta.
Document carried evidence and unproved full-game/dependency qualification.
No experiment-specific lifted-source reuse subsystem or third-party edits.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
SDL2-compat dlopens SDL3 by file name and its own build carries only an
@loader_path rpath. Require LONGSHOT_SDL3 and its internal alias whenever
the selected SDL2 file names SDL3, not only when an rpath spells sdl3; a
classic SDL2 that names no SDL3 still needs none.

The strict-import check read only LC_SYMTAB ordinals. Also decode the
binding dyld applies: LC_DYLD_INFO bind, weak-bind and lazy-bind opcodes
and the LC_DYLD_CHAINED_FIXUPS import table (all three import formats).
Flat-lookup or main-executable imports, and library ordinals that name no
loaded dylib, refuse. C++ weak-definition coalescing (-3) stays two-level.

Validate ar member sizes as plain in-archive decimals before special
members are handled, which also ends the negative-size infinite loop.
A Mach-O member's object is the extent its own load commands describe;
only fewer than eight trailing newlines (Apple ar framing) may follow,
and member SHA-256 covers the object bytes, equal to the original NASM
output. Unknown file-bearing load commands and fat members refuse.

Only alias entries in longshot-macos.json may be symlinks. The maker
clears a dangling staged .new alias with lexists on macOS, so a retry
after a stopped first SDL3 install recovers. Windows/Linux staging is
unchanged.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
dyld keeps the library ordinal across a lazy-stream DONE; DONE only ends
the entry. The reader reset it to 0, so a lazy entry without its own
SET_DYLIB after a flat-lookup (-2) entry was checked as a self import
and admitted while dyld binds it flat. Clear only the symbol on a lazy
DONE and keep the ordinal, with that exact stream as a regression.

Correct the wording for ordinal -3 (BIND_SPECIAL_DYLIB_WEAK_LOOKUP): it
is not two-level. dyld resolves it by a process-wide, load-order search
among images with weak definitions. It stays admitted because ld emits
it only for weak definitions it resolved while linking. The previous
commit message's "stays two-level" was wrong in the same way.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
build(macos): take the architecture and SDK from darwin_toolchain when packaging
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Third-party parts / As their notes say (pull_request) Successful in 11s
Website / Website on Cloudflare (pull_request_target) Has been skipped
fff15660d3
The macOS build flags PR moved system(), architecture(), darwin_sdk()
and version() from a64-kit's toolchain.py into nx2/darwin_toolchain.py,
which main's toolchain.py does not have. The dependency plan and the
post-link check still asked toolchain for them, so on a Mac
dependencies() and post_link() stopped with AttributeError. Ask
darwin_toolchain there, and point the package and preflight tests'
stand-ins at it (the preflight's scope gets _darwin, as make_game.py
names it).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Third-party parts / As their notes say (pull_request) Successful in 11s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request has changes conflicting with the target branch.
  • docs/development.md
  • nx2/build.py
  • nx2/make_game.py
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'mac/build-complete:mooki-mac/build-complete'
git switch 'mooki-mac/build-complete'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!479
No description provided.