WIP: build(macos): classify Darwin helpers and package a relocatable macOS 14.4 dylib set #479
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!479
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:mac/build-complete"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is for: the macOS port (#420). The macOS build-flags PR (#453) forms Mach-O commands, but its source list
takes every
host/posix/darwin_*.cinto the executable, and the macOS game folder gets the executable, the renderer andone SDL file, with the libraries still loaded from wherever they were installed (Homebrew's are arm64 only, macOS 26).
This PR says which macOS helper goes into which image, checks every library the game will load against the target
(architecture, macOS 14.4, the whole dependency closure, how dyld will bind it), and packages a game folder that carries
its own copies, relocated and signed. Windows and Linux are unchanged: their compile and link commands are the same 36
arrays, and their game-folder staging is the same code. What is measured: three game-free tests (the macOS
commands, the package, the game folder's staging) fail on
mainand on its two dependencies alone, and pass here, onarm64 and on x86_64 under Rosetta, two interleaved rounds. Also the full game-free suite on an Apple M4 Max (macOS
26.7.1), compared by test name with
main. Not measured: Windows, Linux, a complete game build or run, macOS 14.4itself, an Intel Mac.
Depends on #453 (
mac/build-flags) and #437 (mac/waits): this branch starts at #453's head4723ee36and merges#437's head
0d62666dunchanged (merged59e4c32). Reviewd59e4c32..fff15660, this PR's own four commits.Why
A Mac game folder has to run on another Mac. With #453 alone:
darwin_*.cthat exists, so a helper that only the executable may own(the wait state, the guest memory's owner, the profiler, the control sockets, the Vulkan loader) would get a second
copy in the renderer, with the host's logger and SDL calls in it;
pkg-configfound them, checked only by a tiny test link. A link that succeeds withwarnings is no proof: Homebrew's FFmpeg and SDL are arm64 only and need macOS 26, so the folder would not start on
x86_64 or on macOS 14.4;
What changed
nx2/darwin_build.py.darwin_fault.c,darwin_fiber.c,darwin_mem.c,darwin_process.c,darwin_wait.c. The executable only:darwin_control_sockets.c,darwin_cpu.c,darwin_guest_mem.c,darwin_prof.c,darwin_vulkan.c,darwin_wait_state.c. Adarwin_*.cthe table does notname stops the selection (
unclassified), instead of being taken into both. Only the files present are compiled,so the helpers of PRs not merged yet are simply absent.
dependencies(): FFmpeg, SDL2, SDL3 and MoltenVK resolved to exact files once, and the same files used for thepreflight, the link and the package. Every non-system
-lbecomes a file path before linking.requires
LONGSHOT_SDL3.nx2/darwin_package.py(new).arfiles; the target slice; the minimum macOS; the recursive dylib closure.LC_DYLD_INFObind, weak-bind and lazy-bind opcodes and theLC_DYLD_CHAINED_FIXUPSimport table as well as the symbol table. Flat-namespace and main-executable imports, andlibrary ordinals that name no loaded library, are refused.
non-system loads to
@loader_path/<name>, removes external rpaths and signs each copy ad hoc (codesign --sign -).Only checked copies replace the build output; the original library files are never written.
longshot-macos.jsonlists every output file with its hash. Only alias entries (libSDL3.dylib→libSDL3.0.dylib) may be symlinks.nx2/darwin_archive.py(new). FFmpeg's NASM objects (x86_64) carry no minimum-macOS command. They are refusedunless
LONGSHOT_DARWIN_ARCHIVE_RECEIPTandLONGSHOT_DARWIN_ARCHIVE_RECEIPT_SHA256name one exact receipt bindingeach such member's bytes, sources, assembler command, imports and instruction set.
nx2/build.pyon macOS: the dependencies are admitted before any compile of a standalone build; theexecutable link gets
-undefined errortoo, and both links-headerpad_max_install_names(room for the rewrittenpaths); the package step runs after both links. The old one-file SDL copy is gone.
nx2/make_game.pyon macOS: the preflight uses the same dependency plan and inspects its test links' products;the game-folder stage copies and checks every file in
longshot-macos.json(aliases kept as links), and removes onlyunchanged files an earlier manifest owned. On Windows and Linux every changed line keeps its old branch.
fff15660is new: #453 movedarchitecture()anddarwin_sdk()from a64-kit'stoolchain.pyintonx2/darwin_toolchain.py, so the dependency plan and the post-link check askdarwin_toolchainfor them. Without itboth stop with
AttributeErroron a Mac.tests/test_macos_build_flags.py: the helper table; strict links (no flat namespace, nodynamic_lookup); thepreflight's commands; and a real link of
darwin_wait_state.cwith the production host flags, read back from twoseparately linked dylibs (the wait PR's shared state).
tests/test_macos_package.py(new): dependency and receipt refusals on authored metadata (every OS), and on a Macreal C/C++ products linked with the production commands, packaged, moved and loaded with their original folder
hidden; real chained-fixup and opcode-stream links.
--archselects the routes,--fixtures-outkeeps them.tests/test_finish_output.py: the macOS stage through the production finish statements: the whole manifest set,a changed or missing file refused before any swap, a partial copy leaving the installed folder as it was, an
unmanaged Vulkan loader not overriding the packaged MoltenVK, symlinked non-alias entries refused, a retry after a
stopped first SDL3 install. The existing fixtures run as Windows/Linux (
DARWIN=False) on every OS.docs/development.md: the macOS build section (inputs, packaging, receipts, the helper table), and the waitsparagraph from #437, rewritten around that table.
Rebase onto
5e072fc8: the three original commits are cherry-picks ofea75ec4a,127ddcb2and5495c3f4. Thefirst met #453's toolchain move in
build.py,darwin_build.pyandmake_game.py(kept #453'sdarwin_toolchaincalls with this PR's additions), and
docs/development.md, where it had also rewritten the guest-memory, host andfault-context sections of PRs this one does not carry: those hunks are left to their own PRs.
Flags and settings
LONGSHOT_SDL2(changed on macOS)libSDL2-2.0.0.dyliband checked against the targetpkg-configLONGSHOT_SDL3(new on macOS)libSDL3.0.dylibwith thelibSDL3.dylibaliasLONGSHOT_SDL2LONGSHOT_MOLTENVK(new)libMoltenVK.dylibLONGSHOT_VK_PREBUILT(on macOS)LONGSHOT_DARWIN_ARCHIVE_RECEIPT,..._SHA256(new)--arch(repeatable),--fixtures-out,--wait-provider,--llvm-prefixEach is in
docs/development.md's macOS build section. All are build switches; the game reads none.Measurements
main:
5e072fc8c2adadba5ea610148be1c47800ba0cca(2026-11-14)dependencies alone:
d59e4c32(#453 + #437, merged, without this PR's commits)this PR:
mac/build-complete@fff15660d355e5caca78a61d7734234cc18fdce5Hardware: Apple M4 Max (16 CPUs, 64 GiB), macOS 26.7.1 (25G241), Apple clang 21.0.0, Python 3.14.10. No GPU, no game.
Scene: none (a build change; its proof is the tests, as CONTRIBUTING's
buildrow allows).Method: the three tests below, with this PR's test files, against each tree's own
nx2/andhost/(git archiveofmainand of the dependencies, with the three test files copied in), in two interleaved rounds (PR, main,dependencies, then again).
--arch arm64 --arch x86_64for the two that take it (x86_64 runs under Rosetta).How to reproduce:
python -I -B tests/test_macos_build_flags.py --arch arm64 --arch x86_64, the same fortests/test_macos_package.py, andpython -I -B tests/test_finish_output.py.5e072fc8d59e4c32fff15660test_macos_build_flags.py(16 tests)test_macos_package.py(24 tests)No module named 'darwin_build'/ the sameNo module named 'darwin_package'/ the sametest_finish_output.py(20 tests, 3 skipped)No module named 'darwin_package') / the sameOn the dependencies alone, the failures are this PR's checks: the helper table (
darwin_control_sockets.cand theother executable-only helpers taken into the renderer), the executable link without
-undefined error, the preflightwithout a dependency plan, and the real link, whose products are checked with
darwin_package.py(absent there).Earlier, on the old base (
127ddcb2, before this rebase; not rerun here): the archive reader read the real macOS14.4 dependency archives built for the port; all 86 unstamped NASM members hashed to their recorded original object
digests, and the 293 stamped x86_64 and 333 arm64 members kept their digests. The reader is byte-identical here.
Testing
Tested:
On an Apple M4 Max with macOS 26.7.1, game-free only, natively and under Rosetta, as above.
The full suite,
tests/run_all.py, 6 at once, on this PR, compared by test name withmainon the same Mac (therunner labels macOS
linux):tests/test_macos_build_flags.pyandtests/test_macos_package.py(new tomain) pass, and sodoes #437's
tests/test_darwin_waits.py, which this branch carries.tests/test_gpu_hang_watch.pyfailed once, on time, not on behaviour: the stuck fixture ended with exit 3 andthe device-lost line as it should, but after 3.7 s, past the check's 3.4 s bound, with the Mac's load average at
12 to 19. Run alone right after, it passed on this PR (12.3 s) and on
main(13.1 s). This PR changes no rendereror host code.
main's own on a Mac, which the macOS port's other pullrequests address:
test_appimage_package_retry,test_collect_wide,test_control,test_framegen_resources,test_game_layout,test_low_memory_notice,test_mem_gate,test_mem_gate_children,test_mirror_trim,test_module_ids,test_oot_state_tap,test_pad_layout,test_pcache_header_read,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_prof_symbols,test_program_cache_cleanup,test_program_cache_one_variant. The same 17 are skipped.Not tested:
main's frozencopies by
tests/test_macos_build_flags.py, and their game-folder staging bytests/test_finish_output.py'sexisting cases, both on this Mac.
maindoes not build for macOS until theport's other pull requests land (#420); the packaged SDL3, MoltenVK and codec copies have not loaded a game from a
moved folder. The relocation tests use small authored products.
--llvm-prefixroute (LLVM 23.1.5) of the build-flags test; Apple's clang only here.Needs other testers:
tests/run_all.py, expecting the same results as onmain, and a normal game build(request to open).
python -I -B tests/test_macos_package.pyandtests/test_macos_build_flags.py, expecting OK (request to open).Screenshots
Nothing on screen.
Risk and rollback
main's frozen copies by the build-flags test,and pass. Every changed line in
build.pyandmake_game.pykeeps its old branch for them. None of the changed filesis one of the lift's sources (
make_game.py'slift_sources), so no platform lifts the game again, and none ishashed into
LONGSHOT_SHADER_STAMP(nx2/build.py:1176-1182).macOS, an SDL2-compat without its SDL3, an unknown helper, an unbound import). Each refusal names the file and why.
codesign --sign -is local and ad hoc: no account, certificate or keychain. The folder is not notarized.Reviewer notes
posix_sourcesanddependenciesindarwin_build.py, thenpackageindarwin_package.py.2a53a8f8,a888ab4aand635ccef6areea75ec4a,127ddcb2and5495c3f4from the port'sbranch, with the conflicts above resolved and the
Assisted-by/Co-Authored-Bytrailers of this rebase added;127ddcb2's "stays two-level" for ordinal -3 is corrected by635ccef6.fff15660is new (see What changed).docs/development.md(#478, #477, #436, and the darwin-hostPR) still say each helper must be added to the macOS source list; whichever of them lands after this PR should say
the table includes it. This PR's table already names all eleven helpers.
main(dc390e59,git merge-tree): one conflict,nx2/build.py's renderer compile block, with#370 (the shader stamp moved into
nx2/translator_files.py) against #453'svflags = renderer_flags(...); it is#453's conflict, unchanged by this PR's commits (the same block when #453 merges alone). Textual: keep both lines.
#371's object store merges without a conflict but is near this PR's object keys; the build-flags test's frozen
arrays need a fresh look in that rebase. #370 changes no claim here: this PR edits no file the stamp hashes, before or
after #370.
mach-o/loader.h,mach-o/fixup-chains.h,ar(5),ld(1),install_name_tool(1)andcodesign(1); nothing copied.Issues
See #420 (the macOS port).
Checklist
For everyone
type(scope): description).main. Branched from5e072fc8, on #453 and #437, which it depends on;mainhas moved since(the one conflict with
dc390e59is under Reviewer notes).python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above).On macOS: 527 passed, 17 skipped, the same 19 failures as
mainon this Mac, andtest_gpu_hang_watchfailing onceon its time bound under load and passing when rerun; named under Testing.
requests are still to open.
the median reported. (For a
buildchange: the tests fail onmainand pass here, 2 interleaved rounds, bothroutes.)
third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.row.)
View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.