WIP: feat(macos): native Darwin threads, fibers, restart, process I/O and profiler #490

Draft

mooki wants to merge 29 commits from mooki/Longshot:mac/darwin-host into main

pull from: mooki/Longshot:mac/darwin-host

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 0 Commits 29 Files changed 82 +7749 -71

mooki commented 2026-11-14 16:30:51 +00:00

Member

• Copy link
• Copy Markdown

What this is for: the macOS port (#420). The POSIX layer (host/posix/win32_posix.c) is written for Linux: on
macOS it does not compile (Linux ucontext fibers, cpu_set_t, /proc, dlinfo), and the profiler, restart,
child processes, the menu's share functions, the memory notice and Discord presence had no macOS route. This PR gives
the host a native Darwin implementation of each: Mach thread identities and times, guarded native fibers that may
migrate between threads, executable and module paths from dyld, same-process restart and child processes through
posix_spawn, Darwin memory counters, the low-memory notice from hw.memsize, and a profiler that samples interrupted
threads through Mach. It also changes Linux in three places, listed under "Changes on Linux": Discord IPC, the
thread-creation error paths, and scripts/prof.py. What is measured: tests/test_darwin_host.py builds the real
host objects and runs seven groups of checks on arm64 and on x86_64 under Rosetta; it fails on main, and on main
with the six pull requests this one carries, and passes here, two interleaved runs each. tests/test_low_memory_notice.py
and tests/test_prof_symbols.py fail on main on macOS and pass here. And the full game-free suite on an Apple M4 Max
(macOS 26.7.1), compared by test name with main. Not measured: a game run on this branch; Windows and Linux runs;
Discord itself; macOS 14.4; an Intel Mac.

Depends on #454 (run-all), #477 (arm64-host), #478 (guest-memory), #436 (fault-contexts), #437 (waits) and #438
(vk-admission). This branch carries their commits unchanged, merged in that order (9ebd6ae8, 56b5ed6b, 9c679145,
7731dc0c, 0d62666d, 91d8cc35); its own commits start at 5d3abc9a.

Why

With the six pull requests above, win32_posix.c still stops on macOS at its Linux-only parts, and the host has no
macOS answer for the thread, process and profiling calls the game and the renderer make:

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ucontext.h:51:2: error: The deprecated ucontext routines require _XOPEN_SOURCE to be defined
host/posix/win32_posix.c:357:5: error: use of undeclared identifier 'cpu_set_t'
host/posix/win32_posix.c:358:5: error: call to undeclared function 'CPU_ZERO'

Problem

• Fibers: the POSIX fibers use ucontext, which Darwin deprecates and does not implement as an arm64 fiber switch.
  The game's nn::os fibers run on them (host/hos_thread.c: CreateFiberEx, SwitchToFiber), and
  hos_setjmp.c's nonlocal slots belong to the running thread, not to the fiber.
• Threads: thread ids come from SYS_gettid, names from pthread_setname_np on any thread, priorities from
  per-thread nice, affinity and topology from cpu_set_t. Darwin has none of these in that form.
• Modules and processes: the executable's path comes from /proc/self/exe and a module's from dlinfo/link_map.
  Restart reads the startup command and environment back from /proc/self/cmdline and /proc/self/environ, closes
  descriptors through /proc/self/fd and re-executes /proc/self/exe; the menu's child processes use pipe2 and
  /proc/self/fd too. macOS has no /proc and no pipe2.
• Memory and diagnostics: GetProcessMemoryInfo reads /proc/self/status, the low-memory notice reads sysinfo,
  and the profiler (HOS_PROFILE, HOS_SAMPLE, HOS_THREAD_TIMES) uses Linux signals and /proc. scripts/prof.py
  reads ELF and Windows maps only.
• Menu and Discord: the share functions run xclip/xdg-open; the Discord client connects with sizeof of the
  socket address and no close-on-exec.

What changed

macOS (everything below is inside #if defined(__APPLE__) or in new Darwin-only files)

• Threads (win32_posix.c): real pthread and Mach identities. The Win32 DWORD thread id is the Mach thread port
  name; the full 64-bit native id is kept separately (darwin_thread_id) and the profiler uses it. Names are set on
  the kernel for the calling thread; a name set on another thread is kept and queryable but returns S_FALSE, since
  Darwin only lets a thread name itself; an oversized name is refused. Priorities use
  THREAD_PRECEDENCE_POLICY (a per-thread relative importance; the process's nice is untouched).
  GetThreadTimes reads live Mach times and keeps an exited thread's. GetSystemInfo reports the real architecture
  and page size. Stacks are rounded to the host page with a guard page. Affinity and logical-CPU topology fail with
  ERROR_NOT_SUPPORTED instead of inventing masks; HOS_PIN_BY_CORE and the renderer's reserved core say once in the
  log that Darwin has no binding mask; CREATE_SUSPENDED is refused.
• Fibers (host/posix/darwin_fiber.c, new): native arm64 and x86-64 System V contexts with guard pages at both ends
  of each stack. A fiber can resume on another thread; a thread's own stack cannot. The outgoing fiber is published as
  resumable only once its registers are saved and the stack pointer is on the new stack. x18, the thread pointer, TLS
  and signal masks are never carried across. Deleting or leaving the running fiber first returns to the thread's root
  stack, so cleanup never unmaps the stack it runs on. hos_setjmp.c takes its nonlocal slots from the current fiber,
  so a suspended fiber's slots outlive the thread that made it.
• Modules (win32_posix.c): the executable's path from dyld and libproc; GetModuleHandleA(NULL) is the
  executable's Mach-O header even when called from the renderer; address lookups use dladdr; loaded handles are
  recorded with their path instead of casting a handle to link_map. A packaged library beside the executable wins
  over an installed one, and a broken packaged file fails instead of falling back. A failed load keeps dlerror()'s
  text for hos_load_error() (main's #34), as on Linux.
• Restart and child processes (host/hos_relaunch.c, host/posix/darwin_process.c, new): main keeps the
  startup command line and a copy of the environment before anything changes them (after #321's hos_game_layout(),
  which changes neither). Restart re-executes that snapshot in the same process through POSIX_SPAWN_SETEXEC, with
  POSIX_SPAWN_CLOEXEC_DEFAULT so only the standard descriptors survive. Child processes remap 0/1/2 explicitly; pipes
  use pipe plus fcntl close-on-exec and F_SETNOSIGPIPE; waits retry on EINTR and reap their children.
• Menu (gpu/vk/vk_menu.cpp): share_spawn, share_clipboard and share_open use /usr/bin/pbcopy and
  /usr/bin/open with separated arguments.
• Memory (host/posix/psapi.h): resident, peak (Darwin's peak is in bytes) and the physical footprint ledger;
  the private and pagefile fields are the footprint, which includes compressed memory. They are not Linux's RssAnon
  or Windows' commit.
• Low-memory notice (host/hos_main.c): RAM from hw.memsize, allocated swap from vm.swapusage. The warning
  depends on physical RAM alone (below 10240 MiB), and its text says macOS grows swap as needed rather than treating
  the current swap as a limit. When the swap query fails, the notice still shows, without a swap figure.
• Profiler (host/posix/darwin_prof.c, new; executable only): HOS_PROFILE, HOS_THREAD_TIMES, HOS_SAMPLE,
  HOS_SAMPLE_ALL, HOS_SAMPLE_FILE and HOS_CHAIN as on Linux. Sampling suspends only its target thread, reads the
  interrupted PC, SP and FP from its Mach thread state, walks the frame records, and resumes it before any logging or
  allocation; no signal handler is installed. This is a frame-record walker, not an unwinder: callers without frame
  records may be missing, though the interrupted PC is always recorded. HOS_WATCH says that hardware watchpoints are
  unavailable and none were armed.
• scripts/prof.py: reads thin 64-bit Mach-O symbols (chosen by file magic, so the extensionless longshot works)
  relative to the image's preferred header address, names C++ frames through c++filt -n, and counts
  longshot_vk.dylib as the renderer. RX_MAPS must name the same build and architecture as the capture; a fat or
  unreadable image is reported on stderr.
• Shutdown (host/hos_main.c): on Apple, after the game thread is joined, main ends through ExitProcess(0), as
  the game's other exits do, instead of returning and running C++ static destructors under the remaining threads.
• Docs (docs/development.md): RX_MAPS on macOS.

Changes on Linux

• host/discord_rpc.c (Linux and macOS share this client):
  • the socket is created with SOCK_CLOEXEC, so a child process no longer inherits the Discord connection;
  • connect gets the exact sockaddr_un length (path plus its NUL) instead of sizeof the structure;
  • a socket path too long for sun_path is skipped with ENAMETOOLONG instead of connecting to the truncated path;
  • a send interrupted by a signal (EINTR) is retried instead of closing the connection.
• host/posix/win32_posix.c, thread error paths: a failed allocation of a thread record now returns NULL with
  ERROR_NOT_ENOUGH_MEMORY from CreateThread (before: a NULL dereference), and GetCurrentThreadId returns 0 in that
  case; a failed pthread_create destroys the record's mutex and condition variable before freeing it;
  GetThreadId of a bad handle sets ERROR_INVALID_HANDLE. The thread record's typedef gains a tag (struct HThread),
  and FreeLibrary/GetProcAddress are reformatted; neither changes behaviour.
• scripts/prof.py, every platform: a frame from any Longshot module (now also longshot_vk.so: and
  longshot_vk.dylib:) decides whether a sample outside Longshot is a wait, and a wait frame matches with its argument
  list too (recording_wait(int), as demangled names print). An empty .map file (not MSVC's) no longer registers a
  Windows module that hides the native executable in hotlist.py. tests/test_prof_symbols.py's Linux ELF fixture
  now also separates a wait-named frame from libc work under a non-wait frame.
• No behaviour change: host/hos_thread.c includes <emmintrin.h> on x86-64 only and reads
  SetThreadAffinityMask's result into an unused variable; host/posix/windows.h defines ERROR_INSUFFICIENT_BUFFER
  (122, Windows' value); comments.

How this was checked: every changed C file, preprocessed with unifdef -U__APPLE__ -D__linux__ -U_WIN32, compared
with main's. host/hos_main.c, host/hos_relaunch.c, host/posix/psapi.h and gpu/vk/vk_menu.cpp are identical;
the differences in the others are exactly the ones listed. That is text, not a Linux build.

Rebase onto main

• #321 (hos_game_layout() in main): kept first, then the startup snapshot. The snapshot reads argv, environ
  and the executable path and writes nothing, and the layout changes neither, so the order keeps "before anything
  writes" for the layout and "before anything changes the environment" for the snapshot.
• #34 (load_error/hos_load_error): load_error is declared for every POSIX host, and the Darwin
  load_darwin_library, which returns before main's dlopen, writes dlerror()'s text into it too. A new check in
  tests/darwin_host_test.c (commit b22debb8) covers it.
• #454's new test_shader_module_guard.py macOS skip is kept in tests/test_run_all_declarations.py, beside this
  PR's removal of test_prof_symbols.py from the macOS exclusions.

Tests

• tests/test_darwin_host.py (new) with darwin_host_test.c, darwin_fiber_registers.S, darwin_process_test.c,
  darwin_ipc_test.c, darwin_share_test.cpp and darwin_memory_notice_test.c: compiles the real host objects
  (win32_posix.c, darwin_fiber.c, darwin_process.c, darwin_prof.c, hos_thread.c, hos_main.c,
  hos_relaunch.c, the IPC client) for each architecture and runs threads, fibers, modules, memory, sampling, restart,
  child processes, the menu's share functions, the IPC client and the memory notice. Child processes, the share targets
  and the Discord listener are local stand-ins: no clipboard, browser, Finder or Discord action happens.
• tests/test_prof_symbols.py: builds Mach-O executables and dylibs for arm64 and x86_64 and checks the symbol
  offsets against otool/nm, then runs prof.py, crit.py, hotlist.py and chain on authored traces. It now
  runs on macOS instead of being skipped.
• tests/test_low_memory_notice.py: a macOS branch with a sysctlbyname stand-in that answers only hw.memsize
  and vm.swapusage; the same nine RAM/swap cases under macOS's rule and wording, both query failures, and --arch.
• Fixtures that cut code this PR changes: tests/test_darwin_waits.py (the thread record is now a tagged
  struct HThread), tests/display_startup_failure_test.cpp (an Apple stand-in for the startup snapshot that checks it
  ran once with the real argv/environ), tests/test_arm64_host.py (links darwin_fiber.c when present),
  tests/test_run_all_declarations.py. Their Windows and Linux text is unchanged.

Flags and settings

None new. On macOS the existing HOS_PROFILE, HOS_THREAD_TIMES, HOS_SAMPLE, HOS_SAMPLE_ALL,
HOS_SAMPLE_FILE, HOS_CHAIN and RX_MAPS now work; HOS_WATCH, HOS_PIN_BY_CORE, HOS_PIN and
HOS_RESERVE_CORE log that they are unavailable there. docs/development.md describes RX_MAPS on macOS.

Measurements

main: 5e072fc8c2adadba5ea610148be1c47800ba0cca (2026-11-14)
this PR: mac/darwin-host @ b22debb8090c4fad3c9ad8f8708bb24afce64b3b
deps: 9439c31a, main plus the six pull requests this one carries (the branch just before its own commits)
Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1; Apple clang 21.0.0; Python 3.14.10. No GPU, no game.
Scene: none. Game-free.
Method: each test from this PR, on this PR's source and on main's (a clean checkout of 5e072fc8, through the
tests' --source), interleaved: PR, main, PR, main. tests/test_darwin_host.py also on the deps tree, interleaved
with two more PR runs. tests/test_prof_symbols.py has no --source: its main side is a copy of main's tree with
this PR's test file. Every run through tools/slot.py, x86_64 executables under Rosetta.
How to reproduce: python -I -B tests/test_darwin_host.py --arch arm64 --arch x86_64 [--source <tree>],
python -I -B tests/test_low_memory_notice.py --cc clang --arch arm64 [--source <tree>/host/hos_main.c],
python -I -B tests/test_prof_symbols.py --cc clang --arch arm64 --arch x86_64.

	main @ 5e072fc8	deps @ 9439c31a	this PR @ b22debb8
tests/test_darwin_host.py, arm64 + x86_64, 2 runs per side (4 on the PR)	exit 1: stops at the CPU prerequisite (hos_setjmp.c:108, x86 assembly)	exit 1: 10 failed steps (win32_posix.c does not compile; darwin_process.c, darwin_prof.c absent; IPC link; memory notice)	exit 0: darwin_host_checks architectures=2 failed=0
tests/test_low_memory_notice.py, arm64 and x86_64, 2 runs each	exit 1: FAIL: host/hos_main.c reads macOS's memory without <sys/sysctl.h>	not run	exit 0: 17 PASS lines
tests/test_prof_symbols.py, arm64 + x86_64, 2 runs	exit 1: FAIL: shared running/waiting classification: longshot_vk.dylib	not run	exit 0
full game-free suite	525 passed, 19 failed, 17 skipped	not run	537 passed, 14 failed, 17 skipped

this PR @ b22debb8, tests/test_darwin_host.py (arm64; x86_64 has 4096-byte guards and 6 GPR checks):
threads native_id_bits=64 win_id_is_mach_port=1 names=kernel_and_remote priority=restored times=live_and_exit guards=16384 affinity=explicitly_unsupported
fibers migration=after_creator_exit nonlocal=root_and_fiber native_TLS=current guards=both_native_pages interrupted_samples=20 exit_modes=3
modules executable=dyld_path truncated=reported opaque_handle=recorded address_base=dladdr packaged_alias=preferred load_error=dlerror_text
memory resident=queried peak=Darwin_bytes private=ledger_footprint short_buffer=refused
sampling requested=100 interrupted_PC=100 caller_chains=100 self_refused=1 live_profiler=1
frameless_sampling interrupted_PC=100 required=100 omitted_callers=not_claimed
restart same_pid=1 original_argv=6 original_env=retained extra_fds=0
process spawn_closure=32/32 concurrent_unflagged_sockets=1 closed_stdio_remap=1 pipe_cloexec=1 epipe_survived=1 exit7_reaped=1 missing_child=ENOENT detached_reaped=1
darwin_share PASS copied=1 opened=1 selected=1 binary_input=1 child_failure=reported all_children=reaped GUI_browser_clipboard_actions=0
darwin_ipc PASS local_substitute=1 exact_sockaddr_length=1 cloexec=1 overlength_refused=1 no_sigpipe=1 Discord_connections=0
darwin_memory_notice PASS native_queries=RAM_and_allocated_swap physical_RAM_floor=1 dynamic_swap_text=1 GUI_calls=0
darwin_host_checks architectures=2 failed=0

this PR @ b22debb8, tests/test_prof_symbols.py:
Mach-O arch=arm64 target=14.4 offsets=preferred-base-relative names=complete running=busy+copy waiting=native-wait consumers=prof+crit+chain+hotlist executed_fixtures=0
Mach-O arch=x86_64 target=14.4 offsets=preferred-base-relative names=complete running=busy+copy waiting=native-wait consumers=prof+crit+chain+hotlist executed_fixtures=0

The sampling and classification figures are fixture checks, not performance measurements.

Testing

Tested:

• The three tests, as above.

• The fixture fixes, before and after, on this PR's source: the waits PR's test_darwin_waits.py stops at
  unknown type name 'HThread', this PR's passes (--shared-only, arm64 and x86_64); main's
  display_startup_failure_test.cpp stops at environ and hos_relaunch_keep_startup undeclared, this PR's passes
  (11 cases, 0 failed).

• The hos_load_error resolution: with load_darwin_library's dlerror() copy removed (a throwaway copy of this
  PR), tests/test_darwin_host.py fails at the new check (FAIL line=444: hos_load_error()[0] && strstr(...)).

• The Linux text, as described under "Changes on Linux" (unifdef).

• The whole suite, tests/run_all.py --jobs 6, on the same Mac:

  main @ 5e072fc8:     525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once)
  this PR @ b22debb8:  537 passed, 14 failed, 17 skipped in 193 s (macos, 6 at once)
  

  • By test name, this PR's own changes: tests/test_darwin_host.py (new) passes, and test_low_memory_notice and
    test_prof_symbols go from FAIL to pass.
  • From the pull requests it carries: test_arm64_host, test_darwin_fault_context, test_darwin_guest_memory,
    test_darwin_waits, test_vk_darwin_admission and test_vk_darwin_sdl (new) pass; test_collect_wide and
    test_mirror_trim FAIL to pass, test_dav1d_av1 skip to pass, test_appimage_package_retry FAIL to skip (#454).
  • No new failure. The 14 that remain fail on main too: test_control, test_framegen_resources,
    test_game_layout, test_mem_gate, test_mem_gate_children, test_module_ids, test_oot_state_tap,
    test_pad_layout, test_pcache_header_read, test_pipeline_cache_file, test_pipeline_cache_save,
    test_pipeline_play_cache, test_program_cache_cleanup, test_program_cache_one_variant.
  • No new crash report; the tree was clean before and after.

• History, not rerun on this branch: these commits were first made on the macOS integration branch (23095796 to
  f465faef), with the same checks on both routes; integration builds that carry them have since built and started the
  game on macOS. That speaks for those builds, not for this branch alone.

Not tested:

• Linux: no build or run here. Not tested there: Discord presence with the new socket code, the thread-creation
  error paths (they need an allocation or pthread_create failure), and scripts/prof.py on a real Linux capture.
• Windows: no build. win32_posix.c and discord_rpc.c's POSIX half are not compiled there.
• A game run on this branch, a real restart from the game's menu, a real share (clipboard, browser, Finder), and a
  real Discord client on macOS.
• macOS 14.4 itself (the deployment target is 14.4; checks ran on 26.7.1), and an Intel Mac.
• Hardware watchpoints (HOS_WATCH) and CPU pinning are unavailable on macOS by design; callers without frame records
  are not recovered by the sampler.

Needs other testers:

• Linux: build and start the game with Discord running; presence appears as before. Then
  python -I scripts/prof.py on a HOS_SAMPLE_FILE capture: the same functions as before at the top (request to open).
• Linux: the game-free suite (request to open).
• Windows (llvm-mingw): build host and renderer; nothing here is compiled there except unchanged text (request to
  open).

Screenshots

Nothing on screen.

Risk and rollback

• Linux: the Discord client and the thread error paths above. The Discord change only adds close-on-exec, the exact
  address length (what connect expects), a long-path guard and an EINTR retry; the thread changes only touch
  failure paths. scripts/prof.py may classify a few more samples as waits on Linux, where a wait frame carries its
  argument list or a Longshot frame is named by its .so.
• macOS: large new code (fibers, sampler, process spawning). Each part is checked on both CPU routes, but not yet in
  a game run on this branch.
• Windows: none.
• Rollback: revert this PR's nine commits (5d3abc9a..b22debb8); the six pull requests it carries stand alone.

Reviewer notes

• Review 9439c31a..b22debb8 (nine commits); the merges before it are the six pull requests this one depends on.
  5d3abc9a is the host itself (conflicts with #321 and #34 resolved as above); b7c9aebd the profiler consumers and
  the joined-main exit; 1bb0a62d docs; 8aa8c182, 7e0f9407, 84959356 and 08f6dba1 fixtures that cut code this
  PR changes; a19cd2ff keeps the low-RAM notice when the swap query fails; b22debb8 the hos_load_error check.
• Start with darwin_fiber.c, then the thread and module parts of win32_posix.c, then darwin_prof.c.
• On macOS, rx_backend.c's "did not load" line (#34) still turns .dll into .so in the file name; the dlerror()
  text after it names the real .dylib. Not changed here (the macOS build PR names the renderer).
• tests/test_game_layout.py (#321) still fails on macOS, now later: win32_posix.c compiles, and the link lacks the
  Darwin helpers (the macOS build PR lists them) and hos_sdl_audio_open (the renderer-correctness PR resolves it on
  macOS). It fails on main too.
• tests/test_oot_state_tap.py needs the control-sockets PR (#456), which this one does not carry; it fails on main
  too.
• Written from Apple's public SDK headers and man pages (posix_spawn, libproc, mach/thread_act.h,
  mach-o/loader.h, sysctl). Nothing was copied from another project.
• Against main at dc390e59 (7 commits after this branch's base), git merge-tree finds conflicts in
  gpu/vk/vk_api.h and host/rx_backend.c (from #438, which this branch carries; the branch before this PR's commits
  meets the same two) and in host/posix/win32_posix.c (this PR's own). A rebase before ready resolves them. #438's
  vk_api.h edit rebuilds the shader caches on 5e072fc8 but not after #370; this PR's own files are not stamped.

Issues

See #420.

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main. Based on 5e072fc8, with the six pull requests it depends on merged in; main has
  since moved to dc390e59 (conflicts named under Reviewer notes). Rebased before ready.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above). On macOS:
  537 passed, 17 skipped, and 14 failures that also fail on main, named under Testing.
• Tested in the game, or what wasn't is named under Testing. No game run on this branch; named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test
  requests are still to open.
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each,
  the median reported. For a feat without a scene: the new tests fail on main and pass here, in two interleaved runs
  each.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• Flags noted, and the switch reference updated (docs/development.md, the README). No new switch; RX_MAPS on
  macOS is in docs/development.md.
• A game-free test added for the fix, failing on main and passing here.
• A check under tests/ that reads changed source updated in this PR, not weakened.
• Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Not built there; requests to open.
• Docs updated.

**What this is for:** the macOS port (#420). The POSIX layer (`host/posix/win32_posix.c`) is written for Linux: on macOS it does not compile (Linux `ucontext` fibers, `cpu_set_t`, `/proc`, `dlinfo`), and the profiler, restart, child processes, the menu's share functions, the memory notice and Discord presence had no macOS route. This PR gives the host a native Darwin implementation of each: Mach thread identities and times, guarded native fibers that may migrate between threads, executable and module paths from dyld, same-process restart and child processes through `posix_spawn`, Darwin memory counters, the low-memory notice from `hw.memsize`, and a profiler that samples interrupted threads through Mach. **It also changes Linux in three places, listed under "Changes on Linux"**: Discord IPC, the thread-creation error paths, and `scripts/prof.py`. **What is measured:** `tests/test_darwin_host.py` builds the real host objects and runs seven groups of checks on arm64 and on x86_64 under Rosetta; it fails on `main`, and on `main` with the six pull requests this one carries, and passes here, two interleaved runs each. `tests/test_low_memory_notice.py` and `tests/test_prof_symbols.py` fail on `main` on macOS and pass here. And the full game-free suite on an Apple M4 Max (macOS 26.7.1), compared by test name with `main`. **Not measured:** a game run on this branch; Windows and Linux runs; Discord itself; macOS 14.4; an Intel Mac. Depends on #454 (run-all), #477 (arm64-host), #478 (guest-memory), #436 (fault-contexts), #437 (waits) and #438 (vk-admission). This branch carries their commits unchanged, merged in that order (`9ebd6ae8`, `56b5ed6b`, `9c679145`, `7731dc0c`, `0d62666d`, `91d8cc35`); its own commits start at `5d3abc9a`. ## Why With the six pull requests above, `win32_posix.c` still stops on macOS at its Linux-only parts, and the host has no macOS answer for the thread, process and profiling calls the game and the renderer make: ``` /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ucontext.h:51:2: error: The deprecated ucontext routines require _XOPEN_SOURCE to be defined host/posix/win32_posix.c:357:5: error: use of undeclared identifier 'cpu_set_t' host/posix/win32_posix.c:358:5: error: call to undeclared function 'CPU_ZERO' ``` ## Problem - **Fibers:** the POSIX fibers use `ucontext`, which Darwin deprecates and does not implement as an arm64 fiber switch. The game's `nn::os` fibers run on them (`host/hos_thread.c`: `CreateFiberEx`, `SwitchToFiber`), and `hos_setjmp.c`'s nonlocal slots belong to the running thread, not to the fiber. - **Threads:** thread ids come from `SYS_gettid`, names from `pthread_setname_np` on any thread, priorities from per-thread nice, affinity and topology from `cpu_set_t`. Darwin has none of these in that form. - **Modules and processes:** the executable's path comes from `/proc/self/exe` and a module's from `dlinfo`/`link_map`. Restart reads the startup command and environment back from `/proc/self/cmdline` and `/proc/self/environ`, closes descriptors through `/proc/self/fd` and re-executes `/proc/self/exe`; the menu's child processes use `pipe2` and `/proc/self/fd` too. macOS has no `/proc` and no `pipe2`. - **Memory and diagnostics:** `GetProcessMemoryInfo` reads `/proc/self/status`, the low-memory notice reads `sysinfo`, and the profiler (`HOS_PROFILE`, `HOS_SAMPLE`, `HOS_THREAD_TIMES`) uses Linux signals and `/proc`. `scripts/prof.py` reads ELF and Windows maps only. - **Menu and Discord:** the share functions run `xclip`/`xdg-open`; the Discord client connects with `sizeof` of the socket address and no close-on-exec. ## What changed ### macOS (everything below is inside `#if defined(__APPLE__)` or in new Darwin-only files) - **Threads** (`win32_posix.c`): real pthread and Mach identities. The Win32 `DWORD` thread id is the Mach thread port name; the full 64-bit native id is kept separately (`darwin_thread_id`) and the profiler uses it. Names are set on the kernel for the calling thread; a name set on another thread is kept and queryable but returns `S_FALSE`, since Darwin only lets a thread name itself; an oversized name is refused. Priorities use `THREAD_PRECEDENCE_POLICY` (a per-thread relative importance; the process's nice is untouched). `GetThreadTimes` reads live Mach times and keeps an exited thread's. `GetSystemInfo` reports the real architecture and page size. Stacks are rounded to the host page with a guard page. Affinity and logical-CPU topology fail with `ERROR_NOT_SUPPORTED` instead of inventing masks; `HOS_PIN_BY_CORE` and the renderer's reserved core say once in the log that Darwin has no binding mask; `CREATE_SUSPENDED` is refused. - **Fibers** (`host/posix/darwin_fiber.c`, new): native arm64 and x86-64 System V contexts with guard pages at both ends of each stack. A fiber can resume on another thread; a thread's own stack cannot. The outgoing fiber is published as resumable only once its registers are saved and the stack pointer is on the new stack. `x18`, the thread pointer, TLS and signal masks are never carried across. Deleting or leaving the running fiber first returns to the thread's root stack, so cleanup never unmaps the stack it runs on. `hos_setjmp.c` takes its nonlocal slots from the current fiber, so a suspended fiber's slots outlive the thread that made it. - **Modules** (`win32_posix.c`): the executable's path from dyld and libproc; `GetModuleHandleA(NULL)` is the executable's Mach-O header even when called from the renderer; address lookups use `dladdr`; loaded handles are recorded with their path instead of casting a handle to `link_map`. A packaged library beside the executable wins over an installed one, and a broken packaged file fails instead of falling back. A failed load keeps `dlerror()`'s text for `hos_load_error()` (main's #34), as on Linux. - **Restart and child processes** (`host/hos_relaunch.c`, `host/posix/darwin_process.c`, new): `main` keeps the startup command line and a copy of the environment before anything changes them (after #321's `hos_game_layout()`, which changes neither). Restart re-executes that snapshot in the same process through `POSIX_SPAWN_SETEXEC`, with `POSIX_SPAWN_CLOEXEC_DEFAULT` so only the standard descriptors survive. Child processes remap 0/1/2 explicitly; pipes use `pipe` plus `fcntl` close-on-exec and `F_SETNOSIGPIPE`; waits retry on `EINTR` and reap their children. - **Menu** (`gpu/vk/vk_menu.cpp`): `share_spawn`, `share_clipboard` and `share_open` use `/usr/bin/pbcopy` and `/usr/bin/open` with separated arguments. - **Memory** (`host/posix/psapi.h`): resident, peak (Darwin's peak is in bytes) and the physical footprint ledger; the private and pagefile fields are the footprint, which includes compressed memory. They are not Linux's `RssAnon` or Windows' commit. - **Low-memory notice** (`host/hos_main.c`): RAM from `hw.memsize`, allocated swap from `vm.swapusage`. The warning depends on physical RAM alone (below 10240 MiB), and its text says macOS grows swap as needed rather than treating the current swap as a limit. When the swap query fails, the notice still shows, without a swap figure. - **Profiler** (`host/posix/darwin_prof.c`, new; executable only): `HOS_PROFILE`, `HOS_THREAD_TIMES`, `HOS_SAMPLE`, `HOS_SAMPLE_ALL`, `HOS_SAMPLE_FILE` and `HOS_CHAIN` as on Linux. Sampling suspends only its target thread, reads the interrupted PC, SP and FP from its Mach thread state, walks the frame records, and resumes it before any logging or allocation; no signal handler is installed. This is a frame-record walker, not an unwinder: callers without frame records may be missing, though the interrupted PC is always recorded. `HOS_WATCH` says that hardware watchpoints are unavailable and none were armed. - **`scripts/prof.py`:** reads thin 64-bit Mach-O symbols (chosen by file magic, so the extensionless `longshot` works) relative to the image's preferred header address, names C++ frames through `c++filt -n`, and counts `longshot_vk.dylib` as the renderer. `RX_MAPS` must name the same build and architecture as the capture; a fat or unreadable image is reported on stderr. - **Shutdown** (`host/hos_main.c`): on Apple, after the game thread is joined, `main` ends through `ExitProcess(0)`, as the game's other exits do, instead of returning and running C++ static destructors under the remaining threads. - **Docs** (`docs/development.md`): `RX_MAPS` on macOS. ### Changes on Linux - **`host/discord_rpc.c`** (Linux and macOS share this client): - the socket is created with `SOCK_CLOEXEC`, so a child process no longer inherits the Discord connection; - `connect` gets the exact `sockaddr_un` length (path plus its NUL) instead of `sizeof` the structure; - a socket path too long for `sun_path` is skipped with `ENAMETOOLONG` instead of connecting to the truncated path; - a `send` interrupted by a signal (`EINTR`) is retried instead of closing the connection. - **`host/posix/win32_posix.c`, thread error paths:** a failed allocation of a thread record now returns `NULL` with `ERROR_NOT_ENOUGH_MEMORY` from `CreateThread` (before: a NULL dereference), and `GetCurrentThreadId` returns 0 in that case; a failed `pthread_create` destroys the record's mutex and condition variable before freeing it; `GetThreadId` of a bad handle sets `ERROR_INVALID_HANDLE`. The thread record's `typedef` gains a tag (`struct HThread`), and `FreeLibrary`/`GetProcAddress` are reformatted; neither changes behaviour. - **`scripts/prof.py`, every platform:** a frame from any Longshot module (now also `longshot_vk.so:` and `longshot_vk.dylib:`) decides whether a sample outside Longshot is a wait, and a wait frame matches with its argument list too (`recording_wait(int)`, as demangled names print). An empty `.map` file (not MSVC's) no longer registers a Windows module that hides the native executable in `hotlist.py`. `tests/test_prof_symbols.py`'s Linux ELF fixture now also separates a wait-named frame from libc work under a non-wait frame. - **No behaviour change:** `host/hos_thread.c` includes `<emmintrin.h>` on x86-64 only and reads `SetThreadAffinityMask`'s result into an unused variable; `host/posix/windows.h` defines `ERROR_INSUFFICIENT_BUFFER` (122, Windows' value); comments. How this was checked: every changed C file, preprocessed with `unifdef -U__APPLE__ -D__linux__ -U_WIN32`, compared with `main`'s. `host/hos_main.c`, `host/hos_relaunch.c`, `host/posix/psapi.h` and `gpu/vk/vk_menu.cpp` are identical; the differences in the others are exactly the ones listed. That is text, not a Linux build. ### Rebase onto `main` - **#321** (`hos_game_layout()` in `main`): kept first, then the startup snapshot. The snapshot reads `argv`, `environ` and the executable path and writes nothing, and the layout changes neither, so the order keeps "before anything writes" for the layout and "before anything changes the environment" for the snapshot. - **#34** (`load_error`/`hos_load_error`): `load_error` is declared for every POSIX host, and the Darwin `load_darwin_library`, which returns before `main`'s `dlopen`, writes `dlerror()`'s text into it too. A new check in `tests/darwin_host_test.c` (commit `b22debb8`) covers it. - **#454's** new `test_shader_module_guard.py` macOS skip is kept in `tests/test_run_all_declarations.py`, beside this PR's removal of `test_prof_symbols.py` from the macOS exclusions. ### Tests - **`tests/test_darwin_host.py` (new)** with `darwin_host_test.c`, `darwin_fiber_registers.S`, `darwin_process_test.c`, `darwin_ipc_test.c`, `darwin_share_test.cpp` and `darwin_memory_notice_test.c`: compiles the real host objects (`win32_posix.c`, `darwin_fiber.c`, `darwin_process.c`, `darwin_prof.c`, `hos_thread.c`, `hos_main.c`, `hos_relaunch.c`, the IPC client) for each architecture and runs threads, fibers, modules, memory, sampling, restart, child processes, the menu's share functions, the IPC client and the memory notice. Child processes, the share targets and the Discord listener are local stand-ins: no clipboard, browser, Finder or Discord action happens. - **`tests/test_prof_symbols.py`:** builds Mach-O executables and dylibs for arm64 and x86_64 and checks the symbol offsets against `otool`/`nm`, then runs `prof.py`, `crit.py`, `hotlist.py` and `chain` on authored traces. It now runs on macOS instead of being skipped. - **`tests/test_low_memory_notice.py`:** a macOS branch with a `sysctlbyname` stand-in that answers only `hw.memsize` and `vm.swapusage`; the same nine RAM/swap cases under macOS's rule and wording, both query failures, and `--arch`. - **Fixtures that cut code this PR changes:** `tests/test_darwin_waits.py` (the thread record is now a tagged `struct HThread`), `tests/display_startup_failure_test.cpp` (an Apple stand-in for the startup snapshot that checks it ran once with the real `argv`/`environ`), `tests/test_arm64_host.py` (links `darwin_fiber.c` when present), `tests/test_run_all_declarations.py`. Their Windows and Linux text is unchanged. ## Flags and settings None new. On macOS the existing `HOS_PROFILE`, `HOS_THREAD_TIMES`, `HOS_SAMPLE`, `HOS_SAMPLE_ALL`, `HOS_SAMPLE_FILE`, `HOS_CHAIN` and `RX_MAPS` now work; `HOS_WATCH`, `HOS_PIN_BY_CORE`, `HOS_PIN` and `HOS_RESERVE_CORE` log that they are unavailable there. `docs/development.md` describes `RX_MAPS` on macOS. ## Measurements main: `5e072fc8c2adadba5ea610148be1c47800ba0cca` (2026-11-14) this PR: `mac/darwin-host` @ `b22debb8090c4fad3c9ad8f8708bb24afce64b3b` deps: `9439c31a`, `main` plus the six pull requests this one carries (the branch just before its own commits) Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1; Apple clang 21.0.0; Python 3.14.10. No GPU, no game. Scene: none. Game-free. Method: each test from this PR, on this PR's source and on `main`'s (a clean checkout of `5e072fc8`, through the tests' `--source`), interleaved: PR, main, PR, main. `tests/test_darwin_host.py` also on the deps tree, interleaved with two more PR runs. `tests/test_prof_symbols.py` has no `--source`: its `main` side is a copy of `main`'s tree with this PR's test file. Every run through `tools/slot.py`, x86_64 executables under Rosetta. How to reproduce: `python -I -B tests/test_darwin_host.py --arch arm64 --arch x86_64 [--source <tree>]`, `python -I -B tests/test_low_memory_notice.py --cc clang --arch arm64 [--source <tree>/host/hos_main.c]`, `python -I -B tests/test_prof_symbols.py --cc clang --arch arm64 --arch x86_64`. | | main @ 5e072fc8 | deps @ 9439c31a | this PR @ b22debb8 | |---|---|---|---| | `tests/test_darwin_host.py`, arm64 + x86_64, 2 runs per side (4 on the PR) | exit 1: stops at the CPU prerequisite (`hos_setjmp.c:108`, x86 assembly) | exit 1: 10 failed steps (`win32_posix.c` does not compile; `darwin_process.c`, `darwin_prof.c` absent; IPC link; memory notice) | exit 0: `darwin_host_checks architectures=2 failed=0` | | `tests/test_low_memory_notice.py`, arm64 and x86_64, 2 runs each | exit 1: `FAIL: host/hos_main.c reads macOS's memory without <sys/sysctl.h>` | not run | exit 0: 17 PASS lines | | `tests/test_prof_symbols.py`, arm64 + x86_64, 2 runs | exit 1: `FAIL: shared running/waiting classification: longshot_vk.dylib` | not run | exit 0 | | full game-free suite | 525 passed, 19 failed, 17 skipped | not run | 537 passed, 14 failed, 17 skipped | ``` this PR @ b22debb8, tests/test_darwin_host.py (arm64; x86_64 has 4096-byte guards and 6 GPR checks): threads native_id_bits=64 win_id_is_mach_port=1 names=kernel_and_remote priority=restored times=live_and_exit guards=16384 affinity=explicitly_unsupported fibers migration=after_creator_exit nonlocal=root_and_fiber native_TLS=current guards=both_native_pages interrupted_samples=20 exit_modes=3 modules executable=dyld_path truncated=reported opaque_handle=recorded address_base=dladdr packaged_alias=preferred load_error=dlerror_text memory resident=queried peak=Darwin_bytes private=ledger_footprint short_buffer=refused sampling requested=100 interrupted_PC=100 caller_chains=100 self_refused=1 live_profiler=1 frameless_sampling interrupted_PC=100 required=100 omitted_callers=not_claimed restart same_pid=1 original_argv=6 original_env=retained extra_fds=0 process spawn_closure=32/32 concurrent_unflagged_sockets=1 closed_stdio_remap=1 pipe_cloexec=1 epipe_survived=1 exit7_reaped=1 missing_child=ENOENT detached_reaped=1 darwin_share PASS copied=1 opened=1 selected=1 binary_input=1 child_failure=reported all_children=reaped GUI_browser_clipboard_actions=0 darwin_ipc PASS local_substitute=1 exact_sockaddr_length=1 cloexec=1 overlength_refused=1 no_sigpipe=1 Discord_connections=0 darwin_memory_notice PASS native_queries=RAM_and_allocated_swap physical_RAM_floor=1 dynamic_swap_text=1 GUI_calls=0 darwin_host_checks architectures=2 failed=0 this PR @ b22debb8, tests/test_prof_symbols.py: Mach-O arch=arm64 target=14.4 offsets=preferred-base-relative names=complete running=busy+copy waiting=native-wait consumers=prof+crit+chain+hotlist executed_fixtures=0 Mach-O arch=x86_64 target=14.4 offsets=preferred-base-relative names=complete running=busy+copy waiting=native-wait consumers=prof+crit+chain+hotlist executed_fixtures=0 ``` The sampling and classification figures are fixture checks, not performance measurements. ## Testing **Tested:** - The three tests, as above. - **The fixture fixes, before and after**, on this PR's source: the waits PR's `test_darwin_waits.py` stops at `unknown type name 'HThread'`, this PR's passes (`--shared-only`, arm64 and x86_64); `main`'s `display_startup_failure_test.cpp` stops at `environ` and `hos_relaunch_keep_startup` undeclared, this PR's passes (11 cases, 0 failed). - **The `hos_load_error` resolution:** with `load_darwin_library`'s `dlerror()` copy removed (a throwaway copy of this PR), `tests/test_darwin_host.py` fails at the new check (`FAIL line=444: hos_load_error()[0] && strstr(...)`). - **The Linux text**, as described under "Changes on Linux" (`unifdef`). - **The whole suite**, `tests/run_all.py --jobs 6`, on the same Mac: ``` main @ 5e072fc8: 525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once) this PR @ b22debb8: 537 passed, 14 failed, 17 skipped in 193 s (macos, 6 at once) ``` - By test name, this PR's own changes: `tests/test_darwin_host.py` (new) passes, and `test_low_memory_notice` and `test_prof_symbols` go from FAIL to pass. - From the pull requests it carries: `test_arm64_host`, `test_darwin_fault_context`, `test_darwin_guest_memory`, `test_darwin_waits`, `test_vk_darwin_admission` and `test_vk_darwin_sdl` (new) pass; `test_collect_wide` and `test_mirror_trim` FAIL to pass, `test_dav1d_av1` skip to pass, `test_appimage_package_retry` FAIL to skip (#454). - No new failure. The 14 that remain fail on `main` too: `test_control`, `test_framegen_resources`, `test_game_layout`, `test_mem_gate`, `test_mem_gate_children`, `test_module_ids`, `test_oot_state_tap`, `test_pad_layout`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_program_cache_cleanup`, `test_program_cache_one_variant`. - No new crash report; the tree was clean before and after. - **History, not rerun on this branch:** these commits were first made on the macOS integration branch (`23095796` to `f465faef`), with the same checks on both routes; integration builds that carry them have since built and started the game on macOS. That speaks for those builds, not for this branch alone. **Not tested:** - **Linux:** no build or run here. Not tested there: Discord presence with the new socket code, the thread-creation error paths (they need an allocation or `pthread_create` failure), and `scripts/prof.py` on a real Linux capture. - Windows: no build. `win32_posix.c` and `discord_rpc.c`'s POSIX half are not compiled there. - A game run on this branch, a real restart from the game's menu, a real share (clipboard, browser, Finder), and a real Discord client on macOS. - macOS 14.4 itself (the deployment target is 14.4; checks ran on 26.7.1), and an Intel Mac. - Hardware watchpoints (`HOS_WATCH`) and CPU pinning are unavailable on macOS by design; callers without frame records are not recovered by the sampler. **Needs other testers:** - **Linux:** build and start the game with Discord running; presence appears as before. Then `python -I scripts/prof.py` on a `HOS_SAMPLE_FILE` capture: the same functions as before at the top (request to open). - **Linux:** the game-free suite (request to open). - **Windows (llvm-mingw):** build host and renderer; nothing here is compiled there except unchanged text (request to open). ## Screenshots Nothing on screen. ## Risk and rollback - **Linux:** the Discord client and the thread error paths above. The Discord change only adds close-on-exec, the exact address length (what `connect` expects), a long-path guard and an `EINTR` retry; the thread changes only touch failure paths. `scripts/prof.py` may classify a few more samples as waits on Linux, where a wait frame carries its argument list or a Longshot frame is named by its `.so`. - **macOS:** large new code (fibers, sampler, process spawning). Each part is checked on both CPU routes, but not yet in a game run on this branch. - **Windows:** none. - **Rollback:** revert this PR's nine commits (`5d3abc9a..b22debb8`); the six pull requests it carries stand alone. ## Reviewer notes - Review `9439c31a..b22debb8` (nine commits); the merges before it are the six pull requests this one depends on. `5d3abc9a` is the host itself (conflicts with #321 and #34 resolved as above); `b7c9aebd` the profiler consumers and the joined-main exit; `1bb0a62d` docs; `8aa8c182`, `7e0f9407`, `84959356` and `08f6dba1` fixtures that cut code this PR changes; `a19cd2ff` keeps the low-RAM notice when the swap query fails; `b22debb8` the `hos_load_error` check. - Start with `darwin_fiber.c`, then the thread and module parts of `win32_posix.c`, then `darwin_prof.c`. - On macOS, `rx_backend.c`'s "did not load" line (#34) still turns `.dll` into `.so` in the file name; the `dlerror()` text after it names the real `.dylib`. Not changed here (the macOS build PR names the renderer). - `tests/test_game_layout.py` (#321) still fails on macOS, now later: `win32_posix.c` compiles, and the link lacks the Darwin helpers (the macOS build PR lists them) and `hos_sdl_audio_open` (the renderer-correctness PR resolves it on macOS). It fails on `main` too. - `tests/test_oot_state_tap.py` needs the control-sockets PR (#456), which this one does not carry; it fails on `main` too. - Written from Apple's public SDK headers and man pages (`posix_spawn`, `libproc`, `mach/thread_act.h`, `mach-o/loader.h`, `sysctl`). Nothing was copied from another project. - Against `main` at `dc390e59` (7 commits after this branch's base), `git merge-tree` finds conflicts in `gpu/vk/vk_api.h` and `host/rx_backend.c` (from #438, which this branch carries; the branch before this PR's commits meets the same two) and in `host/posix/win32_posix.c` (this PR's own). A rebase before ready resolves them. #438's `vk_api.h` edit rebuilds the shader caches on 5e072fc8 but not after #370; this PR's own files are not stamped. ## Issues See #420. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. Based on 5e072fc8, with the six pull requests it depends on merged in; `main` has since moved to dc390e59 (conflicts named under Reviewer notes). Rebased before ready. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). On macOS: 537 passed, 17 skipped, and 14 failures that also fail on `main`, named under Testing. - [x] Tested in the game, or what wasn't is named under Testing. No game run on this branch; named under Testing. - [ ] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test requests are still to open. - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. For a `feat` without a scene: the new tests fail on `main` and pass here, in two interleaved runs each. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). No new switch; `RX_MAPS` on macOS is in `docs/development.md`. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Not built there; requests to open. - [x] Docs updated.

mooki added 29 commits 2026-11-14 16:30:51 +00:00

fix(runtime): preserve guest pages when discarding on macOS ... fe35e00ac6

Reserve the fixed guest window with a non-overwriting Mach mapping. Reclaim
complete owned host pages with fresh anonymous mappings and zero only the
partial-page edges, preserving adjacent 4 KiB guest pages on 16 KiB hosts.
Use the actual host granule for non-guest allocation/protection and report
macOS discard failures instead of returning success.

Add game-free source-linked arm64/Rosetta tests covering collisions, repeated
free/recommit, unaligned ranges, failure paths and resident-memory reduction.
Keep Linux and Windows paths unchanged. Build-list integration is separate.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(runtime): retain guest-page rounding in Darwin memory wrappers ... a350a531fa

Keep the POSIX layer's outward 4 KiB MEM_DECOMMIT and inward 4 KiB MEM_RESET
contracts above the byte-exact Darwin helper. Check arithmetic before rounding,
and keep separate real-memory assertions for helper and wrapper ranges.

Default the Darwin regression test to the native architecture. Both CPU routes
remain explicitly selectable for acceptance without requiring Rosetta or arm64
execution from every ordinary macOS suite run.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(runtime): compile host jumps and pause hints on macOS arm64 ... 56b5ed6b7e

Add a Darwin arm64 host nonlocal jump implementation with the AAPCS64
callee-saved registers, stack and frame pointers, link register and floating
point environment. Leave x18 untouched and reject arm64e authenticated
contexts explicitly instead of using raw saved pointers there.

Keep both x86 assembly implementations unchanged. Select a Darwin pause
hint without changing Windows or Linux code, and guard the x86 header.
Add game-free object builds and assembly-controlled register/return tests,
including a Rosetta route and a test against another source checkout.

The new Darwin source must be added to the macOS host build list separately;
this topic does not enable a full macOS build or change fibers or VM pages.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(runtime): translate macOS fault contexts on both CPU routes ... da567be4b6

Decode Darwin signal contexts and access faults for arm64 and x86_64,
copy supported context edits back on signal return, and prevent recursive
callback dispatch when a fault handler itself faults. Keep the non-Apple
bridge text and the x64-shaped CONTEXT layout unchanged.

Add source-linked, game-free owned-memory fault tests for both targets,
including page repair, PC redirection, prior handlers and an inaccessible
handler stack. Document the native PC/SP contract and remaining limits.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(runtime): keep Darwin fault checks native by default ... 7731dc0c18

Do not make the ordinary Apple Silicon suite depend on Rosetta. Default
to the running Python process's architecture and retain explicit repeated
--arch options for dual-route qualification. Document that command and
count only files when reporting crash-report directory changes.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(runtime): port Win32 waits and deadlines to macOS ... 2e939743d5

Add macOS 14.4 public address waits, awake-uptime deadlines, relative
monitor waits, absolute timer sleeps, and real initialization for zeroed
SRW locks. Keep the Linux source behind each Apple branch byte-identical.

Exercise the extracted layer with real primitives and C/C++ header clients
on arm64 and x86_64. Document the existing broadcast address wakes, timer
restrictions and non-atomic polling, without claiming a complete host port.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(runtime): link Darwin waits into the pointer fixture ... 35ef0284fc

The pointer mailbox test uses the real POSIX SRW wrappers, which now call
into darwin_wait.c. Compile that production C dependency and link it into
the existing native-default fixture on macOS only. Keep the Linux and
Windows compiler invocation and every existing assertion unchanged.

The full-suite failure at 1da4f9b3 is retained. This repair does not resolve
the held cross-image address-wait contract decision or qualify the port.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): share address wait state through the executable ... 3861478f6e

Move the address sequence words into an executable-only provider and resolve
that canonical service once in each shim image. Fail closed when the host
provider is absent; retain full-width comparisons and broadcast ordering.

Extend the existing wait driver with real host/two-dylib behavior coverage,
including upper-half and cross-word changes, same-image controls, image-local
error state, broadcast wakes, and rejection of a dylib-only provider. Keep
the existing timing campaign intact and independently selectable.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(runtime): cut main's int hos_commit in the Darwin guest-memory test ... 9c6791454f

Main's #324 (a4c77174) made hos_commit in host/mkw_glue.c return whether
the range was committed, so its definition now starts "int hos_commit(".
The Darwin guest-memory test cut the function by its old "void" head and
stopped with a ValueError on main's source, on this branch and on main alike,
before compiling anything. Cut it by the new head.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs(runtime): describe the executable-owned macOS address-wait state ... 0d62666d45

The wait section still said each image's darwin_wait.c is self-contained.
Since the address sequence words moved into the executable, the renderer's
copy finds them through RTLD_MAIN_ONLY, a missing provider aborts, and the
build has to link darwin_wait_state.c into the host only. Say so, and name
the test's --shared-only mode.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

feat(vk): add macOS Vulkan loader and device admission ... e294efe9db

Select a local Vulkan loader or MoltenVK with RX_VK_LIBRARY and keep SDL and
renderer startup on that same library. Capture SDL's instance extensions on
the window thread, enable advertised portability extensions, and name missing
admission requirements and VkResults without changing Windows or Linux paths.

Keep the non-Apple host API at 21 for existing prebuilt renderers; append the
two macOS callbacks only in Apple's version 22. Cover pure decisions, legacy
platform expectations, ABI layout, and the SDL2/SDL3 bridge without a GPU.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): use semantic admission contracts ... 81053e98f6

Remove the SDL source-string wiring and source-index ordering assertions while
keeping the sixteen compiled behavioral cases and surface adapter compile.
Replace the comment-sensitive API header hash with fixed baseline field names,
byte offsets, versions and sizes, retaining typed Apple callback checks and
both platforms' actual admission decisions and negative controls.

Production code is unchanged. Prior full-suite and device receipts remain
bound to 818e33b1; only the affected game-free tests were rerun.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): follow main's LoadLibraryA and float-add choice in the admission fixture ... 91d8cc3527

The fixture cuts LoadLibraryA and create_device out of main's source. Since
#34, LoadLibraryA keeps dlerror()'s text in load_error, and since #191
native_fadd_wanted also takes the device ID and driver version, so the cut
code no longer compiled against the fixture's stand-ins on this base. Give
it load_error and a dlerror stand-in, and the four-argument stand-in. No
expectation changed.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test: teach the game-free runner about macOS ... 6e666d70e9

Recognize Darwin separately, retain explicit game/device and Windows-only
exclusions, and leave actual host, cache and memory portability failures
running. Exclude only the Linux AppImage bootstrap and ELF symbol fixture.

Discover macOS dav1d through pkg-config and link its fixture from that prefix.
Keep the existing Linux and Windows paths, declarations and fixture code;
repair the Apple-only memcpy macro and wait lookup collisions.

Exercise detection, tool choices, declarations, skip overrides, summaries
and per-system timing/sharding through the runner's own functions.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test: skip main's new Windows-only guard check on macOS, and undo the SDK's vsnprintf macro in a fixture ... 9ebd6ae863

Two checks main added since the port's base meet the macOS runner:

- tests/test_shader_module_guard.py tests Windows structured exceptions
  and is skipped on Linux for it; it is now skipped on macOS for the same
  reason, and the declaration test's macOS exclusion set names it (18).
- tests/test_oot_state_tap.py's fixture counts vsnprintf calls with a
  macro of its own. The macOS SDK's fortified stdio already defines
  vsnprintf as a macro, so -Werror stopped the compile. Under __APPLE__
  only, the SDK's macro is undefined first, as test_collect_wide does
  for memcpy. The check still fails on macOS at host/control.c's Linux
  socket calls, which the control-bridge PR ports.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/arm64-host (#P07) into mac/darwin-host ... 3c17736281

Carries the arm64-host pull request's commit, which the Darwin host
needs (its native jump ABI and pause hint). No conflict.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/guest-memory (#P08) into mac/darwin-host ... baed9984a8

Carries the guest-memory pull request's commits, which the Darwin host
needs (darwin_mem.h). docs/development.md: both it and the arm64-host
pull request add a section after "Log levels"; both are kept, the CPU
checks first.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/fault-contexts (#436) into mac/darwin-host ... dcb4c52122

Carries the fault-contexts pull request's commits, which the Darwin host
needs (darwin_fault.h). host/posix/win32_posix.c: its Apple include
block keeps both darwin_mem.h and darwin_fault.h.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/waits (#437) into mac/darwin-host ... a63380cafc

Carries the waits pull request's commits, which the Darwin host needs
(darwin_wait.h and the executable-owned wait state). No conflict.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/vk-admission (#438) into mac/darwin-host ... 9439c31af1

Carries the vk-admission pull request's commits, which the Darwin host
needs (darwin_vulkan.h and the Apple MkwVkHost callbacks).
host/posix/win32_posix.c: its Apple include block keeps darwin_mem.h,
darwin_fault.h and darwin_vulkan.h.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

feat(macos): complete Darwin host lifecycle and diagnostics ... 5d3abc9a99

Implement native migrating fibers and fiber-owned nonlocal state, supported restart/spawn descriptor inheritance, Darwin thread/module/memory queries and interrupted-PC profiling. Keep platform limits explicit and cover native arm64 and Rosetta with game-free local fixtures.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): integrate profile consumers and joined-thread shutdown ... b7c9aebd36

Decode thin Mach-O function symbols relative to the preferred image header, retain demangled names and classify native renderer work and waits through the shared consumers. Keep Windows maps and ELF loading, and exercise authored arm64/x86 metadata without executing fixtures. End the Apple joined-main tail through ExitProcess rather than running static destructors under remaining threads.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs(macos): describe Darwin profile symbol loading ... 1bb0a62d1d

RX_MAPS on macOS must name the unstripped thin longshot and
longshot_vk.dylib of the sampled build and architecture. The readers'
commands are unchanged; c++filt names C++ frames. Records the loader's
preferred-header-relative offsets, its refusal of fat or unreadable
images on stderr, and that non-MSVC .map files are ignored, as
scripts/prof.py at 18f1df44 implements them.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): integration fix: cut the tagged HThread in the waits fixture ... 8aa8c182d7

DarwinHost tags the thread record (typedef struct HThread) for its
Darwin thread registry. tests/test_darwin_waits.py still searched for
an untagged "typedef struct {" after the threads marker, found a later
struct and cut an empty HThread definition, so the fixture could not
compile. Found when this branch was merged into macos/integrated
(receipt i2-check-02-darwin-waits-shared; the same fix is 1cb8752f
there). The fixture now cuts the first typedef of the threads section,
tagged or not, and stops with a FAIL when that cut does not hold the
HThread definition. No production source or wait behavior changes.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): integration fix: stub the kept startup in the display fixture ... 7e0f940761

host/hos_main.c's main now keeps its startup command and environment
on Darwin first (hos_relaunch_keep_startup(argc, argv, environ)).
tests/test_display_startup_failure.py compiles that main with authored
stubs and had no declaration for either, so the macOS fixture failed
to build (full-suite receipt full-1cb8752f). An Apple-only stub records
the call, refuses it after display or game work began, and every case
checks it ran once with the application's argv and environ. Windows
and Linux fixture text is unchanged.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): integration fix: prof symbols run on macOS, not a skip ... 84959356df

This branch removed test_prof_symbols.py's macOS RUN_ALL_SKIP: it now
checks Mach-O symbols on macOS. RunAll's declaration test still listed
it among the macOS exclusions, so test_run_all_declarations.py failed
(full-suite receipt full-1cb8752f). Drop it from the expected set.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): the memory-notice fixture runs the Darwin query ... 08f6dba104

darwin-host's memory_notice() reads hw.memsize and vm.swapusage through
sysctlbyname, but the fixture only offered Linux's sysinfo spy, so on
macOS it stopped at an incomplete struct xsw_usage, an undeclared
sysctlbyname and errno (full-suite receipt full-1cb8752f).

Add a macOS spy branch: the real <sys/sysctl.h> struct xsw_usage and an
authored sysctlbyname that answers only hw.memsize and vm.swapusage at
their sizes, in bytes. The same nine cases run under macOS's
physical-RAM rule and wording; the boot log must qualify the allocated
swap; a byte short of the next MiB must truncate; both query failures
must log their diagnostic and show no warning. --arch arm64|x86_64
builds and runs one macOS architecture (-mmacosx-version-min=14.4).
Linux and Windows cases and assertions are unchanged.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): a failed swap query still shows the low-RAM notice ... a19cd2ff86

On macOS the warning depends on physical RAM alone, but when the
vm.swapusage query failed memory_notice() logged the reason and
returned, so a Mac under 10 GiB of RAM got no notice.

Keep the logged reason, then decide and show the RAM notice anyway.
Without a swap figure (UINT64_MAX between the two functions), the
notice says "This Mac has N GiB of RAM; its allocated swap could not be
read." and the boot lines say "allocated swap unavailable" instead of a
number; the dynamic-swap qualifier is logged only with a real figure.
With a figure, the macOS text and logs are unchanged. Linux and Windows
lines are unchanged.

tests/test_low_memory_notice.py: a failed swap query across the RAM
cases logs the reason; below 10240 MiB it shows the notice without a
swap figure (visible and headless), at or above it shows none; no boot
line claims a swap figure.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): the Darwin LoadLibraryA keeps dlerror()'s text for hos_load_error ... b22debb809

main's #34 made the POSIX LoadLibraryA keep dlerror()'s text, which
hos_load_error() returns and host/rx_backend.c's load_vulkan and the
renderer's mkwvk_init log when a library does not load. This branch's
Darwin LoadLibraryA loads through load_darwin_library, which returned
before main's dlopen; the rebase onto main resolved that conflict by
keeping the text there too (load_error is declared for every POSIX host,
and load_darwin_library writes dlerror() into it on a failed dlopen).

darwin_host_test.c's module checks now ask for a packaged library with
no file behind it: LoadLibraryA returns NULL with ERROR_FILE_NOT_FOUND,
and hos_load_error() names the .dylib it tried. A Windows system library
name (refused before any dlopen) leaves the text empty, as on Linux, and
a later successful load clears it. Windows and Linux are unchanged.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

jarynclouatre referenced this pull request 2026-11-14 16:50:33 +00:00

fix(vk): on Linux the texture helpers started inside a replay run on every core again #499

All checks were successful

Hide checks

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 4s

Details

PR description / Description follows the template (pull_request_target) Successful in 5s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 8s

Details

Third-party parts / As their notes say (pull_request) Successful in 11s

Details

This pull request has changes conflicting with the target branch.

• docs/development.md
• gpu/vk/vk_api.h
• host/posix/win32_posix.c
• host/rx_backend.c

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title WIP: feat(macos): native Darwin threads, fibers, restart, process I/O and profiler (#490)

Merge commit body Reviewed-on: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/490

WIP: feat(macos): native Darwin threads, fibers, restart, process I/O and profiler (#490) Reviewed-on: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/490 Copy merge commit message

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'mac/darwin-host:mooki-mac/darwin-host'

git switch 'mooki-mac/darwin-host'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Linux only

  

platform/steam-deck

Steam Deck

  

platform/windows

Windows only

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!490

No description provided.