MagicBeanDev/Longshot
MagicBeanDev/Longshot
Watch
55
68
Fork
You've already forked Longshot
 71
Code Issues 125 Pull requests 195 Projects Releases 8 Packages Wiki Activity Actions 445

WIP: fix(runtime): add macOS waits and deadlines, shared by host and renderer #437

Draft
mooki wants to merge 4 commits from mooki/Longshot:mac/waits into main
pull from: mooki/Longshot:mac/waits
merge into: MagicBeanDev:main
Conversation 0 Commits 4 Files changed 9 +1252 -2
mooki commented 2026-11-14 13:55:09 +00:00
Member

What this is for: the macOS port (#420). The POSIX Win32 layer's waits (host/posix/win32_posix.c) use Linux
futexes, a condition-variable clock attribute and clock_nanosleep, none of which macOS has. This PR gives that layer
macOS 14.4 waits, deadlines and SRW locks, for both CPU routes. The executable and the renderer each link their own copy
of the layer, so the address waits also share one set of wait state through the executable: without that, a wake in
one image never reached a waiter in the other (20 of 20 such waits stuck, per architecture). The Windows and Linux code
is unchanged. On its own it does not make the macOS host build or run. What is measured: a new game-free test builds
the real wait functions (cut from win32_posix.c) with darwin_wait.c and runs them with real locks, waits, wakes and
clocks. Its two-image mode builds two separately linked dylibs plus the executable and checks every waiting image against
every waking image. It fails on main (no Darwin waits) and passes here, on arm64 and on x86_64 under Rosetta, two
interleaved runs each. The cross-image failure is shown on this PR's commits without the shared state. Also measured:
the full game-free suite on an Apple M4 Max (macOS 26.7.1), compared by test name with main. Not measured: Windows,
Linux, a macOS build or game run, macOS 14.4 itself, an Intel Mac, sleep and resume, and the Rosetta timing campaign
on a quiet machine (below).

Why

A Mac cannot build LS's POSIX host while its waits are futex calls. And a renderer and an executable that each keep
private wait state lose each other's wakes.

Problem

  • On main, win32_posix.c includes <linux/futex.h> (line 12), so the whole file stops there on macOS:
    host/posix/win32_posix.c:12:10: fatal error: 'linux/futex.h' file not found.
  • Behind it, the wait group is Linux-only:
    • futex() (:81) is syscall(SYS_futex, …), used by WakeConditionVariable, SleepConditionVariable*,
      WaitOnAddress and both WakeByAddress* (:88-137);
    • monitor_init (:64) sets the condition variable's clock with pthread_condattr_setclock, which Darwin lacks;
    • the timer wait (:489) is clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, …), which Darwin lacks;
    • SRWLOCK is a bare pthread_rwlock_t whose zeroed storage counts as initialized (windows.h:153-156). On Darwin a
      zeroed rwlock is not a valid lock.
  • Darwin's CLOCK_MONOTONIC counts system sleep, unlike Linux's. A deadline on it would expire while the Mac sleeps.
  • The layer is built twice: into the executable and, on its own, into the renderer (win32_posix.c's first comment).
    A futex is keyed by the address, so on Linux the two copies wake each other. macOS's public address wait
    (os_sync_wait_on_address) also keys by address, but a Win32 WaitOnAddress compares 1 to 8 bytes, so the Darwin side
    waits on a sequence word that each wake bumps. With one sequence table per image, a wake through the executable's copy
    never bumped the word that the renderer's copy was waiting on.
  • On main, the renderer's recorder idle (RX_VK_RECORDER_IDLE=wait, gpu/vk/vk_replay.cpp:135-136) uses
    WaitOnAddress and WakeByAddressSingle from the renderer's copy.

What changed

  • host/posix/darwin_wait.c and .h (new), all inside #if defined(__APPLE__):
    • darwin_now_ns: awake uptime (mach_absolute_time), so sleep does not use up a deadline. FILETIME stays realtime.
    • darwin_monitor_init and darwin_cond_timedwait: relative pthread timed waits, recomputed from one fixed uptime
      deadline after every return.
    • darwin_wait_until_ns: mach_wait_until with the timebase rounded up and KERN_ABORTED retried, for the timer.
    • darwin_wait_word / darwin_wake_word: os_sync_wait_on_address with a timeout, for condition variables.
    • darwin_wait_address / darwin_wake_address: Win32 address waits over 1, 2, 4 or 8 bytes, upper-half and
      cross-word changes included. The waiter snapshots its sequence word before comparing; a wake bumps the word, then
      wakes all. Both Win32 wake names broadcast, as the Linux layer does.
    • darwin_srw_init, darwin_srw_native and darwin_srw_trylock: zeroed SRWLOCK storage gets a real
      pthread_rwlock_init the first time it is used. A try-lock never waits for another thread's initialization.
  • host/posix/darwin_wait_state.c (new): hos_posix_address_wait_sequences() returns the process's one array of
    256 zero-initialized sequence words. It belongs in the executable only. Each image's darwin_wait.c finds it once
    with dlsym(RTLD_MAIN_ONLY, …). If the executable lacks it, the first address wait or wake aborts with
    fatal: main executable has no Darwin address-wait state provider; there is no private fallback, and a provider in
    any other image is not used.
  • host/posix/win32_posix.c: #if defined(__APPLE__) branches at the futex include, now_ns, abs_after_ms,
    monitor_init, the monitor wait, the condition-variable wake and wait, SRW release and reacquisition, WaitOnAddress,
    WakeByAddress*, Sleep and the timer wait. Each #else is main's text unchanged. Events, semaphores and the
    multiple-object polling are unchanged.
  • host/posix/windows.h: on Apple only, SRWLOCK is DarwinSRWLock with wrappers that initialize it, and
    QueryPerformanceCounter / GetTickCount* read awake uptime. The pause instruction is untouched.
  • tests/test_darwin_waits.py (new) cuts the synchronization functions out of win32_posix.c and links them with
    darwin_wait.c, darwin_wait_state.c and the real windows.h. No fake locks, waits, wakes or clocks.
    • The default run checks contention without lost wakes (32- and 64-bit, upper-half only), wake counts, fixed deadlines
      under spurious wakes, zero, finite and infinite waits, both SRW modes, events, semaphores, the absolute timer, thread
      completion and the polling limits. Its timing campaign has fixed tolerances: a 10 ms wait may end 0.1 ms early or
      50 ms late.
    • tests/darwin_wait_sharing_test.c (new), with --shared-only: the executable and two separately linked dylibs,
      each with its own copy of the wait functions, and the real provider in the executable. Every waiting image against
      every waking image, 8 layouts from 1 to 8 bytes, both wake names, a waiter already queued and a waiter just entering:
      288 cases of 4 rounds. Also four-waiter broadcasts across images, polls, and nine runs of an executable without the
      provider (a fourth dylib exports one), which must abort.
    • It builds the running Python's architecture by default; repeated --arch selects both routes and --source-tree
      another tree. It skips on Windows and prints SKIP on Linux.
  • tests/test_display_pointer.py: on macOS only, it compiles darwin_wait.c and links it into its fixture, whose
    SRW locks now need it. The Linux and Windows command and every assertion are unchanged.
  • docs/development.md: the macOS wait subset, its clock, the shared address-wait state, the test command, and the
    build's source lists.
  • Unchanged: nx2/build.py. The macOS build PR in this series (build(macos): classify Darwin helpers …) puts
    darwin_wait.c into both images and darwin_wait_state.c into the executable only. The executable has to export
    hos_posix_address_wait_sequences; the macOS build flags PR (build(macos): discover dual-architecture toolchains …)
    exports _hos_posix_* with -exported_symbol, as Linux already does with --export-dynamic-symbol=hos_posix_*.

Flags and settings

No game, renderer or build switch. The new test's own options:

Flag What it does Default now Recommendation Helps on Doesn't help on Untested on
--arch (repeatable) the fixture's targets the running Python's architecture both, --arch arm64 --arch x86_64, where Rosetta is installed macOS Windows, Linux an Intel Mac
--shared-only only the two-image checks and the C++ header checks, without the timing campaign off on for a busy machine or for the cross-image question timing-free runs timing
--source-tree the tree whose wait functions are built this checkout a tree of main, for the failing side comparisons
--out keeps the fixtures and logs in a new folder a temporary folder set it for evidence

Measurements

main: 5e072fc8c2adadba5ea610148be1c47800ba0cca (2026-11-14)
this PR: mac/waits @ 0d62666d45bba74fe344eac19e73b179481e85c1
Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1 (25G241); Apple clang 21.0.0; Python 3.14.10. No GPU used.
Scene: none. Game-free.
Method: tests/test_darwin_waits.py --shared-only --arch arm64 --arch x86_64 on main's tree (--source-tree, a clean
checkout of 5e072fc8) and on this PR, interleaved: main, PR, main, PR. For the cross-image defect, the waits commits
of this PR without the shared state (35ef0284) ran a separate two-dylib probe that links the same wait functions
twice. Then tests/run_all.py --jobs 6 on this PR, compared by test name with a main run on the same Mac.
How to reproduce: python -I -B tests/test_darwin_waits.py --shared-only --arch arm64 --arch x86_64, then the same with
--source-tree <a checkout of main>.

main @ 5e072fc8 this PR @ 35ef0284, without the shared state this PR @ 0d62666d
tests/test_darwin_waits.py --shared-only, both routes, run 1 exit 1: no host/posix/darwin_wait.c (the mode is added with the shared state) exit 0
the same, run 2 exit 1: the same exit 0
a wake in another image, per architecture no Darwin waits at all 20 of 20 waits stuck (10 A→B, 10 B→A); 20 of 20 same-image controls woke 768 of 768 cross-image waits woke, 0 early
same-image waits, per architecture 20 of 20 384 of 384
four-waiter broadcasts across images, per architecture 16 of 16 groups, 64 waiters
executable without the provider, per architecture 9 of 9 abort, as they must
full game-free suite 525 passed, 19 failed, 17 skipped not run 526 passed, 19 failed, 17 skipped 
this PR @ 35ef0284, without the shared state (arm64; x86_64 printed the same):
PAIR A_to_A rounds=10 kernel_wait_observed=10 requested_image_woke=10 still_waiting_after_100ms=0 same_image_rescued=0 early_unjudged=0
PAIR B_to_B rounds=10 kernel_wait_observed=10 requested_image_woke=10 still_waiting_after_100ms=0 same_image_rescued=0 early_unjudged=0
PAIR A_to_B rounds=10 kernel_wait_observed=10 requested_image_woke=0 still_waiting_after_100ms=10 same_image_rescued=10 early_unjudged=0
PAIR B_to_A rounds=10 kernel_wait_observed=10 requested_image_woke=0 still_waiting_after_100ms=10 same_image_rescued=10 early_unjudged=0
RESULT controls_stuck=0 cross_image_stuck=20 expected_cross_image_stuck=0
FAIL cross_image_wait_wake_binding

this PR @ 0d62666d (each run, per architecture; a few of the 288 PAIR lines):
IMAGES distinct_dylibs=2 distinct_wait_entries=3 distinct_wake_entries=3 dladdr_identity=1 image_local_errors=3 main_win32_exports=0
PAIR wait=A wake=B layout=u64-upper-half size=8 wrapper=single phase=queued rounds=4 observed=4 early_unjudged=0 completed=4
PAIR wait=B wake=host layout=u32-cross-word size=4 wrapper=single phase=queued rounds=4 observed=4 early_unjudged=0 completed=4
BROADCAST layout=u8-aligned size=1 wrapper=single kernel_wait_observed=4 completed=4
STATE canonical_wake_increments=1174 decoy_wake_increments=0 stable_provider=1
fatal: main executable has no Darwin address-wait state provider
EXPECTED missing-provider SIGABRT
SHARING_CHECKS arch=arm64 missing_provider_probes=9 expected_exit=86
PASS test_darwin_waits architectures=arm64,x86_64 mode=sharing-only

main @ 5e072fc8 (each run):
FileNotFoundError: [Errno 2] No such file or directory: '…/host/posix/darwin_wait.c'

The stuck waiters at 35ef0284 were each woken through their own image afterwards (same_image_rescued=10), which
shows they were waiting, not lost. decoy_wake_increments=0: a dylib that also exports a provider is never used.

Testing

Tested:

  • The two-image test, both ways, both routes, twice each, and the cross-image probe on 35ef0284, as above.

  • The whole of win32_posix.c now gets past its includes on macOS (syntax only). It still fails, with 75 errors on
    each route, in code that other macOS PRs port: the fiber ucontext.h routines, CPU affinity, thread names,
    MAP_STACK, another thread's CPU clock, dlinfo and the fault context.

  • The whole suite, tests/run_all.py, 6 at once, on the same Mac (the runner labels macOS linux):

    main @ 5e072fc8:     525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once)
this PR @ 0d62666d:  526 passed, 19 failed, 17 skipped in 180 s (linux, 6 at once)
    • By test name, the only difference is the new tests/test_darwin_waits.py (pass, native arm64 with its timing
      campaign, as run_all runs it). tests/test_display_pointer.py passes on both. No other check changed status.
    • The same 19 checks fail on both. They are macOS portability failures of main, which other macOS pull requests
      address: test_appimage_package_retry, test_collect_wide, test_control, test_framegen_resources,
      test_game_layout, test_low_memory_notice, test_mem_gate, test_mem_gate_children, test_mirror_trim,
      test_module_ids, test_oot_state_tap, test_pad_layout, test_pcache_header_read, test_pipeline_cache_file,
      test_pipeline_cache_save, test_pipeline_play_cache, test_prof_symbols, test_program_cache_cleanup,
      test_program_cache_one_variant.
    • test_game_layout gets further here: main stops at linux/futex.h, this PR at the fiber ucontext.h include.

Not tested:

  • Windows and Linux. The non-Apple text of win32_posix.c and windows.h is main's; that is not a build there.
  • A macOS build or game run, and so the executable's export of the provider in a real link.
  • The timing campaign under Rosetta on a quiet machine. Before this rebase, at 1da4f9b3, on a heavily loaded Mac,
    one 10 ms absolute-timer wait under Rosetta took 68.4 ms, over the 60 ms ceiling; native arm64 passed. The ceiling
    stays; the quiet rerun has not been done. The suite runs the timing campaign natively only.
  • macOS 14.4 itself (the deployment target is 14.4; it ran only on 26.7.1), an Intel Mac, sleep and resume, a clock
    adjustment, and every possible thread interleaving: the wake checks are repeated observations.

Needs other testers:

  • Windows (llvm-mingw): build the host and run the game-free suite; expect no change, and the new test skipped (request
    to open).
  • Linux: build the host and run the game-free suite, especially test_display_pointer; expect no change, and the new
    test printing SKIP (request to open).
  • macOS 14.4 on Apple Silicon: python -I -B tests/test_darwin_waits.py --arch arm64 --arch x86_64 on a quiet machine;
    expect PASS, with no late sample (request to open).

Screenshots

Nothing on screen.

Risk and rollback

  • Windows and Linux: no change in behaviour. Every new line is inside #if defined(__APPLE__) or in a new file.
  • macOS: an executable built without darwin_wait_state.c, or without exporting its function, aborts at its first
    address wait. That is on purpose: a private fallback would bring back the lost wakes.
  • Wakes: both wake names wake every waiter on the word, and two addresses can share a sequence word. Callers must
    recheck their condition after a wake, as the Linux layer already requires.
  • Win32 differences kept from the Linux layer: relative-only waitable timers that ignore the wait's millisecond
    limit, periodic and APC timers; a polling wait-all that can take one object before all are ready.
  • Rollback: revert the four commits. There is no switch.

Reviewer notes

  • Review 5e072fc8..0d62666d: four commits and no dependency. 2e939743 is the waits; 35ef0284 links
    darwin_wait.c into the pointer test's fixture; 3861478f moves the address-wait state into the executable;
    0d62666d documents that.
  • Start with darwin_wait.c's address waits and init_address_sequences, then the Apple branches in
    win32_posix.c's monitor and condition-variable waits.
  • Written from Apple's SDK headers and pages (os/os_sync_wait_on_address.h, pthread/pthread.h, clock_gettime(3),
    mach/mach_time.h, dlsym(3)). Nothing was copied from another project.

Issues

See #420.

Checklist

For everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description).
  • Rebased on current main. Based on 5e072fc8.
  • python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above). On macOS:
    526 passed, 17 skipped, and the same 19 failures as main, named under Testing.
  • Tested in the game, or what wasn't is named under Testing. No game run; named under Testing.
  • Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test
    requests are still to open.
  • Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each,
    the median reported. For a fix: the new test fails on main and passes here, in two interleaved runs each.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Flags noted, and the switch reference updated (docs/development.md, the README). Test options only, documented in
    docs/development.md.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened. test_display_pointer.py links
    the new helper on macOS; its assertions are unchanged.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Not built there; requests to open.
  • Docs updated.
**What this is for:** the macOS port (#420). The POSIX Win32 layer's waits (`host/posix/win32_posix.c`) use Linux futexes, a condition-variable clock attribute and `clock_nanosleep`, none of which macOS has. This PR gives that layer macOS 14.4 waits, deadlines and SRW locks, for both CPU routes. The executable and the renderer each link their own copy of the layer, so the address waits also share one set of wait state through the executable: without that, a wake in one image never reached a waiter in the other (20 of 20 such waits stuck, per architecture). The Windows and Linux code is unchanged. On its own it does not make the macOS host build or run. **What is measured:** a new game-free test builds the real wait functions (cut from `win32_posix.c`) with `darwin_wait.c` and runs them with real locks, waits, wakes and clocks. Its two-image mode builds two separately linked dylibs plus the executable and checks every waiting image against every waking image. It fails on `main` (no Darwin waits) and passes here, on arm64 and on x86_64 under Rosetta, two interleaved runs each. The cross-image failure is shown on this PR's commits without the shared state. Also measured: the full game-free suite on an Apple M4 Max (macOS 26.7.1), compared by test name with `main`. **Not measured:** Windows, Linux, a macOS build or game run, macOS 14.4 itself, an Intel Mac, sleep and resume, and the Rosetta timing campaign on a quiet machine (below). ## Why A Mac cannot build LS's POSIX host while its waits are futex calls. And a renderer and an executable that each keep private wait state lose each other's wakes. ## Problem - On `main`, `win32_posix.c` includes `<linux/futex.h>` (line 12), so the whole file stops there on macOS: `host/posix/win32_posix.c:12:10: fatal error: 'linux/futex.h' file not found`. - Behind it, the wait group is Linux-only: - `futex()` (`:81`) is `syscall(SYS_futex, …)`, used by `WakeConditionVariable`, `SleepConditionVariable*`, `WaitOnAddress` and both `WakeByAddress*` (`:88-137`); - `monitor_init` (`:64`) sets the condition variable's clock with `pthread_condattr_setclock`, which Darwin lacks; - the timer wait (`:489`) is `clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, …)`, which Darwin lacks; - `SRWLOCK` is a bare `pthread_rwlock_t` whose zeroed storage counts as initialized (`windows.h:153-156`). On Darwin a zeroed rwlock is not a valid lock. - Darwin's `CLOCK_MONOTONIC` counts system sleep, unlike Linux's. A deadline on it would expire while the Mac sleeps. - The layer is built twice: into the executable and, on its own, into the renderer (`win32_posix.c`'s first comment). A futex is keyed by the address, so on Linux the two copies wake each other. macOS's public address wait (`os_sync_wait_on_address`) also keys by address, but a Win32 `WaitOnAddress` compares 1 to 8 bytes, so the Darwin side waits on a sequence word that each wake bumps. With one sequence table per image, a wake through the executable's copy never bumped the word that the renderer's copy was waiting on. - On `main`, the renderer's recorder idle (`RX_VK_RECORDER_IDLE=wait`, `gpu/vk/vk_replay.cpp:135-136`) uses `WaitOnAddress` and `WakeByAddressSingle` from the renderer's copy. ## What changed - **`host/posix/darwin_wait.c` and `.h` (new)**, all inside `#if defined(__APPLE__)`: - `darwin_now_ns`: awake uptime (`mach_absolute_time`), so sleep does not use up a deadline. FILETIME stays realtime. - `darwin_monitor_init` and `darwin_cond_timedwait`: relative pthread timed waits, recomputed from one fixed uptime deadline after every return. - `darwin_wait_until_ns`: `mach_wait_until` with the timebase rounded up and `KERN_ABORTED` retried, for the timer. - `darwin_wait_word` / `darwin_wake_word`: `os_sync_wait_on_address` with a timeout, for condition variables. - `darwin_wait_address` / `darwin_wake_address`: Win32 address waits over 1, 2, 4 or 8 bytes, upper-half and cross-word changes included. The waiter snapshots its sequence word before comparing; a wake bumps the word, then wakes all. Both Win32 wake names broadcast, as the Linux layer does. - `darwin_srw_init`, `darwin_srw_native` and `darwin_srw_trylock`: zeroed `SRWLOCK` storage gets a real `pthread_rwlock_init` the first time it is used. A try-lock never waits for another thread's initialization. - **`host/posix/darwin_wait_state.c` (new)**: `hos_posix_address_wait_sequences()` returns the process's one array of 256 zero-initialized sequence words. It belongs in the executable only. Each image's `darwin_wait.c` finds it once with `dlsym(RTLD_MAIN_ONLY, …)`. If the executable lacks it, the first address wait or wake aborts with `fatal: main executable has no Darwin address-wait state provider`; there is no private fallback, and a provider in any other image is not used. - **`host/posix/win32_posix.c`:** `#if defined(__APPLE__)` branches at the futex include, `now_ns`, `abs_after_ms`, `monitor_init`, the monitor wait, the condition-variable wake and wait, SRW release and reacquisition, `WaitOnAddress`, `WakeByAddress*`, `Sleep` and the timer wait. Each `#else` is `main`'s text unchanged. Events, semaphores and the multiple-object polling are unchanged. - **`host/posix/windows.h`:** on Apple only, `SRWLOCK` is `DarwinSRWLock` with wrappers that initialize it, and `QueryPerformanceCounter` / `GetTickCount*` read awake uptime. The pause instruction is untouched. - **`tests/test_darwin_waits.py` (new)** cuts the synchronization functions out of `win32_posix.c` and links them with `darwin_wait.c`, `darwin_wait_state.c` and the real `windows.h`. No fake locks, waits, wakes or clocks. - The default run checks contention without lost wakes (32- and 64-bit, upper-half only), wake counts, fixed deadlines under spurious wakes, zero, finite and infinite waits, both SRW modes, events, semaphores, the absolute timer, thread completion and the polling limits. Its timing campaign has fixed tolerances: a 10 ms wait may end 0.1 ms early or 50 ms late. - **`tests/darwin_wait_sharing_test.c` (new)**, with `--shared-only`: the executable and two separately linked dylibs, each with its own copy of the wait functions, and the real provider in the executable. Every waiting image against every waking image, 8 layouts from 1 to 8 bytes, both wake names, a waiter already queued and a waiter just entering: 288 cases of 4 rounds. Also four-waiter broadcasts across images, polls, and nine runs of an executable without the provider (a fourth dylib exports one), which must abort. - It builds the running Python's architecture by default; repeated `--arch` selects both routes and `--source-tree` another tree. It skips on Windows and prints `SKIP` on Linux. - **`tests/test_display_pointer.py`:** on macOS only, it compiles `darwin_wait.c` and links it into its fixture, whose SRW locks now need it. The Linux and Windows command and every assertion are unchanged. - **`docs/development.md`:** the macOS wait subset, its clock, the shared address-wait state, the test command, and the build's source lists. - **Unchanged:** `nx2/build.py`. The macOS build PR in this series (`build(macos): classify Darwin helpers …`) puts `darwin_wait.c` into both images and `darwin_wait_state.c` into the executable only. The executable has to export `hos_posix_address_wait_sequences`; the macOS build flags PR (`build(macos): discover dual-architecture toolchains …`) exports `_hos_posix_*` with `-exported_symbol`, as Linux already does with `--export-dynamic-symbol=hos_posix_*`. ## Flags and settings No game, renderer or build switch. The new test's own options: | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `--arch` (repeatable) | the fixture's targets | the running Python's architecture | both, `--arch arm64 --arch x86_64`, where Rosetta is installed | macOS | Windows, Linux | an Intel Mac | | `--shared-only` | only the two-image checks and the C++ header checks, without the timing campaign | off | on for a busy machine or for the cross-image question | timing-free runs | timing | | | `--source-tree` | the tree whose wait functions are built | this checkout | a tree of `main`, for the failing side | comparisons | | | | `--out` | keeps the fixtures and logs in a new folder | a temporary folder | set it for evidence | | | | ## Measurements main: `5e072fc8c2adadba5ea610148be1c47800ba0cca` (2026-11-14) this PR: `mac/waits` @ `0d62666d45bba74fe344eac19e73b179481e85c1` Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1 (25G241); Apple clang 21.0.0; Python 3.14.10. No GPU used. Scene: none. Game-free. Method: `tests/test_darwin_waits.py --shared-only --arch arm64 --arch x86_64` on `main`'s tree (`--source-tree`, a clean checkout of `5e072fc8`) and on this PR, interleaved: main, PR, main, PR. For the cross-image defect, the waits commits of this PR without the shared state (`35ef0284`) ran a separate two-dylib probe that links the same wait functions twice. Then `tests/run_all.py --jobs 6` on this PR, compared by test name with a `main` run on the same Mac. How to reproduce: `python -I -B tests/test_darwin_waits.py --shared-only --arch arm64 --arch x86_64`, then the same with `--source-tree <a checkout of main>`. | | main @ 5e072fc8 | this PR @ 35ef0284, without the shared state | this PR @ 0d62666d | |---|---|---|---| | `tests/test_darwin_waits.py --shared-only`, both routes, run 1 | exit 1: no `host/posix/darwin_wait.c` | (the mode is added with the shared state) | exit 0 | | the same, run 2 | exit 1: the same | | exit 0 | | a wake in another image, per architecture | no Darwin waits at all | **20 of 20 waits stuck** (10 A→B, 10 B→A); 20 of 20 same-image controls woke | 768 of 768 cross-image waits woke, 0 early | | same-image waits, per architecture | | 20 of 20 | 384 of 384 | | four-waiter broadcasts across images, per architecture | | | 16 of 16 groups, 64 waiters | | executable without the provider, per architecture | | | 9 of 9 abort, as they must | | full game-free suite | 525 passed, 19 failed, 17 skipped | not run | 526 passed, 19 failed, 17 skipped | ``` this PR @ 35ef0284, without the shared state (arm64; x86_64 printed the same): PAIR A_to_A rounds=10 kernel_wait_observed=10 requested_image_woke=10 still_waiting_after_100ms=0 same_image_rescued=0 early_unjudged=0 PAIR B_to_B rounds=10 kernel_wait_observed=10 requested_image_woke=10 still_waiting_after_100ms=0 same_image_rescued=0 early_unjudged=0 PAIR A_to_B rounds=10 kernel_wait_observed=10 requested_image_woke=0 still_waiting_after_100ms=10 same_image_rescued=10 early_unjudged=0 PAIR B_to_A rounds=10 kernel_wait_observed=10 requested_image_woke=0 still_waiting_after_100ms=10 same_image_rescued=10 early_unjudged=0 RESULT controls_stuck=0 cross_image_stuck=20 expected_cross_image_stuck=0 FAIL cross_image_wait_wake_binding this PR @ 0d62666d (each run, per architecture; a few of the 288 PAIR lines): IMAGES distinct_dylibs=2 distinct_wait_entries=3 distinct_wake_entries=3 dladdr_identity=1 image_local_errors=3 main_win32_exports=0 PAIR wait=A wake=B layout=u64-upper-half size=8 wrapper=single phase=queued rounds=4 observed=4 early_unjudged=0 completed=4 PAIR wait=B wake=host layout=u32-cross-word size=4 wrapper=single phase=queued rounds=4 observed=4 early_unjudged=0 completed=4 BROADCAST layout=u8-aligned size=1 wrapper=single kernel_wait_observed=4 completed=4 STATE canonical_wake_increments=1174 decoy_wake_increments=0 stable_provider=1 fatal: main executable has no Darwin address-wait state provider EXPECTED missing-provider SIGABRT SHARING_CHECKS arch=arm64 missing_provider_probes=9 expected_exit=86 PASS test_darwin_waits architectures=arm64,x86_64 mode=sharing-only main @ 5e072fc8 (each run): FileNotFoundError: [Errno 2] No such file or directory: '…/host/posix/darwin_wait.c' ``` The stuck waiters at `35ef0284` were each woken through their own image afterwards (`same_image_rescued=10`), which shows they were waiting, not lost. `decoy_wake_increments=0`: a dylib that also exports a provider is never used. ## Testing **Tested:** - The two-image test, both ways, both routes, twice each, and the cross-image probe on `35ef0284`, as above. - The whole of `win32_posix.c` now gets past its includes on macOS (syntax only). It still fails, with 75 errors on each route, in code that other macOS PRs port: the fiber `ucontext.h` routines, CPU affinity, thread names, `MAP_STACK`, another thread's CPU clock, `dlinfo` and the fault context. - **The whole suite**, `tests/run_all.py`, 6 at once, on the same Mac (the runner labels macOS `linux`): ``` main @ 5e072fc8: 525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once) this PR @ 0d62666d: 526 passed, 19 failed, 17 skipped in 180 s (linux, 6 at once) ``` - By test name, the only difference is the new `tests/test_darwin_waits.py` (pass, native arm64 with its timing campaign, as `run_all` runs it). `tests/test_display_pointer.py` passes on both. No other check changed status. - The same 19 checks fail on both. They are macOS portability failures of `main`, which other macOS pull requests address: `test_appimage_package_retry`, `test_collect_wide`, `test_control`, `test_framegen_resources`, `test_game_layout`, `test_low_memory_notice`, `test_mem_gate`, `test_mem_gate_children`, `test_mirror_trim`, `test_module_ids`, `test_oot_state_tap`, `test_pad_layout`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_prof_symbols`, `test_program_cache_cleanup`, `test_program_cache_one_variant`. - `test_game_layout` gets further here: `main` stops at `linux/futex.h`, this PR at the fiber `ucontext.h` include. **Not tested:** - Windows and Linux. The non-Apple text of `win32_posix.c` and `windows.h` is `main`'s; that is not a build there. - A macOS build or game run, and so the executable's export of the provider in a real link. - **The timing campaign under Rosetta on a quiet machine.** Before this rebase, at `1da4f9b3`, on a heavily loaded Mac, one 10 ms absolute-timer wait under Rosetta took 68.4 ms, over the 60 ms ceiling; native arm64 passed. The ceiling stays; the quiet rerun has not been done. The suite runs the timing campaign natively only. - macOS 14.4 itself (the deployment target is 14.4; it ran only on 26.7.1), an Intel Mac, sleep and resume, a clock adjustment, and every possible thread interleaving: the wake checks are repeated observations. **Needs other testers:** - Windows (llvm-mingw): build the host and run the game-free suite; expect no change, and the new test skipped (request to open). - Linux: build the host and run the game-free suite, especially `test_display_pointer`; expect no change, and the new test printing `SKIP` (request to open). - macOS 14.4 on Apple Silicon: `python -I -B tests/test_darwin_waits.py --arch arm64 --arch x86_64` on a quiet machine; expect `PASS`, with no late sample (request to open). ## Screenshots Nothing on screen. ## Risk and rollback - **Windows and Linux:** no change in behaviour. Every new line is inside `#if defined(__APPLE__)` or in a new file. - **macOS:** an executable built without `darwin_wait_state.c`, or without exporting its function, aborts at its first address wait. That is on purpose: a private fallback would bring back the lost wakes. - **Wakes:** both wake names wake every waiter on the word, and two addresses can share a sequence word. Callers must recheck their condition after a wake, as the Linux layer already requires. - **Win32 differences kept from the Linux layer:** relative-only waitable timers that ignore the wait's millisecond limit, periodic and APC timers; a polling wait-all that can take one object before all are ready. - **Rollback:** revert the four commits. There is no switch. ## Reviewer notes - Review `5e072fc8..0d62666d`: four commits and no dependency. `2e939743` is the waits; `35ef0284` links `darwin_wait.c` into the pointer test's fixture; `3861478f` moves the address-wait state into the executable; `0d62666d` documents that. - Start with `darwin_wait.c`'s address waits and `init_address_sequences`, then the Apple branches in `win32_posix.c`'s monitor and condition-variable waits. - Written from Apple's SDK headers and pages (`os/os_sync_wait_on_address.h`, `pthread/pthread.h`, `clock_gettime(3)`, `mach/mach_time.h`, `dlsym(3)`). Nothing was copied from another project. ## Issues See #420. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. Based on 5e072fc8. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). On macOS: 526 passed, 17 skipped, and the same 19 failures as `main`, named under Testing. - [x] Tested in the game, or what wasn't is named under Testing. No game run; named under Testing. - [ ] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test requests are still to open. - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. For a `fix`: the new test fails on `main` and passes here, in two interleaved runs each. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). Test options only, documented in `docs/development.md`. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. `test_display_pointer.py` links the new helper on macOS; its assertions are unchanged. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Not built there; requests to open. - [x] Docs updated. 
Add macOS 14.4 public address waits, awake-uptime deadlines, relative
monitor waits, absolute timer sleeps, and real initialization for zeroed
SRW locks. Keep the Linux source behind each Apple branch byte-identical.

Exercise the extracted layer with real primitives and C/C++ header clients
on arm64 and x86_64. Document the existing broadcast address wakes, timer
restrictions and non-atomic polling, without claiming a complete host port.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The pointer mailbox test uses the real POSIX SRW wrappers, which now call
into darwin_wait.c. Compile that production C dependency and link it into
the existing native-default fixture on macOS only. Keep the Linux and
Windows compiler invocation and every existing assertion unchanged.

The full-suite failure at 1da4f9b3 is retained. This repair does not resolve
the held cross-image address-wait contract decision or qualify the port.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Move the address sequence words into an executable-only provider and resolve
that canonical service once in each shim image. Fail closed when the host
provider is absent; retain full-width comparisons and broadcast ordering.

Extend the existing wait driver with real host/two-dylib behavior coverage,
including upper-half and cross-word changes, same-image controls, image-local
error state, broadcast wakes, and rejection of a dylib-only provider. Keep
the existing timing campaign intact and independently selectable.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs(runtime): describe the executable-owned macOS address-wait state
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
PR description / Description follows the template (pull_request_target) Successful in 7s
Details
CI / Conventional Commit title (pull_request) Successful in 6s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 7s
Details
Third-party parts / As their notes say (pull_request) Successful in 8s
Details
Website / Website on Cloudflare (pull_request_target) Has been skipped
Details
0d62666d45 
The wait section still said each image's darwin_wait.c is self-contained.
Since the address sequence words moved into the executable, the renderer's
copy finds them through RTLD_MAIN_ONLY, a missing provider aborts, and the
build has to link darwin_wait_state.c into the host only. Say so, and name
the test's --shared-only mode.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
PR description / Description follows the template (pull_request_target) Successful in 7s
Details
CI / Conventional Commit title (pull_request) Successful in 6s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 7s
Details
Third-party parts / As their notes say (pull_request) Successful in 8s
Details
Website / Website on Cloudflare (pull_request_target) Has been skipped
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'mac/waits:mooki-mac/waits'
git switch 'mooki-mac/waits'
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area/accessibility
Barriers for people with disabilities

  
area/audio
Sound and music

  
area/input
Controllers, keyboard, mouse

  
area/install
Installing, updating, building from your game

  
area/menu
In-game menu and settings

  
area/renderer
Vulkan renderer, shaders, effects

  
area/saves
Save files

  
area/upscaling
DLSS, FSR, frame generation

  
good first issue
A good start for a new contributor

  
gpu/amd
AMD GPUs

  
gpu/igpu
Integrated GPUs

  
gpu/intel
Intel GPUs

  
gpu/nvidia
NVIDIA GPUs

  
help wanted
Extra hands or knowledge wanted

  
kind
bug
 Something doesn't work as it should

  
kind
chore
 Build, CI, dependencies, tidying

  
kind
docs
 Documentation

  
kind
feature
 A new feature or an improvement

  
kind
performance
 Frame rate, frame time, loading, memory

  
kind
question
 A question rather than a change

  
needs measurements
Waiting for before/after measurements

  
platform/linux
Linux only

  
platform/steam-deck
Steam Deck

  
platform/windows
Windows only

  
preview
Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  
priority
critical
 Crashes, data loss, or blocks most players

  
priority
high
 Important: next in line

  
priority
low
 Nice to have

  
status
blocked
 Waits on something else first

  
status
confirmed
 Reproduced or agreed on

  
status
duplicate
 Already reported or proposed

  
status
invalid
 Not a problem in this project

  
status
needs-info
 Waiting for more information from the reporter

  
status
wontfix
 Will not be worked on

  
testing wanted
Wanted: a test on other hardware (posts a test request in #test-requests)

No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!437
No description provided.