fix(macos): pin Vulkan host imports until their last owner, and unwind failed startups #491

Closed

mooki wants to merge 25 commits from mooki/Longshot:mac/renderer-correctness into main

pull from: mooki/Longshot:mac/renderer-correctness

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 1 Commits 25 Files changed 49 +5183 -69

mooki commented 2026-11-14 16:30:58 +00:00

Member

• Copy link
• Copy Markdown

Type: fix · Risk: medium · Tested on: macOS 26.7.1, Apple M4 Max (game-free checks, arm64 and Rosetta x86_64; no game run on this branch) · Needs: Windows and Linux, AMD or NVIDIA GPU

Why

On macOS the renderer imports guest memory into Vulkan, and the guest pages behind an import must not be replaced or
discarded while any Vulkan object still uses them; nothing guaranteed that. On every platform a renderer whose startup
fails part-way left behind what it had made. This is part of the macOS port (#420).

• Imports on macOS: a Vulkan buffer imported over guest memory reads those host pages directly. On macOS the
  guest-memory code can replace a page's backing (a discard maps fresh memory there) while the renderer still has a
  buffer over it; the GPU would then read pages the game no longer sees. Nothing told the executable which ranges the
  renderer was importing, or for how long.
• Copies of an import: Pool was copied into the import table and the background-import queue
  (imported.emplace(lo, p), bg_ready.push_back({.., p})). Any owner the copy carried would be released twice, or by
  the wrong copy.
• Failed startup: replay_start returned false on a failed create and left behind what it had made (fences,
  command pools, semaphores, the parallel init pools, the VMA allocator). Several creates were not checked at all
  (vkAllocateCommandBuffers, the frame fence and semaphore).
• Audio and message boxes on macOS: waveOutOpen reached SDL through a weak hos_sdl_audio_open import, which a
  strict two-level macOS link (-undefined error) refuses. MessageBoxA looked for SDL under its Linux names.

How to reproduce

Seen on: macOS 26.7.1, Apple M4 Max (arm64 and x86_64 under Rosetta), in the game-free renderer and audio tests.
The startup rollback and the import copies are in code every platform runs (vk_replay.cpp, vk_vma.cpp,
vk_memory.cpp); the pins and the audio route are macOS only.
Settings and switches: defaults; RX_VK_RECORDER=defer or thread for the recorder boundary.
Steps: on a Mac, python -I -B tests/test_darwin_renderer.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64 --arch x86_64 --source <a tree of main> and python -I -B tests/test_darwin_audio_service.py --cc clang --arch arm64 --arch x86_64 --source <a tree of main>; and read replay_start in main's vk_replay.cpp.
Expected: each pin released once, after its buffer is destroyed; retirement waiting for completion; each failure
point of the startup unwinding exactly what it made; the audio shim linking strictly.
Actual: on main (and on the dependencies alone) both tests stop: no struct DarwinImportBacking and no
WaveAudioOpenFn service route to cut (before the rebase, at 5e072fc8). main's replay_start returns false on a
failed create and leaves its fences, command pools, semaphores, parallel init pools and VMA allocator behind, and does
not check vkAllocateCommandBuffers, the frame fence or its semaphore. On macOS nothing pins an import's pages.

What changed

Every macOS import pins its whole backing in the executable (through two new host callbacks) until the last Vulkan
owner is destroyed, and imports are move-only so an owner is never copied or dropped early. A renderer whose startup
fails part-way gives back what it made, and the recorder drains before that rollback destroys the allocator: those
startup and ownership changes apply on every platform (listed under "Changes on Windows and Linux").

• Read first: DarwinImportBacking and DarwinImportView in vk_memory.cpp, then replay_start_rollback in
  vk_replay.cpp.
• Depends on #438 (vk-admission) and #480 (guest-16k), which carries #478 (guest-memory) and #436 (fault-contexts).
  This branch carries their commits unchanged: guest-16k's head afa7490c, then #438's head 0f2262a3 merged
  (4975b01a); its own commits start at 7d168a5e. It needs guest-16k for the executable's
  hos_posix_guest_import_pin/unpin, which the new callbacks call, and #438 for MkwVkHost's Apple callbacks it
  appends to.
• Commits: review 4975b01a..7bf15899 (seven commits); the commits before it are the pull requests this one depends
  on. 7d168a5e is the change (one conflict with #250's includes, kept both); 1e538827 moves three fixtures onto the
  Windows/Linux Pool layout; 3bdfb929 adds the vmaDestroyAllocator boundary (beside #250's vmaFreeMemory);
  2764a8a6 follows #324 and #250 in the new fixture; f78ae3ed follows the moved Pool in
  test_mirror_gpu_only.py; 0f54963d makes the import pin callbacks macOS host API 24 (below); 7bf15899 makes
  vk_memory.cpp compile for macOS beside main's #324 (below). The renderer-core PR later rewrites
  test_mirror_gpu_only.py's line for its imported_add. The measurements name the first five by their old SHAs:
  107fda29, 1f805dac, 05213268, 91f9a58d, c1f9f884.

macOS (inside #if defined(__APPLE__))

• gpu/vk/vk_api.h: MkwVkHost gains guest_import_pin(lo, hi) and guest_import_unpin(lo, hi) at its end, on
  Apple only, where MKWVK_API_VERSION becomes 25 (24 after #438). Elsewhere it stays main's 23 and the structure
  is unchanged. host/rx_backend.c fills them with guest-16k's hos_posix_guest_import_pin/unpin.
• gpu/vk/vk_memory.cpp: each macOS import has one DarwinImportBacking: the whole outward-rounded range, pinned in
  the executable before the import is prepared and unpinned only in its destructor, after its device memory is freed.
  A Pool's buffer is a move-only DarwinImportView that holds the backing; the buffer is always destroyed before the
  last backing owner unpins. Retirement keeps the existing completion gates (retired_value, dead_imports): a view
  leaves only when the GPU is past it. A failed or unpublished import unwinds its view and pin at once. The published
  tables live in process-owned containers that are never destroyed at exit, as the recorder's are, so no Vulkan call
  runs from static teardown. darwin_guest_imported reports which guest ranges have live owners, from the existing
  tables (no second registry). gpu/vk/darwin_import.h (new) holds the range arithmetic.
• host/posix/win32_posix.c: waveOutOpen resolves the executable's hos_posix_audio_open once through
  dlsym(RTLD_MAIN_ONLY, …) instead of a weak import, so the renderer image links strictly and never starts a second
  SDL. host/hos_sdl.c exports that service. MessageBoxA looks for libSDL3.0.dylib and libSDL2-2.0.0.dylib.

Changes on Windows and Linux

• Startup rollback (gpu/vk/vk_replay.cpp, gpu/vk/vk_vma.cpp, every platform): every fallible create in
  replay_start is checked, and a failure calls the new replay_start_rollback(), which destroys the parallel init
  pools, each recording's fence and command pools, each frame's command pool, fence and semaphore, the three timeline
  semaphores, and then the VMA allocator (vma_startup_rollback()), before returning false as before. vma_init
  publishes the allocator only once vmaCreateAllocator succeeded. A startup that succeeds runs exactly as before.
• Recorder boundary (gpu/vk/vk_recorder.h, vk_work.h, every platform): vmaDestroyAllocator is a drain-first
  boundary, next to main's vmaFreeMemory, so with RX_VK_RECORDER=defer or thread the recorder drains before the
  rollback destroys the allocator. With the recorder off (the default) the boundary is a relaxed load and a branch.
  tests/test_recorder_sync.py caught the missing boundary.
• Import ownership (gpu/vk/vk_memory.cpp, every platform): Pool values are moved, not copied, into the import
  table and the background queue; the queue slot for a retired or evicted pool (and a DeadImport) is allocated
  before the pool leaves the live table, so a failed allocation leaves its owner where it was; the three-line buffer and
  memory release is one destroy_import() helper with the same calls in the same order. Log lines read the moved pool.
• host/posix/win32_posix.c (Linux): MessageBoxA closes the extra reference its RTLD_NOLOAD dlopen takes;
  waveOutOpen checks its calloc and pthread_mutex_init and returns MMSYSERR_ERROR instead of crashing.
• Shader caches: on 6c9edffe vk_api.h is not in the stamp (since #370), so nothing rebuilds. Its preprocessed
  text off Apple is unchanged and the API stays main's 22.

How this was checked: every changed C/C++ file, preprocessed with unifdef for Linux (-U__APPLE__ -D__linux__) and
for Windows (-U__APPLE__ -D_WIN32), compared with the branch before this PR's commits. vk_api.h, hos_sdl.h and
rx_backend.c are identical; the differences in the others are exactly the ones listed (hos_sdl.c by one blank line).
That is text, not a Windows or Linux build.

Rebase onto 5e072fc8 and 6c9edffe, and notes for reviewers

• #250 (sparse mirrors): its vmaFreeMemory boundary and this PR's vmaDestroyAllocator are both kept, in
  RX_REC_BOUNDARIES and vk_work.h; its vk_memory.cpp includes and this PR's are both kept. The renderer fixture
  gains an empty sparse_wanted stand-in, which #250's retire_pool now erases from.
• #324 reworded the comment where the fixture's cut of imports_give_back ends; the cut follows the new words. Same
  code cut. Both in commit 2764a8a6 (was 91f9a58d).
• tests/test_darwin_audio_service.py declares a skip for Windows and Linux. Without the run-all PR (#454), the runner
  labels macOS as linux and skips it in run_all.py; run directly, or with #454, it runs and passes (above).
• Left out: this does not make imports alias-safe by itself (two imports over the same pages); that is a later PR in
  the series.
• Rebased on 6c9edffe: the carried heads are the rebased PRs' heads: #480's afa7490c (with #478's cd383fcb and
  #436's 861d9aba) and #438's 0f2262a3. The merge 4975b01a (was b5da98ba) was made again, with the same include
  block. 7d168a5e (was 107fda29) met #438's end commit in tests/test_vk_darwin_admission.py: the offsets are
  combined (Apple callbacks at 360/368/376/384, size 392). The end commit 0f54963d makes the import pin/unpin
  callbacks macOS host API 24 in vk_api.h, since #438 is now 23 on Apple (#51 took 22 everywhere); the admission
  contract and darwin_renderer_startup_test.cpp pin 24/392, and the renderer test's PASS line says Apple24. If #68
  lands first and takes 23, #438 becomes 24 and this PR 25.
• 7bf15899, the macOS compile fix: main's #324 (a4c77174) declares extern std::deque<…> evicted_pools; above
  memory_guest_imported, and this PR defines it on Apple as a process-owned reference (auto& evicted_pools = *new …), so vk_memory.cpp did not compile for macOS (redefinition of 'evicted_pools' with a different type; the same
  two declarations are already at c1f9f884, on 5e072fc8). The game-free fixtures cut functions out of the file and
  never compile it whole. The extern is now #if !defined(__APPLE__): on macOS memory_guest_imported answers
  through darwin_guest_imported, after the definition. unifdef for Windows and for Linux gives the same text as at
  0f54963d. The same change is 444e05c1 in the renderer-core PR, which merges this one.

Tests

• tests/test_darwin_renderer.py (new) with darwin_import_lifetime_test.cpp and darwin_renderer_startup_test.cpp:
  cuts the real import, view, pin, retirement and eviction code and the real replay_start, its rollback, vma_init and
  vma_startup_rollback from the source and runs them against recording stand-ins for Vulkan and the host pins. It
  checks that every pin has one unpin after the buffer's destruction, that retirement waits for completion, that each
  failure point unwinds exactly what was made, the host callback table's positions, and MessageBoxA on both
  platforms. --legacy-source runs main's import and retirement topology through the same consumer.
• tests/test_darwin_audio_service.py (new): links the real Darwin waveOut shim strictly (-undefined error) and
  checks it reaches only the executable's audio service, never a renderer-local lookalike, with every MMRESULT path.
• Updated: test_vk_darwin_admission.py (Apple API 25, the two callbacks at 384 and 392, size 400),
  test_parallel_init_order.py (the init pools are made last and destroyed by the rollback), test_sdl_audio_library.py,
  test_import_buffer_binding.py and test_texel_buffer_ownership.py (they keep testing the Windows/Linux factories on
  a Mac; the Apple owner has its own test), and test_direct_buffer_writes.py, test_parallel_rb_refresh.py and
  nx2/test_import_windows.py (they cut Pool's and import_pool's Windows/Linux layout, as before), and
  test_mirror_gpu_only.py (its cut of pool_lookup now ends at the moved imported.emplace(p.lo, std::move(p))).

Measurements

Verdict: fixed game-free on both Mac routes, with every platform's startup rollback checked at each failure point:
at this head the two new tests pass on arm64 and x86_64 under Rosetta (PASS: 133262 startup/ABI/resource checks),
and the full suite after the rebase has no check that passes on main failing. Before the rebase both tests failed on
main and on main with the carried pull requests, and passed here, two interleaved runs each; five existing checks
that pass on main and that this PR's first commit broke are shown failing before their repair and passing at the
head. No timing: the rollback runs only after a failed startup.

What is measured: two new game-free tests compile the real import, retirement, startup and audio code against
recording stand-ins, on arm64 and on x86_64 under Rosetta; and the full game-free suite on an Apple M4 Max (macOS
26.7.1), compared by test name with main. Not measured: a game run on this branch; a real Vulkan device; Windows and
Linux runs; macOS 14.4; an Intel Mac.

Commits: main 6c9edffe5d (2026-11-14); this PR mac/renderer-correctness @
7bf1589934 (its parent 0f54963d differs only by the 2-line macOS compile guard).
Before the rebase: main 5e072fc8c2 (2026-11-14); deps b5da98ba (main plus the pull
requests this one carries, the branch just before its own commits); this PR @ c1f9f88499.
Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1; Apple clang 21.0.0; Vulkan-Headers 1.4.345; Python 3.14.10. No GPU, no game.
Scene: none. Game-free.
Method: after the rebase, the two new tests and the admission test on this head (and at 0f54963d/45bd1541), whole
files compiled for macOS, and the full suite at 45bd1541 against main @ 6c9edffe (Testing). Before the rebase: the
two new tests with --arch arm64 --arch x86_64, on this PR, on main and on deps (through --source), interleaved:
PR, main, deps, PR, main, deps. Then the fixtures at each of this PR's commits (git archive), through
tests/run_all.py. Every run through tools/slot.py; x86_64 executables under Rosetta.
How to reproduce: python -I -B tests/test_darwin_renderer.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64 --arch x86_64 [--source <tree>]; python -I -B tests/test_darwin_audio_service.py --cc clang --arch arm64 --arch x86_64 [--source <tree>].

	Before: main @ 6c9edffe	After: this PR @ 7bf15899
test_darwin_renderer.py on the Mac, arm64 and x86_64	absent on main; not re-run against main after the rebase (exit 1 at 5e072fc8, below)	pass (at 0f54963d: 10 PASS lines, ending PASS: 133262 startup/ABI/resource checks)
test_darwin_audio_service.py on the Mac, arm64 and x86_64	absent on main; not re-run against main after the rebase (exit 1 at 5e072fc8, below)	PASS
test_vk_darwin_admission.py on the Mac, arm64, then x86_64 (#438's, updated here)	absent on main	pass: Apple version 24, size 392, typed callbacks at 360/368/376/384 (non-Apple version 22, size 360, all 45 named offsets; that line at 45bd1541)
full game-free suite on macOS (tests/run_all.py --jobs 6)	531 passed, 32 failed, 17 skipped in 181 s	at 45bd1541 (0f54963d but for a PASS label): 537 passed, 31 failed, 18 skipped in 175 s; no check that passes on main fails

Before the rebase (history at the commits the columns name):

	main @ 5e072fc8	deps @ b5da98ba	this PR @ c1f9f884
tests/test_darwin_renderer.py, arm64 + x86_64, runs 1 and 2 (and 3, at the head)	exit 1: no struct DarwinImportBacking to cut	exit 1: the same	exit 0: 5 PASS lines per architecture
tests/test_darwin_audio_service.py, arm64 + x86_64, runs 1 and 2 (and 3, at the head)	exit 1: no WaveAudioOpenFn service route to cut	exit 1: the same	exit 0: PASS per architecture
tests/test_mirror_gpu_only.py	pass	not run	pass (FAIL at 91f9a58d, before its last commit: substring not found)
full game-free suite	525 passed, 19 failed, 17 skipped	not run	530 passed, 19 failed, 18 skipped

Runs 1 and 2 of this PR's side were at 91f9a58d, which differs from the head only in tests/test_mirror_gpu_only.py;
run 3 is at the head.

this PR @ c1f9f884, tests/test_darwin_renderer.py (each architecture, each run):
PASS: explicit shared-owner and destroy-before-reimport consumers; driver alias visibility remains untested
PASS: 688 import-owner checks; full-range pins, actual retirement/error paths, explicit shared-owner consumer and no new whole-pool cap; alias GPU qualification pending
PASS: published import remains process-owned; no Vulkan destruction/unpin during static teardown
PASS: 352 exact startup allocation failures, Apple23 typed host callback wiring, serial/parallel resources and rollback; no Vulkan driver
PASS: 133262 startup/ABI/resource checks; source-only fixture, no SDL window or GPU

this PR @ c1f9f884, tests/test_darwin_audio_service.py (each architecture, each run):
PASS: strict Mach-O waveOut service, executable-only resolution, callback ABI and failure result; no SDL/audio opened

This PR's own commits (git archive of each, the same 11 checks through tests/run_all.py):

Check	107fda29 (the change)	1f805dac (+ fixtures)	05213268 (+ boundary)	91f9a58d (+ fixture follows main)
test_direct_buffer_writes, test_parallel_rb_refresh	FAIL: unknown type name 'DarwinImportView'	pass	pass	pass
nx2/test_import_windows	FAIL: darwin_import_pool undeclared	pass	pass	pass
test_recorder_sync	FAIL: vk_vma.cpp calls ['vmaDestroyAllocator'], which vk_work.h does not drain before	FAIL: the same	pass	pass
test_darwin_renderer	FAIL: #324's cut anchor	FAIL	FAIL	pass
the other 6 (audio service, init order, SDL audio, import binding, texel ownership, admission)	pass	pass	pass	pass

The checks in the first three rows pass on main: this PR's first commit broke them, and its next two commits repair
them. 91f9a58d makes the new fixture follow main. The full suite at 91f9a58d then found one more check the first
commit broke, tests/test_mirror_gpu_only.py (it passes on main); the next commit, c1f9f884 (now f78ae3ed),
repairs it. With the merge's gpu_only mark removed from a throwaway copy, that test fails (a pool that takes others in does not keep their gpu_only and gpu_written marks), so it still checks what it did.

Screenshots: none; nothing on screen.

Flags and settings

No new switch. MKWVK_API_VERSION becomes 25 on macOS only; main's 23 elsewhere. If #68 lands first and takes 24,
#438's becomes 25 and this PR's 26. The other switches the diff touches come from the pull requests it carries.

Flag	What it is	Before this PR	After this PR	Effect	Helps / doesn't help / untested	Recommendation
RX_VK_RECORDER=defer / thread (existing)	The recorder modes	default off	unchanged; vmaDestroyAllocator is now a drain-first boundary	the recorder drains before a failed startup's rollback destroys the allocator; with the recorder off, a relaxed load and a branch	untested through a real failed startup	as before
RX_VK_LIBRARY (carried from #438)	macOS: the first Vulkan loader or MoltenVK dylib to try	-	as in #438	as in #438	described in #438's Flags and settings	as in #438
HOS_ZERO_ALL_PAGES, HOS_KEEP_FREED_PAGES, RX_NVN_WATCH_TEXELS_SRC, RX_NVN_WATCH_AFTER, RX_NVN_WATCH_ALL (carried from #480)	Guest-memory and write-watch switches, changed on macOS only	-	as in #480	as in #480	described in #480's Flags and settings	as in #480

Testing

Tested:

Re-tested after the rebase (head 0f54963d on 6c9edffe): python -I -B tests/test_darwin_renderer.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64 --arch x86_64: 10 PASS lines, ending PASS: 133262 startup/ABI/resource checks. python -I -B tests/test_darwin_audio_service.py --cc clang --arch arm64 --arch x86_64:
PASS. python -I -B tests/test_vk_darwin_admission.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64,
and with --arch x86_64: Apple version 24, size 392, typed callbacks at 360/368/376/384 (non-Apple version 22, size
360, all 45 named offsets). All exit 0. unifdef -U__APPLE__ of every changed C/C++ file, branch against main: the
non-Apple text differs only where it did before the rebase, with the same content. The audio and admission runs and
the unifdef comparison were at 45bd1541, which differs from 0f54963d only by the renderer test's PASS label (now
Apple24 typed host callback wiring).

Whole files on macOS, and re-tested at 7bf15899: every C/C++ file this PR changes (4 renderer, 13 host) was
compiled whole, as an object, for arm64 and x86_64, with nx2/build.py's macOS commands (#453's renderer_flags() and
host_flags(); host/posix files in both the executable's and the renderer's job; the lift's runtime headers made
without a game, as tests/test_arm64_host.py makes them). At 0f54963d, gpu/vk/vk_memory.cpp failed on both
(redefinition of 'evicted_pools' with a different type, see Reviewer notes); at 7bf15899 it compiles.
host/posix/win32_posix.c stops at main's #include <linux/futex.h>, as on main and in #480, #438, #478 and #436;
with the darwin-host PR (#490, which carries #437's guard) merged in (one include-block conflict, #490's block kept),
all 17 files compile whole on both. The other 71 renderer and host files of the macOS build (86 of the 88 include a
header this PR changes) compile, or fail with exactly main's errors (vk_draw.cpp, vk_menu.cpp, vk_progcache.cpp,
vk_programs.cpp, vk_shader.cpp, control.c, hos_crash_record.c, hos_report_run.c, and on arm64 hos_setjmp.c
and hos_thread.c, which other pull requests port). At 7bf15899 the two new tests (arm64 and x86_64), the admission
test (each) and 12 named checks through tests/run_all.py pass.

• The two new tests and the per-commit checks (Measurements).

• Mutants (throwaway copies of this PR): vk_work.h without the vmaDestroyAllocator boundary makes
  tests/test_recorder_sync.py fail (vk_work.h's boundary macros and RX_REC_BOUNDARIES differ: … only in vk_recorder.h ['vmaDestroyAllocator']); vma_startup_rollback that leaves the allocator alive makes
  tests/test_darwin_renderer.py fail (FAIL: failed startup leaked a resource).

• The Windows and Linux text, as described under "Changes on Windows and Linux" (unifdef).

• The whole suite after the rebase, tests/run_all.py --jobs 6, on the same Mac, compared by test name with a full
  main run at 6c9edffe (run at 45bd1541, which differs from 0f54963d only by the renderer test's PASS label):

  main @ 6c9edffe:     531 passed, 32 failed, 17 skipped in 181 s (linux, 6 at once)
  this PR @ 45bd1541:  537 passed, 31 failed, 18 skipped in 175 s (linux, 6 at once)
  

  • By test name, this PR's own: tests/test_darwin_renderer.py (new) passes; tests/test_darwin_audio_service.py
    (new) is skipped, because without the run-all PR (#454) the runner labels macOS linux (it passes run directly,
    above).
  • From the pull requests it carries: test_darwin_fault_context, test_darwin_guest_memory,
    test_vk_darwin_admission and test_vk_darwin_sdl (new) pass; test_collect_wide goes from FAIL to pass.
  • No check that passes on main fails here (test_gpu_hang_watch, which failed under load before the rebase,
    passes). The 31 failures all fail on main too: test_appimage_package_retry, test_appimage_player_build,
    test_control, test_cpm_discovery, test_crash_dialog, test_crash_record, test_env_on, test_game_layout,
    test_lift_cleanup_error, test_linux_preflight, test_longshot_ctl, test_low_memory_notice,
    test_make_game_cache, test_mem_gate, test_mem_gate_children, test_mirror_trim, test_oot_state_tap,
    test_pcache_header_read, test_pipeline_cache_file, test_pipeline_cache_save, test_pipeline_play_cache,
    test_pipeline_tour, test_prof_symbols, test_program_cache_cleanup, test_program_cache_one_variant,
    test_readback_late, test_report_run, test_split_giants, test_vk_pgo, test_vram_budget_row,
    test_walk_key.

• Before the rebase (the runner labels macOS linux): main @ 5e072fc8 had 525 passed, 19 failed, 17 skipped;
  this PR @ c1f9f884 had 530 passed, 19 failed, 18 skipped in 194 s.

  • By test name, this PR's own: tests/test_darwin_renderer.py (new) passed; tests/test_darwin_audio_service.py
    (new) was skipped, because without the run-all PR (#454) the runner labels macOS linux (it passes run directly,
    above).
  • From the pull requests it carries: test_darwin_fault_context, test_darwin_guest_memory,
    test_vk_darwin_admission and test_vk_darwin_sdl (new) passed; test_collect_wide went from FAIL to pass.
  • One check that passes on main failed there: tests/test_gpu_hang_watch.py, a timing check (exit 3, 3.5 s
    against its bound of 3.4 s) on a loaded machine (load average 10 to 16). The code it cuts, gpu_hang_watch_start,
    was the same there as on main, byte for byte. Of two reruns of it alone at that head, the first failed the same
    way and the second passed.
  • The other 18 failures failed on main too: test_appimage_package_retry, test_control,
    test_framegen_resources, test_game_layout, test_low_memory_notice, test_mem_gate, test_mem_gate_children,
    test_mirror_trim, test_module_ids, test_oot_state_tap, test_pad_layout, test_pcache_header_read,
    test_pipeline_cache_file, test_pipeline_cache_save, test_pipeline_play_cache, test_prof_symbols,
    test_program_cache_cleanup, test_program_cache_one_variant. On main @ 6c9edffe,
    test_framegen_resources, test_module_ids and test_pad_layout pass; its 32 failures are the other 15,
    test_collect_wide and 16 new since 5e072fc8.
  • The tree was clean before and after.

• History, not rerun on this branch: these commits were first made on the macOS integration branch (044635ea,
  6cf83191, 605ddf99) with the same checks; integration builds that carry them have since built and started the game
  on macOS (see #420). That speaks for those builds, not for this branch alone.

• After the rebase onto main aa95d35f, at 695f6859 (7bf15899 before the rebases onto e090cc48 and aa95d35f):
  Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they
  apply) gave the same results as before, but the ABI numbers main's #729 moved: the admission test's line now reads
  version 23, size 368, all 46 named offsets; Apple version 25, size 400, typed callbacks at 368/376/384/392 and the
  renderer test's Apple25 typed host callback wiring; python -I -B tests/run_all.py on wave B's merge db0e2e16,
  which carries this head: 569/32/18 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new
  failure (reports/2026-11-15-rebase-sweep-3.md).

Not tested:

• Windows and Linux: no build or run here. Not tested there: a startup that fails part-way (it needs a create to
  fail), and the recorder modes (RX_VK_RECORDER=defer/thread) through such a failure.
• A real Vulkan device on this head: the tests use recording stand-ins. A game run on this branch.
• macOS 14.4 itself (the deployment target is 14.4; checks ran on 26.7.1), and an Intel Mac.
• Windows and Linux: the import ownership changes in a game run (the moves and the queue-slot order keep the same
  owners and release calls; checked only by the fixtures and unifdef text).
• A renderer built before this PR next to a host built with it, on Linux. The non-Apple vk_api.h after unifdef -U__APPLE__ is main's byte for byte (version 23, the same MkwVkHost), so they pair as before; not run.
• tests/test_darwin_renderer.py on Windows or Linux: it has no RUN_ALL_SKIP, so tests/run_all.py runs it there
  (only its --arch is macOS-only); it ran only on the Mac.

Must be tested:

• Windows and Linux, an AMD or NVIDIA GPU, any driver: build this PR (head 695f6859) and main aa95d35f the same way,
  start each, play a minute in a busy outdoor area and quit, then run python -I -B tests/run_all.py on each, expecting
  the same start, picture and frame rate as main, no new error lines in longshot_log.txt, no crash at quit, no program
  translated again, the new test_darwin_renderer to pass (test_darwin_audio_service skips), and no check that passes
  on main to fail (by hand: test request T-K3Q5).

Risks

What could break:

• Every platform: the rollback runs only after a failed startup, and the moves and queue-slot order keep the same owners
  and the same release calls. The new recorder boundary costs a relaxed load and a branch with the recorder off. These
  touch the renderer's import path, which every platform runs; nobody has run them in a game on Windows or Linux.
• Linux: MessageBoxA now drops its extra SDL reference; waveOutOpen fails cleanly where it would have crashed.
• macOS: a pin that the executable refuses fails the import (VK_ERROR_MEMORY_MAP_FAILED) rather than importing
  unprotected pages.
• The API number differs by platform (24 on Apple, 22 elsewhere), as #438 set up. If #68 lands first and takes 23, it
  becomes 25.
• Landing order: after #438 and #480 (with #478 and #436), which this branch carries; the renderer-core PR merges this
  one.
• Needed when both this PR and #431 land: #658 (mac/renderer-core)'s 0030e14a (imported_add(Pool&&) and the
  test_import_reach / test_darwin_renderer fixtures) makes #431 and #491 build together on macOS.

Turning it off: revert this PR's seven commits (4975b01a..7bf15899).

Compatibility: no shader cache rebuilds: on 6c9edffe vk_api.h is not hashed into LONGSHOT_SHADER_STAMP (since
#370; nx2/translator_files.py names what is), and none of the changed renderer files is either, so nothing is
retranslated. The vk ABI: MKWVK_API_VERSION is 25 on macOS only (two more callbacks in MkwVkHost); on Windows and
Linux it stays main's 23 with the same structure. Saves and longshot_options.ini are unchanged. The builder compiles the
host's objects again once (main's nx2/build.py keys them on gpu/vk/vk_api.h and every host/*.h).

Issues

See #420.
See #432 (the every-platform guest-memory follow-up; this PR's carried guest-16k code stays Apple-only).

Checklist

Everyone

• One topic.
• The title is a Conventional Commit (type(scope): description) of the status line's type.
• Why gives the reason plainly and links the issues that fit (the tracker searched).
• Measurements has a verdict, or one line saying why there is nothing to measure.
• Testing names what was tested, what wasn't, and what must still be tested, where and how.
• Rebased on current main: on aa95d35f (#761, 2026-11-15), head 695f6859, on #438 681e3708, on #480
  657b7803, without a new conflict; with #438's ABI move for main's #729, the import pin callbacks are macOS host API
  version 25 (Apple table 400 bytes, callbacks at 368/376/384/392); this PR's own commits, old → new: 4975b01a →
  08304713, 7d168a5e → db82a0f2, 1e538827 → 6cee76eb, 3bdfb929 → 18c05be5, 2764a8a6 → c099f5da,
  f78ae3ed → 165bf7e6, 0f54963d → fa691b55, 7bf15899 → 695f6859; the commit names elsewhere in this
  description are from before the rebase.
• python -I -B tests/run_all.py passes (any failure that also fails on main is named). On macOS, after the
  rebase: 537 passed, 18 skipped, and 31 failures that also fail on main (531 passed, 32 failed, 17 skipped at
  6c9edffe), named under Testing. Run at 45bd1541, which differs from 0f54963d only by a test's PASS label. After
  the rebase onto main aa95d35f (head 695f6859): wave B's merge db0e2e16, 569/32/18 against main aa95d35f's
  543/33/17, no new failure.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. No
  new switch here; the carried ones are documented by #438 (RX_VK_LIBRARY in the switch table) and #480 (the macOS
  guest-memory section), all unset.
• MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed). Only the Apple
  version: 25 on Apple, where MkwVkHost changed; main's 23 elsewhere, where it did not.
• A game-free test added for the fix, failing on main and passing here.
• A check under tests/ that reads changed source updated in this PR, not weakened.
• Docs updated.

**Type:** fix · **Risk:** medium · **Tested on:** macOS 26.7.1, Apple M4 Max (game-free checks, arm64 and Rosetta x86_64; no game run on this branch) · **Needs:** Windows and Linux, AMD or NVIDIA GPU ## Why On macOS the renderer imports guest memory into Vulkan, and the guest pages behind an import must not be replaced or discarded while any Vulkan object still uses them; nothing guaranteed that. On every platform a renderer whose startup fails part-way left behind what it had made. This is part of the macOS port (#420). - **Imports on macOS:** a Vulkan buffer imported over guest memory reads those host pages directly. On macOS the guest-memory code can replace a page's backing (a discard maps fresh memory there) while the renderer still has a buffer over it; the GPU would then read pages the game no longer sees. Nothing told the executable which ranges the renderer was importing, or for how long. - **Copies of an import:** `Pool` was copied into the import table and the background-import queue (`imported.emplace(lo, p)`, `bg_ready.push_back({.., p})`). Any owner the copy carried would be released twice, or by the wrong copy. - **Failed startup:** `replay_start` returned `false` on a failed create and left behind what it had made (fences, command pools, semaphores, the parallel init pools, the VMA allocator). Several creates were not checked at all (`vkAllocateCommandBuffers`, the frame fence and semaphore). - **Audio and message boxes on macOS:** `waveOutOpen` reached SDL through a weak `hos_sdl_audio_open` import, which a strict two-level macOS link (`-undefined error`) refuses. `MessageBoxA` looked for SDL under its Linux names. ## How to reproduce **Seen on:** macOS 26.7.1, Apple M4 Max (arm64 and x86_64 under Rosetta), in the game-free renderer and audio tests. The startup rollback and the import copies are in code every platform runs (`vk_replay.cpp`, `vk_vma.cpp`, `vk_memory.cpp`); the pins and the audio route are macOS only. **Settings and switches:** defaults; `RX_VK_RECORDER=defer` or `thread` for the recorder boundary. **Steps:** on a Mac, `python -I -B tests/test_darwin_renderer.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64 --arch x86_64 --source <a tree of main>` and `python -I -B tests/test_darwin_audio_service.py --cc clang --arch arm64 --arch x86_64 --source <a tree of main>`; and read `replay_start` in `main`'s `vk_replay.cpp`. **Expected:** each pin released once, after its buffer is destroyed; retirement waiting for completion; each failure point of the startup unwinding exactly what it made; the audio shim linking strictly. **Actual:** on `main` (and on the dependencies alone) both tests stop: no `struct DarwinImportBacking` and no `WaveAudioOpenFn` service route to cut (before the rebase, at 5e072fc8). `main`'s `replay_start` returns `false` on a failed create and leaves its fences, command pools, semaphores, parallel init pools and VMA allocator behind, and does not check `vkAllocateCommandBuffers`, the frame fence or its semaphore. On macOS nothing pins an import's pages. ## What changed Every macOS import pins its whole backing in the executable (through two new host callbacks) until the last Vulkan owner is destroyed, and imports are move-only so an owner is never copied or dropped early. A renderer whose startup fails part-way gives back what it made, and the recorder drains before that rollback destroys the allocator: those startup and ownership changes apply on every platform (listed under "Changes on Windows and Linux"). - Read first: `DarwinImportBacking` and `DarwinImportView` in `vk_memory.cpp`, then `replay_start_rollback` in `vk_replay.cpp`. - Depends on #438 (vk-admission) and #480 (guest-16k), which carries #478 (guest-memory) and #436 (fault-contexts). This branch carries their commits unchanged: guest-16k's head `afa7490c`, then #438's head `0f2262a3` merged (`4975b01a`); its own commits start at `7d168a5e`. It needs guest-16k for the executable's `hos_posix_guest_import_pin`/`unpin`, which the new callbacks call, and #438 for `MkwVkHost`'s Apple callbacks it appends to. - Commits: review `4975b01a..7bf15899` (seven commits); the commits before it are the pull requests this one depends on. `7d168a5e` is the change (one conflict with #250's includes, kept both); `1e538827` moves three fixtures onto the Windows/Linux `Pool` layout; `3bdfb929` adds the `vmaDestroyAllocator` boundary (beside #250's `vmaFreeMemory`); `2764a8a6` follows #324 and #250 in the new fixture; `f78ae3ed` follows the moved `Pool` in `test_mirror_gpu_only.py`; `0f54963d` makes the import pin callbacks macOS host API 24 (below); `7bf15899` makes `vk_memory.cpp` compile for macOS beside main's #324 (below). The renderer-core PR later rewrites `test_mirror_gpu_only.py`'s line for its `imported_add`. The measurements name the first five by their old SHAs: `107fda29`, `1f805dac`, `05213268`, `91f9a58d`, `c1f9f884`. ### macOS (inside `#if defined(__APPLE__)`) - **`gpu/vk/vk_api.h`:** `MkwVkHost` gains `guest_import_pin(lo, hi)` and `guest_import_unpin(lo, hi)` at its end, on Apple only, where `MKWVK_API_VERSION` becomes 25 (24 after #438). Elsewhere it stays `main`'s 23 and the structure is unchanged. `host/rx_backend.c` fills them with guest-16k's `hos_posix_guest_import_pin`/`unpin`. - **`gpu/vk/vk_memory.cpp`:** each macOS import has one `DarwinImportBacking`: the whole outward-rounded range, pinned in the executable before the import is prepared and unpinned only in its destructor, after its device memory is freed. A `Pool`'s buffer is a move-only `DarwinImportView` that holds the backing; the buffer is always destroyed before the last backing owner unpins. Retirement keeps the existing completion gates (`retired_value`, `dead_imports`): a view leaves only when the GPU is past it. A failed or unpublished import unwinds its view and pin at once. The published tables live in process-owned containers that are never destroyed at exit, as the recorder's are, so no Vulkan call runs from static teardown. `darwin_guest_imported` reports which guest ranges have live owners, from the existing tables (no second registry). `gpu/vk/darwin_import.h` (new) holds the range arithmetic. - **`host/posix/win32_posix.c`:** `waveOutOpen` resolves the executable's `hos_posix_audio_open` once through `dlsym(RTLD_MAIN_ONLY, …)` instead of a weak import, so the renderer image links strictly and never starts a second SDL. `host/hos_sdl.c` exports that service. `MessageBoxA` looks for `libSDL3.0.dylib` and `libSDL2-2.0.0.dylib`. ### Changes on Windows and Linux - **Startup rollback** (`gpu/vk/vk_replay.cpp`, `gpu/vk/vk_vma.cpp`, every platform): every fallible create in `replay_start` is checked, and a failure calls the new `replay_start_rollback()`, which destroys the parallel init pools, each recording's fence and command pools, each frame's command pool, fence and semaphore, the three timeline semaphores, and then the VMA allocator (`vma_startup_rollback()`), before returning `false` as before. `vma_init` publishes the allocator only once `vmaCreateAllocator` succeeded. A startup that succeeds runs exactly as before. - **Recorder boundary** (`gpu/vk/vk_recorder.h`, `vk_work.h`, every platform): `vmaDestroyAllocator` is a drain-first boundary, next to main's `vmaFreeMemory`, so with `RX_VK_RECORDER=defer` or `thread` the recorder drains before the rollback destroys the allocator. With the recorder off (the default) the boundary is a relaxed load and a branch. `tests/test_recorder_sync.py` caught the missing boundary. - **Import ownership** (`gpu/vk/vk_memory.cpp`, every platform): `Pool` values are moved, not copied, into the import table and the background queue; the queue slot for a retired or evicted pool (and a `DeadImport`) is allocated before the pool leaves the live table, so a failed allocation leaves its owner where it was; the three-line buffer and memory release is one `destroy_import()` helper with the same calls in the same order. Log lines read the moved pool. - **`host/posix/win32_posix.c`** (Linux): `MessageBoxA` closes the extra reference its `RTLD_NOLOAD` `dlopen` takes; `waveOutOpen` checks its `calloc` and `pthread_mutex_init` and returns `MMSYSERR_ERROR` instead of crashing. - **Shader caches:** on `6c9edffe` `vk_api.h` is not in the stamp (since #370), so nothing rebuilds. Its preprocessed text off Apple is unchanged and the API stays `main`'s 22. How this was checked: every changed C/C++ file, preprocessed with `unifdef` for Linux (`-U__APPLE__ -D__linux__`) and for Windows (`-U__APPLE__ -D_WIN32`), compared with the branch before this PR's commits. `vk_api.h`, `hos_sdl.h` and `rx_backend.c` are identical; the differences in the others are exactly the ones listed (`hos_sdl.c` by one blank line). That is text, not a Windows or Linux build. ### Rebase onto `5e072fc8` and `6c9edffe`, and notes for reviewers - **#250** (sparse mirrors): its `vmaFreeMemory` boundary and this PR's `vmaDestroyAllocator` are both kept, in `RX_REC_BOUNDARIES` and `vk_work.h`; its `vk_memory.cpp` includes and this PR's are both kept. The renderer fixture gains an empty `sparse_wanted` stand-in, which #250's `retire_pool` now erases from. - **#324** reworded the comment where the fixture's cut of `imports_give_back` ends; the cut follows the new words. Same code cut. Both in commit `2764a8a6` (was `91f9a58d`). - `tests/test_darwin_audio_service.py` declares a skip for Windows and Linux. Without the run-all PR (#454), the runner labels macOS as `linux` and skips it in `run_all.py`; run directly, or with #454, it runs and passes (above). - Left out: this does not make imports alias-safe by itself (two imports over the same pages); that is a later PR in the series. - **Rebased on `6c9edffe`:** the carried heads are the rebased PRs' heads: #480's `afa7490c` (with #478's `cd383fcb` and #436's `861d9aba`) and #438's `0f2262a3`. The merge `4975b01a` (was `b5da98ba`) was made again, with the same include block. `7d168a5e` (was `107fda29`) met #438's end commit in `tests/test_vk_darwin_admission.py`: the offsets are combined (Apple callbacks at 360/368/376/384, size 392). The end commit `0f54963d` makes the import pin/unpin callbacks macOS host API 24 in `vk_api.h`, since #438 is now 23 on Apple (#51 took 22 everywhere); the admission contract and `darwin_renderer_startup_test.cpp` pin 24/392, and the renderer test's PASS line says `Apple24`. If #68 lands first and takes 23, #438 becomes 24 and this PR 25. - **`7bf15899`, the macOS compile fix:** main's #324 (`a4c77174`) declares `extern std::deque<…> evicted_pools;` above `memory_guest_imported`, and this PR defines it on Apple as a process-owned reference (`auto& evicted_pools = *new …`), so `vk_memory.cpp` did not compile for macOS (`redefinition of 'evicted_pools' with a different type`; the same two declarations are already at `c1f9f884`, on `5e072fc8`). The game-free fixtures cut functions out of the file and never compile it whole. The `extern` is now `#if !defined(__APPLE__)`: on macOS `memory_guest_imported` answers through `darwin_guest_imported`, after the definition. `unifdef` for Windows and for Linux gives the same text as at `0f54963d`. The same change is `444e05c1` in the renderer-core PR, which merges this one. ### Tests - **`tests/test_darwin_renderer.py` (new)** with `darwin_import_lifetime_test.cpp` and `darwin_renderer_startup_test.cpp`: cuts the real import, view, pin, retirement and eviction code and the real `replay_start`, its rollback, `vma_init` and `vma_startup_rollback` from the source and runs them against recording stand-ins for Vulkan and the host pins. It checks that every pin has one unpin after the buffer's destruction, that retirement waits for completion, that each failure point unwinds exactly what was made, the host callback table's positions, and `MessageBoxA` on both platforms. `--legacy-source` runs `main`'s import and retirement topology through the same consumer. - **`tests/test_darwin_audio_service.py` (new):** links the real Darwin `waveOut` shim strictly (`-undefined error`) and checks it reaches only the executable's audio service, never a renderer-local lookalike, with every `MMRESULT` path. - **Updated:** `test_vk_darwin_admission.py` (Apple API 25, the two callbacks at 384 and 392, size 400), `test_parallel_init_order.py` (the init pools are made last and destroyed by the rollback), `test_sdl_audio_library.py`, `test_import_buffer_binding.py` and `test_texel_buffer_ownership.py` (they keep testing the Windows/Linux factories on a Mac; the Apple owner has its own test), and `test_direct_buffer_writes.py`, `test_parallel_rb_refresh.py` and `nx2/test_import_windows.py` (they cut `Pool`'s and `import_pool`'s Windows/Linux layout, as before), and `test_mirror_gpu_only.py` (its cut of `pool_lookup` now ends at the moved `imported.emplace(p.lo, std::move(p))`). ## Measurements **Verdict:** fixed game-free on both Mac routes, with every platform's startup rollback checked at each failure point: at this head the two new tests pass on arm64 and x86_64 under Rosetta (`PASS: 133262 startup/ABI/resource checks`), and the full suite after the rebase has no check that passes on `main` failing. Before the rebase both tests failed on `main` and on `main` with the carried pull requests, and passed here, two interleaved runs each; five existing checks that pass on `main` and that this PR's first commit broke are shown failing before their repair and passing at the head. No timing: the rollback runs only after a failed startup. What is measured: two new game-free tests compile the real import, retirement, startup and audio code against recording stand-ins, on arm64 and on x86_64 under Rosetta; and the full game-free suite on an Apple M4 Max (macOS 26.7.1), compared by test name with `main`. Not measured: a game run on this branch; a real Vulkan device; Windows and Linux runs; macOS 14.4; an Intel Mac. Commits: main 6c9edffe5da1f5b5383977aeab3f544c0de25d4c (2026-11-14); this PR `mac/renderer-correctness` @ 7bf158993418cef4a2f4db61f8a32b48eb532291 (its parent `0f54963d` differs only by the 2-line macOS compile guard). Before the rebase: main 5e072fc8c2adadba5ea610148be1c47800ba0cca (2026-11-14); deps `b5da98ba` (`main` plus the pull requests this one carries, the branch just before its own commits); this PR @ c1f9f884995db23c3d1620f75bab97ea67f19478. Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1; Apple clang 21.0.0; Vulkan-Headers 1.4.345; Python 3.14.10. No GPU, no game. Scene: none. Game-free. Method: after the rebase, the two new tests and the admission test on this head (and at `0f54963d`/`45bd1541`), whole files compiled for macOS, and the full suite at `45bd1541` against `main` @ 6c9edffe (Testing). Before the rebase: the two new tests with `--arch arm64 --arch x86_64`, on this PR, on `main` and on deps (through `--source`), interleaved: PR, main, deps, PR, main, deps. Then the fixtures at each of this PR's commits (`git archive`), through `tests/run_all.py`. Every run through `tools/slot.py`; x86_64 executables under Rosetta. How to reproduce: `python -I -B tests/test_darwin_renderer.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64 --arch x86_64 [--source <tree>]`; `python -I -B tests/test_darwin_audio_service.py --cc clang --arch arm64 --arch x86_64 [--source <tree>]`. | | Before: main @ 6c9edffe | After: this PR @ 7bf15899 | |---|---|---| | `test_darwin_renderer.py` on the Mac, arm64 and x86_64 | absent on `main`; not re-run against `main` after the rebase (exit 1 at 5e072fc8, below) | pass (at `0f54963d`: 10 PASS lines, ending `PASS: 133262 startup/ABI/resource checks`) | | `test_darwin_audio_service.py` on the Mac, arm64 and x86_64 | absent on `main`; not re-run against `main` after the rebase (exit 1 at 5e072fc8, below) | PASS | | `test_vk_darwin_admission.py` on the Mac, arm64, then x86_64 (#438's, updated here) | absent on `main` | pass: `Apple version 24, size 392, typed callbacks at 360/368/376/384` (non-Apple version 22, size 360, all 45 named offsets; that line at `45bd1541`) | | full game-free suite on macOS (`tests/run_all.py --jobs 6`) | 531 passed, 32 failed, 17 skipped in 181 s | at `45bd1541` (`0f54963d` but for a PASS label): 537 passed, 31 failed, 18 skipped in 175 s; no check that passes on `main` fails | Before the rebase (history at the commits the columns name): | | main @ 5e072fc8 | deps @ b5da98ba | this PR @ c1f9f884 | |---|---|---|---| | `tests/test_darwin_renderer.py`, arm64 + x86_64, runs 1 and 2 (and 3, at the head) | exit 1: no `struct DarwinImportBacking` to cut | exit 1: the same | exit 0: 5 PASS lines per architecture | | `tests/test_darwin_audio_service.py`, arm64 + x86_64, runs 1 and 2 (and 3, at the head) | exit 1: no `WaveAudioOpenFn` service route to cut | exit 1: the same | exit 0: PASS per architecture | | `tests/test_mirror_gpu_only.py` | pass | not run | pass (FAIL at `91f9a58d`, before its last commit: `substring not found`) | | full game-free suite | 525 passed, 19 failed, 17 skipped | not run | 530 passed, 19 failed, 18 skipped | Runs 1 and 2 of this PR's side were at `91f9a58d`, which differs from the head only in `tests/test_mirror_gpu_only.py`; run 3 is at the head. ``` this PR @ c1f9f884, tests/test_darwin_renderer.py (each architecture, each run): PASS: explicit shared-owner and destroy-before-reimport consumers; driver alias visibility remains untested PASS: 688 import-owner checks; full-range pins, actual retirement/error paths, explicit shared-owner consumer and no new whole-pool cap; alias GPU qualification pending PASS: published import remains process-owned; no Vulkan destruction/unpin during static teardown PASS: 352 exact startup allocation failures, Apple23 typed host callback wiring, serial/parallel resources and rollback; no Vulkan driver PASS: 133262 startup/ABI/resource checks; source-only fixture, no SDL window or GPU this PR @ c1f9f884, tests/test_darwin_audio_service.py (each architecture, each run): PASS: strict Mach-O waveOut service, executable-only resolution, callback ABI and failure result; no SDL/audio opened ``` **This PR's own commits** (`git archive` of each, the same 11 checks through `tests/run_all.py`): | Check | `107fda29` (the change) | `1f805dac` (+ fixtures) | `05213268` (+ boundary) | `91f9a58d` (+ fixture follows main) | |---|---|---|---|---| | `test_direct_buffer_writes`, `test_parallel_rb_refresh` | FAIL: `unknown type name 'DarwinImportView'` | pass | pass | pass | | `nx2/test_import_windows` | FAIL: `darwin_import_pool` undeclared | pass | pass | pass | | `test_recorder_sync` | FAIL: `vk_vma.cpp calls ['vmaDestroyAllocator'], which vk_work.h does not drain before` | FAIL: the same | pass | pass | | `test_darwin_renderer` | FAIL: #324's cut anchor | FAIL | FAIL | pass | | the other 6 (audio service, init order, SDL audio, import binding, texel ownership, admission) | pass | pass | pass | pass | The checks in the first three rows pass on `main`: this PR's first commit broke them, and its next two commits repair them. `91f9a58d` makes the new fixture follow `main`. The full suite at `91f9a58d` then found one more check the first commit broke, `tests/test_mirror_gpu_only.py` (it passes on `main`); the next commit, `c1f9f884` (now `f78ae3ed`), repairs it. With the merge's `gpu_only` mark removed from a throwaway copy, that test fails (`a pool that takes others in does not keep their gpu_only and gpu_written marks`), so it still checks what it did. Screenshots: none; nothing on screen. ### Flags and settings No new switch. `MKWVK_API_VERSION` becomes 25 on macOS only; `main`'s 23 elsewhere. If #68 lands first and takes 24, #438's becomes 25 and this PR's 26. The other switches the diff touches come from the pull requests it carries. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `RX_VK_RECORDER=defer` / `thread` (existing) | The recorder modes | default off | unchanged; `vmaDestroyAllocator` is now a drain-first boundary | the recorder drains before a failed startup's rollback destroys the allocator; with the recorder off, a relaxed load and a branch | untested through a real failed startup | as before | | `RX_VK_LIBRARY` (carried from #438) | macOS: the first Vulkan loader or MoltenVK dylib to try | - | as in #438 | as in #438 | described in #438's Flags and settings | as in #438 | | `HOS_ZERO_ALL_PAGES`, `HOS_KEEP_FREED_PAGES`, `RX_NVN_WATCH_TEXELS_SRC`, `RX_NVN_WATCH_AFTER`, `RX_NVN_WATCH_ALL` (carried from #480) | Guest-memory and write-watch switches, changed on macOS only | - | as in #480 | as in #480 | described in #480's Flags and settings | as in #480 | ## Testing **Tested:** **Re-tested after the rebase** (head `0f54963d` on `6c9edffe`): `python -I -B tests/test_darwin_renderer.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64 --arch x86_64`: 10 PASS lines, ending `PASS: 133262 startup/ABI/resource checks`. `python -I -B tests/test_darwin_audio_service.py --cc clang --arch arm64 --arch x86_64`: PASS. `python -I -B tests/test_vk_darwin_admission.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64`, and with `--arch x86_64`: `Apple version 24, size 392, typed callbacks at 360/368/376/384` (non-Apple version 22, size 360, all 45 named offsets). All exit 0. `unifdef -U__APPLE__` of every changed C/C++ file, branch against `main`: the non-Apple text differs only where it did before the rebase, with the same content. The audio and admission runs and the `unifdef` comparison were at `45bd1541`, which differs from `0f54963d` only by the renderer test's PASS label (now `Apple24 typed host callback wiring`). **Whole files on macOS, and re-tested at `7bf15899`:** every C/C++ file this PR changes (4 renderer, 13 host) was compiled whole, as an object, for arm64 and x86_64, with `nx2/build.py`'s macOS commands (#453's `renderer_flags()` and `host_flags()`; `host/posix` files in both the executable's and the renderer's job; the lift's runtime headers made without a game, as `tests/test_arm64_host.py` makes them). At `0f54963d`, `gpu/vk/vk_memory.cpp` failed on both (`redefinition of 'evicted_pools' with a different type`, see Reviewer notes); at `7bf15899` it compiles. `host/posix/win32_posix.c` stops at `main`'s `#include <linux/futex.h>`, as on `main` and in #480, #438, #478 and #436; with the darwin-host PR (#490, which carries #437's guard) merged in (one include-block conflict, #490's block kept), all 17 files compile whole on both. The other 71 renderer and host files of the macOS build (86 of the 88 include a header this PR changes) compile, or fail with exactly `main`'s errors (`vk_draw.cpp`, `vk_menu.cpp`, `vk_progcache.cpp`, `vk_programs.cpp`, `vk_shader.cpp`, `control.c`, `hos_crash_record.c`, `hos_report_run.c`, and on arm64 `hos_setjmp.c` and `hos_thread.c`, which other pull requests port). At `7bf15899` the two new tests (arm64 and x86_64), the admission test (each) and 12 named checks through `tests/run_all.py` pass. - The two new tests and the per-commit checks (Measurements). - **Mutants** (throwaway copies of this PR): `vk_work.h` without the `vmaDestroyAllocator` boundary makes `tests/test_recorder_sync.py` fail (`vk_work.h's boundary macros and RX_REC_BOUNDARIES differ: … only in vk_recorder.h ['vmaDestroyAllocator']`); `vma_startup_rollback` that leaves the allocator alive makes `tests/test_darwin_renderer.py` fail (`FAIL: failed startup leaked a resource`). - **The Windows and Linux text**, as described under "Changes on Windows and Linux" (`unifdef`). - **The whole suite after the rebase**, `tests/run_all.py --jobs 6`, on the same Mac, compared by test name with a full `main` run at `6c9edffe` (run at `45bd1541`, which differs from `0f54963d` only by the renderer test's PASS label): ``` main @ 6c9edffe: 531 passed, 32 failed, 17 skipped in 181 s (linux, 6 at once) this PR @ 45bd1541: 537 passed, 31 failed, 18 skipped in 175 s (linux, 6 at once) ``` - By test name, this PR's own: `tests/test_darwin_renderer.py` (new) passes; `tests/test_darwin_audio_service.py` (new) is skipped, because without the run-all PR (#454) the runner labels macOS `linux` (it passes run directly, above). - From the pull requests it carries: `test_darwin_fault_context`, `test_darwin_guest_memory`, `test_vk_darwin_admission` and `test_vk_darwin_sdl` (new) pass; `test_collect_wide` goes from FAIL to pass. - No check that passes on `main` fails here (`test_gpu_hang_watch`, which failed under load before the rebase, passes). The 31 failures all fail on `main` too: `test_appimage_package_retry`, `test_appimage_player_build`, `test_control`, `test_cpm_discovery`, `test_crash_dialog`, `test_crash_record`, `test_env_on`, `test_game_layout`, `test_lift_cleanup_error`, `test_linux_preflight`, `test_longshot_ctl`, `test_low_memory_notice`, `test_make_game_cache`, `test_mem_gate`, `test_mem_gate_children`, `test_mirror_trim`, `test_oot_state_tap`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_pipeline_tour`, `test_prof_symbols`, `test_program_cache_cleanup`, `test_program_cache_one_variant`, `test_readback_late`, `test_report_run`, `test_split_giants`, `test_vk_pgo`, `test_vram_budget_row`, `test_walk_key`. - **Before the rebase** (the runner labels macOS `linux`): `main` @ `5e072fc8` had 525 passed, 19 failed, 17 skipped; this PR @ `c1f9f884` had 530 passed, 19 failed, 18 skipped in 194 s. - By test name, this PR's own: `tests/test_darwin_renderer.py` (new) passed; `tests/test_darwin_audio_service.py` (new) was skipped, because without the run-all PR (#454) the runner labels macOS `linux` (it passes run directly, above). - From the pull requests it carries: `test_darwin_fault_context`, `test_darwin_guest_memory`, `test_vk_darwin_admission` and `test_vk_darwin_sdl` (new) passed; `test_collect_wide` went from FAIL to pass. - One check that passes on `main` failed there: `tests/test_gpu_hang_watch.py`, a timing check (`exit 3, 3.5 s` against its bound of 3.4 s) on a loaded machine (load average 10 to 16). The code it cuts, `gpu_hang_watch_start`, was the same there as on `main`, byte for byte. Of two reruns of it alone at that head, the first failed the same way and the second passed. - The other 18 failures failed on `main` too: `test_appimage_package_retry`, `test_control`, `test_framegen_resources`, `test_game_layout`, `test_low_memory_notice`, `test_mem_gate`, `test_mem_gate_children`, `test_mirror_trim`, `test_module_ids`, `test_oot_state_tap`, `test_pad_layout`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_prof_symbols`, `test_program_cache_cleanup`, `test_program_cache_one_variant`. On `main` @ `6c9edffe`, `test_framegen_resources`, `test_module_ids` and `test_pad_layout` pass; its 32 failures are the other 15, `test_collect_wide` and 16 new since `5e072fc8`. - The tree was clean before and after. - **History, not rerun on this branch:** these commits were first made on the macOS integration branch (`044635ea`, `6cf83191`, `605ddf99`) with the same checks; integration builds that carry them have since built and started the game on macOS (see #420). That speaks for those builds, not for this branch alone. - **After the rebase onto main aa95d35f**, at `695f6859` (`7bf15899` before the rebases onto e090cc48 and aa95d35f): Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they apply) gave the same results as before, but the ABI numbers main's #729 moved: the admission test's line now reads `version 23, size 368, all 46 named offsets; Apple version 25, size 400, typed callbacks at 368/376/384/392` and the renderer test's `Apple25 typed host callback wiring`; `python -I -B tests/run_all.py` on wave B's merge `db0e2e16`, which carries this head: 569/32/18 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure (`reports/2026-11-15-rebase-sweep-3.md`). **Not tested:** - **Windows and Linux:** no build or run here. Not tested there: a startup that fails part-way (it needs a create to fail), and the recorder modes (`RX_VK_RECORDER=defer`/`thread`) through such a failure. - A real Vulkan device on this head: the tests use recording stand-ins. A game run on this branch. - macOS 14.4 itself (the deployment target is 14.4; checks ran on 26.7.1), and an Intel Mac. - Windows and Linux: the import ownership changes in a game run (the moves and the queue-slot order keep the same owners and release calls; checked only by the fixtures and `unifdef` text). - A renderer built before this PR next to a host built with it, on Linux. The non-Apple `vk_api.h` after `unifdef -U__APPLE__` is `main`'s byte for byte (version 23, the same `MkwVkHost`), so they pair as before; not run. - `tests/test_darwin_renderer.py` on Windows or Linux: it has no `RUN_ALL_SKIP`, so `tests/run_all.py` runs it there (only its `--arch` is macOS-only); it ran only on the Mac. **Must be tested:** - Windows and Linux, an AMD or NVIDIA GPU, any driver: build this PR (head 695f6859) and main aa95d35f the same way, start each, play a minute in a busy outdoor area and quit, then run `python -I -B tests/run_all.py` on each, expecting the same start, picture and frame rate as main, no new error lines in `longshot_log.txt`, no crash at quit, no program translated again, the new `test_darwin_renderer` to pass (`test_darwin_audio_service` skips), and no check that passes on main to fail (by hand: test request T-K3Q5). ## Risks **What could break:** - Every platform: the rollback runs only after a failed startup, and the moves and queue-slot order keep the same owners and the same release calls. The new recorder boundary costs a relaxed load and a branch with the recorder off. These touch the renderer's import path, which every platform runs; nobody has run them in a game on Windows or Linux. - Linux: `MessageBoxA` now drops its extra SDL reference; `waveOutOpen` fails cleanly where it would have crashed. - macOS: a pin that the executable refuses fails the import (`VK_ERROR_MEMORY_MAP_FAILED`) rather than importing unprotected pages. - The API number differs by platform (24 on Apple, 22 elsewhere), as #438 set up. If #68 lands first and takes 23, it becomes 25. - Landing order: after #438 and #480 (with #478 and #436), which this branch carries; the renderer-core PR merges this one. - Needed when both this PR and #431 land: #658 (mac/renderer-core)'s `0030e14a` (`imported_add(Pool&&)` and the `test_import_reach` / `test_darwin_renderer` fixtures) makes #431 and #491 build together on macOS. **Turning it off:** revert this PR's seven commits (`4975b01a..7bf15899`). **Compatibility:** no shader cache rebuilds: on `6c9edffe` `vk_api.h` is not hashed into `LONGSHOT_SHADER_STAMP` (since #370; `nx2/translator_files.py` names what is), and none of the changed renderer files is either, so nothing is retranslated. The vk ABI: `MKWVK_API_VERSION` is 25 on macOS only (two more callbacks in `MkwVkHost`); on Windows and Linux it stays main's 23 with the same structure. Saves and `longshot_options.ini` are unchanged. The builder compiles the host's objects again once (`main`'s `nx2/build.py` keys them on `gpu/vk/vk_api.h` and every `host/*.h`). ## Issues See #420. See #432 (the every-platform guest-memory follow-up; this PR's carried guest-16k code stays Apple-only). ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`: on aa95d35f (#761, 2026-11-15), head `695f6859`, on #438 `681e3708`, on #480 `657b7803`, without a new conflict; with #438's ABI move for main's #729, the import pin callbacks are macOS host API version 25 (Apple table 400 bytes, callbacks at 368/376/384/392); this PR's own commits, old → new: `4975b01a` → `08304713`, `7d168a5e` → `db82a0f2`, `1e538827` → `6cee76eb`, `3bdfb929` → `18c05be5`, `2764a8a6` → `c099f5da`, `f78ae3ed` → `165bf7e6`, `0f54963d` → `fa691b55`, `7bf15899` → `695f6859`; the commit names elsewhere in this description are from before the rebase. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). On macOS, after the rebase: 537 passed, 18 skipped, and 31 failures that also fail on `main` (531 passed, 32 failed, 17 skipped at `6c9edffe`), named under Testing. Run at `45bd1541`, which differs from `0f54963d` only by a test's PASS label. After the rebase onto main aa95d35f (head `695f6859`): wave B's merge `db0e2e16`, 569/32/18 against main aa95d35f's 543/33/17, no new failure. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. No new switch here; the carried ones are documented by #438 (`RX_VK_LIBRARY` in the switch table) and #480 (the macOS guest-memory section), all unset. - [x] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). Only the Apple version: 25 on Apple, where `MkwVkHost` changed; `main`'s 23 elsewhere, where it did not. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.

mooki added 19 commits 2026-11-14 16:30:58 +00:00

fix(runtime): preserve guest pages when discarding on macOS ... fe35e00ac6

Reserve the fixed guest window with a non-overwriting Mach mapping. Reclaim
complete owned host pages with fresh anonymous mappings and zero only the
partial-page edges, preserving adjacent 4 KiB guest pages on 16 KiB hosts.
Use the actual host granule for non-guest allocation/protection and report
macOS discard failures instead of returning success.

Add game-free source-linked arm64/Rosetta tests covering collisions, repeated
free/recommit, unaligned ranges, failure paths and resident-memory reduction.
Keep Linux and Windows paths unchanged. Build-list integration is separate.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(runtime): retain guest-page rounding in Darwin memory wrappers ... a350a531fa

Keep the POSIX layer's outward 4 KiB MEM_DECOMMIT and inward 4 KiB MEM_RESET
contracts above the byte-exact Darwin helper. Check arithmetic before rounding,
and keep separate real-memory assertions for helper and wrapper ranges.

Default the Darwin regression test to the native architecture. Both CPU routes
remain explicitly selectable for acceptance without requiring Rosetta or arm64
execution from every ordinary macOS suite run.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(runtime): translate macOS fault contexts on both CPU routes ... da567be4b6

Decode Darwin signal contexts and access faults for arm64 and x86_64,
copy supported context edits back on signal return, and prevent recursive
callback dispatch when a fault handler itself faults. Keep the non-Apple
bridge text and the x64-shaped CONTEXT layout unchanged.

Add source-linked, game-free owned-memory fault tests for both targets,
including page repair, PC redirection, prior handlers and an inaccessible
handler stack. Document the native PC/SP contract and remaining limits.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(runtime): keep Darwin fault checks native by default ... 7731dc0c18

Do not make the ordinary Apple Silicon suite depend on Rosetta. Default
to the running Python process's architecture and retain explicit repeated
--arch options for dual-route qualification. Document that command and
count only files when reporting crash-report directory changes.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(runtime): cut main's int hos_commit in the Darwin guest-memory test ... 9c6791454f

Main's #324 (a4c77174) made hos_commit in host/mkw_glue.c return whether
the range was committed, so its definition now starts "int hos_commit(".
The Darwin guest-memory test cut the function by its old "void" head and
stopped with a ValueError on main's source, on this branch and on main alike,
before compiling anything. Cut it by the new head.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

feat(vk): add macOS Vulkan loader and device admission ... e294efe9db

Select a local Vulkan loader or MoltenVK with RX_VK_LIBRARY and keep SDL and
renderer startup on that same library. Capture SDL's instance extensions on
the window thread, enable advertised portability extensions, and name missing
admission requirements and VkResults without changing Windows or Linux paths.

Keep the non-Apple host API at 21 for existing prebuilt renderers; append the
two macOS callbacks only in Apple's version 22. Cover pure decisions, legacy
platform expectations, ABI layout, and the SDL2/SDL3 bridge without a GPU.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): use semantic admission contracts ... 81053e98f6

Remove the SDL source-string wiring and source-index ordering assertions while
keeping the sixteen compiled behavioral cases and surface adapter compile.
Replace the comment-sensitive API header hash with fixed baseline field names,
byte offsets, versions and sizes, retaining typed Apple callback checks and
both platforms' actual admission decisions and negative controls.

Production code is unchanged. Prior full-suite and device receipts remain
bound to 818e33b1; only the affected game-free tests were rerun.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): follow main's LoadLibraryA and float-add choice in the admission fixture ... 91d8cc3527

The fixture cuts LoadLibraryA and create_device out of main's source. Since
#34, LoadLibraryA keeps dlerror()'s text in load_error, and since #191
native_fadd_wanted also takes the device ID and driver version, so the cut
code no longer compiled against the fixture's stand-ins on this base. Give
it load_error and a dlerror stand-in, and the four-argument stand-in. No
expectation changed.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/fault-contexts (#436) into mac/guest-16k ... aeb30801c2

This PR needs the macOS fault-context PR (#436); its commits are kept as they are.
The one conflict, host/posix/win32_posix.c's Apple include block, keeps both
darwin_mem.h (the guest-memory PR's) and darwin_fault.h (#436's).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): serialize guest backing and native-page watch lifetimes ... faea2408b2

Keep address-region and page frees owned until discard or in-place zeroing
finishes. Route guest mutations from each image to one executable-owned
registry, and pin complete imported backing through duplicate and overlapping
GPU memory lifetimes without querying the renderer under ownership locks.

Use real Mach query ranges and protections, preserve byte-exact 4 KiB guest
ownership and 64 KiB commit semantics, and handle all native watch subpages
with checked fault recovery and conservative dirty coverage. Replace racy
bulk dirty-byte reads with same-granule atomics.

Add game-free native arm64/Rosetta fixtures for two-image dispatch, actual
faults, aliases, failures, and deterministic free/reallocate/import races.
Carry prior RSS evidence rather than repeat unchanged experiments.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): retain pending watch faults across protection changes ... 37901de8c6

Keep a cancelled-writable watch state so an already pending diagnostic fault
is acknowledged without rearming or changing permissions. Check the real
mapping before acknowledgement; intentional read-only/no-access stays closed.

Freeze affected watch pages once, preflight the complete native range against
Mach maximum permissions, and issue one protection operation. Preserve prior
watch states when either preflight or the native operation fails.

Exercise the real Win32 vectored dispatcher, signal-handler retention and the
next genuine watch fault. Cover permission failures both before preflight and
after a deterministic barrier, with real Mach maximum-protection changes and
no manufactured VM results, on native arm64 and Rosetta.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): make the guest-memory changes Apple-only; Windows and Linux keep main's code ... 3366b61b32

The two commits before this one also changed Windows and Linux: the
dirty-page scan read its marks with byte atomics, the renderer's zeroed
guest memory was marked written, an empty range consumed no mark,
allocation failures in the scan and in the address-region bookkeeping
were fatal, hos_region_alloc refused a bad size or alignment,
hos_mark_written clamped its length, and VirtualQuery checked its
pointer and set last_error. Nobody has timed the scan there, so each of
these is now inside #if defined(__APPLE__), with main's code in the
#else. unifdef -U__APPLE__ of every host/ file this branch changes
equals main's byte for byte, and so do their preprocessed Linux and
Windows token streams. Issue #432 is the version for every platform.

test_collect_wide's fixture compiles the scan the platform builds: byte
atomics, and an empty range consuming nothing, on macOS; main's word
loads elsewhere.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs(macos): describe the guest window's one owner, import pins and native-page watches ... 647e90e450

The guest-memory section still described the groundwork alone: it said
the write watch and VirtualQuery were separate work and that the
region allocator's unlock-before-discard race remained. Describe the
executable's single owner and how each image reaches it, frees that
keep their range until the bytes are gone, import pins, the real
VirtualQuery and VirtualProtect, the native-page watch and the atomic
dirty scan, that all of it is macOS only (issue #432), and the test's
two narrower modes.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/vk-admission (#438) into mac/renderer-correctness ... b5da98ba30

Carries the vk-admission pull request's commits, which this one needs
(MkwVkHost's Apple callbacks, darwin_vulkan.h), on top of guest-16k.
host/posix/win32_posix.c: its Apple include block keeps darwin_mem.h,
darwin_fault.h and darwin_vulkan.h.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): pin renderer imports through final resource ownership ... 107fda2904

Append Apple23 full-backing import lifetime callbacks while preserving the
non-Apple21 layout. Keep buffer views move-only, release buffers before the
last memory owner unpins backing, and preserve completion-gated retirement.
Unpublished failures unwind; published roots retain process lifetime rather
than calling Vulkan from unsafe static teardown.

Check and unwind post-device startup allocations. Resolve the shared audio
shim through the executable-owned SDL service on Darwin without unresolved
weak imports or linker relaxation. Retain the existing loader/SDL behavior.

Add game-free owner, ABI, source-topology and strict-link consumers, including
both native arm64 and x86_64 routes. This slice does not claim to fix separate
host-import alias visibility or implement the final alias admission planner.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): cut the non-Apple Pool and import route in three more fixtures ... 1f805dac09

044635ea gave vk_memory.cpp's Pool an Apple-only DarwinImportView member and
import_pool an Apple route (darwin_import_pool) right after its guards. It
moved test_import_buffer_binding and test_texel_buffer_ownership onto the
Windows/Linux layout (#if defined(__APPLE__) -> #if 0: the Apple owner has
its own darwin_import_lifetime_test), but three more fixtures cut the same
code and stopped compiling on macOS. Each passes at 58c5c5aa (044635ea's
parent) and fails at 044635ea:

- test_direct_buffer_writes and test_parallel_rb_refresh: their cut of
  struct Pool named DarwinImportView. They now cut Pool's Windows/Linux
  layout, the same one they compiled before.
- nx2/test_import_windows: its cut of import_pool's guards, which ends at
  the import, took in the unterminated Apple #if. It now ends at the Apple
  route, which must stay one #if just before the import; the fixture's own
  continuation models the Windows/Linux import as before.

No assertion, case or check is removed or loosened. test_parallel_rb_refresh
and test_import_windows also need the renderer-core extractions (next).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(vk): drain the recorder before startup rollback destroys the allocator ... 05213268df

044635ea added vma_startup_rollback (vk_vma.cpp, run after a failed
replay_start), which calls vmaDestroyAllocator. vk_vma.cpp includes
vk_work.h last so that RX_VK_RECORDER=defer/thread drains before its
destructions, and every such boundary call in a vk_work.h file must be in
RX_REC_BOUNDARIES with a drain-first macro. This one was not, and
test_recorder_sync caught it ("vk_vma.cpp calls ['vmaDestroyAllocator'],
which vk_work.h does not drain before"): it passes at 58c5c5aa (044635ea's
parent) and fails at 044635ea.

Add it to the list and to vk_work.h, next to vmaDestroyBuffer and
vmaDestroyImage. With the recorder off (the default) boundary() is a
relaxed load and a branch. Rollback runs before any image or buffer exists,
so nothing should be held there today, but the allocator is no longer
destroyed while a recorder could still hold packets. The test is unchanged
(sources: 42 boundaries drained, was 41).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): follow main's #324 and #250 in the Darwin renderer fixture ... 91f9a58de6

tests/test_darwin_renderer.py cuts vk_memory.cpp's code into
darwin_import_lifetime_test.cpp. Two of main's changes since this
branch's first base broke that cut on the rebase, before any check ran:

- #324 reworded the comment after imports_give_back's drain of the
  evicted pools ("(longshot) Off by default on Windows ..." is now
  "(longshot) Default by OS and vendor ..."). The cut ends at that
  comment, so it now looks for the new words. The cut code is the same
  drain loop as before.
- #250 (sparse mirrors) made retire_pool erase a retired pool from
  sparse_wanted. The fixture's stand-ins gain an empty sparse_wanted,
  beside mirror_wanted, as main declares it (std::vector<Pool*>).

No assertion, case or expected output changes.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): follow the moved Pool in test_mirror_gpu_only's cut ... c1f9f88499

107fda29 moves the new pool into the import table
(imported.emplace(p.lo, std::move(p))) instead of copying it.
tests/test_mirror_gpu_only.py ends its cut of pool_lookup's merge at
"imported.emplace(p.lo, p)", so it stopped with "substring not found"
(full-suite receipt full-91f9a58d: the only check that passes on main
and fails here). It now ends at the moved form. The check is the same:
the merged pool must take in the gpu_only and gpu_written marks of the
pools it absorbs before it goes into the table.

On the macOS integration branch this was hidden: renderer-core's
53b8914d rewrote the same line for its imported_add. When that pull
request rebases onto this one, its line replaces this one.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

mooki added the

kind

bug

priority

high

labels 2026-11-14 17:12:10 +00:00

mooki added the

area/install

area/renderer

platform/macos

labels 2026-11-14 18:24:39 +00:00

mooki force-pushed mac/renderer-correctness from c1f9f88499 to 0f54963d80 2026-11-15 00:34:42 +00:00 Compare

mooki referenced this pull request 2026-11-15 00:35:10 +00:00

fix(runtime): preserve guest pages when discarding on macOS #478

mooki referenced this pull request 2026-11-15 00:35:11 +00:00

fix(runtime): translate macOS fault contexts on both CPU routes #436

mooki referenced this pull request 2026-11-15 00:35:15 +00:00

feat(vk): add macOS Vulkan loader and device admission #438

mooki referenced this pull request 2026-11-15 00:35:18 +00:00

fix(macos): serialize guest memory ownership on 16 KiB pages and pin imported backing #480

mooki referenced this pull request 2026-11-15 01:52:43 +00:00

fix(vk): scan the import table as far back as its largest live import #431

mooki referenced this pull request 2026-11-15 02:16:56 +00:00

fix(macos): on unified memory the GPU's allowance is what the Mac's RAM leaves, not Metal's working set #642

mooki referenced this pull request 2026-11-15 02:17:00 +00:00

fix(vk): give storage buffer tables descriptor sets instead of pushes on MoltenVK #643

mooki referenced this pull request 2026-11-15 02:17:04 +00:00

fix(vk): give programs that read storage buffer lengths descriptor sets on MoltenVK #644

mooki referenced this pull request 2026-11-15 02:17:10 +00:00

fix(vk): use descriptor sets for stages with more than 8 read_write textures on MoltenVK #645

mooki referenced this pull request 2026-11-15 02:17:14 +00:00

fix(vk): use update-after-bind sets past Metal's per-stage limits on MoltenVK #646

mooki referenced this pull request 2026-11-15 02:17:17 +00:00

feat(vk): add a segment planner for guest spans larger than one device allocation #647

mooki referenced this pull request 2026-11-15 02:17:23 +00:00

feat(vk): bind storage and texel descriptors as piece arrays in the SPIR-V back end #648

mooki referenced this pull request 2026-11-15 02:17:29 +00:00

feat(vk): read texel formats MoltenVK cannot view through R8/R16/R32 carrier codecs #649

mooki referenced this pull request 2026-11-15 02:17:34 +00:00

WIP: perf(vk): per-worker pipeline caches, every CPU and no saved pipeline cache for shader preparation on MoltenVK #650

mooki referenced this pull request 2026-11-15 02:17:38 +00:00

WIP: fix(macos): start renderer threads with a 16 MiB stack #651

mooki referenced this pull request 2026-11-15 02:17:42 +00:00

WIP: fix(macos): let MoltenVK's precompile go on behind the title and name slow compiles #652

mooki referenced this pull request 2026-11-15 02:17:46 +00:00

feat(macos): RX_VK_METAL_CACHE_DIR keeps Metal's shader cache in a chosen folder #653

mooki added 1 commit 2026-11-15 02:18:41 +00:00

fix(macos): vk_memory.cpp compiles on macOS: #324's evicted_pools declaration is not Apple's ... 7bf1589934

Main's #324 (a4c77174) declares `extern std::deque<…> evicted_pools;` above
memory_guest_imported, which reads it on Windows and Linux. This branch
defines evicted_pools on macOS as a process-owned reference
(`auto& evicted_pools = *new std::deque<…>;`), so after the rebase onto
6c9edffe vk_memory.cpp did not compile for macOS ("redefinition of
'evicted_pools' with a different type"). The game-free fixtures cut the
functions out and never compile the whole file, so they did not see it.

The declaration is now for Windows and Linux only: on macOS
memory_guest_imported answers through darwin_guest_imported, which comes after
the definition. Windows and Linux compile the same text as before.

First made on mac/renderer-core, where this branch is merged.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 444e05c119a261b36c7c762abf9d26194b0ee417)

mooki referenced this pull request 2026-11-15 02:33:17 +00:00

feat(vk): admit guest buffers per operation and import over-cap spans as segments on macOS #658

mooki changed title from WIP: fix(macos): pin Vulkan host imports until their last owner, and unwind failed startups to fix(macos): pin Vulkan host imports until their last owner, and unwind failed startups 2026-11-15 02:39:47 +00:00

mooki referenced this pull request 2026-11-15 02:50:16 +00:00

feat(vk): add the inert contract for segmented binding consumers #662

mooki referenced this pull request 2026-11-15 02:50:22 +00:00

feat(vk): bind segment-straddling buffers and texel views as piece arrays, and split transfers at segment cuts, on macOS #663

mooki referenced this pull request 2026-11-15 02:55:02 +00:00

WIP: fix(macos): leave MoltenVK's compute pipelines to the precompile and bound its giant compiles #664

mooki referenced this pull request 2026-11-15 02:55:05 +00:00

WIP: fix(macos): hold MoltenVK's giant compiles behind the game until its first frame, at a lower QoS class #665

mooki referenced this pull request 2026-11-15 03:11:09 +00:00

fix(vk): a timeline wait that comes back device-lost is said as a lost device, not as a timeout #670

mooki referenced this pull request 2026-11-15 04:00:40 +00:00

WIP: feat(macos): count NVN's polygon modes and restart-off strip draws on Apple, log-only #677

mooki referenced this pull request 2026-11-15 04:00:45 +00:00

WIP: fix(macos): bind a direct draw's 8-bit indices with their size on MoltenVK #678

mooki referenced this pull request 2026-11-15 04:00:51 +00:00

WIP: fix(macos): count draws rewritten where MoltenVK has no drawIndirectCount, and buffer updates copied from the stream #679

mooki referenced this pull request 2026-11-15 04:00:56 +00:00

WIP: fix(macos): direct draws that read the draw index drawn as one indirect record on MoltenVK #680

mooki referenced this pull request 2026-11-15 04:01:45 +00:00

WIP: feat(macos): RX_VK_LMEM_LOG names each module's private memory and each pipeline's first bind #681

mooki referenced this pull request 2026-11-15 04:33:59 +00:00

WIP: fix(macos): Rosetta host imports at whole hardware pages, and a refused import asked again only when its backing changes #686

mooki referenced this pull request 2026-11-15 05:23:50 +00:00

WIP: fix(macos): fans keyed apart from the other triangle topologies on MoltenVK #701

mooki referenced this pull request 2026-11-15 05:24:19 +00:00

WIP: feat(vk): log-only frame cost lines, a GPU timeline and the bench's A/B exactness arm, all off by default #704

mooki force-pushed mac/renderer-correctness from 7bf1589934 to f66d0a87b8 2026-11-15 07:17:02 +00:00 Compare

mooki referenced this pull request 2026-11-15 11:56:12 +00:00

WIP: fix(macos): Metal's shader cache folder asked at the host's start, before SDL's window #786

mooki force-pushed mac/renderer-correctness from f66d0a87b8 to 695f6859dc 2026-11-15 13:05:59 +00:00 Compare

mooki referenced this pull request from a commit 2026-11-15 17:38:53 +00:00

fix(macos): pin Vulkan host imports until their last owner, and unwind failed startups

mooki referenced this pull request 2026-11-15 17:39:39 +00:00

WIP: feat(macos): Vulkan on Metal: pinned host imports, the unified-memory budget, Metal's cache folder and KosmicKrisp #846

mooki commented 2026-11-15 18:07:38 +00:00

Author

Member

• Copy link
• Copy Markdown

Folded into #846 (feat(macos): Vulkan on Metal: pinned host imports, the unified-memory budget, Metal's cache folder and KosmicKrisp)

Folded into #846 (feat(macos): Vulkan on Metal: pinned host imports, the unified-memory budget, Metal's cache folder and KosmicKrisp)

mooki closed this pull request 2026-11-15 18:07:39 +00:00

Some checks failed

Hide checks

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 4s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s

Details

Third-party parts / As their notes say (pull_request) Successful in 11s

Details

PR description / Description follows the template (pull_request_target) Successful in 5s

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

beanstalk: approved @ 1697603

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 2c5a62e

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 4fd82b7

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 64d6d09

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 6f91fae

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 70a78f9

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 8015c9a

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 8d361dc

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 9d8356e

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ abf2227

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ d82f202

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ e5f1e40

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ ee8ab5f

Set by Beanstalk for show; the farm's database is what counts

  

ci: approved

A maintainer approved CircleCI for this fork PR (sticky; removed after 14 quiet days)

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Affects Linux (a change for every system carries both Windows and Linux)

  

platform/macos

macOS (Apple silicon or Intel Macs)

  

platform/steam-deck

Steam Deck

  

platform/windows

Affects Windows (a change for every system carries both Windows and Linux)

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

regression

Worked in an earlier build, broken in a newer one: highest priority, above other bugs

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

  

ci: approved

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

beanstalk: approved @ 1697603

beanstalk: approved @ 2c5a62e

beanstalk: approved @ 4fd82b7

beanstalk: approved @ 64d6d09

beanstalk: approved @ 6f91fae

beanstalk: approved @ 70a78f9

beanstalk: approved @ 8015c9a

beanstalk: approved @ 8d361dc

beanstalk: approved @ 9d8356e

beanstalk: approved @ abf2227

beanstalk: approved @ d82f202

beanstalk: approved @ e5f1e40

beanstalk: approved @ ee8ab5f

ci: approved

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/macos

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

regression

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

ci: approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!491

No description provided.