WIP: fix(macos): pin Vulkan host imports until their last owner, and unwind failed startups #491

Draft

mooki wants to merge 19 commits from mooki/Longshot:mac/renderer-correctness into main

pull from: mooki/Longshot:mac/renderer-correctness

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 0 Commits 19 Files changed 48 +5171 -67

mooki commented 2026-11-14 16:30:58 +00:00

Member

• Copy link
• Copy Markdown

What this is for: the macOS port (#420). On macOS the renderer imports guest memory into Vulkan, and the guest
pages behind an import must not be replaced or discarded while any Vulkan object still uses them. This PR has every
macOS import pin its whole backing in the executable (through two new host callbacks) until the last Vulkan owner is
destroyed, and makes imports move-only so an owner is never copied or dropped early. It also makes a renderer whose
startup fails part-way give back what it made, and drains the recorder before that rollback destroys the allocator.
Those startup and ownership changes apply on every platform; they are listed under "Changes on Windows and
Linux". Shader caches: this PR edits gpu/vk/vk_api.h. On 5e072fc8 vk_api.h is stamped, so every platform's
caches rebuild once; after #370 it is not, so they do not. What is measured: two new game-free tests compile the real import, retirement, startup and
audio code against recording stand-ins, on arm64 and on x86_64 under Rosetta. They fail on main, and on main with
the pull requests this one carries, and pass here, two interleaved runs each. Five existing checks that pass on main
and that this PR's first commit broke are shown failing before their repair and passing at the head. And the full game-free suite on an Apple
M4 Max (macOS 26.7.1), compared by test name with main. Not measured: a game run on this branch; a real Vulkan
device; Windows and Linux runs; macOS 14.4; an Intel Mac.

Depends on #438 (vk-admission) and #480 (guest-16k), which carries #478 (guest-memory) and #436 (fault-contexts). This
branch carries their commits unchanged: guest-16k's head 647e90e4, then #438 merged (91d8cc35); its own commits
start at 107fda29. It needs guest-16k for the executable's hos_posix_guest_import_pin/unpin, which the new
callbacks call, and #438 for MkwVkHost's Apple callbacks it appends to.

Why

• Imports on macOS: a Vulkan buffer imported over guest memory reads those host pages directly. On macOS the
  guest-memory code can replace a page's backing (a discard maps fresh memory there) while the renderer still has a
  buffer over it; the GPU would then read pages the game no longer sees. Nothing told the executable which ranges the
  renderer was importing, or for how long.
• Copies of an import: Pool was copied into the import table and the background-import queue
  (imported.emplace(lo, p), bg_ready.push_back({.., p})). Any owner the copy carried would be released twice, or by
  the wrong copy.
• Failed startup: replay_start returned false on a failed create and left behind what it had made (fences,
  command pools, semaphores, the parallel init pools, the VMA allocator). Several creates were not checked at all
  (vkAllocateCommandBuffers, the frame fence and semaphore).
• Audio and message boxes on macOS: waveOutOpen reached SDL through a weak hos_sdl_audio_open import, which a
  strict two-level macOS link (-undefined error) refuses. MessageBoxA looked for SDL under its Linux names.

What changed

macOS (inside #if defined(__APPLE__))

• gpu/vk/vk_api.h: MkwVkHost gains guest_import_pin(lo, hi) and guest_import_unpin(lo, hi) at its end, on
  Apple only, where MKWVK_API_VERSION becomes 23 (22 after #438). Elsewhere it stays 21 and the structure is
  unchanged. host/rx_backend.c fills them with guest-16k's hos_posix_guest_import_pin/unpin.
• gpu/vk/vk_memory.cpp: each macOS import has one DarwinImportBacking: the whole outward-rounded range, pinned in
  the executable before the import is prepared and unpinned only in its destructor, after its device memory is freed.
  A Pool's buffer is a move-only DarwinImportView that holds the backing; the buffer is always destroyed before the
  last backing owner unpins. Retirement keeps the existing completion gates (retired_value, dead_imports): a view
  leaves only when the GPU is past it. A failed or unpublished import unwinds its view and pin at once. The published
  tables live in process-owned containers that are never destroyed at exit, as the recorder's are, so no Vulkan call
  runs from static teardown. darwin_guest_imported reports which guest ranges have live owners, from the existing
  tables (no second registry). gpu/vk/darwin_import.h (new) holds the range arithmetic.
• host/posix/win32_posix.c: waveOutOpen resolves the executable's hos_posix_audio_open once through
  dlsym(RTLD_MAIN_ONLY, …) instead of a weak import, so the renderer image links strictly and never starts a second
  SDL. host/hos_sdl.c exports that service. MessageBoxA looks for libSDL3.0.dylib and libSDL2-2.0.0.dylib.

Changes on Windows and Linux

• Startup rollback (gpu/vk/vk_replay.cpp, gpu/vk/vk_vma.cpp, every platform): every fallible create in
  replay_start is checked, and a failure calls the new replay_start_rollback(), which destroys the parallel init
  pools, each recording's fence and command pools, each frame's command pool, fence and semaphore, the three timeline
  semaphores, and then the VMA allocator (vma_startup_rollback()), before returning false as before. vma_init
  publishes the allocator only once vmaCreateAllocator succeeded. A startup that succeeds runs exactly as before.
• Recorder boundary (gpu/vk/vk_recorder.h, vk_work.h, every platform): vmaDestroyAllocator is a drain-first
  boundary, next to main's vmaFreeMemory, so with RX_VK_RECORDER=defer or thread the recorder drains before the
  rollback destroys the allocator. With the recorder off (the default) the boundary is a relaxed load and a branch.
  tests/test_recorder_sync.py caught the missing boundary.
• Import ownership (gpu/vk/vk_memory.cpp, every platform): Pool values are moved, not copied, into the import
  table and the background queue; the queue slot for a retired or evicted pool (and a DeadImport) is allocated
  before the pool leaves the live table, so a failed allocation leaves its owner where it was; the three-line buffer and
  memory release is one destroy_import() helper with the same calls in the same order. Log lines read the moved pool.
• host/posix/win32_posix.c (Linux): MessageBoxA closes the extra reference its RTLD_NOLOAD dlopen takes;
  waveOutOpen checks its calloc and pthread_mutex_init and returns MMSYSERR_ERROR instead of crashing.
• Shader caches: on 5e072fc8 vk_api.h is in the stamp, so every platform retranslates once; after #370 it is not,
  and nothing rebuilds. Its preprocessed text off Apple is unchanged and the API stays 21.

How this was checked: every changed C/C++ file, preprocessed with unifdef for Linux (-U__APPLE__ -D__linux__) and
for Windows (-U__APPLE__ -D_WIN32), compared with the branch before this PR's commits. vk_api.h, hos_sdl.h and
rx_backend.c are identical; the differences in the others are exactly the ones listed (hos_sdl.c by one blank line).
That is text, not a Windows or Linux build.

Rebase onto main

• #250 (sparse mirrors): its vmaFreeMemory boundary and this PR's vmaDestroyAllocator are both kept, in
  RX_REC_BOUNDARIES and vk_work.h; its vk_memory.cpp includes and this PR's are both kept. The renderer fixture
  gains an empty sparse_wanted stand-in, which #250's retire_pool now erases from.
• #324 reworded the comment where the fixture's cut of imports_give_back ends; the cut follows the new words. Same
  code cut. Both in commit 91f9a58d.

Tests

• tests/test_darwin_renderer.py (new) with darwin_import_lifetime_test.cpp and darwin_renderer_startup_test.cpp:
  cuts the real import, view, pin, retirement and eviction code and the real replay_start, its rollback, vma_init and
  vma_startup_rollback from the source and runs them against recording stand-ins for Vulkan and the host pins. It
  checks that every pin has one unpin after the buffer's destruction, that retirement waits for completion, that each
  failure point unwinds exactly what was made, the host callback table's positions, and MessageBoxA on both
  platforms. --legacy-source runs main's import and retirement topology through the same consumer.
• tests/test_darwin_audio_service.py (new): links the real Darwin waveOut shim strictly (-undefined error) and
  checks it reaches only the executable's audio service, never a renderer-local lookalike, with every MMRESULT path.
• Updated: test_vk_darwin_admission.py (Apple API 23, the two callbacks at 368 and 376, size 384),
  test_parallel_init_order.py (the init pools are made last and destroyed by the rollback), test_sdl_audio_library.py,
  test_import_buffer_binding.py and test_texel_buffer_ownership.py (they keep testing the Windows/Linux factories on
  a Mac; the Apple owner has its own test), and test_direct_buffer_writes.py, test_parallel_rb_refresh.py and
  nx2/test_import_windows.py (they cut Pool's and import_pool's Windows/Linux layout, as before), and
  test_mirror_gpu_only.py (its cut of pool_lookup now ends at the moved imported.emplace(p.lo, std::move(p))).

Flags and settings

None new. MKWVK_API_VERSION becomes 23 on macOS only; 21 elsewhere.

Measurements

main: 5e072fc8c2adadba5ea610148be1c47800ba0cca (2026-11-14)
this PR: mac/renderer-correctness @ c1f9f884995db23c3d1620f75bab97ea67f19478
deps: b5da98ba, main plus the pull requests this one carries (the branch just before its own commits)
Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1; Apple clang 21.0.0; Vulkan-Headers 1.4.345; Python 3.14.10. No
GPU, no game.
Scene: none. Game-free.
Method: the two new tests with --arch arm64 --arch x86_64, on this PR, on main and on deps (through --source),
interleaved: PR, main, deps, PR, main, deps. Then the fixtures at each of this PR's commits (git archive), through
tests/run_all.py. Every run through tools/slot.py; x86_64 executables under Rosetta.
How to reproduce: python -I -B tests/test_darwin_renderer.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64 --arch x86_64 [--source <tree>]; python -I -B tests/test_darwin_audio_service.py --cc clang --arch arm64 --arch x86_64 [--source <tree>].

	main @ 5e072fc8	deps @ b5da98ba	this PR @ c1f9f884
tests/test_darwin_renderer.py, arm64 + x86_64, runs 1 and 2 (and 3, at the head)	exit 1: no struct DarwinImportBacking to cut	exit 1: the same	exit 0: 5 PASS lines per architecture
tests/test_darwin_audio_service.py, arm64 + x86_64, runs 1 and 2 (and 3, at the head)	exit 1: no WaveAudioOpenFn service route to cut	exit 1: the same	exit 0: PASS per architecture
tests/test_mirror_gpu_only.py	pass	not run	pass (FAIL at 91f9a58d, before its last commit: substring not found)
full game-free suite	525 passed, 19 failed, 17 skipped	not run	530 passed, 19 failed, 18 skipped

Runs 1 and 2 of this PR's side were at 91f9a58d, which differs from the head only in tests/test_mirror_gpu_only.py;
run 3 is at the head.

this PR @ c1f9f884, tests/test_darwin_renderer.py (each architecture, each run):
PASS: explicit shared-owner and destroy-before-reimport consumers; driver alias visibility remains untested
PASS: 688 import-owner checks; full-range pins, actual retirement/error paths, explicit shared-owner consumer and no new whole-pool cap; alias GPU qualification pending
PASS: published import remains process-owned; no Vulkan destruction/unpin during static teardown
PASS: 352 exact startup allocation failures, Apple23 typed host callback wiring, serial/parallel resources and rollback; no Vulkan driver
PASS: 133262 startup/ABI/resource checks; source-only fixture, no SDL window or GPU

this PR @ c1f9f884, tests/test_darwin_audio_service.py (each architecture, each run):
PASS: strict Mach-O waveOut service, executable-only resolution, callback ABI and failure result; no SDL/audio opened

This PR's own commits (git archive of each, the same 11 checks through tests/run_all.py):

Check	107fda29 (the change)	1f805dac (+ fixtures)	05213268 (+ boundary)	91f9a58d (+ fixture follows main)
test_direct_buffer_writes, test_parallel_rb_refresh	FAIL: unknown type name 'DarwinImportView'	pass	pass	pass
nx2/test_import_windows	FAIL: darwin_import_pool undeclared	pass	pass	pass
test_recorder_sync	FAIL: vk_vma.cpp calls ['vmaDestroyAllocator'], which vk_work.h does not drain before	FAIL: the same	pass	pass
test_darwin_renderer	FAIL: #324's cut anchor	FAIL	FAIL	pass
the other 6 (audio service, init order, SDL audio, import binding, texel ownership, admission)	pass	pass	pass	pass

The checks in the first three rows pass on main: this PR's first commit broke them, and its next two commits repair
them. 91f9a58d makes the new fixture follow main. The full suite at 91f9a58d then found one more check the first
commit broke, tests/test_mirror_gpu_only.py (it passes on main); the last commit, c1f9f884, repairs it. With
the merge's gpu_only mark removed from a throwaway copy, that test fails (a pool that takes others in does not keep their gpu_only and gpu_written marks), so it still checks what it did.

Testing

Tested:

• The two new tests and the per-commit checks, as above.

• Mutants (throwaway copies of this PR): vk_work.h without the vmaDestroyAllocator boundary makes
  tests/test_recorder_sync.py fail (vk_work.h's boundary macros and RX_REC_BOUNDARIES differ: … only in vk_recorder.h ['vmaDestroyAllocator']); vma_startup_rollback that leaves the allocator alive makes
  tests/test_darwin_renderer.py fail (FAIL: failed startup leaked a resource).

• The Windows and Linux text, as described under "Changes on Windows and Linux" (unifdef).

• The whole suite, tests/run_all.py --jobs 6, on the same Mac:

  main @ 5e072fc8:     525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once)
  this PR @ c1f9f884:  530 passed, 19 failed, 18 skipped in 194 s (linux, 6 at once)
  

  • By test name, this PR's own: tests/test_darwin_renderer.py (new) passes; tests/test_darwin_audio_service.py
    (new) is skipped, because without the run-all PR (#454) the runner labels macOS linux (it passes run directly,
    above).
  • From the pull requests it carries: test_darwin_fault_context, test_darwin_guest_memory,
    test_vk_darwin_admission and test_vk_darwin_sdl (new) pass; test_collect_wide goes from FAIL to pass.
  • One check that passes on main failed here: tests/test_gpu_hang_watch.py, a timing check (exit 3, 3.5 s
    against its bound of 3.4 s) on a loaded machine (load average 10 to 16). The code it cuts, gpu_hang_watch_start,
    is the same here as on main, byte for byte. Of two reruns of it alone at this head, the first failed the same way
    and the second passed.
  • The other 18 failures fail on main too: test_appimage_package_retry, test_control, test_framegen_resources,
    test_game_layout, test_low_memory_notice, test_mem_gate, test_mem_gate_children, test_mirror_trim,
    test_module_ids, test_oot_state_tap, test_pad_layout, test_pcache_header_read, test_pipeline_cache_file,
    test_pipeline_cache_save, test_pipeline_play_cache, test_prof_symbols, test_program_cache_cleanup,
    test_program_cache_one_variant.
  • The tree was clean before and after.

• History, not rerun on this branch: these commits were first made on the macOS integration branch (044635ea,
  6cf83191, 605ddf99) with the same checks; integration builds that carry them have since built and started the game
  on macOS (see #420). That speaks for those builds, not for this branch alone.

Not tested:

• Windows and Linux: no build or run here. Not tested there: a startup that fails part-way (it needs a create to
  fail), and the recorder modes (RX_VK_RECORDER=defer/thread) through such a failure.
• A real Vulkan device on this head: the tests use recording stand-ins. A game run on this branch.
• macOS 14.4 itself (the deployment target is 14.4; checks ran on 26.7.1), and an Intel Mac.

Needs other testers:

• Windows and Linux: build and start the game once; nothing changes (on 5e072fc8 the shader caches rebuild once
  first). Then a
  forced startup failure (for example a renderer started with too little device memory, or a driver that refuses a
  fence), which must log the failure and fall back as before, with no leak or crash on exit (request to open).
• Linux: a start with a renderer built before this PR next to a host built with it, which must still pair at API 21
  (request to open).

Screenshots

Nothing on screen.

Risk and rollback

• Every platform: on 5e072fc8 vk_api.h is hashed into LONGSHOT_SHADER_STAMP (nx2/build.py:1176-1182), so the
  first start after updating retranslates the shader programs; after #370 it is not hashed and nothing is retranslated.
  The rollback runs only after a failed startup, and the moves and
  queue-slot order keep the same owners and the same release calls. The new recorder boundary costs a relaxed load and
  a branch with the recorder off.
• Linux: MessageBoxA now drops its extra SDL reference; waveOutOpen fails cleanly where it would have crashed.
• macOS: a pin that the executable refuses fails the import (VK_ERROR_MEMORY_MAP_FAILED) rather than importing
  unprotected pages.
• The API number differs by platform (23 on Apple, 21 elsewhere), as #438 set up.
• Rollback: revert this PR's five commits (107fda29..c1f9f884).

Reviewer notes

• Review b5da98ba..c1f9f884 (five commits); the commits before it are the pull requests this one depends on.
  107fda29 is the change (one conflict with #250's includes, kept both); 1f805dac moves three fixtures onto the
  Windows/Linux Pool layout; 05213268 adds the vmaDestroyAllocator boundary (beside #250's vmaFreeMemory);
  91f9a58d follows #324 and #250 in the new fixture; c1f9f884 follows the moved Pool in test_mirror_gpu_only.py.
  The renderer-core PR later rewrites that same line for its imported_add.
• tests/test_darwin_audio_service.py declares a skip for Windows and Linux. Without the run-all PR (#454), the runner
  labels macOS as linux and skips it in run_all.py; run directly, or with #454, it runs and passes (above).
• Start with DarwinImportBacking and DarwinImportView in vk_memory.cpp, then replay_start_rollback in
  vk_replay.cpp.
• This does not make imports alias-safe by itself (two imports over the same pages); that is a later PR in the series.
• Against main at dc390e59 (7 commits after this branch's base), git merge-tree finds conflicts in
  gpu/vk/vk_api.h and host/rx_backend.c. The branch before this PR's commits (#438) meets the same two; a rebase
  before ready resolves them.

Issues

See #420.

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main. Based on 5e072fc8, with the pull requests it depends on merged in; main has since
  moved to dc390e59 (conflicts named under Reviewer notes). Rebased before ready.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above). On macOS:
  530 passed, 18 skipped, 18 failures that also fail on main, and test_gpu_hang_watch, a timing check that failed
  under load here and passed on a rerun (named under Testing).
• Tested in the game, or what wasn't is named under Testing. No game run on this branch; named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test
  requests are still to open.
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each,
  the median reported. For a fix: the new tests fail on main and pass here, in two interleaved runs each.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed). 23 on Apple, where
  MkwVkHost changed; 21 elsewhere, where it did not.
• A game-free test added for the fix, failing on main and passing here.
• A check under tests/ that reads changed source updated in this PR, not weakened.
• Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Not built there; requests to open.

**What this is for:** the macOS port (#420). On macOS the renderer imports guest memory into Vulkan, and the guest pages behind an import must not be replaced or discarded while any Vulkan object still uses them. This PR has every macOS import pin its whole backing in the executable (through two new host callbacks) until the last Vulkan owner is destroyed, and makes imports move-only so an owner is never copied or dropped early. It also makes a renderer whose startup fails part-way give back what it made, and drains the recorder before that rollback destroys the allocator. **Those startup and ownership changes apply on every platform; they are listed under "Changes on Windows and Linux".** **Shader caches:** this PR edits `gpu/vk/vk_api.h`. On 5e072fc8 `vk_api.h` is stamped, so every platform's caches rebuild once; after #370 it is not, so they do not. **What is measured:** two new game-free tests compile the real import, retirement, startup and audio code against recording stand-ins, on arm64 and on x86_64 under Rosetta. They fail on `main`, and on `main` with the pull requests this one carries, and pass here, two interleaved runs each. Five existing checks that pass on `main` and that this PR's first commit broke are shown failing before their repair and passing at the head. And the full game-free suite on an Apple M4 Max (macOS 26.7.1), compared by test name with `main`. **Not measured:** a game run on this branch; a real Vulkan device; Windows and Linux runs; macOS 14.4; an Intel Mac. Depends on #438 (vk-admission) and #480 (guest-16k), which carries #478 (guest-memory) and #436 (fault-contexts). This branch carries their commits unchanged: guest-16k's head `647e90e4`, then #438 merged (`91d8cc35`); its own commits start at `107fda29`. It needs guest-16k for the executable's `hos_posix_guest_import_pin`/`unpin`, which the new callbacks call, and #438 for `MkwVkHost`'s Apple callbacks it appends to. ## Why - **Imports on macOS:** a Vulkan buffer imported over guest memory reads those host pages directly. On macOS the guest-memory code can replace a page's backing (a discard maps fresh memory there) while the renderer still has a buffer over it; the GPU would then read pages the game no longer sees. Nothing told the executable which ranges the renderer was importing, or for how long. - **Copies of an import:** `Pool` was copied into the import table and the background-import queue (`imported.emplace(lo, p)`, `bg_ready.push_back({.., p})`). Any owner the copy carried would be released twice, or by the wrong copy. - **Failed startup:** `replay_start` returned `false` on a failed create and left behind what it had made (fences, command pools, semaphores, the parallel init pools, the VMA allocator). Several creates were not checked at all (`vkAllocateCommandBuffers`, the frame fence and semaphore). - **Audio and message boxes on macOS:** `waveOutOpen` reached SDL through a weak `hos_sdl_audio_open` import, which a strict two-level macOS link (`-undefined error`) refuses. `MessageBoxA` looked for SDL under its Linux names. ## What changed ### macOS (inside `#if defined(__APPLE__)`) - **`gpu/vk/vk_api.h`:** `MkwVkHost` gains `guest_import_pin(lo, hi)` and `guest_import_unpin(lo, hi)` at its end, on Apple only, where `MKWVK_API_VERSION` becomes 23 (22 after #438). Elsewhere it stays 21 and the structure is unchanged. `host/rx_backend.c` fills them with guest-16k's `hos_posix_guest_import_pin`/`unpin`. - **`gpu/vk/vk_memory.cpp`:** each macOS import has one `DarwinImportBacking`: the whole outward-rounded range, pinned in the executable before the import is prepared and unpinned only in its destructor, after its device memory is freed. A `Pool`'s buffer is a move-only `DarwinImportView` that holds the backing; the buffer is always destroyed before the last backing owner unpins. Retirement keeps the existing completion gates (`retired_value`, `dead_imports`): a view leaves only when the GPU is past it. A failed or unpublished import unwinds its view and pin at once. The published tables live in process-owned containers that are never destroyed at exit, as the recorder's are, so no Vulkan call runs from static teardown. `darwin_guest_imported` reports which guest ranges have live owners, from the existing tables (no second registry). `gpu/vk/darwin_import.h` (new) holds the range arithmetic. - **`host/posix/win32_posix.c`:** `waveOutOpen` resolves the executable's `hos_posix_audio_open` once through `dlsym(RTLD_MAIN_ONLY, …)` instead of a weak import, so the renderer image links strictly and never starts a second SDL. `host/hos_sdl.c` exports that service. `MessageBoxA` looks for `libSDL3.0.dylib` and `libSDL2-2.0.0.dylib`. ### Changes on Windows and Linux - **Startup rollback** (`gpu/vk/vk_replay.cpp`, `gpu/vk/vk_vma.cpp`, every platform): every fallible create in `replay_start` is checked, and a failure calls the new `replay_start_rollback()`, which destroys the parallel init pools, each recording's fence and command pools, each frame's command pool, fence and semaphore, the three timeline semaphores, and then the VMA allocator (`vma_startup_rollback()`), before returning `false` as before. `vma_init` publishes the allocator only once `vmaCreateAllocator` succeeded. A startup that succeeds runs exactly as before. - **Recorder boundary** (`gpu/vk/vk_recorder.h`, `vk_work.h`, every platform): `vmaDestroyAllocator` is a drain-first boundary, next to main's `vmaFreeMemory`, so with `RX_VK_RECORDER=defer` or `thread` the recorder drains before the rollback destroys the allocator. With the recorder off (the default) the boundary is a relaxed load and a branch. `tests/test_recorder_sync.py` caught the missing boundary. - **Import ownership** (`gpu/vk/vk_memory.cpp`, every platform): `Pool` values are moved, not copied, into the import table and the background queue; the queue slot for a retired or evicted pool (and a `DeadImport`) is allocated before the pool leaves the live table, so a failed allocation leaves its owner where it was; the three-line buffer and memory release is one `destroy_import()` helper with the same calls in the same order. Log lines read the moved pool. - **`host/posix/win32_posix.c`** (Linux): `MessageBoxA` closes the extra reference its `RTLD_NOLOAD` `dlopen` takes; `waveOutOpen` checks its `calloc` and `pthread_mutex_init` and returns `MMSYSERR_ERROR` instead of crashing. - **Shader caches:** on 5e072fc8 `vk_api.h` is in the stamp, so every platform retranslates once; after #370 it is not, and nothing rebuilds. Its preprocessed text off Apple is unchanged and the API stays 21. How this was checked: every changed C/C++ file, preprocessed with `unifdef` for Linux (`-U__APPLE__ -D__linux__`) and for Windows (`-U__APPLE__ -D_WIN32`), compared with the branch before this PR's commits. `vk_api.h`, `hos_sdl.h` and `rx_backend.c` are identical; the differences in the others are exactly the ones listed (`hos_sdl.c` by one blank line). That is text, not a Windows or Linux build. ### Rebase onto `main` - **#250** (sparse mirrors): its `vmaFreeMemory` boundary and this PR's `vmaDestroyAllocator` are both kept, in `RX_REC_BOUNDARIES` and `vk_work.h`; its `vk_memory.cpp` includes and this PR's are both kept. The renderer fixture gains an empty `sparse_wanted` stand-in, which #250's `retire_pool` now erases from. - **#324** reworded the comment where the fixture's cut of `imports_give_back` ends; the cut follows the new words. Same code cut. Both in commit `91f9a58d`. ### Tests - **`tests/test_darwin_renderer.py` (new)** with `darwin_import_lifetime_test.cpp` and `darwin_renderer_startup_test.cpp`: cuts the real import, view, pin, retirement and eviction code and the real `replay_start`, its rollback, `vma_init` and `vma_startup_rollback` from the source and runs them against recording stand-ins for Vulkan and the host pins. It checks that every pin has one unpin after the buffer's destruction, that retirement waits for completion, that each failure point unwinds exactly what was made, the host callback table's positions, and `MessageBoxA` on both platforms. `--legacy-source` runs `main`'s import and retirement topology through the same consumer. - **`tests/test_darwin_audio_service.py` (new):** links the real Darwin `waveOut` shim strictly (`-undefined error`) and checks it reaches only the executable's audio service, never a renderer-local lookalike, with every `MMRESULT` path. - **Updated:** `test_vk_darwin_admission.py` (Apple API 23, the two callbacks at 368 and 376, size 384), `test_parallel_init_order.py` (the init pools are made last and destroyed by the rollback), `test_sdl_audio_library.py`, `test_import_buffer_binding.py` and `test_texel_buffer_ownership.py` (they keep testing the Windows/Linux factories on a Mac; the Apple owner has its own test), and `test_direct_buffer_writes.py`, `test_parallel_rb_refresh.py` and `nx2/test_import_windows.py` (they cut `Pool`'s and `import_pool`'s Windows/Linux layout, as before), and `test_mirror_gpu_only.py` (its cut of `pool_lookup` now ends at the moved `imported.emplace(p.lo, std::move(p))`). ## Flags and settings None new. `MKWVK_API_VERSION` becomes 23 on macOS only; 21 elsewhere. ## Measurements main: `5e072fc8c2adadba5ea610148be1c47800ba0cca` (2026-11-14) this PR: `mac/renderer-correctness` @ `c1f9f884995db23c3d1620f75bab97ea67f19478` deps: `b5da98ba`, `main` plus the pull requests this one carries (the branch just before its own commits) Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1; Apple clang 21.0.0; Vulkan-Headers 1.4.345; Python 3.14.10. No GPU, no game. Scene: none. Game-free. Method: the two new tests with `--arch arm64 --arch x86_64`, on this PR, on `main` and on deps (through `--source`), interleaved: PR, main, deps, PR, main, deps. Then the fixtures at each of this PR's commits (`git archive`), through `tests/run_all.py`. Every run through `tools/slot.py`; x86_64 executables under Rosetta. How to reproduce: `python -I -B tests/test_darwin_renderer.py --cxx clang++ --headers <Vulkan-Headers/include> --arch arm64 --arch x86_64 [--source <tree>]`; `python -I -B tests/test_darwin_audio_service.py --cc clang --arch arm64 --arch x86_64 [--source <tree>]`. | | main @ 5e072fc8 | deps @ b5da98ba | this PR @ c1f9f884 | |---|---|---|---| | `tests/test_darwin_renderer.py`, arm64 + x86_64, runs 1 and 2 (and 3, at the head) | exit 1: no `struct DarwinImportBacking` to cut | exit 1: the same | exit 0: 5 PASS lines per architecture | | `tests/test_darwin_audio_service.py`, arm64 + x86_64, runs 1 and 2 (and 3, at the head) | exit 1: no `WaveAudioOpenFn` service route to cut | exit 1: the same | exit 0: PASS per architecture | | `tests/test_mirror_gpu_only.py` | pass | not run | pass (FAIL at `91f9a58d`, before its last commit: `substring not found`) | | full game-free suite | 525 passed, 19 failed, 17 skipped | not run | 530 passed, 19 failed, 18 skipped | Runs 1 and 2 of this PR's side were at `91f9a58d`, which differs from the head only in `tests/test_mirror_gpu_only.py`; run 3 is at the head. ``` this PR @ c1f9f884, tests/test_darwin_renderer.py (each architecture, each run): PASS: explicit shared-owner and destroy-before-reimport consumers; driver alias visibility remains untested PASS: 688 import-owner checks; full-range pins, actual retirement/error paths, explicit shared-owner consumer and no new whole-pool cap; alias GPU qualification pending PASS: published import remains process-owned; no Vulkan destruction/unpin during static teardown PASS: 352 exact startup allocation failures, Apple23 typed host callback wiring, serial/parallel resources and rollback; no Vulkan driver PASS: 133262 startup/ABI/resource checks; source-only fixture, no SDL window or GPU this PR @ c1f9f884, tests/test_darwin_audio_service.py (each architecture, each run): PASS: strict Mach-O waveOut service, executable-only resolution, callback ABI and failure result; no SDL/audio opened ``` **This PR's own commits** (`git archive` of each, the same 11 checks through `tests/run_all.py`): | Check | `107fda29` (the change) | `1f805dac` (+ fixtures) | `05213268` (+ boundary) | `91f9a58d` (+ fixture follows main) | |---|---|---|---|---| | `test_direct_buffer_writes`, `test_parallel_rb_refresh` | FAIL: `unknown type name 'DarwinImportView'` | pass | pass | pass | | `nx2/test_import_windows` | FAIL: `darwin_import_pool` undeclared | pass | pass | pass | | `test_recorder_sync` | FAIL: `vk_vma.cpp calls ['vmaDestroyAllocator'], which vk_work.h does not drain before` | FAIL: the same | pass | pass | | `test_darwin_renderer` | FAIL: #324's cut anchor | FAIL | FAIL | pass | | the other 6 (audio service, init order, SDL audio, import binding, texel ownership, admission) | pass | pass | pass | pass | The checks in the first three rows pass on `main`: this PR's first commit broke them, and its next two commits repair them. `91f9a58d` makes the new fixture follow `main`. The full suite at `91f9a58d` then found one more check the first commit broke, `tests/test_mirror_gpu_only.py` (it passes on `main`); the last commit, `c1f9f884`, repairs it. With the merge's `gpu_only` mark removed from a throwaway copy, that test fails (`a pool that takes others in does not keep their gpu_only and gpu_written marks`), so it still checks what it did. ## Testing **Tested:** - The two new tests and the per-commit checks, as above. - **Mutants** (throwaway copies of this PR): `vk_work.h` without the `vmaDestroyAllocator` boundary makes `tests/test_recorder_sync.py` fail (`vk_work.h's boundary macros and RX_REC_BOUNDARIES differ: … only in vk_recorder.h ['vmaDestroyAllocator']`); `vma_startup_rollback` that leaves the allocator alive makes `tests/test_darwin_renderer.py` fail (`FAIL: failed startup leaked a resource`). - **The Windows and Linux text**, as described under "Changes on Windows and Linux" (`unifdef`). - **The whole suite**, `tests/run_all.py --jobs 6`, on the same Mac: ``` main @ 5e072fc8: 525 passed, 19 failed, 17 skipped in 177 s (linux, 6 at once) this PR @ c1f9f884: 530 passed, 19 failed, 18 skipped in 194 s (linux, 6 at once) ``` - By test name, this PR's own: `tests/test_darwin_renderer.py` (new) passes; `tests/test_darwin_audio_service.py` (new) is skipped, because without the run-all PR (#454) the runner labels macOS `linux` (it passes run directly, above). - From the pull requests it carries: `test_darwin_fault_context`, `test_darwin_guest_memory`, `test_vk_darwin_admission` and `test_vk_darwin_sdl` (new) pass; `test_collect_wide` goes from FAIL to pass. - One check that passes on `main` failed here: `tests/test_gpu_hang_watch.py`, a timing check (`exit 3, 3.5 s` against its bound of 3.4 s) on a loaded machine (load average 10 to 16). The code it cuts, `gpu_hang_watch_start`, is the same here as on `main`, byte for byte. Of two reruns of it alone at this head, the first failed the same way and the second passed. - The other 18 failures fail on `main` too: `test_appimage_package_retry`, `test_control`, `test_framegen_resources`, `test_game_layout`, `test_low_memory_notice`, `test_mem_gate`, `test_mem_gate_children`, `test_mirror_trim`, `test_module_ids`, `test_oot_state_tap`, `test_pad_layout`, `test_pcache_header_read`, `test_pipeline_cache_file`, `test_pipeline_cache_save`, `test_pipeline_play_cache`, `test_prof_symbols`, `test_program_cache_cleanup`, `test_program_cache_one_variant`. - The tree was clean before and after. - **History, not rerun on this branch:** these commits were first made on the macOS integration branch (`044635ea`, `6cf83191`, `605ddf99`) with the same checks; integration builds that carry them have since built and started the game on macOS (see #420). That speaks for those builds, not for this branch alone. **Not tested:** - **Windows and Linux:** no build or run here. Not tested there: a startup that fails part-way (it needs a create to fail), and the recorder modes (`RX_VK_RECORDER=defer`/`thread`) through such a failure. - A real Vulkan device on this head: the tests use recording stand-ins. A game run on this branch. - macOS 14.4 itself (the deployment target is 14.4; checks ran on 26.7.1), and an Intel Mac. **Needs other testers:** - **Windows and Linux:** build and start the game once; nothing changes (on 5e072fc8 the shader caches rebuild once first). Then a forced startup failure (for example a renderer started with too little device memory, or a driver that refuses a fence), which must log the failure and fall back as before, with no leak or crash on exit (request to open). - **Linux:** a start with a renderer built before this PR next to a host built with it, which must still pair at API 21 (request to open). ## Screenshots Nothing on screen. ## Risk and rollback - **Every platform:** on 5e072fc8 `vk_api.h` is hashed into `LONGSHOT_SHADER_STAMP` (`nx2/build.py:1176-1182`), so the first start after updating retranslates the shader programs; after #370 it is not hashed and nothing is retranslated. The rollback runs only after a failed startup, and the moves and queue-slot order keep the same owners and the same release calls. The new recorder boundary costs a relaxed load and a branch with the recorder off. - **Linux:** `MessageBoxA` now drops its extra SDL reference; `waveOutOpen` fails cleanly where it would have crashed. - **macOS:** a pin that the executable refuses fails the import (`VK_ERROR_MEMORY_MAP_FAILED`) rather than importing unprotected pages. - **The API number differs by platform** (23 on Apple, 21 elsewhere), as #438 set up. - **Rollback:** revert this PR's five commits (`107fda29..c1f9f884`). ## Reviewer notes - Review `b5da98ba..c1f9f884` (five commits); the commits before it are the pull requests this one depends on. `107fda29` is the change (one conflict with #250's includes, kept both); `1f805dac` moves three fixtures onto the Windows/Linux `Pool` layout; `05213268` adds the `vmaDestroyAllocator` boundary (beside #250's `vmaFreeMemory`); `91f9a58d` follows #324 and #250 in the new fixture; `c1f9f884` follows the moved `Pool` in `test_mirror_gpu_only.py`. The renderer-core PR later rewrites that same line for its `imported_add`. - `tests/test_darwin_audio_service.py` declares a skip for Windows and Linux. Without the run-all PR (#454), the runner labels macOS as `linux` and skips it in `run_all.py`; run directly, or with #454, it runs and passes (above). - Start with `DarwinImportBacking` and `DarwinImportView` in `vk_memory.cpp`, then `replay_start_rollback` in `vk_replay.cpp`. - This does not make imports alias-safe by itself (two imports over the same pages); that is a later PR in the series. - Against `main` at `dc390e59` (7 commits after this branch's base), `git merge-tree` finds conflicts in `gpu/vk/vk_api.h` and `host/rx_backend.c`. The branch before this PR's commits (#438) meets the same two; a rebase before ready resolves them. ## Issues See #420. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. Based on 5e072fc8, with the pull requests it depends on merged in; `main` has since moved to dc390e59 (conflicts named under Reviewer notes). Rebased before ready. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). On macOS: 530 passed, 18 skipped, 18 failures that also fail on `main`, and `test_gpu_hang_watch`, a timing check that failed under load here and passed on a rerun (named under Testing). - [x] Tested in the game, or what wasn't is named under Testing. No game run on this branch; named under Testing. - [ ] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). The test requests are still to open. - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. For a `fix`: the new tests fail on `main` and pass here, in two interleaved runs each. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). 23 on Apple, where `MkwVkHost` changed; 21 elsewhere, where it did not. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. Not built there; requests to open.

mooki added 19 commits 2026-11-14 16:30:58 +00:00

fix(runtime): preserve guest pages when discarding on macOS ... fe35e00ac6

Reserve the fixed guest window with a non-overwriting Mach mapping. Reclaim
complete owned host pages with fresh anonymous mappings and zero only the
partial-page edges, preserving adjacent 4 KiB guest pages on 16 KiB hosts.
Use the actual host granule for non-guest allocation/protection and report
macOS discard failures instead of returning success.

Add game-free source-linked arm64/Rosetta tests covering collisions, repeated
free/recommit, unaligned ranges, failure paths and resident-memory reduction.
Keep Linux and Windows paths unchanged. Build-list integration is separate.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(runtime): retain guest-page rounding in Darwin memory wrappers ... a350a531fa

Keep the POSIX layer's outward 4 KiB MEM_DECOMMIT and inward 4 KiB MEM_RESET
contracts above the byte-exact Darwin helper. Check arithmetic before rounding,
and keep separate real-memory assertions for helper and wrapper ranges.

Default the Darwin regression test to the native architecture. Both CPU routes
remain explicitly selectable for acceptance without requiring Rosetta or arm64
execution from every ordinary macOS suite run.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(runtime): translate macOS fault contexts on both CPU routes ... da567be4b6

Decode Darwin signal contexts and access faults for arm64 and x86_64,
copy supported context edits back on signal return, and prevent recursive
callback dispatch when a fault handler itself faults. Keep the non-Apple
bridge text and the x64-shaped CONTEXT layout unchanged.

Add source-linked, game-free owned-memory fault tests for both targets,
including page repair, PC redirection, prior handlers and an inaccessible
handler stack. Document the native PC/SP contract and remaining limits.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(runtime): keep Darwin fault checks native by default ... 7731dc0c18

Do not make the ordinary Apple Silicon suite depend on Rosetta. Default
to the running Python process's architecture and retain explicit repeated
--arch options for dual-route qualification. Document that command and
count only files when reporting crash-report directory changes.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(runtime): cut main's int hos_commit in the Darwin guest-memory test ... 9c6791454f

Main's #324 (a4c77174) made hos_commit in host/mkw_glue.c return whether
the range was committed, so its definition now starts "int hos_commit(".
The Darwin guest-memory test cut the function by its old "void" head and
stopped with a ValueError on main's source, on this branch and on main alike,
before compiling anything. Cut it by the new head.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

feat(vk): add macOS Vulkan loader and device admission ... e294efe9db

Select a local Vulkan loader or MoltenVK with RX_VK_LIBRARY and keep SDL and
renderer startup on that same library. Capture SDL's instance extensions on
the window thread, enable advertised portability extensions, and name missing
admission requirements and VkResults without changing Windows or Linux paths.

Keep the non-Apple host API at 21 for existing prebuilt renderers; append the
two macOS callbacks only in Apple's version 22. Cover pure decisions, legacy
platform expectations, ABI layout, and the SDL2/SDL3 bridge without a GPU.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): use semantic admission contracts ... 81053e98f6

Remove the SDL source-string wiring and source-index ordering assertions while
keeping the sixteen compiled behavioral cases and surface adapter compile.
Replace the comment-sensitive API header hash with fixed baseline field names,
byte offsets, versions and sizes, retaining typed Apple callback checks and
both platforms' actual admission decisions and negative controls.

Production code is unchanged. Prior full-suite and device receipts remain
bound to 818e33b1; only the affected game-free tests were rerun.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): follow main's LoadLibraryA and float-add choice in the admission fixture ... 91d8cc3527

The fixture cuts LoadLibraryA and create_device out of main's source. Since
#34, LoadLibraryA keeps dlerror()'s text in load_error, and since #191
native_fadd_wanted also takes the device ID and driver version, so the cut
code no longer compiled against the fixture's stand-ins on this base. Give
it load_error and a dlerror stand-in, and the four-argument stand-in. No
expectation changed.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/fault-contexts (#436) into mac/guest-16k ... aeb30801c2

This PR needs the macOS fault-context PR (#436); its commits are kept as they are.
The one conflict, host/posix/win32_posix.c's Apple include block, keeps both
darwin_mem.h (the guest-memory PR's) and darwin_fault.h (#436's).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): serialize guest backing and native-page watch lifetimes ... faea2408b2

Keep address-region and page frees owned until discard or in-place zeroing
finishes. Route guest mutations from each image to one executable-owned
registry, and pin complete imported backing through duplicate and overlapping
GPU memory lifetimes without querying the renderer under ownership locks.

Use real Mach query ranges and protections, preserve byte-exact 4 KiB guest
ownership and 64 KiB commit semantics, and handle all native watch subpages
with checked fault recovery and conservative dirty coverage. Replace racy
bulk dirty-byte reads with same-granule atomics.

Add game-free native arm64/Rosetta fixtures for two-image dispatch, actual
faults, aliases, failures, and deterministic free/reallocate/import races.
Carry prior RSS evidence rather than repeat unchanged experiments.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): retain pending watch faults across protection changes ... 37901de8c6

Keep a cancelled-writable watch state so an already pending diagnostic fault
is acknowledged without rearming or changing permissions. Check the real
mapping before acknowledgement; intentional read-only/no-access stays closed.

Freeze affected watch pages once, preflight the complete native range against
Mach maximum permissions, and issue one protection operation. Preserve prior
watch states when either preflight or the native operation fails.

Exercise the real Win32 vectored dispatcher, signal-handler retention and the
next genuine watch fault. Cover permission failures both before preflight and
after a deterministic barrier, with real Mach maximum-protection changes and
no manufactured VM results, on native arm64 and Rosetta.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): make the guest-memory changes Apple-only; Windows and Linux keep main's code ... 3366b61b32

The two commits before this one also changed Windows and Linux: the
dirty-page scan read its marks with byte atomics, the renderer's zeroed
guest memory was marked written, an empty range consumed no mark,
allocation failures in the scan and in the address-region bookkeeping
were fatal, hos_region_alloc refused a bad size or alignment,
hos_mark_written clamped its length, and VirtualQuery checked its
pointer and set last_error. Nobody has timed the scan there, so each of
these is now inside #if defined(__APPLE__), with main's code in the
#else. unifdef -U__APPLE__ of every host/ file this branch changes
equals main's byte for byte, and so do their preprocessed Linux and
Windows token streams. Issue #432 is the version for every platform.

test_collect_wide's fixture compiles the scan the platform builds: byte
atomics, and an empty range consuming nothing, on macOS; main's word
loads elsewhere.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs(macos): describe the guest window's one owner, import pins and native-page watches ... 647e90e450

The guest-memory section still described the groundwork alone: it said
the write watch and VirtualQuery were separate work and that the
region allocator's unlock-before-discard race remained. Describe the
executable's single owner and how each image reaches it, frees that
keep their range until the bytes are gone, import pins, the real
VirtualQuery and VirtualProtect, the native-page watch and the atomic
dirty scan, that all of it is macOS only (issue #432), and the test's
two narrower modes.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

Merge mac/vk-admission (#438) into mac/renderer-correctness ... b5da98ba30

Carries the vk-admission pull request's commits, which this one needs
(MkwVkHost's Apple callbacks, darwin_vulkan.h), on top of guest-16k.
host/posix/win32_posix.c: its Apple include block keeps darwin_mem.h,
darwin_fault.h and darwin_vulkan.h.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(macos): pin renderer imports through final resource ownership ... 107fda2904

Append Apple23 full-backing import lifetime callbacks while preserving the
non-Apple21 layout. Keep buffer views move-only, release buffers before the
last memory owner unpins backing, and preserve completion-gated retirement.
Unpublished failures unwind; published roots retain process lifetime rather
than calling Vulkan from unsafe static teardown.

Check and unwind post-device startup allocations. Resolve the shared audio
shim through the executable-owned SDL service on Darwin without unresolved
weak imports or linker relaxation. Retain the existing loader/SDL behavior.

Add game-free owner, ABI, source-topology and strict-link consumers, including
both native arm64 and x86_64 routes. This slice does not claim to fix separate
host-import alias visibility or implement the final alias admission planner.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): cut the non-Apple Pool and import route in three more fixtures ... 1f805dac09

044635ea gave vk_memory.cpp's Pool an Apple-only DarwinImportView member and
import_pool an Apple route (darwin_import_pool) right after its guards. It
moved test_import_buffer_binding and test_texel_buffer_ownership onto the
Windows/Linux layout (#if defined(__APPLE__) -> #if 0: the Apple owner has
its own darwin_import_lifetime_test), but three more fixtures cut the same
code and stopped compiling on macOS. Each passes at 58c5c5aa (044635ea's
parent) and fails at 044635ea:

- test_direct_buffer_writes and test_parallel_rb_refresh: their cut of
  struct Pool named DarwinImportView. They now cut Pool's Windows/Linux
  layout, the same one they compiled before.
- nx2/test_import_windows: its cut of import_pool's guards, which ends at
  the import, took in the unterminated Apple #if. It now ends at the Apple
  route, which must stay one #if just before the import; the fixture's own
  continuation models the Windows/Linux import as before.

No assertion, case or check is removed or loosened. test_parallel_rb_refresh
and test_import_windows also need the renderer-core extractions (next).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(vk): drain the recorder before startup rollback destroys the allocator ... 05213268df

044635ea added vma_startup_rollback (vk_vma.cpp, run after a failed
replay_start), which calls vmaDestroyAllocator. vk_vma.cpp includes
vk_work.h last so that RX_VK_RECORDER=defer/thread drains before its
destructions, and every such boundary call in a vk_work.h file must be in
RX_REC_BOUNDARIES with a drain-first macro. This one was not, and
test_recorder_sync caught it ("vk_vma.cpp calls ['vmaDestroyAllocator'],
which vk_work.h does not drain before"): it passes at 58c5c5aa (044635ea's
parent) and fails at 044635ea.

Add it to the list and to vk_work.h, next to vmaDestroyBuffer and
vmaDestroyImage. With the recorder off (the default) boundary() is a
relaxed load and a branch. Rollback runs before any image or buffer exists,
so nothing should be held there today, but the allocator is no longer
destroyed while a recorder could still hold packets. The test is unchanged
(sources: 42 boundaries drained, was 41).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(macos): follow main's #324 and #250 in the Darwin renderer fixture ... 91f9a58de6

tests/test_darwin_renderer.py cuts vk_memory.cpp's code into
darwin_import_lifetime_test.cpp. Two of main's changes since this
branch's first base broke that cut on the rebase, before any check ran:

- #324 reworded the comment after imports_give_back's drain of the
  evicted pools ("(longshot) Off by default on Windows ..." is now
  "(longshot) Default by OS and vendor ..."). The cut ends at that
  comment, so it now looks for the new words. The cut code is the same
  drain loop as before.
- #250 (sparse mirrors) made retire_pool erase a retired pool from
  sparse_wanted. The fixture's stand-ins gain an empty sparse_wanted,
  beside mirror_wanted, as main declares it (std::vector<Pool*>).

No assertion, case or expected output changes.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(vk): follow the moved Pool in test_mirror_gpu_only's cut ... c1f9f88499

107fda29 moves the new pool into the import table
(imported.emplace(p.lo, std::move(p))) instead of copying it.
tests/test_mirror_gpu_only.py ends its cut of pool_lookup's merge at
"imported.emplace(p.lo, p)", so it stopped with "substring not found"
(full-suite receipt full-91f9a58d: the only check that passes on main
and fails here). It now ends at the moved form. The check is the same:
the merged pool must take in the gpu_only and gpu_written marks of the
pools it absorbs before it goes into the table.

On the macOS integration branch this was hidden: renderer-core's
53b8914d rewrote the same line for its imported_add. When that pull
request rebases onto this one, its line replaces this one.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

All checks were successful

Hide checks

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

PR description / Description follows the template (pull_request_target) Successful in 7s

Details

CI / Conventional Commit title (pull_request) Successful in 7s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 7s

Details

Third-party parts / As their notes say (pull_request) Successful in 9s

Details

This pull request has changes conflicting with the target branch.

• gpu/vk/vk_api.h
• host/rx_backend.c

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title WIP: fix(macos): pin Vulkan host imports until their last owner, and unwind failed startups (#491)

Merge commit body Reviewed-on: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/491

WIP: fix(macos): pin Vulkan host imports until their last owner, and unwind failed startups (#491) Reviewed-on: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/491 Copy merge commit message

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'mac/renderer-correctness:mooki-mac/renderer-correctness'

git switch 'mooki-mac/renderer-correctness'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Linux only

  

platform/steam-deck

Steam Deck

  

platform/windows

Windows only

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!491

No description provided.