WIP: feat(macos): add a macOS build #966
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!966
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:mac/add-macos-build"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: macOS 26.7.1, Apple M4 Max (arm64; x86_64 under Rosetta) · Needs: Windows and Linux, any GPU (and one GPU on the mesh compute fallback)
Current revision:
594d5296merges maina8a30c22into the combined macOS head6343f29e. The PR remains WIP: final game-build acceptance, Windows/Linux qualification and guide/UI review are still open. Work is paused after this push at the owner's request.Why
Nobody can build or run LS on a Mac today. The builder takes the Linux route there, the host's POSIX layer is written for
Linux and x86, and the Vulkan renderer neither finds a driver on macOS nor works within what MoltenVK and Metal allow: shaders
Metal refuses, descriptor sets MoltenVK cannot push, a compiler service that dies under the game's giant programs, draws
MoltenVK draws differently or not at all, and a GPU budget that ran a 64 GB Mac out of memory. This PR is the macOS port: LS
builds on a Mac (natively on Apple silicon, and as x86_64 under Rosetta), packages a game folder for macOS 14.4 and later, and
plays on Metal through MoltenVK (KosmicKrisp opt-in), held in play for over 15 minutes on both routes. Two kinds of draw the Mac
work found missing on every driver are drawn here for everyone: indirect quad draws, and mesh count draws on GPUs that use LS's
mesh compute fallback. It replaces the macOS topic PRs #845-#852 (71 earlier PRs in all) with one PR; the commits stay one per
original change. See #420.
What changed
The build scripts get a Mach-O route and a packager; the host gets native Darwin memory, waits, faults, threads, control and
crash records; the renderer loads MoltenVK (or KosmicKrisp through LunarG's loader), budgets from unified memory, and works
around what MoltenVK and Metal lack, area by area below. Almost everything is behind
__APPLE__or a run-time gate on the driver(
dev.driver_id, vendor fixes,Profileflags); what changes on Windows and Linux is listed under Risks. The original6343f29eseries below contains 104 non-merge commits and 12 merges;594d5296additionally integrates main's newerrenderer, host, build and shared-tooling changes while preserving the Mac platform gates.
1. macOS host (was #845)
tests/run_all.py:macos, its skip reasons and timing).host/posix/darwin_cpu.c).host/posix/darwin_mem.c: a Mach reservation, exact zeroing, 4 KiB guest rules on 16 KiB pages).darwin_fault.c, forwin32_posix.c's fault bridge).darwin_wait.c,darwin_wait_state.c).gpu/vk/vk_loader.h,vk_main.cpp,host/posix/darwin_vulkan.c;RX_VK_LIBRARY).posix_spawn, memory counters, a Mach sampling profiler; on Linux, DiscordIPC (
host/discord_rpc.c) and the thread error paths (win32_posix.c) hardened.nx2/darwin_toolchain.py,nx2/darwin_build.py;RX_ARCH); Windows' and Linux' arrays moveinto four functions unchanged, element for element (
tests/build_flags_main.json).(
nx2/darwin_package.py,nx2/darwin_archive.py).nx2/darwin_memory.py).host/posix/darwin_control_sockets.c), and the control test's quit case.F_PREALLOCATEinpipeline_cache_file.h,F_RDADVISEinvk_progcache.cpp).darwin_guest_mem.c).vk_programs.cpp,vk_shader.cppand the debug tools build on Darwin.hos_report_run.c,hos_crash_record.c).longshot_ctlsessions (scripts/longshot_ctl.py).2. Vulkan on Metal (was #846)
gpu/vk/darwin_import.h,vk_memory.cpp); every platform: astartup that fails part-way unwinds (
replay_start_rollback(),vma_startup_rollback()),vmaDestroyAllocatoris a recorderdrain boundary, import pools are moved, not copied; on Linux
MessageBoxAandwaveOutOpenchecks.gpu/vk/unified_budget.h,system_memory.h,vram_usage();RX_VK_UNIFIED_BUDGET).gpu/vk/metal_cache.h;RX_VK_METAL_CACHE_DIR).RX_VK_DRIVER;LONGSHOT_VULKAN_LOADER,LONGSHOT_KOSMICKRISP).80a10018: test_darwin_renderer's fixture declares main'sevicted_import_bytes(#223).3. MoltenVK descriptor limits (was #847)
On MoltenVK a program a pushed set breaks gets an ordinary update-after-bind set (
layout_pushes()invk_draw.cpp); everyother driver pushes as before, and the SPIR-V and the caches are unchanged.
set_for_tables,RX_VK_SET_FOR_TABLES).gpu/vk/spirv_lengths.h;set_for_lengths,RX_VK_SET_FOR_LENGTHS).b56a6cee: main's Linux pressure fixture includesspirv_lengths.h(#223).4. The SPIR-V back end and guest spans on MoltenVK (was #848)
Each part is chosen by a
Profileflagmake_profilesets on Apple, or byalias_isolation(macOS); with them off everymodule, layout and command is main's (the fixtures check it byte for byte).
gpu/vk/span_segments.h).RuntimeInfo::segmented).texel_codec.h;RX_VK_TEXEL_CODECS).span_xfer.h), and stand-ins for empty slots (null_dummies.h).kProgramCacheFormat3).msl_depth.cpp).msl_scope.cpp).import_hold.h).RX_VK_SPV_PROFILE, what the big modules are made of (log only).RX_VK_GM_LOOPon MoltenVK and KosmicKrisp).RX_VK_LOCAL_ZERO_LOOP).8e3c5951: a stand-in clear not done in 5 s is waited for before its fence and pool go (the Mac-only part of #788);81eff678keeps it inside#if defined(__APPLE__).7f9a99fa(the profile fixtures where #694, #696, #697 and #698 meet),74d8a42b(the segment fixtures take main'sFlatMap
c1_at, #231),30627309(main's Linux pressure fixture stands in forSegSig'sSegEntry).5. Precompile and pipelines on Metal (was #849)
Under the vendor fix
compiler_service(MoltenVK and KosmicKrisp only) the preparation compiles no compute pipeline, theprecompile orders the work for Metal's compiler service, lets the title come while the last giants build behind the game, bounds
giants and heavy compiles, and retries a failed compile alone (
gpu/vk/precompile_handoff.h, the precompile invk_draw.cpp).worker_caches.h,prepare_workers.h).gpu/vk/thread_stack.h).seed_compute).skip_log.h).RX_VK_METAL_ARCHIVE_DIR, Metal binary archives (metal_archive.h; off).RX_VK_LMEM_LOG(private_memory.h; log only).host/posix/darwin_metal_cache.h).e1965d56(a never-ending compile holds the heavy ones back 240 s at most),3c23a8e9(slow-first: what an earlier runfound slow compiles first),
cb01f764(metal_archive.h's notes).30252cf5: Metal's compiler service out of reach: compiles wait or are asked for again, no try alone spent (ServiceReach;run N3 lost 7,932 compute pipelines that way).
ad1147cb,69be3fd1,d450bc17(tests where two members met),42cc9868(main's Linux pressure fixture includesprivate_memory.h).6. Draw-path emulation on MoltenVK (was #850)
mode_census.h; log only).VK_KHR_maintenance5;RX_VK_INDEX8_SIZE).drawIndirectCount, buffer updates copied from the stream (count_draws.h).drawid_direct.h).docs/macos-moltenvk.md).fan_key.h; the program cache stamp).fan_indirect.h).provoking.h).8e3cbac6(test_fan_indirect's cut),6551f4c5(draw_index8_report's brace),e0cc1407(test_fan_key reads main'spredicted_keys_read(), #713).7. Indirect quads and mesh count draws on every driver, with their Mac halves (was #851)
a10d6c62(was #705, every driver): indirect quad draws drawn: their records rewritten on the GPU into indexed triangle lists(
gpu/vk/quad_draws.h,quad_draws.comp,quad_rewriteinvk_draw.cpp).fbaba871(was #707, every GPU on the mesh compute fallback): mesh count draws drawn, the records area grown past 64 MB whenthe allowance holds it, the cuts counted (
mesh_area.h,count_draws.h,host/nvn_native.c).a39f027d(#705 with area 6's censuses and checks),af4f2999(test_drawid_direct pins #707's DrawIndex condition).8. Mac performance (was #852)
vk_perf.cpp,perf_lines.h).506752c2(measured under the noise).RX_VK_INDEX8_RAW); its look-ahead dropped by95aa6b91(measured under the noise).
mesh_batch.h;RX_VK_MESH_BATCH).ebf9d527,615e698e(no provoking-vertex draw left out in a mesh batch; the every-platform middle commit is #853),f9b1c81b(macOS's write-mark walk, 64 marks a test in 16-byte loads;collect_ininhost/rx_backend.c,HOS_COLLECT_VECTOR).df8c1602,18bf99b2,325a9147,7c0a8884(tests where two members met),b0326575(the perf-lines andstream-room tests read main's
split_partand FlatMap, #443, #231).The join:
8d337c08merges area 5 into areas 6-8's line (Reviewer notes);6343f29enames #707's new fallback bind underRX_VK_LMEM_LOG, as #681 names the other fallback binds (log only; found bytest_private_memoryat the join).Docs:
docs/development.md(the macOS build and every switch's row),docs/control.md,docs/pipeline-cache-storage.md,docs/macos-moltenvk.md,gpu/vk/VULKAN.md.Measurements
Verdict: works. Main cannot build on a Mac; with this PR's commits (integration builds) the game held 1,197 s in play on
arm64 and 18 m 22 s under Rosetta from the field-dawn save, every draw drawn, all eight heavy compiles made, no device loss, a
clean exit. Two changes measured faster on the same build and ship on: MeshBatch (−1.0 ms a frame) and macOS's vector walk (the
replay's memory sync 3.25 → 0.25 ms a frame); two measured under the noise and are dropped (LoadClear, Widen8's look-ahead).
Hardware: macOS 26.7.1 (25G241), Apple M4 Max (40-core GPU), 16-core CPU, 64 GB; MoltenVK main 5189f8de + argbuf-slots (arm64
c170a745; its x86_64 slice96313ca8under Rosetta)Scene: Hyrule Field from the field-dawn save (slot 00, START then A), windowless unless named, LS's defaults; frame cap 60 for I39's E-series (E7: 240, which never binds),
HOS_FPS_MAX=240for E9Method: each run's longshot_log.txt (the precompile, hand-off, compile and close lines; the FPS counter's Dev level;
RX_VK_PERF_LINES' L1-L6), medians over the hold's 120-present periods from +20 s; one start each (E7 against E2 measured the run-to-run noise: 0.17 ms of frame)How to reproduce:
tools/ls-run.py(the port's runner) on a macOS build of these commits, the field-dawn save, START and A at the title; the switches in each columnBaseline: main @
7bacf846does not build or start on macOS, so the after columns are the port's integration builds I4178ec1c47and I4263fc3d2c(I41 plus slow-first), which carry these commits as they were before the rebase onto7bacf846(without
30252cf5and the Sweep4 test fixes; with the every-platform #788 and #853, which are separate PRs). The A/Bs belowcompare one build with a switch off and on. No game run at this PR's head yet (Testing).
7bacf846[mvk-error]The windowed run #38 (I42) held 540 s, 0 skipped draws, the tree leafy from its shot after 44511d69 was made; its lane's death
cut the hold at 542 s (the game quit on request, exit 0). The owner looked at E9a's shots in Preview.
The speed-ups (same build, switch off against on, one 600 s hold each):
7a7726fe, E4 → E2: fps / frame ms / GPU ms78ec1c47, E9b (HOS_COLLECT_VECTOR=0) → E9c: the replay's memory sync, ms a frameAcross builds, not A/Bs: MoltenVK 1.4.2 → main 5189f8de + argbuf-slots, GPU 23.1 → 20.25 ms a frame (#691's starts); with
RX_VK_INDEX8_RAW, MoltenVK's 8-bit pass splits 57.5 → 0 a frame (I38 → I39). MoltenVK's asynchronous submits measured 0.63ms slower (E6): not adopted. The every-platform present-nowait (#656) measured no gain on this Mac (E10).
What each area fixed, as each original PR measured it (its build or GPU probe named; history):
Flags and settings
The switches and build inputs, in area order: new ones read on macOS or MoltenVK/KosmicKrisp only; three measuring
switches (
RX_VK_PERF_LINES,RX_VK_GPU_TIMELINE,RX_VK_BENCH_AB) on every platform, off by default; nothing changes off the Macwith them unset.
RX_VK_LOAD_CLEARandRX_VK_WIDEN8are gone again (measured under the noise). Main's switches the diff shows,read as before:
RX_VK_DLSS_ACTORS,RX_VK_GPU_DUMP,RX_VK_MESH_IND_COUNT_WORD,RX_VK_NO_UPSCALE_STUB,HOS_HEADLESS,RX_VK_NO_WARM,RX_VK_PCACHE_FETCH_LOCK,RX_VK_SKIP_PROG,RX_VK_BENCH,RX_VK_CHUNK,RX_VK_DRAW_STATS,RX_VK_PASS_BARRIER,RX_VK_PASS_TIMING,RX_VK_BARRIER_DEDUPE_OFF,RX_VK_BARRIER_EVERY_DRAW,RX_VK_PUSH_STATE_OLD,HOS_COLLECT_WIDE.RX_VK_LIBRARY=<path>RX_ARCH=arm64orx86_64LONGSHOT_MOLTENVKlibMoltenVK.dylibdocs/macos-moltenvk.md; 0 lost pipelines); 1.4.2 acceptedLONGSHOT_SDL3libSDL3.0.dylibLONGSHOT_SDL2LONGSHOT_DARWIN_ARCHIVE_RECEIPT,LONGSHOT_DARWIN_ARCHIVE_RECEIPT_SHA256LONGSHOT_VULKAN_LOADER,LONGSHOT_KOSMICKRISPRX_MARCH,EMSDK,SDKROOT,LONGSHOT_FFMPEG,LONGSHOT_SDL2,RX_VK_ONLY(existing)LONGSHOT_NGX_SDK,LONGSHOT_FSR(existing)HOS_PROFILE,HOS_THREAD_TIMES,HOS_SAMPLE,HOS_SAMPLE_ALL,HOS_SAMPLE_FILE,HOS_CHAIN,RX_MAPS(existing)HOS_WATCH,HOS_PIN_BY_CORE,HOS_PIN,HOS_RESERVE_CORE(existing)HOS_ZERO_ALL_PAGES,HOS_KEEP_FREED_PAGES,RX_NVN_WATCH_TEXELS_SRC,RX_NVN_WATCH_AFTER,RX_NVN_WATCH_ALL(existing)RX_VK_UNIFIED_BUDGET=0vk: unified memoryline=0only to compareRX_VK_METAL_CACHE_DIR=<folder>/var/foldersRX_VK_DRIVERkosmickrispormoltenvkon a loader packagekosmickrisp: measurement only (run #20, no title)RX_VK_SET_FOR_TABLES,RX_VK_SET_FOR_LENGTHS0or1: ordinary sets for the programs a push breaks on MoltenVKRX_VK_TEXEL_CODECS0or1: texel buffers through carrier codecs=1elsewhereRX_VK_SPV_PROFILE,RX_VK_SPV_PROFILE_MINRX_VK_GM_LOOP(existing)RX_VK_LOCAL_ZERO_LOOP=0RX_VK_COMPILER_SERVICE0or1: the compiler-service preparation and precompile=0to compareRX_VK_WORKER_CACHESRX_VK_PCACHE_LOAD(existing default)RX_VK_SLOW_COMPILE_S,RX_VK_PRECOMPILE_STALL_S(the second existing)RX_VK_GIANT_COMPILES,RX_VK_GIANT_WORDSRX_VK_GIANT_HOLD_PRESENTS,RX_VK_GIANT_HOLD_S,RX_VK_GIANT_ROOM_MBRX_VK_COMPILE_QOSdefaultin the gameRX_VK_ALONE_TRIES,RX_VK_SKIP_LOGRX_VK_HEAVY_WEIGHT,RX_VK_HEAVY_WORDS,RX_VK_STALE_COMPILE_SRX_VK_SLOW_FIRST_SRX_VK_SERVICE_BURST,RX_VK_SERVICE_FAST_MSRX_VK_METAL_ARCHIVE_DIRRX_VK_LMEM_LOG,RX_VK_LMEM_LOG_MINRX_VK_INDEX8_SIZE0or1: a direct draw's 8-bit indices bound with their sizeRX_VK_COUNT_REWRITE=1drawIndirectCounttoo (macOS)RX_VK_PERF_LINES=1,RX_VK_GPU_TIMELINE=1or2,RX_VK_BENCH_AB=<arm>RX_VK_INDEX8_RAW0or1: restart-off 8-bit draws fetch 0xFF as vertex 255RX_VK_MESH_BATCH0or<n>: the mesh fallback's draws of a pass batchedHOS_COLLECT_VECTOR=0=0for A/BTesting
Current revision validation (
594d5296)Tested: game-free validation of the
594d5296source on this Mac, as detailed below; no current-revision game or Windows/Linux run is claimed.a8a30c22at 584/43/18. Six merge-related fixture failures were repaired and all six passed focused reruns. A Windows-only execution test gained its explicit macOS skip; the ordinary runner excludes it and still includes the passing portable report test. No new full-suite total is claimed after those fixture-only edits.Earlier validation (
6343f29e)Earlier tested revision: macOS 26.7.1, Apple M4 Max, 16-core CPU, 64 GB, Apple clang 21.0.0, at
6343f29eon main7bacf846,arm64 and x86_64 under Rosetta; these results are historical, not a fresh run of
594d5296:python -I -B tests/run_all.pythroughtools/ls-suite.py), compared by test name with main7bacf846's on this Mac: 649 passed, 16 failed, 17 skipped, against 558/34/17. New and passing: the 73 new tests. Fixed(FAIL on main): 17, the host's macOS repairs (
test_collect_wide,test_control,test_crash_record,test_longshot_ctl,test_low_memory_notice,test_prof_symbols,test_mem_gate,test_mem_gate_children,test_mirror_trim,test_oot_state_tap,test_report_runand six cache checks).test_appimage_package_retryis skipped on macOS by #454'sdeclaration, and
test_dav1d_av1runs and passes (#454 links dav1d). Nothing that passes on main fails. The 16 failures failon main too, on this Mac:
test_appimage_player_build,test_crash_dialog,test_env_on,test_game_layout,test_lift_cleanup_error,test_linux_preflight,test_make_game_cache,test_missing_nup,test_options_budget_once,test_options_path,test_pipeline_tour,test_readback_late,test_split_giants,test_vk_occlusion_queries,test_vk_pgo,test_walk_key.without an architecture), 459 pass. The 23 others are explained, none a defect of this head: 16 base-tree checks of area 4
(
test_seg_shader --base-treeagainst seven old member bases,test_local_zero --base) differ from those bases only in themesh fallback's modules, which #707 (area 7) changes on purpose (they pass up to area 6); the consumer-check job, 54 of its 60
cases as required; 4 runs of main's own failing
test_vk_occlusion_queries; 2 of #667's oldtest_metal_cache_dir --headersform (#786's form passes). 33 jobs passed on a second run: Rosetta SIGKILLs under load (exit -9, no crash report), and runs
repeated after a fix.
seg-bindpositives (both arches) report "common path differs from B2's cut"because they compare with B2
918e6214, from before main7bacf846changed the functions they cut (the same at area 4's owntip); the
seg-gathersandseg-xferpositives cannot build B2 with today's fixtures, which include areas 6 and 8's headers(they pass at area 4's tip).
test_seg_bind,test_seg_gathersandtest_seg_xferthemselves pass at this head, both arches.nx2/build.pymacOS commands(104 compiles, arm64 and x86_64) and the 12
gpu/vk/edenback-end files with the renderer library's flags (24 compiles): 128compiles, all exit 0;
nx2/bake_ds/bake_ds.cpp(a Linux lift tool) through its member test.gpu/vkfile compiled with the tree'srenderer_flags(), arm64 and x86_64): 68 of68 objects, 610 cross-file
mkwvk::symbols per architecture, every one defined.unifdef -U__APPLE__of the 130 changed non-test C/C++ files against main, compared as tokens): 16 the same, 55differ, 59 new (11 of them only Apple blocks). Each area's off-Apple change, compared at its tip with its original PRs' before
the rebase, is that change or differs by the rebase's hand resolutions only (Reviewer notes); the join adds nothing off Apple
but
6343f29e's log-only line. What differs is listed under Risks.589/18/17, 559/35/17, 614/20/17, 627/20/17, 621/21/17, 626/22/17, 629/23/17 (areas 1-8), whose new failures led to the
seven fixture fixes named above (beside the load-sensitive
test_gpu_hang_watch).s) and #38 on I42 (windowed, 540 s); earlier, #32 on I38 (arm64, 900 s) and #34 on I38 (Rosetta, 16 m 52 s).
Not tested: Windows and Linux execution of the current revision; a game run of
594d5296. The game runs below are earlierintegration builds without
30252cf5. No scene with an indirect quad draw or a mesh count draw (field-dawn draws neither:E9a has no
vk: quad drawsline andvk: mesh draws0.0 a frame with a count), so area 7's every-driver paths ran only in thetests and the QD-01 and MC-01 probes, never on a GPU with mesh shaders or another driver's fallback. A segmented domain in a
game; KosmicKrisp in a game beyond run #20; a warm start from Metal binary archives as shipped (off); a Rosetta frame rate in a
quiet hold on I42; Macs with 8-32 GB; macOS 14.4 and 15 (every run was on 26.7.1); an Intel Mac; SDL2-compat loading the
packaged SDL3 in a game.
Must be tested:
594d5296) and maina8a30c22the same way, runpython -I -B tests/run_all.pyon each and start each to the title, expecting successful builds and no regression against that main, with every unsupported
check explicitly declared (by hand: compare the suite results and renderer/cache startup logs). T-3F2Q was opened for the
earlier
6343f29e; it is not acceptance evidence for this updated head. Current-head qualification is required before ready.Linux by default, Navi 21/22 on Windows) play to the field with
RX_VK_VALIDATION=1, expectingvk: quad draws (…)with 0left out once a quad is drawn,
vk: mesh draws (…)with none left out,vk: mesh fallback cuts so farall zero on thefallback GPU, no validation error, no
vk: provoking vertexline and the scenes as on main (by hand: T-3F2Q, withscreenshots of anything that differs)
RX_VK_PERF_LINES=1 RX_VK_GPU_TIMELINE=1and play a minute,expecting the
vk: L1-L6lines and a T1 line every 120 presents, the frame times as without them (by hand: T-3F2Q)Risks
What could break: on Windows and Linux, by intent: every driver now draws indirect quad draws and every GPU on the mesh
compute fallback draws mesh count draws, both left out before (a pass break per indirect quad draw, device-wide buffers made at
the first one; the fallback's records area can grow past 64 MB; fallback stages that load DrawID read it pushed); a startup that
fails part-way unwinds what it made; on Linux, Discord presence's socket code and the thread-creation error paths; the build's
Windows and Linux arrays moved into functions (pinned element for element). Compiled everywhere but gated off at run time: the
back end's new emit paths, the descriptor-set choice, the compiler-service precompile, the draw-path fixes, MeshBatch; a gate
held wrong would change modules or commands off the Mac (the fixtures check the modules byte for byte against main's). The perf
hooks cost one flag load each with their switches off (about 1 ns a call). On macOS everything is new and has run game-free and
in the integration builds only: the start's order under the compiler service; sets instead of pushes, null-descriptor guards,
stored image reads and one-lane warps cost what no frame has measured.
Turning it off: the macOS code is behind
__APPLE__or a driver gate; the switches above turn its parts off(
RX_VK_COMPILER_SERVICE=0,RX_VK_MESH_BATCH=0,RX_VK_UNIFIED_BUDGET=0,RX_VK_SET_FOR_TABLES=0, …). The every-driverdraws of area 7 have no switch: revert their two commits (was #705, #707), or the PR.
Compatibility: every platform lifts the game again once (
nx2/kit.pyand the newnx2/darwin_toolchain.pyare liftsources), prepares its shaders again once (the translator's stamp changes:
vk_shader.cpp, the back end, the fallback's DrawID),rebuilds its program cache once (
kProgramCacheFormat1 → 3) and compiles the host's objects again once. The vk ABI:MKWVK_API_VERSIONis 27 on macOS only (host slots appended by #438, #491, #686 and #786; the macOS executable and renderer arebuilt together); Windows and Linux keep main's 23 with the same structure. Saves and
longshot_options.iniare unchanged. Everyplatform's log gains fields: the
vk: vendor fixesline (0 off MoltenVK), the quad and mesh draw reports. On MoltenVK thepipeline cache file is no longer read or written, and
longshot_pipelines.bin.slowlists slow compiles.Reviewer notes
Reading order: the areas in commit order. The every-platform parts to read closely: area 7's two commits (was #705, #707),
#491's startup rollback (area 2), #490's Linux changes (area 1) and #704's measuring hooks (area 8). Then the Mac's: the build
scripts and
host/posix/darwin_*.c(1),vk_memory.cpp's imports and the budget (2),layout_pushes()(3), the back end byProfileflag (4),precompile_handoff.h(5),draw()invk_draw.cpp(6, 7, 8).By commit:
git log --reverse --no-merges --format='%h %s' 7bacf846..6343f29e. Each original PR's commit names it (Was: #NNNwith its branch; area 7's two say "#705 restacked …", "#707 …"); that PR's page keeps its review and measurements. The named
fix commits sit after their area's members. The maintainers squash at merge.
Topology: each topic's commits are linear on its base; the merges are the stack's own.
#848 sits on #846 and merges #847 and #431; seven "Merge topic/ into topic/" commits carry Sweep4's test fixes up
the stack. #849 and #850 both sit on #848, so one merge commit,
8d337c08, joins #849's line to #850-#852's: in its fiveconflicted files (
vk_draw.cpp,vk_main.cpp,vk_progcache.cpp,vk_replay.cpp,vk_vendor.h) both sides add at oneplace, and the merge keeps both, in integration build I42's (
63fc3d2c) order. Two more merges (562d10b3,e9404d81) takeareas 4 and 5's last fixture fixes up, and
6343f29eadds the one line where areas 5 and 7 meet.Rebased onto main
7bacf846(fromaa95d35f); the hand resolutions, each also in its commit's message:d85dac1a(#453),0e7ee58e(#479): main dropped the WebGPU renderer (#577), so the macOS build has none; main's host-sourceline stays verbatim (main's
test_no_webgpu.pyunchanged), and the link takesresources=exe_resources()(#857).81c21324(#491): main'sevicted_import_bytesaccounting (#223) kept beforedestroy_import(p);ec5731cc(#642): main'sLinux pressure kept beside the unified-memory
keep_now.88e914ad);vk_shader.hkeeps main'srecord_at/record_sizeandarray_length.a9888f20(#663): main's FlatMapc1_at(#231) with the segment arena; by the owner's decision on #403, main's generic dummiesstay on every device without
nullDescriptor(make_dummiesand its test unchanged) and the typed stand-ins move tomake_stand_ins(), Apple only. #431's merge takes its earlier resolution (vk_memory.cppand two tests).64c14f29(#651),9ab883a4(#665): include lines only.256ddf7e(#704): the perf op before main'ssplit_part(#443), its reason line inside it;e26f6c09(#783): main'sc1_at.reset()before the mesh batch's start (test_mesh_batchasks for it, the owner's decision);14cc564d,644770fc,506752c2: the LoadClear and Widen8 lines around main's split.Main integration:
594d5296has parents6343f29eanda8a30c22; the previously listed 19 conflicts with that main areresolved. Current-revision validation is listed above. Any newer upstream changes and the live mergeability state must be
checked again before this PR is marked ready; no additional rebase or acceptance run is scheduled during the owner's hold.
Related every-platform PRs, kept separate: #431 (merged inside #848; review it there; its two commits drop out of this diff
once it lands); #853 (a direct quad draw's triangles grow the stream under
RX_VK_STREAM_GROW_OFF: the every-platform half ofarea 8's stream-room port; merges cleanly with this PR); #907 (a direct non-indexed quad draw's BaseVertex is its first vertex,
as area 7's indirect quads read it; conflicts with this PR in
vk_draw.cpp); #936 (the write-mark walk vectorised on everyplatform, of which area 8's macOS walk is the Mac form; conflicts in
host/rx_backend.c,tests/test_collect_wide.pyanddocs/development.md: whichever lands second takes the other's walk); #788 (async-safe frees; area 4 carries its Mac-only part;merges cleanly); #656 (present-nowait; no gain on this Mac). Independent of this PR: #423 #428 #434 #451 #452 #474 #475 #498 #670
#700 #912. These overlap notes were assessed at
6343f29e; recheck them before integrating another PR.Coming separately: the macOS nightly builder (a zip for Apple silicon and CircleCI jobs), as its own PR on top of this one.
Issues
See #420.
Checklist
Everyone
type(scope): description) of the status line's type.main: maina8a30c22is merged at594d5296; a final live-main check is owed before ready.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). See current-revision full-run and focused-repair evidence under Testing; Windows/Linux and final game-build acceptance remain open.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.MKWVK_API_VERSIONbumped (gpu/vk/vk_api.h'sMkwVkHostorMkwVkRendererchanged). (macOS 27; Windows and Linux 23.)mainand passing here.tests/that reads changed source updated in this PR, not weakened.Every scan down vk_memory.cpp's import table by start (pool_lookup's covering import and merge plan, bg_drain, prefetch_chunks, import_in_background, memory_note_gpu_write and memory_guest_imported) stopped once an import ended more than 1 GiB below its address, on the belief that no import is larger than that. pool_lookup's own merge escape leaves an import above 1 GiB over the smaller ones inside it ("left overlapping"), and the device allows imports up to max_import_size: a scan stopped at a small import inside a large one, missed the large one, and the same bytes were imported again, two memories over the same bytes. import_reach.h counts the live imports by span class; its bound is above every live span, within twice the largest. imported_add and imported_take, now the table's only ways in and out, keep the count. A scan stops only where no import further down can reach its address, so it also stops sooner than before when the imports are small. The covering-import checks in pool_lookup and import_in_background look past the last import at or below the range for the same reason. Tests: test_import_reach.py runs the actual scans with a 4 GiB import over a 16 MiB chunk, left by the merge escape (imports are recorded ranges: no payload, no Vulkan, no game). Its lookup, query, note, drain, background and prefetch cases fail on main5e072fc8and pass here, and a source check keeps every change to the table in the two helpers. test_import_windows runs pool_lookup with the helpers; test_mirror_gpu_only names the new publication point. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]>Possibly related, for context:
a8a30c22into macOS buildTest report from a second Mac: an M1 Max (24-core GPU), 32 GB, macOS 27.0.1, at this PR's head
594d5296. It builds and plays. The scenes I reached look right. Three problems showed up that the M4 Max runs in the description don't show: the compiler service runs out of memory, every start is slow, and gameplay runs at single-digit frame rates.(Disclosure: the testing and this write-up were done by an AI agent, Claude, on Matthew's Mac, with his permission. Every number below comes from the runs' own logs.)
Setup
xcrungives a different MoltenVK hash.5189f8de+moltenvk-argbuf-slots.patch, built with the script indocs/macos-moltenvk.md. Universal SHA-256ac45f2b0…, arm64 slicef961a4e1…, not the documentedb4a59f07…; I believe that's the SDK difference. The renderer's line readsMoltenVK 1.4.3.make_game.py --checkcaught my own FFmpeg's NEON objects declaring macOS 27 (requires macOS 27.0.0, newer than target 14.4). After a rebuild withMACOSX_DEPLOYMENT_TARGETset, it passed, and the build finished in one go.What works
kokiri-startsave loads.1. Metal's compiler service runs out of memory: 8 mesh fallback programs not made, their draws left out
not made (-3)withits draws are left out:ea4dd64df9c8f2b7and93e3e999f2fb9fd8: 1,087,607 SPIR-V words each, about 57 s per attempt.29a8af226b5afad2,2d7d43c7cb9c077b,38a34ca000b39f70,682b5e6954b55941,7b9d6679d826f98aand7c6ca92f187ecd19: 85-93k words.4f0c0b40ab3a32cd: 66k words.Compilation failed due to an interrupted connection: XPC_ERROR_CONNECTION_INTERRUPTED. This error occurred after multiple retries.MTLCompilerServicecrash reports overnight. They'reSIGABRTfromllvm::report_bad_alloc_error(inStringMapImpl::RehashTable,SROAPass::promoteAllocas/PromoteMemToReg,MachineRegisterInfo::createIncompleteVirtualRegister), plusSIGBUSinMachineOperand::setReg. In other words, the compiler runs out of memory on these programs.make_game.pyhit the same XPC errors (14 compute pipelines), but its retries made them all.RX_VK_ALONE_TRIESas untested for lack of such a failure. On 32 GB that failure is real.2. Every start takes 18-28 minutes before the first frame
vk: precompile: ... programs ... in N ms:RX_VK_METAL_CACHE_DIRset to a new empty folder.RX_VK_METAL_CACHE_DIR, Metal's files stopped growing atfunctions.data= 1,118,912,512 bytes andlibraries.data≈ 1.16 GB. That's within a few hundred KB of what the default per-user cache holds. My guess, not checked against Apple's docs: Metal caps each cache file near 1 GiB, so this game's ~38,000 pipelines never all stay cached and each start compiles much of the set again.vk: driver shader cache at the start: 0 MB (no folder found)only looks at NVIDIA's and Mesa's folders. It may want the Metal folder there instead.3. Frame rate in play: 5-10 fps
RX_VK_PERF_LINES=1period in play readGPU time 124.6 ms a frame (the longest submission 1247.8 ms). The menus read 12-23 ms, at 28-33 fps.done, 43896 of 43896 items) before most of these readings, so the giant compiles aren't what's using the GPU.Logs for any of these are available on request. I'll keep testing adult Link and the dungeon interiors and add to this.
Follow-up from the same M1 Max (32 GB), same head
594d5296: adult Link and the dungeons, plus one device loss.Adult world and dungeons draw correctly. The bridge's
age adultworked, and the teleport reached every adult destination except the Crater, which I skipped so Link wouldn't burn:The Sheik scene in the Meadow played through, including the Minuet's ocarina prompt.
Device lost once, about 70 s into a start. MoltenVK reported:
Two things were unusual about that start:
longshot_ctl quit, and it loggedprogram cache ...: another Longshot is using it: this one runs without it, so its precompile was0 programs ... in 500 msand everything compiled at first draw.The next start, with the cache, played for about an hour with no device loss. I can't say which of the two matters, or whether either does.
More numbers for the earlier three points
RX_VK_METAL_CACHE_DIRfolder.MTLCompilerServicecrash reports. The same 8 mesh fallback programs were not made, plus the M4 Max run's 895-s giant44511d69b52282ad.vk: GPU timeover 524 periods: median 90.5 ms a frame, max 228.5 ms (menus about 1-23 ms).Frame time scales with render resolution, so it's GPU work, not paging. In adult Kokiri Forest, standing still (about 800-880 draws), I alternated the Insert menu's render resolution twice:
vk: GPU timeper frameFour times the pixels cost 2.4 times the GPU time. Even at 50% the GPU needs about 88 ms a frame, against 19.4 ms on the M4 Max in the description (I don't know the resolution of that run). A 24-core M1 Max should be slower than a 40-core M4 Max, but probably not 4-10x slower, so something in the frame may be much more expensive on this GPU family than on the M4. I tried to find which work it is:
RX_VK_PASS_TIMING=2logs0.00 msfor every span on MoltenVK, so its in-pass timestamps don't seem to work there.xctrace, Metal System Trace template, same spot, 50%, about 40 frames) attributes about 1.83 s of GPU time to fragment work, 1.17 s to compute and 0.16 s to vertex: roughly 46, 29 and 4 ms a frame. These are lower bounds: rows the export lists by reference were skipped.A caveat I should have put in the first comment: this Mac had other apps open (a browser among them), and swap was already about 11 GB before the game started. The memory numbers come from an ordinary desktop, not a clean machine. That's realistic for a player, but it makes the frame times and the compiler's out-of-memory failures worse than a fresh boot would.
View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.