WIP: docs(macos): MoltenVK main 5189f8de with the argbuf-slots patch as the recommended MoltenVK input, and how to build it #691
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!691
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:mac/moltenvk-main"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: docs · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (MoltenVK main 5189f8de + argbuf-slots, start #31; the documented build run as written at this head) · Needs: maintainers' approval of the MoltenVK input; nothing on Windows or Linux (docs and a Python comment only)
Why
The macOS port now builds and runs with a MoltenVK built from MoltenVK's
main(5189f8de) plus one small patch,argbuf-slots, instead of the 1.4.2 release.mainhas what 1.4.2 lacks (drawIndirectCount, MoltenVK PR #2775, and thedraw-index fix bc0d5bf733), but on its own it lost 39 of the game's graphics pipelines at a Mac start (
Vertex shader requires buffer size buffer, but there is no free slot to pass it). With the patch, start #31 (arm64) had 0 such errors, 0failed graphics pipelines and 0 skipped draws in all 174 hold groups, with count draws drawn by MoltenVK itself. This PR
puts the patch in the repository, documents exactly how to build that dylib with Apple's Command Line Tools, and names it
in the
LONGSHOT_MOLTENVKrow as the macOS work's recommended input, pending the maintainers' approval (the row asks for anapproved dylib); 1.4.2 stays an option. It changes no code. It is part of the macOS port. See #420.
How to reproduce
Seen on: macOS 26.7.1, Apple M4 Max, LS
macos/integratedb8170e55packaged with MoltenVKmain5189f8de, arm64(start #28); the cause is in MoltenVK, not LS
Settings and switches: defaults
Steps: build MoltenVK
main5189f8de without the patch (docs/macos-moltenvk.md, leaving out step 2), pass it asLONGSHOT_MOLTENVK, start the game and let the precompile runExpected: every graphics pipeline is made, as with 1.4.2
Actual: 39 graphics pipelines are not made (78
[mvk-error] VK_ERROR_INITIALIZATION_FAILED: Vertex shader requires buffer size buffer, but there is no free slot to pass it.lines, each pipeline at its compile and its retry); with theargbuf-slotspatch (start #31) noneWhat changed
The patch MoltenVK needs is now in the repository with a document that builds the recommended MoltenVK dylib from pinned
sources with the Command Line Tools alone; the
LONGSHOT_MOLTENVKrow names it as the recommended input and points there.gpu/vk/VULKAN.mdsays whatmainchanges for LS's count draws and buffer updates. No code changes.docs/patches/moltenvk-argbuf-slots.patch(new, SHA-25674f517e3ed9f2b0e473e811c48802fa54c2c627cc9a144471f143cd633bd99fc,43 lines): in
MoltenVK/MoltenVK/GPUObjects/MVKPipeline.mm,MVKPipelineLayout::Createreserves Metal buffer indicesfor argument buffers only up to the layout's last argument-buffer set instead of
kMVKMaxDescriptorSetCount(16 sinceSPIRV-Cross 763015417e raised
kMaxArgumentBuffersfrom 8), plus a line in MoltenVK'sDocs/Whats_New.md. LS had nofolder for third-party patches, hence
docs/patches/.docs/macos-moltenvk.md(new): the two known inputs; whymainand why the patch; one bash recipe: the six sourcearchives with their SHA-256 (MoltenVK 5189f8de and the revisions its
ExternalRevisions/*_repo_revisionfiles list), thepatch applied with no fuzz, the revision string that gives a patched build its own
pipelineCacheUUID, the four statictargets' sources and flags (the Release settings of MoltenVK's Xcode projects:
-O2,NDEBUG, hidden visibility, C++17with libc++, no ARC, no modules, macOS 12.0, plus
-ffile-prefix-map), the dead-stripped link with its frameworks, thead-hoc signature,
lipo, the output file to set asLONGSHOT_MOLTENVK, and its expected SHA-256 for Apple clang 21.0.0and the macOS 27.0 SDK.
docs/development.md, theLONGSHOT_MOLTENVKrow: the recommended input (pending approval), the patch and its SHA-256, alink to
macos-moltenvk.md, 1.4.2 an option.gpu/vk/VULKAN.md: the count-draw paragraph saysmain5189f8de or later draws count draws itself (the rewrite runs onlywhere
drawIndirectCountis missing, as before); the buffer-update paragraph says MoltenVK fixed the draw-index zeroingafter 1.4.2 and that LS's stream copy stays on every MoltenVK, keyed on the driver.
nx2/darwin_package.py: a comment (ICD_API_VERSIONis 1.4.0 in the manifests of 1.4.2 and ofmain5189f8de).Measurements
Verdict: nothing to measure in this PR's own change (documentation and a patch file). The recipe was run as written and
made the very dylib start #31 ran, byte for byte (universal
b4a59f07…). The input it recommends rests on these Mac starts.Hardware: macOS 26.7.1, Apple M4 Max, 64 GB, Apple clang 21.0.0 (clang-2100.3.34.2), macOS 27.0 SDK
Scene: the field-dawn save, played into the field (game data stays on the owner's machines; only renderer log lines read)
Method: start #28 (
mac-arm64-b8170e55-mvk01, unpatchedmain) and start #31 (mac-arm64-b4732392-02, I36 withmainargbuf-slots, universal dylibb4a59f07…), each one 900 s hold; the patch's probe MS-01 (one admission, arm64) on 1.4.2,mainand patchedmainHow to reproduce:
docs/macos-moltenvk.md's recipe, then build LS withLONGSHOT_MOLTENVKset to its output and start thegame
Baseline: MoltenVK 1.4.2 (start #26) and unpatched
main(start #28)no free sloterrorslen_b15refused; binding 14 read the push constants from the vertex buffer)Flags and settings
None in this PR's own commits. The diff against main also shows the switches of the PRs it stacks on (#679, #654 and the
PRs they carry), theirs to list.
Testing
Tested: at
3cf8bcd8, this PR's head before the rebase ontoe090cc48(main6c9edffe, #679's stack + #654, this PR),on macOS 26.7.1, Apple M4 Max:
docs/macos-moltenvk.md's bash block run exactly as written (extracted from the file and piped to bash,LSthischeckout, an empty work folder, the six archives already downloaded, so their SHA-256 checks ran but no download did):
the universal
libMoltenVK.dylibis byte for byte the one start #31 ran (b4a59f07e12a1f7e25963f27d6126fadf26b8e3f6ae57137eb005862307edb83;arm64 slice
c170a745…, x86_64 slice96313ca8…), so the document reproduces the tested input exactly.docs/patches/moltenvk-argbuf-slots.patchis byte for byte the patch the starts and MS-01 were built with (SHA-256above).
tests/test_macos_package.py --arch arm64 --arch x86_64(the package code whose comment changed),
tests/test_check_pr_description.py,tests/test_vk_darwin_admission.pyonarm64 and x86_64 (the driver choice, 27 cases, through the merge of #654 into #679's stack),
tests/test_count_draws.py: allexit 0. The full suite at this head is in the run_all box.
reports/PubWave-files/PubNext/(proof-md-02/,mvk-docs/,mvk-recipe/).e090cc48, at04be6e14(was3cf8bcd8): Sweep2's own checks of this head were stillrunning at writing (
reports/Sweep2-files/lanes/691.txt);python -I -B tests/run_all.pyon wave C6's merge82bdf719, which carries this head: 586/31/18 (passed/failed/skipped) against main e090cc48's 544/32/17 on this Mac,no new failure (
reports/2026-11-14-rebase-sweep-2.md).Not tested: nothing new runs in LS: no code changes. The recipe with another compiler or SDK (other bytes are expected).
Windows and Linux: nothing they build changes (a Python comment in a macOS packaging module and Markdown files).
Must be tested:
docs/macos-moltenvk.md's recipe, buildLS with
LONGSHOT_MOLTENVKset to its output and start the game, expecting thevk:device line to name MoltenVK 1.4.3 andno
no free slotline (by hand: test request T-XXXX)Risks
What could break: nothing at run time (docs, a patch file and a comment). The recommended input needs a source build;
1.4.2 stays accepted.
Turning it off: revert the PR.
Compatibility: no change to saves,
longshot_options.ini, the translator stamp or the vk ABI.Reviewer notes
c45636d2..HEADwithout the merge:96367c48ismacos/moltenvk-main's 1106a4da (-x), itsVULKAN.mdcount-draw paragraph merged with #679's macOS-only wording; the last commit adds the patch,
macos-moltenvk.mdand the row.c45636d2: theVULKAN.mdparagraphs) and #654 (kosmickrisp, merged inea99f478, clean: theLONGSHOT_MOLTENVKrow anddarwin_package.py). Until they merge, this PR's diff shows theirs too.macos-moltenvk.mdand the row can name that release and the patch can go.Issues
See #420
Checklist
Everyone
type(scope): description) of the status line's type.main(maine090cc48; head04be6e14, was3cf8bcd8: this PR's own commits replayed one for one, old → new:ea99f478→8273e75e,96367c48→eb89b847,3cf8bcd8→04be6e14; the commit names elsewhere in this description are from before the rebase).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). (At this head3cf8bcd8: 562 passed, 31 failed, 18 skipped against main 6c9edffe's 531/32/17 on this Mac; the 31 failures are main's own and nothing that passes on main fails;test_collect_widenow passes; 30 new tests (the stacked PRs') pass;test_darwin_audio_service, Darwin-only, is skipped by the runner;reports/PubWave-files/PubNext/mvk-docs/full-3cf8bcd8/. After the rebase onto maine090cc48(head04be6e14): wave C6's merge82bdf719, 586/31/18 against main e090cc48's 544/32/17, no new failure.)third_party/untouched (a new folder with aPROVENANCE.mdinstead).docs/as a build input; LS does not compile it.)When it applies
Assisted-by:/Co-Authored-By:trailers.Every scan down vk_memory.cpp's import table by start (pool_lookup's covering import and merge plan, bg_drain, prefetch_chunks, import_in_background, memory_note_gpu_write and memory_guest_imported) stopped once an import ended more than 1 GiB below its address, on the belief that no import is larger than that. pool_lookup's own merge escape leaves an import above 1 GiB over the smaller ones inside it ("left overlapping"), and the device allows imports up to max_import_size: a scan stopped at a small import inside a large one, missed the large one, and the same bytes were imported again, two memories over the same bytes. import_reach.h counts the live imports by span class; its bound is above every live span, within twice the largest. imported_add and imported_take, now the table's only ways in and out, keep the count. A scan stops only where no import further down can reach its address, so it also stops sooner than before when the imports are small. The covering-import checks in pool_lookup and import_in_background look past the last import at or below the range for the same reason. Tests: test_import_reach.py runs the actual scans with a 4 GiB import over a 16 MiB chunk, left by the merge escape (imports are recorded ranges: no payload, no Vulkan, no game). Its lookup, query, note, drain, background and prefetch cases fail on main5e072fc8and pass here, and a source check keeps every change to the table in the two helpers. test_import_windows runs pool_lookup with the helpers; test_mirror_gpu_only names the new publication point. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]>044635ea added vma_startup_rollback (vk_vma.cpp, run after a failed replay_start), which calls vmaDestroyAllocator. vk_vma.cpp includes vk_work.h last so that RX_VK_RECORDER=defer/thread drains before its destructions, and every such boundary call in a vk_work.h file must be in RX_REC_BOUNDARIES with a drain-first macro. This one was not, and test_recorder_sync caught it ("vk_vma.cpp calls ['vmaDestroyAllocator'], which vk_work.h does not drain before"): it passes at 58c5c5aa (044635ea's parent) and fails at 044635ea. Add it to the list and to vk_work.h, next to vmaDestroyBuffer and vmaDestroyImage. With the recorder off (the default) boundary() is a relaxed load and a branch. Rollback runs before any image or buffer exists, so nothing should be held there today, but the allocator is no longer destroyed while a recorder could still hold packets. The test is unchanged (sources: 42 boundaries drained, was 41). Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]>A guest span larger than one device allocation (the cap on MoltenVK/M4 Max is 41,747,087,360 bytes) is bound as pieces: one descriptor per allocation segment, at the descriptor's own binding (reports/2026-11-14-full-range-design.md sections 4.1-4.4). This is the shader half; the draw side (descriptors, layout, gathers) comes later. - runtime_info.h: SegEntry {Storage|Table|TexBuf|ImgBuf, desc, pieces} and RuntimeInfo::segmented. - Back end: a segmented storage buffer is an array of pieces at its binding (a table count * pieces, entry k's piece j at k * pieces + j); its loads, stores and atomics call DontInline helpers made after main, which find the piece by the pieces' OpArrayLength in order, with constant indices. The global memory fallbacks sum the pieces' lengths for a segmented descriptor's bound end and branch to its piece (single, native candidate, table written out and in the loop). Texel and image buffers are arrays too, their fetches, reads, writes and atomics through helpers that pick the piece by OpImageQuerySize, and a texel buffer's size query sums the pieces. ImageQuery was declared for every module already. With RuntimeInfo::segmented empty nothing new is made: the SPIR-V is byte-identical. - vk_shader: SegSig (entries, hash) on Program, seg_sig and seg_variant; program_segmented finds or translates the variant of a plain program with plain's texture questions and answers and checks its descriptors are plain's; program_pick, mesh_program_locked, program_find_plain, loaded_variants and portable_variant keep the plain and segmented variants apart; every stage's RuntimeInfo::segmented comes from Program::seg. - vk_progcache: a segmented variant is never written to the program cache nor listed among the loaded programs. - tests/test_seg_shader.py + tests/seg_shader_ir.cpp: synthetic IR programs through the real back end, plain and segmented; structure checks, spirv-val (Vulkan 1.3, uniform buffer standard layout), and with --base-tree a byte comparison of the plain modules against an older tree. Five source-reading checks follow the new Program field and the atomics' new branch. The translator stamp changes with this (eden/, vk_shader.cpp, sizeof(Program)): every program cache is rebuilt once. Land it in the same release as the vk_api.h change that already does so. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]> (cherry picked from commit b7928876dab846485f667c1c934d34d25ecef035)Carries the vk-admission pull request's commits, which the packaged driver choice needs (host/posix/darwin_vulkan.{c,h} and test_vk_darwin_admission). #479 (mac/build-complete, this branch's start) brings darwin_build.py and darwin_package.py; the two merge without a conflict. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]>The interfaces the four segmented-consumer lanes (SegBind, SegXfer, TexelDesc, NullShader; reports/2026-11-14-seg-consumer-plan.md section 3) code against, frozen before they fork. Inert: no caller of anything new, no SPIR-V, layout, push range or stream change, sizeof(Program) unchanged. - vk_internal.h / vk_memory.cpp, SegMem's consumer interface: gpu_span_any() (a relaxed count of the live domains, stored in seg_commit; domains are kept for the run), gpu_span (gpu_buffer's flavour of gpu_span_bind: import buffers, no written marks, no mirror), SpanLayout::bind_stamp (gpu_bind_stamp(): a segment can be imported again with its cuts kept, so cut_gen alone never keys piece buffers), span_unused_piece's semantics ({null_storage, 0, 1}), and the SegArena API (SegArenaRef, seg_arena_take, seg_gather, seg_scatter; stub bodies after stream_alloc that give no room and log, SegBind's to fill). - Review fixes in the same interface (renderer-core review P2-3, P3.2): every piece of gpu_span_bind carries its buffer's device address (the import's or the mirror's; 0 only with bufferDeviceAddress off), and its one-piece path stores the translated guest address, not a virtual one. - vk_texture.h/.cpp: texel_view_pieces / texel_piece_views, stubs that return 1 and false (TexelDesc's to fill). - vk_shader.h: Program::presence_words, in what was padding. - Back end: PRESENCE_PUSH_OFFSET (88), Profile::emulate_null_descriptors, Bindings::presence / presence_read with the bit numbering rule. - tests: test_resolver_coverage lists gpu_span_bind / gpu_span sites too (scoped to gpu/vk: gpu/render has an rx::gpu_span of its own), checks that gpu_span follows its own gpu_buffer, and splits SITES into SITES_BASE plus one block per lane. test_segmented_domains executes the contract (the count 0, 1, 2 -> 1 on a merge, 0; gpu_span's pieces; addresses with the fixture's gpu_buffer_bind returning none as the real one does; the bind stamp; a virtual address's guest). test_seg_shader accepts a base tree of this back end's generation and compares every module. - docs/segmented-binding.md: the gate (every new behaviour Apple-only, other platforms byte-identical) and one heading per lane. Conflict on this branch (not with main): tests/test_seg_shader.py's docstring, against the array_pointer_calls paragraph #P20's msl-arrayptr port added; both kept, this paragraph first. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]> (cherry picked from commit 95fa06d4e6526601bd2e1bc88c2f8f5720855350)The segmented single storage array's case (plan 4.3 item 5) had no negative: the check always cut its own tree. With --source <B2 archive> it fails as required ("moltenvk.segmented_single want=set got=push"). Conflict with #P28 (read_write textures; not with main): the docstring keeps #P28's paragraph with this usage line, and main() cuts from --source's vk_draw.cpp with #P28's read_write count (as in the macOS integration, Integration 17). Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]> (cherry picked from commit 5824f6828e21cf810dcceaa6be425a9224b15994)plaine37c097da2plaina997b3629aKosmicKrisp has no fillModeNonSolid and MoltenVK no POINT polygons, and Metal restarts an indexed strip or fan at the all-ones index whatever Vulkan's restart enable says. Before anything is fixed, count whether the game needs it (mode_census.h, Apple only; nothing built or drawn changes): - the graphics pipelines built, where st_pipelines counts them, and the draws recorded, where st_draws counts them, by NVN's polygon mode, with the first 8 LINE and POINT pipelines (program, pipeline key, topology) and programs named; - the indexed strip and fan draws with restart off; a direct one's indices are read whole, a page at a time, through memory_read_as_submitted for the index type's all-ones value (0xFF, 0xFFFF, 0xFFFFFFFF). An indirect one's range is the GPU's and a range the CPU cannot read is not guessed: both are counted as not scanned. Any other draw costs restart_off_strip's compares and its polygon mode's count on the thread's own counters. Printed with the frame statistics every 120 presents ("at present N") and at close ("at close"). The two tests that compile build_graphics and draw()'s binds from the source get a stand-in for the counts. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]> (cherry picked from commit 1bd30dceb348909ca03c4cf12709dccbb0900f4a)3cf8bcd81d04be6e14ae04be6e14aeebd6b127e2Folded into #850 (fix(macos): draw-path emulation on MoltenVK)
Pull request closed