feat(vk): bind storage and texel descriptors as piece arrays in the SPIR-V back end #648

Closed
mooki wants to merge 2 commits from mooki/Longshot:mac/seg-shader into main
Member

Type: feat · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (MoltenVK 1.4.2), game-free at this head; GPU probes on both CPU routes (history) · Needs: Windows, Linux

Why

On a Mac, MoltenVK caps one device allocation at 41,747,087,360 bytes (M4 Max). The guest memory spans the game binds as
storage and texel buffers can be larger, so the macOS port imports such a span as several allocations (segments), and a
descriptor whose range crosses a cut cannot be one buffer. This PR is the shader half of binding it anyway: the SPIR-V back
end can declare such a descriptor as an array of pieces, one per segment, at its own binding, and every access finds its
piece. It also carries the fix for the one pattern of it MoltenVK refused: a segmented program with a compare-exchange
loop (an increment, a decrement, or a float add the GPU has no atomic for) failed pipeline creation with "no matching
function for call". Nothing is drawn with pieces until the draw side (#663) asks for a segmented variant, so on its own
this PR changes no frame on any platform. It is part of the macOS port. See #420.

What changed

The back end learns RuntimeInfo::segmented: a list of descriptors to bind as pieces. With the list empty, which is every
program today, it emits byte for byte what it did. With it, the descriptor is an array of pieces at its binding, and each
load, store, atomic, texel fetch and size query goes through a small helper function that walks the pieces by their lengths
with constant indices. The renderer keeps such a variant apart from the plain program (never cached on disk, never picked for
a plain draw). The second commit writes the compare-exchange loops out inside those helpers, because SPIRV-Cross's MSL cannot
pass a pointer to one piece of a buffer array to a function.

  • back end, gpu/vk/eden/shader_recompiler/runtime_info.h: SegEntry {Storage|Table|TexBuf|ImgBuf, desc, pieces} and
    RuntimeInfo::segmented.
  • back end, backend/spirv/spirv_emit_context.{h,cpp}: DefineSegmentedDescriptors checks the list against the program
    (refused with LogicError otherwise); DefineSsbos, DefineTextureBuffers and DefineImageBuffers declare a flagged
    descriptor as TypeArray(…, pieces) at the same binding (a table's count * pieces, entry k's piece j at
    k * pieces + j); SegmentedStorageCall, SegmentedTextureBufferCall, SegmentedImageBufferCall and
    SegmentedTextureBufferSize ask for one DontInline helper per (view, op, result type), made after main by
    DefineSegmentedHelpers and named in the calls by PatchSegmentedCalls; DefineGlobalMemoryFunctions sums a segmented
    descriptor's piece lengths for its bound end and branches to the piece (single, native candidate, table written out and in
    the loop).
  • back end, emit_spirv_memory.cpp, emit_spirv_atomic.cpp, emit_spirv_image.cpp, emit_spirv_image_atomic.cpp: the
    storage loads, stores and atomics (CAS loops, float adds, the 64-bit and 32x2 fallbacks), texel fetches, image buffer
    reads, writes and atomics dispatch to the helpers for a flagged descriptor; emit_spirv.cpp calls the two new steps,
    which do nothing without helpers.
  • back end, the MslArrayPtr fix (second commit): EmitContext::CasWordHere, CasLoopHere and CasStoreHere write
    CasLoop's loop (and write_storage_cas_loop_func's) out in the piece, attempt for attempt, with the same
    CAS_LOOP_MAX_TRIES ceiling; CasLoop records each loop's operation (cas_operations); SegmentedCasLoop and
    WriteStorageByCasLoop use them, and DefineSegmentedHelpers makes the loop's two variables first in the helper.
  • renderer, gpu/vk/vk_shader.{h,cpp}: SegSig (sorted entries and a hash) on Program, seg_sig(), seg_variant(),
    program_segmented() (finds or translates the variant of a plain program with plain's questions and answers, and fails
    it loudly if its descriptors are not plain's); program_pick, mesh_program_locked, program_find_plain,
    loaded_variants and portable_variant keep the variants apart; every stage translation sets RuntimeInfo::segmented
    from Program::seg (empty in every plain variant).
  • renderer, gpu/vk/vk_progcache.cpp: a segmented variant is never written to the program cache nor listed among the
    loaded programs.
  • tests: tests/test_seg_shader.py and tests/seg_shader_ir.cpp (new) run synthetic IR programs through the real back
    end, plain and in pieces, check the segmented modules' structure and capabilities, run every module through spirv-val
    (Vulkan 1.3, uniform buffer standard layout, as CI validates the back end), check that no module passes a pointer into a
    buffer descriptor array to a function, and with --base-tree compare the plain modules byte for byte with an older
    tree's. Seven source-reading checks follow the new field, the atomics' new branch and the new loop:
    test_native_float_atomic_add, test_pipeline_state_portable, test_program_cache_cleanup,
    test_program_cache_one_variant, test_seed_kept_loaded, test_seed_variant_cleanup, test_cas_loop_ceiling.

Measurements

Verdict: passes. The plain SPIR-V is byte for byte main's on both CPU routes, every module validates, and the array
pointer check fails without the fix (3 segmented modules, 54 calls) and passes with it. No speed to measure: nothing is
drawn with pieces until #663, and the plain path's SPIR-V is unchanged.

Hardware: macOS 26.7.1, Apple M4 Max, 64 GB, Apple clang 21 (game-free: no GPU, no game)
Scene: none (an emitter change: its proof is the fixture's modules)
Method: tests/test_seg_shader.py at this head with --base-tree a git archive of main 6c9edffe's back end, on arm64
and with --arch x86_64 (compiled for Intel, run under Rosetta); the same test file against the first commit's back end
(cbcdb34d, the pieces without the fix). Every output is a byte comparison or a hash, so one run per route and side; the
two routes' 29 modules have equal SHA-256.
How to reproduce: python -I -B tests/test_seg_shader.py --headers <SPIRV-Headers/include> --spirv-val spirv-val --base-tree <git archive of main> [--arch x86_64]
Baseline: the middle column is this PR's first commit alone, the pieces before the fix; main has no segmented module to
check.

Before: main @ 6c9edffe After, first commit only (no fix): this PR @ cbcdb34d After: this PR @ 9d48901a
plain modules, arm64 and x86_64 14 emitted (the base tree) — 14 of 14 byte-identical to main's on both routes
modules passing spirv-val — 29 of 29 29 of 29, both routes
modules passing a pointer into a buffer array to a function — 3 of 29 (seg-storage 18 calls, seg-storage-native-fadd 15, seg-storage-narrow 21): exit 1 0 of 29: exit 0
arm E (SegProbe's module) SHA-256 — 01a0f43f… 01a0f43f…, as SegProbe pinned
this PR @ 9d48901a, arm64 (x86_64 the same lines):
array pointers: 29 of 29 modules pass no pointer into a buffer descriptor array to a function
base tree …/base-6c9edffe: 14 of 14 plain modules byte-identical
PASS: segmented descriptors through the real back end; every module validates
this PR's test against cbcdb34d's back end, arm64:
seg-storage-narrow.spv: 21 calls pass a pointer into a buffer array: %105(ssbo0), %132(ssbo0), %162(ssbo0), %191(ssbo0), %219(ssbo0), %240(ssbo0), %74(ssbo0)
seg-storage-native-fadd.spv: 15 calls pass a pointer into a buffer array: %118(ssbo0), %149(ssbo0), %176(ssbo0), %205(ssbo0), %233(ssbo0)
seg-storage.spv: 18 calls pass a pointer into a buffer array: %110(ssbo0), %141(ssbo0), %168(ssbo0), %198(ssbo0), %227(ssbo0), %255(ssbo0)
array pointers: 26 of 29 modules pass no pointer into a buffer descriptor array to a function

History, GPU (MoltenVK 1.4.2, M4 Max, arm64 and x86_64 under Rosetta), and why it holds for this head:

  • SegProbe revision 02 ran this emitter's arm E module (a 4-piece storage buffer, read and written through the helpers):
    E.read_g1_layout and E.write_g2_layout PASS on both routes, 0 [mvk-error] (reports/2026-11-14-seg-probe.md
    section 3d). This head emits the same module (SHA-256 01a0f43f7da317c61ab1ee0dab18448d8dcafb798a1fcabdf5d08a0db8b69cc0).
  • MslProbe revision 02 ran the fix's modules (reports/2026-11-14-msl-arrayptr.md sections 11 and 12): every pipeline was
    made, and the written-out loops (increment, decrement, float add, f32x2 add/min/max, the narrow profile's 8- and 16-bit
    stores) gave the CPU interpreter's bytes in every piece; ma-cas-seg, ma-cas-narrow-seg, their whole controls and the
    table program mc-table MATCH on both routes. Those five modules were built from the NullShader branch's back end; this
    head's back end emits the same five programs byte for byte (reports/PubWave-files/PubShader/compare-mslprobe.txt), so
    the result is this head's.
  • Without the fix, that branch's segmented program NS2 failed pipeline creation on MoltenVK with no matching function for call, &ssbo0_4[0u] given to a device _66 * parameter (NullProbe revision 02; the cause in SPIRV-Cross's source is in
    reports/2026-11-14-msl-arrayptr.md section 1).

Flags and settings

None.

Testing

Tested: on macOS 26.7.1, Apple M4 Max, at 9d48901a, this PR's head before the rebases onto e090cc48 and aa95d35f
(main 6c9edffe plus this PR), game-free:

  • tests/test_seg_shader.py --base-tree <main 6c9edffe> on arm64 and x86_64 (Rosetta): exit 0 both, as in Measurements;
    the 29 modules of the two routes are byte-identical, and byte-identical to the macOS port's integration tree after the
    fix merged (7cd720da, both routes: every plain and segmented module and arm E). Without the fix (the first commit's back
    end) the three segmented storage modules are the topic's original bytes (b7928876 on a4802cb5) and fail the array
    pointer check.
  • The five MslProbe programs (build_cas_run whole and in 3 pieces, both profiles, and build_table_run, copied from the
    NullShader branch's fixture into a scratch copy of this head's, bit off) through this head's back end: byte-identical to
    MslProbe revision 02's frozen GPU inputs; all validate and pass the array pointer check.
  • Renderer files that include vk_shader.h (vk_shader.cpp, vk_progcache.cpp, vk_draw.cpp, vk_predict.cpp,
    vk_replay.cpp, vk_programs.cpp) compiled for arm64 and x86_64 at this head and on main 6c9edffe with main's renderer
    flags (nx2/build.py's vflags plus -arch, and main's non-Windows -fPIC -I host/posix -D_GNU_SOURCE). Every tree got
    #492's two Darwin fixes (environ, RUSAGE_THREAD) and a compile-only prelude declaring posix_fallocate and
    posix_fadvise, which main's renderer calls and macOS lacks (#476 gives them Darwin paths): 12 of 12 objects on each,
    exit 0.
  • The seven source-reading checks this PR updates (tools/ls-suite.py named run): 5 pass. test_program_cache_cleanup and
    test_program_cache_one_variant cannot compile their fixture on this Mac, exactly as on main 6c9edffe (main's
    vk_progcache.cpp uses POSIX_FADV_WILLNEED; main's own macOS suite fails both the same way). With #476's two Darwin
    hunks (vk_progcache.cpp, pipeline_cache_file.h) applied to scratch copies of main and this head, both pass on both
    (main 9 tests OK and the same variant for 9 picks; this head the same).
  • Logs: reports/PubWave-files/PubShader/proof/p20-*.log, proof/fadv-*.log, compare-p20.txt, compare-mslprobe.txt,
    compile-*.out, named/, merge-tree.txt, stamps.txt.
  • After the rebase onto main aa95d35f, at 9b73e829 (9d48901a before the rebases onto e090cc48 and aa95d35f):
    Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they
    apply) gave the same results as before; python -I -B tests/run_all.py on wave B's merge db0e2e16, which carries
    this head: 569/32/18 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure
    (reports/2026-11-15-rebase-sweep-3.md).

Not tested: the game at this head: per-PR game runs are not planned, and nothing here runs before #663 asks for a
segmented variant. The GPU at this head: the probes above are history whose modules this head reproduces byte for byte. The
f16x2 compare-exchange loops have no fixture program (the CPU interpreter has no 16-bit float vectors); they share the f32x2
loops' code path. Windows and Linux: not built or run there (11 changed renderer and back-end files differ from main's off
Apple, unifdef -U__APPLE__); its test request covers the build, run_all on Linux and a start.

Must be tested:

  • Linux and Windows, any GPU (Linux runs the SPIR-V test): build this PR (head 9b73e829) and main aa95d35f the same way
    (the 11 every-platform files: the back end's emit_spirv.cpp, emit_spirv_atomic.cpp, emit_spirv_image.cpp,
    emit_spirv_image_atomic.cpp, emit_spirv_memory.cpp, spirv_emit_context.{h,cpp}, runtime_info.h;
    vk_shader.{h,cpp}, vk_progcache.cpp); on Linux run python -I -B tests/run_all.py, expecting
    tests/test_seg_shader.py to pass (it needs spirv-val and clang++; Windows skips it); start each build to the
    title, expecting this PR to translate the shader caches again once (the stamp changed) and nothing else to differ from
    main, in the log or on screen (by hand: test request T-WLDY)

Risks

What could break: nothing runs differently until #663 calls program_segmented: every program's
RuntimeInfo::segmented is empty, and with it empty the back end's output is byte for byte main's (checked on 14 programs,
both routes). Program grows by a SegSig (an empty vector and a hash). Once #663 uses it on macOS: a segmented variant is
translated on the drawing thread under shader_mu at its first draw, and one whose descriptors differ from plain's is failed
and logged, never drawn with pieces.

Turning it off: revert the commits (there is no switch: nothing asks for pieces before #663).

Compatibility: every platform's shader caches are translated again once: eden/ and vk_shader.cpp are in the
translator's stamp (nx2/translator_files.py: ebbf160ce9626041 on main, 40b414b30dad80e1 here), so each player's
program cache, prepare record and pipeline cache are dropped at the first start after updating. kProgramCacheFormat is
unchanged here (#649 raises it). Nothing changes in saves, longshot_options.ini or the vk ABI (MKWVK_API_VERSION).

Reviewer notes

  • Review cbcdb34d..9d48901a: this PR's two commits, cbcdb34d (the pieces, -x from macos/seg-shader b7928876)
    and 9d48901a (the MslArrayPtr fix, -x from macos/msl-arrayptr bd396010). Read runtime_info.h, then
    DefineSegmentedHelpers and SegmentedStorageCall in spirv_emit_context.cpp, then program_segmented in
    vk_shader.cpp; then CasWordHere/CasLoopHere.
  • One PR for two topics (owner decision): the pieces and their compare-exchange fix ship together, so no state of main has
    segmented atomics that MoltenVK refuses.
  • How the fix was ported: bd396010 was committed on the NullShader branch, where a storage descriptor bound whole can also
    reach the segmented helpers (StorageThroughHelper). Here every access that reaches a helper is in pieces, so the loops are
    always written out; the bound-whole branch, the guarded helper's cas_loop, and the tests that need NullShader or
    tests/spirv_run.h (the loops' and a table's CPU runs, --changed-segmented, the *.cpurun comparison) stay for #674.
    The cherry-pick's conflicts (emit_spirv_memory.cpp, spirv_emit_context.{h,cpp}, seg_shader_ir.cpp,
    test_seg_shader.py, spirv_run.h absent here) were all that difference.
  • Stacking: on main 6c9edffe, no dependency. #649 stacks on it; the draw side (#662, #663, #663) and #674 use it.
  • Conflicts with main: none in the cherry-picks. No semantic rework: main changed vk_shader.cpp (#252 make_module's
    Intel gate and driver_faulted_program in translate_into, which program_segmented reaches through translate_into,
    so a program the driver faulted on is not asked again as a segmented variant either), vk_progcache.cpp (#321's paths,
    #252) and test_native_float_atomic_add.py (#191, #444) beside this PR's lines. #370 moved the stamp's file list to
    nx2/translator_files.py and took vk_api.h out of it: the first commit's message still says to land it with a
    vk_api.h change; the stamp changes here either way.
  • With the open macOS PRs (#431, #437, #438, #479, #480, #490, #491, #492) and this wave's branches on main 6c9edffe:
    none (git merge-tree).
  • Code comments cite reports/2026-11-14-*.md (the port's design notes and the MslArrayPtr report), which are not in this
    repository.

Issues

See #420

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how. (Test request T-WLDY.)
  • Rebased on current main: on aa95d35f (#761, 2026-11-15), head 9b73e829, without a conflict; this PR's own
    commits, old → new: cbcdb34d → 1f9183c2, 9d48901a → 9b73e829; the commit names elsewhere in this description
    are from before the rebase.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). (On the wave's merge of every macOS PR built with this one, 453eb678: 553 passed, 32 failed, 18 skipped against main 6c9edffe's 531/32/17 on this Mac; the failures are main's own except test_split_ub_flush, which #658's tree at the time broke and its head fixes; test_collect_wide now passes; the new tests pass, test_seg_shader among them, and test_darwin_audio_service, Darwin-only, is skipped by the runner; reports/PubWave-files/waveA/. After the rebase onto main aa95d35f (head 9b73e829): wave B's merge db0e2e16, 569/32/18 against main aa95d35f's 543/33/17, no new failure.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here. (It fails on this PR's first commit, the
    pieces without the fix; main has no segmented module.)
  • A check under tests/ that reads changed source updated in this PR, not weakened.
**Type:** feat · **Risk:** low · **Tested on:** macOS 26.7.1, Apple M4 Max (MoltenVK 1.4.2), game-free at this head; GPU probes on both CPU routes (history) · **Needs:** Windows, Linux ## Why On a Mac, MoltenVK caps one device allocation at 41,747,087,360 bytes (M4 Max). The guest memory spans the game binds as storage and texel buffers can be larger, so the macOS port imports such a span as several allocations (segments), and a descriptor whose range crosses a cut cannot be one buffer. This PR is the shader half of binding it anyway: the SPIR-V back end can declare such a descriptor as an array of pieces, one per segment, at its own binding, and every access finds its piece. It also carries the fix for the one pattern of it MoltenVK refused: a segmented program with a compare-exchange loop (an increment, a decrement, or a float add the GPU has no atomic for) failed pipeline creation with "no matching function for call". Nothing is drawn with pieces until the draw side (#663) asks for a segmented variant, so on its own this PR changes no frame on any platform. It is part of the macOS port. See #420. ## What changed The back end learns `RuntimeInfo::segmented`: a list of descriptors to bind as pieces. With the list empty, which is every program today, it emits byte for byte what it did. With it, the descriptor is an array of pieces at its binding, and each load, store, atomic, texel fetch and size query goes through a small helper function that walks the pieces by their lengths with constant indices. The renderer keeps such a variant apart from the plain program (never cached on disk, never picked for a plain draw). The second commit writes the compare-exchange loops out inside those helpers, because SPIRV-Cross's MSL cannot pass a pointer to one piece of a buffer array to a function. - back end, `gpu/vk/eden/shader_recompiler/runtime_info.h`: `SegEntry {Storage|Table|TexBuf|ImgBuf, desc, pieces}` and `RuntimeInfo::segmented`. - back end, `backend/spirv/spirv_emit_context.{h,cpp}`: `DefineSegmentedDescriptors` checks the list against the program (refused with `LogicError` otherwise); `DefineSsbos`, `DefineTextureBuffers` and `DefineImageBuffers` declare a flagged descriptor as `TypeArray(…, pieces)` at the same binding (a table's `count * pieces`, entry k's piece j at `k * pieces + j`); `SegmentedStorageCall`, `SegmentedTextureBufferCall`, `SegmentedImageBufferCall` and `SegmentedTextureBufferSize` ask for one DontInline helper per (view, op, result type), made after main by `DefineSegmentedHelpers` and named in the calls by `PatchSegmentedCalls`; `DefineGlobalMemoryFunctions` sums a segmented descriptor's piece lengths for its bound end and branches to the piece (single, native candidate, table written out and in the loop). - back end, `emit_spirv_memory.cpp`, `emit_spirv_atomic.cpp`, `emit_spirv_image.cpp`, `emit_spirv_image_atomic.cpp`: the storage loads, stores and atomics (CAS loops, float adds, the 64-bit and 32x2 fallbacks), texel fetches, image buffer reads, writes and atomics dispatch to the helpers for a flagged descriptor; `emit_spirv.cpp` calls the two new steps, which do nothing without helpers. - back end, the MslArrayPtr fix (second commit): `EmitContext::CasWordHere`, `CasLoopHere` and `CasStoreHere` write CasLoop's loop (and `write_storage_cas_loop_func`'s) out in the piece, attempt for attempt, with the same `CAS_LOOP_MAX_TRIES` ceiling; `CasLoop` records each loop's operation (`cas_operations`); `SegmentedCasLoop` and `WriteStorageByCasLoop` use them, and `DefineSegmentedHelpers` makes the loop's two variables first in the helper. - renderer, `gpu/vk/vk_shader.{h,cpp}`: `SegSig` (sorted entries and a hash) on `Program`, `seg_sig()`, `seg_variant()`, `program_segmented()` (finds or translates the variant of a plain program with plain's questions and answers, and fails it loudly if its descriptors are not plain's); `program_pick`, `mesh_program_locked`, `program_find_plain`, `loaded_variants` and `portable_variant` keep the variants apart; every stage translation sets `RuntimeInfo::segmented` from `Program::seg` (empty in every plain variant). - renderer, `gpu/vk/vk_progcache.cpp`: a segmented variant is never written to the program cache nor listed among the loaded programs. - tests: `tests/test_seg_shader.py` and `tests/seg_shader_ir.cpp` (new) run synthetic IR programs through the real back end, plain and in pieces, check the segmented modules' structure and capabilities, run every module through `spirv-val` (Vulkan 1.3, uniform buffer standard layout, as CI validates the back end), check that no module passes a pointer into a buffer descriptor array to a function, and with `--base-tree` compare the plain modules byte for byte with an older tree's. Seven source-reading checks follow the new field, the atomics' new branch and the new loop: `test_native_float_atomic_add`, `test_pipeline_state_portable`, `test_program_cache_cleanup`, `test_program_cache_one_variant`, `test_seed_kept_loaded`, `test_seed_variant_cleanup`, `test_cas_loop_ceiling`. ## Measurements **Verdict:** passes. The plain SPIR-V is byte for byte main's on both CPU routes, every module validates, and the array pointer check fails without the fix (3 segmented modules, 54 calls) and passes with it. No speed to measure: nothing is drawn with pieces until #663, and the plain path's SPIR-V is unchanged. Hardware: macOS 26.7.1, Apple M4 Max, 64 GB, Apple clang 21 (game-free: no GPU, no game) Scene: none (an emitter change: its proof is the fixture's modules) Method: `tests/test_seg_shader.py` at this head with `--base-tree` a `git archive` of main 6c9edffe's back end, on arm64 and with `--arch x86_64` (compiled for Intel, run under Rosetta); the same test file against the first commit's back end (`cbcdb34d`, the pieces without the fix). Every output is a byte comparison or a hash, so one run per route and side; the two routes' 29 modules have equal SHA-256. How to reproduce: `python -I -B tests/test_seg_shader.py --headers <SPIRV-Headers/include> --spirv-val spirv-val --base-tree <git archive of main> [--arch x86_64]` Baseline: the middle column is this PR's first commit alone, the pieces before the fix; main has no segmented module to check. | | Before: main @ 6c9edffe | After, first commit only (no fix): this PR @ cbcdb34d | After: this PR @ 9d48901a | |---|---|---|---| | plain modules, arm64 and x86_64 | 14 emitted (the base tree) | — | 14 of 14 byte-identical to main's on both routes | | modules passing `spirv-val` | — | 29 of 29 | 29 of 29, both routes | | modules passing a pointer into a buffer array to a function | — | 3 of 29 (`seg-storage` 18 calls, `seg-storage-native-fadd` 15, `seg-storage-narrow` 21): exit 1 | 0 of 29: exit 0 | | arm E (SegProbe's module) SHA-256 | — | `01a0f43f…` | `01a0f43f…`, as SegProbe pinned | ``` this PR @ 9d48901a, arm64 (x86_64 the same lines): array pointers: 29 of 29 modules pass no pointer into a buffer descriptor array to a function base tree …/base-6c9edffe: 14 of 14 plain modules byte-identical PASS: segmented descriptors through the real back end; every module validates this PR's test against cbcdb34d's back end, arm64: seg-storage-narrow.spv: 21 calls pass a pointer into a buffer array: %105(ssbo0), %132(ssbo0), %162(ssbo0), %191(ssbo0), %219(ssbo0), %240(ssbo0), %74(ssbo0) seg-storage-native-fadd.spv: 15 calls pass a pointer into a buffer array: %118(ssbo0), %149(ssbo0), %176(ssbo0), %205(ssbo0), %233(ssbo0) seg-storage.spv: 18 calls pass a pointer into a buffer array: %110(ssbo0), %141(ssbo0), %168(ssbo0), %198(ssbo0), %227(ssbo0), %255(ssbo0) array pointers: 26 of 29 modules pass no pointer into a buffer descriptor array to a function ``` History, GPU (MoltenVK 1.4.2, M4 Max, arm64 and x86_64 under Rosetta), and why it holds for this head: - SegProbe revision 02 ran this emitter's arm E module (a 4-piece storage buffer, read and written through the helpers): `E.read_g1_layout` and `E.write_g2_layout` PASS on both routes, 0 `[mvk-error]` (`reports/2026-11-14-seg-probe.md` section 3d). This head emits the same module (SHA-256 `01a0f43f7da317c61ab1ee0dab18448d8dcafb798a1fcabdf5d08a0db8b69cc0`). - MslProbe revision 02 ran the fix's modules (`reports/2026-11-14-msl-arrayptr.md` sections 11 and 12): every pipeline was made, and the written-out loops (increment, decrement, float add, f32x2 add/min/max, the narrow profile's 8- and 16-bit stores) gave the CPU interpreter's bytes in every piece; `ma-cas-seg`, `ma-cas-narrow-seg`, their whole controls and the table program `mc-table` MATCH on both routes. Those five modules were built from the NullShader branch's back end; this head's back end emits the same five programs byte for byte (`reports/PubWave-files/PubShader/compare-mslprobe.txt`), so the result is this head's. - Without the fix, that branch's segmented program NS2 failed pipeline creation on MoltenVK with `no matching function for call`, `&ssbo0_4[0u]` given to a `device _66 *` parameter (NullProbe revision 02; the cause in SPIRV-Cross's source is in `reports/2026-11-14-msl-arrayptr.md` section 1). ### Flags and settings None. ## Testing **Tested:** on macOS 26.7.1, Apple M4 Max, at `9d48901a`, this PR's head before the rebases onto e090cc48 and aa95d35f (main 6c9edffe plus this PR), game-free: - `tests/test_seg_shader.py --base-tree <main 6c9edffe>` on arm64 and x86_64 (Rosetta): exit 0 both, as in Measurements; the 29 modules of the two routes are byte-identical, and byte-identical to the macOS port's integration tree after the fix merged (`7cd720da`, both routes: every plain and segmented module and arm E). Without the fix (the first commit's back end) the three segmented storage modules are the topic's original bytes (`b7928876` on `a4802cb5`) and fail the array pointer check. - The five MslProbe programs (`build_cas_run` whole and in 3 pieces, both profiles, and `build_table_run`, copied from the NullShader branch's fixture into a scratch copy of this head's, bit off) through this head's back end: byte-identical to MslProbe revision 02's frozen GPU inputs; all validate and pass the array pointer check. - Renderer files that include `vk_shader.h` (`vk_shader.cpp`, `vk_progcache.cpp`, `vk_draw.cpp`, `vk_predict.cpp`, `vk_replay.cpp`, `vk_programs.cpp`) compiled for arm64 and x86_64 at this head and on main 6c9edffe with main's renderer flags (`nx2/build.py`'s `vflags` plus `-arch`, and main's non-Windows `-fPIC -I host/posix -D_GNU_SOURCE`). Every tree got #492's two Darwin fixes (`environ`, `RUSAGE_THREAD`) and a compile-only prelude declaring `posix_fallocate` and `posix_fadvise`, which main's renderer calls and macOS lacks (#476 gives them Darwin paths): 12 of 12 objects on each, exit 0. - The seven source-reading checks this PR updates (`tools/ls-suite.py` named run): 5 pass. `test_program_cache_cleanup` and `test_program_cache_one_variant` cannot compile their fixture on this Mac, exactly as on main 6c9edffe (main's `vk_progcache.cpp` uses `POSIX_FADV_WILLNEED`; main's own macOS suite fails both the same way). With #476's two Darwin hunks (`vk_progcache.cpp`, `pipeline_cache_file.h`) applied to scratch copies of main and this head, both pass on both (main 9 tests OK and the same variant for 9 picks; this head the same). - Logs: `reports/PubWave-files/PubShader/proof/p20-*.log`, `proof/fadv-*.log`, `compare-p20.txt`, `compare-mslprobe.txt`, `compile-*.out`, `named/`, `merge-tree.txt`, `stamps.txt`. - **After the rebase onto main aa95d35f**, at `9b73e829` (`9d48901a` before the rebases onto e090cc48 and aa95d35f): Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they apply) gave the same results as before; `python -I -B tests/run_all.py` on wave B's merge `db0e2e16`, which carries this head: 569/32/18 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure (`reports/2026-11-15-rebase-sweep-3.md`). **Not tested:** the game at this head: per-PR game runs are not planned, and nothing here runs before #663 asks for a segmented variant. The GPU at this head: the probes above are history whose modules this head reproduces byte for byte. The f16x2 compare-exchange loops have no fixture program (the CPU interpreter has no 16-bit float vectors); they share the f32x2 loops' code path. Windows and Linux: not built or run there (11 changed renderer and back-end files differ from main's off Apple, `unifdef -U__APPLE__`); its test request covers the build, run_all on Linux and a start. **Must be tested:** - Linux and Windows, any GPU (Linux runs the SPIR-V test): build this PR (head 9b73e829) and main aa95d35f the same way (the 11 every-platform files: the back end's `emit_spirv.cpp`, `emit_spirv_atomic.cpp`, `emit_spirv_image.cpp`, `emit_spirv_image_atomic.cpp`, `emit_spirv_memory.cpp`, `spirv_emit_context.{h,cpp}`, `runtime_info.h`; `vk_shader.{h,cpp}`, `vk_progcache.cpp`); on Linux run `python -I -B tests/run_all.py`, expecting `tests/test_seg_shader.py` to pass (it needs `spirv-val` and clang++; Windows skips it); start each build to the title, expecting this PR to translate the shader caches again once (the stamp changed) and nothing else to differ from main, in the log or on screen (by hand: test request T-WLDY) ## Risks **What could break:** nothing runs differently until #663 calls `program_segmented`: every program's `RuntimeInfo::segmented` is empty, and with it empty the back end's output is byte for byte main's (checked on 14 programs, both routes). `Program` grows by a `SegSig` (an empty vector and a hash). Once #663 uses it on macOS: a segmented variant is translated on the drawing thread under `shader_mu` at its first draw, and one whose descriptors differ from plain's is failed and logged, never drawn with pieces. **Turning it off:** revert the commits (there is no switch: nothing asks for pieces before #663). **Compatibility:** every platform's shader caches are translated again once: `eden/` and `vk_shader.cpp` are in the translator's stamp (`nx2/translator_files.py`: `ebbf160ce9626041` on main, `40b414b30dad80e1` here), so each player's program cache, prepare record and pipeline cache are dropped at the first start after updating. `kProgramCacheFormat` is unchanged here (#649 raises it). Nothing changes in saves, `longshot_options.ini` or the vk ABI (`MKWVK_API_VERSION`). ## Reviewer notes - Review `cbcdb34d..9d48901a`: this PR's two commits, `cbcdb34d` (the pieces, `-x` from `macos/seg-shader` `b7928876`) and `9d48901a` (the MslArrayPtr fix, `-x` from `macos/msl-arrayptr` `bd396010`). Read `runtime_info.h`, then `DefineSegmentedHelpers` and `SegmentedStorageCall` in `spirv_emit_context.cpp`, then `program_segmented` in `vk_shader.cpp`; then `CasWordHere`/`CasLoopHere`. - One PR for two topics (owner decision): the pieces and their compare-exchange fix ship together, so no state of `main` has segmented atomics that MoltenVK refuses. - How the fix was ported: `bd396010` was committed on the NullShader branch, where a storage descriptor bound whole can also reach the segmented helpers (`StorageThroughHelper`). Here every access that reaches a helper is in pieces, so the loops are always written out; the bound-whole branch, the guarded helper's `cas_loop`, and the tests that need NullShader or `tests/spirv_run.h` (the loops' and a table's CPU runs, `--changed-segmented`, the `*.cpurun` comparison) stay for #674. The cherry-pick's conflicts (`emit_spirv_memory.cpp`, `spirv_emit_context.{h,cpp}`, `seg_shader_ir.cpp`, `test_seg_shader.py`, `spirv_run.h` absent here) were all that difference. - Stacking: on main 6c9edffe, no dependency. #649 stacks on it; the draw side (#662, #663, #663) and #674 use it. - Conflicts with main: none in the cherry-picks. No semantic rework: main changed `vk_shader.cpp` (#252 `make_module`'s Intel gate and `driver_faulted_program` in `translate_into`, which `program_segmented` reaches through `translate_into`, so a program the driver faulted on is not asked again as a segmented variant either), `vk_progcache.cpp` (#321's paths, #252) and `test_native_float_atomic_add.py` (#191, #444) beside this PR's lines. #370 moved the stamp's file list to `nx2/translator_files.py` and took `vk_api.h` out of it: the first commit's message still says to land it with a `vk_api.h` change; the stamp changes here either way. - With the open macOS PRs (#431, #437, #438, #479, #480, #490, #491, #492) and this wave's branches on main 6c9edffe: none (`git merge-tree`). - Code comments cite `reports/2026-11-14-*.md` (the port's design notes and the MslArrayPtr report), which are not in this repository. ## Issues See #420 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. (Test request T-WLDY.) - [x] Rebased on current `main`: on aa95d35f (#761, 2026-11-15), head `9b73e829`, without a conflict; this PR's own commits, old → new: `cbcdb34d` → `1f9183c2`, `9d48901a` → `9b73e829`; the commit names elsewhere in this description are from before the rebase. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (On the wave's merge of every macOS PR built with this one, `453eb678`: 553 passed, 32 failed, 18 skipped against main 6c9edffe's 531/32/17 on this Mac; the failures are main's own except `test_split_ub_flush`, which #658's tree at the time broke and its head fixes; `test_collect_wide` now passes; the new tests pass, `test_seg_shader` among them, and `test_darwin_audio_service`, Darwin-only, is skipped by the runner; `reports/PubWave-files/waveA/`. After the rebase onto main aa95d35f (head `9b73e829`): wave B's merge `db0e2e16`, 569/32/18 against main aa95d35f's 543/33/17, no new failure.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. (It fails on this PR's first commit, the pieces without the fix; `main` has no segmented module.) - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.
A guest span larger than one device allocation (the cap on MoltenVK/M4 Max is 41,747,087,360 bytes) is bound as
pieces: one descriptor per allocation segment, at the descriptor's own binding (reports/2026-11-14-full-range-design.md
sections 4.1-4.4). This is the shader half; the draw side (descriptors, layout, gathers) comes later.

- runtime_info.h: SegEntry {Storage|Table|TexBuf|ImgBuf, desc, pieces} and RuntimeInfo::segmented.
- Back end: a segmented storage buffer is an array of pieces at its binding (a table count * pieces, entry k's piece j
  at k * pieces + j); its loads, stores and atomics call DontInline helpers made after main, which find the piece by
  the pieces' OpArrayLength in order, with constant indices. The global memory fallbacks sum the pieces' lengths for a
  segmented descriptor's bound end and branch to its piece (single, native candidate, table written out and in the
  loop). Texel and image buffers are arrays too, their fetches, reads, writes and atomics through helpers that pick
  the piece by OpImageQuerySize, and a texel buffer's size query sums the pieces. ImageQuery was declared for every
  module already. With RuntimeInfo::segmented empty nothing new is made: the SPIR-V is byte-identical.
- vk_shader: SegSig (entries, hash) on Program, seg_sig and seg_variant; program_segmented finds or translates the
  variant of a plain program with plain's texture questions and answers and checks its descriptors are plain's;
  program_pick, mesh_program_locked, program_find_plain, loaded_variants and portable_variant keep the plain and
  segmented variants apart; every stage's RuntimeInfo::segmented comes from Program::seg.
- vk_progcache: a segmented variant is never written to the program cache nor listed among the loaded programs.
- tests/test_seg_shader.py + tests/seg_shader_ir.cpp: synthetic IR programs through the real back end, plain and
  segmented; structure checks, spirv-val (Vulkan 1.3, uniform buffer standard layout), and with --base-tree a byte
  comparison of the plain modules against an older tree. Five source-reading checks follow the new Program field and
  the atomics' new branch.

The translator stamp changes with this (eden/, vk_shader.cpp, sizeof(Program)): every program cache is rebuilt once.
Land it in the same release as the vk_api.h change that already does so.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit b7928876dab846485f667c1c934d34d25ecef035)
fix(vk): MslArrayPtr: no pointer into a storage buffer array is passed to a function
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
Third-party parts / As their notes say (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 4s
9d48901abf
On MoltenVK a segmented storage program with a compare-exchange loop (an
increment, decrement or float add the device does not have) failed pipeline
creation: NullProbe revision 02's NS2, "no matching function for call",
&ssbo0_4[0u] given to a device _66 * parameter. SPIRV-Cross's MSL (6c09849f,
MoltenVK 1.4.2) passes a storage buffer pointer to a function by value, and
takes the address of the argument's expression: for an element of a buffer
array, which MSL holds as an array of pointers, that is the address of the
pointer (reports/2026-11-14-msl-arrayptr.md).

A descriptor bound as pieces now has its loops written out in the piece,
inside the segmented helper (CasLoopHere, CasStoreHere: CasLoop's loop,
attempt for attempt, its variables made first in the helper), instead of
calling the CasLoop function or write_storage_cas_loop_func with a pointer
into the pieces' array. Every other program is as before byte for byte.

Tests: test_seg_shader.py checks that no module passes a pointer into a
buffer descriptor array to a function. The two checks that cut the back
end's source follow the new loop and parameter.

Ported from macos/msl-arrayptr, where it was committed on NullShader, to the
pieces alone: here every storage access that reaches a segmented helper is
in pieces, so the loops are always written out. Left for the NullShader
change: the bound-whole branch that still calls the function with its
block, the guarded helper's cas_loop, and the tests that need NullShader or
tests/spirv_run.h (the CPU runs of the loops and of a storage buffer table,
--changed-segmented and the *.cpurun comparison).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit bd396010887afab9520181009db1b56a4cc425e3)
mooki changed title from WIP: feat(vk): bind storage and texel descriptors as piece arrays in the SPIR-V back end to feat(vk): bind storage and texel descriptors as piece arrays in the SPIR-V back end 2026-11-15 05:25:32 +00:00
mooki force-pushed mac/seg-shader from 9d48901abf
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 4s
Third-party parts / As their notes say (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 4s
to 407a0cfc93
All checks were successful
PR description / Description follows the template (pull_request_target) Successful in 5s
2026-11-15 07:16:33 +00:00
Compare
mooki force-pushed mac/seg-shader from 407a0cfc93
All checks were successful
PR description / Description follows the template (pull_request_target) Successful in 5s
to 9b73e82942
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s
Third-party parts / As their notes say (pull_request) Successful in 11s
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
2026-11-15 13:10:01 +00:00
Compare
Author
Member

Folded into #848 (feat(vk): the SPIR-V back end on MoltenVK and guest spans past one device allocation)

Folded into #848 (feat(vk): the SPIR-V back end on MoltenVK and guest spans past one device allocation)
mooki closed this pull request 2026-11-15 18:07:52 +00:00
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s
Third-party parts / As their notes say (pull_request) Successful in 11s
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!648
No description provided.