MagicBeanDev/Longshot

Watch

Fork 99

WIP: fix(vk): MoltenVK's descriptor limits in descriptor sets #847

Draft

mooki wants to merge 4 commits from mooki/Longshot:mac/topic-mvk-descriptor-limits into main

mooki commented

2026-11-15 17:39:46 +00:00

Member

Type: fix · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (arm64; x86_64 under Rosetta) · Needs: Windows and Linux, any GPU

Why

On a Mac the Vulkan renderer runs on MoltenVK, and MoltenVK binds a pushed descriptor set straight into a Metal function's own
resource tables, outside Metal's argument buffers. Four kinds of shader program break there when their set is pushed: a
program that reads the game's memory through a storage buffer table fails to build (the first Mac start lost 256 compute
pipelines that way); a program that reads a storage buffer's length reads 0 for every buffer, so its writes and atomics to
game memory are silently dropped; a stage with more than 8 storage images and storage texel buffers fails to compile (Metal
allows 8 read_write textures a function); and a stage with more buffers, textures or samplers than a Metal function's
tables hold fails to compile. Whatever those programs draw or compute is missing or wrong. In an ordinary descriptor set,
which MoltenVK keeps in a Metal 3 argument buffer, every one of them works. This PR gives those programs ordinary sets on
MoltenVK; every other driver is unchanged. It is part of the macOS port. See #420.

How to reproduce

Seen on: macOS 26.7.1, Apple M4 Max, MoltenVK 1.4.2, arm64 and x86_64 under Rosetta (main does not build on macOS yet:
seen on the macOS port's integration builds and in game-free tests and GPU probes)
Settings and switches: defaults
Steps: start the game on a fresh profile and let the first start prepare its shaders, then read longshot_log.txt.
Game-free: copy this PR's tests/test_set_for_tables.py, tests/test_set_for_lengths.py, tests/set_for_lengths_ir.cpp
and tests/test_vendor_fixes.py into main's tree and run python -I -B tests/test_set_for_tables.py --cxx clang++ --headers <Vulkan-Headers/include> and python -I -B tests/test_set_for_lengths.py --cxx clang++ --spirv-headers <SPIRV-Headers/include> --headers <Vulkan-Headers/include>.
Expected: every program's pipeline is made and reads exact storage buffer lengths; the tests pass.
Actual: 256 pairs of [mvk-error] SPIR-V to MSL conversion error: Attempting to alias same binding with a descriptor array which is not implemented through argument buffers and Compute shader function could not be compiled into pipeline
(the first start, mac-arm64-a5f2d6f9-01); the NullRangeProbe GPU check read 0 for every pushed storage buffer's
OpArrayLength on both CPU routes; a pushed compute stage with 40 storage images and 9 storage texel buffers is refused with
number of textures with read_write access exceeds maximum supported (8); a stage past a Metal function's table is refused
at its out-of-range [[buffer]], [[texture]] or [[sampler]] index. The tests push layouts that need a set (Measurements).

What changed

On MoltenVK a program is no longer pushed when its set holds a storage buffer table, when any stage's SPIR-V reads a storage
buffer's length, when one stage has more than 8 storage images and storage texel buffers, or when one stage would not fit a
Metal function's buffer, texture or sampler table; it gets an ordinary descriptor set from the recording's pool instead
(update-after-bind, from update-after-bind pools), the path programs over the push limit already take. Every other program
is still pushed, the bindings are the same either way, and the SPIR-V, the translator's stamp and the program cache's format
are unchanged. Two switches force it either way; the vk: vendor fixes line says whether each is on, and a line at close
counts the programs moved. The four original pull requests, in commit order:

storage buffer tables (063d87c4; Was: #643):
layout_pushes() in gpu/vk/vk_draw.cpp sends a set with a storage buffer array to an ordinary set under the new vendor
fix set_for_tables (gpu/vk/vk_vendor.h, on with MoltenVK only; RX_VK_SET_FOR_TABLES); gpu/vk/vk_main.cpp's
vk: vendor fixes line gains its field. Tests: tests/test_set_for_tables.py (new), tests/test_vendor_fixes.py.
storage buffer lengths (02ad928d; Was: #644):
gpu/vk/spirv_lengths.h (new) scans a stage's SPIR-V once for OpArrayLength where the program cache already handles the
code (gpu/vk/vk_progcache.cpp, Stage::array_length in gpu/vk/vk_shader.h); such a program gets a set under the vendor
fix set_for_lengths (RX_VK_SET_FOR_LENGTHS, MoltenVK only); a compute program read from the cache without its code is
decided by its global memory use until the code is read; draw_layouts_report() (gpu/vk/vk_draw.h) prints the close
line. Tests: tests/test_set_for_lengths.py, tests/set_for_lengths_ir.cpp (new); the program cache fixtures follow.
more than 8 read_write textures (5b523fef; Was: #645):
under set_for_tables, storage images and storage texel buffers are counted per stage (every array element; the mesh
fallback's task and mesh stages as two functions, from stage_first), and a stage over 8 gets a set.
Metal's per-stage tables (792a7d8a; Was: #646): under
set_for_tables, a stage past 7 buffers (vertex), 23 (fragment) or 27 (compute), 128 textures or 16 samplers gets a set
(MoltenVK's top buffer slots kept), every set under the fix is update-after-bind from update-after-bind pools, and the
close line counts the moves by class. tests/recording_admission_test.cpp follows the new dev.fixes field.
docs: docs/development.md (the two switches' rows), gpu/vk/VULKAN.md.

Measurements

Verdict: passes. Each member's new test fails before it and passes after it (two interleaved runs each in the members'
own runs), and every one passes at this PR's head on arm64 and on x86_64 under Rosetta; in the game (history, integration
builds) the 256 alias errors went to 0 on arm64 and under Rosetta, and at close 2,023 programs were moved for lengths and
3,366 for buffers, 0 for textures and 1,755 for samplers. The full suite at this head: 545 passed, 33 failed, 17 skipped against
main's 543/33/17 on this Mac, the only differences this PR's two new tests passing. No timing claimed: the cost
of a set against a push is not measured.

Baseline: the rows of #644, #645 and #646 compare with each member's own base in the stack (the heads of #643, #644 and
#645), as those pull requests measured them: main lacks the layout_pushes they extend. The game rows are the macOS port's
integration builds (main does not run on macOS).

Hardware: macOS 26.7.1, Apple M4 Max, 16-core CPU, 64 GB, Apple clang 21.0.0; MoltenVK 1.4.2 for the game rows
Scene: none for the tests (a layout fix: its proof is the test); the game rows: the start-up precompile of integration
starts 01 and 12, windowless, defaults
Method: each test before and after its pull request, two interleaved runs each (the members' runs); at this head each test
once on arm64 and once with its fixture built for x86_64 and run under Rosetta; the suite once through tools/ls-suite.py
How to reproduce: python -I -B tests/test_set_for_tables.py --cxx clang++ --headers <Vulkan-Headers/include>,
python -I -B tests/test_set_for_lengths.py --cxx clang++ --spirv-headers <SPIRV-Headers/include> --headers <Vulkan-Headers/include> --jobs 1, python -I -B tests/test_vendor_fixes.py --cxx clang++ --headers <Vulkan-Headers/include>

	Before: main @ `aa95d35f`	After: this PR @ `792a7d8a`
#643: `test_set_for_tables.py`, storage buffer table layouts pushed where a set is needed	7 of 79 (main)	0 of 237: passes, arm64 and x86_64
#644: `test_set_for_lengths.py`, length-reading layouts pushed	17 of 138, no scan (#643's head)	0 of 138: passes, arm64 and x86_64
#645: stages over 8 read_write textures pushed	7 of 134 (#644's head)	0: passes (in the 237)
#646: stages past a Metal table pushed, sets without update-after-bind	26 of 237 (#645's head)	0 of 237: passes
game: alias-error pipelines at the first start	256 (integration a5f2d6f9)	0, arm64 and Rosetta (integration builds)
game: programs moved, close line of start 12	none	lengths 2,023; buffers 3,366, textures 0, samplers 1,755
full game-free suite on this Mac	543 passed, 33 failed, 17 skipped	545 passed, 33 failed, 17 skipped

c3-settables-arm64: ok: 237 layouts; on MoltenVK a set with a storage buffer table, a stage with more than 8 storage images
  and storage texel buffers, or a stage past a Metal function's buffer, texture or sampler table is not pushed, and only such
  a set; every other driver as before; RX_VK_SET_FOR_TABLES forces it both ways
c3-setlengths-x86_64: layouts: 138 cases, 0 not as expected
c3-vendor-x86_64: ok: 17 devices, each fix on only where it is for, none on NVIDIA's driver; every switch forces its fix both ways

Flags and settings

Two new switches; by default both are on with MoltenVK only (unset), off with every other driver.

Flag	What it is	Before this PR	After this PR	Effect	Helps / doesn't help / untested	Recommendation
`RX_VK_SET_FOR_TABLES`	`0` or `1`: programs with a storage buffer table, a stage over 8 storage images and storage texel buffers, or a stage past a Metal function's buffer, texture or sampler table get an ordinary (update-after-bind) set instead of a push	(no switch: pushed wherever the set fits the push limit)	unset: on with MoltenVK, off elsewhere; `0` off, `1` on whatever the driver	those programs' sets are allocated and bound per draw or dispatch; their pipelines are made on MoltenVK; the close line counts the moves by class	helps: MoltenVK (the pipelines are made instead of failing); doesn't help: other drivers, which push these fine; untested: the cost of the extra sets, `=1` on another driver in the game	leave unset; `=0` only to see the failures on a Mac
`RX_VK_SET_FOR_LENGTHS`	`0` or `1`: a program whose SPIR-V reads a storage buffer's length in any stage gets an ordinary set	(no switch)	unset: on with MoltenVK, off elsewhere; `0` off, `1` on whatever the driver	the lengths are exact on MoltenVK; a close line counts the programs moved	helps: MoltenVK (global memory stores, atomics and loads reach their buffers); doesn't help: other drivers, whose pushed lengths are right; untested: the game with it off against on	leave unset; `=0` only to compare

Testing

Tested: on macOS 26.7.1, Apple M4 Max, at this head 792a7d8a (a clean worktree):

The four member pull requests' own tests, deduplicated, on arm64 and on x86_64 under Rosetta (the fixture built with
clang++ -arch x86_64): test_set_for_tables, test_set_for_lengths, test_vendor_fixes, test_program_cache_one_variant
and test_program_cache_cleanup (with a no-op posix_fadvise prelude, as the members ran them), test_recording_admission:
12 runs, 12 exit 0 (237 layouts; 138 layouts, 0 not as expected; 17 devices; the program cache's byte counts as before;
22 PASS lines). test_set_for_tables' x86_64 fixture was killed once by the system (exit -9, no crash report), as in the
members' own runs of #643-#645, and passed on its one rerun.
python -I -B tests/run_all.py through tools/ls-suite.py at this head: 545 passed, 33 failed, 17 skipped, against main
aa95d35f's 543/33/17 on this Mac, compared by test name: the only differences are test_set_for_tables and
test_set_for_lengths, new and passing. The 33 failures are main's own on this Mac (main without the macOS host topic: the
runner still calls a Mac linux, and main's renderer and host checks that need the macOS PRs fail as on main).
The renderer's link check at this head, which is on main (main's renderer does not build on macOS, so the check ran on a
scratch merge of this head with the macOS PRs that make it build, #476, #492 and #490, with main's renderer flags): 0 of 595
declared mkwvk:: names lack a definition, 66 of 66 objects compiled (arm64 and x86_64), 542 cross-file symbols per
architecture all defined.
Off Apple (unifdef -U__APPLE__ of every changed non-test C/C++ file, main against this head, compared as tokens): the
renderer files differ there by design, since this fix is chosen at run time by the driver (dev.fixes), not at compile
time: vk_draw.cpp (layout_pushes, the counts, the update-after-bind layout and pool, draw_layouts_report),
vk_draw.h, vk_main.cpp (the vk: vendor fixes line's two new fields), vk_vendor.h (the two fixes, off but for
MoltenVK), vk_progcache.cpp and vk_shader.h (the OpArrayLength scan, run on every platform), and the new
spirv_lengths.h. With the fixes off, other drivers' layouts are the same (the tests' ten other drivers).
macOS game runs with these commits in the stack (integration builds carrying this topic's members): arm64, windowless: run
E9a on I41 78ec1c47: a 1,197 s hold in play, 12 of 12 must-shows, 0 skipped draws, a clean end
(reports/2026-11-15-mac-perf-E9.md); x86_64 under Rosetta: run #34 on I38 3f872b0c: 16 m 52 s of play, every draw
drawn, 19 of 19 checks, a clean exit (reports/2026-11-14-mac-startup-34-x86_64.md).

Not tested: Windows and Linux: nothing was built or run there (the test request below). The cost of a set against a push
in a real scene (RX_VK_PROF_FINE=1 counts the sets allocated); the game with either switch at 0; KosmicKrisp; an Intel
Mac (where MoltenVK may run without argument buffers); a GPU probe of a pushed stage with 17 samplers, 129 textures or 31
buffers (that a set takes them rests on MoltenVK's source).

Must be tested:

Windows and Linux, any GPU: build this PR and main aa95d35f the same way, run python -I -B tests/run_all.py on each and
start each to the title, expecting test_set_for_tables and test_set_for_lengths to pass, no other check to change status,
and the vk: lines to match main's but for the vk: vendor fixes line's two new fields at 0 (Test Farm: @Beanstalk windows
linux, build, run_all and start to the title)

Risks

What could break: on MoltenVK many more programs now allocate, write and bind a set per draw or dispatch instead of a
push (2,023, 3,366 and 1,755 programs in start 12), a cost not measured; a recording's pool that runs out is followed by
another of the same size. The buffer budgets keep every slot MoltenVK may take for itself, so some programs that would have
pushed safely get a set too. On other drivers: the layouts are identical, each stage's SPIR-V gets one more walk over its words
where the program cache already handles it, and the vk: vendor fixes line has two more fields.

Turning it off: RX_VK_SET_FOR_TABLES=0 and RX_VK_SET_FOR_LENGTHS=0 (MoltenVK then fails or misreads these programs as
on main), or revert the PR.

Compatibility: nothing changes in saves, longshot_options.ini or the vk ABI (MKWVK_API_VERSION). The shader caches are
kept: none of the changed files is in the translator's stamp (ebbf160ce9626041 before and after) and the program cache's
format is the same. On Windows and Linux the vk: vendor fixes line has two more fields (0). On macOS the moved programs'
pipelines change layout, so Metal compiles them once.

Issues

See #420. The macOS topic PRs, merged in this order; "topic N" in each description means these: 1 #845 (macos-host), 2 #846 (vulkan-on-metal, on 1), 3 #847 (mvk-descriptor-limits), 4 #848 (shader-backend, on 2, 3 and #431), 5 #849 (precompile-metal, on 4), 6 #850 (draw-path, on 4), 7 #851 (draw-path-quads-mesh, on 6, #705 and #707), 8 #852 (mac-perf, on 7).

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main: on aa95d35f.
python -I -B tests/run_all.py passes (any failure that also fails on main is named). (macOS: 545/33/17, the 33 failures exactly main's on this Mac, reports/Baseline-files/main-aa95d35f; Windows and Linux: the test request.)
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Not built there yet: the test request.)
Docs updated.

**Type:** fix · **Risk:** low · **Tested on:** macOS 26.7.1, Apple M4 Max (arm64; x86_64 under Rosetta) · **Needs:** Windows and Linux, any GPU ## Why On a Mac the Vulkan renderer runs on MoltenVK, and MoltenVK binds a pushed descriptor set straight into a Metal function's own resource tables, outside Metal's argument buffers. Four kinds of shader program break there when their set is pushed: a program that reads the game's memory through a storage buffer table fails to build (the first Mac start lost 256 compute pipelines that way); a program that reads a storage buffer's length reads 0 for every buffer, so its writes and atomics to game memory are silently dropped; a stage with more than 8 storage images and storage texel buffers fails to compile (Metal allows 8 read_write textures a function); and a stage with more buffers, textures or samplers than a Metal function's tables hold fails to compile. Whatever those programs draw or compute is missing or wrong. In an ordinary descriptor set, which MoltenVK keeps in a Metal 3 argument buffer, every one of them works. This PR gives those programs ordinary sets on MoltenVK; every other driver is unchanged. It is part of the macOS port. See #420. ## How to reproduce **Seen on:** macOS 26.7.1, Apple M4 Max, MoltenVK 1.4.2, arm64 and x86_64 under Rosetta (main does not build on macOS yet: seen on the macOS port's integration builds and in game-free tests and GPU probes) **Settings and switches:** defaults **Steps:** start the game on a fresh profile and let the first start prepare its shaders, then read longshot_log.txt. Game-free: copy this PR's `tests/test_set_for_tables.py`, `tests/test_set_for_lengths.py`, `tests/set_for_lengths_ir.cpp` and `tests/test_vendor_fixes.py` into main's tree and run `python -I -B tests/test_set_for_tables.py --cxx clang++ --headers <Vulkan-Headers/include>` and `python -I -B tests/test_set_for_lengths.py --cxx clang++ --spirv-headers <SPIRV-Headers/include> --headers <Vulkan-Headers/include>`. **Expected:** every program's pipeline is made and reads exact storage buffer lengths; the tests pass. **Actual:** 256 pairs of `[mvk-error] SPIR-V to MSL conversion error: Attempting to alias same binding with a descriptor array which is not implemented through argument buffers` and `Compute shader function could not be compiled into pipeline` (the first start, `mac-arm64-a5f2d6f9-01`); the NullRangeProbe GPU check read 0 for every pushed storage buffer's `OpArrayLength` on both CPU routes; a pushed compute stage with 40 storage images and 9 storage texel buffers is refused with `number of textures with read_write access exceeds maximum supported (8)`; a stage past a Metal function's table is refused at its out-of-range `[[buffer]]`, `[[texture]]` or `[[sampler]]` index. The tests push layouts that need a set (Measurements). ## What changed On MoltenVK a program is no longer pushed when its set holds a storage buffer table, when any stage's SPIR-V reads a storage buffer's length, when one stage has more than 8 storage images and storage texel buffers, or when one stage would not fit a Metal function's buffer, texture or sampler table; it gets an ordinary descriptor set from the recording's pool instead (update-after-bind, from update-after-bind pools), the path programs over the push limit already take. Every other program is still pushed, the bindings are the same either way, and the SPIR-V, the translator's stamp and the program cache's format are unchanged. Two switches force it either way; the `vk: vendor fixes` line says whether each is on, and a line at close counts the programs moved. The four original pull requests, in commit order: - storage buffer tables (`063d87c4`; Was: [#643](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/643)): `layout_pushes()` in `gpu/vk/vk_draw.cpp` sends a set with a storage buffer array to an ordinary set under the new vendor fix `set_for_tables` (`gpu/vk/vk_vendor.h`, on with MoltenVK only; `RX_VK_SET_FOR_TABLES`); `gpu/vk/vk_main.cpp`'s `vk: vendor fixes` line gains its field. Tests: `tests/test_set_for_tables.py` (new), `tests/test_vendor_fixes.py`. - storage buffer lengths (`02ad928d`; Was: [#644](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/644)): `gpu/vk/spirv_lengths.h` (new) scans a stage's SPIR-V once for `OpArrayLength` where the program cache already handles the code (`gpu/vk/vk_progcache.cpp`, `Stage::array_length` in `gpu/vk/vk_shader.h`); such a program gets a set under the vendor fix `set_for_lengths` (`RX_VK_SET_FOR_LENGTHS`, MoltenVK only); a compute program read from the cache without its code is decided by its global memory use until the code is read; `draw_layouts_report()` (`gpu/vk/vk_draw.h`) prints the close line. Tests: `tests/test_set_for_lengths.py`, `tests/set_for_lengths_ir.cpp` (new); the program cache fixtures follow. - more than 8 read_write textures (`5b523fef`; Was: [#645](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/645)): under `set_for_tables`, storage images and storage texel buffers are counted per stage (every array element; the mesh fallback's task and mesh stages as two functions, from `stage_first`), and a stage over 8 gets a set. - Metal's per-stage tables (`792a7d8a`; Was: [#646](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/646)): under `set_for_tables`, a stage past 7 buffers (vertex), 23 (fragment) or 27 (compute), 128 textures or 16 samplers gets a set (MoltenVK's top buffer slots kept), every set under the fix is update-after-bind from update-after-bind pools, and the close line counts the moves by class. `tests/recording_admission_test.cpp` follows the new `dev.fixes` field. - docs: `docs/development.md` (the two switches' rows), `gpu/vk/VULKAN.md`. ## Measurements **Verdict:** passes. Each member's new test fails before it and passes after it (two interleaved runs each in the members' own runs), and every one passes at this PR's head on arm64 and on x86_64 under Rosetta; in the game (history, integration builds) the 256 alias errors went to 0 on arm64 and under Rosetta, and at close 2,023 programs were moved for lengths and 3,366 for buffers, 0 for textures and 1,755 for samplers. The full suite at this head: 545 passed, 33 failed, 17 skipped against main's 543/33/17 on this Mac, the only differences this PR's two new tests passing. No timing claimed: the cost of a set against a push is not measured. Baseline: the rows of #644, #645 and #646 compare with each member's own base in the stack (the heads of #643, #644 and #645), as those pull requests measured them: main lacks the `layout_pushes` they extend. The game rows are the macOS port's integration builds (main does not run on macOS). Hardware: macOS 26.7.1, Apple M4 Max, 16-core CPU, 64 GB, Apple clang 21.0.0; MoltenVK 1.4.2 for the game rows Scene: none for the tests (a layout fix: its proof is the test); the game rows: the start-up precompile of integration starts 01 and 12, windowless, defaults Method: each test before and after its pull request, two interleaved runs each (the members' runs); at this head each test once on arm64 and once with its fixture built for x86_64 and run under Rosetta; the suite once through `tools/ls-suite.py` How to reproduce: `python -I -B tests/test_set_for_tables.py --cxx clang++ --headers <Vulkan-Headers/include>`, `python -I -B tests/test_set_for_lengths.py --cxx clang++ --spirv-headers <SPIRV-Headers/include> --headers <Vulkan-Headers/include> --jobs 1`, `python -I -B tests/test_vendor_fixes.py --cxx clang++ --headers <Vulkan-Headers/include>` | | Before: main @ aa95d35f | After: this PR @ 792a7d8a | |---|---|---| | #643: `test_set_for_tables.py`, storage buffer table layouts pushed where a set is needed | 7 of 79 (main) | 0 of 237: passes, arm64 and x86_64 | | #644: `test_set_for_lengths.py`, length-reading layouts pushed | 17 of 138, no scan (#643's head) | 0 of 138: passes, arm64 and x86_64 | | #645: stages over 8 read_write textures pushed | 7 of 134 (#644's head) | 0: passes (in the 237) | | #646: stages past a Metal table pushed, sets without update-after-bind | 26 of 237 (#645's head) | 0 of 237: passes | | game: alias-error pipelines at the first start | 256 (integration a5f2d6f9) | 0, arm64 and Rosetta (integration builds) | | game: programs moved, close line of start 12 | none | lengths 2,023; buffers 3,366, textures 0, samplers 1,755 | | full game-free suite on this Mac | 543 passed, 33 failed, 17 skipped | 545 passed, 33 failed, 17 skipped | ``` c3-settables-arm64: ok: 237 layouts; on MoltenVK a set with a storage buffer table, a stage with more than 8 storage images and storage texel buffers, or a stage past a Metal function's buffer, texture or sampler table is not pushed, and only such a set; every other driver as before; RX_VK_SET_FOR_TABLES forces it both ways c3-setlengths-x86_64: layouts: 138 cases, 0 not as expected c3-vendor-x86_64: ok: 17 devices, each fix on only where it is for, none on NVIDIA's driver; every switch forces its fix both ways ``` ### Flags and settings Two new switches; by default both are on with MoltenVK only (unset), off with every other driver. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `RX_VK_SET_FOR_TABLES` | `0` or `1`: programs with a storage buffer table, a stage over 8 storage images and storage texel buffers, or a stage past a Metal function's buffer, texture or sampler table get an ordinary (update-after-bind) set instead of a push | (no switch: pushed wherever the set fits the push limit) | unset: on with MoltenVK, off elsewhere; `0` off, `1` on whatever the driver | those programs' sets are allocated and bound per draw or dispatch; their pipelines are made on MoltenVK; the close line counts the moves by class | helps: MoltenVK (the pipelines are made instead of failing); doesn't help: other drivers, which push these fine; untested: the cost of the extra sets, `=1` on another driver in the game | leave unset; `=0` only to see the failures on a Mac | | `RX_VK_SET_FOR_LENGTHS` | `0` or `1`: a program whose SPIR-V reads a storage buffer's length in any stage gets an ordinary set | (no switch) | unset: on with MoltenVK, off elsewhere; `0` off, `1` on whatever the driver | the lengths are exact on MoltenVK; a close line counts the programs moved | helps: MoltenVK (global memory stores, atomics and loads reach their buffers); doesn't help: other drivers, whose pushed lengths are right; untested: the game with it off against on | leave unset; `=0` only to compare | ## Testing **Tested:** on macOS 26.7.1, Apple M4 Max, at this head `792a7d8a` (a clean worktree): - The four member pull requests' own tests, deduplicated, on arm64 and on x86_64 under Rosetta (the fixture built with `clang++ -arch x86_64`): `test_set_for_tables`, `test_set_for_lengths`, `test_vendor_fixes`, `test_program_cache_one_variant` and `test_program_cache_cleanup` (with a no-op `posix_fadvise` prelude, as the members ran them), `test_recording_admission`: 12 runs, 12 exit 0 (237 layouts; 138 layouts, 0 not as expected; 17 devices; the program cache's byte counts as before; 22 PASS lines). `test_set_for_tables`' x86_64 fixture was killed once by the system (exit -9, no crash report), as in the members' own runs of #643-#645, and passed on its one rerun. - `python -I -B tests/run_all.py` through `tools/ls-suite.py` at this head: 545 passed, 33 failed, 17 skipped, against main `aa95d35f`'s 543/33/17 on this Mac, compared by test name: the only differences are `test_set_for_tables` and `test_set_for_lengths`, new and passing. The 33 failures are main's own on this Mac (main without the macOS host topic: the runner still calls a Mac `linux`, and main's renderer and host checks that need the macOS PRs fail as on main). - The renderer's link check at this head, which is on main (main's renderer does not build on macOS, so the check ran on a scratch merge of this head with the macOS PRs that make it build, #476, #492 and #490, with main's renderer flags): 0 of 595 declared `mkwvk::` names lack a definition, 66 of 66 objects compiled (arm64 and x86_64), 542 cross-file symbols per architecture all defined. - Off Apple (`unifdef -U__APPLE__` of every changed non-test C/C++ file, main against this head, compared as tokens): the renderer files differ there by design, since this fix is chosen at run time by the driver (`dev.fixes`), not at compile time: `vk_draw.cpp` (`layout_pushes`, the counts, the update-after-bind layout and pool, `draw_layouts_report`), `vk_draw.h`, `vk_main.cpp` (the `vk: vendor fixes` line's two new fields), `vk_vendor.h` (the two fixes, off but for MoltenVK), `vk_progcache.cpp` and `vk_shader.h` (the `OpArrayLength` scan, run on every platform), and the new `spirv_lengths.h`. With the fixes off, other drivers' layouts are the same (the tests' ten other drivers). - macOS game runs with these commits in the stack (integration builds carrying this topic's members): arm64, windowless: run E9a on I41 `78ec1c47`: a 1,197 s hold in play, 12 of 12 must-shows, 0 skipped draws, a clean end (`reports/2026-11-15-mac-perf-E9.md`); x86_64 under Rosetta: run #34 on I38 `3f872b0c`: 16 m 52 s of play, every draw drawn, 19 of 19 checks, a clean exit (`reports/2026-11-14-mac-startup-34-x86_64.md`). **Not tested:** Windows and Linux: nothing was built or run there (the test request below). The cost of a set against a push in a real scene (`RX_VK_PROF_FINE=1` counts the sets allocated); the game with either switch at `0`; KosmicKrisp; an Intel Mac (where MoltenVK may run without argument buffers); a GPU probe of a pushed stage with 17 samplers, 129 textures or 31 buffers (that a set takes them rests on MoltenVK's source). **Must be tested:** - Windows and Linux, any GPU: build this PR and main `aa95d35f` the same way, run `python -I -B tests/run_all.py` on each and start each to the title, expecting `test_set_for_tables` and `test_set_for_lengths` to pass, no other check to change status, and the `vk:` lines to match main's but for the `vk: vendor fixes` line's two new fields at 0 (Test Farm: @Beanstalk windows linux, build, run_all and start to the title) ## Risks **What could break:** on MoltenVK many more programs now allocate, write and bind a set per draw or dispatch instead of a push (2,023, 3,366 and 1,755 programs in start 12), a cost not measured; a recording's pool that runs out is followed by another of the same size. The buffer budgets keep every slot MoltenVK may take for itself, so some programs that would have pushed safely get a set too. On other drivers: the layouts are identical, each stage's SPIR-V gets one more walk over its words where the program cache already handles it, and the `vk: vendor fixes` line has two more fields. **Turning it off:** `RX_VK_SET_FOR_TABLES=0` and `RX_VK_SET_FOR_LENGTHS=0` (MoltenVK then fails or misreads these programs as on main), or revert the PR. **Compatibility:** nothing changes in saves, `longshot_options.ini` or the vk ABI (`MKWVK_API_VERSION`). The shader caches are kept: none of the changed files is in the translator's stamp (`ebbf160ce9626041` before and after) and the program cache's format is the same. On Windows and Linux the `vk: vendor fixes` line has two more fields (`0`). On macOS the moved programs' pipelines change layout, so Metal compiles them once. ## Issues See #420. The macOS topic PRs, merged in this order; "topic N" in each description means these: 1 #845 (macos-host), 2 #846 (vulkan-on-metal, on 1), 3 #847 (mvk-descriptor-limits), 4 #848 (shader-backend, on 2, 3 and #431), 5 #849 (precompile-metal, on 4), 6 #850 (draw-path, on 4), 7 #851 (draw-path-quads-mesh, on 6, #705 and #707), 8 #852 (mac-perf, on 7). ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`: on `aa95d35f`. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (macOS: 545/33/17, the 33 failures exactly main's on this Mac, `reports/Baseline-files/main-aa95d35f`; Windows and Linux: the test request.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Not built there yet: the test request.) - [x] Docs updated.

mooki added 4 commits

2026-11-15 17:39:47 +00:00

fix(vk): give storage buffer tables descriptor sets instead of pushes on MoltenVK 063d87c431

Was: #643 (#643), branch mac/mvk-push at ca17df3be6 (Sweep3, on main aa95d35f).

Its own commits, squashed:
- fix(macos): storage buffer tables get descriptor sets on MoltenVK (ca17df3be)

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(vk): give programs that read storage buffer lengths descriptor sets on MoltenVK 02ad928d81

Was: #644 (#644), branch mac/mvk-sizes at f08eb70206 (Sweep3, on main aa95d35f).

It was stacked on #643; their changes are the earlier commits here.

Its own commits, squashed:
- fix(macos): programs reading storage buffer lengths get descriptor sets on MoltenVK (f08eb7020)

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(vk): use descriptor sets for stages with more than 8 read_write textures on MoltenVK 5b523fef19

Was: #645 (#645), branch mac/mvk-rwtex at b8b1ba998a (Sweep3, on main aa95d35f).

It was stacked on #644; their changes are the earlier commits here.

Its own commits, squashed:
- fix(macos): stages with more than 8 read_write textures get descriptor sets on MoltenVK (b8b1ba998)

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(vk): use update-after-bind sets past Metal's per-stage limits on MoltenVK

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 15s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 16s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s

Details

Third-party parts / As their notes say (pull_request) Successful in 14s

Details

PR description / Description follows the template (pull_request_target) Successful in 16s

Details

792a7d8a32

Was: #646 (#646), branch mac/mvk-pushlimits at 374c55a031 (Sweep3, on main aa95d35f).

It was stacked on #645; their changes are the earlier commits here.

Its own commits, squashed:
- fix(macos): stages past a Metal function's buffer, texture or sampler table get descriptor sets on MoltenVK (6c9e3c9fa)
- fix(macos): MoltenVK's descriptor sets are update-after-bind layouts, within Vulkan's limits for them (3e7d8d3d3)
- test(vk): recording_admission's device stub has the fixes make_pool now reads (374c55a03)

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

BeanIssues added the

labels

2026-11-15 17:39:51 +00:00

mooki referenced this pull request

2026-11-15 17:41:20 +00:00

WIP: feat(macos): run the host on macOS: build, runtime, control, crash records and the Vulkan loader #845

mooki referenced this pull request

2026-11-15 17:41:24 +00:00

WIP: feat(macos): Vulkan on Metal: pinned host imports, the unified-memory budget, Metal's cache folder and KosmicKrisp #846

mooki referenced this pull request

2026-11-15 17:41:31 +00:00

WIP: feat(vk): the SPIR-V back end on MoltenVK and guest spans past one device allocation #848

mooki referenced this pull request

2026-11-15 17:41:35 +00:00

WIP: fix(macos): MoltenVK's precompile and pipelines on Metal #849

mooki referenced this pull request

2026-11-15 17:41:40 +00:00

WIP: fix(macos): draw-path emulation on MoltenVK #850

mooki referenced this pull request

2026-11-15 17:41:44 +00:00

WIP: fix(macos): the Mac halves of indirect quads and mesh count draws #851

mooki referenced this pull request

2026-11-15 17:41:48 +00:00

WIP: perf(macos): Mac performance on MoltenVK #852

MaikelJSQ referenced this pull request

2026-11-15 18:05:02 +00:00

Game: Longshot closed unexpectedly while you were playing #859

mooki referenced this pull request

2026-11-15 18:07:45 +00:00

fix(vk): give storage buffer tables descriptor sets instead of pushes on MoltenVK #643

mooki referenced this pull request

2026-11-15 18:07:47 +00:00

fix(vk): give programs that read storage buffer lengths descriptor sets on MoltenVK #644

mooki referenced this pull request

2026-11-15 18:07:48 +00:00

fix(vk): use descriptor sets for stages with more than 8 read_write textures on MoltenVK #645

mooki referenced this pull request

2026-11-15 18:07:49 +00:00

fix(vk): use update-after-bind sets past Metal's per-stage limits on MoltenVK #646

mooki referenced this pull request

2026-11-15 18:14:44 +00:00

Idea: a native macOS build (Apple silicon, x86-64 under Rosetta) with Vulkan on Metal through MoltenVK #420

HumanBean force-pushed mac/topic-mvk-descriptor-limits from 792a7d8a32

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 15s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 16s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s

Details

Third-party parts / As their notes say (pull_request) Successful in 14s

Details

PR description / Description follows the template (pull_request_target) Successful in 16s

Details

to 88e914ad4e

PR description / Description follows the template (pull_request_target) Waiting to run

Details

CI / Conventional Commit title (pull_request) Waiting to run

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

CI / Workflow lint (actionlint) (pull_request) Waiting to run

Details

Guide screenshots / Guide images and local UI source review (pull_request) Waiting to run

Details

Third-party parts / As their notes say (pull_request) Waiting to run

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

2026-11-15 18:47:17 +00:00

Compare

HumanBean commented

2026-11-15 18:49:25 +00:00

Owner

Rebased onto main @ 7bacf846. Conflicts in gpu/vk/vk_shader.h in three commits (main's code_at/record_at members of Stage and this PR's array_length member, both added at the end of the struct): kept both. No tests were run here; the test boxes stay yours.

Rebased onto `main` @ 7bacf846. Conflicts in `gpu/vk/vk_shader.h` in three commits (`main`'s `code_at`/`record_at` members of `Stage` and this PR's `array_length` member, both added at the end of the struct): kept both. No tests were run here; the test boxes stay yours.