WIP: fix(macos): KosmicKrisp's compiles go through the compiler service's precompile, as MoltenVK's do #785

Closed
mooki wants to merge 23 commits from mooki/Longshot:mac/kk-compiler-service into main
Member

Type: fix · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (MoltenVK 1.4.2; KosmicKrisp from the LunarG SDK 1.4.363.1 in probes KK-03 and KK-04), game-free on arm64 and x86_64 under Rosetta · Needs: macOS, Apple silicon (a start with KosmicKrisp); Windows and Linux, any GPU (build and start)

Why

On a Mac, KosmicKrisp (Mesa's Vulkan driver on Metal 4, opt-in since #654: RX_VK_DRIVER=kosmickrisp) compiles every
pipeline in Metal's compiler service, as MoltenVK does, but the game treated it as any other driver: its shader preparation
compiled every compute program's pipelines itself, the place where nothing can hand a slow compile to the game. The first
start with KosmicKrisp (start #20) reached 1,607 of 38,809 programs in 29.5 minutes and never showed the title; on MoltenVK
(start #19, Integration 27) the preparation translated all 38,809 in 75 s and the precompile compiled them, then the game
started. This PR gives KosmicKrisp the same compiler-service path as MoltenVK: the preparation only translates, the
precompile compiles (a few giants at a time, the rest behind the game), and a compile that Metal's service failed is tried
again. It is part of the macOS port. See #420.

How to reproduce

Seen on: macOS 26.7.1, Apple M4 Max, KosmicKrisp from the LunarG Vulkan SDK 1.4.363.1 (Mesa 269a955c); main does not
start on macOS yet: seen on the macOS port's integration build 09f9a171, which carries the open macOS PRs
Settings and switches: RX_VK_DRIVER=kosmickrisp (start #20 chose it by itself: that build picked KosmicKrisp on
macOS 26; MoltenVK has been the default since)
Steps: start the game with KosmicKrisp and a translated program cache missing (a new translator stamp), and watch the
vk: start-up: N of 38809 lines.
Expected: the preparation translates the programs in about a minute, then the precompile compiles their pipelines and
hands the rest to the game, as on MoltenVK (start #19).
Actual: the preparation compiles every compute program's pipelines itself: 0.9 programs a second, 1,607 of 38,809 after
29.5 minutes, no title (reports/2026-11-14-mac-startup-20.md, reports/2026-11-14-kk-slow.md §3).

What changed

On KosmicKrisp's driver (VK_DRIVER_ID_MESA_KOSMICKRISP) compiler_service is on by default, as on MoltenVK's, and its
failed-compile code is read as MoltenVK's where the service's compiles are made. Every other driver, MoltenVK's included, runs
the code it ran before.

  • gpu/vk/vk_vendor.h vendor_fixes: compiler_service on for MoltenVK's and KosmicKrisp's drivers;
    RX_VK_COMPILER_SERVICE=0 still turns it off. The comment carries KK-03's numbers (below).
  • gpu/vk/vk_draw.cpp service_failed: KosmicKrisp reports every Metal library or pipeline it could not make, a compiler
    service that ended under it included (start #20 had three), as VK_ERROR_INVALID_SHADER_NV. Where the service's compiles
    are made (create_watched with compiler_service on, and compile_again's result) that code is read as
    VK_ERROR_INITIALIZATION_FAILED, MoltenVK's, on KosmicKrisp only, and said once (vk: KosmicKrisp: a pipeline compile failed (VK_ERROR_INVALID_SHADER_NV: …); read as VK_ERROR_INITIALIZATION_FAILED from here on …). So #665's tries again apply
    unchanged: once more with a new pipeline cache, or a giant's alone. KosmicKrisp's pipeline cache keeps no failed compile
    (Mesa vk_pipeline.c), so the new cache costs it the one compile alone. compile_again's line still prints the driver's
    own code. The comments on the section and on prepare_run's warm name KosmicKrisp and start #20.
  • docs/development.md: the RX_VK_COMPILER_SERVICE and RX_VK_PRECOMPILE_STALL_S rows.
  • tests/test_precompile_handoff.py: service_failed, compile_again and create_watched compiled from the source against a
    made-up KosmicKrisp (its failed compile is VK_ERROR_INVALID_SHADER_NV, its cache keeps none): a compile that failed once is
    made by the second try and said once; one that always fails is tried twice and said not made (-1000012000); a giant's
    failure is left for its try alone; on MoltenVK's driver the same code is returned as it is; on KosmicKrisp without
    compiler_service too. The vendor keying (KosmicKrisp on, the switch off on it, HoneyKrisp off), and the source's wiring
    (service_failed( three times: its definition and the two places).

Measurements

Verdict: the cause is measured, the fix's start is not yet. Probe KK-03 measured KosmicKrisp's compiles on identical
synthetic SPIR-V against MoltenVK's: 1.0× MoltenVK's at 3,003 words, 2.5× at 29,310, 5.8× at 110,406 (2.4× at the middle size
with robustness off on both), 16 at once 4.7×; its pipeline cache saves 8% in a new process, Metal's own cache 94-95% on both
drivers. So the compiler-service path that keeps MoltenVK's giants out of the preparation is needed on KosmicKrisp at least as
much. Probe KK-04 confirmed the driver facts the fix keys on (5 of 5 predictions): for a module Metal cannot compile,
KosmicKrisp returns VK_ERROR_INVALID_SHADER_NV both times and compiles it again on the second ask (its cache keeps no
failure), while MoltenVK returns VK_ERROR_INITIALIZATION_FAILED and answers the second ask from its cache in 0.0 ms (why
compile_again takes a new cache). A game start with this PR is in the table's After column.

Baseline: main does not start on macOS. The "before" is the integration build 09f9a171 (start #20, KosmicKrisp, without this
change; MoltenVK's start #19 on Integration 27 for comparison); KK-03 compares the two drivers in one standalone process each,
with no LS code.

Hardware: macOS 26.7.1 (25G241), Apple M4 Max (40-core GPU), 16-core CPU, 64 GB; KosmicKrisp from the LunarG SDK 1.4.363.1
(Mesa 269a955c), MoltenVK 1.4.2, both through the SDK's loader
Scene: start #20: the preparation from a program cache to translate again in full (a new stamp), windowless; KK-03: one
compute module each of 3,003, 29,310 and 110,406 SPIR-V words, cold and warm Metal caches, 16 at once
Method: start #20: the vk: start-up: N of 38809 readings every 60 s; KK-03: the Vulkan call's time per compile in each
phase's probe.log (reports/KkSlow-files/kk03/runs-kk03-kk/run.json 449dfd7b…, runs-kk03-mvk/run.json 7acdbc04…),
one run per route; KK-04: the result and time of each vkCreateComputePipelines for a 300-deep (past Metal's bracket nesting of
256) and a 200-deep module, twice each with the same VkPipelineCache, one run per route, released by the parent at 05:02-05:03 CST
How to reproduce: KK-03's run commands (reports/KkSlow-files/kk03/FROZEN-COMMAND.md); KK-04's run and score commands
(reports/2026-11-14-kk-slow.md §5.5); a start with RX_VK_DRIVER=kosmickrisp reading the preparation's and the precompile's lines

Before: integration @ 09f9a171 (start #20, KosmicKrisp) After: this PR @ a4d31a59
the preparation (38,809 programs) 1,607 after 1,770 s (0.9 a second), compute pipelines compiled there; no title (E-series numbers: pending)
KK-03, the Vulkan call's ms (cold Metal cache) KosmicKrisp MoltenVK 1.4.2 KosmicKrisp / MoltenVK
3,003 SPIR-V words 38.8 38.4 1.01
29,310 456.6 185.3 2.46
110,406 4,570.3 787.2 5.81
16 new 29,310-word modules at once, wall 2,094 449 4.66
warm Metal cache, 110,406 words 276.1 39.4
KK-04 (VkResult, ms of the call) KosmicKrisp MoltenVK 1.4.2
nest-300 (Metal cannot compile it), first -1000012000 (VK_ERROR_INVALID_SHADER_NV), 190.1 -3 (VK_ERROR_INITIALIZATION_FAILED), 14.9
nest-300, again, same cache -1000012000, 118.5 (compiled again) -3, 0.0 (the failure from its cache)
nest-200, first / again 0, 422.9 / 0, 0.0 0, 69.2 / 0, 0.1
#20, the watch's readings of `vk: start-up: N of 38809`: N = 38 at 11:17:23.6, 1,607 at 11:46:32.9 (the preparation began 11:17:13.5)
#19 (MoltenVK, Integration 27): 38,809 translated in 74.8 s, then 42,234 precompile items in 626.5 s
KK-03: ROUTE route=kk safe=1 complete=1 phases=6/6 compiler_crash_reports=0; ROUTE route=mvk safe=1 complete=1 phases=6/6 compiler_crash_reports=0; COMPARE predictions=9/12
KK-04: ROUTE route=kk safe=1 exit=1 nest-300#1=-1000012000/190.1ms nest-300#2=-1000012000/118.5ms nest-200#1=0/422.9ms nest-200#2=0/0.0ms nesting_lines=2 new_reports=0
KK-04: ROUTE route=mvk safe=1 exit=1 nest-300#1=-3/14.9ms nest-300#2=-3/0.0ms nest-200#1=0/69.2ms nest-200#2=0/0.1ms nesting_lines=1 new_reports=0
KK-04: reports/KkSlow-files/kk04/score-kk04/score.json (sha256 bb0b1e2747d4d20f0385a541a1b1bfeb4b1f7f85cb1815d70a4943acb9525ef7): K04-1 … K04-5 MATCH
(the probe exits 1 on each route because nest-300 fails on purpose)

Reports: reports/2026-11-14-kk-slow.md (KosmicKrisp's compile path from the SDK's Mesa source, KK-03, the fix, KK-04),
reports/2026-11-14-mac-startup-20.md.

Flags and settings

One default changes, on KosmicKrisp only: compiler_service, which RX_VK_COMPILER_SERVICE overrides. The diff against main
also shows the switches of the PRs it stacks on, theirs to list: RX_VK_COMPILE_QOS, RX_VK_GIANT_COMPILES,
RX_VK_GIANT_HOLD_PRESENTS, RX_VK_GIANT_HOLD_S, RX_VK_GIANT_WORDS, RX_VK_SLOW_COMPILE_S, RX_VK_WORKER_CACHES.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
RX_VK_COMPILER_SERVICE=0 or 1 where Metal's compiler service compiles: the preparation translates, the precompile compiles and hands the rest to the game, a failed compile tried again on by default on MoltenVK only on by default on MoltenVK and KosmicKrisp on KosmicKrisp the preparation no longer compiles the compute pipelines; failed compiles there are tried again helps: start #20's shape (KK-03: KosmicKrisp's compiles 1-5.8× MoltenVK's, which needs it); untested: a KosmicKrisp start with it as it ships (on for both Apple drivers); KosmicKrisp itself stays opt-in

Testing

Tested: on macOS 26.7.1, Apple M4 Max:

  • At this head (main e090cc48 plus #650, #651, #652, #664, #665 and this PR), every run under the shared check slot with a fresh
    HOME and TMPDIR (reports/PubWave-files/PubLate5/StartTrio/proof/kk-*):
    • tests/test_precompile_handoff.py --headers <Vulkan-Headers 1.4.345>, --arch arm64 and --arch x86_64 (Rosetta): exit 0
      on both (PASS: the precompile goes on behind the game … every other driver as before). On the base #665 (40edc9da) with
      this PR's test file: exit 1 on both (create_watched/compile_again: KosmicKrisp's failure code not read …, vk_vendor.h: compiler_service not MoltenVK's and KosmicKrisp's alone …, vk_draw.cpp: no service_failed …).
    • test_warm_handoff, test_vendor_fixes and test_worker_caches, arm64 and x86_64: exit 0 on all six
      (test_vendor_fixes is #665's, unchanged here: the KosmicKrisp row is #698's, a sibling item).
    • gpu/vk/vk_draw.cpp compiled whole for arm64 and x86_64 with main's renderer flags (#476's, #492's and #490's macOS compile
      pieces merged in a scratch tree only): exit 0 (…/StartTrio/compile/kk-compiler-service/).
  • The topic as written (macos/kk-compiler-service ecbeefa5, on the integration branch 79e2e006): the two changed tests
    failed on 79e2e006 and passed at ecbeefa5, with 8 related checks passing (test_precompile_handoff, test_vendor_fixes,
    test_warm_handoff, test_loading_stall, test_retry_alone, test_giant_memory, test_worker_caches,
    test_index8_bind); vk_draw.cpp compiled for arm64 and x86_64 (reports/KkSlow-files/fix/).
  • Off Apple (unifdef -U__APPLE__, base against head): vk_draw.cpp and vk_vendor.h differ: service_failed is compiled
    everywhere and returns every code as it is unless the driver is KosmicKrisp's, and the vendor keying names KosmicKrisp's
    driver id, which no Windows or Linux device reports
    (reports/PubWave-files/PubLate5/StartTrio/offapple-kk-compiler-service.txt).
  • KK-03 as in Measurements (history, standalone, both drivers).
  • Probe KK-04 (frozen packet, review a4f716dc…, build 129a73de…; both routes released once by the parent, 05:02-05:03 CST):
    reports/KkSlow-files/kk04/score-kk04/score.json (sha256 bb0b1e2747d4d20f0385a541a1b1bfeb4b1f7f85cb1815d70a4943acb9525ef7)
    K04-1 to K04-5 MATCH, routes runs-kk04-kk/run.json (5d5e98d4…) and runs-kk04-mvk/run.json (8c3d0873…): the code
    service_failed reads is the one KosmicKrisp gives (K04-1), MoltenVK's is the one compile_again already took (K04-2), and
    each cache behaves as the comments in compile_again and service_failed say (K04-4, K04-5).

Not tested: a game start with KosmicKrisp and this PR; Windows and Linux (no behaviour changes there: the new code returns
at its first test off KosmicKrisp): not built there; its test request covers it. KK-04 has passed, so no probe holds this PR in
WIP any more; the Testing box waits for the test requests.

Must be tested:

  • macOS, Apple silicon, on the rebased macOS stack: a start with RX_VK_DRIVER=kosmickrisp in a KosmicKrisp-only cache folder,
    expecting vk: preparation: the compute programs are translated and kept …, a preparation of about #19's length, then a
    precompile that hands off to the title (by hand: the parent's game run)
  • Windows and Linux, any GPU: build this PR (head a4d31a59, it carries #665, #664, #652, #650 and #651) and #665's head the
    same way and start each to the title, expecting every line to match #665's (no KosmicKrisp: line, the same
    vendor fixes line) (by hand: test request T-XXXX)

Risks

What could break: only starts on KosmicKrisp (opt-in): its compute pipelines move from the preparation to the precompile
and behind the game, as MoltenVK's did, and its failed compiles are tried again. A KosmicKrisp failure for another reason
(a program its MSL cannot express) is tried once more and then said not made, as before but one compile later. KK-03 found
KosmicKrisp 2-3.4× MoltenVK per graphics pipeline and up to 5.8× per large compute pipeline: its first start stays long;
MoltenVK stays the default.

Turning it off: RX_VK_COMPILER_SERVICE=0, or revert the PR.

Compatibility: Nothing changes for players: KosmicKrisp is opt-in, and on every other driver the code is as before.
Neither changed file is a shader-stamp input, so no program is translated again; no MKWVK_API_VERSION change.

Reviewer notes

  • Review 40edc9da..a4d31a59, one commit: macos/kk-compiler-service ecbeefa5 picked with -x. Read service_failed and
    its two calls first, then vendor_fixes.
  • Depends on #665 (its compile_again, create_watched with Retry::Fresh/Later and compile_failed_later: the tries again
    the mapping feeds, which the test exercises) and what #665 carries (#664, #652, #650, #651). Not on #666 or #669: their tries
    alone and gates apply to KosmicKrisp unchanged once both land. It needs no code of #654 (KosmicKrisp's packaging): only the
    driver id, from the Vulkan headers (1.4.345).
  • The pick's conflicts were adjacency only: vk_vendor.h (main's and the integration branch's neighbouring fields; only this
    topic's line kept), docs/development.md (#665's text of the two rows, with this topic's words). Its code and its
    test_precompile_handoff.py changes are the topic's line for line.
  • Sibling item, with #698 (kk-tex-order): whichever lands second updates tests/test_vendor_fixes.py's
    check_apple_drivers: KosmicKrisp's Apple columns 1000000011 → 1000000111 (compiler_service on), its comment, the
    module docstring and the ok: line, as the topic has them (ecbeefa5). That check is #698's; this PR's base does not have it,
    so the pick leaves it out.
  • Sibling conflicts (git merge-tree): docs/development.md's RX_VK_COMPILER_SERVICE row with #666, #667, #669 and the PRs
    that carry them (keep their text, with this PR's "MoltenVK and KosmicKrisp" and its failure-code clause); gpu/vk/vk_vendor.h
    with #784 (neighbouring fields: keep both).
  • Conflicts with main (35465d12): inherited from the base (vk_draw.cpp, test_pipeline_play_cache.py); none of this PR's own.
  • Code comments cite reports/2026-11-14-*.md (the port's design notes), which are not in this repository.

Issues

See #420

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how. (Test requests not open yet; KK-04 passed: no probe holds it WIP.)
  • Rebased on current main. (On main e090cc48 through #665 40edc9da; main has moved since (35465d12): a delta sweep follows.)
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). (On wave S c08ae52d = #784 694edb10 with #786 5c37ec10 and this head a4d31a59 merged: 586 passed, 31 failed, 18 skipped against main e090cc48's 544/32/17 on this Mac; the 31 failures are main's own and nothing that passes on main fails; test_collect_wide now passes; 41 new tests pass; reports/PubWave-files/PubLate5/wave/full-S-c08ae52d/.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • A game-free test added for the fix, failing on main and passing here. (It fails on this PR's base, #665, with this PR's test file; main has no compiler service path.)
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.
**Type:** fix · **Risk:** low · **Tested on:** macOS 26.7.1, Apple M4 Max (MoltenVK 1.4.2; KosmicKrisp from the LunarG SDK 1.4.363.1 in probes KK-03 and KK-04), game-free on arm64 and x86_64 under Rosetta · **Needs:** macOS, Apple silicon (a start with KosmicKrisp); Windows and Linux, any GPU (build and start) ## Why On a Mac, KosmicKrisp (Mesa's Vulkan driver on Metal 4, opt-in since #654: `RX_VK_DRIVER=kosmickrisp`) compiles every pipeline in Metal's compiler service, as MoltenVK does, but the game treated it as any other driver: its shader preparation compiled every compute program's pipelines itself, the place where nothing can hand a slow compile to the game. The first start with KosmicKrisp (start #20) reached 1,607 of 38,809 programs in 29.5 minutes and never showed the title; on MoltenVK (start #19, Integration 27) the preparation translated all 38,809 in 75 s and the precompile compiled them, then the game started. This PR gives KosmicKrisp the same compiler-service path as MoltenVK: the preparation only translates, the precompile compiles (a few giants at a time, the rest behind the game), and a compile that Metal's service failed is tried again. It is part of the macOS port. See #420. ## How to reproduce **Seen on:** macOS 26.7.1, Apple M4 Max, KosmicKrisp from the LunarG Vulkan SDK 1.4.363.1 (Mesa 269a955c); main does not start on macOS yet: seen on the macOS port's integration build 09f9a171, which carries the open macOS PRs **Settings and switches:** `RX_VK_DRIVER=kosmickrisp` (start #20 chose it by itself: that build picked KosmicKrisp on macOS 26; MoltenVK has been the default since) **Steps:** start the game with KosmicKrisp and a translated program cache missing (a new translator stamp), and watch the `vk: start-up: N of 38809` lines. **Expected:** the preparation translates the programs in about a minute, then the precompile compiles their pipelines and hands the rest to the game, as on MoltenVK (start #19). **Actual:** the preparation compiles every compute program's pipelines itself: 0.9 programs a second, 1,607 of 38,809 after 29.5 minutes, no title (`reports/2026-11-14-mac-startup-20.md`, `reports/2026-11-14-kk-slow.md` §3). ## What changed On KosmicKrisp's driver (`VK_DRIVER_ID_MESA_KOSMICKRISP`) `compiler_service` is on by default, as on MoltenVK's, and its failed-compile code is read as MoltenVK's where the service's compiles are made. Every other driver, MoltenVK's included, runs the code it ran before. - `gpu/vk/vk_vendor.h` `vendor_fixes`: `compiler_service` on for MoltenVK's and KosmicKrisp's drivers; `RX_VK_COMPILER_SERVICE=0` still turns it off. The comment carries KK-03's numbers (below). - `gpu/vk/vk_draw.cpp` `service_failed`: KosmicKrisp reports every Metal library or pipeline it could not make, a compiler service that ended under it included (start #20 had three), as `VK_ERROR_INVALID_SHADER_NV`. Where the service's compiles are made (`create_watched` with `compiler_service` on, and `compile_again`'s result) that code is read as `VK_ERROR_INITIALIZATION_FAILED`, MoltenVK's, on KosmicKrisp only, and said once (`vk: KosmicKrisp: a pipeline compile failed (VK_ERROR_INVALID_SHADER_NV: …); read as VK_ERROR_INITIALIZATION_FAILED from here on …`). So #665's tries again apply unchanged: once more with a new pipeline cache, or a giant's alone. KosmicKrisp's pipeline cache keeps no failed compile (Mesa `vk_pipeline.c`), so the new cache costs it the one compile alone. `compile_again`'s line still prints the driver's own code. The comments on the section and on `prepare_run`'s warm name KosmicKrisp and start #20. - `docs/development.md`: the `RX_VK_COMPILER_SERVICE` and `RX_VK_PRECOMPILE_STALL_S` rows. - `tests/test_precompile_handoff.py`: `service_failed`, `compile_again` and `create_watched` compiled from the source against a made-up KosmicKrisp (its failed compile is `VK_ERROR_INVALID_SHADER_NV`, its cache keeps none): a compile that failed once is made by the second try and said once; one that always fails is tried twice and said `not made (-1000012000)`; a giant's failure is left for its try alone; on MoltenVK's driver the same code is returned as it is; on KosmicKrisp without `compiler_service` too. The vendor keying (KosmicKrisp on, the switch off on it, HoneyKrisp off), and the source's wiring (`service_failed(` three times: its definition and the two places). ## Measurements **Verdict:** the cause is measured, the fix's start is not yet. Probe KK-03 measured KosmicKrisp's compiles on identical synthetic SPIR-V against MoltenVK's: 1.0× MoltenVK's at 3,003 words, 2.5× at 29,310, 5.8× at 110,406 (2.4× at the middle size with robustness off on both), 16 at once 4.7×; its pipeline cache saves 8% in a new process, Metal's own cache 94-95% on both drivers. So the compiler-service path that keeps MoltenVK's giants out of the preparation is needed on KosmicKrisp at least as much. Probe KK-04 confirmed the driver facts the fix keys on (5 of 5 predictions): for a module Metal cannot compile, KosmicKrisp returns `VK_ERROR_INVALID_SHADER_NV` both times and compiles it again on the second ask (its cache keeps no failure), while MoltenVK returns `VK_ERROR_INITIALIZATION_FAILED` and answers the second ask from its cache in 0.0 ms (why `compile_again` takes a new cache). A game start with this PR is in the table's After column. Baseline: main does not start on macOS. The "before" is the integration build 09f9a171 (start #20, KosmicKrisp, without this change; MoltenVK's start #19 on Integration 27 for comparison); KK-03 compares the two drivers in one standalone process each, with no LS code. Hardware: macOS 26.7.1 (25G241), Apple M4 Max (40-core GPU), 16-core CPU, 64 GB; KosmicKrisp from the LunarG SDK 1.4.363.1 (Mesa 269a955c), MoltenVK 1.4.2, both through the SDK's loader Scene: start #20: the preparation from a program cache to translate again in full (a new stamp), windowless; KK-03: one compute module each of 3,003, 29,310 and 110,406 SPIR-V words, cold and warm Metal caches, 16 at once Method: start #20: the `vk: start-up: N of 38809` readings every 60 s; KK-03: the Vulkan call's time per compile in each phase's `probe.log` (`reports/KkSlow-files/kk03/runs-kk03-kk/run.json` `449dfd7b…`, `runs-kk03-mvk/run.json` `7acdbc04…`), one run per route; KK-04: the result and time of each `vkCreateComputePipelines` for a 300-deep (past Metal's bracket nesting of 256) and a 200-deep module, twice each with the same `VkPipelineCache`, one run per route, released by the parent at 05:02-05:03 CST How to reproduce: KK-03's `run` commands (`reports/KkSlow-files/kk03/FROZEN-COMMAND.md`); KK-04's `run` and `score` commands (`reports/2026-11-14-kk-slow.md` §5.5); a start with `RX_VK_DRIVER=kosmickrisp` reading the preparation's and the precompile's lines | | Before: integration @ 09f9a171 (start #20, KosmicKrisp) | After: this PR @ a4d31a59 | |---|---|---| | the preparation (38,809 programs) | 1,607 after 1,770 s (0.9 a second), compute pipelines compiled there; no title | (E-series numbers: pending) | | KK-03, the Vulkan call's ms (cold Metal cache) | KosmicKrisp | MoltenVK 1.4.2 | KosmicKrisp / MoltenVK | |---|---|---|---| | 3,003 SPIR-V words | 38.8 | 38.4 | 1.01 | | 29,310 | 456.6 | 185.3 | 2.46 | | 110,406 | 4,570.3 | 787.2 | 5.81 | | 16 new 29,310-word modules at once, wall | 2,094 | 449 | 4.66 | | warm Metal cache, 110,406 words | 276.1 | 39.4 | | | KK-04 (VkResult, ms of the call) | KosmicKrisp | MoltenVK 1.4.2 | |---|---|---| | `nest-300` (Metal cannot compile it), first | -1000012000 (`VK_ERROR_INVALID_SHADER_NV`), 190.1 | -3 (`VK_ERROR_INITIALIZATION_FAILED`), 14.9 | | `nest-300`, again, same cache | -1000012000, 118.5 (compiled again) | -3, 0.0 (the failure from its cache) | | `nest-200`, first / again | 0, 422.9 / 0, 0.0 | 0, 69.2 / 0, 0.1 | ``` #20, the watch's readings of `vk: start-up: N of 38809`: N = 38 at 11:17:23.6, 1,607 at 11:46:32.9 (the preparation began 11:17:13.5) #19 (MoltenVK, Integration 27): 38,809 translated in 74.8 s, then 42,234 precompile items in 626.5 s KK-03: ROUTE route=kk safe=1 complete=1 phases=6/6 compiler_crash_reports=0; ROUTE route=mvk safe=1 complete=1 phases=6/6 compiler_crash_reports=0; COMPARE predictions=9/12 KK-04: ROUTE route=kk safe=1 exit=1 nest-300#1=-1000012000/190.1ms nest-300#2=-1000012000/118.5ms nest-200#1=0/422.9ms nest-200#2=0/0.0ms nesting_lines=2 new_reports=0 KK-04: ROUTE route=mvk safe=1 exit=1 nest-300#1=-3/14.9ms nest-300#2=-3/0.0ms nest-200#1=0/69.2ms nest-200#2=0/0.1ms nesting_lines=1 new_reports=0 KK-04: reports/KkSlow-files/kk04/score-kk04/score.json (sha256 bb0b1e2747d4d20f0385a541a1b1bfeb4b1f7f85cb1815d70a4943acb9525ef7): K04-1 … K04-5 MATCH (the probe exits 1 on each route because nest-300 fails on purpose) ``` Reports: `reports/2026-11-14-kk-slow.md` (KosmicKrisp's compile path from the SDK's Mesa source, KK-03, the fix, KK-04), `reports/2026-11-14-mac-startup-20.md`. ### Flags and settings One default changes, on KosmicKrisp only: `compiler_service`, which `RX_VK_COMPILER_SERVICE` overrides. The diff against main also shows the switches of the PRs it stacks on, theirs to list: `RX_VK_COMPILE_QOS`, `RX_VK_GIANT_COMPILES`, `RX_VK_GIANT_HOLD_PRESENTS`, `RX_VK_GIANT_HOLD_S`, `RX_VK_GIANT_WORDS`, `RX_VK_SLOW_COMPILE_S`, `RX_VK_WORKER_CACHES`. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `RX_VK_COMPILER_SERVICE=0` or `1` | where Metal's compiler service compiles: the preparation translates, the precompile compiles and hands the rest to the game, a failed compile tried again | on by default on MoltenVK only | on by default on MoltenVK and KosmicKrisp | on KosmicKrisp the preparation no longer compiles the compute pipelines; failed compiles there are tried again | helps: start #20's shape (KK-03: KosmicKrisp's compiles 1-5.8× MoltenVK's, which needs it); untested: a KosmicKrisp start with it | as it ships (on for both Apple drivers); KosmicKrisp itself stays opt-in | ## Testing **Tested:** on macOS 26.7.1, Apple M4 Max: - At this head (main e090cc48 plus #650, #651, #652, #664, #665 and this PR), every run under the shared check slot with a fresh HOME and TMPDIR (`reports/PubWave-files/PubLate5/StartTrio/proof/kk-*`): - `tests/test_precompile_handoff.py --headers <Vulkan-Headers 1.4.345>`, `--arch arm64` and `--arch x86_64` (Rosetta): exit 0 on both (`PASS: the precompile goes on behind the game … every other driver as before`). On the base #665 (`40edc9da`) with this PR's test file: exit 1 on both (`create_watched/compile_again: KosmicKrisp's failure code not read …`, `vk_vendor.h: compiler_service not MoltenVK's and KosmicKrisp's alone …`, `vk_draw.cpp: no service_failed …`). - `test_warm_handoff`, `test_vendor_fixes` and `test_worker_caches`, arm64 and x86_64: exit 0 on all six (`test_vendor_fixes` is #665's, unchanged here: the KosmicKrisp row is #698's, a sibling item). - `gpu/vk/vk_draw.cpp` compiled whole for arm64 and x86_64 with main's renderer flags (#476's, #492's and #490's macOS compile pieces merged in a scratch tree only): exit 0 (`…/StartTrio/compile/kk-compiler-service/`). - The topic as written (`macos/kk-compiler-service` `ecbeefa5`, on the integration branch `79e2e006`): the two changed tests failed on `79e2e006` and passed at `ecbeefa5`, with 8 related checks passing (`test_precompile_handoff`, `test_vendor_fixes`, `test_warm_handoff`, `test_loading_stall`, `test_retry_alone`, `test_giant_memory`, `test_worker_caches`, `test_index8_bind`); `vk_draw.cpp` compiled for arm64 and x86_64 (`reports/KkSlow-files/fix/`). - Off Apple (`unifdef -U__APPLE__`, base against head): `vk_draw.cpp` and `vk_vendor.h` differ: `service_failed` is compiled everywhere and returns every code as it is unless the driver is KosmicKrisp's, and the vendor keying names KosmicKrisp's driver id, which no Windows or Linux device reports (`reports/PubWave-files/PubLate5/StartTrio/offapple-kk-compiler-service.txt`). - KK-03 as in Measurements (history, standalone, both drivers). - Probe KK-04 (frozen packet, review `a4f716dc…`, build `129a73de…`; both routes released once by the parent, 05:02-05:03 CST): `reports/KkSlow-files/kk04/score-kk04/score.json` (sha256 `bb0b1e2747d4d20f0385a541a1b1bfeb4b1f7f85cb1815d70a4943acb9525ef7`) K04-1 to K04-5 MATCH, routes `runs-kk04-kk/run.json` (`5d5e98d4…`) and `runs-kk04-mvk/run.json` (`8c3d0873…`): the code `service_failed` reads is the one KosmicKrisp gives (K04-1), MoltenVK's is the one `compile_again` already took (K04-2), and each cache behaves as the comments in `compile_again` and `service_failed` say (K04-4, K04-5). **Not tested:** a game start with KosmicKrisp and this PR; Windows and Linux (no behaviour changes there: the new code returns at its first test off KosmicKrisp): not built there; its test request covers it. KK-04 has passed, so no probe holds this PR in WIP any more; the Testing box waits for the test requests. **Must be tested:** - macOS, Apple silicon, on the rebased macOS stack: a start with `RX_VK_DRIVER=kosmickrisp` in a KosmicKrisp-only cache folder, expecting `vk: preparation: the compute programs are translated and kept …`, a preparation of about #19's length, then a precompile that hands off to the title (by hand: the parent's game run) - Windows and Linux, any GPU: build this PR (head `a4d31a59`, it carries #665, #664, #652, #650 and #651) and #665's head the same way and start each to the title, expecting every line to match #665's (no `KosmicKrisp:` line, the same `vendor fixes` line) (by hand: test request T-XXXX) ## Risks **What could break:** only starts on KosmicKrisp (opt-in): its compute pipelines move from the preparation to the precompile and behind the game, as MoltenVK's did, and its failed compiles are tried again. A KosmicKrisp failure for another reason (a program its MSL cannot express) is tried once more and then said not made, as before but one compile later. KK-03 found KosmicKrisp 2-3.4× MoltenVK per graphics pipeline and up to 5.8× per large compute pipeline: its first start stays long; MoltenVK stays the default. **Turning it off:** `RX_VK_COMPILER_SERVICE=0`, or revert the PR. **Compatibility:** Nothing changes for players: KosmicKrisp is opt-in, and on every other driver the code is as before. Neither changed file is a shader-stamp input, so no program is translated again; no `MKWVK_API_VERSION` change. ## Reviewer notes - Review `40edc9da..a4d31a59`, one commit: `macos/kk-compiler-service` `ecbeefa5` picked with `-x`. Read `service_failed` and its two calls first, then `vendor_fixes`. - Depends on #665 (its `compile_again`, `create_watched` with `Retry::Fresh`/`Later` and `compile_failed_later`: the tries again the mapping feeds, which the test exercises) and what #665 carries (#664, #652, #650, #651). Not on #666 or #669: their tries alone and gates apply to KosmicKrisp unchanged once both land. It needs no code of #654 (KosmicKrisp's packaging): only the driver id, from the Vulkan headers (1.4.345). - The pick's conflicts were adjacency only: `vk_vendor.h` (main's and the integration branch's neighbouring fields; only this topic's line kept), `docs/development.md` (#665's text of the two rows, with this topic's words). Its code and its `test_precompile_handoff.py` changes are the topic's line for line. - **Sibling item, with #698 (kk-tex-order): whichever lands second** updates `tests/test_vendor_fixes.py`'s `check_apple_drivers`: KosmicKrisp's Apple columns `1000000011` → `1000000111` (`compiler_service` on), its comment, the module docstring and the `ok:` line, as the topic has them (`ecbeefa5`). That check is #698's; this PR's base does not have it, so the pick leaves it out. - Sibling conflicts (`git merge-tree`): `docs/development.md`'s `RX_VK_COMPILER_SERVICE` row with #666, #667, #669 and the PRs that carry them (keep their text, with this PR's "MoltenVK and KosmicKrisp" and its failure-code clause); `gpu/vk/vk_vendor.h` with #784 (neighbouring fields: keep both). - Conflicts with main (`35465d12`): inherited from the base (`vk_draw.cpp`, `test_pipeline_play_cache.py`); none of this PR's own. - Code comments cite `reports/2026-11-14-*.md` (the port's design notes), which are not in this repository. ## Issues See #420 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. (Test requests not open yet; KK-04 passed: no probe holds it WIP.) - [ ] Rebased on current `main`. (On main e090cc48 through #665 `40edc9da`; main has moved since (`35465d12`): a delta sweep follows.) - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (On wave S `c08ae52d` = #784 `694edb10` with #786 `5c37ec10` and this head `a4d31a59` merged: 586 passed, 31 failed, 18 skipped against main e090cc48's 544/32/17 on this Mac; the 31 failures are main's own and nothing that passes on main fails; `test_collect_wide` now passes; 41 new tests pass; `reports/PubWave-files/PubLate5/wave/full-S-c08ae52d/`.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A game-free test added for the fix, failing on `main` and passing here. (It fails on this PR's base, #665, with this PR's test file; main has no compiler service path.) - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.
prepare_run's first part (every shader translated, and each compute
variant compiled for the pipeline cache) ran on half the CPUs
everywhere: prepare_cpus gives half_cpus unless the Windows process heap
is the segment heap. Half exists for the NT heap's lock that an
in-process driver compiler queued on, and for whole-PC freezes when
those compilers ran on every CPU at High priority. Neither holds on
macOS: MoltenVK converts SPIR-V to MSL in a few ms and hands every
compile to Metal's compiler service (MTLCompilerService, a launchd
service at nice 0), so the workers mostly wait on it. The Mac's first
start logged "8 workers for the shader programs (16 CPUs usable; half
the CPUs)".

prepare_workers::count takes an `every` argument (default false): each
usable CPU gets a worker, at least half_cpus and 1, at most kMaxWorkers.
kEveryCpu is true where __APPLE__ is defined, and prepare_cpus passes
it; the preparation's log line says "every one: macOS". In a synthetic
MoltenVK 1.4.2 benchmark on an M4 Max (16 CPUs, macOS 26.7.1) 16 threads
compiled 56-61 % more compute and 35-39 % more graphics pipelines a
second than 8, but only with a pipeline cache per thread; with the one
shared cache LS uses today the rate is flat from 4 to 16 threads. The
next commit gives each worker its own cache.

tests/test_prepare_every_cpu.py compiles the real prepare_workers.h and
prepare_cpus() cut from vk_progcache.cpp: on this compiler's target
prepare_cpus() gives 16 of 16 usable CPUs on macOS (8 elsewhere); count
with and without `every`; prepare_cpus with kEveryCpu forced either way.
On the base it fails: prepare_cpus() gives 8 of 16 on macOS.

Windows and Linux: unchanged (kEveryCpu false; the existing
test_prepare_workers and test_prepare_cpus cases are untouched).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 3573e736e15078554543e4740c5e2d53c719fa89)
MoltenVK 1.4.2 holds one mutex per VkPipelineCache across the SPIR-V to
MSL conversion and Metal's library compile (MVKPipeline.mm
getShaderLibrary, _shaderCacheLock). Every worker of the start's
preparation compiled with the one start cache, pcache, so they compiled
one pipeline at a time: the Mac's first start (mac-arm64-a5f2d6f9-01,
M4 Max) logged strict pairs of Metal compiles every ~195 ms, 5.05
compute variants a second, an estimated 33 minutes for the compute
programs alone. A synthetic benchmark on that Mac (PrepSpeed, MoltenVK
1.4.2, macOS 26.7.1): one shared cache stays at 10.5 compute and 9.6
graphics pipelines a second from 4 to 16 threads; a cache per thread,
merged after, gives 31.5 / 29.7 at 8 threads and 49-51 / 40-41 at 16.

gpu/vk/worker_caches.h: a Set of empty caches, one per worker of a
phase, that a worker enters with Scope; create() asks the shared cache
first under VK_PIPELINE_CREATE_FAIL_ON_PIPELINE_COMPILE_REQUIRED_BIT
and compiles what it lacks with the worker's own; merge_into() merges
them into the start cache with one vkMergePipelineCaches in slot order
(MoltenVK retains each Metal library, no compile). The caches are not
externally synchronized, so a shared slot stays correct.

vk_draw.cpp:
- create_cached: before play, a worker in a phase with a set goes
  through worker_caches::create (pcache asked first, so a saved file's
  or an earlier phase's pipelines are found as before); every use of
  pcache holds pcache_mu shared, and worker_caches_merge holds it alone
  (vkMergePipelineCaches's dstCache is externally synchronized:
  vk.xml externsync without INTERNALLY_SYNCHRONIZED_MERGE). A save's
  reads of pcache's data hold it shared too.
- prepare_run (the compute warm), precompile_run, precompile_warm (both
  passes) and precompile_at_create make a set under worker_caches_on,
  enter it in their workers and merge it right after the join; the
  precompile's stragglers left running are merged as far as they got.
- precompile_run, with the caches on, sorts the notes by program
  variant and hands a variant's notes to one worker in runs of up to
  32 (worker_caches::runs/claim), so its shader libraries are compiled
  once, not by every worker that met one of its notes at the same time.
- In play, with the caches on, a compile goes into a new cache of its
  own merged into the generation's at once under the generation's lock
  (made_alone), which the asks of the generation hold shared: the asks
  keep their order and find what any thread compiled.

vk_vendor.h: worker_caches, on for VK_DRIVER_ID_MOLTENVK only, and only
with pipelineCreationCacheControl; RX_VK_WORKER_CACHES forces it either
way. Elsewhere nothing measured says a driver's cache is locked across
its compiles, and the caches would only add merges. The pipelines built,
the persisted cache's format and its stamp are the same either way.

tests/test_worker_caches.py runs vk_draw.cpp's own create_cached,
play_cache_make, worker_caches_on and worker_caches_merge and the real
header against a made-up driver whose caches lock across a compile:
four workers compile at once (without a set they cannot); pcache asked
first, nothing compiled with it; ask-only callers never compiled; caches
not externally synchronized; one merge in slot order whatever order the
slots were taken in, with no ask of pcache beside it; play's compiles
merged into the generation at once; runs and claims. The four-workers
part runs against the base's create_cached and fails there (1 of 4
compiles met the others). tests/test_pipeline_play_cache.py gets an
"alone" case, and both play-cache and cache-save harnesses the new
stand-ins (pcache_mu, worker_caches_on).

Windows and Linux: the same code paths as before (no set, one shared
cache), plus an uncontended shared lock around pcache's uses.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 2174e33f14ccce8c1635565e734773a33bc32455)
tests/test_recorder_sync.py holds every header to no recorder boundary
call: vk_work.h, included last by each renderer .cpp, turns
vkDestroyPipelineCache into a drain of RX_VK_RECORDER's packets and then
the call, and a header's calls would be wrapped or not depending on what
included it. worker_caches.h called vkCreatePipelineCache,
vkDestroyPipelineCache and vkMergePipelineCaches itself, from vk_draw.cpp
before vk_work.h.

worker_caches::Set and made_alone now take a Calls struct with the
three operations; vk_draw.cpp defines worker_cache_calls after
vk_work.h, so a set's cache destruction goes through the recorder's
boundary like every other vkDestroy* in the file. Behaviour is
otherwise the same: the same caches, made, merged and destroyed in the
same order.

tests/test_worker_caches.py and tests/pipeline_play_cache_test.cpp take
worker_cache_calls from the source and the model respectively.
test_recorder_sync now fails only as on the base (vk_vma.cpp's
vmaDestroyAllocator), and the four tests that touch the caches pass.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit aa8ecba546465f1ce65ac872ecf21d232b65cef0)
MoltenVK 1.4.2 keeps MSL, not GPU code, in a VkPipelineCache, and
vkCreatePipelineCache with saved data compiles every stored library one
after another on the calling thread (MVKPipeline.mm readData).
pcache_init hands it all of mkw_vk_pipelines.bin, and play_files_merge
every play file, at every start. The PrepFast GPU check on the M4 Max,
with Metal's per-user cache blocked as in the runner, ran three later
starts of the same 64 pipelines (80 libraries):
- reading the one file: 5.7 s of serial compiles, 6.6 s in all;
- 16 shards read in parallel: 1.4 s;
- nothing read, pipelines compiled from SPIR-V with the worker caches:
  1.4 s.
At the game's scale the read is tens of minutes per start.

vk_vendor.h gets pcache_no_load, on for VK_DRIVER_ID_MOLTENVK only and
logged in the vendor fixes' line with worker_caches. It is the device's
default for RX_VK_PCACHE_LOAD: pcache_load_off reads the switch first
(any value, as before) and falls back to the fix, so
RX_VK_PCACHE_LOAD=1 reads the files on MoltenVK as everywhere else. With
the fix on and the switch unset or off, pcache_files_unused holds:
- pcache_init reads neither the main file nor the play files and says
  so in the log; the cache lives in memory for the run;
- nothing is saved (pcache_no_save, as RX_VK_PCACHE_LOAD=0 already
  did), and play_cache_save writes no play file and keeps the
  generation in memory: unread play files would pile up, since they
  are compacted only where they are read;
- the stamp is left alone (pcache_drop_stale returns as for any
  start that does not read the files);
- precompile_run drops RX_VK_LOAD=create's longshot_pipelines.bin.built
  marker at each start, as it does when the files were dropped: no
  cache a later start reads holds the notes, so the warm builds them
  into the in-memory cache again before loads at create ask for them.

Saving does not continue on MoltenVK because nothing reads the file or
its stamp there:
- the prepared record (prepared_for) has its own file and stamp;
- --prepare-shaders' saved cache is useful only to a start that reads
  it;
- pcache_saved_now serves graphics pipeline libraries, which MoltenVK
  does not have;
- a start with RX_VK_PCACHE_LOAD=1 makes, saves and stamps its own.
Every other driver is unchanged, RX_VK_PCACHE_LOAD=0 there included
(main file unread and unsaved, play files still merged and written).

tests/test_vendor_fixes.py checks the two cache fixes on its 16 devices
and on MoltenVK: on for MoltenVK alone, RX_VK_WORKER_CACHES forcing
the first, vendor_fixes_env leaving pcache_no_load to the files'
switch. It also checks the log line and pcache_load_off's order.
tests/test_pcache_stamp.py runs pcache_load_off, pcache_files_unused,
pcache_drop_stale and pcache_init, cut from the source, in eight starts:
other drivers unset, 0, empty and 1 as before, and MoltenVK unset, 0,
off and 1. test_pipeline_play_cache gets an "unused" case. All three
fail on aa8ecba5. tests/test_warm_bounded.py's source check of the
marker's removal takes the new condition.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 61bfbf1ae7c0966a9436ab0c556a9ee547bb3b5f)
macOS starts a std::thread with 512 KiB of stack, where Linux starts one
with 8 MiB and Windows with longshot.exe's 8 MiB. On macOS MoltenVK
converts each shader with SPIRV-Cross on the thread that creates the
pipeline, recursing once per block of a program's control flow, and a
prepare_run worker overran its 512 KiB there (CFG::walk_from under
vkCreateComputePipelines from seed_compute, start #6, 2026-11-14).

gpu/vk/thread_stack.h: thread_stack::thread is std::thread itself off
macOS; on macOS a std::thread-like thread (construction with decayed
arguments, joinable, join, detach, native_handle, moves) started by
pthread_create with thread_stack::kBytes (16 MiB) of stack. Every
thread the renderer starts goes through it: the preparation's pools
(prepare_run, state_list_prepare, warm_libraries, precompile_at_create,
precompile_run, predict_run, warm_bounded::run), the job workers and
the serial builder, the replay, parallel-replay, presenter and
frame-generation threads, the program cache's loaders and the rest.
mkwvk_init logs the size once per start on macOS.

tests/test_thread_stack.py: no file under gpu/vk starts a thread any
other way; each kind of thread runs a recursion 2 MiB deep through the
helper, and on macOS each overflows as a std::thread (caught, no crash
report). Seven source checks follow the new spelling.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 48b49054b1c64851b3e8c974e64dc0b757b2ced3)
The first thread whose stack overflows says so and ends the process;
the others wait in their handlers, so the message is not interleaved.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 62d0991e0cfb5402202f8f14e9eb799d5c20d262)
test: the checks that compile renderer code with its threads see thread_stack.h
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 16s
CI / Conventional Commit title (pull_request) Successful in 14s
CI / Workflow lint (actionlint) (pull_request) Successful in 19s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 18s
Third-party parts / As their notes say (pull_request) Successful in 15s
01ffd9a938
Eight checks cut renderer functions that start threads (or copy
vk_jobs.cpp and vk_recorder.h) into fixtures of their own; they now
build with gpu/vk/thread_stack.h beside them: test_capture_copy_helpers,
test_gpu_hang_watch, test_jobs_failure, test_jobs_wait_runs_queued,
test_pipeline_cache_save, test_prepare_stall, test_present_source_crop
and test_recorder_sync.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit fd46ff431f9f62b1125582a4a3434a8cc92c551e)
fix(macos): the worker caches follow main's fetch lock (#190) and Intel's gate (#252)
Some checks failed
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 17s
CI / Conventional Commit title (pull_request) Successful in 15s
CI / Workflow lint (actionlint) (pull_request) Successful in 22s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 20s
Third-party parts / As their notes say (pull_request) Successful in 27s
eee107e2f1
Main's #190 (f41a5c59) put a fetch lock (PcacheFetch, PcacheUse) on the pipeline cache's save and compile path, the
very path where the worker caches add the lock their merges need, and #252 (57f4eaec) put Intel's compiler gate
(IntelDriverGate) around the compiles. The worker caches now sit on top of both:

- pcache_mu, the merge lock, is taken only where the worker caches are on (pcache_use_held: MoltenVK by default).
  Elsewhere create_cached, the save's fetch and pcache_saved_now take exactly #190's locks, as on main: no new lock
  on any other driver. A play generation's lock is likewise taken only where play's compiles are merged into it
  (g->alone).
- The worker caches are never on with the fetch lock (worker_caches_on: not with pcache_fetch_lock_on, Intel's
  Windows driver or RX_VK_PCACHE_FETCH_LOCK=1): a merge reads its caches' data as a fetch does, which that lock keeps
  apart from the builds there, and the gate lets one compile at a time into Intel's compiler there anyway.
- The order of the locks: Intel's gate, then pcache_mu, then the fetch lock. The save's fetch and pcache_saved_now
  take pcache_mu before PcacheFetch; a merge takes pcache_mu alone.
- prepare_run's compute warm keeps #252's gate around its compile and, with the caches, goes through create_cached
  under it (cached_compute would take the gate a second time).
- tests: test_worker_caches checks the gating (with the caches off a pipeline is made while pcache_mu or the
  generation's lock is held alone elsewhere; with them on it waits for pcache_mu), worker_caches_on against the
  fetch lock, the warm's gate and pcache_mu ahead of PcacheFetch in the save. test_pipeline_play_cache's route case
  checks that no lock of theirs is taken off MoltenVK; pipeline_play_cache_test.cpp and pipeline_cache_save_test.cpp
  get pcache_use_held stand-ins. test_pcache_stamp's load fixture follows #321's cache_path and #190's PcacheFetch in
  pcache_init, and play_cache_save's new parameters.
- docs: #321's cache/longshot_vk_pipelines.bin in the RX_VK_WORKER_CACHES row and in vk_vendor.h, and a row for
  RX_VK_PCACHE_LOAD, whose default this PR changes on MoltenVK.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The precompile hand-off starts its workers and its compile watcher through thread_stack::thread. One conflict file,
gpu/vk/vk_draw.cpp (4 blocks: the lines P31 had to resolve against main, the other way round): the include block keeps prepare_workers.h,
worker_caches.h and thread_stack.h; prepare_run's, precompile_at_create's and precompile_run's thread lines take
thread_stack::thread and keep the worker caches' merges beside them.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
On the Mac's start 12 every precompile worker was held by a Metal compile of minutes while 37,600 of
42,234 items were still to hand out, so RX_VK_PRECOMPILE_STALL_S could not apply (it waits until every
item is handed out) and the title never came. With the new vendor fix compiler_service (MoltenVK only,
RX_VK_COMPILER_SERVICE):
- the precompile hands the rest to the game once nothing has finished for the stall time, or once all
  that is left is compiles running past RX_VK_SLOW_COMPILE_S; a quarter of the workers build the rest
  behind the game, every item still built, each straggler named when it ends;
- its order: noted pipelines first, then fallbacks and computes by SPIR-V words, what an earlier run
  found slow (longshot_pipelines.bin.slow) last;
- a compile over the slow time (10 s) is named with key, variant, kind, stage and SPIR-V words while it
  runs and when it ends, and written to the slow list;
- a pipeline whose compile failed is compiled once more with a new pipeline cache (MoltenVK keeps a
  failed library in the cache it was compiled with), and said made or not made.
Every other driver: the classic worker, wait and log line as they were.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 77d4971cbdb6356d9064a44cdee923f4bde354f9)
Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit f9a179c08d3f9e102729ec8fe5fe923336f581b8)
Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 194a230dd959706797203501edad3b5a2331118a)
test_fallback_pipeline_state compiles build_graphics cut from the source; the noted pipelines of behind_worker and precompile_warm name what they build instead.

Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 528784dc5c0c293da1b692f6fd71da070a940d8b)
fix(vk): the precompile hand-off on main's Intel gate and quick builds (#252), fetch lock (#190) and game folder (#321)
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 5s
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s
Third-party parts / As their notes say (pull_request) Successful in 12s
02cb2bc35f
The hand-off was written on the macOS integration branch, before main's #190 (f41a5c59), #252 (57f4eaec) and #321
(76699aad), and before the worker caches followed them (this branch's base). What it needs on main's code:

- cached_graphics and cached_compute keep #252's gate and main's `return create_cached(guarded_*_held, ci, pl);`, and go
  through create_watched only where compiles are watched (compiles_watched: compiler_service, MoltenVK's, or
  RX_VK_SLOW_COMPILE_S set). Every other driver makes its pipelines exactly as on main, and main's own source checks
  (test_intel_compiler_gate, test_pcache_fetch_lock) read the lines they read there.
- prepare_run's compute warm: with the worker caches and watched compiles through create_watched, under the gate it holds;
  otherwise P22's create_cached or main's guarded_compute_held, as they are.
- behind_worker, the precompile's worker under compiler_service, builds a compute pipeline as the classic worker now
  does (#252): on Intel's one-at-a-time builds (serial_builds_on) without the optimizer first, the optimized build owed
  to its first dispatch (precompile_compute_quick, optimize_owed). On MoltenVK, where compiler_service is on, that never
  applies unless RX_VK_COMPUTE_SERIAL=1; it keeps the two workers' builds the same.
- test_precompile_handoff's source checks follow main's spellings: the watched and plain lines in cached_* after the gate,
  compiles_watched, the warm, the dispatch's inline build named as main spells it (p, not the integration branch's plain),
  and behind_worker's quick build.
- docs: the slow list is cache/longshot_pipelines.bin.slow since #321 moved the notes into the game folder's cache/.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
On the Mac's cold start 14, prepare_run's compute warm kept 13 of its 16 workers on compute compiles of
122k-209k SPIR-V words for 205-274 s each. The preparation reached 278 of 38,809 programs in 430 s, and the
precompile, whose hand-off lets the title start while such compiles go on behind it, never began.

With compiler_service (MoltenVK only) the warm compiles nothing. Each compute variant is still translated and
kept by seed_compute (program_cache_save), and the state list and the prepared record follow as before. The
precompile compiles the pipelines: every kept compute program on a first start (computes_plan), in its order
and with its hand-off; one it does not take is compiled at its first dispatch. Every other driver's warm is
unchanged.

tests/test_warm_handoff.py runs prepare_run's own decision and warm_compute against stand-ins on both arches,
and pins the source chain that keeps each variant for the precompile.

Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 84c32d68c536468b61af24230ea8c283c5f091ef)
Start 14 had 13 compute compiles of 122k-209k SPIR-V words in flight at once while the other programs went
at 0.1-0.3 a second, and Metal's compiler service aborted twice (LLVM bad_alloc).

precompile_handoff.h Giants: at most RX_VK_GIANT_COMPILES (4) compiles of more than RX_VK_GIANT_WORDS
(100,000) SPIR-V words in flight at once among behind_worker's workers, before the game and behind it. A
giant claimed while the bound is reached is deferred, never dropped, and named once with a count; its
worker takes the next item, and the deferred ones are taken first once a giant ends. A worker with nothing
else waits for them and leaves only when none is deferred. behind_worker claims with claim_below, which
never passes the end, and a worker adds its one "ran out" to next as it leaves, so the wait's
next >= total + threads still means every worker has left. The classic worker (every other driver) is
unchanged.

The precompile_handoff fixture runs the gate with blocking giants: at most 2 of 6 in flight, the small
items going on meanwhile, each deferral named once with its count, every item built once, before the game
and behind it with the hand-off.

Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit f1e028c8db05ef31383fa44c05fa156984c0f6c9)
Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 8aaa2f33d51b6b6f8db335cac69639137833537e)
The bound on giant compiles in flight is now min(RX_VK_GIANT_COMPILES, workers in the phase - 1), and at
least 1. Before the hand-off the phase is the precompile's workers (16 on an M4 Max: 4); at the hand-off
precompile_run sets it to the share behind the game (4 workers: 3), so one worker there always stays on the
smaller items. Giants already in flight over the new bound finish; no more start until they are under it.
RX_VK_GIANT_COMPILES=0 is still no bound. The order line, the hand-off line and each deferral line give the
bound in use; docs/development.md says so.

The precompile_handoff fixture gains a hand-off by the stall time with three giants (4 workers: 3) and a
stuck small item holding every worker and most of the list still to hand out: behind the game, on a share
of 2, one giant at a time and the other worker on the small items, every item built once. Plus limit_for
and phase.

Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit acd3a71a2717b1e1bfc83c1952932fdcf1afd9de)
test(vk): test_warm_handoff follows main's warm and dispatch (#252, #190)
All checks were successful
PR description / Description follows the template (pull_request_target) Successful in 8s
658b854052
The fixture cuts prepare_run's warm out of vk_draw.cpp. On main the warm builds
under Intel's compiler gate (#252 IntelDriverGate) through guarded_compute_held,
with the worker caches through create_cached (#190, PrepFast) or create_watched
where compiles are watched (PrecompileStall's compiles_watched). The fixture now
has stand-ins for those; without worker caches, as there, the warm still goes to
guarded_compute_held into pcache and is counted the same way.

The source check for a dispatch's build of an unmade compute pipeline reads
main's name for the dispatch's program, p (the topic's tree had a later
rename to plain).

Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Run 17 (MoltenVK 1.4.2, M4 Max): the game thread's first vkQueueSubmit2 waited about four minutes in
MVKCmdBufferImageCopy::encode. The AGX driver compiled a blit program variant for the copy synchronously
(findOrCreateBlitProgramVariant -> MTLCompilerScheduler::buildRequest -> MTLXPCCompilerConnection::tryBoost ->
dispatch_sync), and its request waited on one of Metal's 16 compiler connections behind a giant compute compile
that ended only when the compiler service, crashing under 16 of them, gave up on it.

Under compiler_service (MoltenVK) only:
- Giants::hold: from the hand-off, no compile of more than RX_VK_GIANT_WORDS starts until the game has shown
  RX_VK_GIANT_HOLD_PRESENTS frames (1), or RX_VK_GIANT_HOLD_S (120 s) at most; the ones in flight finish, the
  deferred ones go first after it. Nothing is dropped.
- RX_VK_COMPILE_QOS (utility): behind the game every run, and every giant before it, is built at QoS class
  utility; the rest, which the title waits for, at the thread's own (DEFAULT, the game threads' class).
  MoltenVK dispatches each Metal compile from the calling thread to the default global queue, whose QoS is a
  fallback only, so the request carries the worker's class.

The classic worker and every other driver are unchanged; compile_qos_apply does nothing off macOS.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 4f319a0d9a658d4205d52fe25be645443764ed75)
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 36e04c5eb85f15398ec3a82afdef7205bd37e304)
fix(vk): a giant compile that failed is tried once more alone, not at once among the other giants
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 8s
Third-party parts / As their notes say (pull_request) Successful in 14s
40edc9da95
Runs 17 and 18 (MoltenVK 1.4.2, M4 Max): Metal's compiler service aborted 13 and 7 times on an LLVM
allocation failure while it held giant compute compiles, and each abort takes every compile of that
instance with it (up to 8). A giant that came back VK_ERROR_INITIALIZATION_FAILED was compiled again
at once, with a new pipeline cache, among the giants still in flight.

Under compiler_service (MoltenVK) only, in the precompile's behind_worker: a giant's failed compile is
only said (create_watched, Retry::Later); its worker gives its place back (Giants::solo_ask) and waits
until no other giant is in flight and none is held, while no giant starts (Giants::solo), then
compiles it once more with a new pipeline cache (Retry::Fresh) and says so. Its compute_state stays
claimed meanwhile; a stop cancels its turn. Every other compile, and every other driver, as before.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit b6069bbab0719980b61f26c6b1df469f989f930a)
fix(macos): KosmicKrisp's compiles go through the compiler service's precompile, as MoltenVK's do
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s
Third-party parts / As their notes say (pull_request) Successful in 11s
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
a4d31a5904
KosmicKrisp (Mesa's Vulkan driver on Metal 4, LunarG SDK 1.4.363.1) compiles Metal source in Metal's compiler
service at pipeline creation, as MoltenVK does, but vk_vendor.h keyed compiler_service on MoltenVK alone. So on
KosmicKrisp the preparation compiled every compute variant itself: start 20 reached 1,607 of 38,809 programs in
1,770 s, the shape of MoltenVK's start 14 before the precompile hand-off existed.

KK-03 (reports/2026-11-14-kk-slow.md section 5) measured the same synthetic SPIR-V on both drivers: KosmicKrisp's
cold compute compiles are 1.0x MoltenVK's at 3,003 words, 2.5x at 29,310 and 5.8x at 110,406 (2.4x with robustness
off on both), 16 at once take 4.7x MoltenVK's wall time, and its pipeline cache carries no compile across processes
(a hit from saved data took 92% of the cold time; Metal's own cache 5-6%).

- vk_vendor.h: compiler_service on for KosmicKrisp's driver too: the preparation translates and keeps the compute
  programs, the precompile compiles them (giants bounded, the rest handed to the game), slow compiles are named.
- vk_draw.cpp: KosmicKrisp gives VK_ERROR_INVALID_SHADER_NV for every Metal library or pipeline it could not make,
  a compiler service that ended under it included (start 20's three XPC_ERROR_CONNECTION_INTERRUPTED libraries);
  where the service's compiles are made (create_watched, compile_again) it is read as MoltenVK's
  VK_ERROR_INITIALIZATION_FAILED, said once, so the same tries again and tries alone apply. Every other driver and
  code as before.
- Tests: test_precompile_handoff (a made-up KosmicKrisp whose failed compile is VK_ERROR_INVALID_SHADER_NV and whose
  cache keeps none; MoltenVK and KosmicKrisp without the service unchanged; the vendor keying, HoneyKrisp off) and
  test_vendor_fixes (KosmicKrisp's Apple columns). Both fail on 79e2e006 and pass here.

KosmicKrisp stays opt-in (RX_VK_DRIVER=kosmickrisp); MoltenVK and every other driver behave as before. Neither file
is a shader-stamp input, so no program is translated again.

(Picked into mac/kk-compiler-service, on mac/title-stall (the precompile's tries again and tries alone), without the
part that needs a branch not in it: test_vendor_fixes.py's check_apple_drivers, its KosmicKrisp row (1000000011 ->
1000000111) and its words, are mac/kk-tex-order's; that check is not on this base. A sibling merge item, named in
the PR. vk_vendor.h and docs/development.md: only this topic's lines, on this base's text.)

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit ecbeefa597d65f931e7f878eacac6cf7c5152928)
mooki force-pushed mac/kk-compiler-service from a4d31a5904
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s
Third-party parts / As their notes say (pull_request) Successful in 11s
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
to 588d5f3851
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s
Third-party parts / As their notes say (pull_request) Successful in 9s
2026-11-15 15:05:39 +00:00
Compare
Author
Member

Folded into #849 (fix(macos): MoltenVK's precompile and pipelines on Metal)

Folded into #849 (fix(macos): MoltenVK's precompile and pipelines on Metal)
mooki closed this pull request 2026-11-15 18:08:49 +00:00
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s
Third-party parts / As their notes say (pull_request) Successful in 9s

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!785
No description provided.