MagicBeanDev/Longshot
MagicBeanDev/Longshot
Watch
74
95
Fork
You've already forked Longshot
 98
Code Issues 230 Pull requests 280 Projects Releases 9 Packages Wiki Activity Actions 796

feat(macos): RX_VK_METAL_CACHE_DIR keeps Metal's shader cache in a chosen folder #653

Closed
mooki wants to merge 6 commits from mooki/Longshot:mac/metal-cache into main
pull from: mooki/Longshot:mac/metal-cache
merge into: MagicBeanDev:main
Conversation 1 Commits 6 Files changed 15 +1584 -1
mooki commented 2026-11-15 02:17:46 +00:00
Member

Type: feat · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (MoltenVK 1.4.2) · Needs: a windowed macOS start

Why

On a Mac, Metal keeps the GPU code it compiles for the game's shaders in a per-user folder under /var/folders, outside the
game folder. A start that runs in a sandbox which allows only the game's own folder (as the macOS port's test runner does)
cannot open that cache: Metal compiled every pipeline again on every start, and the log filled with
fopen failed for list file: errno = 1 (2,489 such lines in one start, about 1.5 per compile). There was no way to point
Metal elsewhere: Metal has no cache-path variable, and the per-user folder ignores HOME, TMPDIR and
CFFIXED_USER_HOME. This PR adds an opt-in switch that tells Metal which folder to use. It is part of the macOS port. See
#420.

What changed

RX_VK_METAL_CACHE_DIR=<folder> (macOS only, off by default) hands that folder to Metal before the Vulkan library is
loaded, and after the device is made logs whether Metal took it. Unset or empty, nothing is loaded, called or logged.
Every refusal is logged once and the start goes on as without the switch. All of it is inside #if defined(__APPLE__).

  • Metal's cache folder: gpu/vk/metal_cache.h (new). ask() checks the folder (absolute, at most 1,023 bytes, an existing
    directory: the game never creates it), looks up Metal's exported MTLSetShaderCachePath/MTLGetShaderCachePath and
    CoreFoundation's string calls with dlopen/dlsym, and makes the call with a CFString it releases. check() compares
    MTLGetShaderCachePath() with the folder after create_device. No new link dependency and no Objective-C.
  • the renderer: gpu/vk/vk_main.cpp: load_global()'s macOS branch asks right before mkwvk::host->vulkan_library()
    (Metal fixes its folder on first use and silently ignores a later call), and mkwvk_init() checks right after
    create_device. Log lines: vk: Metal shader cache: asked for <folder> (…), then vk: Metal shader cache: <folder> (MTLGetShaderCachePath, as RX_VK_METAL_CACHE_DIR asked), or vk: RX_VK_METAL_CACHE_DIR NOT applied: ….
  • docs: docs/development.md, the RX_VK_METAL_CACHE_DIR row.
  • tests: tests/test_metal_cache_dir.py compiles the real load_global() and header against made-up Metal,
    CoreFoundation and host loader calls on any OS.

MTLSetShaderCachePath is Apple SPI: exported by Metal and listed in the SDK's Metal.tbd, declared in WebKit's
MetalSPI.h and used by Chromium, but in no public header. If a macOS update removes it, the switch logs NOT applied and
the start compiles as it does without the switch.

Measurements

Verdict: works: Metal took the chosen folder in every start that set it (game runs #5-#9, arm64 and x86_64 under
Rosetta), and the denied cache opens went from about 1.5 per compile to 0. In a game-free MoltenVK probe a second process
made each pipeline from that folder 83× faster (2.6 ms against 214.8 ms), and even the first process compiled about twice as
fast as with Metal's folder denied (215 against 395 ms). Not measured in the game against main: main does not run on macOS.

Baseline: main does not start on macOS, so the game columns are the macOS port's integration builds, arm64, MoltenVK
1.4.2, the runner's sandbox (which denies /var/folders): before is run #4 on aefe6b0a, the last build without this
commit; after are runs #5-#8 on f32adfa2 and a7711f3c, which carry it (the topic's d7586df3) and set the switch to a
folder in the run tree. The probe below is the same call made from a standalone MoltenVK process, with the switch's two
"before" cases (unset, and the call made after vkCreateDevice, which Metal ignores).

Hardware: macOS 26.7.1 (25G241), Apple M4 Max (40-core GPU), MoltenVK 1.4.2, 16-core CPU, 64 GB
Scene: game runs #4-#8 from a fresh profile (#6 and #8 warm: seeded from #5 and #7), windowless, to 180 s of preparation;
the probe: 24 distinct synthetic compute pipelines per process, no VkPipelineCache
Method: game runs: the renderer's two vk: Metal shader cache lines, the OS log's errno = 1 count and the runner's
published Metal files, one run each; probe reports/MetalCacheDir-files/probe/runs-01 (one gpu slot admission,
2026-11-14 07:52 UTC, 2 processes per route, A then B; per-pipeline times from each process's own SUMMARY line, the
median over 24 pipelines; run.json 323d4ef4…)
How to reproduce: set RX_VK_METAL_CACHE_DIR to an existing folder and read longshot_log.txt's vk: Metal shader cache
lines; the probe: reports/2026-11-14-metal-cache-dir.md §5 (the exact driver.py run command)

Before: integration @ aefe6b0a (run #4) After: integration @ f32adfa2 (runs #5, #6) After: integration @ a7711f3c (runs #7, #8)
vk: Metal shader cache lines none (no switch) asked for, then taken, in both runs asked for, then taken, in both runs
OS log fopen failed for list file: errno = 1 6,371 (1.499 per compile) 0 in both runs 0 in both runs
Metal files the runner carried to the next start none (Metal's folder unreadable) after #6: functions.data 235,225,088 B, libraries.data 404,455,424 B after #8: functions.data 379,764,736 B, libraries.data 603,750,400 B

The probe (metal-cache-probe.cpp, MoltenVK 1.4.2 compute pipelines, process A then B):

switch unset (default) call after the device (setpath-late) call before MoltenVK loads (setpath) folder carried by clonefile (carried)
MTLGetShaderCachePath in A / B (null) / (null) (null) / (null) the chosen folder / the same the chosen folder / its clone
A: median per pipeline 394.8 ms 396.5 ms 214.8 ms 212.5 ms
B: median per pipeline 415.7 ms 415.2 ms 2.59 ms 2.73 ms
B / A 1.053 1.047 0.0121 0.0129 
probe runs-01 setpath: GETPATH …/runs-01/setpath/metal-cache (matches_chosen=1 in A and B); after A: 16777235_467/functions.data 1,572,864, functions.list 2,732, 32024/libraries.data 1,720,320, libraries.list 1,444, modules.timestamp 0
game runs #5-#9, lines 8 and 233: vk: Metal shader cache: asked for <run>/cache/metal (RX_VK_METAL_CACHE_DIR, before the Vulkan library is loaded)
                                  vk: Metal shader cache: <run>/cache/metal (MTLGetShaderCachePath, as RX_VK_METAL_CACHE_DIR asked)
OS log, runs #5-#8: libCoreFSCache "fopen failed for list file: errno = 1": 0 (runs #1-#4, without the switch: about 1.5 per compile; 2,489 in one start)

Run reports: reports/2026-11-14-metal-cache-dir.md (§1 the denied opens, §6 the probe), -metal-cache-impl.md,
-mac-startup-05.md (#5 cold, #6 warm), -mac-startup-07.md (#7 cold, #8 warm: the carried folder opened with no
invalidation), -mac-startup-09.md (#9, x86_64 under Rosetta). The lines above are quoted from them (renderer lines and
counts only).

What it does not do: Metal holds at most 1 GiB per cache file and never evicts what the running start itself touched, so a
start whose pipelines pass 1 GiB stores nothing after that, folder or not (runs #17-#22 reached the cap:
functions.data/libraries.data about 1.12 GB; reports/2026-11-14-metal-cache2.md §1). The follow-up #667
keeps MoltenVK's pipelines in Metal binary archives in a second folder for that.

Flags and settings

One new switch, macOS only, off by default (the owner's decision: keep it opt-in).

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
RX_VK_METAL_CACHE_DIR=<folder> the folder Metal keeps its shader cache in (an existing absolute path) no switch: Metal's per-user folder under /var/folders unset: unchanged; set: that folder, or a NOT applied line and unchanged Metal reads and writes its compiled functions and libraries there; a sandboxed or carried start gets a warm cache helps: starts in a sandbox that denies /var/folders, caches carried between runs (measured); doesn't help: a normal player start, whose per-user folder already works; untested: a start that made a window first (SDL), other macOS builds off by default; set it only for sandboxed or reproducible runs (the macOS test runner sets it per run)

Testing

Tested: on macOS 26.7.1, Apple M4 Max, at fc8ee75d, this PR's head before the rebases onto e090cc48 and aa95d35f
(#438 at 0f2262a3 plus this commit):

  • tests/test_metal_cache_dir.py --cxx /usr/bin/clang++ (arm64) and --cxx clang++ -arch x86_64 (the fixture run under
    Rosetta): pass. The same test pointed at the base 0f2262a3 (--source) fails on both arches, and at main 6c9edffe
    on arm64 (header missing, no ask in load_global, no check in mkwvk_init, no docs row). Logs:
    reports/PubWave-files/PubMetal/proof/mc-*.log.
  • The topic's three mutation runs (ask moved after the library load, empty treated as set, check always succeeding; each
    fails the test, reports/MetalCacheImpl-files/negative-*.log) carry over: this commit's added and removed lines, its
    test, its header and load_global()'s block are byte-identical to the topic's d7586df3 (checked,
    reports/PubWave-files/PubMetal/mc-non-apple-text.txt).
  • vk_main.cpp compiled for arm64 and x86_64 with main's renderer flags (nx2/build.py's vflags plus -arch):
    exit 0 both; each object holds the switch and both SPI names (reports/PubWave-files/PubMetal/objects-mc/).
  • vk_main.cpp's Linux and Windows text (every __APPLE__ block resolved away) is byte-identical to 0f2262a3, and the new
    metal_cache.h is included only inside an __APPLE__ block: Windows and Linux compile exactly what they compile without this
    PR (unifdef -U__APPLE__ token check, reports/PubWave-files/offapple/metal-cache.txt), so the host-build box does not apply.
  • Through tools/ls-suite.py: test_metal_cache_dir, test_vk_darwin_admission, test_renderer_load_error,
    test_gpu_choice, test_vendor_fixes, test_load_default, test_log_levels, test_check_pr_description,
    test_dev_packs pass; test_crash_dialog fails as it does on main 6c9edffe on macOS
    (reports/PubWave-files/PubMetal/named-mc/).
  • In the game (integration builds carrying this commit, not this head): runs #5-#9 set the switch and Metal took the folder
    each time (no NOT applied line); #6 and #8 opened the folder carried from #5 and #7 with no invalidation. Later runner
    starts (#21, #22) kept setting it; their Metal files reached the 1 GiB cap described above.
  • After the rebase onto main aa95d35f, at db50cf3d (fc8ee75d before the rebases onto e090cc48 and aa95d35f):
    Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they
    apply) gave the same results as before; python -I -B tests/run_all.py on wave C1's merge 2540345b, which carries
    this head: 556/33/17 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure
    (reports/2026-11-15-rebase-sweep-3.md).

Not tested: a start that creates its window before the renderer loads Vulkan (every run here was windowless; if SDL
already used Metal, the check logs NOT applied); macOS versions other than 26.7.1; an Intel Mac. Windows and Linux were not
built or run, and need nothing: this PR's own range is main's text there (vk_main.cpp the same off Apple, metal_cache.h
included only inside __APPLE__; reports/PubWave-files/offapple/metal-cache.txt), and #438, which this branch carries, has
its own test request.

Must be tested:

  • macOS, windowed: start with RX_VK_METAL_CACHE_DIR set to an existing empty folder, expecting vk: Metal shader cache: <folder> (MTLGetShaderCachePath, …) and files in the folder after quitting (by hand: a Mac tester, the log and the
    folder).

Risks

What could break: only starts that set the switch on macOS: a future macOS without the SPI, or a host that used Metal
before the renderer, logs NOT applied and compiles as without it. With the switch unset nothing runs.

Turning it off: leave RX_VK_METAL_CACHE_DIR unset (the default), or revert the PR.

Compatibility: Nothing changes for players: the switch is off by default and Apple-only. vk_main.cpp and
metal_cache.h are not hashed into the shader stamp, so no cache is translated again; no MKWVK_API_VERSION change.

Reviewer notes

  • Review 0f2262a3..fc8ee75d (one commit, cherry-picked with -x from macos/metal-cache d7586df3 with no conflict).
  • Depends on #438 (vk-admission): load_global()'s macOS branch and mkwvk::host->vulkan_library() come from it. The
    branch carries #438's four commits (6c9edffe..0f2262a3) until #438 merges; then rebase with
    git rebase --onto main 0f2262a3 mac/metal-cache.
  • Start with metal_cache.h's ask() and check(), then the two call sites in vk_main.cpp.
  • Conflicts with main 6c9edffe: none. With the open macOS PRs (git merge-tree): none with #431, #437, #479, #490, #491,
    #492; #480 conflicts in host/posix/win32_posix.c, which comes from #438's commits, not from this one (#438's own head
    meets the same conflict). With later PRs: #651 (renderer threads' 16 MiB stack) edits vk_main.cpp near these lines and
    rebases after this one (textual only). #667 stacks on this PR.

Issues

See #420

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how. (No test request: nothing changes for Windows or Linux; the macOS item is by hand.)
  • Rebased on current main: on aa95d35f (#761, 2026-11-15), head db50cf3d, on #438 681e3708, without a
    conflict; this PR's own commits, old → new: fc8ee75d → db50cf3d; the commit names elsewhere in this description
    are from before the rebase.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). (On the wave's merge of every macOS PR built with this one, 453eb678: 553 passed, 32 failed, 18 skipped against main 6c9edffe's 531/32/17 on this Mac; the failures are main's own except test_split_ub_flush, which #658's tree at the time broke and its head fixes; test_collect_wide now passes; the new tests pass and test_darwin_audio_service, Darwin-only, is skipped by the runner; reports/PubWave-files/waveA/. After the rebase onto main aa95d35f (head db50cf3d): wave C1's merge 2540345b, 556/33/17 against main aa95d35f's 543/33/17, no new failure.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • A game-free test added for the fix, failing on main and passing here.
  • Docs updated.
**Type:** feat · **Risk:** low · **Tested on:** macOS 26.7.1, Apple M4 Max (MoltenVK 1.4.2) · **Needs:** a windowed macOS start ## Why On a Mac, Metal keeps the GPU code it compiles for the game's shaders in a per-user folder under `/var/folders`, outside the game folder. A start that runs in a sandbox which allows only the game's own folder (as the macOS port's test runner does) cannot open that cache: Metal compiled every pipeline again on every start, and the log filled with `fopen failed for list file: errno = 1` (2,489 such lines in one start, about 1.5 per compile). There was no way to point Metal elsewhere: Metal has no cache-path variable, and the per-user folder ignores `HOME`, `TMPDIR` and `CFFIXED_USER_HOME`. This PR adds an opt-in switch that tells Metal which folder to use. It is part of the macOS port. See #420. ## What changed `RX_VK_METAL_CACHE_DIR=<folder>` (macOS only, off by default) hands that folder to Metal before the Vulkan library is loaded, and after the device is made logs whether Metal took it. Unset or empty, nothing is loaded, called or logged. Every refusal is logged once and the start goes on as without the switch. All of it is inside `#if defined(__APPLE__)`. - Metal's cache folder: `gpu/vk/metal_cache.h` (new). `ask()` checks the folder (absolute, at most 1,023 bytes, an existing directory: the game never creates it), looks up Metal's exported `MTLSetShaderCachePath`/`MTLGetShaderCachePath` and CoreFoundation's string calls with `dlopen`/`dlsym`, and makes the call with a CFString it releases. `check()` compares `MTLGetShaderCachePath()` with the folder after `create_device`. No new link dependency and no Objective-C. - the renderer: `gpu/vk/vk_main.cpp`: `load_global()`'s macOS branch asks right before `mkwvk::host->vulkan_library()` (Metal fixes its folder on first use and silently ignores a later call), and `mkwvk_init()` checks right after `create_device`. Log lines: `vk: Metal shader cache: asked for <folder> (…)`, then `vk: Metal shader cache: <folder> (MTLGetShaderCachePath, as RX_VK_METAL_CACHE_DIR asked)`, or `vk: RX_VK_METAL_CACHE_DIR NOT applied: …`. - docs: `docs/development.md`, the `RX_VK_METAL_CACHE_DIR` row. - tests: `tests/test_metal_cache_dir.py` compiles the real `load_global()` and header against made-up Metal, CoreFoundation and host loader calls on any OS. `MTLSetShaderCachePath` is Apple SPI: exported by Metal and listed in the SDK's `Metal.tbd`, declared in WebKit's `MetalSPI.h` and used by Chromium, but in no public header. If a macOS update removes it, the switch logs `NOT applied` and the start compiles as it does without the switch. ## Measurements **Verdict:** works: Metal took the chosen folder in every start that set it (game runs #5-#9, arm64 and x86_64 under Rosetta), and the denied cache opens went from about 1.5 per compile to 0. In a game-free MoltenVK probe a second process made each pipeline from that folder 83× faster (2.6 ms against 214.8 ms), and even the first process compiled about twice as fast as with Metal's folder denied (215 against 395 ms). Not measured in the game against main: main does not run on macOS. Baseline: main does not start on macOS, so the game columns are the macOS port's integration builds, arm64, MoltenVK 1.4.2, the runner's sandbox (which denies `/var/folders`): before is run #4 on `aefe6b0a`, the last build without this commit; after are runs #5-#8 on `f32adfa2` and `a7711f3c`, which carry it (the topic's `d7586df3`) and set the switch to a folder in the run tree. The probe below is the same call made from a standalone MoltenVK process, with the switch's two "before" cases (unset, and the call made after `vkCreateDevice`, which Metal ignores). Hardware: macOS 26.7.1 (25G241), Apple M4 Max (40-core GPU), MoltenVK 1.4.2, 16-core CPU, 64 GB Scene: game runs #4-#8 from a fresh profile (#6 and #8 warm: seeded from #5 and #7), windowless, to 180 s of preparation; the probe: 24 distinct synthetic compute pipelines per process, no `VkPipelineCache` Method: game runs: the renderer's two `vk: Metal shader cache` lines, the OS log's `errno = 1` count and the runner's published Metal files, one run each; probe `reports/MetalCacheDir-files/probe/runs-01` (one `gpu` slot admission, 2026-11-14 07:52 UTC, 2 processes per route, A then B; per-pipeline times from each process's own `SUMMARY` line, the median over 24 pipelines; `run.json` `323d4ef4…`) How to reproduce: set `RX_VK_METAL_CACHE_DIR` to an existing folder and read `longshot_log.txt`'s `vk: Metal shader cache` lines; the probe: `reports/2026-11-14-metal-cache-dir.md` §5 (the exact `driver.py run` command) | | Before: integration @ aefe6b0a (run #4) | After: integration @ f32adfa2 (runs #5, #6) | After: integration @ a7711f3c (runs #7, #8) | |---|---|---|---| | `vk: Metal shader cache` lines | none (no switch) | asked for, then taken, in both runs | asked for, then taken, in both runs | | OS log `fopen failed for list file: errno = 1` | 6,371 (1.499 per compile) | 0 in both runs | 0 in both runs | | Metal files the runner carried to the next start | none (Metal's folder unreadable) | after #6: `functions.data` 235,225,088 B, `libraries.data` 404,455,424 B | after #8: `functions.data` 379,764,736 B, `libraries.data` 603,750,400 B | The probe (`metal-cache-probe.cpp`, MoltenVK 1.4.2 compute pipelines, process A then B): | | switch unset (`default`) | call after the device (`setpath-late`) | call before MoltenVK loads (`setpath`) | folder carried by `clonefile` (`carried`) | |---|---|---|---|---| | `MTLGetShaderCachePath` in A / B | (null) / (null) | (null) / (null) | the chosen folder / the same | the chosen folder / its clone | | A: median per pipeline | 394.8 ms | 396.5 ms | 214.8 ms | 212.5 ms | | B: median per pipeline | 415.7 ms | 415.2 ms | 2.59 ms | 2.73 ms | | B / A | 1.053 | 1.047 | 0.0121 | 0.0129 | ``` probe runs-01 setpath: GETPATH …/runs-01/setpath/metal-cache (matches_chosen=1 in A and B); after A: 16777235_467/functions.data 1,572,864, functions.list 2,732, 32024/libraries.data 1,720,320, libraries.list 1,444, modules.timestamp 0 game runs #5-#9, lines 8 and 233: vk: Metal shader cache: asked for <run>/cache/metal (RX_VK_METAL_CACHE_DIR, before the Vulkan library is loaded) vk: Metal shader cache: <run>/cache/metal (MTLGetShaderCachePath, as RX_VK_METAL_CACHE_DIR asked) OS log, runs #5-#8: libCoreFSCache "fopen failed for list file: errno = 1": 0 (runs #1-#4, without the switch: about 1.5 per compile; 2,489 in one start) ``` Run reports: `reports/2026-11-14-metal-cache-dir.md` (§1 the denied opens, §6 the probe), `-metal-cache-impl.md`, `-mac-startup-05.md` (#5 cold, #6 warm), `-mac-startup-07.md` (#7 cold, #8 warm: the carried folder opened with no invalidation), `-mac-startup-09.md` (#9, x86_64 under Rosetta). The lines above are quoted from them (renderer lines and counts only). What it does not do: Metal holds at most 1 GiB per cache file and never evicts what the running start itself touched, so a start whose pipelines pass 1 GiB stores nothing after that, folder or not (runs #17-#22 reached the cap: `functions.data`/`libraries.data` about 1.12 GB; `reports/2026-11-14-metal-cache2.md` §1). The follow-up #667 keeps MoltenVK's pipelines in Metal binary archives in a second folder for that. ### Flags and settings One new switch, macOS only, off by default (the owner's decision: keep it opt-in). | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `RX_VK_METAL_CACHE_DIR=<folder>` | the folder Metal keeps its shader cache in (an existing absolute path) | no switch: Metal's per-user folder under `/var/folders` | unset: unchanged; set: that folder, or a `NOT applied` line and unchanged | Metal reads and writes its compiled functions and libraries there; a sandboxed or carried start gets a warm cache | helps: starts in a sandbox that denies `/var/folders`, caches carried between runs (measured); doesn't help: a normal player start, whose per-user folder already works; untested: a start that made a window first (SDL), other macOS builds | off by default; set it only for sandboxed or reproducible runs (the macOS test runner sets it per run) | ## Testing **Tested:** on macOS 26.7.1, Apple M4 Max, at `fc8ee75d`, this PR's head before the rebases onto e090cc48 and aa95d35f (#438 at `0f2262a3` plus this commit): - `tests/test_metal_cache_dir.py --cxx /usr/bin/clang++` (arm64) and `--cxx` `clang++ -arch x86_64` (the fixture run under Rosetta): pass. The same test pointed at the base `0f2262a3` (`--source`) fails on both arches, and at main `6c9edffe` on arm64 (header missing, no ask in `load_global`, no check in `mkwvk_init`, no docs row). Logs: `reports/PubWave-files/PubMetal/proof/mc-*.log`. - The topic's three mutation runs (ask moved after the library load, empty treated as set, check always succeeding; each fails the test, `reports/MetalCacheImpl-files/negative-*.log`) carry over: this commit's added and removed lines, its test, its header and `load_global()`'s block are byte-identical to the topic's `d7586df3` (checked, `reports/PubWave-files/PubMetal/mc-non-apple-text.txt`). - `vk_main.cpp` compiled for arm64 and x86_64 with main's renderer flags (`nx2/build.py`'s `vflags` plus `-arch`): exit 0 both; each object holds the switch and both SPI names (`reports/PubWave-files/PubMetal/objects-mc/`). - `vk_main.cpp`'s Linux and Windows text (every `__APPLE__` block resolved away) is byte-identical to `0f2262a3`, and the new `metal_cache.h` is included only inside an `__APPLE__` block: Windows and Linux compile exactly what they compile without this PR (`unifdef -U__APPLE__` token check, `reports/PubWave-files/offapple/metal-cache.txt`), so the host-build box does not apply. - Through `tools/ls-suite.py`: `test_metal_cache_dir`, `test_vk_darwin_admission`, `test_renderer_load_error`, `test_gpu_choice`, `test_vendor_fixes`, `test_load_default`, `test_log_levels`, `test_check_pr_description`, `test_dev_packs` pass; `test_crash_dialog` fails as it does on main `6c9edffe` on macOS (`reports/PubWave-files/PubMetal/named-mc/`). - In the game (integration builds carrying this commit, not this head): runs #5-#9 set the switch and Metal took the folder each time (no `NOT applied` line); #6 and #8 opened the folder carried from #5 and #7 with no invalidation. Later runner starts (#21, #22) kept setting it; their Metal files reached the 1 GiB cap described above. - **After the rebase onto main aa95d35f**, at `db50cf3d` (`fc8ee75d` before the rebases onto e090cc48 and aa95d35f): Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they apply) gave the same results as before; `python -I -B tests/run_all.py` on wave C1's merge `2540345b`, which carries this head: 556/33/17 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure (`reports/2026-11-15-rebase-sweep-3.md`). **Not tested:** a start that creates its window before the renderer loads Vulkan (every run here was windowless; if SDL already used Metal, the check logs `NOT applied`); macOS versions other than 26.7.1; an Intel Mac. Windows and Linux were not built or run, and need nothing: this PR's own range is main's text there (`vk_main.cpp` the same off Apple, `metal_cache.h` included only inside `__APPLE__`; `reports/PubWave-files/offapple/metal-cache.txt`), and #438, which this branch carries, has its own test request. **Must be tested:** - macOS, windowed: start with `RX_VK_METAL_CACHE_DIR` set to an existing empty folder, expecting `vk: Metal shader cache: <folder> (MTLGetShaderCachePath, …)` and files in the folder after quitting (by hand: a Mac tester, the log and the folder). ## Risks **What could break:** only starts that set the switch on macOS: a future macOS without the SPI, or a host that used Metal before the renderer, logs `NOT applied` and compiles as without it. With the switch unset nothing runs. **Turning it off:** leave `RX_VK_METAL_CACHE_DIR` unset (the default), or revert the PR. **Compatibility:** Nothing changes for players: the switch is off by default and Apple-only. `vk_main.cpp` and `metal_cache.h` are not hashed into the shader stamp, so no cache is translated again; no `MKWVK_API_VERSION` change. ## Reviewer notes - Review `0f2262a3..fc8ee75d` (one commit, cherry-picked with `-x` from `macos/metal-cache` `d7586df3` with no conflict). - Depends on #438 (vk-admission): `load_global()`'s macOS branch and `mkwvk::host->vulkan_library()` come from it. The branch carries #438's four commits (`6c9edffe..0f2262a3`) until #438 merges; then rebase with `git rebase --onto main 0f2262a3 mac/metal-cache`. - Start with `metal_cache.h`'s `ask()` and `check()`, then the two call sites in `vk_main.cpp`. - Conflicts with main `6c9edffe`: none. With the open macOS PRs (`git merge-tree`): none with #431, #437, #479, #490, #491, #492; #480 conflicts in `host/posix/win32_posix.c`, which comes from #438's commits, not from this one (#438's own head meets the same conflict). With later PRs: #651 (renderer threads' 16 MiB stack) edits `vk_main.cpp` near these lines and rebases after this one (textual only). #667 stacks on this PR. ## Issues See #420 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. (No test request: nothing changes for Windows or Linux; the macOS item is by hand.) - [x] Rebased on current `main`: on aa95d35f (#761, 2026-11-15), head `db50cf3d`, on #438 `681e3708`, without a conflict; this PR's own commits, old → new: `fc8ee75d` → `db50cf3d`; the commit names elsewhere in this description are from before the rebase. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (On the wave's merge of every macOS PR built with this one, `453eb678`: 553 passed, 32 failed, 18 skipped against main 6c9edffe's 531/32/17 on this Mac; the failures are main's own except `test_split_ub_flush`, which #658's tree at the time broke and its head fixes; `test_collect_wide` now passes; the new tests pass and `test_darwin_audio_service`, Darwin-only, is skipped by the runner; `reports/PubWave-files/waveA/`. After the rebase onto main aa95d35f (head `db50cf3d`): wave C1's merge `2540345b`, 556/33/17 against main aa95d35f's 543/33/17, no new failure.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] Docs updated. 
Select a local Vulkan loader or MoltenVK with RX_VK_LIBRARY and keep SDL and
renderer startup on that same library. Capture SDL's instance extensions on
the window thread, enable advertised portability extensions, and name missing
admission requirements and VkResults without changing Windows or Linux paths.

Keep the non-Apple host API at 21 for existing prebuilt renderers; append the
two macOS callbacks only in Apple's version 22. Cover pure decisions, legacy
platform expectations, ABI layout, and the SDL2/SDL3 bridge without a GPU.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Remove the SDL source-string wiring and source-index ordering assertions while
keeping the sixteen compiled behavioral cases and surface adapter compile.
Replace the comment-sensitive API header hash with fixed baseline field names,
byte offsets, versions and sizes, retaining typed Apple callback checks and
both platforms' actual admission decisions and negative controls.

Production code is unchanged. Prior full-suite and device receipts remain
bound to 818e33b1; only the affected game-free tests were rerun.

Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra <[email protected]>
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The fixture cuts LoadLibraryA and create_device out of main's source. Since
#34, LoadLibraryA keeps dlerror()'s text in load_error, and since #191
native_fadd_wanted also takes the device ID and driver version, so the cut
code no longer compiled against the fixture's stand-ins on this base. Give
it load_error and a dlerror stand-in, and the four-argument stand-in. No
expectation changed.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
test(vk): pin the admission ABI at main's version 22 and macOS 23
All checks were successful
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
Website / Website on Cloudflare (pull_request) Has been skipped
Details
CI / Conventional Commit title (pull_request) Successful in 4s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Details
Third-party parts / As their notes say (pull_request) Successful in 10s
Details
PR description / Description follows the template (pull_request_target) Successful in 4s
Details
0f2262a378 
Main's custom output size (#51) took MKWVK_API_VERSION 22 on every
platform and appended take_text at offset 352. The macOS loader
callbacks now follow it: the rebased header says 23 on Apple and 22
elsewhere. The fixed contract follows: non-Apple version 22, size 360
and 45 named offsets (take_text added); Apple version 23, size 376,
vulkan_library and vulkan_extensions at 360 and 368.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
feat(macos): Metal's shader cache in a folder the caller names
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
CI / Conventional Commit title (pull_request) Successful in 3s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Details
Third-party parts / As their notes say (pull_request) Successful in 8s
Details
PR description / Description follows the template (pull_request_target) Successful in 3s
Details
fc8ee75dc7 
RX_VK_METAL_CACHE_DIR=<folder>, macOS only and off by default, puts
Metal's own shader cache (compiled functions, libraries, front-end
modules) in that folder instead of Metal's per-user folder under
/var/folders, which a sandbox that keeps every game file in one folder
(tools/ls-run.py) denies. With it denied, every start compiled all of
its pipelines again; the game run's 2,489 "fopen failed for list file:
errno = 1" lines came from that.

gpu/vk/metal_cache.h looks up Metal's exported MTLSetShaderCachePath
and MTLGetShaderCachePath (SPI: in the SDK's Metal.tbd, declared by
WebKit, in no public header) and CoreFoundation's string calls with
dlopen/dlsym, so the renderer links nothing new and has no Objective-C.
load_global asks for the folder before the host loads the Vulkan
library (Metal fixes its folder on first use and silently ignores a
later call), and mkwvk_init logs after create_device whether Metal took
it. A relative or over-long path, a folder that does not exist (the
game never makes it), Metal or a symbol missing, or a folder Metal had
already chosen is logged as "RX_VK_METAL_CACHE_DIR NOT applied" and the
start goes on as without the switch. Unset or empty, nothing is loaded,
called or logged. All of it is under #if defined(__APPLE__): vk_main.cpp's
Windows and Linux text is byte-identical to 5fd1c74b.

Measured with a game-free MoltenVK 1.4.2 probe on the M4 Max (macOS
26.7.1, reports/2026-11-14-metal-cache-dir.md): a synthetic compute
pipeline compiled in ~400 ms with Metal's folder denied, ~213 ms with
a writable empty folder, and 2.6 ms in a second process that found it
there, the folder carried by clonefile; a call after vkCreateDevice was
ignored. Not run in the game.

tests/test_metal_cache_dir.py compiles the real load_global and header
against made-up Metal, CoreFoundation and host loader calls: the order
(the folder is handed over before the library is loaded), the unset,
empty, refused, missing-symbol and mismatch paths, and the source (macOS
only, the check after create_device, the docs row). It fails on
5fd1c74b and on three mutations of the change.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit d7586df37f65b60e507cc3ffb79f38e0f7f7faf0)
mooki changed title from WIP: feat(macos): RX_VK_METAL_CACHE_DIR keeps Metal's shader cache in a chosen folder to feat(macos): RX_VK_METAL_CACHE_DIR keeps Metal's shader cache in a chosen folder 2026-11-15 02:44:47 +00:00
mooki force-pushed mac/metal-cache from fc8ee75dc7
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
CI / Conventional Commit title (pull_request) Successful in 3s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Details
Third-party parts / As their notes say (pull_request) Successful in 8s
Details
PR description / Description follows the template (pull_request_target) Successful in 3s
Details
to c39e0fb4e7
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
Website / Website on Cloudflare (pull_request) Has been skipped
Details
CI / Conventional Commit title (pull_request) Successful in 5s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Details
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
Details
Third-party parts / As their notes say (pull_request) Successful in 15s
Details
PR description / Description follows the template (pull_request_target) Successful in 4s
Details
2026-11-15 07:25:34 +00:00 Compare
mooki force-pushed mac/metal-cache from c39e0fb4e7
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
Website / Website on Cloudflare (pull_request) Has been skipped
Details
CI / Conventional Commit title (pull_request) Successful in 5s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Details
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
Details
Third-party parts / As their notes say (pull_request) Successful in 15s
Details
PR description / Description follows the template (pull_request_target) Successful in 4s
Details
to db50cf3d26
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
Website / Website on Cloudflare (pull_request) Has been skipped
Details
CI / Conventional Commit title (pull_request) Successful in 4s
Details
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 7s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 8s
Details
Third-party parts / As their notes say (pull_request) Successful in 12s
Details
Website / Website on Cloudflare (pull_request_target) Has been skipped
Details
PR description / Description follows the template (pull_request_target) Successful in 7s
Details
2026-11-15 13:54:40 +00:00 Compare
mooki commented 2026-11-15 18:07:42 +00:00
Author
Member

Folded into #846 (feat(macos): Vulkan on Metal: pinned host imports, the unified-memory budget, Metal's cache folder and KosmicKrisp)

Folded into #846 (feat(macos): Vulkan on Metal: pinned host imports, the unified-memory budget, Metal's cache folder and KosmicKrisp)
mooki closed this pull request 2026-11-15 18:07:43 +00:00
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
Website / Website on Cloudflare (pull_request) Has been skipped
Details
CI / Conventional Commit title (pull_request) Successful in 4s
Details
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 7s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 8s
Details
Third-party parts / As their notes say (pull_request) Successful in 12s
Details
Website / Website on Cloudflare (pull_request_target) Has been skipped
Details
PR description / Description follows the template (pull_request_target) Successful in 7s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area/accessibility
Barriers for people with disabilities

  
area/audio
Sound and music

  
area/input
Controllers, keyboard, mouse

  
area/install
Installing, updating, building from your game

  
area/menu
In-game menu and settings

  
area/renderer
Vulkan renderer, shaders, effects

  
area/saves
Save files

  
area/upscaling
DLSS, FSR, frame generation

  
beanstalk: approved @ 1697603
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ 2c5a62e
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ 4fd82b7
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ 64d6d09
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ 6f91fae
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ 70a78f9
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ 8d361dc
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ 9d8356e
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ abf2227
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ d82f202
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ e5f1e40
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: approved @ ee8ab5f
Set by Beanstalk for show; the farm's database is what counts

  
ci: approved
A maintainer approved CircleCI for this fork PR (sticky; removed after 14 quiet days)

  
good first issue
A good start for a new contributor

  
gpu/amd
AMD GPUs

  
gpu/igpu
Integrated GPUs

  
gpu/intel
Intel GPUs

  
gpu/nvidia
NVIDIA GPUs

  
help wanted
Extra hands or knowledge wanted

  
kind
bug
 Something doesn't work as it should

  
kind
chore
 Build, CI, dependencies, tidying

  
kind
docs
 Documentation

  
kind
feature
 A new feature or an improvement

  
kind
performance
 Frame rate, frame time, loading, memory

  
kind
question
 A question rather than a change

  
needs measurements
Waiting for before/after measurements

  
platform/linux
Affects Linux (a change for every system carries both Windows and Linux)

  
platform/macos
macOS (Apple silicon or Intel Macs)

  
platform/steam-deck
Steam Deck

  
platform/windows
Affects Windows (a change for every system carries both Windows and Linux)

  
preview
Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  
priority
critical
 Crashes, data loss, or blocks most players

  
priority
high
 Important: next in line

  
priority
low
 Nice to have

  
regression
Worked in an earlier build, broken in a newer one: highest priority, above other bugs

  
status
blocked
 Waits on something else first

  
status
confirmed
 Reproduced or agreed on

  
status
duplicate
 Already reported or proposed

  
status
invalid
 Not a problem in this project

  
status
needs-info
 Waiting for more information from the reporter

  
status
wontfix
 Will not be worked on

  
testing wanted
Wanted: a test on other hardware (posts a test request in #test-requests)

  
ci: approved
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!653
No description provided.