WIP: fix(vk): preserve cache file reservations on macOS #476
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!476
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:mac/cache-files"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
The renderer's two cache files do not compile on macOS.
gpu/vk/pipeline_cache_file.hreserves the pipeline cache'sdisk space with Linux's
posix_fallocate, andgpu/vk/vk_progcache.cppasks for read-ahead withPOSIX_FADV_WILLNEED;macOS has neither. So six of
main's game-free cache checks fail on a Mac, and no Mac renderer can build. The reservationitself has to stay: the driver writes the pipeline cache through a shared mapping, and on a sparse file a full disk is a
SIGBUSin the middle of a write instead of a save that fails cleanly.What changed
PipelineCacheFile::writableingpu/vk/pipeline_cache_file.h.fcntl(F_PREALLOCATE)(F_ALLOCATEALL, all ornothing;
F_PEOFPOSMODE). Only then doesftruncateset its length and is it mapped.vk_draw.cpp:839and:10352) already handle: thetemporary is removed and the published cache is left as it was. There is no sparse-file fallback.
program_cache_loadingpu/vk/vk_progcache.cpp. On macOS each record's two read hints (its first 4 KiB, its last8 KiB) use
fcntl(F_RDADVISE)for the same ranges as Linux'sposix_fadvise. Best effort, as before: the reads thatfollow do the I/O and check it either way.
F_RDAHEADwas not used, because it changes the file's read-ahead policyrather than asking for these ranges.
#if defined(__APPLE__), and#elsekeepsmain's linesas they are.
tests/pipeline_cache_io_fault_test.cpp. On macOS its injectedENOSPCreaches theF_PREALLOCATEcall(through an
fcntlstand-in), so the existing native failure stages run there too. Linux'sposix_fallocatestand-in is unchanged.
tests/test_pipeline_cache_allocation.py(new; macOS only, skipped elsewhere). It compilespipeline_cache_file.h(--source-rootselects another tree) and checks, on a real file:blocks of 512 bytes, against 0 for a control file sized with
ftruncatealone);temporaries are removed, and a retry succeeds.
docs/pipeline-cache-storage.mdsays what macOS reserves and hints, and what the new test checks.Flags and settings
None.
Measurements
main:
5e072fc8c2adadba5ea610148be1c47800ba0cca(2026-11-14)this PR:
mac/cache-files@587afb479e2005b0ba29bb64e6bed2059d6f680aHardware: Apple M4 Max (16 CPUs, 64 GiB), APFS, macOS 26.7.1 (25G241), Apple clang 21.0.0. Game-free: no GPU, no game
Scene: none (a portability fix; its proof is the tests, as CONTRIBUTING's
fixrow allows)Method: the six cache checks and the new test, compiled for arm64 and for x86_64 (run under Rosetta), against
main'ssource (
git archive 5e072fc8) and against this PR. Two interleaved rounds per route: PR, main, PR, main. Then the fullgame-free suite on this PR, compared by test name with
main's on the same Mac.How to reproduce:
python -I -B tests/run_all.py pipeline_cache pipeline_play_cache pcache_header program_cache_cleanup program_cache_one_varianton each side;python -I -B tests/test_pipeline_cache_allocation.py --cxx clang++ --source-root <a tree of main>for the new test againstmain's header. For Rosetta, pass a compiler that adds-arch x86_64.5e072fc8587afb47test_pipeline_cache_file.pyposix_fallocatetest_pipeline_cache_save.pyposix_fallocatetest_pipeline_play_cache.pyposix_fallocatetest_pcache_header_read.pyposix_fallocatetest_program_cache_cleanup.pyPOSIX_FADV_WILLNEEDtest_program_cache_one_variant.pyPOSIX_FADV_WILLNEEDtest_pipeline_cache_allocation.py(new)posix_fallocateEach row is the same on arm64 and under Rosetta, in both rounds: 7 of 7 fail on
mainand 7 of 7 pass here, 28 runs inall.
The new test also catches a missing reservation, not only
main's compile error. With this PR's header and only theF_PREALLOCATEcall removed (a throwaway copy, not committed), it compiles and fails on both routes:Testing
Tested:
On an Apple M4 Max with macOS 26.7.1, game-free only, on arm64 and under Rosetta (x86_64), as above.
The full suite,
tests/run_all.py, 6 at once, on this PR, compared by test name withmainon the same Mac (therunner labels macOS
linux):main(
test_pcache_header_read,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_program_cache_cleanup,test_program_cache_one_variant) and the newtest_pipeline_cache_allocation. No checkthat passes on
mainfails here, and the same 17 are skipped.main's own on a Mac, the same on both sides; the macOS port's other pull requestsaddress them:
test_appimage_package_retry,test_collect_wide,test_control,test_framegen_resources,test_game_layout,test_low_memory_notice,test_mem_gate,test_mem_gate_children,test_mirror_trim,test_module_ids,test_oot_state_tap,test_pad_layout,test_prof_symbols.main's run left one, fromtest_recorder_sync's fixture, which crasheson purpose).
Not tested:
main's text, in the#elsebranches.main's renderer does not build on macOS yet (the macOS port's other pull requests,#420); this PR is the cache-file part.
ENOSPCis injected at theF_PREALLOCATEcall, andthe file-size limit is real. A filesystem that refuses
F_PREALLOCATEfails the save and keeps the old cache, bydesign.
-mmacosx-version-min=14.4, but it ran on 26.7.1 only.F_RDADVISEis a hint, asposix_fadviseis; no timing is claimed.Needs other testers:
python -I -B tests/run_all.py pipeline_cache pipeline_play_cache pcache_header program_cache, expecting the same results as onmain(request to open).tests/test_pipeline_cache_allocation.py, expecting all to pass(request to open).
Screenshots
Nothing on screen.
Risk and rollback
main's.F_PREALLOCATEcannot save the pipeline cache: the save fails cleanly and the oldcache stays. That is deliberate. A sparse mapping there could
SIGBUSmid-write on a full disk.LONGSHOT_SHADER_STAMP(nx2/build.py:1176-1182), so no shader cache rebuilds. The cacheformats are unchanged.
Reviewer notes
PipelineCacheFile::writable's order: create the empty temporary, reserve all of it or nothing, set its length,then map it. The two callers' handling of an empty result is unchanged.
F_RDADVISE's two ranges match theposix_fadviseranges line for line.main's #321 moved the caches into the game folder'scache/. The temporary is created beside its destination(
create_temporary), so the reservation goes where the cache goes and nothing else here changes.587afb47(one commit). It ismacos/cache-files'98b6d7cfcherry-picked ontomainwithout a conflict(the same patch id), with the
Assisted-byandCo-Authored-Bytrailers of this rebase added.Issues
See #420 (the macOS port).
Checklist
For everyone
type(scope): description).main. Branched from5e072fc8; main has moved since, and it is rebased in one sweep before ready.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above). On macOS:532 passed, 17 skipped; the 13 failures are among
main's 19 on this Mac, named under Testing. None is new.requests are still to open.
the median reported. For a
fix: the tests fail onmainand pass here, 2 interleaved rounds on each route.third_party/untouched (a new folder with aPROVENANCE.mdinstead).fcntl(2)and
truncate(2)manuals andsys/fcntl.h; nothing copied.If applicable
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened. (pipeline_cache_io_fault_test.cppinjects its failure at the new call.)
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.