WIP: feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing #737

Draft

dyst wants to merge 2 commits from dyst/Longshot:feat/shader-cache-store into main

pull from: dyst/Longshot:feat/shader-cache-store

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 3 Commits 2 Files changed 20 +1395 -20

dyst commented 2026-11-15 08:16:04 +00:00

Member

• Copy link
• Copy Markdown

Type: feat · Risk: medium · Tested on: Windows 11, game-free checks and a renderer build (not yet in the game) · Needs: a run in the game with two game folders (Windows and Linux, any GPU)

Why

Every game folder prepares its shaders on its own, about 25 minutes with Compile shaders All on a fast PC, even when another folder on the same PC translated the same shaders with the same translator a minute before: a stable and a nightly side by side, a rebuild into a new folder, each dev_packs.py runtime (which also carries a 10 GB copy of the caches). The caches are already stamped by what they are good for (program_cache_stamp), so a store keyed by that stamp lets every folder with the same translation use them in place.

The design was agreed with the maintainers, including moving these files out of the game folder, where #321 put them. It complements MagicBeanDev/LongshotLauncher#57 (an update in place keeps its cache/ anyway) and replaces copying caches between folders (MagicBeanDev/LongshotLauncher#78). It mirrors the object and lift stores (#371, #372). No Longshot issue fits (the tracker searched).

What changed

With LONGSHOT_SHADER_CACHE=<folder> (or --shader-cache <folder>), the game keeps its shader caches in <folder>/<stamp>/ for what one translation made and <folder>/shared/ for the lists good for any build, so a game folder whose stamp is in the store starts with its programs and pipelines and prepares nothing. Unset, empty or 0, every file stays in the game folder's cache/ as before, and the stamp is not taken any sooner. Nothing sets it for players yet: a launcher PR follows, setting it beside its builds and cleaning up the stamps no build uses.

• runtime: gpu/vk/shader_store.h (new, standard library and the platform's file lock, no Vulkan):
  • open_stamped (at the first stamped file asked for, where the game took the stamp before) and open_shared (at the first list); each part held by <part>/.lock for the process's life: open with no sharing on Windows, flock elsewhere, so a tool can tell a part in use whatever it calls the path. A second run of the same stamp keeps that part's files in its own cache/ for that run.
  • consume: a game folder's cache/ of the same stamp is taken in at a start that holds the store. Files the store lacks are moved in (a rename, else a copy as .part renamed in); the program-side files only when cache/'s program cache carries this stamp, the pipeline cache only when its stamp file does. Pipeline notes and lists the store has get the records it lacks (merge_records, by their heads: the notes' stamp and record size, each list's magic and layout). A pipeline cache the store has a main file for goes in as a play file, merged at the next start by play_files_merge. A second run's program cache is left in its cache/: merging two program caches record by record is not done here.
  • A stamp's folder is named by the program cache's stamp and the toolchain (folder_key: the compiler's version and the C++ library's), not the stamp alone: the program cache keeps its records' Shader::Info as it lies in memory, and the stamp has the library's version for libc++ and Microsoft's but none for libstdc++ (HumanBean's point on MagicBeanDev/LongshotLauncher#78). Before the store each version had its folder and its tools, so a cache never met another toolchain's; now one never does in the store either. The stamp itself is unchanged, so nobody's caches go.
  • An update in place, by default (hand_over): each game folder registers in the stamp it uses (<stamp>/users/, a file per folder naming it) and records it (cache/longshot_shader_store.txt). When a folder starts under another stamp (a new translator, another toolchain), the stamp it used before is renamed aside at once and deleted on a thread of its own, unless another game folder that is still there and still on it uses it, a running game holds it, or keeping was asked for (LONGSHOT_SHADER_CACHE_KEEP=1, --shader-cache-keep: both kept, for an A/B). So a folder holds one set of caches at a time, as without the store. Its old files are not reused under the new stamp: they are of no use to another translator, and a toolchain's change keeps the stamp while the records' layout may differ. A folder an earlier run set aside is swept at a later start.
  • The rest of the clean-up is the launcher's (a deleted build's stamp, kept ones past what the player keeps): the record and <stamp>/longshot_last_used say what each folder uses and when a stamp was last used.
• runtime: gpu/vk/vk_paths.h shader_cache_path / shared_cache_path. vk_draw.cpp (the pipeline notes and what goes with them, the preparation's record, the pipeline cache's folder and its stamp check, the compute and state lists), vk_progcache.cpp disk_init (the program cache), vk_programs.cpp (the module ids) take their files through them. The module ids now go by path beside the programs: before, they used program_cache_path()'s folder, which is "" with the program cache off or held by another Longshot, and the file landed in the working folder. The game folder's own files stay on cache_path: the preparation-skipped marker, the driver's faults, the loader's notes go with the notes.
• host: host/hos_main.c --shader-cache <folder|0> sets LONGSHOT_SHADER_CACHE for the run, over the environment, and --shader-cache-keep sets LONGSHOT_SHADER_CACHE_KEEP=1; both entry points skip their values, and host/options_path.c's scan of the arguments (#729) no longer takes either for the image.
• build: nx2/make_game.py GAME_ENV_KEEP: the build's shader preparation keeps LONGSHOT_SHADER_CACHE and LONGSHOT_SHADER_CACHE_KEEP (the other LONGSHOT_ names still go), so the preparation fills the store the game will use; builder/linux/longshot-launch keeps it too.
• dev: scripts/dev_packs.py gives every game it starts one store (~/.cache/longshot-shaders, %LOCALAPPDATA%\Longshot\shaders) unless LONGSHOT_SHADER_CACHE says otherwise; scripts/shader_store.py (new) ls lists a store's stamps (size, last use, in use) and prune --keep N lets go of the stamps no running game holds and no game folder still uses (its users/ and record; --game <folder> names more), the newest N of the rest kept.
• docs: docs/development.md The shader cache store (the layout, what the game does with it, the A/B case, clean-up), the switch's row, the dev_packs bullet; the README's argument table; docs/design.md's cache note.
• tests: tests/pipeline_play_cache_test.cpp's stand-ins have shader_cache_path beside cache_path (it compiles pcache_init and the stamp check cut from vk_draw.cpp). tests/test_shader_store.py with tests/test_shader_store.cpp (the header against temporary folders: off and on, the parts, what cache/ gives the store, the record, a part held; and the renderer's call sites, game_env, dev_packs' names), tests/test_shader_store_tool.py (prune against a lock a child process holds as the game does, and the users' markers). The hand-over's cases in tests/test_shader_store.cpp: the old stamp let go; kept for another folder still on it, with keep, or held; a stale marker not counted; a set-aside folder swept. tests/test_options_path.py: neither argument is taken for the image (it fails before the fix). tests/test_prepare_at_build.py follows GAME_ENV_KEEP in both the Python and the shell launcher.

Measurements

Verdict: game-free: the store's logic passes against temporary folders on Windows (the stamp's and shared parts, take-in, merges, the record, a held part falling back, prune never letting go of a held or recorded stamp), and the renderer builds with it. What it saves (a second game folder that prepares nothing) is not measured in the game yet: it will be added here when the runs under Must be tested are made, by me or by anyone who runs them first.

Hardware: Windows 11 Pro, Ryzen 9 3900X, 64 GB; llvm-mingw clang 23 (the Builder's v2.0.14 tools)
Scene: none (game-free checks; a renderer-only build, RX_VK_ONLY)
Method: tests/test_shader_store.py --cxx <clang++> and tests/test_shader_store_tool.py on this branch; the checks pass or fail, so there is nothing to take a median of
How to reproduce: the two commands above; RX_VK_ONLY=1 python -I nx2/build.py --standalone for the renderer

	Before: main @ a00c485c	After: this PR @ a45e8c89
shader caches of two game folders with the same translation	two sets, each prepared (store: none)	one set in the store (test_shader_store's checks)
a game folder's existing cache/ when the store is first used	n/a	moved in, not prepared again (the checks)
renderer build (RX_VK_ONLY)	builds	builds: 0 failed, longshot_vk.dll linked

shader store: ok
test_shader_store: ok
test_shader_store_tool: Ran 4 tests ... OK
compiled in 35s, 0 failed
linked ...\work\sa\build\longshot_vk.dll: 13 MB (0s)

Flags and settings

One switch and one argument for the same thing, off by default: the launcher PR turns it on for the builds it makes.

Flag	What it is	Before this PR	After this PR	Effect	Helps / doesn't help / untested	Recommendation
LONGSHOT_SHADER_CACHE=<folder> / --shader-cache <folder>	the shader cache store	(none)	unset: the game folder's cache/	set: the shader caches shared by every game folder with the same stamp; a folder's old stamp let go when it moves to another	helps: side-by-side builds, rebuilds into new folders, dev_packs runtimes; doesn't help: one folder updated in place (its cache/ stays anyway); untested: in the game	off for now; set by the Longshot Launcher beside its builds (its PR), and by dev_packs.py
LONGSHOT_SHADER_CACHE_KEEP=1 / --shader-cache-keep	keep a folder's old stamp	(none)	unset: let go when no other folder uses it	set: kept, so going back to it prepares nothing	helps: A/B of two translations in one folder; costs a whole set of caches each	off: for development; the launcher's "keep older shader caches" sets it

Testing

Tested: tests/test_shader_store.py (the C++ checks with the Builder's llvm-mingw clang) and tests/test_shader_store_tool.py pass, Windows 11. python -I -B tests/run_all.py on this branch (before its rebase onto a00c485c, which adds only a skill file) and on main @ 9e6f231d, with the Builder's llvm-mingw clang (12 at once): this branch 542 passed, 18 failed, 35 skipped; main 540 passed, 18 failed, 35 skipped; the same 18 fail on both (test_appimage_player_build, test_cpm_discovery, test_cpp_preflight, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_options_budget_once, test_readback_late, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_walk_key). tests/test_shader_store.py also with g++ (libstdc++), for the library's branch of the folder's name. The renderer builds alone (RX_VK_ONLY). The Linux launcher's filter run under a shell: it drops LONGSHOT_X and keeps LONGSHOT_SHADER_CACHE.

Not tested:

• In the game: no run with a store yet.
• The host: host/hos_main.c's --shader-cache is not compiled yet. It needs the lift's runtime headers, so a whole build: the first run under Must be tested is one. It is two short blocks, as --compile-shaders' beside them.
• Linux: the flock path of the lock, and tests/test_longshot_launch.py (it runs only there).
• A second run's program cache merged into the store (not done here, see What changed).

Must be tested:

• Windows, any GPU, by hand: anyone with a build tree and their own game files (I will run it too and add what I see here). Build one game folder of this PR with LONGSHOT_SHADER_CACHE=<store> set for the build (its preparation fills the store), then a second game folder of the same commit with the same switch: expecting the second's longshot_log.txt to say vk: shader store: stamp <hex>: the shader caches are in <store>\<hex>\ and its first start to prepare nothing, and the store to hold one stamp folder. Then start both folders at once: the second says another Longshot uses the store's and runs from its own cache/. Then start a folder that has a full cache/ from main with the switch set: its files move into the store and it does not prepare again. Then a build of another translator into the second folder: its start says let go for the old stamp only if the first folder no longer uses it, and with --shader-cache-keep both stay. Each reaching play from a save.
• Linux, any GPU, by hand: the same steps, expecting the same log lines and a second folder that prepares nothing; and, with both folders started at once, <store>/<stamp>/.lock held by the first (the flock lock), the second running from its own cache/; and the AppImage's Play button (longshot-launch) starting the game with the switch still set.

Risks

What could break: with the switch set, a folder's old stamp is deleted when it moves to another and no other existing folder still uses it: a folder whose record or marker went missing could let go of a stamp another build still wanted (that build then prepares again; the shared lists stay). With the switch set, a game that read or wrote a shader cache file in the wrong place would prepare again, or lose what it compiled in play; the call sites are all checked by text, and unset nothing moves. Two runs of one stamp could clash: each part is held for the process's life, and the second run uses its own cache/. A store on another drive than the game folder makes the first take-in a copy (several GB) instead of a rename. The store grows by a whole set of caches for each translation until something lets the unused ones go: the launcher PR for players, scripts/shader_store.py prune for development stores.

Turning it off: unset LONGSHOT_SHADER_CACHE, or start with --shader-cache 0: the game folder's cache/ again. A store's files stay in the store until pruned.

Compatibility: nothing changes for players until the launcher sets the switch: unset, every file is where it was, and nothing takes the stamp sooner than before. With it set, a game folder's existing caches of the same stamp move into the store at its first start. Saves, the options file and the vk ABI are unchanged.

Checklist

Everyone

• One topic.
• The title is a Conventional Commit (type(scope): description) of the status line's type.
• Why gives the reason plainly and links the issues that fit (the tracker searched).
• Measurements has a verdict, or one line saying why there is nothing to measure.
• Testing names what was tested, what wasn't, and what must still be tested, where and how.
• Rebased on current main.
• python -I -B tests/run_all.py passes (any failure that also fails on main is named).
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
• A check under tests/ that reads changed source updated in this PR, not weakened.
• Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (not yet: the host change needs a whole build, under Must be tested)
• Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** medium · **Tested on:** Windows 11, game-free checks and a renderer build (not yet in the game) · **Needs:** a run in the game with two game folders (Windows and Linux, any GPU) ## Why Every game folder prepares its shaders on its own, about 25 minutes with Compile shaders All on a fast PC, even when another folder on the same PC translated the same shaders with the same translator a minute before: a stable and a nightly side by side, a rebuild into a new folder, each `dev_packs.py` runtime (which also carries a 10 GB copy of the caches). The caches are already stamped by what they are good for (`program_cache_stamp`), so a store keyed by that stamp lets every folder with the same translation use them in place. The design was agreed with the maintainers, including moving these files out of the game folder, where https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/321 put them. It complements https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/57 (an update in place keeps its `cache/` anyway) and replaces copying caches between folders (https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/78). It mirrors the object and lift stores (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/371, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/372). No Longshot issue fits (the tracker searched). ## What changed With `LONGSHOT_SHADER_CACHE=<folder>` (or `--shader-cache <folder>`), the game keeps its shader caches in `<folder>/<stamp>/` for what one translation made and `<folder>/shared/` for the lists good for any build, so a game folder whose stamp is in the store starts with its programs and pipelines and prepares nothing. Unset, empty or `0`, every file stays in the game folder's `cache/` as before, and the stamp is not taken any sooner. Nothing sets it for players yet: a launcher PR follows, setting it beside its builds and cleaning up the stamps no build uses. - runtime: `gpu/vk/shader_store.h` (new, standard library and the platform's file lock, no Vulkan): - `open_stamped` (at the first stamped file asked for, where the game took the stamp before) and `open_shared` (at the first list); each part held by `<part>/.lock` for the process's life: open with no sharing on Windows, `flock` elsewhere, so a tool can tell a part in use whatever it calls the path. A second run of the same stamp keeps that part's files in its own `cache/` for that run. - `consume`: a game folder's `cache/` of the same stamp is taken in at a start that holds the store. Files the store lacks are moved in (a rename, else a copy as `.part` renamed in); the program-side files only when `cache/`'s program cache carries this stamp, the pipeline cache only when its stamp file does. Pipeline notes and lists the store has get the records it lacks (`merge_records`, by their heads: the notes' stamp and record size, each list's magic and layout). A pipeline cache the store has a main file for goes in as a play file, merged at the next start by `play_files_merge`. A second run's program cache is left in its `cache/`: merging two program caches record by record is not done here. - A stamp's folder is named by the program cache's stamp and the toolchain (`folder_key`: the compiler's version and the C++ library's), not the stamp alone: the program cache keeps its records' `Shader::Info` as it lies in memory, and the stamp has the library's version for libc++ and Microsoft's but none for libstdc++ (HumanBean's point on https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/78). Before the store each version had its folder and its tools, so a cache never met another toolchain's; now one never does in the store either. The stamp itself is unchanged, so nobody's caches go. - An update in place, by default (`hand_over`): each game folder registers in the stamp it uses (`<stamp>/users/`, a file per folder naming it) and records it (`cache/longshot_shader_store.txt`). When a folder starts under another stamp (a new translator, another toolchain), the stamp it used before is renamed aside at once and deleted on a thread of its own, unless another game folder that is still there and still on it uses it, a running game holds it, or keeping was asked for (`LONGSHOT_SHADER_CACHE_KEEP=1`, `--shader-cache-keep`: both kept, for an A/B). So a folder holds one set of caches at a time, as without the store. Its old files are not reused under the new stamp: they are of no use to another translator, and a toolchain's change keeps the stamp while the records' layout may differ. A folder an earlier run set aside is swept at a later start. - The rest of the clean-up is the launcher's (a deleted build's stamp, kept ones past what the player keeps): the record and `<stamp>/longshot_last_used` say what each folder uses and when a stamp was last used. - runtime: `gpu/vk/vk_paths.h` `shader_cache_path` / `shared_cache_path`. `vk_draw.cpp` (the pipeline notes and what goes with them, the preparation's record, the pipeline cache's folder and its stamp check, the compute and state lists), `vk_progcache.cpp` `disk_init` (the program cache), `vk_programs.cpp` (the module ids) take their files through them. The module ids now go by path beside the programs: before, they used `program_cache_path()`'s folder, which is "" with the program cache off or held by another Longshot, and the file landed in the working folder. The game folder's own files stay on `cache_path`: the preparation-skipped marker, the driver's faults, the loader's notes go with the notes. - host: `host/hos_main.c` `--shader-cache <folder|0>` sets `LONGSHOT_SHADER_CACHE` for the run, over the environment, and `--shader-cache-keep` sets `LONGSHOT_SHADER_CACHE_KEEP=1`; both entry points skip their values, and `host/options_path.c`'s scan of the arguments (#729) no longer takes either for the image. - build: `nx2/make_game.py` `GAME_ENV_KEEP`: the build's shader preparation keeps `LONGSHOT_SHADER_CACHE` and `LONGSHOT_SHADER_CACHE_KEEP` (the other `LONGSHOT_` names still go), so the preparation fills the store the game will use; `builder/linux/longshot-launch` keeps it too. - dev: `scripts/dev_packs.py` gives every game it starts one store (`~/.cache/longshot-shaders`, `%LOCALAPPDATA%\Longshot\shaders`) unless `LONGSHOT_SHADER_CACHE` says otherwise; `scripts/shader_store.py` (new) `ls` lists a store's stamps (size, last use, in use) and `prune --keep N` lets go of the stamps no running game holds and no game folder still uses (its users/ and record; `--game <folder>` names more), the newest N of the rest kept. - docs: `docs/development.md` *The shader cache store* (the layout, what the game does with it, the A/B case, clean-up), the switch's row, the dev_packs bullet; the README's argument table; `docs/design.md`'s cache note. - tests: `tests/pipeline_play_cache_test.cpp`'s stand-ins have `shader_cache_path` beside `cache_path` (it compiles `pcache_init` and the stamp check cut from `vk_draw.cpp`). `tests/test_shader_store.py` with `tests/test_shader_store.cpp` (the header against temporary folders: off and on, the parts, what `cache/` gives the store, the record, a part held; and the renderer's call sites, `game_env`, dev_packs' names), `tests/test_shader_store_tool.py` (prune against a lock a child process holds as the game does, and the users' markers). The hand-over's cases in `tests/test_shader_store.cpp`: the old stamp let go; kept for another folder still on it, with keep, or held; a stale marker not counted; a set-aside folder swept. `tests/test_options_path.py`: neither argument is taken for the image (it fails before the fix). `tests/test_prepare_at_build.py` follows `GAME_ENV_KEEP` in both the Python and the shell launcher. ## Measurements **Verdict:** game-free: the store's logic passes against temporary folders on Windows (the stamp's and shared parts, take-in, merges, the record, a held part falling back, prune never letting go of a held or recorded stamp), and the renderer builds with it. What it saves (a second game folder that prepares nothing) is not measured in the game yet: it will be added here when the runs under Must be tested are made, by me or by anyone who runs them first. Hardware: Windows 11 Pro, Ryzen 9 3900X, 64 GB; llvm-mingw clang 23 (the Builder's v2.0.14 tools) Scene: none (game-free checks; a renderer-only build, `RX_VK_ONLY`) Method: `tests/test_shader_store.py --cxx <clang++>` and `tests/test_shader_store_tool.py` on this branch; the checks pass or fail, so there is nothing to take a median of How to reproduce: the two commands above; `RX_VK_ONLY=1 python -I nx2/build.py --standalone` for the renderer | | Before: main @ a00c485c | After: this PR @ a45e8c89 | |---|---|---| | shader caches of two game folders with the same translation | two sets, each prepared (store: none) | one set in the store (test_shader_store's checks) | | a game folder's existing `cache/` when the store is first used | n/a | moved in, not prepared again (the checks) | | renderer build (`RX_VK_ONLY`) | builds | builds: 0 failed, `longshot_vk.dll` linked | ``` shader store: ok test_shader_store: ok test_shader_store_tool: Ran 4 tests ... OK compiled in 35s, 0 failed linked ...\work\sa\build\longshot_vk.dll: 13 MB (0s) ``` ### Flags and settings One switch and one argument for the same thing, off by default: the launcher PR turns it on for the builds it makes. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `LONGSHOT_SHADER_CACHE=<folder>` / `--shader-cache <folder>` | the shader cache store | (none) | unset: the game folder's `cache/` | set: the shader caches shared by every game folder with the same stamp; a folder's old stamp let go when it moves to another | helps: side-by-side builds, rebuilds into new folders, dev_packs runtimes; doesn't help: one folder updated in place (its `cache/` stays anyway); untested: in the game | off for now; set by the Longshot Launcher beside its builds (its PR), and by dev_packs.py | | `LONGSHOT_SHADER_CACHE_KEEP=1` / `--shader-cache-keep` | keep a folder's old stamp | (none) | unset: let go when no other folder uses it | set: kept, so going back to it prepares nothing | helps: A/B of two translations in one folder; costs a whole set of caches each | off: for development; the launcher's "keep older shader caches" sets it | ## Testing **Tested:** `tests/test_shader_store.py` (the C++ checks with the Builder's llvm-mingw clang) and `tests/test_shader_store_tool.py` pass, Windows 11. `python -I -B tests/run_all.py` on this branch (before its rebase onto a00c485c, which adds only a skill file) and on main @ 9e6f231d, with the Builder's llvm-mingw clang (12 at once): this branch 542 passed, 18 failed, 35 skipped; main 540 passed, 18 failed, 35 skipped; the same 18 fail on both (test_appimage_player_build, test_cpm_discovery, test_cpp_preflight, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_options_budget_once, test_readback_late, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_walk_key). `tests/test_shader_store.py` also with g++ (libstdc++), for the library's branch of the folder's name. The renderer builds alone (`RX_VK_ONLY`). The Linux launcher's filter run under a shell: it drops `LONGSHOT_X` and keeps `LONGSHOT_SHADER_CACHE`. **Not tested:** - In the game: no run with a store yet. - The host: `host/hos_main.c`'s `--shader-cache` is not compiled yet. It needs the lift's runtime headers, so a whole build: the first run under Must be tested is one. It is two short blocks, as `--compile-shaders`' beside them. - Linux: the `flock` path of the lock, and `tests/test_longshot_launch.py` (it runs only there). - A second run's program cache merged into the store (not done here, see What changed). **Must be tested:** - Windows, any GPU, by hand: anyone with a build tree and their own game files (I will run it too and add what I see here). Build one game folder of this PR with `LONGSHOT_SHADER_CACHE=<store>` set for the build (its preparation fills the store), then a second game folder of the same commit with the same switch: expecting the second's `longshot_log.txt` to say `vk: shader store: stamp <hex>: the shader caches are in <store>\<hex>\` and its first start to prepare nothing, and the store to hold one stamp folder. Then start both folders at once: the second says `another Longshot uses the store's` and runs from its own `cache/`. Then start a folder that has a full `cache/` from main with the switch set: its files move into the store and it does not prepare again. Then a build of another translator into the second folder: its start says `let go` for the old stamp only if the first folder no longer uses it, and with `--shader-cache-keep` both stay. Each reaching play from a save. - Linux, any GPU, by hand: the same steps, expecting the same log lines and a second folder that prepares nothing; and, with both folders started at once, `<store>/<stamp>/.lock` held by the first (the `flock` lock), the second running from its own `cache/`; and the AppImage's Play button (`longshot-launch`) starting the game with the switch still set. ## Risks **What could break:** with the switch set, a folder's old stamp is deleted when it moves to another and no other existing folder still uses it: a folder whose record or marker went missing could let go of a stamp another build still wanted (that build then prepares again; the shared lists stay). With the switch set, a game that read or wrote a shader cache file in the wrong place would prepare again, or lose what it compiled in play; the call sites are all checked by text, and unset nothing moves. Two runs of one stamp could clash: each part is held for the process's life, and the second run uses its own `cache/`. A store on another drive than the game folder makes the first take-in a copy (several GB) instead of a rename. The store grows by a whole set of caches for each translation until something lets the unused ones go: the launcher PR for players, `scripts/shader_store.py prune` for development stores. **Turning it off:** unset `LONGSHOT_SHADER_CACHE`, or start with `--shader-cache 0`: the game folder's `cache/` again. A store's files stay in the store until pruned. **Compatibility:** nothing changes for players until the launcher sets the switch: unset, every file is where it was, and nothing takes the stamp sooner than before. With it set, a game folder's existing caches of the same stamp move into the store at its first start. Saves, the options file and the vk ABI are unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (not yet: the host change needs a whole build, under Must be tested) - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit 2026-11-15 08:16:05 +00:00

feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing ... b34e61c30b

LONGSHOT_SHADER_CACHE=<folder> (or --shader-cache <folder>): the game keeps
its shader caches in a store every game folder shares (gpu/vk/shader_store.h).
<store>/<stamp>/ holds what one translation made (the programs and their
index, the module ids, the pipeline notes and predictions, the preparation's
record, the driver's pipeline cache and its play files), by the program
cache's stamp, so a game folder whose stamp is there starts with them: no
preparation, no copy. <store>/shared/ holds the lists good for any build (the
compute and pipeline-state lists). Unset, empty or 0: the game folder's
cache/, as before, and nothing takes the stamp any sooner.

- The renderer takes those files by shader_cache_path / shared_cache_path
  (vk_paths.h); the store's part for the stamp opens at the first of them,
  where the game took the stamp before. The game folder's own markers, the
  driver's faults and the button prompts stay in cache/. The module ids now
  go beside the programs by path, not by program_cache_path's folder, which
  was "" with the program cache off or held, and put the file in the
  working folder.
- One game at a time on each part (<part>/.lock: open with no sharing on
  Windows, flock elsewhere, so a tool can tell); a second run of the same
  stamp keeps that part's files in its own cache/.
- A game folder's cache/ of the same stamp is taken in at the start that
  holds the store: files it lacks moved in (rename, else copy), notes and
  lists merged record by record, a pipeline cache as a play file. A second
  run's program cache stays in its cache/ (no record-by-record merge yet).
- The game never deletes from the store: it records the stamp it used
  (cache/longshot_shader_store.txt) and the stamp's last use, for the
  launcher. scripts/shader_store.py ls / prune --keep N cleans a
  development store.
- make_game.py's preparation and builder/linux/longshot-launch keep
  LONGSHOT_SHADER_CACHE (GAME_ENV_KEEP), so a build's preparation fills the
  store the game uses. scripts/dev_packs.py gives its games one store.

tests/test_shader_store.py (+ .cpp: the header against temporary folders:
off and on, the parts, what cache/ gives, the record, a part held),
tests/test_shader_store_tool.py (prune against a lock a child holds);
test_prepare_at_build.py follows GAME_ENV_KEEP. Docs: development.md's
"The shader cache store", the switch row, the dev_packs bullet; the
README's argument; design.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst force-pushed feat/shader-cache-store from b34e61c30b to 0e8569be74 2026-11-15 08:29:16 +00:00 Compare

dyst commented 2026-11-15 08:29:17 +00:00

Author

Member

• Copy link
• Copy Markdown

Rebased onto main 9e6f231d (past #729, which added options_path beside the store's paths in vk_paths.h: both kept). New: a stamp's folder is named by the stamp and the toolchain (folder_key: the compiler's version and the C++ library's), so caches never cross a toolchain in the store; HumanBean's point on LongshotLauncher#78 about libstdc++ (the stamp carries libc++'s and Microsoft's library version, not libstdc++'s) applies here too. The stamp itself is unchanged, so nobody's caches go. The record now says both (stamp=, translation=). Full suite again on this branch and main: no new failure; the store's check passes with clang and g++.

Rebased onto main 9e6f231d (past #729, which added options_path beside the store's paths in vk_paths.h: both kept). New: a stamp's folder is named by the stamp and the toolchain (folder_key: the compiler's version and the C++ library's), so caches never cross a toolchain in the store; HumanBean's point on LongshotLauncher#78 about libstdc++ (the stamp carries libc++'s and Microsoft's library version, not libstdc++'s) applies here too. The stamp itself is unchanged, so nobody's caches go. The record now says both (stamp=<folder>, translation=<program cache stamp>). Full suite again on this branch and main: no new failure; the store's check passes with clang and g++.

dyst force-pushed feat/shader-cache-store from 0e8569be74 to a45e8c89d0 2026-11-15 09:35:48 +00:00 Compare

dyst commented 2026-11-15 09:35:49 +00:00

Author

Member

• Copy link
• Copy Markdown

Pushed a45e8c89 (rebased onto main a00c485c): an update in place, in the store. Each game folder registers in the stamp it uses (/users/) and records it; when a folder starts under another stamp, its old one is renamed aside and deleted in the background unless another game folder still on it uses it, a running game holds it, or LONGSHOT_SHADER_CACHE_KEEP=1 / --shader-cache-keep asks to keep both (an A/B). So a folder holds one set of caches at a time, as without the store; the old files are not reused under the new stamp (another toolchain keeps the stamp, not the records' layout). Also fixed: host/options_path.c (#729) took --shader-cache's value for the image; it now skips both arguments, with a case in test_options_path.py that fails before. scripts/shader_store.py prune now reads the stamps' users itself. Renderer builds; full suite: no failure main doesn't have. Still WIP for the whole build (the host change) and the runs in the game under Must be tested.

Pushed a45e8c89 (rebased onto main a00c485c): an update in place, in the store. Each game folder registers in the stamp it uses (<stamp>/users/) and records it; when a folder starts under another stamp, its old one is renamed aside and deleted in the background unless another game folder still on it uses it, a running game holds it, or LONGSHOT_SHADER_CACHE_KEEP=1 / --shader-cache-keep asks to keep both (an A/B). So a folder holds one set of caches at a time, as without the store; the old files are not reused under the new stamp (another toolchain keeps the stamp, not the records' layout). Also fixed: host/options_path.c (#729) took --shader-cache's value for the image; it now skips both arguments, with a case in test_options_path.py that fails before. scripts/shader_store.py prune now reads the stamps' users itself. Renderer builds; full suite: no failure main doesn't have. Still WIP for the whole build (the host change) and the runs in the game under Must be tested.

mooki added the

area/renderer

kind

feature

platform/linux

platform/windows

priority

low

labels 2026-11-15 13:59:15 +00:00

HumanBean force-pushed feat/shader-cache-store from a45e8c89d0 to fea52a0be3 2026-11-15 18:46:09 +00:00 Compare

HumanBean commented 2026-11-15 18:48:06 +00:00

Owner

• Copy link
• Copy Markdown

Rebased onto main @ 7bacf846 (no conflicts); pushed as fea52a0b. No tests were run here; the test boxes stay yours.

Rebased onto `main` @ 7bacf846 (no conflicts); pushed as fea52a0b. No tests were run here; the test boxes stay yours.

Some checks are pending

Hide checks

PR description / Description follows the template (pull_request_target) Waiting to run

Details

CI / Conventional Commit title (pull_request) Waiting to run

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

CI / Workflow lint (actionlint) (pull_request) Waiting to run

Details

Guide screenshots / Guide images and local UI source review (pull_request) Waiting to run

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Third-party parts / As their notes say (pull_request) Waiting to run

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'feat/shader-cache-store:dyst-feat/shader-cache-store'

git switch 'dyst-feat/shader-cache-store'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

beanstalk: approved @ 1697603

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 2c5a62e

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 4fd82b7

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 64d6d09

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 6f91fae

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 70a78f9

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 8d361dc

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ 9d8356e

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ abf2227

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ d82f202

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ e5f1e40

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: approved @ ee8ab5f

Set by Beanstalk for show; the farm's database is what counts

  

ci: approved

A maintainer approved CircleCI for this fork PR (sticky; removed after 14 quiet days)

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Affects Linux (a change for every system carries both Windows and Linux)

  

platform/macos

macOS (Apple silicon or Intel Macs)

  

platform/steam-deck

Steam Deck

  

platform/windows

Affects Windows (a change for every system carries both Windows and Linux)

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

regression

Worked in an earlier build, broken in a newer one: highest priority, above other bugs

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

  

ci: approved

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

beanstalk: approved @ 1697603

beanstalk: approved @ 2c5a62e

beanstalk: approved @ 4fd82b7

beanstalk: approved @ 64d6d09

beanstalk: approved @ 6f91fae

beanstalk: approved @ 70a78f9

beanstalk: approved @ 8d361dc

beanstalk: approved @ 9d8356e

beanstalk: approved @ abf2227

beanstalk: approved @ d82f202

beanstalk: approved @ e5f1e40

beanstalk: approved @ ee8ab5f

ci: approved

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/macos

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

regression

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

ci: approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!737

No description provided.