perf(cache): one shader store for every game folder: each program's SPIR-V kept once, by its content #1809

Merged
HumanBean merged 10 commits from madebyjordan/Longshot:perf/shared-spirv-pack into main 2026-11-20 16:59:50 +00:00
Member

Summary

Every Longshot build kept its own ~8 GB of translated shader code (cache\longshot_programs.bin), although a stable and a nightly build translate 99.6% of the game's shaders to exactly the same SPIR-V: a player's v3.0.0 and nightly-486 held 47,592 byte-identical programs of 47,798, 14.7 GB stored for 3.5 GB distinct (#1649, issuecomment-55045). Each further build (a test or pull request build, a dev runtime) added another 8 GB, which players see as "every build takes a whole game's worth of space".

This keeps each program's SPIR-V once, by its content, in one shader store every game folder of the player shares (%LOCALAPPDATA%\Longshot\shaders, Linux $XDG_CACHE_HOME/longshot/shaders or ~/.cache/longshot/shaders); each build's longshot_programs.bin keeps its records with a place in the store instead of the words. A build's program cache goes from 8.0 GB to 0.15 GB, and further builds, on the same translator or another, add nothing to the 3.7 GB store.

Read first:

  • gpu/vk/spirv_pack.h (new, header only): the store. spirv.pack (blobs: a 24-byte head with the 128-bit hash and the word count, then the words) and spirv.idx (their places), both behind a head with the store's random id, only ever appended to. An append takes a lock on spirv.lock (waited for a minute at most), reads what other Longshots added since, and writes the blob, then its entry. A blob is used only when its words hash to its head's; one that does not (cut short, zeros a crash left) is translated and written again, and its place is not given out again. Appends land past every place an entry names, and an index entry cut short by a full disk is padded to a whole one, so one bad write spoils no other blob. A store file another program holds at the start is not taken for a missing one: that run goes without the store.
  • gpu/vk/vk_progcache.cpp: kProgramCacheFormat 1 -> 2; a stage's code is [u64 words][words] as before or [u64 words | 1<<63][i64 place]; get_code / read_code serve every reader of a stage's code; the file's head names its store, so a cache of a store that is gone starts over, and one whose store did not open this run is kept for the next.
  • scripts/devtools/packs.py: a runtime prepares again when kProgramCacheFormat changes, not only the translator's stamp (the Test Farm's first run played this PR unprepared because of it); scripts/devtools/vcs.py gains show() for that.
  • scripts/validate_spirv.py reads stages from the store (--store).

Fixes #1759. See #1649 (its option B, across game folders; its option A, compressing each blob, fits on top in the store) and #737 (dyst's stamp-keyed store, which shares whole caches between folders of the same translation; the two compose, issuecomment-55159). The launcher side is LongshotLauncher #210.

Testing

Tested: Windows 11 Home 26200, AMD Radeon RX 6700 XT (driver 0x800161), Ryzen 5 5600, headless 1920x1080, from the repository's kokiri-start save (title, Continue, Load).

  • On this head (720c6988, on main 89c3d21e: futondestroyer's review fixed, the records' hash, kProgramCacheFormat 3), as a dev.py pack over the base at main 8e9cfc3a, into an empty default store: the runtime's preparation prepared 38809 shader programs (46598 new, 69 failed, ...) in 118.7 s, as main's; its longshot_programs.bin 150,840,657 bytes (each stored stage's record 16 bytes longer for its hash) and longshot_programs.idx 1,133,936 bytes; the store filled to 3,736,516,412 bytes (spirv.pack, 50,685 blobs, the same as before to the byte) and 1,621,936 bytes (spirv.idx). Then played with the default load at create: 47798 program keys kept, 147305 KB, 50685 SPIR-V blobs, 10914 shader modules let go once their programs' noted pipelines were built, Kokiri Forest, Hyrule Field, the Lost Woods, the Death Mountain Trail and Zora's Fountain drawn, 0 draws left out, 0 has no SPIR-V / did not read / not whole / another shader store lines, quit cleanly. The fixes' game-free checks: last item.
  • Before the review's fixes (993b9a04, on main 1780fbe0, with #1844's drop of loaded modules), as a dev.py pack over the base at main 8e9cfc3a, into the default store already filled: the runtime's preparation prepared 38809 shader programs (46598 new, 69 failed, ...), as main's, its longshot_programs.bin 149,112,385 bytes, the store unchanged at 3,736,516,412 bytes (50,685 blobs). Then played with the default load at create: 47798 program keys kept, 145617 KB, 50685 SPIR-V blobs, 13594 shader modules let go once their programs' noted pipelines were built, Kokiri Forest, Hyrule Field, the Lost Woods, the Death Mountain Trail and Zora's Fountain drawn, 0 draws left out, 0 has no SPIR-V / did not read / not whole lines, quit cleanly. The same tour on the head before the last rebase (f78a5324, main a3565ac5) with RX_VK_PRECOMPILE=0 read every program from the store the same way (translation 187 ms in all).
  • Rebased since onto main 89c3d21e (head e4c12951): main's six new commits touch none of this PR's files but gpu/vk/switches.def (merged without a conflict); the renderer built again on it (dev.py vk, its exports checked) and the store's and the cache's checks pass, so the run above stands.
  • After the cold review's fixes (a1c39f91, before the rebase): two starts on an empty store of their own with RX_VK_PRECOMPILE=0. The first translated 1,614 blobs (129 MB) in 14,063 ms; the second kept 1,137 programs and translated 228 ms in all, reading the rest from the store with the hash check; 0 read failures.
  • Before (7c88d606): three game folders prepared one after another into one empty store, two of them on different translator stamps (the table below); one played through Kokiri Forest, Hyrule Field, Kakariko, Goron City, Zora's Domain, the Deku Tree and the Forest Temple, every scene drawn, 0 read failures.
  • Game-free, the review's fixes (720c6988): test_spirv_pack adds another blob at a place asked for by another hash, a pack cut under an entry whose place a blob of the same length takes, an index entry 64 TB past the pack's end (nothing written out to it), and a store made again while another Store reads it (Windows: not replaced; Linux: new files, the reader keeps its own), each caught when its fix is undone (a read ignoring the hash, the store remade in place); test_prepare_at_build (make_game.game_env keeps LONGSHOT_SHADER_STORE, a folder or 0, and longshot-launch keeps the same name) and test_longshot_launch (the AppImage's Play passes it on: Linux, CI); test_validate_spirv (a cache passes with its store, fails with none or another, and a reference with another hash is not read); test_loaded_modules_drop, test_program_cache_cleanup, test_program_cache_one_variant again. tests/run_all.py on Windows with llvm-mingw: 696 passed, 62 failed, 124 skipped; the same 62 fail on main 89c3d21e on this PC, each the same way.
  • Game-free, before: test_spirv_pack (the store alone: one blob per content, two writers each finding the other's, the next start, a write and a pack cut short, words changed or zeroed under an entry, an index entry cut short, a store file held at the start, a deleted store's new id; disk_init's choices by its source), run against mutations of the store's fixes (the hash check, the bad places, the head check, the padding of the pack and of the index, the remade store), each caught; test_program_cache_cleanup and test_program_cache_one_variant (every case with and without the store, the whole read's code marked as the store's), test_validate_spirv, test_dev_packs' format check, test_switch_table, test_doc_anchors, and #1844's test_loaded_modules_drop run again with the store (a dropped program's modules remade from their places in it). mise run test (tests/run_all.py --strict) on Windows with llvm-mingw: 682 passed, 138 failed, 48 skipped; main @ a3565ac5 on the same PC: 681 passed and the same 138 failed (most need tools this PC lacks, such as the Vulkan headers and spirv-val, which --strict counts as failures), so none fails here that passes on main.

Still to test:

  • linux (any GPU): build this branch, let the runtime prepare the shaders, play about a minute from the kokiri-start save, expecting vk: shader store ...: N SPIR-V blobs with N in the tens of thousands, vk: program cache ...: N program keys kept, K KB with K about 150,000 (about 8,000,000 on main), and no has no SPIR-V line (@Beanstalk)
  • windows, nvidia: the same, expecting the same (@Beanstalk)

Measurements

Verdict: a build's program cache is 0.15 GB instead of 8.0 GB, and further builds share the 3.7 GB store: a second build added 0 bytes to it on the same translator stamp and 0 bytes on another. The same programs are prepared.

Setup: Windows 11, RX 6700 XT, as Testing. Before: the dev base at main 8e9cfc3a, prepared by main. After: this PR as a pack over that base (runtime A); runtimes B and C its folder with longshot_vk.dll built from this PR plus a test-only commit (B: the same translator stamp; C: another, 30bc04d7a5198418 against c2295d1c2762bd43), prepared one after another into one empty store. Sizes from the files, counts from the log; sizes, not timings, so one run each. Measured on 7c88d606; this head (993b9a04, on main 1780fbe0) prepared the same cache to the byte and added nothing to the store (Testing).

Before: main @ 8e9cfc3a After: this PR @ 7c88d606
longshot_programs.bin of a prepared game folder 8,027,711,307 bytes 149,112,385 bytes
shader store after the first folder (none) 3,736,516,412 bytes, 50,685 blobs
what a second folder adds, same stamp (B) another 8.0 GB 149 MB of its own, store +0 bytes
what a third folder adds, another stamp (C) another 8.0 GB 149 MB of its own, store +0 bytes
three folders' program caches together ~24.1 GB 4.18 GB
everything on disk for three folders, indexes included ~24.1 GB + 3 × longshot_programs.idx 4.18 GB + 3 × longshot_programs.idx (1.1 MB each) + spirv.idx (1.6 MB)
programs prepared 38,809 (46,598 new, 69 failed) 38,809 (46,598 new, 69 failed), in each of A, B and C

Nothing per frame changes: a program's SPIR-V is read from another file at its load, the same size, and checked against its hash once. Since the review's fixes (720c6988) a stored stage's record is 16 bytes longer for its hash: the program file of a prepared folder is 150,840,657 bytes (+1.2%), the store the same to the byte, spirv.idx 1,621,936 bytes (16 + 32 bytes a blob). The format change still costs every player one preparation after updating (Risks).

Risks

  • Every player prepares their shaders once more after updating: the cache's format changes (the game's prepare record holds it, so the first start does it).
  • A store deleted while no game runs (by hand, or the launcher's Free up space) makes each game folder's next start prepare again; a damaged blob is translated and written again.
  • The store only grows: SPIR-V no build uses any more stays until the store is deleted. Pruning it fits #737's cleanup tool or the launcher's Free up space.
  • Turning it off: LONGSHOT_SHADER_STORE=0 (each record keeps its SPIR-V, the cache starting over once), or a revert. The builders keep the switch for the game they start (the preparation at the build, the AppImage's Play).
  • With #737 (one shared record folder per translator stamp): two games with different LONGSHOT_SHADER_STORE folders would each start the shared record file over for their own store. Before both are on, the store's identity needs to be one per shared record folder (#737 picks the store, or a record names its store per record); not an issue while each game folder has its own records, as here.
  • Deleting the store is the launcher's Free up space (LongshotLauncher #210): the contract is the three files deleted together. A delete while a game has them open is refused on Windows, and on Linux the game keeps the files it has open; a half-deleted store is made again (new files renamed into place) and every folder prepares again. #210's side to be checked against this before both merge.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A switch added or changed: it ships at the default the measurements chose, with its row in gpu/vk/switches.def.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows built and ran. Not built on Linux here: this PC has no Linux build, so the Linux build and run are left to the Test Farm and the team's Linux testers, Still to test; marked ready for review without it on purpose)
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

🤖 Generated with Claude Code

## Summary Every Longshot build kept its own ~8 GB of translated shader code (`cache\longshot_programs.bin`), although a stable and a nightly build translate 99.6% of the game's shaders to exactly the same SPIR-V: a player's v3.0.0 and nightly-486 held 47,592 byte-identical programs of 47,798, 14.7 GB stored for 3.5 GB distinct (#1649, issuecomment-55045). Each further build (a test or pull request build, a dev runtime) added another 8 GB, which players see as "every build takes a whole game's worth of space". This keeps each program's SPIR-V once, by its content, in one shader store every game folder of the player shares (`%LOCALAPPDATA%\Longshot\shaders`, Linux `$XDG_CACHE_HOME/longshot/shaders` or `~/.cache/longshot/shaders`); each build's `longshot_programs.bin` keeps its records with a place in the store instead of the words. A build's program cache goes from 8.0 GB to 0.15 GB, and further builds, on the same translator or another, add nothing to the 3.7 GB store. Read first: - `gpu/vk/spirv_pack.h` (new, header only): the store. `spirv.pack` (blobs: a 24-byte head with the 128-bit hash and the word count, then the words) and `spirv.idx` (their places), both behind a head with the store's random id, only ever appended to. An append takes a lock on `spirv.lock` (waited for a minute at most), reads what other Longshots added since, and writes the blob, then its entry. A blob is used only when its words hash to its head's; one that does not (cut short, zeros a crash left) is translated and written again, and its place is not given out again. Appends land past every place an entry names, and an index entry cut short by a full disk is padded to a whole one, so one bad write spoils no other blob. A store file another program holds at the start is not taken for a missing one: that run goes without the store. - `gpu/vk/vk_progcache.cpp`: `kProgramCacheFormat` 1 -> 2; a stage's code is `[u64 words][words]` as before or `[u64 words | 1<<63][i64 place]`; `get_code` / `read_code` serve every reader of a stage's code; the file's head names its store, so a cache of a store that is gone starts over, and one whose store did not open this run is kept for the next. - `scripts/devtools/packs.py`: a runtime prepares again when `kProgramCacheFormat` changes, not only the translator's stamp (the Test Farm's first run played this PR unprepared because of it); `scripts/devtools/vcs.py` gains `show()` for that. - `scripts/validate_spirv.py` reads stages from the store (`--store`). Fixes #1759. See #1649 (its option B, across game folders; its option A, compressing each blob, fits on top in the store) and #737 (dyst's stamp-keyed store, which shares whole caches between folders of the same translation; the two compose, issuecomment-55159). The launcher side is LongshotLauncher #210. ## Testing **Tested:** Windows 11 Home 26200, AMD Radeon RX 6700 XT (driver 0x800161), Ryzen 5 5600, headless 1920x1080, from the repository's kokiri-start save (title, Continue, Load). - On this head (720c6988, on main 89c3d21e: futondestroyer's review fixed, the records' hash, kProgramCacheFormat 3), as a `dev.py` pack over the base at main 8e9cfc3a, into an **empty** default store: the runtime's preparation `prepared 38809 shader programs (46598 new, 69 failed, ...)` in 118.7 s, as main's; its `longshot_programs.bin` 150,840,657 bytes (each stored stage's record 16 bytes longer for its hash) and `longshot_programs.idx` 1,133,936 bytes; the store filled to 3,736,516,412 bytes (`spirv.pack`, 50,685 blobs, the same as before to the byte) and 1,621,936 bytes (`spirv.idx`). Then played with the default load at create: `47798 program keys kept, 147305 KB`, `50685 SPIR-V blobs`, `10914 shader modules let go once their programs' noted pipelines were built`, Kokiri Forest, Hyrule Field, the Lost Woods, the Death Mountain Trail and Zora's Fountain drawn, 0 draws left out, 0 `has no SPIR-V` / `did not read` / `not whole` / `another shader store` lines, quit cleanly. The fixes' game-free checks: last item. - Before the review's fixes (993b9a04, on main 1780fbe0, with #1844's drop of loaded modules), as a `dev.py` pack over the base at main 8e9cfc3a, into the default store already filled: the runtime's preparation `prepared 38809 shader programs (46598 new, 69 failed, ...)`, as main's, its `longshot_programs.bin` 149,112,385 bytes, the store unchanged at 3,736,516,412 bytes (50,685 blobs). Then played with the default load at create: `47798 program keys kept, 145617 KB`, `50685 SPIR-V blobs`, `13594 shader modules let go once their programs' noted pipelines were built`, Kokiri Forest, Hyrule Field, the Lost Woods, the Death Mountain Trail and Zora's Fountain drawn, 0 draws left out, 0 `has no SPIR-V` / `did not read` / `not whole` lines, quit cleanly. The same tour on the head before the last rebase (f78a5324, main a3565ac5) with `RX_VK_PRECOMPILE=0` read every program from the store the same way (translation 187 ms in all). - Rebased since onto main 89c3d21e (head e4c12951): main's six new commits touch none of this PR's files but `gpu/vk/switches.def` (merged without a conflict); the renderer built again on it (`dev.py vk`, its exports checked) and the store's and the cache's checks pass, so the run above stands. - After the cold review's fixes (a1c39f91, before the rebase): two starts on an empty store of their own with `RX_VK_PRECOMPILE=0`. The first translated 1,614 blobs (129 MB) in 14,063 ms; the second kept 1,137 programs and translated 228 ms in all, reading the rest from the store with the hash check; 0 read failures. - Before (7c88d606): three game folders prepared one after another into one empty store, two of them on different translator stamps (the table below); one played through Kokiri Forest, Hyrule Field, Kakariko, Goron City, Zora's Domain, the Deku Tree and the Forest Temple, every scene drawn, 0 read failures. - Game-free, the review's fixes (720c6988): `test_spirv_pack` adds another blob at a place asked for by another hash, a pack cut under an entry whose place a blob of the same length takes, an index entry 64 TB past the pack's end (nothing written out to it), and a store made again while another Store reads it (Windows: not replaced; Linux: new files, the reader keeps its own), each caught when its fix is undone (a read ignoring the hash, the store remade in place); `test_prepare_at_build` (`make_game.game_env` keeps `LONGSHOT_SHADER_STORE`, a folder or 0, and `longshot-launch` keeps the same name) and `test_longshot_launch` (the AppImage's Play passes it on: Linux, CI); `test_validate_spirv` (a cache passes with its store, fails with none or another, and a reference with another hash is not read); `test_loaded_modules_drop`, `test_program_cache_cleanup`, `test_program_cache_one_variant` again. `tests/run_all.py` on Windows with llvm-mingw: 696 passed, 62 failed, 124 skipped; the same 62 fail on main 89c3d21e on this PC, each the same way. - Game-free, before: `test_spirv_pack` (the store alone: one blob per content, two writers each finding the other's, the next start, a write and a pack cut short, words changed or zeroed under an entry, an index entry cut short, a store file held at the start, a deleted store's new id; `disk_init`'s choices by its source), run against mutations of the store's fixes (the hash check, the bad places, the head check, the padding of the pack and of the index, the remade store), each caught; `test_program_cache_cleanup` and `test_program_cache_one_variant` (every case with and without the store, the whole read's code marked as the store's), `test_validate_spirv`, `test_dev_packs`' format check, `test_switch_table`, `test_doc_anchors`, and #1844's `test_loaded_modules_drop` run again with the store (a dropped program's modules remade from their places in it). `mise run test` (`tests/run_all.py --strict`) on Windows with llvm-mingw: 682 passed, 138 failed, 48 skipped; main @ a3565ac5 on the same PC: 681 passed and the same 138 failed (most need tools this PC lacks, such as the Vulkan headers and spirv-val, which --strict counts as failures), so none fails here that passes on main. **Still to test:** - linux (any GPU): build this branch, let the runtime prepare the shaders, play about a minute from the kokiri-start save, expecting `vk: shader store ...: N SPIR-V blobs` with N in the tens of thousands, `vk: program cache ...: N program keys kept, K KB` with K about 150,000 (about 8,000,000 on main), and no `has no SPIR-V` line (@Beanstalk) - windows, nvidia: the same, expecting the same (@Beanstalk) ## Measurements **Verdict:** a build's program cache is 0.15 GB instead of 8.0 GB, and further builds share the 3.7 GB store: a second build added 0 bytes to it on the same translator stamp and 0 bytes on another. The same programs are prepared. Setup: Windows 11, RX 6700 XT, as Testing. Before: the dev base at main 8e9cfc3a, prepared by main. After: this PR as a pack over that base (runtime A); runtimes B and C its folder with `longshot_vk.dll` built from this PR plus a test-only commit (B: the same translator stamp; C: another, `30bc04d7a5198418` against `c2295d1c2762bd43`), prepared one after another into one empty store. Sizes from the files, counts from the log; sizes, not timings, so one run each. Measured on 7c88d606; this head (993b9a04, on main 1780fbe0) prepared the same cache to the byte and added nothing to the store (Testing). | | Before: main @ 8e9cfc3a | After: this PR @ 7c88d606 | |---|---|---| | `longshot_programs.bin` of a prepared game folder | 8,027,711,307 bytes | 149,112,385 bytes | | shader store after the first folder | (none) | 3,736,516,412 bytes, 50,685 blobs | | what a second folder adds, same stamp (B) | another 8.0 GB | 149 MB of its own, store +0 bytes | | what a third folder adds, another stamp (C) | another 8.0 GB | 149 MB of its own, store +0 bytes | | three folders' program caches together | ~24.1 GB | 4.18 GB | | everything on disk for three folders, indexes included | ~24.1 GB + 3 × `longshot_programs.idx` | 4.18 GB + 3 × `longshot_programs.idx` (1.1 MB each) + `spirv.idx` (1.6 MB) | | programs prepared | 38,809 (46,598 new, 69 failed) | 38,809 (46,598 new, 69 failed), in each of A, B and C | Nothing per frame changes: a program's SPIR-V is read from another file at its load, the same size, and checked against its hash once. Since the review's fixes (720c6988) a stored stage's record is 16 bytes longer for its hash: the program file of a prepared folder is 150,840,657 bytes (+1.2%), the store the same to the byte, `spirv.idx` 1,621,936 bytes (16 + 32 bytes a blob). The format change still costs every player one preparation after updating (Risks). ## Risks - Every player prepares their shaders once more after updating: the cache's format changes (the game's prepare record holds it, so the first start does it). - A store deleted while no game runs (by hand, or the launcher's Free up space) makes each game folder's next start prepare again; a damaged blob is translated and written again. - The store only grows: SPIR-V no build uses any more stays until the store is deleted. Pruning it fits #737's cleanup tool or the launcher's Free up space. - Turning it off: `LONGSHOT_SHADER_STORE=0` (each record keeps its SPIR-V, the cache starting over once), or a revert. The builders keep the switch for the game they start (the preparation at the build, the AppImage's Play). - With #737 (one shared record folder per translator stamp): two games with different `LONGSHOT_SHADER_STORE` folders would each start the shared record file over for their own store. Before both are on, the store's identity needs to be one per shared record folder (#737 picks the store, or a record names its store per record); not an issue while each game folder has its own records, as here. - Deleting the store is the launcher's Free up space (LongshotLauncher #210): the contract is the three files deleted together. A delete while a game has them open is refused on Windows, and on Linux the game keeps the files it has open; a half-deleted store is made again (new files renamed into place) and every folder prepares again. #210's side to be checked against this before both merge. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [x] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A switch added or changed: it ships at the default the measurements chose, with its row in `gpu/vk/switches.def`. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [ ] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows built and ran. Not built on Linux here: this PC has no Linux build, so the Linux build and run are left to the Test Farm and the team's Linux testers, Still to test; marked ready for review without it on purpose) - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
A game folder's longshot_programs.bin held each stage's SPIR-V itself: ~7.5 GB a build, half of it the same SPIR-V
again, and the same again in every other build the player keeps. Two installs on one PC (stable v3.0.0 and nightly-486,
different translator stamps): 47,592 of 47,798 programs translated to byte-identical SPIR-V, 14.7 GB stored for 3.5 GB
distinct.

- gpu/vk/spirv_pack.h: the shader store, %LOCALAPPDATA%\Longshot\shaders (Linux: $XDG_CACHE_HOME/longshot/shaders;
  LONGSHOT_SHADER_STORE names another folder, 0 for none). spirv.pack holds each distinct SPIR-V once (128-bit hash of
  its words), spirv.idx their places; only appended to. Several Longshots may write at once: an append takes the store's
  lock (a named mutex on Windows, flock elsewhere), reads what the others added first, then writes the blob and its entry.
- vk_progcache.cpp: a stage's code in a record is its words, as before, or [count | bit 63][place] in the store.
  kProgramCacheFormat 1 -> 2 (one more preparation for every player). The file's head names the store (its unused second
  word): a cache of a store that is gone (deleted, the launcher's Free up space) starts over instead of naming blobs
  that are not there. Every reader of a stage's code takes the store's references: the whole read, the light read of a
  compute variant, the code by its place (complete_from, complete_graphics), the prediction. A put that fails (a full
  disk) keeps the words in the record.
- vk_programs.cpp: a light-read record's bytes without its code count none for code in the store.
- scripts/validate_spirv.py reads a stage from the store (--store, default the player's).
- Tests: tests/test_spirv_pack.py (the store alone: dedupe, two writers, restart, a write or a pack cut short, a deleted
  store); test_program_cache_cleanup and test_program_cache_one_variant run every case with and without the store;
  test_validate_spirv has a stage in the store.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs(cache): the shader store's switch, and its run on Windows
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 10s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 13s
beanstalk/approval approved by madebyjordan (reviewed)
beanstalk/linux 2/1 fail
beanstalk/windows+nvidia 1/1 fail
e2ac1c128c
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

Tested in game: RX 6700 XT, driver 0x800161, Windows 11: three game folders prepared into one store (two translator stamps): each folder's longshot_programs.bin 149 MB instead of 8.0 GB, the second and third added 0 bytes to the 3.7 GB store; one played headless from kokiri-start through Kokiri Forest, Hyrule Field, Kakariko, Goron City, Zora's Domain, the Deku Tree and the Forest Temple, every scene drawn, no failed read

Tested in game: RX 6700 XT, driver 0x800161, Windows 11: three game folders prepared into one store (two translator stamps): each folder's longshot_programs.bin 149 MB instead of 8.0 GB, the second and third added 0 bytes to the 3.7 GB store; one played headless from kokiri-start through Kokiri Forest, Hyrule Field, Kakariko, Goron City, Zora's Domain, the Deku Tree and the Forest Temple, every scene drawn, no failed read
Author
Member

@Beanstalk linux (any GPU): build this branch, prepare the shaders, then play about a minute from the kokiri-start save. Please report the log's vk: shader store ...: N SPIR-V blobs line, the size of cache/longshot_programs.bin (about 0.15 GB expected, against about 8 GB on main) and any has no SPIR-V line (none expected). This checks the flock lock and the XDG path, which I could not run.

@Beanstalk windows nvidia: the same steps and the same three things to report.

🤖 Generated with Claude Code

@Beanstalk linux (any GPU): build this branch, prepare the shaders, then play about a minute from the kokiri-start save. Please report the log's `vk: shader store ...: N SPIR-V blobs` line, the size of cache/longshot_programs.bin (about 0.15 GB expected, against about 8 GB on main) and any `has no SPIR-V` line (none expected). This checks the flock lock and the XDG path, which I could not run. @Beanstalk windows nvidia: the same steps and the same three things to report. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Member
Beanstalk test farm · an earlier run: commit e2ac1c1 (the newer results are in a comment below)

Commit e2ac1c1 · tier reviewed · approved by madebyjordan · dashboard

What will be tested · AI plan (gpt-6.1-terra)

On Linux with any GPU and on Windows with NVIDIA, the plan compares main and the PR after shader preparation, then runs about a minute from the Kokiri start save. It records the shader-store blob count and the program-cache size reported at startup, and compares a gameplay image and frame times. A missing store log, an unexpectedly sized PR program cache, a missing-SPIR-V message, a crash, or a significant visual or frame-time regression fails the PR. The plan cannot directly inspect file sizes, set a custom XDG path, or run concurrent writers, so those aspects of the request are not fully tested.

  • Where: linux (1-2 reports), windows+nvidia (1-2 reports).
  • kokiri-store: from the repository's save kokiri-start (switches RX_VK_PRECOMPILE=0), then move the stick, take the picture kokiri to compare with main, measure play for 60 s (after 0 s warm-up); 1 run on each side.
  • Measured: crashes, frame times, stutters and hitches, the game's log, numbers from the game's log.
  • The PR fails if: it crashes; or a picture differs from main by more than 5%; or frame time (p99) is more than 20% above main; or forbidden log lines is above 0; or expected log lines missing is above 0; or value_program_cache_kb is above 300000; or value_program_cache_kb is below 10000.
  • Time: about 12 min of game runs per tester (2 game starts), plus the two builds.

Done (started 22:51 UTC, last report 23:28 UTC). Comment @Beanstalk to run it again.

Target Reports Crash Frame p99 (ms) vs main VRAM peak (MB) vs main Result
linux · 1: linux+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 5070, 620.70.6; 2: linux+amd+radv+igpu AMD Radeon 760M Graphics (RADV PHOENIX), 26.0.8 2/1 1 2 none 35.3 vs 60.5 (−42%) — fail ⚠ outlier
windows+nvidia · 1: windows+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0; 2: windows+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 3070, 621.24.0 2/1 1 2 none 19.6 vs 26.3 (−26%) — fail

Measurements · linux

Verdict: fail on linux (2/1 report): kokiri-store value_program_cache_kb.min; kokiri-store screenshot.max_diff failed.

Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 5070, 620.70.6 AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM; 2: linux+amd+radv+igpu AMD Radeon 760M Graphics (RADV PHOENIX), 26.0.8 AMD Ryzen 5 230 w/ Radeon 760M Graphics, 16 GB RAM
Scene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (@Beanstalk on this PR runs it again)

Before: main @ 621dae5 After: this PR @ e2ac1c1
kokiri-store · play · frame time (p99) (ms) 60.5 35.3 (−42%)
kokiri-store · forbidden log lines 0.0 0.0
kokiri-store · expected log lines missing 1.0 0.0 (−100%)
kokiri-store · value_program_cache_kb 7,941,497 0.0 (−100%)

Flags and settings

None: the plan varied no setting, switch or build option.

Markdown for the PR description
## Measurements

**Verdict:** fail on linux (2/1 report): `kokiri-store value_program_cache_kb.min`; `kokiri-store screenshot.max_diff` failed.

Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5070, 620.70.6` `AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM`; 2: linux+amd+radv+igpu `AMD Radeon 760M Graphics (RADV PHOENIX), 26.0.8` `AMD Ryzen 5 230 w/ Radeon 760M Graphics, 16 GB RAM`  
Scene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them  
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports  
How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again)

| | Before: main @ 621dae5 | After: this PR @ e2ac1c1 |
|---|---|---|
| kokiri-store · play · frame time (p99) (ms) | 60.5 | 35.3 (−42%) |
| kokiri-store · forbidden log lines | 0.0 | 0.0 |
| kokiri-store · expected log lines missing | 1.0 | 0.0 (−100%) |
| kokiri-store · value_program_cache_kb | 7,941,497 | 0.0 (−100%) |

### Flags and settings

None: the plan varied no setting, switch or build option.

Measurements · windows+nvidia

Verdict: fail on windows+nvidia (2/1 report): kokiri-store value_program_cache_kb.min failed.

Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0 AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM; 2: windows+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 3070, 621.24.0 AMD Ryzen 5 5600X3D 6-Core Processor, 32 GB RAM
Scene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (@Beanstalk on this PR runs it again)

Before: main @ 621dae5 After: this PR @ e2ac1c1
kokiri-store · play · frame time (p99) (ms) 26.3 19.6 (−26%)
kokiri-store · forbidden log lines 0.0 0.0
kokiri-store · expected log lines missing 1.0 0.0 (−100%)
kokiri-store · value_program_cache_kb 8,113,415 0.0 (−100%)

Flags and settings

None: the plan varied no setting, switch or build option.

Markdown for the PR description
## Measurements

**Verdict:** fail on windows+nvidia (2/1 report): `kokiri-store value_program_cache_kb.min` failed.

Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0` `AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM`; 2: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3070, 621.24.0` `AMD Ryzen 5 5600X3D 6-Core Processor, 32 GB RAM`  
Scene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them  
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports  
How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again)

| | Before: main @ 621dae5 | After: this PR @ e2ac1c1 |
|---|---|---|
| kokiri-store · play · frame time (p99) (ms) | 26.3 | 19.6 (−26%) |
| kokiri-store · forbidden log lines | 0.0 | 0.0 |
| kokiri-store · expected log lines missing | 1.0 | 0.0 (−100%) |
| kokiri-store · value_program_cache_kb | 8,113,415 | 0.0 (−100%) |

### Flags and settings

None: the plan varied no setting, switch or build option.
<!-- beanstalk:previous --> <details><summary>Beanstalk test farm · an earlier run: commit <code>e2ac1c1</code> (the newer results are in a comment below)</summary> Commit `e2ac1c1` · tier **reviewed** · approved by madebyjordan · [dashboard](https://beanstalk.magicbean.dev/prs/1809) **What will be tested** · AI plan (`gpt-6.1-terra`) On Linux with any GPU and on Windows with NVIDIA, the plan compares main and the PR after shader preparation, then runs about a minute from the Kokiri start save. It records the shader-store blob count and the program-cache size reported at startup, and compares a gameplay image and frame times. A missing store log, an unexpectedly sized PR program cache, a missing-SPIR-V message, a crash, or a significant visual or frame-time regression fails the PR. The plan cannot directly inspect file sizes, set a custom XDG path, or run concurrent writers, so those aspects of the request are not fully tested. - **Where:** linux (1-2 reports), windows+nvidia (1-2 reports). - **kokiri-store:** from the repository's save `kokiri-start` (switches `RX_VK_PRECOMPILE=0`), then move the stick, take the picture kokiri to compare with main, measure play for 60 s (after 0 s warm-up); 1 run on each side. - **Measured:** crashes, frame times, stutters and hitches, the game's log, numbers from the game's log. - **The PR fails if:** it crashes; or a picture differs from main by more than 5%; or frame time (p99) is more than 20% above main; or forbidden log lines is above 0; or expected log lines missing is above 0; or value_program_cache_kb is above 300000; or value_program_cache_kb is below 10000. - **Time:** about 12 min of game runs per tester (2 game starts), plus the two builds. Done (started 22:51 UTC, last report 23:28 UTC). Comment `@Beanstalk` to run it again. | Target | Reports | Crash | Frame p99 (ms) vs main | VRAM peak (MB) vs main | Result | | --- | --- | --- | --- | --- | --- | | linux · 1: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5070, 620.70.6`; 2: linux+amd+radv+igpu `AMD Radeon 760M Graphics (RADV PHOENIX), 26.0.8` | 2/1 [1](https://beanstalk.magicbean.dev/reports/rp_d3e11024f23b0547) [2](https://beanstalk.magicbean.dev/reports/rp_36f898530734c353) | none | 35.3 vs 60.5 (−42%) | — | fail ⚠ outlier | | windows+nvidia · 1: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0`; 2: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3070, 621.24.0` | 2/1 [1](https://beanstalk.magicbean.dev/reports/rp_f8d8dd2a1d0e957a) [2](https://beanstalk.magicbean.dev/reports/rp_fa2e3df980ec6f1b) | none | 19.6 vs 26.3 (−26%) | — | fail | #### Measurements · linux **Verdict:** fail on linux (2/1 report): `kokiri-store value_program_cache_kb.min`; `kokiri-store screenshot.max_diff` failed. Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5070, 620.70.6` `AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM`; 2: linux+amd+radv+igpu `AMD Radeon 760M Graphics (RADV PHOENIX), 26.0.8` `AMD Ryzen 5 230 w/ Radeon 760M Graphics, 16 GB RAM` Scene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) | | Before: main @ 621dae5 | After: this PR @ e2ac1c1 | |---|---|---| | kokiri-store · play · frame time (p99) (ms) | 60.5 | 35.3 (−42%) | | kokiri-store · forbidden log lines | 0.0 | 0.0 | | kokiri-store · expected log lines missing | 1.0 | 0.0 (−100%) | | kokiri-store · value_program_cache_kb | 7,941,497 | 0.0 (−100%) | ### Flags and settings None: the plan varied no setting, switch or build option. <details><summary>Markdown for the PR description</summary> ```markdown ## Measurements **Verdict:** fail on linux (2/1 report): `kokiri-store value_program_cache_kb.min`; `kokiri-store screenshot.max_diff` failed. Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5070, 620.70.6` `AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM`; 2: linux+amd+radv+igpu `AMD Radeon 760M Graphics (RADV PHOENIX), 26.0.8` `AMD Ryzen 5 230 w/ Radeon 760M Graphics, 16 GB RAM` Scene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) | | Before: main @ 621dae5 | After: this PR @ e2ac1c1 | |---|---|---| | kokiri-store · play · frame time (p99) (ms) | 60.5 | 35.3 (−42%) | | kokiri-store · forbidden log lines | 0.0 | 0.0 | | kokiri-store · expected log lines missing | 1.0 | 0.0 (−100%) | | kokiri-store · value_program_cache_kb | 7,941,497 | 0.0 (−100%) | ### Flags and settings None: the plan varied no setting, switch or build option. ``` </details> #### Measurements · windows+nvidia **Verdict:** fail on windows+nvidia (2/1 report): `kokiri-store value_program_cache_kb.min` failed. Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0` `AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM`; 2: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3070, 621.24.0` `AMD Ryzen 5 5600X3D 6-Core Processor, 32 GB RAM` Scene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) | | Before: main @ 621dae5 | After: this PR @ e2ac1c1 | |---|---|---| | kokiri-store · play · frame time (p99) (ms) | 26.3 | 19.6 (−26%) | | kokiri-store · forbidden log lines | 0.0 | 0.0 | | kokiri-store · expected log lines missing | 1.0 | 0.0 (−100%) | | kokiri-store · value_program_cache_kb | 8,113,415 | 0.0 (−100%) | ### Flags and settings None: the plan varied no setting, switch or build option. <details><summary>Markdown for the PR description</summary> ```markdown ## Measurements **Verdict:** fail on windows+nvidia (2/1 report): `kokiri-store value_program_cache_kb.min` failed. Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0` `AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM`; 2: windows+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 3070, 621.24.0` `AMD Ryzen 5 5600X3D 6-Core Processor, 32 GB RAM` Scene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) | | Before: main @ 621dae5 | After: this PR @ e2ac1c1 | |---|---|---| | kokiri-store · play · frame time (p99) (ms) | 26.3 | 19.6 (−26%) | | kokiri-store · forbidden log lines | 0.0 | 0.0 | | kokiri-store · expected log lines missing | 1.0 | 0.0 (−100%) | | kokiri-store · value_program_cache_kb | 8,113,415 | 0.0 (−100%) | ### Flags and settings None: the plan varied no setting, switch or build option. ``` </details> </details>
fix(dev_packs): a runtime prepares again when the program cache's format changes
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
beanstalk/approval not approved yet
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 14s
CI / Conventional Commit title (pull_request) Successful in 13s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 24s
CI / Workflow lint (actionlint) (pull_request) Successful in 26s
Third-party parts / As their notes say (pull_request) Successful in 28s
1788a31eef
A pack prepared again only when the translator's stamp moved. A new
kProgramCacheFormat keeps that stamp but drops every cache the base
prepared (program_cache_stamp holds the format), so the runtime played
with no prepared shaders: the Test Farm's PR side logged an empty
program cache and store at its start.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Beanstalk test farm

Commit 720c698 · approved by madebyjordan (untrusted) · dashboard

This diff touches sensitive paths, so it needs two members of BigBeans: nx2/make_game.py, scripts/devtools/packs.py, scripts/devtools/vcs.py

A member of BigBeans can review this commit and comment @Beanstalk to test it.

<!-- beanstalk:sticky --> ### Beanstalk test farm Commit `720c698` · approved by madebyjordan (untrusted) · [dashboard](https://beanstalk.magicbean.dev/prs/1809) This diff touches sensitive paths, so it needs two members of `BigBeans`: `nx2/make_game.py`, `scripts/devtools/packs.py`, `scripts/devtools/vcs.py` A member of `BigBeans` can review this commit and comment `@Beanstalk` to test it.
fix(cache): the shader store checks each blob's hash and keeps one bad write from spoiling others
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
beanstalk/approval not approved yet
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
PR description / Description follows the template (pull_request_target) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 13s
CI / Workflow lint (actionlint) (pull_request) Successful in 14s
Third-party parts / As their notes say (pull_request) Successful in 16s
a1c39f919e
From a cold review of the store:

- a blob is read only when its words hash to its head's; a place whose
  words did not (a blob cut short, zeros a crash left, another blob under
  an old entry) is not given out again, and a put checks the head of the
  blob it would give out: the words are written again rather than wrong
  SPIR-V reaching the driver or the same program translated every start;
- appends land past every place an index entry names, so an old entry of
  a pack cut short never names a newer blob;
- an index entry cut short by a full disk is padded to a whole entry of
  zeros before the next, so the entries after it keep their grid;
- a store file that cannot be read now (held by another program) is no
  longer taken for a missing one: the run goes without the store instead
  of making it again, and a game folder's cache naming that store is
  kept for the next run rather than started over;
- the lock is a lock on spirv.lock on Windows too (one lock for any
  spelling of the folder, let go when a Longshot ends), waited for a
  minute at most on both systems; open() no longer goes on without it;
- a stage is at most 64 MB of SPIR-V, places at most 256 TB, so a damaged
  entry neither allocates 1 GB nor overflows;
- the global store is never destroyed, as the other globals there.

Tests: each of these, run against mutations of the store (the hash
check, the bad places, the head check, the padding of the pack and of
the index, the remade store); the whole read's code marked as the
store's; disk_init's three choices by the file's store.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs(cache): the shader store's second run, with the hash check
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
beanstalk/approval needs a second approval (sensitive paths)
Website / Website on Cloudflare (pull_request_target) Has been skipped
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 12s
eaada6083c
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

Why the first farm run failed: the PR's runtime was never prepared. scripts/dev_packs.py prepared a runtime again only when the translator's stamp changed, and this PR changes the program cache's format (kProgramCacheFormat) without it. So the runtime started from main's caches, the game dropped them as another format, and play began from an empty cache: value_program_cache_kb 0 and store_blobs 0 at the start. 1788a31e makes a pack prepare again when the format changes. A player's game was never affected: its prepare record holds the format, so the first start after an update prepares.

Since then, a1c39f91 fixes what a cold review of the store found. A blob is read only when its words hash to its head's, and appends land past every place an entry names. An index entry cut short stays on its grid, a store file held at the start no longer remakes the store, and the lock is a file lock on Windows too. That build ran again here in the game: run02 in docs/tested/.

@Beanstalk linux (any GPU) and windows nvidia: the same plan as before, now on eaada608. Build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Report the log's vk: shader store ...: N SPIR-V blobs line (N in the tens of thousands expected), the vk: program cache ...: N program keys kept, K KB line (about 150,000 KB expected, against about 8,000,000 on main), and any has no SPIR-V line (none expected).

Why the first farm run failed: the PR's runtime was never prepared. `scripts/dev_packs.py` prepared a runtime again only when the translator's stamp changed, and this PR changes the program cache's format (`kProgramCacheFormat`) without it. So the runtime started from main's caches, the game dropped them as another format, and play began from an empty cache: `value_program_cache_kb` 0 and `store_blobs` 0 at the start. 1788a31e makes a pack prepare again when the format changes. A player's game was never affected: its prepare record holds the format, so the first start after an update prepares. Since then, a1c39f91 fixes what a cold review of the store found. A blob is read only when its words hash to its head's, and appends land past every place an entry names. An index entry cut short stays on its grid, a store file held at the start no longer remakes the store, and the lock is a file lock on Windows too. That build ran again here in the game: run02 in `docs/tested/`. @Beanstalk linux (any GPU) and windows nvidia: the same plan as before, now on eaada608. Build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Report the log's `vk: shader store ...: N SPIR-V blobs` line (N in the tens of thousands expected), the `vk: program cache ...: N program keys kept, K KB` line (about 150,000 KB expected, against about 8,000,000 on main), and any `has no SPIR-V` line (none expected).
Member

@madebyjordan approval of eaada60 recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@madebyjordan approval of `eaada60` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
madebyjordan force-pushed perf/shared-spirv-pack from eaada6083c
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
beanstalk/approval needs a second approval (sensitive paths)
Website / Website on Cloudflare (pull_request_target) Has been skipped
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 12s
to f78a53248b
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 13s
beanstalk/approval not approved yet
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Website build and checks (pull_request) Has been cancelled
2026-11-20 06:15:15 +00:00
Compare
madebyjordan force-pushed perf/shared-spirv-pack from f78a53248b
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 13s
beanstalk/approval not approved yet
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Website build and checks (pull_request) Has been cancelled
to 993b9a04b8
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Successful in 7s
beanstalk/approval needs a second approval (sensitive paths)
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Website build and checks (pull_request) Has been cancelled
2026-11-20 06:58:06 +00:00
Compare
Author
Member

@dyst could you look at this one for the Test Farm's second BigBeans approval? It needs two because it changes scripts/devtools/packs.py and scripts/devtools/vcs.py, which the farm itself runs:

  • packs.py: cache_format_changed(). A pack now prepares its runtime again when kProgramCacheFormat differs from the base's, not only when the translator's stamp does. This PR raises the format, so without this the farm's PR side started from main's caches, dropped them as another format, and played unprepared: that is why the first farm run read 0 for the cache and the store.
  • vcs.py: show(tree, commit, path), a file's text at a commit through jj or git, for that check.

Both are about 30 lines in all, with test_dev_packs' check of the format. The rest of the PR is the shader store itself; it composes with your #737 (issuecomment-55159). Since the last run: rebased onto main (now 993b9a04, on main 1780fbe0); the store's cold-review fixes (hash-checked reads, padded appends); and #1844's module drop checked with the store, in its test and in the game. Testing has the runs.

@Beanstalk linux (any GPU) and windows nvidia, on 993b9a04: build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Report the log's vk: shader store ...: N SPIR-V blobs line (N in the tens of thousands expected), the vk: program cache ...: N program keys kept, K KB line (K about 150,000 expected, against about 8,000,000 on main), and any has no SPIR-V line (none expected).

@dyst could you look at this one for the Test Farm's second BigBeans approval? It needs two because it changes `scripts/devtools/packs.py` and `scripts/devtools/vcs.py`, which the farm itself runs: - `packs.py`: `cache_format_changed()`. A pack now prepares its runtime again when `kProgramCacheFormat` differs from the base's, not only when the translator's stamp does. This PR raises the format, so without this the farm's PR side started from main's caches, dropped them as another format, and played unprepared: that is why the first farm run read 0 for the cache and the store. - `vcs.py`: `show(tree, commit, path)`, a file's text at a commit through jj or git, for that check. Both are about 30 lines in all, with `test_dev_packs`' check of the format. The rest of the PR is the shader store itself; it composes with your #737 (issuecomment-55159). Since the last run: rebased onto main (now 993b9a04, on main 1780fbe0); the store's cold-review fixes (hash-checked reads, padded appends); and #1844's module drop checked with the store, in its test and in the game. Testing has the runs. @Beanstalk linux (any GPU) and windows nvidia, on 993b9a04: build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Report the log's `vk: shader store ...: N SPIR-V blobs` line (N in the tens of thousands expected), the `vk: program cache ...: N program keys kept, K KB` line (K about 150,000 expected, against about 8,000,000 on main), and any `has no SPIR-V` line (none expected).
Member

@madebyjordan approval of 993b9a0 recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@madebyjordan approval of `993b9a0` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
madebyjordan force-pushed perf/shared-spirv-pack from 993b9a04b8
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Successful in 7s
beanstalk/approval needs a second approval (sensitive paths)
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Website build and checks (pull_request) Has been cancelled
to e4c129516c
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 24s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 32s
CI / Workflow lint (actionlint) (pull_request) Successful in 38s
beanstalk/approval needs a second approval (sensitive paths)
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 43s
PR description / Description follows the template (pull_request_target) Failing after 22s
CI / Website build and checks (pull_request) Has been cancelled
2026-11-20 09:24:17 +00:00
Compare
madebyjordan changed title from WIP: perf(cache): one shader store for every game folder: each program's SPIR-V kept once, by its content to perf(cache): one shader store for every game folder: each program's SPIR-V kept once, by its content 2026-11-20 09:24:45 +00:00
Author
Member

Rebased onto main 89c3d21e (no conflicts; main's new commits touch none of this PR's files but switches.def) and marked ready for review: the head is now e4c12951. @dyst, the farm approval request above is for this head; packs.py and vcs.py are the same as at 993b9a0.

The Linux build is left to the Test Farm and the team's Linux testers (Still to test): nothing here builds on Linux.

@Beanstalk linux (any GPU) and windows nvidia, on e4c12951: the same plan as before. Build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save, and report the vk: shader store ...: N SPIR-V blobs line, the vk: program cache ...: N program keys kept, K KB line (K about 150,000 expected), and any has no SPIR-V line (none expected).

Rebased onto main 89c3d21e (no conflicts; main's new commits touch none of this PR's files but `switches.def`) and marked ready for review: the head is now e4c12951. @dyst, the farm approval request above is for this head; `packs.py` and `vcs.py` are the same as at 993b9a0. The Linux build is left to the Test Farm and the team's Linux testers (Still to test): nothing here builds on Linux. @Beanstalk linux (any GPU) and windows nvidia, on e4c12951: the same plan as before. Build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save, and report the `vk: shader store ...: N SPIR-V blobs` line, the `vk: program cache ...: N program keys kept, K KB` line (K about 150,000 expected), and any `has no SPIR-V` line (none expected).
Member

@madebyjordan approval of e4c1295 recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@madebyjordan approval of `e4c1295` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
futondestroyer left a comment

Request changes. Reviewed e4c12951 against main `89c3d21e). The Windows AMD evidence demonstrates substantial disk savings for the sampled builds. A v2 program file of 149,112,385 bytes plus the 3,736,516,412-byte shared pack replaces an 8,027,711,307-byte per-build v1 file. Please report index and total footprint separately from the program file and pack. The measured zero additional pack bytes in two follow-on builds describes those samples. The format upgrade still incurs preparation.

The override must reach preparation and play consistently, and established readers must retain their pack generation during repair. Please also bound malformed-index recovery and fail validation when referenced stages cannot be read, as detailed inline.

The change can land independently of #737 once fixed and tested. Before enabling both, make the pack identity stable for a shared record directory. With independent root overrides, two games can otherwise alternate the store ID of the same stamped program cache and rebuild it repeatedly. Coordinate the cleanup contract with Launcher #210.

Please add game-free checks for the override and opt-out in builder and AppImage, concurrent reset with a live reader, out-of-range index, and validator failure when the store is missing or mismatched. Then verify cold v1-to-v2 preparation and warm startup on Linux and Windows NVIDIA with an explicit root. Check stage and program counts, preparation/load time and total bytes including the index. This is a source review using exact-revision Graphify and direct source checks. I ran no PR code or gameplay.

**Request changes.** Reviewed `e4c12951` against main `89c3d21e). The Windows AMD evidence demonstrates substantial disk savings for the sampled builds. A v2 program file of 149,112,385 bytes plus the 3,736,516,412-byte shared pack replaces an 8,027,711,307-byte per-build v1 file. Please report index and total footprint separately from the program file and pack. The measured zero additional pack bytes in two follow-on builds describes those samples. The format upgrade still incurs preparation. The override must reach preparation and play consistently, and established readers must retain their pack generation during repair. Please also bound malformed-index recovery and fail validation when referenced stages cannot be read, as detailed inline. The change can land independently of #737 once fixed and tested. Before enabling both, make the pack identity stable for a shared record directory. With independent root overrides, two games can otherwise alternate the store ID of the same stamped program cache and rebuild it repeatedly. Coordinate the cleanup contract with Launcher #210. Please add game-free checks for the override and opt-out in builder and AppImage, concurrent reset with a live reader, out-of-range index, and validator failure when the store is missing or mismatched. Then verify cold v1-to-v2 preparation and warm startup on Linux and Windows NVIDIA with an explicit root. Check stage and program counts, preparation/load time and total bytes including the index. This is a source review using exact-revision Graphify and direct source checks. I ran no PR code or gameplay.
@ -0,0 +321,4 @@
const Head ph = read_head(pack_path, kPackMagic, pid), ih = read_head(idx_path, kIdxMagic, iid);
// (a file there that could not be read now, held by another program: none this run, the store kept as it is)
if (ph == Head::unreadable || ih == Head::unreadable) return false;
if (ph != Head::ok || ih != Head::ok || pid != iid) { // (none yet, one of the two damaged or of another store)
Member

[P2] Do not truncate a generation that another process still reads. The lock here covers open and append, but an existing game retains offsets beyond that lock. A missing, mismatched or different-version index can trigger in-place truncation while it plays. If an offset is later reused for a same-length blob, read() checks that blob against its own hash and can return the wrong shader. Refuse destructive repair while readers exist or pin immutable generation files for their lifetime.

[P2] Do not truncate a generation that another process still reads. The lock here covers open and append, but an existing game retains offsets beyond that lock. A missing, mismatched or different-version index can trigger in-place truncation while it plays. If an offset is later reused for a same-length blob, read() checks that blob against its own hash and can return the wrong shader. Refuse destructive repair while readers exist or pin immutable generation files for their lifetime.
@ -0,0 +361,4 @@
const uint64_t* x = &e[i * 4];
if (!x[3] || x[3] >= kMostWords || x[2] < kHead + kBlobHead || x[2] >= (uint64_t)kMostAt) continue;
const int64_t at = (int64_t)x[2], end = at + (int64_t)x[3] * 4; // (both bounded: no overflow)
named_end = std::max(named_end, end);
Member

[P2] Bound recovery by the data that actually exists. An index entry beyond EOF still raises named_end. The next put fills that gap with zeros under the writer lock, even when the nominal offset is far beyond the pack. Reject or quarantine unverified extents with a bounded recovery policy, and process index records in bounded batches.

[P2] Bound recovery by the data that actually exists. An index entry beyond EOF still raises named_end. The next put fills that gap with zeros under the writer lock, even when the nominal offset is far beyond the pack. Reject or quarantine unverified extents with a bounded recovery policy, and process index records in bounded batches.
@ -241,6 +241,7 @@ SW(LONGSHOT_DRIVER_ENV_SET, diag, string, "not set", "Set by the game itself bef
SW(LONGSHOT_NFC_DIR, player, string, "Windows: %LOCALAPPDATA%\\Longshot\\nfc; Linux: $XDG_DATA_HOME/longshot/nfc (~/.local/share/longshot/nfc)", "The folder of NFC tag files the Insert menu's NFC tab lists, instead of the user's own.")
SW(LONGSHOT_NO_AV1_NOTICE, kill, bool, "the notice window is shown once (Windows, no AV1 decoder)", "No pop-up asking to install the AV1 Video Extension; the log line is still written.")
SW(LONGSHOT_PREPARE_ONLY, player, bool, "off: a normal game run", "Marks a --prepare-shaders run: the renderer only compiles and saves shaders, skipping work a played run does.")
SW(LONGSHOT_SHADER_STORE, player, string, "Windows: %LOCALAPPDATA%\\Longshot\\shaders; Linux: $XDG_CACHE_HOME/longshot/shaders (~/.cache/longshot/shaders)", "The shader store's folder, the SPIR-V every game folder's program cache shares (gpu/vk/spirv_pack.h), instead of the user's own; 0: none, each record keeps its SPIR-V.")
Member

[P2] Carry this override through all launch paths. nx2/make_game.py:1600–1612 and :1691 strip LONGSHOT_* for preparation. scripts/devtools/packs.py uses that filter and builder/linux/longshot-launch:25 strips it for AppImage play. Preparing against the default and playing with a custom path or 0 gives another store ID and can discard the just-prepared program records. Preserve both forms through preparation and launch.

[P2] Carry this override through all launch paths. nx2/make_game.py:1600–1612 and :1691 strip LONGSHOT_* for preparation. scripts/devtools/packs.py uses that filter and builder/linux/longshot-launch:25 strips it for AppImage play. Preparing against the default and playing with a custom path or 0 gives another store ID and can discard the just-prepared program records. Preserve both forms through preparation and launch.
@ -201,3 +261,3 @@
programs += 1
for nvn, vk, spirv in stages(data):
for nvn, vk, spirv in stages(data, store):
job = (key, nvn, vk, spirv)
Member

[P2] Fail validation when a referenced stage cannot be inspected. A missing or unreadable pack makes stages() omit references silently, and the tool can report zero failed stages and exit 0 despite checking none of them. Check the pack generation against the program-cache header and count unreadable references as failures. Report stages validated separately from program records encountered.

[P2] Fail validation when a referenced stage cannot be inspected. A missing or unreadable pack makes stages() omit references silently, and the tool can report zero failed stages and exit 0 despite checking none of them. Check the pack generation against the program-cache header and count unreadable references as failures. Report stages validated separately from program records encountered.
futondestroyer's review of #1809: a record named only a blob's place and length, and a read checked the
words against the head found there. A store made again (truncated in place) under a running game, or a
pack cut short by hand, could put another blob of the same length at that place, which then passed.

- A record in the store keeps the words' hash after their place (kProgramCacheFormat 3), Stage keeps it
  for the reads after the load (a light read, a dropped module remade), and Store::read takes it: whatever
  is at a place now is read only when it is those words.
- With that, appends no longer pad the pack out to the furthest place an index entry names: a damaged
  entry far past the pack's end wrote zeros up to it (64 TB in the check). Such an entry is left out, and
  the index is read a few thousand entries at a time.
- A store made again is new files renamed over the old ones, never the old ones cut short: a Longshot
  reading them keeps them (Linux), and while one has them open they are not replaced (Windows: the other
  run goes without the store).

Checks (tests/spirv_pack_test.cpp): another blob at a place asked for by another hash, the pack cut under
an entry and the place taken by a blob of the same length, an entry 64 TB past the end, and a store made
again while another Store reads it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
make_game.game_env and builder/linux/longshot-launch drop the LONGSHOT_ build switches before starting
the game, LONGSHOT_SHADER_STORE among them: the shaders were prepared in the default store and played
with the folder (or 0) the player set, whose other store id started the cache over. It is kept now
(GAME_ENV_KEEP), in both, with checks of each.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(validate_spirv): a stage in the shader store that can't be read fails the run
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 13s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 15s
PR description / Description follows the template (pull_request_target) Failing after 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval needs a second approval (sensitive paths)
CI / Website build and checks (pull_request) Failing after 3m7s
720c698822
A missing or other store left its stages out silently, and the run passed having checked none of them.
The cache's head names its store, which must be the one found; each stage reference the store does not
give (no store, another store, other words at its place: the record's hash is checked) counts as not
read and fails the run. The report says how many stages it checked, apart from the programs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

@futondestroyer thank you. All four inline points were real, and they're fixed in three commits on top. Head 720c6988, still on main 89c3d21e.

[P2] switches.def:244, the override through every launch path (issuecomment-58382). 50efd93e: make_game.GAME_ENV_KEEP keeps LONGSHOT_SHADER_STORE in game_env(), which covers the preparation at the build and packs.py. builder/linux/longshot-launch keeps the same name for the AppImage's Play. Checks: test_prepare_at_build (a folder and 0 both survive game_env, and the shell script's kept name matches GAME_ENV_KEEP) and test_longshot_launch (the AppImage's Play passes both on; it runs on Linux CI).

[P2] spirv_pack.h:324, truncating a generation another process reads (issuecomment-58383). e2eb3e8f, two parts:

  • A record now keeps the words' 128-bit hash after their place (kProgramCacheFormat 3), and Stage keeps it for the reads after the load (a light read, a dropped module remade). Store::read(at, n, want) gives the words only when the blob at that place has that hash and the words hash to it. Whatever lies at a reused place is never handed to the driver in another blob's stead.
  • A store made again is new files written beside the old ones and renamed into place, never the old ones cut short. On Linux a running reader keeps its old files. On Windows the rename is refused while another Longshot has them open, and that run goes without the store.

[P2] spirv_pack.h:364, bounded recovery (issuecomment-58384). With the hash in the record, the padding out to the furthest named place has no job left, so it's gone. An entry naming bytes the pack doesn't have is left out, and nothing is written for it. The index is read 4,096 entries at a time.

The new checks in tests/spirv_pack_test.cpp cover:

  • another blob at a place asked for under another hash;
  • a pack cut under an entry, with that place taken by a blob of the same length;
  • an index entry 64 TB past the end (the next blob goes at the pack's end, and the file grows by that blob alone);
  • a store made again while another Store reads it.

Undoing either fix (a read ignoring the hash, or the store remade in place) fails them.

[P2] validate_spirv.py:204, fail when stages can't be read (issuecomment-58385). 720c6988: the cache's head names its store, and the store found must be that one. Each stage reference the store doesn't give (no store, another store, or another hash at the place) counts as not read and fails the run. The report gives stages checked apart from programs. test_validate_spirv has a cache that passes with its store and fails with none or another.

From the summary:

  • Footprint, index included: after the fixes, a prepared folder's longshot_programs.bin is 150,840,657 bytes (+1.2% for the hashes) and its longshot_programs.idx 1,133,936 bytes. The store is 3,736,516,412 bytes (spirv.pack, unchanged to the byte) plus 1,621,936 bytes (spirv.idx). Measurements has the row.
  • The format upgrade still prepares: yes, every player prepares once after updating. That's in Risks. The "+0 bytes" rows describe those two follow-on folders, as you say.
  • #737: agreed. It's in Risks: before both are on, the store's identity has to be one per shared record folder, or alternating roots start the shared file over.
  • Launcher #210: the cleanup contract is in Risks: the three files deleted together. #210 will be checked against it before both merge.

Run in the game on 720c6988 (Windows 11, RX 6700 XT), from an empty store:

  • the runtime prepared the same 38,809 programs (46,598 new, 69 failed) in 118.7 s;
  • the play from kokiri-start kept 47,798 program keys (147,305 KB) and read 50,685 blobs, through Hyrule Field, the Lost Woods, the Death Mountain Trail and Zora's Fountain;
  • 0 draws left out, 0 read failures, and the game quit cleanly.

Not done here: the cold v1-to-v2 and warm runs on Linux and Windows NVIDIA with an explicit root. Those are in Still to test.

@Beanstalk linux (any GPU) and windows nvidia, on 720c6988: build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Then do one more start with LONGSHOT_SHADER_STORE set to a new empty folder, both for the build's preparation and for the play. For each, report:

  • the vk: shader store ...: N SPIR-V blobs line (N about 50,000 expected after preparing);
  • the vk: program cache ...: N program keys kept, K KB line (K about 150,000 expected);
  • the preparation's time;
  • the size of spirv.pack and spirv.idx in the store folder;
  • any has no SPIR-V, did not read or another shader store line (none expected).
@futondestroyer thank you. All four inline points were real, and they're fixed in three commits on top. Head **720c6988**, still on main 89c3d21e. **[P2] switches.def:244, the override through every launch path** (issuecomment-58382). `50efd93e`: `make_game.GAME_ENV_KEEP` keeps `LONGSHOT_SHADER_STORE` in `game_env()`, which covers the preparation at the build and `packs.py`. `builder/linux/longshot-launch` keeps the same name for the AppImage's Play. Checks: `test_prepare_at_build` (a folder and `0` both survive `game_env`, and the shell script's kept name matches `GAME_ENV_KEEP`) and `test_longshot_launch` (the AppImage's Play passes both on; it runs on Linux CI). **[P2] spirv_pack.h:324, truncating a generation another process reads** (issuecomment-58383). `e2eb3e8f`, two parts: - A record now keeps the words' 128-bit hash after their place (`kProgramCacheFormat` 3), and `Stage` keeps it for the reads after the load (a light read, a dropped module remade). `Store::read(at, n, want)` gives the words only when the blob at that place has that hash and the words hash to it. Whatever lies at a reused place is never handed to the driver in another blob's stead. - A store made again is new files written beside the old ones and renamed into place, never the old ones cut short. On Linux a running reader keeps its old files. On Windows the rename is refused while another Longshot has them open, and that run goes without the store. **[P2] spirv_pack.h:364, bounded recovery** (issuecomment-58384). With the hash in the record, the padding out to the furthest named place has no job left, so it's gone. An entry naming bytes the pack doesn't have is left out, and nothing is written for it. The index is read 4,096 entries at a time. The new checks in `tests/spirv_pack_test.cpp` cover: - another blob at a place asked for under another hash; - a pack cut under an entry, with that place taken by a blob of the same length; - an index entry 64 TB past the end (the next blob goes at the pack's end, and the file grows by that blob alone); - a store made again while another `Store` reads it. Undoing either fix (a read ignoring the hash, or the store remade in place) fails them. **[P2] validate_spirv.py:204, fail when stages can't be read** (issuecomment-58385). `720c6988`: the cache's head names its store, and the store found must be that one. Each stage reference the store doesn't give (no store, another store, or another hash at the place) counts as not read and fails the run. The report gives stages checked apart from programs. `test_validate_spirv` has a cache that passes with its store and fails with none or another. **From the summary:** - **Footprint, index included:** after the fixes, a prepared folder's `longshot_programs.bin` is 150,840,657 bytes (+1.2% for the hashes) and its `longshot_programs.idx` 1,133,936 bytes. The store is 3,736,516,412 bytes (`spirv.pack`, unchanged to the byte) plus 1,621,936 bytes (`spirv.idx`). Measurements has the row. - **The format upgrade still prepares:** yes, every player prepares once after updating. That's in Risks. The "+0 bytes" rows describe those two follow-on folders, as you say. - **#737:** agreed. It's in Risks: before both are on, the store's identity has to be one per shared record folder, or alternating roots start the shared file over. - **Launcher #210:** the cleanup contract is in Risks: the three files deleted together. #210 will be checked against it before both merge. **Run in the game** on 720c6988 (Windows 11, RX 6700 XT), from an **empty** store: - the runtime prepared the same 38,809 programs (46,598 new, 69 failed) in 118.7 s; - the play from kokiri-start kept 47,798 program keys (147,305 KB) and read 50,685 blobs, through Hyrule Field, the Lost Woods, the Death Mountain Trail and Zora's Fountain; - 0 draws left out, 0 read failures, and the game quit cleanly. Not done here: the cold v1-to-v2 and warm runs on Linux and Windows NVIDIA with an explicit root. Those are in Still to test. @Beanstalk linux (any GPU) and windows nvidia, on 720c6988: build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Then do one more start with `LONGSHOT_SHADER_STORE` set to a new empty folder, both for the build's preparation and for the play. For each, report: - the `vk: shader store ...: N SPIR-V blobs` line (N about 50,000 expected after preparing); - the `vk: program cache ...: N program keys kept, K KB` line (K about 150,000 expected); - the preparation's time; - the size of `spirv.pack` and `spirv.idx` in the store folder; - any `has no SPIR-V`, `did not read` or `another shader store` line (none expected).
Member

@madebyjordan approval of 720c698 recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@madebyjordan approval of `720c698` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
HumanBean deleted branch perf/shared-spirv-pack 2026-11-20 16:59:51 +00:00
Author
Member

Correction to this PR's Risks. I wrote that "every player prepares once after updating" and measured it as a dev runtime's translation-only preparation (118.7 s). That understated the real cost. program_cache_stamp() mixes kProgramCacheFormat into the stamp that every cache file is judged by. So nightly-622's first start also dropped the driver's pipeline cache (longshot_vk_pipelines.bin), truncated the pipeline notes, and started the module identifiers over. The launcher's --compile-shaders most then compiles every pipeline cold: over 40 minutes and counting on an RX 6700 XT, and players on Discord report it too. None of those files depends on the program file's layout. The fix keeps them across a format change for the same translator: branch fix/pipelines-outlive-cache-format on my fork, with a game-free test that fails on main. It opens as a PR once it has run in the game. It helps players who haven't started 622 yet; those who have already lost their files.

Correction to this PR's Risks. I wrote that "every player prepares once after updating" and measured it as a dev runtime's translation-only preparation (118.7 s). That understated the real cost. `program_cache_stamp()` mixes `kProgramCacheFormat` into the stamp that every cache file is judged by. So nightly-622's first start also dropped the driver's pipeline cache (`longshot_vk_pipelines.bin`), truncated the pipeline notes, and started the module identifiers over. The launcher's `--compile-shaders most` then compiles every pipeline cold: over 40 minutes and counting on an RX 6700 XT, and players on Discord report it too. None of those files depends on the program file's layout. The fix keeps them across a format change for the same translator: branch `fix/pipelines-outlive-cache-format` on my fork, with a game-free test that fails on main. It opens as a PR once it has run in the game. It helps players who haven't started 622 yet; those who have already lost their files.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1809
No description provided.