perf(cache): one shader store for every game folder: each program's SPIR-V kept once, by its content #1809
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1809
Loading…
Reference in a new issue
No description provided.
Delete branch "madebyjordan/Longshot:perf/shared-spirv-pack"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Every Longshot build kept its own ~8 GB of translated shader code (
cache\longshot_programs.bin), although a stable and a nightly build translate 99.6% of the game's shaders to exactly the same SPIR-V: a player's v3.0.0 and nightly-486 held 47,592 byte-identical programs of 47,798, 14.7 GB stored for 3.5 GB distinct (#1649, issuecomment-55045). Each further build (a test or pull request build, a dev runtime) added another 8 GB, which players see as "every build takes a whole game's worth of space".This keeps each program's SPIR-V once, by its content, in one shader store every game folder of the player shares (
%LOCALAPPDATA%\Longshot\shaders, Linux$XDG_CACHE_HOME/longshot/shadersor~/.cache/longshot/shaders); each build'slongshot_programs.binkeeps its records with a place in the store instead of the words. A build's program cache goes from 8.0 GB to 0.15 GB, and further builds, on the same translator or another, add nothing to the 3.7 GB store.Read first:
gpu/vk/spirv_pack.h(new, header only): the store.spirv.pack(blobs: a 24-byte head with the 128-bit hash and the word count, then the words) andspirv.idx(their places), both behind a head with the store's random id, only ever appended to. An append takes a lock onspirv.lock(waited for a minute at most), reads what other Longshots added since, and writes the blob, then its entry. A blob is used only when its words hash to its head's; one that does not (cut short, zeros a crash left) is translated and written again, and its place is not given out again. Appends land past every place an entry names, and an index entry cut short by a full disk is padded to a whole one, so one bad write spoils no other blob. A store file another program holds at the start is not taken for a missing one: that run goes without the store.gpu/vk/vk_progcache.cpp:kProgramCacheFormat1 -> 2; a stage's code is[u64 words][words]as before or[u64 words | 1<<63][i64 place];get_code/read_codeserve every reader of a stage's code; the file's head names its store, so a cache of a store that is gone starts over, and one whose store did not open this run is kept for the next.scripts/devtools/packs.py: a runtime prepares again whenkProgramCacheFormatchanges, not only the translator's stamp (the Test Farm's first run played this PR unprepared because of it);scripts/devtools/vcs.pygainsshow()for that.scripts/validate_spirv.pyreads stages from the store (--store).Fixes #1759. See #1649 (its option B, across game folders; its option A, compressing each blob, fits on top in the store) and #737 (dyst's stamp-keyed store, which shares whole caches between folders of the same translation; the two compose, issuecomment-55159). The launcher side is LongshotLauncher #210.
Testing
Tested: Windows 11 Home 26200, AMD Radeon RX 6700 XT (driver 0x800161), Ryzen 5 5600, headless 1920x1080, from the repository's kokiri-start save (title, Continue, Load).
720c6988, on main89c3d21e: futondestroyer's review fixed, the records' hash, kProgramCacheFormat 3), as adev.pypack over the base at main8e9cfc3a, into an empty default store: the runtime's preparationprepared 38809 shader programs (46598 new, 69 failed, ...)in 118.7 s, as main's; itslongshot_programs.bin150,840,657 bytes (each stored stage's record 16 bytes longer for its hash) andlongshot_programs.idx1,133,936 bytes; the store filled to 3,736,516,412 bytes (spirv.pack, 50,685 blobs, the same as before to the byte) and 1,621,936 bytes (spirv.idx). Then played with the default load at create:47798 program keys kept, 147305 KB,50685 SPIR-V blobs,10914 shader modules let go once their programs' noted pipelines were built, Kokiri Forest, Hyrule Field, the Lost Woods, the Death Mountain Trail and Zora's Fountain drawn, 0 draws left out, 0has no SPIR-V/did not read/not whole/another shader storelines, quit cleanly. The fixes' game-free checks: last item.993b9a04, on main1780fbe0, with #1844's drop of loaded modules), as adev.pypack over the base at main8e9cfc3a, into the default store already filled: the runtime's preparationprepared 38809 shader programs (46598 new, 69 failed, ...), as main's, itslongshot_programs.bin149,112,385 bytes, the store unchanged at 3,736,516,412 bytes (50,685 blobs). Then played with the default load at create:47798 program keys kept, 145617 KB,50685 SPIR-V blobs,13594 shader modules let go once their programs' noted pipelines were built, Kokiri Forest, Hyrule Field, the Lost Woods, the Death Mountain Trail and Zora's Fountain drawn, 0 draws left out, 0has no SPIR-V/did not read/not wholelines, quit cleanly. The same tour on the head before the last rebase (f78a5324, maina3565ac5) withRX_VK_PRECOMPILE=0read every program from the store the same way (translation 187 ms in all).89c3d21e(heade4c12951): main's six new commits touch none of this PR's files butgpu/vk/switches.def(merged without a conflict); the renderer built again on it (dev.py vk, its exports checked) and the store's and the cache's checks pass, so the run above stands.a1c39f91, before the rebase): two starts on an empty store of their own withRX_VK_PRECOMPILE=0. The first translated 1,614 blobs (129 MB) in 14,063 ms; the second kept 1,137 programs and translated 228 ms in all, reading the rest from the store with the hash check; 0 read failures.720c6988):test_spirv_packadds another blob at a place asked for by another hash, a pack cut under an entry whose place a blob of the same length takes, an index entry 64 TB past the pack's end (nothing written out to it), and a store made again while another Store reads it (Windows: not replaced; Linux: new files, the reader keeps its own), each caught when its fix is undone (a read ignoring the hash, the store remade in place);test_prepare_at_build(make_game.game_envkeepsLONGSHOT_SHADER_STORE, a folder or 0, andlongshot-launchkeeps the same name) andtest_longshot_launch(the AppImage's Play passes it on: Linux, CI);test_validate_spirv(a cache passes with its store, fails with none or another, and a reference with another hash is not read);test_loaded_modules_drop,test_program_cache_cleanup,test_program_cache_one_variantagain.tests/run_all.pyon Windows with llvm-mingw: 696 passed, 62 failed, 124 skipped; the same 62 fail on main89c3d21eon this PC, each the same way.test_spirv_pack(the store alone: one blob per content, two writers each finding the other's, the next start, a write and a pack cut short, words changed or zeroed under an entry, an index entry cut short, a store file held at the start, a deleted store's new id;disk_init's choices by its source), run against mutations of the store's fixes (the hash check, the bad places, the head check, the padding of the pack and of the index, the remade store), each caught;test_program_cache_cleanupandtest_program_cache_one_variant(every case with and without the store, the whole read's code marked as the store's),test_validate_spirv,test_dev_packs' format check,test_switch_table,test_doc_anchors, and #1844'stest_loaded_modules_droprun again with the store (a dropped program's modules remade from their places in it).mise run test(tests/run_all.py --strict) on Windows with llvm-mingw: 682 passed, 138 failed, 48 skipped; main @a3565ac5on the same PC: 681 passed and the same 138 failed (most need tools this PC lacks, such as the Vulkan headers and spirv-val, which --strict counts as failures), so none fails here that passes on main.Still to test:
vk: shader store ...: N SPIR-V blobswith N in the tens of thousands,vk: program cache ...: N program keys kept, K KBwith K about 150,000 (about 8,000,000 on main), and nohas no SPIR-Vline (@Beanstalk)Measurements
Verdict: a build's program cache is 0.15 GB instead of 8.0 GB, and further builds share the 3.7 GB store: a second build added 0 bytes to it on the same translator stamp and 0 bytes on another. The same programs are prepared.
Setup: Windows 11, RX 6700 XT, as Testing. Before: the dev base at main
8e9cfc3a, prepared by main. After: this PR as a pack over that base (runtime A); runtimes B and C its folder withlongshot_vk.dllbuilt from this PR plus a test-only commit (B: the same translator stamp; C: another,30bc04d7a5198418againstc2295d1c2762bd43), prepared one after another into one empty store. Sizes from the files, counts from the log; sizes, not timings, so one run each. Measured on 7c88d606; this head (993b9a04, on main1780fbe0) prepared the same cache to the byte and added nothing to the store (Testing).8e9cfc3alongshot_programs.binof a prepared game folderlongshot_programs.idxlongshot_programs.idx(1.1 MB each) +spirv.idx(1.6 MB)Nothing per frame changes: a program's SPIR-V is read from another file at its load, the same size, and checked against its hash once. Since the review's fixes (
720c6988) a stored stage's record is 16 bytes longer for its hash: the program file of a prepared folder is 150,840,657 bytes (+1.2%), the store the same to the byte,spirv.idx1,621,936 bytes (16 + 32 bytes a blob). The format change still costs every player one preparation after updating (Risks).Risks
LONGSHOT_SHADER_STORE=0(each record keeps its SPIR-V, the cache starting over once), or a revert. The builders keep the switch for the game they start (the preparation at the build, the AppImage's Play).LONGSHOT_SHADER_STOREfolders would each start the shared record file over for their own store. Before both are on, the store's identity needs to be one per shared record folder (#737 picks the store, or a record names its store per record); not an issue while each game folder has its own records, as here.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Tested in game: RX 6700 XT, driver 0x800161, Windows 11: three game folders prepared into one store (two translator stamps): each folder's longshot_programs.bin 149 MB instead of 8.0 GB, the second and third added 0 bytes to the 3.7 GB store; one played headless from kokiri-start through Kokiri Forest, Hyrule Field, Kakariko, Goron City, Zora's Domain, the Deku Tree and the Forest Temple, every scene drawn, no failed read
@Beanstalk linux (any GPU): build this branch, prepare the shaders, then play about a minute from the kokiri-start save. Please report the log's
vk: shader store ...: N SPIR-V blobsline, the size of cache/longshot_programs.bin (about 0.15 GB expected, against about 8 GB on main) and anyhas no SPIR-Vline (none expected). This checks the flock lock and the XDG path, which I could not run.@Beanstalk windows nvidia: the same steps and the same three things to report.
🤖 Generated with Claude Code
Beanstalk test farm · an earlier run: commit
e2ac1c1(the newer results are in a comment below)Commit
e2ac1c1· tier reviewed · approved by madebyjordan · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)On Linux with any GPU and on Windows with NVIDIA, the plan compares main and the PR after shader preparation, then runs about a minute from the Kokiri start save. It records the shader-store blob count and the program-cache size reported at startup, and compares a gameplay image and frame times. A missing store log, an unexpectedly sized PR program cache, a missing-SPIR-V message, a crash, or a significant visual or frame-time regression fails the PR. The plan cannot directly inspect file sizes, set a custom XDG path, or run concurrent writers, so those aspects of the request are not fully tested.
kokiri-start(switchesRX_VK_PRECOMPILE=0), then move the stick, take the picture kokiri to compare with main, measure play for 60 s (after 0 s warm-up); 1 run on each side.Done (started 22:51 UTC, last report 23:28 UTC). Comment
@Beanstalkto run it again.NVIDIA GeForce RTX 5070, 620.70.6; 2: linux+amd+radv+igpuAMD Radeon 760M Graphics (RADV PHOENIX), 26.0.8NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0; 2: windows+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 3070, 621.24.0Measurements · linux
Verdict: fail on linux (2/1 report):
kokiri-store value_program_cache_kb.min;kokiri-store screenshot.max_difffailed.Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu
NVIDIA GeForce RTX 5070, 620.70.6AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM; 2: linux+amd+radv+igpuAMD Radeon 760M Graphics (RADV PHOENIX), 26.0.8AMD Ryzen 5 230 w/ Radeon 760M Graphics, 16 GB RAMScene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)621dae5e2ac1c1Flags and settings
None: the plan varied no setting, switch or build option.
Markdown for the PR description
Measurements · windows+nvidia
Verdict: fail on windows+nvidia (2/1 report):
kokiri-store value_program_cache_kb.minfailed.Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu
NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM; 2: windows+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 3070, 621.24.0AMD Ryzen 5 5600X3D 6-Core Processor, 32 GB RAMScene: kokiri-store, from the repository's save kokiri-start (RX_VK_PRECOMPILE=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); kokiri-store: 1 game start a side, play 60 s after 0 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)621dae5e2ac1c1Flags and settings
None: the plan varied no setting, switch or build option.
Markdown for the PR description
Beanstalk test farm
Commit
720c698· approved by madebyjordan (untrusted) · dashboardThis diff touches sensitive paths, so it needs two members of
BigBeans:nx2/make_game.py,scripts/devtools/packs.py,scripts/devtools/vcs.pyA member of
BigBeanscan review this commit and comment@Beanstalkto test it.Why the first farm run failed: the PR's runtime was never prepared.
scripts/dev_packs.pyprepared a runtime again only when the translator's stamp changed, and this PR changes the program cache's format (kProgramCacheFormat) without it. So the runtime started from main's caches, the game dropped them as another format, and play began from an empty cache:value_program_cache_kb0 andstore_blobs0 at the start.1788a31emakes a pack prepare again when the format changes. A player's game was never affected: its prepare record holds the format, so the first start after an update prepares.Since then,
a1c39f91fixes what a cold review of the store found. A blob is read only when its words hash to its head's, and appends land past every place an entry names. An index entry cut short stays on its grid, a store file held at the start no longer remakes the store, and the lock is a file lock on Windows too. That build ran again here in the game: run02 indocs/tested/.@Beanstalk linux (any GPU) and windows nvidia: the same plan as before, now on
eaada608. Build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Report the log'svk: shader store ...: N SPIR-V blobsline (N in the tens of thousands expected), thevk: program cache ...: N program keys kept, K KBline (about 150,000 KB expected, against about 8,000,000 on main), and anyhas no SPIR-Vline (none expected).@madebyjordan approval of
eaada60recorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.eaada6083cf78a53248bf78a53248b993b9a04b8@dyst could you look at this one for the Test Farm's second BigBeans approval? It needs two because it changes
scripts/devtools/packs.pyandscripts/devtools/vcs.py, which the farm itself runs:packs.py:cache_format_changed(). A pack now prepares its runtime again whenkProgramCacheFormatdiffers from the base's, not only when the translator's stamp does. This PR raises the format, so without this the farm's PR side started from main's caches, dropped them as another format, and played unprepared: that is why the first farm run read 0 for the cache and the store.vcs.py:show(tree, commit, path), a file's text at a commit through jj or git, for that check.Both are about 30 lines in all, with
test_dev_packs' check of the format. The rest of the PR is the shader store itself; it composes with your #737 (issuecomment-55159). Since the last run: rebased onto main (now993b9a04, on main1780fbe0); the store's cold-review fixes (hash-checked reads, padded appends); and #1844's module drop checked with the store, in its test and in the game. Testing has the runs.@Beanstalk linux (any GPU) and windows nvidia, on
993b9a04: build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Report the log'svk: shader store ...: N SPIR-V blobsline (N in the tens of thousands expected), thevk: program cache ...: N program keys kept, K KBline (K about 150,000 expected, against about 8,000,000 on main), and anyhas no SPIR-Vline (none expected).@madebyjordan approval of
993b9a0recorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.993b9a04b8e4c129516cWIP: perf(cache): one shader store for every game folder: each program's SPIR-V kept once, by its contentto perf(cache): one shader store for every game folder: each program's SPIR-V kept once, by its contentRebased onto main
89c3d21e(no conflicts; main's new commits touch none of this PR's files butswitches.def) and marked ready for review: the head is nowe4c12951. @dyst, the farm approval request above is for this head;packs.pyandvcs.pyare the same as at993b9a0.The Linux build is left to the Test Farm and the team's Linux testers (Still to test): nothing here builds on Linux.
@Beanstalk linux (any GPU) and windows nvidia, on
e4c12951: the same plan as before. Build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save, and report thevk: shader store ...: N SPIR-V blobsline, thevk: program cache ...: N program keys kept, K KBline (K about 150,000 expected), and anyhas no SPIR-Vline (none expected).@madebyjordan approval of
e4c1295recorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.Request changes. Reviewed
e4c12951against main `89c3d21e). The Windows AMD evidence demonstrates substantial disk savings for the sampled builds. A v2 program file of 149,112,385 bytes plus the 3,736,516,412-byte shared pack replaces an 8,027,711,307-byte per-build v1 file. Please report index and total footprint separately from the program file and pack. The measured zero additional pack bytes in two follow-on builds describes those samples. The format upgrade still incurs preparation.The override must reach preparation and play consistently, and established readers must retain their pack generation during repair. Please also bound malformed-index recovery and fail validation when referenced stages cannot be read, as detailed inline.
The change can land independently of #737 once fixed and tested. Before enabling both, make the pack identity stable for a shared record directory. With independent root overrides, two games can otherwise alternate the store ID of the same stamped program cache and rebuild it repeatedly. Coordinate the cleanup contract with Launcher #210.
Please add game-free checks for the override and opt-out in builder and AppImage, concurrent reset with a live reader, out-of-range index, and validator failure when the store is missing or mismatched. Then verify cold v1-to-v2 preparation and warm startup on Linux and Windows NVIDIA with an explicit root. Check stage and program counts, preparation/load time and total bytes including the index. This is a source review using exact-revision Graphify and direct source checks. I ran no PR code or gameplay.
@ -0,0 +321,4 @@const Head ph = read_head(pack_path, kPackMagic, pid), ih = read_head(idx_path, kIdxMagic, iid);// (a file there that could not be read now, held by another program: none this run, the store kept as it is)if (ph == Head::unreadable || ih == Head::unreadable) return false;if (ph != Head::ok || ih != Head::ok || pid != iid) { // (none yet, one of the two damaged or of another store)[P2] Do not truncate a generation that another process still reads. The lock here covers open and append, but an existing game retains offsets beyond that lock. A missing, mismatched or different-version index can trigger in-place truncation while it plays. If an offset is later reused for a same-length blob, read() checks that blob against its own hash and can return the wrong shader. Refuse destructive repair while readers exist or pin immutable generation files for their lifetime.
@ -0,0 +361,4 @@const uint64_t* x = &e[i * 4];if (!x[3] || x[3] >= kMostWords || x[2] < kHead + kBlobHead || x[2] >= (uint64_t)kMostAt) continue;const int64_t at = (int64_t)x[2], end = at + (int64_t)x[3] * 4; // (both bounded: no overflow)named_end = std::max(named_end, end);[P2] Bound recovery by the data that actually exists. An index entry beyond EOF still raises named_end. The next put fills that gap with zeros under the writer lock, even when the nominal offset is far beyond the pack. Reject or quarantine unverified extents with a bounded recovery policy, and process index records in bounded batches.
@ -241,6 +241,7 @@ SW(LONGSHOT_DRIVER_ENV_SET, diag, string, "not set", "Set by the game itself befSW(LONGSHOT_NFC_DIR, player, string, "Windows: %LOCALAPPDATA%\\Longshot\\nfc; Linux: $XDG_DATA_HOME/longshot/nfc (~/.local/share/longshot/nfc)", "The folder of NFC tag files the Insert menu's NFC tab lists, instead of the user's own.")SW(LONGSHOT_NO_AV1_NOTICE, kill, bool, "the notice window is shown once (Windows, no AV1 decoder)", "No pop-up asking to install the AV1 Video Extension; the log line is still written.")SW(LONGSHOT_PREPARE_ONLY, player, bool, "off: a normal game run", "Marks a --prepare-shaders run: the renderer only compiles and saves shaders, skipping work a played run does.")SW(LONGSHOT_SHADER_STORE, player, string, "Windows: %LOCALAPPDATA%\\Longshot\\shaders; Linux: $XDG_CACHE_HOME/longshot/shaders (~/.cache/longshot/shaders)", "The shader store's folder, the SPIR-V every game folder's program cache shares (gpu/vk/spirv_pack.h), instead of the user's own; 0: none, each record keeps its SPIR-V.")[P2] Carry this override through all launch paths. nx2/make_game.py:1600–1612 and :1691 strip LONGSHOT_* for preparation. scripts/devtools/packs.py uses that filter and builder/linux/longshot-launch:25 strips it for AppImage play. Preparing against the default and playing with a custom path or 0 gives another store ID and can discard the just-prepared program records. Preserve both forms through preparation and launch.
@ -201,3 +261,3 @@programs += 1for nvn, vk, spirv in stages(data):for nvn, vk, spirv in stages(data, store):job = (key, nvn, vk, spirv)[P2] Fail validation when a referenced stage cannot be inspected. A missing or unreadable pack makes stages() omit references silently, and the tool can report zero failed stages and exit 0 despite checking none of them. Check the pack generation against the program-cache header and count unreadable references as failures. Report stages validated separately from program records encountered.
@futondestroyer thank you. All four inline points were real, and they're fixed in three commits on top. Head
720c6988, still on main89c3d21e.[P2] switches.def:244, the override through every launch path (issuecomment-58382).
50efd93e:make_game.GAME_ENV_KEEPkeepsLONGSHOT_SHADER_STOREingame_env(), which covers the preparation at the build andpacks.py.builder/linux/longshot-launchkeeps the same name for the AppImage's Play. Checks:test_prepare_at_build(a folder and0both survivegame_env, and the shell script's kept name matchesGAME_ENV_KEEP) andtest_longshot_launch(the AppImage's Play passes both on; it runs on Linux CI).[P2] spirv_pack.h:324, truncating a generation another process reads (issuecomment-58383).
e2eb3e8f, two parts:kProgramCacheFormat3), andStagekeeps it for the reads after the load (a light read, a dropped module remade).Store::read(at, n, want)gives the words only when the blob at that place has that hash and the words hash to it. Whatever lies at a reused place is never handed to the driver in another blob's stead.[P2] spirv_pack.h:364, bounded recovery (issuecomment-58384). With the hash in the record, the padding out to the furthest named place has no job left, so it's gone. An entry naming bytes the pack doesn't have is left out, and nothing is written for it. The index is read 4,096 entries at a time.
The new checks in
tests/spirv_pack_test.cppcover:Storereads it.Undoing either fix (a read ignoring the hash, or the store remade in place) fails them.
[P2] validate_spirv.py:204, fail when stages can't be read (issuecomment-58385).
720c6988: the cache's head names its store, and the store found must be that one. Each stage reference the store doesn't give (no store, another store, or another hash at the place) counts as not read and fails the run. The report gives stages checked apart from programs.test_validate_spirvhas a cache that passes with its store and fails with none or another.From the summary:
longshot_programs.binis 150,840,657 bytes (+1.2% for the hashes) and itslongshot_programs.idx1,133,936 bytes. The store is 3,736,516,412 bytes (spirv.pack, unchanged to the byte) plus 1,621,936 bytes (spirv.idx). Measurements has the row.Run in the game on
720c6988(Windows 11, RX 6700 XT), from an empty store:Not done here: the cold v1-to-v2 and warm runs on Linux and Windows NVIDIA with an explicit root. Those are in Still to test.
@Beanstalk linux (any GPU) and windows nvidia, on
720c6988: build this branch, let the runtime prepare the shaders, then play about a minute from the kokiri-start save. Then do one more start withLONGSHOT_SHADER_STOREset to a new empty folder, both for the build's preparation and for the play. For each, report:vk: shader store ...: N SPIR-V blobsline (N about 50,000 expected after preparing);vk: program cache ...: N program keys kept, K KBline (K about 150,000 expected);spirv.packandspirv.idxin the store folder;has no SPIR-V,did not readoranother shader storeline (none expected).@madebyjordan approval of
720c698recorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.Correction to this PR's Risks. I wrote that "every player prepares once after updating" and measured it as a dev runtime's translation-only preparation (118.7 s). That understated the real cost.
program_cache_stamp()mixeskProgramCacheFormatinto the stamp that every cache file is judged by. So nightly-622's first start also dropped the driver's pipeline cache (longshot_vk_pipelines.bin), truncated the pipeline notes, and started the module identifiers over. The launcher's--compile-shaders mostthen compiles every pipeline cold: over 40 minutes and counting on an RX 6700 XT, and players on Discord report it too. None of those files depends on the program file's layout. The fix keeps them across a format change for the same translator: branchfix/pipelines-outlive-cache-formaton my fork, with a game-free test that fails on main. It opens as a PR once it has run in the game. It helps players who haven't started 622 yet; those who have already lost their files.