Shader caches take 9.5 GB after a fresh preparation: half the program cache is the same SPIR-V twice (proposal: compress each stage, notes without zeros) #1649

Open
opened 2026-11-18 20:25:30 +00:00 by TrashPanda · 2 comments
Contributor

For a decision: a fresh preparation leaves 9.5 GB of shader caches (plus 1.1 GB of NVIDIA's own), which players with small drives notice. Half of longshot_programs.bin is the same SPIR-V stored twice, and the pipeline notes are 99% zeros. Every fix changes the cache format, so every player would prepare once more: that is the team's call, ideally with a release that changes the translator anyway. The proposal below is written, not implemented.

Measured on existing caches only (no new runs), on Linux, RTX 4070 Ti: a fresh Most preparation of v3.0.0-rc.5-era main, then played
(its longshot_programs.bin the same size, 7,913,087,763 bytes, as fresh preparations of main 1cc59e37 on 2026-11-18), and a
second folder prepared and played longer. Read with two read-only scripts over the files' records.

What a fresh Most preparation leaves in cache/ (9.5 GB) plus NVIDIA's GLCache (1.1 GB)

File Size What it is Compresses (zstd -3)
longshot_programs.bin 7.91 GB 56,722 program records (47,867 keys) 3.7x a record alone; 6.8x as a stream; 9.8x with --long=27
longshot_vk_pipelines.bin 1.1-1.4 GB the Vulkan pipeline cache (NVIDIA's binaries) 1.0x
longshot_pipelines.bin (notes) 0.84-0.88 GB 32,390 pipeline notes of 27,144 bytes, ~105 non-zero bytes each 790x (879.6 MB -> 1.1 MB)
NVIDIA GLCache 1.1 GB the driver's own not ours

longshot_programs.bin

Layout (vk_progcache.cpp program_cache_save, format 1, uncompressed): a 16-byte file head, then records
[key u64][size u64][payload]; the payload is put_head (flags, answers, questions, workgroup, fallback words), the stage
count, and per stage: NVN and Vulkan stage, the SPIR-V (u64 count + words), c1 constants, Shader::Info as it lies in
memory (816 bytes, heap pointers included), then Info's lists.

Part MB Share
SPIR-V 7,765.6 98.1%
Shader::Info as it lies (816 B a stage) 86.2 1.1%
Info's lists 42.0 0.5%
c1 constants 10.1 0.1%
heads (record, program, stage) 9.2 0.1%

By kind: graphics 44,511 records 5,894 MB; compute 9,927 records (4,128 programs, up to 64 texture variants) 984 MB; mesh
2,284 records 1,036 MB. SPIR-V by stage: vertex 1,690 MB (44,511), fragment 4,346 MB (46,795), compute 971 MB, task 242 MB,
mesh 515 MB.

Duplicate and dead data

  • The same SPIR-V stored again: 3,976 MB (51% of the file). 105,591 stages hold only 50,878 distinct blobs (3,789 MB).
    By stage: vertex 975 MB of 1,690 (58%), fragment 2,376 of 4,346 (55%), task 241 of 242 (99%), mesh 385 of 515 (75%),
    compute 0.7 of 971. Mostly across program keys (programs sharing a vertex or fragment shader: the seed's 33,369 vertex uses
    are of 15,852 stages, its 34,681 fragment uses of 22,356; and the bone-input variants, which repeat the fragment stage).
    36,263 blobs are stored once, 8,245 twice, 339 twenty times or more.
  • Superseded records: 220 records (57 MB) of a key and answers recorded again in the same preparation (221 after play).
    Their earlier copy is never read ("of two records of one variant, the later is the one kept").
  • Never shrinks: the file is only appended to (disk_init: append; a record cut short at the end is cut off). It starts
    over only when the translator stamp or kProgramCacheFormat changes, then holds everything again. No compaction exists.
  • Not deterministic: Shader::Info is written as it lies, its vectors' heap pointers included (1.1% of the file), so two
    preparations write different bytes for the same programs (sizes and keys match: the index's (key, size, kind) multiset is
    identical across runs).
  • Variants never used: not measurable from the files (the compute warm plan keeps all 64 variants of a program at the limit;
    play's longshot_compute_variants.bin names the variants dispatched: 159 in game-rs and 276 in game-br, against 9,927 kept)
    • compute is 12% of the file.

Compression (python zstandard 0.25, one thread, 3,038 sampled records compressed one at a time as program_cache_save
does with RX_VK_PROGRAM_CACHE_ZSTD=1): level 1 x3.8 (416 MB/s in, 1,202 MB/s out), level 3 x3.7 (314 / 1,102 MB/s),
level 9 x4.0, level 19 x5.0. The distinct blobs alone: x3.7 at level 3. As a stream (zstd CLI, 512 MB): x6.8 at -3,
x9.8 with --long=27, decompressing at 1.8 GB/s a thread: the variants of one program are near copies of each other.

Options

Option programs.bin Load-time cost Risk
RX_VK_PROGRAM_CACHE_ZSTD=1 by default (exists: whole records compressed) ~2.1 GB forces RX_VK_LOAD=start (records only read whole): every listed program before the title again, the memory the load at create saves high
A: each stage's SPIR-V compressed (zstd 1-3) inside the record, heads and tails left in place ~2.1 GB (-5.8 GB) +1 ms per MB of SPIR-V read (decompress), on the load workers and at a compute program's first dispatch (+~0.1 ms for a 145 KB variant); the light reads by position keep working medium: format 2
B: identical SPIR-V stored once, later records referring to it ~4.0 GB (-3.9 GB) none in CPU; a referred stage is one more read elsewhere in the file medium: format 2, a hash map at save
A + B ~1.05 GB (-6.9 GB) as A medium-high
C: pipeline notes without their zeros (run-length) notes 0.84 GB -> ~1-4 MB none to speak of low-medium: the notes' own format; no translation again (notes are made again from the state lists)

Proposed: A, then C. A has the best size for the least load-time risk: it keeps every reader's positional access (the
create path's light reads find a record's head and descriptors where they are; only complete_from, the whole-record read,
complete_graphics and program_cache_each decompress the stage they read), and the compressed bytes are fewer to read from
disk, which a cold start on an HDD or SATA SSD gains back. B adds most of its win again on top of A but needs a blob map
at save and references in four readers; do it after A, if wanted. C is independent and cheap in risk (0.84 GB).

A, in code (vk_progcache.cpp only, outside the translator stamp)

  • kProgramCacheFormat 1 -> 2 (the program cache, the notes, the prepared record and the pipeline cache files go with it:
    every player prepares again once; ship it with a release that changes the translator anyway).
  • program_cache_save: each stage's SPIR-V as [u64 words | 1 << 63][u64 stored bytes][zstd frame] (level 1 or 3; a stage
    that does not shrink stays plain, count without the bit). zstd is already linked (program_cache_payload).
  • program_cache_read: decompress the stage (code_at/code_words point at the stored field; Stage's and Program's sizes stay
    as they are: they are in the translator stamp's ABI words).
  • program_cache_read_light: the stored length, not words x 4, finds the tail.
  • complete_from, program_cache_complete_graphics: read the field at code_at, decompress, then make the module.
  • program_cache_each (the prediction, All only): decompress.
  • vk_programs.cpp: its byte counts use the stored size.
  • Tests to update: program_cache_cleanup_test.cpp (cuts save, read, payload, load), test_program_cache_one_variant.py,
    test_program_cache_reader.py, test_program_cache_format.py, test_program_cache_index.py; a new one: a record with a
    compressed stage read back whole, by the light read, by complete_from and by complete_graphics, byte for byte.
  • Not contained enough to do without the team: it touches the cache format every player has, forces one re-preparation,
    and changes six readers and five tests. Proposed, not implemented.

What an in-game check of A must confirm

A cold preparation finishes with the same counts (programs, pipelines, "0 not made"); the first start after it loads at
create (programs are loaded when the game makes them (create)), plays Kokiri Forest and Hyrule Field with no draws left
out for programs that did not read (program cache: ... did not read lines stay absent); a compute-heavy scene (the movie
upscaler, the culling programs) shows no first-dispatch hitch beyond main's; and the cache folder's size.

Found while looking at Preparing shaders on 4-core / 8-16 GB PCs (worklist 3.2).

🤖 Generated with Claude Code

**For a decision:** a fresh preparation leaves 9.5 GB of shader caches (plus 1.1 GB of NVIDIA's own), which players with small drives notice. Half of `longshot_programs.bin` is the same SPIR-V stored twice, and the pipeline notes are 99% zeros. Every fix changes the cache format, so every player would prepare once more: that is the team's call, ideally with a release that changes the translator anyway. The proposal below is written, not implemented. Measured on existing caches only (no new runs), on Linux, RTX 4070 Ti: a fresh Most preparation of v3.0.0-rc.5-era main, then played (its longshot_programs.bin the same size, 7,913,087,763 bytes, as fresh preparations of main 1cc59e37 on 2026-11-18), and a second folder prepared and played longer. Read with two read-only scripts over the files' records. ## What a fresh Most preparation leaves in cache/ (9.5 GB) plus NVIDIA's GLCache (1.1 GB) | File | Size | What it is | Compresses (zstd -3) | |---|---|---|---| | longshot_programs.bin | 7.91 GB | 56,722 program records (47,867 keys) | 3.7x a record alone; 6.8x as a stream; 9.8x with --long=27 | | longshot_vk_pipelines.bin | 1.1-1.4 GB | the Vulkan pipeline cache (NVIDIA's binaries) | 1.0x | | longshot_pipelines.bin (notes) | 0.84-0.88 GB | 32,390 pipeline notes of 27,144 bytes, ~105 non-zero bytes each | 790x (879.6 MB -> 1.1 MB) | | NVIDIA GLCache | 1.1 GB | the driver's own | not ours | ## longshot_programs.bin Layout (vk_progcache.cpp program_cache_save, format 1, uncompressed): a 16-byte file head, then records `[key u64][size u64][payload]`; the payload is put_head (flags, answers, questions, workgroup, fallback words), the stage count, and per stage: NVN and Vulkan stage, the SPIR-V (u64 count + words), c1 constants, Shader::Info as it lies in memory (816 bytes, heap pointers included), then Info's lists. | Part | MB | Share | |---|---:|---:| | SPIR-V | 7,765.6 | 98.1% | | Shader::Info as it lies (816 B a stage) | 86.2 | 1.1% | | Info's lists | 42.0 | 0.5% | | c1 constants | 10.1 | 0.1% | | heads (record, program, stage) | 9.2 | 0.1% | By kind: graphics 44,511 records 5,894 MB; compute 9,927 records (4,128 programs, up to 64 texture variants) 984 MB; mesh 2,284 records 1,036 MB. SPIR-V by stage: vertex 1,690 MB (44,511), fragment 4,346 MB (46,795), compute 971 MB, task 242 MB, mesh 515 MB. **Duplicate and dead data** - **The same SPIR-V stored again: 3,976 MB (51% of the file).** 105,591 stages hold only 50,878 distinct blobs (3,789 MB). By stage: vertex 975 MB of 1,690 (58%), fragment 2,376 of 4,346 (55%), task 241 of 242 (99%), mesh 385 of 515 (75%), compute 0.7 of 971. Mostly across program keys (programs sharing a vertex or fragment shader: the seed's 33,369 vertex uses are of 15,852 stages, its 34,681 fragment uses of 22,356; and the bone-input variants, which repeat the fragment stage). 36,263 blobs are stored once, 8,245 twice, 339 twenty times or more. - **Superseded records**: 220 records (57 MB) of a key and answers recorded again in the same preparation (221 after play). Their earlier copy is never read ("of two records of one variant, the later is the one kept"). - **Never shrinks**: the file is only appended to (disk_init: append; a record cut short at the end is cut off). It starts over only when the translator stamp or kProgramCacheFormat changes, then holds everything again. No compaction exists. - **Not deterministic**: Shader::Info is written as it lies, its vectors' heap pointers included (1.1% of the file), so two preparations write different bytes for the same programs (sizes and keys match: the index's (key, size, kind) multiset is identical across runs). - Variants never used: not measurable from the files (the compute warm plan keeps all 64 variants of a program at the limit; play's longshot_compute_variants.bin names the variants dispatched: 159 in game-rs and 276 in game-br, against 9,927 kept) - compute is 12% of the file. **Compression** (python zstandard 0.25, one thread, 3,038 sampled records compressed one at a time as program_cache_save does with RX_VK_PROGRAM_CACHE_ZSTD=1): level 1 x3.8 (416 MB/s in, 1,202 MB/s out), level 3 x3.7 (314 / 1,102 MB/s), level 9 x4.0, level 19 x5.0. The distinct blobs alone: x3.7 at level 3. As a stream (zstd CLI, 512 MB): x6.8 at -3, x9.8 with --long=27, decompressing at 1.8 GB/s a thread: the variants of one program are near copies of each other. ## Options | Option | programs.bin | Load-time cost | Risk | |---|---:|---|---| | RX_VK_PROGRAM_CACHE_ZSTD=1 by default (exists: whole records compressed) | ~2.1 GB | forces RX_VK_LOAD=start (records only read whole): every listed program before the title again, the memory the load at create saves | high | | A: each stage's SPIR-V compressed (zstd 1-3) inside the record, heads and tails left in place | ~2.1 GB (-5.8 GB) | +1 ms per MB of SPIR-V read (decompress), on the load workers and at a compute program's first dispatch (+~0.1 ms for a 145 KB variant); the light reads by position keep working | medium: format 2 | | B: identical SPIR-V stored once, later records referring to it | ~4.0 GB (-3.9 GB) | none in CPU; a referred stage is one more read elsewhere in the file | medium: format 2, a hash map at save | | A + B | ~1.05 GB (-6.9 GB) | as A | medium-high | | C: pipeline notes without their zeros (run-length) | notes 0.84 GB -> ~1-4 MB | none to speak of | low-medium: the notes' own format; no translation again (notes are made again from the state lists) | **Proposed: A, then C.** A has the best size for the least load-time risk: it keeps every reader's positional access (the create path's light reads find a record's head and descriptors where they are; only complete_from, the whole-record read, complete_graphics and program_cache_each decompress the stage they read), and the compressed bytes are fewer to read from disk, which a cold start on an HDD or SATA SSD gains back. B adds most of its win again on top of A but needs a blob map at save and references in four readers; do it after A, if wanted. C is independent and cheap in risk (0.84 GB). ### A, in code (vk_progcache.cpp only, outside the translator stamp) - kProgramCacheFormat 1 -> 2 (the program cache, the notes, the prepared record and the pipeline cache files go with it: **every player prepares again once**; ship it with a release that changes the translator anyway). - program_cache_save: each stage's SPIR-V as `[u64 words | 1 << 63][u64 stored bytes][zstd frame]` (level 1 or 3; a stage that does not shrink stays plain, count without the bit). zstd is already linked (program_cache_payload). - program_cache_read: decompress the stage (code_at/code_words point at the stored field; Stage's and Program's sizes stay as they are: they are in the translator stamp's ABI words). - program_cache_read_light: the stored length, not words x 4, finds the tail. - complete_from, program_cache_complete_graphics: read the field at code_at, decompress, then make the module. - program_cache_each (the prediction, All only): decompress. - vk_programs.cpp: its byte counts use the stored size. - Tests to update: program_cache_cleanup_test.cpp (cuts save, read, payload, load), test_program_cache_one_variant.py, test_program_cache_reader.py, test_program_cache_format.py, test_program_cache_index.py; a new one: a record with a compressed stage read back whole, by the light read, by complete_from and by complete_graphics, byte for byte. - Not contained enough to do without the team: it touches the cache format every player has, forces one re-preparation, and changes six readers and five tests. Proposed, not implemented. ### What an in-game check of A must confirm A cold preparation finishes with the same counts (programs, pipelines, "0 not made"); the first start after it loads at create (`programs are loaded when the game makes them (create)`), plays Kokiri Forest and Hyrule Field with no draws left out for programs that did not read (`program cache: ... did not read` lines stay absent); a compute-heavy scene (the movie upscaler, the culling programs) shows no first-dispatch hitch beyond main's; and the cache folder's size. Found while looking at Preparing shaders on 4-core / 8-16 GB PCs (worklist 3.2). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Member

The same files on AMD, for the cross-vendor side of this. Windows 11, RX 9070 XT (driver 26.11.1), main 1055077d. The cache folder is our test install's, not a fresh Most preparation. Measured read-only, by sampling records in memory. No game runs were made for this.

File Here (AMD) Issue (NVIDIA, Linux)
longshot_pipelines.bin (notes) 885,518,728 bytes, 32,623 notes of 27,144 bytes 0.84-0.88 GB, 32,390 notes
non-zero bytes per note (mean of 256 evenly spaced notes) 83 ~105
longshot_programs.bin 7,730,356,807 bytes 7,913,087,763 bytes
whole program records, zstd -3, one at a time (256 evenly spaced records) 3.75x 3.7x

For C: zstd -3 on each note on its own, with no shared dictionary and no stream, took the 256 sampled notes from 6,948,864 bytes to 35,493. Over 32,623 notes that projects to about 4.5 MB of payload before any framing or index, about 0.88 GB less. That is close to the 1-4 MB estimated for run-length notes. Each note stays independent, so reading a note by position still works. On this AMD install the notes shrink about as much as on NVIDIA.

Not measured here: the duplicated SPIR-V (option B) and the per-stage compression (option A), so I have no AMD numbers for either. There is also no timing of the note reader or the cold open with compact notes.

🤖 Generated with Claude Code

The same files on AMD, for the cross-vendor side of this. Windows 11, RX 9070 XT (driver 26.11.1), main 1055077d. The cache folder is our test install's, not a fresh Most preparation. Measured read-only, by sampling records in memory. No game runs were made for this. | File | Here (AMD) | Issue (NVIDIA, Linux) | |---|---:|---:| | longshot_pipelines.bin (notes) | 885,518,728 bytes, 32,623 notes of 27,144 bytes | 0.84-0.88 GB, 32,390 notes | | non-zero bytes per note (mean of 256 evenly spaced notes) | 83 | ~105 | | longshot_programs.bin | 7,730,356,807 bytes | 7,913,087,763 bytes | | whole program records, zstd -3, one at a time (256 evenly spaced records) | 3.75x | 3.7x | **For C:** zstd -3 on each note on its own, with no shared dictionary and no stream, took the 256 sampled notes from 6,948,864 bytes to 35,493. Over 32,623 notes that projects to about 4.5 MB of payload before any framing or index, about 0.88 GB less. That is close to the 1-4 MB estimated for run-length notes. Each note stays independent, so reading a note by position still works. On this AMD install the notes shrink about as much as on NVIDIA. Not measured here: the duplicated SPIR-V (option B) and the per-stage compression (option A), so I have no AMD numbers for either. There is also no timing of the note reader or the cold open with compact notes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Member

Option B across game folders, measured: the same SPIR-V is stored again in every build, not only within one file.

Windows 11, RX 6700 XT. Two of a player's installs, prepared and played: stable v3.0.0 (program cache stamp 6cc34dcd…) and nightly-486 (stamp 33f15f4a…). Different stamps, so today each keeps a whole cache of its own. Read-only, every record of both files, SPIR-V found by its magic and hashed (BLAKE2b-128):

v3.0.0 nightly-486
program keys 47,798 47,798
SPIR-V stored 7.18 GB 7.53 GB
SPIR-V distinct 3.48 GB 3.48 GB
  • Keys in both: 47,798. Keys whose SPIR-V is byte-identical in both: 47,592 (99.6%). The translator changed between the two, but its output for almost every program did not.
  • One store of the distinct SPIR-V for both: 3.51 GB, against 14.71 GB stored today.

So for a player with stable and nightly (or test builds, or a pull request build), B is worth far more than within one file: each further build adds only the few programs whose translation changed, not ~7.5 GB.

I'd like to do B as a pack shared by every game folder, under #1759:

  • A content-addressed SPIR-V pack outside the game folder (%LOCALAPPDATA%\Longshot\shaders\, $XDG_CACHE_HOME/longshot/shaders/ on Linux), appended to, never rewritten; its index read once at the start.
  • longshot_programs.bin keeps every record as now (heads, Info, lists, the order, the index), with each stage's SPIR-V replaced by a reference (hash, words, the pack's offset). The four readers that take SPIR-V (the whole read, the light read by position, complete_from/complete_graphics, program_cache_each) read it from the pack.
  • A process that can't take the pack (another Longshot writing it, a read-only disk) writes its stages inline, as format 1 does, so nothing waits and nothing breaks.
  • kProgramCacheFormat 1 -> 2: one more preparation for every player, as you wrote for A. I'd ship it with a translator change.
  • A fits on top unchanged: compress each blob in the pack, which takes the shared 3.5 GB to about 1 GB.

@TrashPanda, are you already on A or B in vk_progcache.cpp? If not, I'll start B-shared now and keep the pack's blob format open for A. If you'd rather do it together, or in another order, say so.

🤖 Generated with Claude Code

Option B across game folders, measured: the same SPIR-V is stored again in **every** build, not only within one file. Windows 11, RX 6700 XT. Two of a player's installs, prepared and played: stable v3.0.0 (program cache stamp `6cc34dcd…`) and nightly-486 (stamp `33f15f4a…`). Different stamps, so today each keeps a whole cache of its own. Read-only, every record of both files, SPIR-V found by its magic and hashed (BLAKE2b-128): | | v3.0.0 | nightly-486 | |---|---:|---:| | program keys | 47,798 | 47,798 | | SPIR-V stored | 7.18 GB | 7.53 GB | | SPIR-V distinct | 3.48 GB | 3.48 GB | - Keys in both: 47,798. Keys whose SPIR-V is byte-identical in both: **47,592 (99.6%)**. The translator changed between the two, but its output for almost every program did not. - One store of the distinct SPIR-V for both: **3.51 GB, against 14.71 GB stored today.** So for a player with stable and nightly (or test builds, or a pull request build), B is worth far more than within one file: each further build adds only the few programs whose translation changed, not ~7.5 GB. I'd like to do B as a pack shared by every game folder, under #1759: - A content-addressed SPIR-V pack outside the game folder (`%LOCALAPPDATA%\Longshot\shaders\`, `$XDG_CACHE_HOME/longshot/shaders/` on Linux), appended to, never rewritten; its index read once at the start. - `longshot_programs.bin` keeps every record as now (heads, Info, lists, the order, the index), with each stage's SPIR-V replaced by a reference (hash, words, the pack's offset). The four readers that take SPIR-V (the whole read, the light read by position, complete_from/complete_graphics, program_cache_each) read it from the pack. - A process that can't take the pack (another Longshot writing it, a read-only disk) writes its stages inline, as format 1 does, so nothing waits and nothing breaks. - kProgramCacheFormat 1 -> 2: one more preparation for every player, as you wrote for A. I'd ship it with a translator change. - A fits on top unchanged: compress each blob in the pack, which takes the shared 3.5 GB to about 1 GB. @TrashPanda, are you already on A or B in `vk_progcache.cpp`? If not, I'll start B-shared now and keep the pack's blob format open for A. If you'd rather do it together, or in another order, say so. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1649
No description provided.