The shader cache stamp hashes the translator's source byte for byte, so a comment edit makes every player prepare all shaders again #1611

Open
opened 2026-11-18 15:43:50 +00:00 by mooki · 1 comment
Member

Type: performance · Area: renderer, shader caches, build · Part of: #1493

TL;DR

The stamp that decides whether the shader caches survive an update hashes every file of the translator whole, comments and blank lines included. A change to one comment in vk_shader.cpp therefore makes every player prepare all programs again at the next start. #1510 takes the struct size out of the key; this is the other input, the text. We propose to hash a conservative normal form of the source, so an edit that cannot change a translation keeps the caches, and a check that proves it on a sample of programs.

Breakdown

  • Who it affects: every player at the first start after an update that touched a stamped file. Slow PCs pay minutes: #1510 records 251.6 s on a Steam Deck in a tester's run and about 5 minutes on a fast desktop.
  • What is wrong: stamp() reads the bytes of the three folders and three files and hashes them, with Windows line ends turned into Unix ones and nothing else. vk_shader.cpp alone was touched by 112 commits since 6 Nov.
  • What we propose: normalise before hashing. First only whole-line comments, block comments and blank lines; trailing spaces; later, marked regions known not to affect translation. Every rule has a test. Any doubt about a text keeps it as it is, so a mistake can only make the stamp move more often, never less.
  • What does not change: the stamp's other inputs (GPU facts, switches, the cache format number), the cache files, and #1510's change.
  • What we do not claim: how many of the 131 commits would have kept the caches. Counting that is the first step.
  • How we would prove it: a fixture over the stamp function, then a sample of programs translated by two builds with equal normal forms and compared byte for byte.

Evidence

Code and history on forge/main at 430eef4f2:

  • nx2/translator_files.py:20: the docstring says files are stamped "byte for byte, comments included". :27-28: the folders (eden, sirit, compat) and the files (vk_shader.cpp, vk_bank.cpp, vk_state.h). :64-69: stamp() is a SHA-1 over each file's relative path and bytes.
  • gpu/vk/vk_shader.cpp:1259: translator_stamp() mixes LONGSHOT_SHADER_STAMP with the GPU facts and the switches.
  • scripts/dev_packs.py prepares a runtime again by the same files (stamped() at :59), so a normal form changes what a development pack re-prepares as well.
  • git log on forge/main since 2026-11-06 touching a stamped file: vk_shader.cpp 112 commits, vk_bank.cpp 12, vk_state.h 9, the three folders 138; 222 commits in all, and 131 of the 1,420 first-parent commits on main. A commit that touches a stamped file moves the stamp whether or not it changes a translation. How many of them changed no translation was not counted.
  • Already done around it: #370 took vk_api.h out of the stamp; #1418 keeps the pipeline marker across an update that changes no shader; #1390 deleted switches the translation read; #1510 and #1542 take sizeof(Program) out of the key.

The fork's claim, not ours: an outside audit of a fork says that in the fork 14 of 17 stamp changes between two releases changed no translation, and that each cost about 46,600 programs of preparation (75 seconds). It says the fork fixed this by hashing the sources without comments, blank lines and trailing spaces, with marked regions left out. We have not measured our own share.

Proposed change

  1. A normaliser in nx2/translator_files.py, used by stamp(): it removes whole-line // comments, /* */ blocks that start at the beginning of a line (after white space) and nothing else, blank lines, and trailing white space. It leaves any line it is unsure about, including lines with a quote or a backslash before a comment marker, exactly as it is.
  2. A test that pins what is removed and what is not, in the project's style: a comment edit keeps the stamp; a token edit, an added line of code, a changed string literal, a changed macro or a changed #include moves it.
  3. A check that proves the rule on real code: translate a sample of programs with two builds whose normal forms are equal but whose bytes differ, and compare the SPIR-V. A mismatch means the normaliser hid a real change.
  4. Later, and only with the check in place: a region marker for code that is known not to affect translation (cache input and output), each use with a test.
  5. Ship with a release that re-prepares anyway: the change moves the stamp once.

How we would test it

  • Game-free, fails today: build two copies of the stamped files in a temporary folder that differ by one comment in vk_shader.cpp. The stamps must be equal; today they differ. A second pair differing by one token must differ.
  • Game-free: the normaliser against a small table of hard cases (a comment marker inside a string, a line continuation, a raw string, a comment after code, #if 0 blocks). Each case is either left alone or removed, and a case that is left alone is correct by definition.
  • In the game: three builds started on one prepared folder: main; main with one comment added to vk_shader.cpp; this change with the same comment. The second drops the caches at its first start, the third keeps them. A yes or no in the log, not a timing.
  • The SPIR-V comparison of step 3 on a sample of programs from each stage kind. No speed claim is made.
  • #1510 and #1542 (open): the struct size in the key. Complements this one; neither depends on the other.
  • #1418, #370, #1390 (merged): the other stamp inputs.
  • #737 (open): one shader cache store keyed by the stamp; every avoided stamp change is also an avoided store.
  • #1493 slice d: its exit rule says an unrelated header edit must not invalidate caches; this is a way to meet it.
  • The issue about the program pick (to be linked) touches the same file.

New on this forge; not opened on GitHub.

**Type:** performance · **Area:** renderer, shader caches, build · **Part of:** #1493 ## TL;DR The stamp that decides whether the shader caches survive an update hashes every file of the translator whole, comments and blank lines included. A change to one comment in `vk_shader.cpp` therefore makes every player prepare all programs again at the next start. #1510 takes the struct size out of the key; this is the other input, the text. We propose to hash a conservative normal form of the source, so an edit that cannot change a translation keeps the caches, and a check that proves it on a sample of programs. ## Breakdown - **Who it affects:** every player at the first start after an update that touched a stamped file. Slow PCs pay minutes: #1510 records 251.6 s on a Steam Deck in a tester's run and about 5 minutes on a fast desktop. - **What is wrong:** `stamp()` reads the bytes of the three folders and three files and hashes them, with Windows line ends turned into Unix ones and nothing else. `vk_shader.cpp` alone was touched by 112 commits since 6 Nov. - **What we propose:** normalise before hashing. First only whole-line comments, block comments and blank lines; trailing spaces; later, marked regions known not to affect translation. Every rule has a test. Any doubt about a text keeps it as it is, so a mistake can only make the stamp move more often, never less. - **What does not change:** the stamp's other inputs (GPU facts, switches, the cache format number), the cache files, and #1510's change. - **What we do not claim:** how many of the 131 commits would have kept the caches. Counting that is the first step. - **How we would prove it:** a fixture over the stamp function, then a sample of programs translated by two builds with equal normal forms and compared byte for byte. ## Evidence Code and history on `forge/main` at `430eef4f2`: - `nx2/translator_files.py:20`: the docstring says files are stamped "byte for byte, comments included". `:27-28`: the folders (`eden`, `sirit`, `compat`) and the files (`vk_shader.cpp`, `vk_bank.cpp`, `vk_state.h`). `:64-69`: `stamp()` is a SHA-1 over each file's relative path and bytes. - `gpu/vk/vk_shader.cpp:1259`: `translator_stamp()` mixes `LONGSHOT_SHADER_STAMP` with the GPU facts and the switches. - `scripts/dev_packs.py` prepares a runtime again by the same files (`stamped()` at `:59`), so a normal form changes what a development pack re-prepares as well. - `git log` on `forge/main` since 2026-11-06 touching a stamped file: `vk_shader.cpp` 112 commits, `vk_bank.cpp` 12, `vk_state.h` 9, the three folders 138; 222 commits in all, and 131 of the 1,420 first-parent commits on main. A commit that touches a stamped file moves the stamp whether or not it changes a translation. How many of them changed no translation was not counted. - Already done around it: #370 took `vk_api.h` out of the stamp; #1418 keeps the pipeline marker across an update that changes no shader; #1390 deleted switches the translation read; #1510 and #1542 take `sizeof(Program)` out of the key. The fork's claim, not ours: an outside audit of a fork says that in the fork 14 of 17 stamp changes between two releases changed no translation, and that each cost about 46,600 programs of preparation (75 seconds). It says the fork fixed this by hashing the sources without comments, blank lines and trailing spaces, with marked regions left out. We have not measured our own share. ## Proposed change 1. A normaliser in `nx2/translator_files.py`, used by `stamp()`: it removes whole-line `//` comments, `/* */` blocks that start at the beginning of a line (after white space) and nothing else, blank lines, and trailing white space. It leaves any line it is unsure about, including lines with a quote or a backslash before a comment marker, exactly as it is. 2. A test that pins what is removed and what is not, in the project's style: a comment edit keeps the stamp; a token edit, an added line of code, a changed string literal, a changed macro or a changed `#include` moves it. 3. A check that proves the rule on real code: translate a sample of programs with two builds whose normal forms are equal but whose bytes differ, and compare the SPIR-V. A mismatch means the normaliser hid a real change. 4. Later, and only with the check in place: a region marker for code that is known not to affect translation (cache input and output), each use with a test. 5. Ship with a release that re-prepares anyway: the change moves the stamp once. ## How we would test it - Game-free, fails today: build two copies of the stamped files in a temporary folder that differ by one comment in `vk_shader.cpp`. The stamps must be equal; today they differ. A second pair differing by one token must differ. - Game-free: the normaliser against a small table of hard cases (a comment marker inside a string, a line continuation, a raw string, a comment after code, `#if 0` blocks). Each case is either left alone or removed, and a case that is left alone is correct by definition. - In the game: three builds started on one prepared folder: main; main with one comment added to `vk_shader.cpp`; this change with the same comment. The second drops the caches at its first start, the third keeps them. A yes or no in the log, not a timing. - The SPIR-V comparison of step 3 on a sample of programs from each stage kind. No speed claim is made. ## Related - #1510 and #1542 (open): the struct size in the key. Complements this one; neither depends on the other. - #1418, #370, #1390 (merged): the other stamp inputs. - #737 (open): one shader cache store keyed by the stamp; every avoided stamp change is also an avoided store. - #1493 slice d: its exit rule says an unrelated header edit must not invalidate caches; this is a way to meet it. - The issue about the program pick (to be linked) touches the same file. New on this forge; not opened on GitHub.
Collaborator

Possibly related, for context:

  • issue #1510 The shader cache key includes the size of the program struct, so adding any member to it makes every player prepare all shaders again — Separate shader-stamp input causes the same full cache invalidation on changes to Program layout.
  • issue #1609 A program pick reads and decodes every stage of a new program under the shader lock, even when the same bytes were decoded before — Related shader translation path and same renderer file; distinct per-program decode cost.
  • issue #1493 Renderer rewrite — Renderer rewrite umbrella; shader-cache stamp work is within its scope.
  • PR #737 feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing — Open PR for cache storage keyed by translator stamp; benefits from avoiding unnecessary stamp changes.
  • PR #370 perf(vk): the shader caches' stamp leaves out vk_api.h, so a menu, input or audio change keeps every player's translated programs — Merged PR changed translator stamp inputs to prevent unrelated header changes invalidating caches.
Possibly related, for context: - issue #1510 The shader cache key includes the size of the program struct, so adding any member to it makes every player prepare all shaders again — Separate shader-stamp input causes the same full cache invalidation on changes to Program layout. - issue #1609 A program pick reads and decodes every stage of a new program under the shader lock, even when the same bytes were decoded before — Related shader translation path and same renderer file; distinct per-program decode cost. - issue #1493 Renderer rewrite — Renderer rewrite umbrella; shader-cache stamp work is within its scope. - PR #737 feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing — Open PR for cache storage keyed by translator stamp; benefits from avoiding unnecessary stamp changes. - PR #370 perf(vk): the shader caches' stamp leaves out vk_api.h, so a menu, input or audio change keeps every player's translated programs — Merged PR changed translator stamp inputs to prevent unrelated header changes invalidating caches.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1611
No description provided.