fix(build): the translator stamp hashes a C or C++ file's tokens, so a comment edit no longer re-prepares every shader #1689

Merged
HumanBean merged 2 commits from enum/Longshot:fix/stamp-normal-form into main 2026-11-19 06:22:30 +00:00
Contributor

Type: fix · Risk: low · Tested on: Windows 11, RTX 4070; Linux (Bazzite, game-free checks) · Needs: none

TL;DR

Every edit to the shader translator's source, even a comment, made every player prepare all shaders again at their next start: up to 5 minutes on a fast desktop and over 4 on a Steam Deck. This PR makes the stamp that decides it read the code's tokens instead of its bytes, so comments, blank lines and spacing no longer count. Anything it is not sure of still counts. A new test fails on main and passes here. In the game, a build with one comment added kept its caches here, where main with the same comment threw them away and prepared 38,809 programs again.

Breakdown

  • Who it helps: every player, at each update whose translator changes are only comments or formatting, and most on slow PCs and the Steam Deck.
  • What was wrong: stamp() hashed the translator's files byte for byte, comments included, so a comment edit in vk_shader.cpp (112 commits since 6 Nov) moved LONGSHOT_SHADER_STAMP.
  • What changes: a C or C++ file is stamped by its tokens, as the compiler's phase 3 leaves them, so comments, blank lines, trailing white space and indentation keep the stamp. Every other file, and any text the scanner is unsure of, stays byte for byte.
  • What does not change: the stamp's other inputs (GPU facts, switches, kProgramCacheFormat, #1510's abi[]), the cache files, which files are stamped, and the renderer's code.
  • How it was checked: a game-free test (14 cases fail on main), 8 mutations of the fix each caught, clang's lexer agreeing with the normal form on all 243 non-empty stamped files, the stamp's 22 checks and the whole suite against main, and the in-game keep-or-drop pair.
  • What is still open: scripts/dev_packs.py still re-prepares a dev runtime by path (see Not tested).
  • Risk: low. The worst case is a scanner mistake that keeps the stamp over a real change, and the test, the mutations and the clang comparison are there against that.

Why

The shader caches (longshot_programs.bin, the pipeline caches, the prepare record) are dropped whole when LONGSHOT_SHADER_STAMP changes, and the stamp is a hash of the translator's source. A commit that only edits a comment there costs every player a full preparation at the next start, though the compiled translator is the same. Fixes #1611 (items 1 and 2 of its proposal, plus its step 3 done game-free with clang's lexer in place of a SPIR-V comparison; item 4, region markers, is left for later as the issue says). See #1510 and #1542 for the stamp's other half (sizeof(Program)), which this does not touch.

How to reproduce

Seen on: any OS, any GPU (the stamp is computed at build time)
Settings and switches: defaults
Steps: build the game, start it once so it prepares, add one comment line to gpu/vk/vk_shader.cpp, build the renderer again over the same game folder, start it.
Expected: the caches are kept.
Actual: the log says pipeline cache: 2 files made for another translator dropped (1093 MB), 0 program keys kept, then prepared 38809 shader programs ... in 41.9 s (here, on an RTX 4070; minutes on slower PCs).

What changed

nx2/translator_files.py's stamp() hashes normal(rel, data) of each file instead of its bytes. For a C or C++ file (C_FAMILY: .c .cc .cpp .cxx .h .hh .hpp .inc .inl), normal() returns its tokens after a tag. Any other file comes back as it is.

  • build: normal() and _tokens() in nx2/translator_files.py. A scanner over bytes reads literals (with escapes and the u8/u/U/L prefixes), pp-numbers with digit separators, identifiers and punctuation, and keeps them as they are. A comment becomes a space. A run of white space becomes one space, or one line break when it holds one, since a line break ends a directive. A block comment over lines with tokens on both sides becomes a separator of its own (_SPLIT), because whether a # after it starts a directive is the compiler's choice. Line splices are kept as they are.
  • build: _Unsure makes a file byte for byte when the scanner cannot be sure: a raw string, a line break in a literal, a // comment continued by a splice, a splice inside a block comment or beside / or *, a backslash followed by blanks and then a line end (a splice to GCC and clang, which only warn), or a header name in angle brackets holding a comment marker or a quote. On the current tree no file is unsure.
  • build: NORMAL_TAG goes before a normal form in the hash, so a normal form is never taken for a file's bytes.
  • tests: tests/test_translator_stamp_text.py is new. tests/test_bake_skeleton.py's own copy of the stamp hash (the appended-order trap) takes normal() too.

What this keeps by design: __LINE__ moves when blank lines go. In the stamped files it only feeds ASSERT's message, INSERT_PADDING's member names and ANONYMOUS_VARIABLE's variable names (scope_exit.h), so a translation cannot change with it.

Measurements

Verdict: a comment edit in the translator keeps the shader caches now; on main it drops them. The new test fails on main (14 cases) and passes here.

Hardware: Windows 11 Pro (build 26300), NVIDIA GeForce RTX 4070 (616.56), Ryzen 9 7900X, 64 GB; llvm-mingw clang
Scene: the game's first 300 frames from a cold start (title), 1280x720, defaults
Method: the test's output, both ways. In the game, one shared base (310e7c9df) with four renderer packs: A = this PR, B = this PR + one comment line in vk_shader.cpp, M = main 2f1c0c9c, M′ = main + the same comment. A and M were each started once (both kept 47,867 keys). B was then started on a copy of A's cache/, and M′ on a copy of M's, and the cache lines of each log were read: a yes or no, not a timing.
How to reproduce: python -I -B tests/test_translator_stamp_text.py; the pairs as above with scripts/dev_build.py pack / rt.

Before: main @ 83b161e3 After: this PR @ 52f213f1 (the test: 53e46345 on 09cbdd97)
tests/test_translator_stamp_text.py FAIL: 14 cases (every kept edit moved the stamp) ok
one comment added to vk_shader.cpp, started on the caches of the build without it 2 files made for another translator dropped (1093 MB), 0 program keys kept, prepared 38809 shader programs in 41.9 s (main 2f1c0c9c) 47867 program keys kept, nothing prepared
Before (main + comment, on main's caches):
[hos MainThread] vk: pipeline cache: 2 files made for another translator dropped (1093 MB); it starts empty
[hos MainThread] vk: program cache ...longshot_programs.bin: 0 program keys kept, 0 KB
[hos MainThread] vk: prepared 38809 shader programs (46667 new, 0 failed, 3656 compute pipelines; 9927 compute variants, 831 programs with texture variants, 59 at the limit) in 41.9 s
After (this PR + comment, on this PR's caches):
[hos MainThread] vk: pipeline cache: 1 play files merged (2389 KB)
[hos MainThread] vk: program cache ...longshot_programs.bin: 47867 program keys kept, 7617817 KB

$ python -I -B tests/test_translator_stamp_text.py        (main @ 83b161e3)
FAIL: 14 case(s)
  a comment line added to vk_shader.cpp: the stamp moved (b183c6d3ec72e5ec, affb2f5ae1114655)
  a comment's words changed in vk_shader.cpp: the stamp moved (b183c6d3ec72e5ec, 187d77a7fc4e06c6)
  ...
$ python -I -B tests/test_translator_stamp_text.py        (this PR @ 53e46345)
ok: the translator stamp keeps comment, blank-line and white-space edits and moves with every other

The in-game pair was built before the rebase onto 83b161e3 (A and B at 331da00f, the same two commits on 1055077d). Since rebased again onto 09cbdd97 (#1697, now 21d4d106). Neither rebase changed a file of this PR. Review then found a false keep (a backslash and blanks before a line end, which GCC and clang splice), and 53e46345 gives those files up to bytes: the real tree has none, so its stamp is the same at 21d4d106 and 53e46345 (05a03c2d2a9d178e), and the in-game pair still holds.

Flags and settings

None.

Testing

Tested:

  • In the game, Windows 11, RTX 4070 (616.56): the four builds above, each started through the control bridge to frame 300 and quit. The keep-or-drop result is in Measurements.
  • The new test, run both ways (Measurements). Then the fix checked against broken versions of itself, eight mutations, each failing the test: a comment as nothing, a comment over lines as a plain space, no literals, no digit separators, raw strings read, a comment's splice not seen, line breaks as spaces, every file normalised. A ninth, from review: without the blank-splice give-up, the two cases for it (a // comment ended by \ and a block comment ended early by *\ and / on the next line) fail, the stamp stayed. The cases for the raw string and the splice-continued comment (a raw string with quotes and // on one line, and code after a comment continued by a splice) were added because the first run showed those two mutations passing.
  • clang's own lexer (clang++ -Xclang -dump-raw-tokens, llvm-mingw) over every stamped C/C++ file and its normal form: the same tokens with the same separators (none, space, comment over lines, line break) in all 243 non-empty files. sirit/src/stream.cpp is empty. The comparison does fail on a broken normaliser: with line breaks read as spaces, all 244 files differ.
  • The 22 checks that read the stamp or translator_files (test_bake_skeleton, test_dev_packs, test_shader_stamp_*, test_program_cache_format, nx2/test_version_long_paths, ...): the same 3 failures as main (test_dev_packs, test_linux_pressure, test_nonuniform_image_arrays), the rest pass.
  • The whole suite, Windows, run_all.py --jobs 4: this PR 721 passed, 54 failed, 49 skipped; main 721 passed, 53 failed, 49 skipped. The same 53 failures, plus test_gpu_hang_watch on this PR. That one fails on main too when run alone (3 of 3, a stuck timeline did not end the process after the limit (exit 3, 3.6 s)) while game runs share the GPU: a timing check, not this change.
  • Linux (Bazzite, kernel 7.3, Python standard library only), rebased onto main 09cbdd97, and again at 53e46345: tests/test_translator_stamp_text.py says ok (main with the test: FAIL: 14 case(s)), run_all.py translator_files stamp bake_skeleton 8 passed. Whole suite: this PR (21d4d106) 652 passed, 41 failed, 133 skipped; main 651 passed, 41 failed, 133 skipped. The same 41 failures, and the extra pass is the new test. After the review fix (53e46345): 655 passed, 38 failed, 133 skipped, no failure main does not have (test_bug_report, test_crash_groups and test_hardware_report passed this time; none of them reads the stamp). The skips are GPU checks without Vulkan headers here.

Not tested:

  • scripts/dev_packs.py still decides that a dev runtime prepares again by which paths changed (stamped()), not by the stamp. A comment edit still re-prepares a development pack, though no player's build does. Comparing stamps there is a follow-up.

Must be tested:
None.

Risks

What could break: a scanner mistake that maps two different translators to one stamp would let a player keep caches from another translator. Against it: the give-ups, the hard cases in the test, the mutations, and clang's lexer agreeing on every real file. A file the scanner cannot read gets the old byte-for-byte hash.

Turning it off: revert the commit.

Compatibility: the stamp moves once with this change, so every player prepares once at the first start after it, as after any translator change (ship it with a release that re-prepares anyway, as #1611 says). Saves, longshot_options.ini, MKWVK_API_VERSION and the cache formats don't change. The bake tools (nx2/bake.py, bake_mimo.py, bake_ds/bake_all.py) stamp their banks with the same stamp(), so a bank baked before this change no longer matches and is baked again.

Issues

Fixes #1611
See #1510, #1542

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** fix · **Risk:** low · **Tested on:** Windows 11, RTX 4070; Linux (Bazzite, game-free checks) · **Needs:** none ## TL;DR Every edit to the shader translator's source, even a comment, made every player prepare all shaders again at their next start: up to 5 minutes on a fast desktop and over 4 on a Steam Deck. This PR makes the stamp that decides it read the code's tokens instead of its bytes, so comments, blank lines and spacing no longer count. Anything it is not sure of still counts. A new test fails on main and passes here. In the game, a build with one comment added kept its caches here, where main with the same comment threw them away and prepared 38,809 programs again. ## Breakdown - **Who it helps:** every player, at each update whose translator changes are only comments or formatting, and most on slow PCs and the Steam Deck. - **What was wrong:** `stamp()` hashed the translator's files byte for byte, comments included, so a comment edit in `vk_shader.cpp` (112 commits since 6 Nov) moved `LONGSHOT_SHADER_STAMP`. - **What changes:** a C or C++ file is stamped by its tokens, as the compiler's phase 3 leaves them, so comments, blank lines, trailing white space and indentation keep the stamp. Every other file, and any text the scanner is unsure of, stays byte for byte. - **What does not change:** the stamp's other inputs (GPU facts, switches, `kProgramCacheFormat`, #1510's `abi[]`), the cache files, which files are stamped, and the renderer's code. - **How it was checked:** a game-free test (14 cases fail on main), 8 mutations of the fix each caught, clang's lexer agreeing with the normal form on all 243 non-empty stamped files, the stamp's 22 checks and the whole suite against main, and the in-game keep-or-drop pair. - **What is still open:** `scripts/dev_packs.py` still re-prepares a dev runtime by path (see Not tested). - **Risk:** low. The worst case is a scanner mistake that keeps the stamp over a real change, and the test, the mutations and the clang comparison are there against that. ## Why The shader caches (`longshot_programs.bin`, the pipeline caches, the prepare record) are dropped whole when `LONGSHOT_SHADER_STAMP` changes, and the stamp is a hash of the translator's source. A commit that only edits a comment there costs every player a full preparation at the next start, though the compiled translator is the same. Fixes #1611 (items 1 and 2 of its proposal, plus its step 3 done game-free with clang's lexer in place of a SPIR-V comparison; item 4, region markers, is left for later as the issue says). See #1510 and #1542 for the stamp's other half (`sizeof(Program)`), which this does not touch. ## How to reproduce **Seen on:** any OS, any GPU (the stamp is computed at build time) **Settings and switches:** defaults **Steps:** build the game, start it once so it prepares, add one comment line to `gpu/vk/vk_shader.cpp`, build the renderer again over the same game folder, start it. **Expected:** the caches are kept. **Actual:** the log says `pipeline cache: 2 files made for another translator dropped (1093 MB)`, `0 program keys kept`, then `prepared 38809 shader programs ... in 41.9 s` (here, on an RTX 4070; minutes on slower PCs). ## What changed `nx2/translator_files.py`'s `stamp()` hashes `normal(rel, data)` of each file instead of its bytes. For a C or C++ file (`C_FAMILY`: `.c .cc .cpp .cxx .h .hh .hpp .inc .inl`), `normal()` returns its tokens after a tag. Any other file comes back as it is. - build: `normal()` and `_tokens()` in `nx2/translator_files.py`. A scanner over bytes reads literals (with escapes and the `u8`/`u`/`U`/`L` prefixes), pp-numbers with digit separators, identifiers and punctuation, and keeps them as they are. A comment becomes a space. A run of white space becomes one space, or one line break when it holds one, since a line break ends a directive. A block comment over lines with tokens on both sides becomes a separator of its own (`_SPLIT`), because whether a `#` after it starts a directive is the compiler's choice. Line splices are kept as they are. - build: `_Unsure` makes a file byte for byte when the scanner cannot be sure: a raw string, a line break in a literal, a `//` comment continued by a splice, a splice inside a block comment or beside `/` or `*`, a backslash followed by blanks and then a line end (a splice to GCC and clang, which only warn), or a header name in angle brackets holding a comment marker or a quote. On the current tree no file is unsure. - build: `NORMAL_TAG` goes before a normal form in the hash, so a normal form is never taken for a file's bytes. - tests: `tests/test_translator_stamp_text.py` is new. `tests/test_bake_skeleton.py`'s own copy of the stamp hash (the appended-order trap) takes `normal()` too. What this keeps by design: `__LINE__` moves when blank lines go. In the stamped files it only feeds `ASSERT`'s message, `INSERT_PADDING`'s member names and `ANONYMOUS_VARIABLE`'s variable names (`scope_exit.h`), so a translation cannot change with it. ## Measurements **Verdict:** a comment edit in the translator keeps the shader caches now; on main it drops them. The new test fails on main (14 cases) and passes here. Hardware: Windows 11 Pro (build 26300), NVIDIA GeForce RTX 4070 (616.56), Ryzen 9 7900X, 64 GB; llvm-mingw clang Scene: the game's first 300 frames from a cold start (title), 1280x720, defaults Method: the test's output, both ways. In the game, one shared base (`310e7c9df`) with four renderer packs: A = this PR, B = this PR + one comment line in `vk_shader.cpp`, M = main `2f1c0c9c`, M′ = main + the same comment. A and M were each started once (both kept 47,867 keys). B was then started on a copy of A's `cache/`, and M′ on a copy of M's, and the cache lines of each log were read: a yes or no, not a timing. How to reproduce: `python -I -B tests/test_translator_stamp_text.py`; the pairs as above with `scripts/dev_build.py pack` / `rt`. | | Before: main @ 83b161e3 | After: this PR @ 52f213f1 (the test: 53e46345 on 09cbdd97) | |---|---|---| | `tests/test_translator_stamp_text.py` | FAIL: 14 cases (every kept edit moved the stamp) | ok | | one comment added to `vk_shader.cpp`, started on the caches of the build without it | `2 files made for another translator dropped (1093 MB)`, `0 program keys kept`, `prepared 38809 shader programs` in 41.9 s (main `2f1c0c9c`) | `47867 program keys kept`, nothing prepared | ``` Before (main + comment, on main's caches): [hos MainThread] vk: pipeline cache: 2 files made for another translator dropped (1093 MB); it starts empty [hos MainThread] vk: program cache ...longshot_programs.bin: 0 program keys kept, 0 KB [hos MainThread] vk: prepared 38809 shader programs (46667 new, 0 failed, 3656 compute pipelines; 9927 compute variants, 831 programs with texture variants, 59 at the limit) in 41.9 s After (this PR + comment, on this PR's caches): [hos MainThread] vk: pipeline cache: 1 play files merged (2389 KB) [hos MainThread] vk: program cache ...longshot_programs.bin: 47867 program keys kept, 7617817 KB $ python -I -B tests/test_translator_stamp_text.py (main @ 83b161e3) FAIL: 14 case(s) a comment line added to vk_shader.cpp: the stamp moved (b183c6d3ec72e5ec, affb2f5ae1114655) a comment's words changed in vk_shader.cpp: the stamp moved (b183c6d3ec72e5ec, 187d77a7fc4e06c6) ... $ python -I -B tests/test_translator_stamp_text.py (this PR @ 53e46345) ok: the translator stamp keeps comment, blank-line and white-space edits and moves with every other ``` The in-game pair was built before the rebase onto `83b161e3` (A and B at `331da00f`, the same two commits on `1055077d`). Since rebased again onto `09cbdd97` (#1697, now `21d4d106`). Neither rebase changed a file of this PR. Review then found a false keep (a backslash and blanks before a line end, which GCC and clang splice), and `53e46345` gives those files up to bytes: the real tree has none, so its stamp is the same at `21d4d106` and `53e46345` (`05a03c2d2a9d178e`), and the in-game pair still holds. ### Flags and settings None. ## Testing **Tested:** - In the game, Windows 11, RTX 4070 (616.56): the four builds above, each started through the control bridge to frame 300 and quit. The keep-or-drop result is in Measurements. - The new test, run both ways (Measurements). Then the fix checked against broken versions of itself, eight mutations, each failing the test: a comment as nothing, a comment over lines as a plain space, no literals, no digit separators, raw strings read, a comment's splice not seen, line breaks as spaces, every file normalised. A ninth, from review: without the blank-splice give-up, the two cases for it (a `//` comment ended by `\ ` and a block comment ended early by `*\ ` and `/` on the next line) fail, `the stamp stayed`. The cases for the raw string and the splice-continued comment (a raw string with quotes and `//` on one line, and code after a comment continued by a splice) were added because the first run showed those two mutations passing. - clang's own lexer (`clang++ -Xclang -dump-raw-tokens`, llvm-mingw) over every stamped C/C++ file and its normal form: the same tokens with the same separators (none, space, comment over lines, line break) in all 243 non-empty files. `sirit/src/stream.cpp` is empty. The comparison does fail on a broken normaliser: with line breaks read as spaces, all 244 files differ. - The 22 checks that read the stamp or `translator_files` (`test_bake_skeleton`, `test_dev_packs`, `test_shader_stamp_*`, `test_program_cache_format`, `nx2/test_version_long_paths`, ...): the same 3 failures as main (`test_dev_packs`, `test_linux_pressure`, `test_nonuniform_image_arrays`), the rest pass. - The whole suite, Windows, `run_all.py --jobs 4`: this PR 721 passed, 54 failed, 49 skipped; main 721 passed, 53 failed, 49 skipped. The same 53 failures, plus `test_gpu_hang_watch` on this PR. That one fails on main too when run alone (3 of 3, `a stuck timeline did not end the process after the limit (exit 3, 3.6 s)`) while game runs share the GPU: a timing check, not this change. - Linux (Bazzite, kernel 7.3, Python standard library only), rebased onto main `09cbdd97`, and again at `53e46345`: `tests/test_translator_stamp_text.py` says `ok` (main with the test: `FAIL: 14 case(s)`), `run_all.py translator_files stamp bake_skeleton` 8 passed. Whole suite: this PR (`21d4d106`) 652 passed, 41 failed, 133 skipped; main 651 passed, 41 failed, 133 skipped. The same 41 failures, and the extra pass is the new test. After the review fix (`53e46345`): 655 passed, 38 failed, 133 skipped, no failure main does not have (`test_bug_report`, `test_crash_groups` and `test_hardware_report` passed this time; none of them reads the stamp). The skips are GPU checks without Vulkan headers here. **Not tested:** - `scripts/dev_packs.py` still decides that a dev runtime prepares again by which paths changed (`stamped()`), not by the stamp. A comment edit still re-prepares a development pack, though no player's build does. Comparing stamps there is a follow-up. **Must be tested:** None. ## Risks **What could break:** a scanner mistake that maps two different translators to one stamp would let a player keep caches from another translator. Against it: the give-ups, the hard cases in the test, the mutations, and clang's lexer agreeing on every real file. A file the scanner cannot read gets the old byte-for-byte hash. **Turning it off:** revert the commit. **Compatibility:** the stamp moves once with this change, so every player prepares once at the first start after it, as after any translator change (ship it with a release that re-prepares anyway, as #1611 says). Saves, `longshot_options.ini`, `MKWVK_API_VERSION` and the cache formats don't change. The bake tools (`nx2/bake.py`, `bake_mimo.py`, `bake_ds/bake_all.py`) stamp their banks with the same `stamp()`, so a bank baked before this change no longer matches and is baked again. ## Issues Fixes #1611 See #1510, #1542 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Author
Contributor

Tested in game: Windows 11, RTX 4070 (NVIDIA 616.56): main + one comment in vk_shader.cpp, started on main's prepared caches, dropped them (2 files made for another translator, 0 program keys kept, 38809 programs prepared again in 41.9 s); this PR + the same comment, on this PR's caches, kept 47867 program keys and prepared nothing. Cold start to frame 300, title screen.

Tested in game: Windows 11, RTX 4070 (NVIDIA 616.56): main + one comment in vk_shader.cpp, started on main's prepared caches, dropped them (2 files made for another translator, 0 program keys kept, 38809 programs prepared again in 41.9 s); this PR + the same comment, on this PR's caches, kept 47867 program keys and prepared nothing. Cold start to frame 300, title screen.
Member

Possibly related, for context:

  • PR #737 feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing — Open shader-cache store also keyed by the translator stamp
Possibly related, for context: - PR #737 feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing — Open shader-cache store also keyed by the translator stamp
enum force-pushed fix/stamp-normal-form from 52f213f1c4
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 3s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to 21d4d1066a
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 5s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
2026-11-19 02:30:56 +00:00
Compare
Author
Contributor

Rebased onto main 09cbdd97 (#1697), no conflicts and no file of this PR changed (head now 21d4d106). Ran the Linux game-free checks: test_translator_stamp_text ok (main with the test: FAIL, 14 cases), the stamp group 8 of 8 passing. Whole suite on Linux: this PR 652 passed / 41 failed / 133 skipped, main 651 / 41 / 133, the same 41 failures, and the extra pass is the new test. Testing updated, and nothing is left under Must be tested.

Rebased onto main 09cbdd97 (#1697), no conflicts and no file of this PR changed (head now 21d4d106). Ran the Linux game-free checks: test_translator_stamp_text ok (main with the test: FAIL, 14 cases), the stamp group 8 of 8 passing. Whole suite on Linux: this PR 652 passed / 41 failed / 133 skipped, main 651 / 41 / 133, the same 41 failures, and the extra pass is the new test. Testing updated, and nothing is left under Must be tested.
enum changed title from WIP: fix(build): the translator stamp hashes a C or C++ file's tokens, so a comment edit no longer re-prepares every shader to fix(build): the translator stamp hashes a C or C++ file's tokens, so a comment edit no longer re-prepares every shader 2026-11-19 02:34:56 +00:00
enum left a comment

Code review of 21d4d106 (read, not run). This is a comment because Forgejo does not let a PR's author approve it or request changes on it, and I reviewed as the author's own account. One point blocks it, and the fix is small.

Blocking: a \ followed by blanks and then a line end is a line splice in GCC and clang, and the scanner does not see it

_tokens() treats only \ followed directly by a line end as a splice: data[j - 1] == 0x5C for a // comment (line 112), b"\\\n" everywhere else. GCC and clang also splice a backslash followed by spaces or tabs and then a line end (clang only warns, -Wbackslash-newline-escape). Checked with g++ on Linux:

int f() {
    int x = 0;
    // note \ ⏎        <- a backslash, then a space, then the line end
    x = 1;
    return x;
}

g++ -E comments out x = 1;. The scanner ends the comment at the line end and keeps x = 1; as code, so this file has the same normal form as one whose comment is plain // note. Adding \ at the end of a comment then removes a line of code from the compiled translator without moving the stamp. That is the false keep that Risks rules out ("a doubt can only make the stamp move more often, never less"). The same backslash and blanks can also end a block comment early (*\ ⏎/ is */ to the compiler but not to the scanner's find(b"*/")).

Nothing in the current tree has a backslash followed by blanks at a line end, so the fix doesn't move today's stamp.

Fix: a file that holds a backslash, blanks and a line end is unsure, next to _ANGLED's check at line 78:

if re.search(rb"\\[ \t\f\v]+\n", data):   # (a splice with blanks before its line end: GCC and clang splice it)
    raise _Unsure()

A test that fails without the fix: the existing case "the line after a comment that ends in a splice" passes with this bug, because both of its sides keep int b = … as code. The pair that catches it is:

moved(tmp, "a comment that ends in a backslash and a blank (GCC and clang splice it)",
      *file_pair(k, b"int a; // note\nint b = 1;\n", b"int a; // note \\ \nint b = 1;\n"))

Not blocking

  • What this keeps by design: besides ASSERT and INSERT_PADDING, __LINE__ also feeds ANONYMOUS_VARIABLE in compat/common/scope_exit.h, which names a variable only. It is harmless, but that paragraph should list it.
  • The other give-ups, checked by reading them: a raw string, a line break in a literal, splices next to / or *, and comment markers in an angled header name all go to bytes. Form feed, vertical tab and a lone CR are kept as tokens, so they can only move the stamp. _SPLIT cannot be confused with real tokens, because /* in the normal form appears only inside a literal, which keeps its quotes.
  • test_bake_skeleton.py: the trap check now imports normal() rather than keeping its own copy. That is right, since the check is about file order, not about the text.
  • The test's side effects: it writes only to a temp folder of its own and removes it. Nothing reaches outside the repository.

Overlaps

#737 (one shader cache store, keyed by this stamp) touches scripts/dev_packs.py but not translator_files.py. They don't conflict, and #737 gains from this PR.

Testing

Tested on Windows in the game and on Linux game-free (see the PR's Testing). After the fix, the new test case and tests/run_all.py translator_files stamp bake_skeleton need to be run again, and the clang comparison over the real tree should still agree, since no file there has a backslash followed by blanks.

Code review of 21d4d106 (read, not run). This is a comment because Forgejo does not let a PR's author approve it or request changes on it, and I reviewed as the author's own account. **One point blocks it**, and the fix is small. ## Blocking: a `\` followed by blanks and then a line end is a line splice in GCC and clang, and the scanner does not see it `_tokens()` treats only `\` followed directly by a line end as a splice: `data[j - 1] == 0x5C` for a `//` comment (line 112), `b"\\\n"` everywhere else. GCC and clang also splice a backslash followed by spaces or tabs and then a line end (clang only warns, `-Wbackslash-newline-escape`). Checked with g++ on Linux: ``` int f() { int x = 0; // note \ ⏎ <- a backslash, then a space, then the line end x = 1; return x; } ``` `g++ -E` comments out `x = 1;`. The scanner ends the comment at the line end and keeps `x = 1;` as code, so this file has the same normal form as one whose comment is plain `// note`. Adding `\ ` at the end of a comment then removes a line of code from the compiled translator without moving the stamp. That is the false keep that Risks rules out ("a doubt can only make the stamp move more often, never less"). The same backslash and blanks can also end a block comment early (`*\ ⏎/` is `*/` to the compiler but not to the scanner's `find(b"*/")`). Nothing in the current tree has a backslash followed by blanks at a line end, so the fix doesn't move today's stamp. **Fix:** a file that holds a backslash, blanks and a line end is unsure, next to `_ANGLED`'s check at line 78: ```python if re.search(rb"\\[ \t\f\v]+\n", data): # (a splice with blanks before its line end: GCC and clang splice it) raise _Unsure() ``` **A test that fails without the fix:** the existing case "the line after a comment that ends in a splice" passes with this bug, because both of its sides keep `int b = …` as code. The pair that catches it is: ```python moved(tmp, "a comment that ends in a backslash and a blank (GCC and clang splice it)", *file_pair(k, b"int a; // note\nint b = 1;\n", b"int a; // note \\ \nint b = 1;\n")) ``` ## Not blocking - **What this keeps by design:** besides `ASSERT` and `INSERT_PADDING`, `__LINE__` also feeds `ANONYMOUS_VARIABLE` in `compat/common/scope_exit.h`, which names a variable only. It is harmless, but that paragraph should list it. - **The other give-ups**, checked by reading them: a raw string, a line break in a literal, splices next to `/` or `*`, and comment markers in an angled header name all go to bytes. Form feed, vertical tab and a lone CR are kept as tokens, so they can only move the stamp. `_SPLIT` cannot be confused with real tokens, because `/*` in the normal form appears only inside a literal, which keeps its quotes. - **`test_bake_skeleton.py`:** the trap check now imports `normal()` rather than keeping its own copy. That is right, since the check is about file order, not about the text. - **The test's side effects:** it writes only to a temp folder of its own and removes it. Nothing reaches outside the repository. ## Overlaps https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/737 (one shader cache store, keyed by this stamp) touches `scripts/dev_packs.py` but not `translator_files.py`. They don't conflict, and #737 gains from this PR. ## Testing Tested on Windows in the game and on Linux game-free (see the PR's Testing). After the fix, the new test case and `tests/run_all.py translator_files stamp bake_skeleton` need to be run again, and the clang comparison over the real tree should still agree, since no file there has a backslash followed by blanks.
@ -29,0 +109,4 @@
elif data.startswith(b"//", i):
j = data.find(b"\n", i)
j = n if j < 0 else j
if data[j - 1] == 0x5C: # (the comment goes on over the splice)
Author
Contributor

Only a backslash right before the line end is seen here. GCC and clang also splice a backslash, then spaces or tabs, then the line end (g++ -E checked), so // note \ comments out the next line and the stamp stays. See the review.

Only a backslash right before the line end is seen here. GCC and clang also splice a backslash, then spaces or tabs, then the line end (g++ -E checked), so `// note \ ` comments out the next line and the stamp stays. See the review.
enum force-pushed fix/stamp-normal-form from 21d4d1066a
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 5s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to 53e46345cb
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
2026-11-19 02:54:35 +00:00
Compare
Author
Contributor

Review fix pushed (53e46345, the branch rewritten so it still reads test, then fix): a backslash followed by blanks and then a line end is a splice to GCC and clang (g++ -E checked), and the scanner read it as a line end, so // note \ above a line of code kept the stamp while the compiler commented that line out. A file holding one is now unsure (byte for byte), and the test has two new cases for it: a // comment and a block comment ended early by it. Both fail with the fix as it was (the stamp stayed) and pass now. The real tree has no such line, so its stamp is the same as at 21d4d106 (05a03c2d2a9d178e), and the in-game pair and the clang comparison still hold. Linux: the stamp group 8 passed; the whole suite 655 passed, 38 failed, 133 skipped, with no failure main lacks. The LINE paragraph now names ANONYMOUS_VARIABLE too.

Review fix pushed (53e46345, the branch rewritten so it still reads test, then fix): a backslash followed by blanks and then a line end is a splice to GCC and clang (g++ -E checked), and the scanner read it as a line end, so `// note \ ` above a line of code kept the stamp while the compiler commented that line out. A file holding one is now unsure (byte for byte), and the test has two new cases for it: a `//` comment and a block comment ended early by it. Both fail with the fix as it was (the stamp stayed) and pass now. The real tree has no such line, so its stamp is the same as at 21d4d106 (05a03c2d2a9d178e), and the in-game pair and the clang comparison still hold. Linux: the stamp group 8 passed; the whole suite 655 passed, 38 failed, 133 skipped, with no failure main lacks. The __LINE__ paragraph now names ANONYMOUS_VARIABLE too.
HumanBean deleted branch fix/stamp-normal-form 2026-11-19 06:22:31 +00:00
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
Some workflows are waiting to be reviewed.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1689
No description provided.