fix(build): the translator stamp hashes a C or C++ file's tokens, so a comment edit no longer re-prepares every shader #1689
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1689
Loading…
Reference in a new issue
No description provided.
Delete branch "enum/Longshot:fix/stamp-normal-form"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Windows 11, RTX 4070; Linux (Bazzite, game-free checks) · Needs: none
TL;DR
Every edit to the shader translator's source, even a comment, made every player prepare all shaders again at their next start: up to 5 minutes on a fast desktop and over 4 on a Steam Deck. This PR makes the stamp that decides it read the code's tokens instead of its bytes, so comments, blank lines and spacing no longer count. Anything it is not sure of still counts. A new test fails on main and passes here. In the game, a build with one comment added kept its caches here, where main with the same comment threw them away and prepared 38,809 programs again.
Breakdown
stamp()hashed the translator's files byte for byte, comments included, so a comment edit invk_shader.cpp(112 commits since 6 Nov) movedLONGSHOT_SHADER_STAMP.kProgramCacheFormat, #1510'sabi[]), the cache files, which files are stamped, and the renderer's code.scripts/dev_packs.pystill re-prepares a dev runtime by path (see Not tested).Why
The shader caches (
longshot_programs.bin, the pipeline caches, the prepare record) are dropped whole whenLONGSHOT_SHADER_STAMPchanges, and the stamp is a hash of the translator's source. A commit that only edits a comment there costs every player a full preparation at the next start, though the compiled translator is the same. Fixes #1611 (items 1 and 2 of its proposal, plus its step 3 done game-free with clang's lexer in place of a SPIR-V comparison; item 4, region markers, is left for later as the issue says). See #1510 and #1542 for the stamp's other half (sizeof(Program)), which this does not touch.How to reproduce
Seen on: any OS, any GPU (the stamp is computed at build time)
Settings and switches: defaults
Steps: build the game, start it once so it prepares, add one comment line to
gpu/vk/vk_shader.cpp, build the renderer again over the same game folder, start it.Expected: the caches are kept.
Actual: the log says
pipeline cache: 2 files made for another translator dropped (1093 MB),0 program keys kept, thenprepared 38809 shader programs ... in 41.9 s(here, on an RTX 4070; minutes on slower PCs).What changed
nx2/translator_files.py'sstamp()hashesnormal(rel, data)of each file instead of its bytes. For a C or C++ file (C_FAMILY:.c .cc .cpp .cxx .h .hh .hpp .inc .inl),normal()returns its tokens after a tag. Any other file comes back as it is.normal()and_tokens()innx2/translator_files.py. A scanner over bytes reads literals (with escapes and theu8/u/U/Lprefixes), pp-numbers with digit separators, identifiers and punctuation, and keeps them as they are. A comment becomes a space. A run of white space becomes one space, or one line break when it holds one, since a line break ends a directive. A block comment over lines with tokens on both sides becomes a separator of its own (_SPLIT), because whether a#after it starts a directive is the compiler's choice. Line splices are kept as they are._Unsuremakes a file byte for byte when the scanner cannot be sure: a raw string, a line break in a literal, a//comment continued by a splice, a splice inside a block comment or beside/or*, a backslash followed by blanks and then a line end (a splice to GCC and clang, which only warn), or a header name in angle brackets holding a comment marker or a quote. On the current tree no file is unsure.NORMAL_TAGgoes before a normal form in the hash, so a normal form is never taken for a file's bytes.tests/test_translator_stamp_text.pyis new.tests/test_bake_skeleton.py's own copy of the stamp hash (the appended-order trap) takesnormal()too.What this keeps by design:
__LINE__moves when blank lines go. In the stamped files it only feedsASSERT's message,INSERT_PADDING's member names andANONYMOUS_VARIABLE's variable names (scope_exit.h), so a translation cannot change with it.Measurements
Verdict: a comment edit in the translator keeps the shader caches now; on main it drops them. The new test fails on main (14 cases) and passes here.
Hardware: Windows 11 Pro (build 26300), NVIDIA GeForce RTX 4070 (616.56), Ryzen 9 7900X, 64 GB; llvm-mingw clang
Scene: the game's first 300 frames from a cold start (title), 1280x720, defaults
Method: the test's output, both ways. In the game, one shared base (
310e7c9df) with four renderer packs: A = this PR, B = this PR + one comment line invk_shader.cpp, M = main2f1c0c9c, M′ = main + the same comment. A and M were each started once (both kept 47,867 keys). B was then started on a copy of A'scache/, and M′ on a copy of M's, and the cache lines of each log were read: a yes or no, not a timing.How to reproduce:
python -I -B tests/test_translator_stamp_text.py; the pairs as above withscripts/dev_build.py pack/rt.83b161e353e46345on09cbdd97)tests/test_translator_stamp_text.pyvk_shader.cpp, started on the caches of the build without it2 files made for another translator dropped (1093 MB),0 program keys kept,prepared 38809 shader programsin 41.9 s (main2f1c0c9c)47867 program keys kept, nothing preparedThe in-game pair was built before the rebase onto
83b161e3(A and B at331da00f, the same two commits on1055077d). Since rebased again onto09cbdd97(#1697, now21d4d106). Neither rebase changed a file of this PR. Review then found a false keep (a backslash and blanks before a line end, which GCC and clang splice), and53e46345gives those files up to bytes: the real tree has none, so its stamp is the same at21d4d106and53e46345(05a03c2d2a9d178e), and the in-game pair still holds.Flags and settings
None.
Testing
Tested:
//comment ended by\and a block comment ended early by*\and/on the next line) fail,the stamp stayed. The cases for the raw string and the splice-continued comment (a raw string with quotes and//on one line, and code after a comment continued by a splice) were added because the first run showed those two mutations passing.clang++ -Xclang -dump-raw-tokens, llvm-mingw) over every stamped C/C++ file and its normal form: the same tokens with the same separators (none, space, comment over lines, line break) in all 243 non-empty files.sirit/src/stream.cppis empty. The comparison does fail on a broken normaliser: with line breaks read as spaces, all 244 files differ.translator_files(test_bake_skeleton,test_dev_packs,test_shader_stamp_*,test_program_cache_format,nx2/test_version_long_paths, ...): the same 3 failures as main (test_dev_packs,test_linux_pressure,test_nonuniform_image_arrays), the rest pass.run_all.py --jobs 4: this PR 721 passed, 54 failed, 49 skipped; main 721 passed, 53 failed, 49 skipped. The same 53 failures, plustest_gpu_hang_watchon this PR. That one fails on main too when run alone (3 of 3,a stuck timeline did not end the process after the limit (exit 3, 3.6 s)) while game runs share the GPU: a timing check, not this change.09cbdd97, and again at53e46345:tests/test_translator_stamp_text.pysaysok(main with the test:FAIL: 14 case(s)),run_all.py translator_files stamp bake_skeleton8 passed. Whole suite: this PR (21d4d106) 652 passed, 41 failed, 133 skipped; main 651 passed, 41 failed, 133 skipped. The same 41 failures, and the extra pass is the new test. After the review fix (53e46345): 655 passed, 38 failed, 133 skipped, no failure main does not have (test_bug_report,test_crash_groupsandtest_hardware_reportpassed this time; none of them reads the stamp). The skips are GPU checks without Vulkan headers here.Not tested:
scripts/dev_packs.pystill decides that a dev runtime prepares again by which paths changed (stamped()), not by the stamp. A comment edit still re-prepares a development pack, though no player's build does. Comparing stamps there is a follow-up.Must be tested:
None.
Risks
What could break: a scanner mistake that maps two different translators to one stamp would let a player keep caches from another translator. Against it: the give-ups, the hard cases in the test, the mutations, and clang's lexer agreeing on every real file. A file the scanner cannot read gets the old byte-for-byte hash.
Turning it off: revert the commit.
Compatibility: the stamp moves once with this change, so every player prepares once at the first start after it, as after any translator change (ship it with a release that re-prepares anyway, as #1611 says). Saves,
longshot_options.ini,MKWVK_API_VERSIONand the cache formats don't change. The bake tools (nx2/bake.py,bake_mimo.py,bake_ds/bake_all.py) stamp their banks with the samestamp(), so a bank baked before this change no longer matches and is baked again.Issues
Fixes #1611
See #1510, #1542
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Tested in game: Windows 11, RTX 4070 (NVIDIA 616.56): main + one comment in vk_shader.cpp, started on main's prepared caches, dropped them (2 files made for another translator, 0 program keys kept, 38809 programs prepared again in 41.9 s); this PR + the same comment, on this PR's caches, kept 47867 program keys and prepared nothing. Cold start to frame 300, title screen.
Possibly related, for context:
52f213f1c421d4d1066aRebased onto main
09cbdd97(#1697), no conflicts and no file of this PR changed (head now 21d4d106). Ran the Linux game-free checks: test_translator_stamp_text ok (main with the test: FAIL, 14 cases), the stamp group 8 of 8 passing. Whole suite on Linux: this PR 652 passed / 41 failed / 133 skipped, main 651 / 41 / 133, the same 41 failures, and the extra pass is the new test. Testing updated, and nothing is left under Must be tested.WIP: fix(build): the translator stamp hashes a C or C++ file's tokens, so a comment edit no longer re-prepares every shaderto fix(build): the translator stamp hashes a C or C++ file's tokens, so a comment edit no longer re-prepares every shaderCode review of 21d4d106 (read, not run). This is a comment because Forgejo does not let a PR's author approve it or request changes on it, and I reviewed as the author's own account. One point blocks it, and the fix is small.
Blocking: a
\followed by blanks and then a line end is a line splice in GCC and clang, and the scanner does not see it_tokens()treats only\followed directly by a line end as a splice:data[j - 1] == 0x5Cfor a//comment (line 112),b"\\\n"everywhere else. GCC and clang also splice a backslash followed by spaces or tabs and then a line end (clang only warns,-Wbackslash-newline-escape). Checked with g++ on Linux:g++ -Ecomments outx = 1;. The scanner ends the comment at the line end and keepsx = 1;as code, so this file has the same normal form as one whose comment is plain// note. Adding\at the end of a comment then removes a line of code from the compiled translator without moving the stamp. That is the false keep that Risks rules out ("a doubt can only make the stamp move more often, never less"). The same backslash and blanks can also end a block comment early (*\ ⏎/is*/to the compiler but not to the scanner'sfind(b"*/")).Nothing in the current tree has a backslash followed by blanks at a line end, so the fix doesn't move today's stamp.
Fix: a file that holds a backslash, blanks and a line end is unsure, next to
_ANGLED's check at line 78:A test that fails without the fix: the existing case "the line after a comment that ends in a splice" passes with this bug, because both of its sides keep
int b = …as code. The pair that catches it is:Not blocking
ASSERTandINSERT_PADDING,__LINE__also feedsANONYMOUS_VARIABLEincompat/common/scope_exit.h, which names a variable only. It is harmless, but that paragraph should list it./or*, and comment markers in an angled header name all go to bytes. Form feed, vertical tab and a lone CR are kept as tokens, so they can only move the stamp._SPLITcannot be confused with real tokens, because/*in the normal form appears only inside a literal, which keeps its quotes.test_bake_skeleton.py: the trap check now importsnormal()rather than keeping its own copy. That is right, since the check is about file order, not about the text.Overlaps
#737 (one shader cache store, keyed by this stamp) touches
scripts/dev_packs.pybut nottranslator_files.py. They don't conflict, and #737 gains from this PR.Testing
Tested on Windows in the game and on Linux game-free (see the PR's Testing). After the fix, the new test case and
tests/run_all.py translator_files stamp bake_skeletonneed to be run again, and the clang comparison over the real tree should still agree, since no file there has a backslash followed by blanks.@ -29,0 +109,4 @@elif data.startswith(b"//", i):j = data.find(b"\n", i)j = n if j < 0 else jif data[j - 1] == 0x5C: # (the comment goes on over the splice)Only a backslash right before the line end is seen here. GCC and clang also splice a backslash, then spaces or tabs, then the line end (g++ -E checked), so
// note \comments out the next line and the stamp stays. See the review.21d4d1066a53e46345cbReview fix pushed (
53e46345, the branch rewritten so it still reads test, then fix): a backslash followed by blanks and then a line end is a splice to GCC and clang (g++ -E checked), and the scanner read it as a line end, so// note \above a line of code kept the stamp while the compiler commented that line out. A file holding one is now unsure (byte for byte), and the test has two new cases for it: a//comment and a block comment ended early by it. Both fail with the fix as it was (the stamp stayed) and pass now. The real tree has no such line, so its stamp is the same as at 21d4d106 (05a03c2d2a9d178e), and the in-game pair and the clang comparison still hold. Linux: the stamp group 8 passed; the whole suite 655 passed, 38 failed, 133 skipped, with no failure main lacks. The LINE paragraph now names ANONYMOUS_VARIABLE too.