fix(vk): a program cache format change keeps the programs, the driver's pipelines and the notes #1926
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1926
Loading…
Reference in a new issue
No description provided.
Delete branch "madebyjordan/Longshot:fix/pipelines-outlive-cache-format"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
What was wrong: after updating to nightly-622, players with full shader caches saw the preparation start again and crawl. On my PC (RX 6700 XT) the launcher's preparation took 83 minutes (18:30 to 19:53, after a 30-minute build). People on Discord report the same.
The cause, my own #1809:
kProgramCacheFormatfrom 1 to 3. That's a change to howlongshot_programs.binlays out a record; the SPIR-V it holds is identical.program_cache_stamp(), which mixes the format into the translator's stamp.longshot_vk_pipelines.bin, 1.88 GB here);longshot_pipelines.bin, truncated to its head);.builtrecord;--compile-shaders mostpath, where the cost is, wasn't measured. That's my mistake.How to see it on main: take a folder prepared by nightly-618 or earlier and update it to nightly-622. The first start logs
vk: pipeline cache: N files made for another translator dropped, although the translator's stamped files (nx2/translator_files.py) are byte-identical between nightly-612 and nightly-622. The preparation then builds every pipeline cold.What this changes:
pipelines_stamp_ok(), which takes this translator's stamp under the current program cache format or any earlier one. What these files hold (pipelines, program keys and variants, module identifiers) depends on the SPIR-V, not on the program file's layout.longshot_prepared.binstill start over with the format: the programs are translated again (about 2 minutes), and the preparation finds every pipeline in the driver's cache. (An earlier head also adopted format-1 program files as they were. #1448 has since moved the translator's stamp, so no format-1 file can be of a current translator any more, and that code would never run: it's gone. BeanBot's review and fix for it are answered by its removal.)The result: a future format bump costs a translation pass (about 2 minutes here) and a warm against a full driver cache, instead of a cold compile of every pipeline. Players who already started 622 have lost theirs; this can't bring them back. The next nightly also moves the translator's stamp (#1448): keeping the caches across that is #1927 with LongshotLauncher #220.
Read first:
gpu/vk/vk_progcache.cpp:program_cache_stamp_of,pipelines_stamp_ok, and the header comment's rule: a format bump can no longer drop the pipeline side, so a change to the SPIR-V, the keys or the variants belongs in the translator's stamp.gpu/vk/pcache_stamp.h(check'salso) andgpu/vk/module_ids.h(open'salso).gpu/vk/vk_draw.cpp: every reader of a pipeline-side head.See #1510 (mooki's proposal to take
sizeof(Program)out of the same stamp, the other way it drops every cache needlessly) and #1809.Testing
Tested: Windows 11 Home 26200, AMD Radeon RX 6700 XT, Ryzen 5 5600.
In the game, on the launcher's path (
longshot.exe --prepare-shaders --compile-shaders most, asmake_game.pyruns it). This was a dev runtime of the stacked branch #1927, which contains this one, with a test-only commit raisingkProgramCacheFormatto 4: a format bump on one translator.cache/was seeded with a copy of my prepared nightly-622 folder's caches: format 3, 1.88 GB driver cache, 1 GB notes.files made for another translator droppedline;pipeline notes ...: stamped as the notes' own;program cache ...: 0 program keys kept(translated again, as intended).prepared 38809 shader programs ... in 125.3 sandprecompile: 38024 noted pipelines ... built into the pipeline cache ... (0 and 0 not made ...) in 6.7 s, withthe pipeline cache not written: nothing new(every pipeline a hit).New check, its own commit:
tests/test_pipelines_outlive_cache_format.py, with clang++ and g++ from llvm-mingw.pcache_drop_stale()andnotes_open_locked(), cut fromvk_draw.cpp, withprogram_cache_stamp()cut fromvk_progcache.cpp, as builds of one translator at format 1, the current format minus one, and the current one, plus a build of another translator.pipelines_stamp_ok.a104182b), it fails: format 1 → 3 givespipeline cache dropped (-1 bytes), 0 notes kept, notes file 16 bytes, and all six other readers are still on the exact stamp.pipeline cache kept (3000 bytes), 3 notes kept, notes file 400 bytes, and another translator is stilldropped.Broken variants of the fix, each caught:
pipe_notes_refsleft on the exact stamp.Checks that read the changed source:
test_pcache_stamp(pluscheck'salso),test_module_ids(plusopen'salso, appended to as it is),test_program_cache_format(plus the rule for earlier formats),test_predicted_check(now "another translator"), and stand-ins intest_note_refs_once,test_precompile_warm_computesandtest_state_list_part_record.program_cache_stamp_of:test_dev_packsandtest_native_float_atomic_add. The latter was run with the cpm headers; Windows skips it by default.The whole suite (
tests/run_all.py --strict, whatmise run testruns; Windows, llvm-mingw), on this head and on main87c0d7b4: 709 passed, 143 failed, 48 skipped here; 708, 143, 48 on main. The same 143 fail and the same 48 skip on both (tools this PC lacks, such as the Vulkan headers and spirv-val, and Windows-only differences); the one more pass is this PR's check. Rebased onto87c0d7b4(#1448) with no conflict.Compiled whole:
vk_progcache.cpp,vk_programs.cppandvk_draw.cppwith-fsyntax-only -Wall -Wextraand the build's include folders. They give the same 132 warnings and the same 4 errors as main (the errors come from target flags my command lacks).Cold review by a second agent (code only). It found nothing unsafe:
NvnStateandPipeNoteare unchanged;program_cache_stamp()user is handled.It caught
test_dev_packs,test_native_float_atomic_addand the untested readers, all fixed here.Still to test:
kProgramCacheFormatraised by one: expecting nofiles made for another translator droppedline and a preparation of minutes, not a cold compile (@Beanstalk)Measurements
Verdict: after a format bump the launcher's preparation takes about 2¼ minutes instead of 83: the driver cache and the notes are kept, so only the programs are translated again.
Setup: Windows 11, RX 6700 XT, Ryzen 5 5600; the launcher's preparation (
--prepare-shaders --compile-shaders most) over a folder prepared by nightly-622, AMD's own driver cache cold (a new folder path). One run each, since the gap is 37×.fb14ba0b(nightly-622, the format bump from 1 to 3)744a2329(run as #1927's06028484, which contains it, plus a test commit raising the format from 3 to 4)Risks
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Tested in game: Windows 11, RX 6700 XT: the launcher's preparation (--prepare-shaders --compile-shaders most) of a dev runtime of the stacked translator branch (contains this one) with kProgramCacheFormat raised to 4, over a copy of my prepared nightly-622 caches, AMD's own driver cache cold: no 'dropped' line, notes kept and restamped, 38809 programs translated in 125.3 s, 38024 noted pipelines from the cache in 6.7 s: about 2 1/4 minutes against 83 for nightly-622.
Possibly related, for context:
Recommendation: Request changes. Format-1 adoption remains enabled after any future program-cache format bump, even when the new format changes the record layout. The actual adoption and restamping path also lacks a behavior test or an in-game run.
The change keeps pipeline-side caches across program-cache format changes and adopts compatible format-1 program files. That could avoid long shader preparation, but unconditional future adoption risks reading incompatible records, and the main migration path has not been demonstrated end to end.
Does it help: Unproven for the affected format-1 upgrade. The author reports about 2¼ minutes versus 83 minutes on an RX 6700 XT, but those are single runs of different format transitions and the faster run used a stacked branch. It did not exercise format-1 adoption.
Other systems: The change affects pipeline caches, notes, predictions and module identifiers on Windows and Linux across GPU vendors. The reported run covers Windows AMD; Linux, NVIDIA, Steam Deck and an actual format-1 upgrade remain unverified. The module-identifier reader still checks the driver UUID.
Switches: None added, changed or removed.
Reviewed:
37715c273dagainsta104182b40: Read AGENTS.md, CONTRIBUTING.md, the PR template, standards, commits and the production diff. Read the changed cache headers and relevant loader, stamp and notes code; skimmed the changed tests and the remaining large production files. Tooling's AGENTS.md was unavailable beside the checkout, and docs/review-checklist.md was absent. Not all changed code files were read in full.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Suggestions (the author may take or leave)
Testing: No code, tests or project scripts were run, as requested. The author's reported checks and hardware run were reviewed as evidence.
Overlaps and merge order: The description mentions #1510, #1809 and a stacked #NEXT; their current status and merge order could not be verified offline.
Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.
@ -82,0 +96,4 @@FileHead program_file_head(bool whole, uint32_t magic, uint32_t want_magic, uint32_t store, uint64_t kept, bool zstd) {if (!whole || magic != want_magic) return FileHead::Over;if (kept == program_cache_stamp()) return FileHead::Take;if (!zstd && !store && kept == program_cache_stamp_of(1)) return FileHead::Adopt;This also adopts format 1 after any future format bump. If format 4 changes the record layout without changing the translator, a direct format-1 upgrade will restamp incompatible records as format 4. Please limit adoption to explicitly compatible destination formats and require a decision at the next bump.
@ -0,0 +117,4 @@return 2;}"""This exercises the header decision but not the migration itself. Please test a representative format-1 file through disk_init, including loading its inline stages, restamping, appending and reopening, so the check catches a broken adoption path.
WIP: fix(vk): a program cache format change keeps the programs, the driver's pipelines and the notesto fix(vk): a program cache format change keeps the programs, the driver's pipelines and the notes@fix
I changed this pull request as the review asked: the new head is
f5d014207c.Changed
Nothing was built or run: check the change. It will be reviewed again.
Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.
Recommendation: Request changes. Keeping compatible caches addresses the cause of the long preparation, but the Builder can still overwrite compatible notes and several game-free checks break at this head. The reported timing used a different format transition on a stacked build, and current main changes the translator stamp this migration requires.
The renderer keeps pipeline caches, notes, prediction files and module identifiers across program-cache format bumps when the translator stamp matches. It also adopts inline format-1 program records into format 3. This can save translation and cold pipeline compilation, at the cost of keeping larger local cache files and maintaining an explicit migration path.
Does it help: Unproven for the release upgrade. The author reports 83 minutes on nightly-622 for 1→3 and about 2¼ minutes on a stacked #1927 dev build for 3→4 on the same Windows RX 6700 XT. The faster run translated 38,809 programs in 125.3 seconds and warmed 38,024 noted pipelines in 6.7 seconds. Each side ran once. The comparison does not exercise format-1 adoption or identify the gain on this head against its base. Current main also changes the translator stamp.
Other systems: The change reaches pipeline load and warm-up, load-at-create, prediction and module ID lookup on every Vulkan vendor and both OSes. The Vulkan cache still checks the vendor, device and cache UUID, and module IDs still check the algorithm UUID. The Builder's optional seed copy can replace old notes before the renderer checks them. Adoption keeps the old inline SPIR-V on disk even with the shared store enabled. Linux, RADV, NVIDIA and Steam Deck runtime coverage remains pending. The diff leaves replay ordering, saves, audio and input code untouched.
Switches: None added, changed or removed. The migration fixture disables the existing shader store for its test processes.
Reviewed:
f5d014207cagainsta104182b40: Read AGENTS.md, CONTRIBUTING.md, the Forgejo PR template, developer standards, the relevant replay rules and test and measurement guidance. Read all commits and changed hunks, both cache headers in full, the new test in full, and the affected runtime readers, writers, stamp inputs, Builder seed path and source-cut harnesses. Read affected sections of vk_draw.cpp, vk_progcache.cpp, vk_programs.cpp and vk_shader.h and skimmed their other sections and the unchanged portions of the existing changed tests. Reviewed the supplied comments and stale review. No docs/review-checklist.md or sibling Tooling checkout was available.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Waiting on the author (a run, a measurement or a decision; no edit does it)
Testing: Read-only review. No builds, tests or project scripts were run. The author's reported cache-retention checks and Windows AMD timing predate the migration fixture changes at
f5d014207c.Overlaps and merge order: #1809 introduced the format bump. #1927 is stacked on this work according to the PR body. Land this prerequisite first if the changes remain separate, and measure the combined release path against current main. #1448 is already on local main and changes the translator stamp. #1940 is also on local main and touches the pipeline warm/save path. The status of #1510 was unavailable offline.
Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.
@ -1073,3 +1075,3 @@static const bool dropped = [] {const std::string dir = cache_path(""); // (where pcache_init keeps them: the game folder's cache/, vk_paths.h)const pcache_stamp::Dropped gone = pcache_stamp::check(dir, program_cache_stamp());const pcache_stamp::Dropped gone = pcache_stamp::check(dir, program_cache_stamp(), pipelines_stamp_ok);The pipeline-play-cache fixture extracts this call. Update its two-argument check stub and supply pipelines_stamp_ok so the generated C++ compiles.
@ -1864,2 +1866,3 @@}const bool same = head[0] == stamp && head[1] == sizeof(PipeNote);// (a head of this translator under an earlier program cache format is the same: its notes are about the pipelines, kept)const bool same = pipelines_stamp_ok(head[0]) && head[1] == sizeof(PipeNote);With a shipped seed, nx2/make_game.py:1557 can overwrite the player's compatible format-1 notes before this runs. Preserve them through the Builder path too.
@ -11290,3 +11293,3 @@if (FILE* f = fopen(predicted_keys_path().c_str(), "rb")) {uint64_t head[2]{};if (fread(head, 8, 2, f) == 2 && head[0] == program_cache_stamp() && head[1] <= (64u << 20) &&if (fread(head, 8, 2, f) == 2 && pipelines_stamp_ok(head[0]) && head[1] <= (64u << 20) &&The Builder cache-save fixture extracts this reader and has no pipelines_stamp_ok. Update the fixture and keep its stale and malformed record cases.
@ -279,6 +323,19 @@ void disk_init() {}fclose(f);}if (good && adopted) {The adopted file keeps all inline SPIR-V even with the shared store on. Measure the per-folder and shared-store sizes after an actual format-1 migration.
@ -0,0 +64,4 @@va_end(a);std::printf("\n");}static void cut_part_record(const std::string&, size_t, size_t) {}The extracted notes opener calls _fseeki64. Map it to fseeko in this harness on Linux.
@ -0,0 +413,4 @@("another translator: dropped", new, other, "drop"),("another translator and format 1: dropped", old, other, "drop"),]failed += ["the program file's head: " + r for r in check_heads(args.cxx, tmp, progcache_src, now)]On base, check_migration raises on the missing FileHead marker before the pipeline cases run. Make the base test reach the behavioral assertions.
f5d014207c744a2329f0@HumanBean thanks for sending BeanBot's fix. I've replaced its commit (
f5d01420) rather than kept it: the adoption it guarded is gone from this PR. #1448 (merged) moved the translator's stamp, so no format-1 program file can belong to a current translator any more, and the adoption path could never run. Its guard and test had nothing left to guard. BeanBot's migration test also didn't compile here (it redefined format 1's Shader::Info by hand, which the translator stamp already pins). Both blocking points are answered by the removal. What's left: a format bump keeps the driver's pipelines and the notes (pipelines_stamp_ok), with the in-game numbers in Testing. Rebased onto87c0d7b4(head744a2329). The full suite (run_all --strict) fails the same 143 checks as main on this PC, plus one more passing. The more urgent half is #1927 with LongshotLauncher #220: the next nightly moves the translator (#1448), and without them every player's update compiles every pipeline cold again.