feat(update): keep the shader caches when the translator is the same #78
No reviewers
Labels
No labels
area/build
area/install
area/saves
area/ui
area/updates
ci: approved
good first issue
gpu/amd
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
platform/linux
platform/steam-deck
platform/windows
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/LongshotLauncher!78
Loading…
Reference in a new issue
No description provided.
Delete branch "cgize/LongshotLauncher:feat/keep-shader-caches"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Every update makes the game prepare all its shaders again (about 25 minutes here), even when the new version translates
shaders exactly as the old one did: the update builds into a new folder, whose
cache/starts empty, while the oldfolder's 10 GB of caches would have been valid as they are.
What changed
After an update's build finishes and before Ready shows, the launcher compares the shader translator of the version being
replaced with the new one's and, when they are the same, takes the old build's caches into the new game folder: moved
when both folders are on one drive, copied otherwise.
Core/ShaderCaches.cs(new)Translator(toolsRoot): the stamp Longshot'snx2/translator_files.pymakes of the translator's sources belowtools/gpu/vk(the valuenx2/build.pycompiles in asLONGSHOT_SHADER_STAMP), followed by a hash oftranslator_files.pyitself, so two versions whose recipes differ never compare equal by this one's. Null when the toolshave no such file or lack a source.
Keep(from, to, fromTools, toTools, ...): takes the cache files (the names of Longshot'sscripts/dev_packs.pyCACHESandPLAY_FILES, as its--seeddoes) when the new folder has none and both translators match. On onedrive they are moved: each file gets its name in the new folder by a hard link (
FileLinks.TryHardLink), all of themor none, then loses its old one, so it is instant and needs no room. Where a link cannot be made (another drive) they
are copied and the old folder keeps its own. Either way each file is made as
<name>.partand renamed only once allare there, so a cancelled or failed one leaves nothing half made.
Skipped, with the reason, when: the old build has no translated programs, the new folder has caches of its own, the
translators differ or either version's tools are not on this PC, the game is running, or (another drive only) there is
no room for the copy plus 1 GiB.
App/ViewModels/BuildStepViewModel.cs:Exitedgoes throughKeepCachesfor an update beforeComplete(whichprunes the old version's tools, needed for its stamp). The Finish row shows "Keeping the prepared shaders: N%" while it
copies; Cancel stops it as it stops the build; the details log says what was kept or why not. New builds, rebuilds and
replays are unchanged.
tests/launcher_core/ShaderCacheChecks.cs(the stamp against a hand-computed SHA-1, line ends, anothertranslator, another recipe, missing tools, every refusal, a cancelled copy, the copy, the move, half a move) and
tests/test_launcher_core.py --longshotnow checks the cache names and the stamped folders and files against Longshot'sdev_packs.pyandtranslator_files.py.es.json: the two new texts; the other catalogs show them in English.The move follows the review: after an update the old folder is no longer listed, so its caches have no other user. Both
folders never keep a link to the same file: the game appends to and rewrites these files.
What it cannot know
The game folds more into its stamp at start than the translator's sources (the GPU, its switches, the records' layout).
The launcher cannot see that, so when it differs the game drops the copied files, exactly as it would have dropped them
in place, and prepares as before: never a wrong cache.
How it was tested
Windows 11, .NET SDK 10.0.404.
python -I tests/test_launcher_core.py --longshot <Longshot main @ e090cc48>:launcher core: 1050 checks passed.dotnet build LongshotLauncher.slnx -c Release: 0 warnings, 0 errors.ShaderCaches.Translatorgivesebbf160ce9626041for both nightly-170-6c9edffe andnightly-171-b80a68f1, the same value Longshot's own
translator_files.stamp()prints for them. Those two builds' cachescarry the same 8-byte stamp (
52 92 c2 c8 e9 04 28 c1) and theirlongshot_programs.binare the same size to the byte(7,790,801,803), which is the case this is for.
cache/into an empty folder, 6 files, 10,177,506,912 bytes in 32.7 s(296 MB/s, NVMe), sizes and hashes equal. On the same PC nightly-171 prepared from nothing for about 25 minutes
(20:34 to 20:59 by its files' times).
Not tested:
It needs the next nightly whose translator matches.
tests/test_release.pydoes not pass on this Windows PC (6 of 28: it starts a shell-script stub and compares textwritten with CRLF); this change does not touch
scripts/release.py.Thanks, this fixes the case players hit most (25 minutes of shader preparation after an update whose translator is unchanged), and checking the stamp against
translator_files.pyis the right gate.One suggestion: move the files rather than copy them, when both folders are on the same drive. After an update the old folder is no longer listed and the launcher can't start it (
BuildRegistry.Replace; the README: "The old folder stays on the disk; nothing deletes it yet"), so its caches have no other user. A rename per file is instant, needs no second 10 GB, and makes the "no room for a copy" refusal unnecessary. Only a new folder on another drive would still need the copy, and your.part-then-rename scheme carries over to it as it is.Where this goes next, on the Longshot side: I'm starting on a store for these caches that every game folder shares, keyed by the game's own stamps, as
LONGSHOT_OBJ_STOREandLONGSHOT_LIFT_STOREare for the build (MagicBeanDev/Longshot#371, MagicBeanDev/Longshot#372). The program cache's stamp is the translator plus the cache format, and the pipeline cache adds the GPU and driver. Builds with the same stamps would use the caches in place, with no copy or move: an update, stable and nightly side by side,dev_packs.pyruntimes. It needs:longshot_programs.binand drops the pipeline files, so a single shared folder would have versions wiping each other;cache/;The maintainers are fine with it moving the caches out of the game folder, which is where #321 put them (MagicBeanDev/Longshot#321). It ships off by default first, then the launcher sets it for the builds it makes, and this step can go then. That will take a while to land and be tested, so I'd merge this one meanwhile, with the move.
One more thing I missed above: #57 (Victor, WIP) builds an update in its build's own folder instead of a new one. With it, the update's
cache/simply stays where it is, along withlongshot_options.ini,modsand the driver's per-program settings, and this step would have nothing to copy or move. It's worth agreeing between the two which goes in: #57 covers this case and more, and this PR (with the move) only helps while updates still go to a new folder.0d2965d7d66ffdb721d7Thanks for the review! Done in
6ffdb72, and rebased on main (11dcb67).FileLinks), all of them or none, then the old names go. No second 10 GB and no room check.launcher core: 1050 checks passed; the move and the copy are both covered inShaderCacheChecks.cs.On #57: agreed, an update that stays in its folder makes this step unnecessary, and so will the shared store. I'm happy for this to be a stopgap until either lands, or to close it if #57 goes in first; your call and Victor's. Still not tested in the game.
Review notes (read from the source; no build or game run):
LauncherSettings.PrepareShaders), so make_game starts the new game at the end of the build to prepare its shaders (nx2/make_game.py,prepare_shadersafter the build's times). That writes the new folder'scache/beforeKeepCachesruns, soKeepreturnsHasCachesafter the 25 minutes are already spent. The copy has to happen before the prepare, which needs make_game's help (for example a variable naming the folder to seed the caches from, asdev_packs --seeddoes).Stop()cancels and goes home, andKeepCachesthen returns withoutComplete(). The new version is built on disk, but it never takes the old row's place and its folder is left orphaned. Cancel should stop the copy and still complete the build.File.Movefails, the files already moved stay and the remaining.partfiles are deleted (for exampleprograms.binwithout its.idx). That contradicts "nothing half made".Translator()hashes the translator's sources andtranslator_files.py, but not what compiles them: clang (llvm-mingw/ the Linux LLVM), the Linux sysroot's libstdc++, thecpmSPIR-V headers, orbuild.py's flags. On Windows the game'stranslator_stamp()catches a different LLVM version through_LIBCPP_VERSION. On Linux (libstdc++, the3u, 0ubranch) it catches nothing, while the records keepShader::Infoexactly as laid out in memory. Until now, caches never crossed a toolchain, because each version had its own tools and folder. Suggest comparing thetools.locksha256 of the compiler parts as well.Also: if #57 lands (updates stay in their folder), the caches stay where they are and this has almost nothing left to do. It may be better rescoped to seeding a new build of another channel.
🤖 Review drafted with Claude Code
My review above was written against
0d2965d7, before your6ffdb72(moving instead of copying on one drive). I re-checked it against6ffdb721: all four points still hold. Fixes are on HumanBean's fork asreview/pr78(ab8a34a9), three commits on6ffdb721. Take what you like:BuildStepViewModel.Runnow callsShaderCaches.Keepbefore make_game starts; the post-buildKeepCachesandExited(keep)are gone.seed_pipelineswritescache/longshot_pipelines.binwhenever the tools ship a pipeline list, so even with "Prepare shaders now" off the old step could give up withHasCaches.cache/and keeps it (check_writableusesexist_ok). Onlyseed_pipelinesreplaceslongshot_pipelines.bin, and only when its stamp differs from the shipped list's.ShaderCaches.Undoputs the caches back if the build fails, fails to start or is cancelled. Moved files go back under their old names; copies are deleted. That also fixes point 2: nothing runs after the build any more, so cancel can't lose a finished build.ShaderCaches.Room).longshot_programs.idx.ShaderCaches.Compilerhashes the SHA-256 of each version's toolchain parts from itstoolchain.json(Toolchains.PartShas):llvm-mingw.emsdk(clang),sysroot(libstdc++) andcpm.test_launcher_core.py --longshotchecks those folders against tools.lock.build.py's flags aren't hashed: they're built at run time, and hashing the whole file would make nearly every pair of versions differ. Windowscpm(the SPIR-V headers) is a one-line addition toWindowsCompilerif you want it.None of it has been compiled or run here; the tests were updated by reading them.
As dyst said, if #57 lands the caches simply stay in place for an update. This would then mainly matter for an update moved to another folder.
🤖 Drafted with Claude Code
A second, independent read of the fix branch found one regression in it, now fixed.
review/pr78is at07feb749:longshot_pipelines.binwas moved too. make_game'sseed_pipelinesoverwrites that file in place with the shipped list whenever its stamp differs, and the stamp depends on the GPU, so it does for most players. When the build then failed or was cancelled,Undomoved the overwritten file back, so the old build lost its own pipeline notes. Before this PR the old folder was never touched. The notes file is now copied, not moved, so the old one stays. Tests and README updated.Worth a look, not changed:
MakeGameRunkills the process tree but waits only for the root, andUndoruns right after. A child (the game's prepare) can still hold a cache file,File.Moveback fails silently, and the old build loses that file. A short retry inUndowould close the gap.cpm. On Windows,build.py'sheader_packagetakes the Vulkan and SPIR-V headers fromRX_CPMtoo. It's a one-line addition toWindowsCompiler.Checked and fine: make_game keeps a target holding only
cache/, the toolchain.json keys match tools.lock's dests, the translator stamp matchestranslator_files.py,Undoruns at most once, and placement is all or nothing. Still not compiled or run.🤖 Drafted with Claude Code
6ffdb721d7bf3d65a524WIP: feat(update): keep the shader caches when the translator is the sameto feat(update): keep the shader caches when the translator is the same