perf(vk): a warm writes the pipeline cache every ten minutes, so a preparation stopped early keeps what it built #1940
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1940
Loading…
Reference in a new issue
No description provided.
Delete branch "madebyjordan/Longshot:perf/warm-saves-as-it-goes"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
What was wrong: the warm-ups that build the noted pipelines wrote the driver's pipeline cache (
longshot_vk_pipelines.bin) once, after their last build. On a cold driver cache that's over an hour of compiling on AMD: 71 minutes in nightly-622's preparation on my RX 6700 XT. A preparation killed, crashed, or stopped before its end kept none of it, and the next one compiled it all again. Stops includemake_game.py's silence timeout, the player closing it, and a driver fault.What this changes:
warm_save_tickwrites the pipeline cache every ten minutes while a warm builds, in the background (pipeline_cache_save(false): its own thread, the builds go on, nothing written when nothing is new). It's called from the progress ticks of the three loops that build for many minutes:precompile_warm's noted and compute pipelines, and the start's noted pipelines inprecompile_run. The forced save at a warm's end and the.builtmarker after it are as they were, so a warm cut short runs again at the next start, now finding what it had built in the cache.Read first:
gpu/vk/vk_draw.cppwarm_save_dueandwarm_save_tick, and their three callers.Related: #1926 and #1927 make an update keep the pipeline cache across a format or translator change; this keeps what a cold warm builds when it doesn't get to its end.
Testing
Tested: Windows 11 Home 26200, AMD Radeon RX 6700 XT, Ryzen 5 5600.
dev.py rt, which runslongshot.exe --prepare-shaders --compile-shaders most). The base's caches belong to another translator, sopipeline cache: 1 files made for another translator dropped (1770 MB); it starts empty.pipeline cache savedlines, against the progress line before each:precompile: 37837 noted pipelines ... (0 and 0 not made ...).tests/test_warm_saves_as_it_goes.py.warm_save_dueover a made-up clock: the first tick sets the next save; saves fall at 10 and 20 minutes; a tick that comes late (33 minutes) saves once and counts on from there.warm_save_ticksaves in the background, not forced.warm_save_due. It passes here with clang++ and g++.precompile_warm,precompile_runandprepare_progress_log: 29 pass.test_linux_pressureandtest_prepare_cache_savefail the same way on main on this PC.Still to test:
pipeline cache ... (N KB)read with N from the last save and the remaining warm shorter (by hand)pipeline cache savedline every ten minutes and the warm's time as main's (@Beanstalk)Measurements
Verdict: nothing to speed up when a preparation finishes: the warm's time is the same, within the noise of one cold run each. 4,034 s here against 71 minutes (4,260 s) in nightly-622's preparation on this PC, a different folder and run. The gain is what a preparation keeps when it doesn't finish: up to the last ten minutes' work instead of nothing (1.6 GB of pipelines at 60 minutes here).
a104182b97cfdf10Risks
PcacheFetch), as play's saves do. It may be put off, never forced.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Tested in game: Windows 11, RX 6700 XT: a cold preparation of a dev runtime of this branch (dev.py rt: --prepare-shaders --compile-shaders most, the base's cache dropped as another translator's): 'pipeline cache saved' at 592 s (452 MB), 1195, 1796, 2398, 3001, 3603 s (1.6 GB), then the end's forced save at 4034 s (1.83 GB); 37837 noted pipelines built, 0 not made; the warm 67 min, as main's 71 in nightly-622's preparation.
Possibly related, for context:
Recommendation: Request changes. The AMD run shows that periodic checkpoints can be written, but a checkpoint can fail while the cache is growing and leave the warm with no saved progress. The added startup cost and the benefit on restart still need a matched comparison.
Long shader preparations and first starts can leave a cache for the next launch to reuse. Reusing the existing writer keeps the patch small and preserves atomic replacement, but that writer needs to handle a cache that grows while it takes a snapshot. The reported AMD run wrote about 6.1 GB across six checkpoints, plus 1.83 GB at completion.
Does it help: Unproven for interrupted preparation. The PR and first comment report six completed checkpoints on Windows with an RX 6700 XT, reaching about 1.6 GB at 3,603 s. A stop at 3,600 s would have left the 1.264 GB checkpoint reported at 3,001 s in that run. No stop-and-restart result shows how much compilation was avoided. The 4,034 s head run and 4,260 s nightly-622 run came from different builds and folders, with one run each. They establish neither a speedup nor the cost of checkpointing.
Other systems: The Builder and normal startup both use the new ticks. The preparation compute workers call cached_compute directly, so Intel's retry between jobs on the serial queue is not guaranteed to run during preparation. The writer also shares save state with the final forced save and in-play saves. Windows NVIDIA, Linux NVIDIA and RADV, Steam Deck, and Intel have no supplied checkpoint measurements. Measure cold and warm startup, peak RAM, temporary disk headroom, and first-frame pacing if a save carries into play. The diff does not change replay ordering, saves, audio, or input.
Switches: None added, changed, or removed. The ten-minute interval is fixed. The existing cache-load policy still suppresses writes where the startup cache is disabled.
Reviewed:
97cfdf1051againsta104182b40: Read the full diff and both commits, the full new Python test and srccut.py, AGENTS.md, CONTRIBUTING.md, the Forgejo template, and docs/dev/standards.md. Read the changed functions and the relevant warm worker, startup and close callers, cache counter and save path, file publication, and Intel fetch policy and serial queue. Read the supplied test report and first comment. No earlier reviews, review checklist, or local Tooling checkout were present. Unrelated vk_draw.cpp sections and other source-reading tests were outside the full read. No translations or generated files changed.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Suggestions (the author may take or leave)
fix(vk)if the intended result is recovery after interruption. The supplied measurements do not establish a faster completed warm.Testing: Read-only review. No builds, tests, or project scripts ran. The author reports Clang and GCC passes for the clock harness, two caught timing mutations, 29 related passes, and matching main failures in test_linux_pressure and test_prepare_cache_save. The source assertions can pass even when the real save never publishes. Full test, lint, and head CI results were not supplied. The in-game opening requirement is supported by the first comment, but the interrupted restart and remaining Tier 1 results are pending.
Overlaps and merge order: #1926 and #1927 are described as preserving pipeline caches across updates. This PR checkpoints during a warm. No merge-order dependency is established by the supplied material.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -10317,6 +10317,26 @@ void prepare_background() {/// (longshot) A line every 10 s while the preparation builds pipelines: their count and an estimate of the time left. The/// pipeline steps wrote nothing to the log for 10-30 minutes on some AMD cards, and players pasted a log whose last lines (the/// vertex formats) looked like a hang (Discord 2026-11-10, "stuck on preparing shaders"); the Builder's bar moved, the log did not/// (longshot) The pipeline cache written while a warm builds, every kWarmSaveEvery, not only at its end. A preparation on a coldMove the progress-log comment back beside prepare_progress_log so each comment sits with the code it describes.
@ -10320,0 +10334,4 @@return true;}void warm_save_tick(std::chrono::steady_clock::time_point& next) {if (warm_save_due(std::chrono::steady_clock::now(), next)) pipeline_cache_save(false);A compile between the size query and data fetch can make the exact-size snapshot return VK_INCOMPLETE. The attempt consumes the dirty count and the deadline moves ten minutes. Please add bounded retry for growth and preserve the pending checkpoint on failure.
@ -10458,6 +10479,7 @@ void precompile_warm(void (*progress)(size_t done, size_t total)) {}, [] { return prepare_stopped(); }, [&](size_t) {if (progress) progress(done.load(), all_n);prepare_progress_log("noted pipelines", done.load(), total, t0, log_at);warm_save_tick(save_at);These checkpoints wrote about 6.1 GB before the final save on AMD. Please measure the interrupted restart benefit and the full-warm time, peak RAM and temporary disk cost against the actual base, including Steam Deck.
@ -10482,6 +10504,7 @@ void precompile_warm(void (*progress)(size_t done, size_t total)) {}, [] { return prepare_stopped(); }, [&](size_t) {if (progress) progress(total + computes_done.load(), all_n);prepare_progress_log("compute pipelines", computes_done.load(), computes.size(), c0, log_at);warm_save_tick(save_at);These compute workers call cached_compute directly. If Intel's fetch times out, the owed save is picked up between jobs on a different queue that can be idle throughout preparation. Please give the warm its own bounded retry.
@ -0,0 +70,4 @@bad.append("%s does not write the cache as it goes (%s)" % (marker.split("(")[0], need.split("(")[0]))if "bool warm_save_due(" not in draw:return bad + ["vk_draw.cpp has no warm_save_due: the cache is written at the warm's end only"]tick = block(draw, "void warm_save_tick(std::chrono::steady_clock::time_point& next)")This can pass when the save call is unreachable or when every snapshot fails to publish. Keep the clock test and exercise the checkpoint behavior with a stub cache that can grow between the two fetches.
WIP: perf(vk): a warm writes the pipeline cache every ten minutes, so a preparation stopped early keeps what it builtto perf(vk): a warm writes the pipeline cache every ten minutes, so a preparation stopped early keeps what it built