perf(vk): a warm writes the pipeline cache every ten minutes, so a preparation stopped early keeps what it built #1940

Merged
HumanBean merged 2 commits from madebyjordan/Longshot:perf/warm-saves-as-it-goes into main 2026-11-21 00:01:27 +00:00
Member

Summary

What was wrong: the warm-ups that build the noted pipelines wrote the driver's pipeline cache (longshot_vk_pipelines.bin) once, after their last build. On a cold driver cache that's over an hour of compiling on AMD: 71 minutes in nightly-622's preparation on my RX 6700 XT. A preparation killed, crashed, or stopped before its end kept none of it, and the next one compiled it all again. Stops include make_game.py's silence timeout, the player closing it, and a driver fault.

What this changes: warm_save_tick writes the pipeline cache every ten minutes while a warm builds, in the background (pipeline_cache_save(false): its own thread, the builds go on, nothing written when nothing is new). It's called from the progress ticks of the three loops that build for many minutes: precompile_warm's noted and compute pipelines, and the start's noted pipelines in precompile_run. The forced save at a warm's end and the .built marker after it are as they were, so a warm cut short runs again at the next start, now finding what it had built in the cache.

Read first: gpu/vk/vk_draw.cpp warm_save_due and warm_save_tick, and their three callers.

Related: #1926 and #1927 make an update keep the pipeline cache across a format or translator change; this keeps what a cold warm builds when it doesn't get to its end.

Testing

Tested: Windows 11 Home 26200, AMD Radeon RX 6700 XT, Ryzen 5 5600.

  • In the game: a dev runtime of this branch prepared cold (dev.py rt, which runs longshot.exe --prepare-shaders --compile-shaders most). The base's caches belong to another translator, so pipeline cache: 1 files made for another translator dropped (1770 MB); it starts empty.
    • The noted pipelines took 4,034 s.
    • The log's pipeline cache saved lines, against the progress line before each:
at noted pipelines built the cache written
592 s 8,304 of 37,474 452 MB
1,195 s 14,464 731 MB
1,796 s 17,794 939 MB
2,398 s 18,799 1,092 MB
3,001 s 20,298 1,264 MB
3,603 s 29,023 1,598 MB
4,034 s (the end's forced save, as before) 37,472 1,833 MB, in 6,050 ms
  • Then precompile: 37837 noted pipelines ... (0 and 0 not made ...).
  • The background saves didn't stall the builds: the progress lines go on between them.
  • New check, its own commit: tests/test_warm_saves_as_it_goes.py.
    • It runs the real warm_save_due over a made-up clock: the first tick sets the next save; saves fall at 10 and 20 minutes; a tick that comes late (33 minutes) saves once and counts on from there.
    • It also checks that the three loops tick it, and that warm_save_tick saves in the background, not forced.
    • It fails on main: none of the three loops saves, and there's no warm_save_due. It passes here with clang++ and g++.
  • Broken variants, each caught: the next save never moved on (a save at every tick); a save one tick late.
  • Checks that read precompile_warm, precompile_run and prepare_progress_log: 29 pass. test_linux_pressure and test_prepare_cache_save fail the same way on main on this PC.

Still to test:

  • windows AMD: stop a cold preparation after 15 minutes, start it again, expecting pipeline cache ... (N KB) read with N from the last save and the remaining warm shorter (by hand)
  • linux and windows nvidia: a cold preparation, expecting a pipeline cache saved line every ten minutes and the warm's time as main's (@Beanstalk)

Measurements

Verdict: nothing to speed up when a preparation finishes: the warm's time is the same, within the noise of one cold run each. 4,034 s here against 71 minutes (4,260 s) in nightly-622's preparation on this PC, a different folder and run. The gain is what a preparation keeps when it doesn't finish: up to the last ten minutes' work instead of nothing (1.6 GB of pipelines at 60 minutes here).

Before: main @ a104182b After: this PR @ 97cfdf10
pipeline cache written during a cold warm once, at its end every 10 minutes (6 times in 67 minutes), then at its end
kept by a preparation stopped at 60 minutes nothing 1.6 GB

Risks

  • Disk writes: a 0.5 to 1.8 GB write every ten minutes during a cold warm, in the background. A warm cache doesn't write (nothing new).
  • Intel: a background save on Intel waits for a gap between the serial compute builds (PcacheFetch), as play's saves do. It may be put off, never forced.
  • To turn it off: a revert. There's no switch.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • A check under tests/ that reads changed source is updated, not weakened.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

🤖 Generated with Claude Code

## Summary **What was wrong:** the warm-ups that build the noted pipelines wrote the driver's pipeline cache (`longshot_vk_pipelines.bin`) once, after their last build. On a cold driver cache that's over an hour of compiling on AMD: 71 minutes in nightly-622's preparation on my RX 6700 XT. A preparation killed, crashed, or stopped before its end kept none of it, and the next one compiled it all again. Stops include `make_game.py`'s silence timeout, the player closing it, and a driver fault. **What this changes:** `warm_save_tick` writes the pipeline cache every ten minutes while a warm builds, in the background (`pipeline_cache_save(false)`: its own thread, the builds go on, nothing written when nothing is new). It's called from the progress ticks of the three loops that build for many minutes: `precompile_warm`'s noted and compute pipelines, and the start's noted pipelines in `precompile_run`. The forced save at a warm's end and the `.built` marker after it are as they were, so a warm cut short runs again at the next start, now finding what it had built in the cache. Read first: `gpu/vk/vk_draw.cpp` `warm_save_due` and `warm_save_tick`, and their three callers. Related: #1926 and #1927 make an update keep the pipeline cache across a format or translator change; this keeps what a cold warm builds when it doesn't get to its end. ## Testing **Tested:** Windows 11 Home 26200, AMD Radeon RX 6700 XT, Ryzen 5 5600. - **In the game:** a dev runtime of this branch prepared cold (`dev.py rt`, which runs `longshot.exe --prepare-shaders --compile-shaders most`). The base's caches belong to another translator, so `pipeline cache: 1 files made for another translator dropped (1770 MB); it starts empty`. - The noted pipelines took 4,034 s. - The log's `pipeline cache saved` lines, against the progress line before each: | at | noted pipelines built | the cache written | |---|---|---| | 592 s | 8,304 of 37,474 | 452 MB | | 1,195 s | 14,464 | 731 MB | | 1,796 s | 17,794 | 939 MB | | 2,398 s | 18,799 | 1,092 MB | | 3,001 s | 20,298 | 1,264 MB | | 3,603 s | 29,023 | 1,598 MB | | 4,034 s (the end's forced save, as before) | 37,472 | 1,833 MB, in 6,050 ms | - Then `precompile: 37837 noted pipelines ... (0 and 0 not made ...)`. - The background saves didn't stall the builds: the progress lines go on between them. - **New check, its own commit:** `tests/test_warm_saves_as_it_goes.py`. - It runs the real `warm_save_due` over a made-up clock: the first tick sets the next save; saves fall at 10 and 20 minutes; a tick that comes late (33 minutes) saves once and counts on from there. - It also checks that the three loops tick it, and that `warm_save_tick` saves in the background, not forced. - **It fails on main:** none of the three loops saves, and there's no `warm_save_due`. It passes here with clang++ and g++. - **Broken variants, each caught:** the next save never moved on (a save at every tick); a save one tick late. - **Checks that read `precompile_warm`, `precompile_run` and `prepare_progress_log`:** 29 pass. `test_linux_pressure` and `test_prepare_cache_save` fail the same way on main on this PC. **Still to test:** - windows AMD: stop a cold preparation after 15 minutes, start it again, expecting `pipeline cache ... (N KB)` read with N from the last save and the remaining warm shorter (by hand) - linux and windows nvidia: a cold preparation, expecting a `pipeline cache saved` line every ten minutes and the warm's time as main's (@Beanstalk) ## Measurements **Verdict:** nothing to speed up when a preparation finishes: the warm's time is the same, within the noise of one cold run each. 4,034 s here against 71 minutes (4,260 s) in nightly-622's preparation on this PC, a different folder and run. The gain is what a preparation keeps when it doesn't finish: up to the last ten minutes' work instead of nothing (1.6 GB of pipelines at 60 minutes here). | | Before: main @ a104182b | After: this PR @ 97cfdf10 | |---|---|---| | pipeline cache written during a cold warm | once, at its end | every 10 minutes (6 times in 67 minutes), then at its end | | kept by a preparation stopped at 60 minutes | nothing | 1.6 GB | ## Risks - **Disk writes:** a 0.5 to 1.8 GB write every ten minutes during a cold warm, in the background. A warm cache doesn't write (nothing new). - **Intel:** a background save on Intel waits for a gap between the serial compute builds (`PcacheFetch`), as play's saves do. It may be put off, never forced. - **To turn it off:** a revert. There's no switch. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [ ] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
On a cold driver cache a preparation is hours of compiles (AMD:
nightly-622's, 80 minutes in with no longshot_vk_pipelines.bin written
yet), and the cache was written once, after the last build: a
preparation killed, crashed or stopped before its end kept nothing.

The check runs vk_draw.cpp's warm_save_due over a made-up clock (a save
every ten minutes, a late tick saving once and counting on from there),
and checks that the three loops that build for many minutes tick it and
that the save runs in the background. It fails on main: there is none.

Assisted-by: Claude Code (Claude Opus 5.5)
perf(vk): a warm writes the pipeline cache every ten minutes, so a preparation stopped early keeps what it built
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 13s
CI / Workflow lint (actionlint) (pull_request) Successful in 14s
Third-party parts / As their notes say (pull_request) Successful in 18s
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Failing after 10s
CI / Website build and checks (pull_request) Failing after 2m34s
97cfdf1051
The pipeline cache was written once, after a warm's last build. On a cold
driver cache a preparation is hours of compiles (AMD: nightly-622's, 80
minutes in with no longshot_vk_pipelines.bin written yet), and one killed,
crashed, or stopped before its end (make_game's silence timeout, the
player closing it) kept none of them: the next one compiled it all again.

warm_save_tick, from the progress ticks of the loops that build for many
minutes (precompile_warm's noted and compute pipelines, the start's noted
pipelines in precompile_run), writes it every ten minutes in the
background (pipeline_cache_save(false): its own thread, the builds go on,
nothing written when nothing new). The warm's own forced save at its end
and the .built marker after it are as they were, so a warm cut short
still runs again at the next start, now finding what it had built in the
cache.

Assisted-by: Claude Code (Claude Opus 5.5)
Author
Member

Tested in game: Windows 11, RX 6700 XT: a cold preparation of a dev runtime of this branch (dev.py rt: --prepare-shaders --compile-shaders most, the base's cache dropped as another translator's): 'pipeline cache saved' at 592 s (452 MB), 1195, 1796, 2398, 3001, 3603 s (1.6 GB), then the end's forced save at 4034 s (1.83 GB); 37837 noted pipelines built, 0 not made; the warm 67 min, as main's 71 in nightly-622's preparation.

Tested in game: Windows 11, RX 6700 XT: a cold preparation of a dev runtime of this branch (dev.py rt: --prepare-shaders --compile-shaders most, the base's cache dropped as another translator's): 'pipeline cache saved' at 592 s (452 MB), 1195, 1796, 2398, 3001, 3603 s (1.6 GB), then the end's forced save at 4034 s (1.83 GB); 37837 noted pipelines built, 0 not made; the warm 67 min, as main's 71 in nightly-622's preparation.
Member

Possibly related, for context:

  • issue #1908 BUG: Vulkan Shader Preparation and Pipeline Cache Generation Extremely Slow on Linux — Open report of very slow Linux Vulkan pipeline preparation
  • PR #903 fix(vk): a start that compiled its noted pipelines writes the pipeline cache before the title — Merged change saves the cache at warm completion; this adds periodic saves
Possibly related, for context: - issue #1908 BUG: Vulkan Shader Preparation and Pipeline Cache Generation Extremely Slow on Linux — Open report of very slow Linux Vulkan pipeline preparation - PR #903 fix(vk): a start that compiled its noted pipelines writes the pipeline cache before the title — Merged change saves the cache at warm completion; this adds periodic saves
BeanBot left a comment

Recommendation: Request changes. The AMD run shows that periodic checkpoints can be written, but a checkpoint can fail while the cache is growing and leave the warm with no saved progress. The added startup cost and the benefit on restart still need a matched comparison.

Long shader preparations and first starts can leave a cache for the next launch to reuse. Reusing the existing writer keeps the patch small and preserves atomic replacement, but that writer needs to handle a cache that grows while it takes a snapshot. The reported AMD run wrote about 6.1 GB across six checkpoints, plus 1.83 GB at completion.

Does it help: Unproven for interrupted preparation. The PR and first comment report six completed checkpoints on Windows with an RX 6700 XT, reaching about 1.6 GB at 3,603 s. A stop at 3,600 s would have left the 1.264 GB checkpoint reported at 3,001 s in that run. No stop-and-restart result shows how much compilation was avoided. The 4,034 s head run and 4,260 s nightly-622 run came from different builds and folders, with one run each. They establish neither a speedup nor the cost of checkpointing.
Other systems: The Builder and normal startup both use the new ticks. The preparation compute workers call cached_compute directly, so Intel's retry between jobs on the serial queue is not guaranteed to run during preparation. The writer also shares save state with the final forced save and in-play saves. Windows NVIDIA, Linux NVIDIA and RADV, Steam Deck, and Intel have no supplied checkpoint measurements. Measure cold and warm startup, peak RAM, temporary disk headroom, and first-frame pacing if a save carries into play. The diff does not change replay ordering, saves, audio, or input.
Switches: None added, changed, or removed. The ten-minute interval is fixed. The existing cache-load policy still suppresses writes where the startup cache is disabled.

Reviewed: 97cfdf1051 against a104182b40: Read the full diff and both commits, the full new Python test and srccut.py, AGENTS.md, CONTRIBUTING.md, the Forgejo template, and docs/dev/standards.md. Read the changed functions and the relevant warm worker, startup and close callers, cache counter and save path, file publication, and Intel fetch policy and serial queue. Read the supplied test report and first comment. No earlier reviews, review checklist, or local Tooling checkout were present. Unrelated vk_draw.cpp sections and other source-reading tests were outside the full read. No translations or generated files changed.; nothing run (a code review).
Checked and fine:

  • One topic. The diff adds no game data, generated output, vendor SDKs, or third-party changes. Both commits carry assistance trailers.
  • The warm callbacks run on the controlling thread, so the local deadline needs no synchronization and carries from graphics into compute. The final forced save and marker paths are preserved.
  • The existing writer publishes through a temporary file and preserves the previous cache when a write fails. It retains the existing Windows UTF-8 path conversion and Linux file handling.
  • The test uses the standard library, the existing source cutter and a local compiler in a temporary directory, as described. Its clock harness checks boundaries and late ticks. The diff adds no network operation.

Blocking (to change before merge)

  1. gpu/vk/vk_draw.cpp:10337: The background writer sizes the cache, creates an exact-size mapping, then fetches the data. If a compiler adds a pipeline in between, the fetch can return VK_INCOMPLETE. The writer drops the attempt and consumes the dirty count, while the warm has already moved its deadline ten minutes out. On a busy warm this can recur and a killed preparation can still keep nothing. Make the warm checkpoint tolerate growth with a bounded resize and retry and preserve pending work when it cannot publish. Cover a growing cache and a failed publication with a game-free test.
  2. gpu/vk/vk_draw.cpp:10507: The Intel fallback described in Risks relies on pipeline_cache_between_builds, which runs on the serial job queue. This warm calls cached_compute from its own workers. With the Windows Intel fetch lock enabled, a long compile can exhaust the fetch window and leave the checkpoint owed to a queue that has no work during preparation. Give the warm a bounded retry path of its own and test a fetch deferred by an active compile.
  3. gpu/vk/vk_draw.cpp:10482: The unconditional checkpoints rewrite gigabytes, and the report only shows that progress continued. For a Steam Deck or another setup with slower storage or tight memory, mapping and flushing a growing cache can compete with compilation, and publication temporarily needs space for both files. Run at least two interleaved base/head pairs on matched builds and settings, including a stop and restart, and report elapsed time, cache reuse, peak RAM and disk headroom on affected Tier 1 setups. Adjust the default behavior if those costs outweigh the recovery benefit.

Suggestions (the author may take or leave)

  • Use fix(vk) if the intended result is recovery after interruption. The supplied measurements do not establish a faster completed warm.
  • Add the driver version and actual baseline commit to Measurements, and replace the estimated stop-at-60-minutes result with the restart result when available.

Testing: Read-only review. No builds, tests, or project scripts ran. The author reports Clang and GCC passes for the clock harness, two caught timing mutations, 29 related passes, and matching main failures in test_linux_pressure and test_prepare_cache_save. The source assertions can pass even when the real save never publishes. Full test, lint, and head CI results were not supplied. The in-game opening requirement is supported by the first comment, but the interrupted restart and remaining Tier 1 results are pending.
Overlaps and merge order: #1926 and #1927 are described as preserving pipeline caches across updates. This PR checkpoints during a warm. No merge-order dependency is established by the supplied material.


Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The AMD run shows that periodic checkpoints can be written, but a checkpoint can fail while the cache is growing and leave the warm with no saved progress. The added startup cost and the benefit on restart still need a matched comparison. Long shader preparations and first starts can leave a cache for the next launch to reuse. Reusing the existing writer keeps the patch small and preserves atomic replacement, but that writer needs to handle a cache that grows while it takes a snapshot. The reported AMD run wrote about 6.1 GB across six checkpoints, plus 1.83 GB at completion. **Does it help:** Unproven for interrupted preparation. The PR and first comment report six completed checkpoints on Windows with an RX 6700 XT, reaching about 1.6 GB at 3,603 s. A stop at 3,600 s would have left the 1.264 GB checkpoint reported at 3,001 s in that run. No stop-and-restart result shows how much compilation was avoided. The 4,034 s head run and 4,260 s nightly-622 run came from different builds and folders, with one run each. They establish neither a speedup nor the cost of checkpointing. **Other systems:** The Builder and normal startup both use the new ticks. The preparation compute workers call cached_compute directly, so Intel's retry between jobs on the serial queue is not guaranteed to run during preparation. The writer also shares save state with the final forced save and in-play saves. Windows NVIDIA, Linux NVIDIA and RADV, Steam Deck, and Intel have no supplied checkpoint measurements. Measure cold and warm startup, peak RAM, temporary disk headroom, and first-frame pacing if a save carries into play. The diff does not change replay ordering, saves, audio, or input. **Switches:** None added, changed, or removed. The ten-minute interval is fixed. The existing cache-load policy still suppresses writes where the startup cache is disabled. **Reviewed:** 97cfdf105159 against a104182b4037: Read the full diff and both commits, the full new Python test and srccut.py, AGENTS.md, CONTRIBUTING.md, the Forgejo template, and docs/dev/standards.md. Read the changed functions and the relevant warm worker, startup and close callers, cache counter and save path, file publication, and Intel fetch policy and serial queue. Read the supplied test report and first comment. No earlier reviews, review checklist, or local Tooling checkout were present. Unrelated vk_draw.cpp sections and other source-reading tests were outside the full read. No translations or generated files changed.; nothing run (a code review). **Checked and fine:** - One topic. The diff adds no game data, generated output, vendor SDKs, or third-party changes. Both commits carry assistance trailers. - The warm callbacks run on the controlling thread, so the local deadline needs no synchronization and carries from graphics into compute. The final forced save and marker paths are preserved. - The existing writer publishes through a temporary file and preserves the previous cache when a write fails. It retains the existing Windows UTF-8 path conversion and Linux file handling. - The test uses the standard library, the existing source cutter and a local compiler in a temporary directory, as described. Its clock harness checks boundaries and late ticks. The diff adds no network operation. **Blocking** (to change before merge) 1. gpu/vk/vk_draw.cpp:10337: The background writer sizes the cache, creates an exact-size mapping, then fetches the data. If a compiler adds a pipeline in between, the fetch can return VK_INCOMPLETE. The writer drops the attempt and consumes the dirty count, while the warm has already moved its deadline ten minutes out. On a busy warm this can recur and a killed preparation can still keep nothing. Make the warm checkpoint tolerate growth with a bounded resize and retry and preserve pending work when it cannot publish. Cover a growing cache and a failed publication with a game-free test. 2. gpu/vk/vk_draw.cpp:10507: The Intel fallback described in Risks relies on pipeline_cache_between_builds, which runs on the serial job queue. This warm calls cached_compute from its own workers. With the Windows Intel fetch lock enabled, a long compile can exhaust the fetch window and leave the checkpoint owed to a queue that has no work during preparation. Give the warm a bounded retry path of its own and test a fetch deferred by an active compile. 3. gpu/vk/vk_draw.cpp:10482: The unconditional checkpoints rewrite gigabytes, and the report only shows that progress continued. For a Steam Deck or another setup with slower storage or tight memory, mapping and flushing a growing cache can compete with compilation, and publication temporarily needs space for both files. Run at least two interleaved base/head pairs on matched builds and settings, including a stop and restart, and report elapsed time, cache reuse, peak RAM and disk headroom on affected Tier 1 setups. Adjust the default behavior if those costs outweigh the recovery benefit. **Suggestions** (the author may take or leave) - Use `fix(vk)` if the intended result is recovery after interruption. The supplied measurements do not establish a faster completed warm. - Add the driver version and actual baseline commit to Measurements, and replace the estimated stop-at-60-minutes result with the restart result when available. **Testing:** Read-only review. No builds, tests, or project scripts ran. The author reports Clang and GCC passes for the clock harness, two caught timing mutations, 29 related passes, and matching main failures in test_linux_pressure and test_prepare_cache_save. The source assertions can pass even when the real save never publishes. Full test, lint, and head CI results were not supplied. The in-game opening requirement is supported by the first comment, but the interrupted restart and remaining Tier 1 results are pending. **Overlaps and merge order:** #1926 and #1927 are described as preserving pipeline caches across updates. This PR checkpoints during a warm. No merge-order dependency is established by the supplied material. --- <sub>Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -10317,6 +10317,26 @@ void prepare_background() {
/// (longshot) A line every 10 s while the preparation builds pipelines: their count and an estimate of the time left. The
/// pipeline steps wrote nothing to the log for 10-30 minutes on some AMD cards, and players pasted a log whose last lines (the
/// vertex formats) looked like a hang (Discord 2026-11-10, "stuck on preparing shaders"); the Builder's bar moved, the log did not
/// (longshot) The pipeline cache written while a warm builds, every kWarmSaveEvery, not only at its end. A preparation on a cold
Member

Move the progress-log comment back beside prepare_progress_log so each comment sits with the code it describes.

Move the progress-log comment back beside prepare_progress_log so each comment sits with the code it describes.
@ -10320,0 +10334,4 @@
return true;
}
void warm_save_tick(std::chrono::steady_clock::time_point& next) {
if (warm_save_due(std::chrono::steady_clock::now(), next)) pipeline_cache_save(false);
Member

A compile between the size query and data fetch can make the exact-size snapshot return VK_INCOMPLETE. The attempt consumes the dirty count and the deadline moves ten minutes. Please add bounded retry for growth and preserve the pending checkpoint on failure.

A compile between the size query and data fetch can make the exact-size snapshot return VK_INCOMPLETE. The attempt consumes the dirty count and the deadline moves ten minutes. Please add bounded retry for growth and preserve the pending checkpoint on failure.
@ -10458,6 +10479,7 @@ void precompile_warm(void (*progress)(size_t done, size_t total)) {
}, [] { return prepare_stopped(); }, [&](size_t) {
if (progress) progress(done.load(), all_n);
prepare_progress_log("noted pipelines", done.load(), total, t0, log_at);
warm_save_tick(save_at);
Member

These checkpoints wrote about 6.1 GB before the final save on AMD. Please measure the interrupted restart benefit and the full-warm time, peak RAM and temporary disk cost against the actual base, including Steam Deck.

These checkpoints wrote about 6.1 GB before the final save on AMD. Please measure the interrupted restart benefit and the full-warm time, peak RAM and temporary disk cost against the actual base, including Steam Deck.
@ -10482,6 +10504,7 @@ void precompile_warm(void (*progress)(size_t done, size_t total)) {
}, [] { return prepare_stopped(); }, [&](size_t) {
if (progress) progress(total + computes_done.load(), all_n);
prepare_progress_log("compute pipelines", computes_done.load(), computes.size(), c0, log_at);
warm_save_tick(save_at);
Member

These compute workers call cached_compute directly. If Intel's fetch times out, the owed save is picked up between jobs on a different queue that can be idle throughout preparation. Please give the warm its own bounded retry.

These compute workers call cached_compute directly. If Intel's fetch times out, the owed save is picked up between jobs on a different queue that can be idle throughout preparation. Please give the warm its own bounded retry.
@ -0,0 +70,4 @@
bad.append("%s does not write the cache as it goes (%s)" % (marker.split("(")[0], need.split("(")[0]))
if "bool warm_save_due(" not in draw:
return bad + ["vk_draw.cpp has no warm_save_due: the cache is written at the warm's end only"]
tick = block(draw, "void warm_save_tick(std::chrono::steady_clock::time_point& next)")
Member

This can pass when the save call is unreachable or when every snapshot fails to publish. Keep the clock test and exercise the checkpoint behavior with a stub cache that can grow between the two fetches.

This can pass when the save call is unreachable or when every snapshot fails to publish. Keep the clock test and exercise the checkpoint behavior with a stub cache that can grow between the two fetches.
HumanBean changed title from WIP: perf(vk): a warm writes the pipeline cache every ten minutes, so a preparation stopped early keeps what it built to perf(vk): a warm writes the pipeline cache every ten minutes, so a preparation stopped early keeps what it built 2026-11-21 00:00:43 +00:00
HumanBean deleted branch perf/warm-saves-as-it-goes 2026-11-21 00:01:28 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1940
No description provided.