MagicBeanDev/Longshot

Watch

Fork 102

WIP: fix(vk): a start that compiled its noted pipelines writes the pipeline cache before the title #903

Draft

zeldaootr wants to merge 2 commits from zeldaootr/Longshot:precompile-save-before-title into main

zeldaootr commented

2026-11-15 19:51:22 +00:00

Contributor

Type: fix · Risk: medium · Tested on: Linux game-free checks; Steam Deck OLED (SteamOS 3.9.5, RADV) with no long precompile, so no save · Needs: Steam Deck (a start that compiles, to the title, and the start after it); Windows, any GPU (the write's time)

Why

A start that had to compile its noted pipelines should keep them, so the next start finds them. On a Steam Deck a start compiled all 32,438 of them for about 20 minutes and kept none in our pipeline cache, because in play the cache is written only once it has grown 64 MB, and those pipelines grew it by 4.3 MB (RADV keeps a pipeline as its key and its shaders' hashes, and the code was already there). The next start could find them only in the driver's own folder.

The compile in that case came from Game Mode's RADV_FORCE_VRS_CONFIG_FILE, which the sibling PR "fix(linux): a start from the Steam Deck's Game Mode no longer compiles every pipeline the preparation built" removes. A start still compiles everything after a driver update, or with RX_VK_HALF_RATE_SHADING=1. #789 (a driver update warms the noted pipelines again under RX_VK_LOAD=create) is related; that path already writes the cache, and this PR covers the other one. No issue on the tracker covers it (searched: pipeline cache save, before the title, noted pipelines).

How to reproduce

Seen on: SteamOS 3.9.5 (Steam Deck OLED), RADV VANGOGH, Mesa 26.2.0-devel
Settings and switches: defaults; RADV_FORCE_VRS_CONFIG_FILE naming a file (as Game Mode does) on main, to make the start compile
Steps: prepare outside Game Mode, start with the variable set, wait through "Loading pipelines" to the title, quit, and start the same way again.
Expected: the second start finds the pipelines in our cache.
Actual: vk: pipeline cache 4303 KB larger in play: written once it is 64 MB larger, or by a forced save; the cache file is not written, and the second start depends on what the driver's folder kept.

What changed

A start whose noted pipelines took RX_VK_PRECOMPILE_SAVE_S (30 s) or more to build writes the pipeline cache before the title, with the start-up screen showing it as saving. The log says what the save did and how long it took. The warm pass under RX_VK_LOAD=create, which already wrote the cache, now reports the same way.

renderer: gpu/vk/precompile_save.h (new, standard library only): after_s reads the switch; due decides the save (the build took that long, the precompile reached its end, and this is a game start, since the preparation's process writes the cache anyway); said puts what a save did into words.
renderer: gpu/vk/vk_draw.cpp:
- precompile_run: when due, it sets prepare_saving_now (the start-up screen's saving state, as precompile_warm does), calls pipeline_cache_save(true), and logs vk: precompile: its pipelines took N s (compiled, not found in the caches): the pipeline cache <what it did>, in T ms, before the title.
- precompile_warm: its forced save logs the pipeline cache <what it did>, in T ms instead of an unconditional "written out".
- pipeline_cache_save records how each save ended in pcache_save_outcome, thread-local: a forced save runs on its caller's thread.
docs: docs/development.md's row for RX_VK_PRECOMPILE_SAVE_S.
tests: tests/test_precompile_save.py with tests/precompile_save_test.cpp. tests/pipeline_cache_save_test.cpp, which compiles the real pipeline_cache_save, also checks the recorded outcome: written with its size, nothing new, fetch failed, write failed.

This changes behaviour on every platform: after any precompile of 30 s or more (the first start after a driver update on Windows, for one) the whole cache is written before the title, only when it grew. That costs seconds once, and what was compiled is no longer lost if the game is closed or crashes before play writes it.

Measurements

Verdict: not yet measured: the save before the title has not run in the game. The Deck run that verified the sibling fix had no precompile of 30 s or more (12-21 s), so no save, as intended. The run that times the save (a start with RX_VK_HALF_RATE_SHADING=1 and the file, to the title, then again from that cache) is queued.

Baseline: the After column is the local integration vrs-verify (bfc69759): main @ aa95d35f with this PR and its two siblings.
Hardware: Steam Deck OLED, AMD Custom GPU 0932 (RADV VANGOGH), Mesa 26.2.0-devel (git-5c65d4fc31), SteamOS 3.9.5 (20261023.100), 16 GB
Scene: the start-up, up to the title; scratch game folders, each start with a fresh copy of the same warm caches; headless, defaults
Method: the log's vk: precompile: and vk: pipeline cache lines, the time to the title and the cache file's size; 4 starts interleaved a, b, b, a
How to reproduce: as in the sibling fix's run; for the save, RX_VK_HALF_RATE_SHADING=1 with RADV_FORCE_VRS_CONFIG_FILE naming a 1x1 file

	Before: main @ `aa95d35f` (with other open PRs)	After: this PR @ `53c0be91` (in `vrs-verify` bfc69759)
a start that found its pipelines	22,350 ms; no save	12,481-20,670 ms over 4 starts; no save before the title (below 30 s)
a start that compiled them (1,222 s), the cache	grew 4.3 MB, `written once it is 64 MB larger`: never written	not run yet
the write's time before the title, Deck internal storage / Windows	none	not run yet / not run yet
the start after that one	depends on the driver's folder	not run yet

Before: vk: pipeline cache 4303 KB larger in play: written once it is 64 MB larger, or by a forced save
After:  vk: precompile: 27660 programs, 32438 graphics and 3623 compute pipelines, 2284 mesh fallback programs in 12481 ms (11974 ms of it the caches opened, the notes and programs read)

Flags and settings

One new switch, on by default.

Flag	What it is	Before this PR	After this PR	Effect	Helps / doesn't help / untested	Recommendation
`RX_VK_PRECOMPILE_SAVE_S=<s>`	the precompile build time from which a start writes the pipeline cache before the title	none: written in play at 64 MB of growth	30 (`0`: never)	after a long precompile, a write of the whole cache before the title, only if it grew	untested in the game (the run is queued)	30
`LONGSHOT_PREPARE_ONLY`	the preparation's own process (set by `vk_main.cpp`, not a switch)	as is	also read here: no early save there, the preparation writes the cache anyway	none	n/a	leave as is

Testing

Tested:

tests/test_precompile_save.py: fails at the test commit (49d11cab) and passes at the fix.
tests/test_pipeline_cache_save.py, which compiles the real pipeline_cache_save, passes.
gpu/vk/precompile_save.h compiled alone with g++ (libstdc++) and llvm-mingw clang (libc++, Windows target).
Steam Deck OLED: 4 starts with this PR (in vrs-verify) that found their pipelines; none saved before the title, as intended.
gpu/vk/vk_draw.cpp whole-file syntax check with the build's renderer flags, Linux (clang) and Windows (llvm-mingw): the same as main @ f637b42d (Linux 103 warnings and main's one narrowing error at an unchanged line; Windows 0 errors, 105 warnings).
The full tests/run_all.py on Linux: 529 passed, 18 failed, 79 skipped (tools this PC lacks), 8 at once, on this PR merged with its two siblings onto main @ f637b42d. The 18 fail the same way on main @ f637b42d run alone: test_appimage_player_build, test_bake_portability, test_crash_dialog, test_crt_options, test_env_on, test_lift_cache, test_lift_cleanup_error, test_linux_preflight, test_linux_update, test_make_game_cache, test_missing_nup, test_mods_size_table, test_options_budget_once, test_pipeline_tour, test_readback_late, test_vk_occlusion_queries, test_vk_pgo, test_walk_key. The checks that read this PR's files pass, among them test_driver_env_choose, test_driver_cache_env, test_precompile_save, test_pipeline_cache_save, test_prepare_stall, test_display_startup_failure and test_tested_configurations.

Not tested: the save before the title in the game; its time on a Deck's internal storage, a microSD card and Windows; a full renderer build and run on Windows (the change is platform-neutral code).

Must be tested:

steamdeck (SteamOS, RADV), by hand: with a backup of cache/ restored before each start, set RX_VK_HALF_RATE_SHADING=1 and RADV_FORCE_VRS_CONFIG_FILE naming a file holding 1x1, start to the title, expecting vk: precompile: its pipelines took ~1200 s (compiled, not found in the caches): the pipeline cache written, N MB, in T ms, before the title with the start-up screen saying it saves; then start again the same way from that cache, not the backup, expecting the precompile in seconds. 2 runs.
Windows, any GPU, by hand: move longshot_vk_pipelines.bin out of the cache folder (or start after a driver update), start the game, expecting the same line when the precompile took 30 s or more and nothing otherwise, and report T.

Risks

What could break: a start that compiled for 30 s or more now writes the whole pipeline cache before the title, on every platform: seconds, more on a microSD card or a slow disk. A start that took 30 s while finding its pipelines (a slow disk) asks the driver for the cache's size and writes nothing when it didn't grow.

Turning it off: RX_VK_PRECOMPILE_SAVE_S=0.

Compatibility: nothing changes in the caches beyond longshot_vk_pipelines.bin being written earlier; saves, longshot_options.ini and the vk ABI are unchanged.

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure. (the save before the title not run yet in the game: the Deck run with RX_VK_HALF_RATE_SHADING=1 is queued)
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main.
python -I -B tests/run_all.py passes (any failure that also fails on main is named).
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (waits on the Deck run above)
Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md). (waits on the Deck run above)
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
Docs updated.

**Type:** fix · **Risk:** medium · **Tested on:** Linux game-free checks; Steam Deck OLED (SteamOS 3.9.5, RADV) with no long precompile, so no save · **Needs:** Steam Deck (a start that compiles, to the title, and the start after it); Windows, any GPU (the write's time) ## Why A start that had to compile its noted pipelines should keep them, so the next start finds them. On a Steam Deck a start compiled all 32,438 of them for about 20 minutes and kept none in our pipeline cache, because in play the cache is written only once it has grown 64 MB, and those pipelines grew it by 4.3 MB (RADV keeps a pipeline as its key and its shaders' hashes, and the code was already there). The next start could find them only in the driver's own folder. The compile in that case came from Game Mode's `RADV_FORCE_VRS_CONFIG_FILE`, which the sibling PR "fix(linux): a start from the Steam Deck's Game Mode no longer compiles every pipeline the preparation built" removes. A start still compiles everything after a driver update, or with `RX_VK_HALF_RATE_SHADING=1`. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/789 (a driver update warms the noted pipelines again under `RX_VK_LOAD=create`) is related; that path already writes the cache, and this PR covers the other one. No issue on the tracker covers it (searched: pipeline cache save, before the title, noted pipelines). ## How to reproduce **Seen on:** SteamOS 3.9.5 (Steam Deck OLED), RADV VANGOGH, Mesa 26.2.0-devel **Settings and switches:** defaults; `RADV_FORCE_VRS_CONFIG_FILE` naming a file (as Game Mode does) on main, to make the start compile **Steps:** prepare outside Game Mode, start with the variable set, wait through "Loading pipelines" to the title, quit, and start the same way again. **Expected:** the second start finds the pipelines in our cache. **Actual:** `vk: pipeline cache 4303 KB larger in play: written once it is 64 MB larger, or by a forced save`; the cache file is not written, and the second start depends on what the driver's folder kept. ## What changed A start whose noted pipelines took `RX_VK_PRECOMPILE_SAVE_S` (30 s) or more to build writes the pipeline cache before the title, with the start-up screen showing it as saving. The log says what the save did and how long it took. The warm pass under `RX_VK_LOAD=create`, which already wrote the cache, now reports the same way. - renderer: `gpu/vk/precompile_save.h` (new, standard library only): `after_s` reads the switch; `due` decides the save (the build took that long, the precompile reached its end, and this is a game start, since the preparation's process writes the cache anyway); `said` puts what a save did into words. - renderer: `gpu/vk/vk_draw.cpp`: - `precompile_run`: when `due`, it sets `prepare_saving_now` (the start-up screen's saving state, as `precompile_warm` does), calls `pipeline_cache_save(true)`, and logs `vk: precompile: its pipelines took N s (compiled, not found in the caches): the pipeline cache <what it did>, in T ms, before the title`. - `precompile_warm`: its forced save logs `the pipeline cache <what it did>, in T ms` instead of an unconditional "written out". - `pipeline_cache_save` records how each save ended in `pcache_save_outcome`, thread-local: a forced save runs on its caller's thread. - docs: `docs/development.md`'s row for `RX_VK_PRECOMPILE_SAVE_S`. - tests: `tests/test_precompile_save.py` with `tests/precompile_save_test.cpp`. `tests/pipeline_cache_save_test.cpp`, which compiles the real `pipeline_cache_save`, also checks the recorded outcome: written with its size, nothing new, fetch failed, write failed. **This changes behaviour on every platform:** after any precompile of 30 s or more (the first start after a driver update on Windows, for one) the whole cache is written before the title, only when it grew. That costs seconds once, and what was compiled is no longer lost if the game is closed or crashes before play writes it. ## Measurements **Verdict:** not yet measured: the save before the title has not run in the game. The Deck run that verified the sibling fix had no precompile of 30 s or more (12-21 s), so no save, as intended. The run that times the save (a start with `RX_VK_HALF_RATE_SHADING=1` and the file, to the title, then again from that cache) is queued. Baseline: the After column is the local integration `vrs-verify` (bfc69759): main @ aa95d35f with this PR and its two siblings. Hardware: Steam Deck OLED, AMD Custom GPU 0932 (RADV VANGOGH), Mesa 26.2.0-devel (git-5c65d4fc31), SteamOS 3.9.5 (20261023.100), 16 GB Scene: the start-up, up to the title; scratch game folders, each start with a fresh copy of the same warm caches; headless, defaults Method: the log's `vk: precompile:` and `vk: pipeline cache` lines, the time to the title and the cache file's size; 4 starts interleaved a, b, b, a How to reproduce: as in the sibling fix's run; for the save, `RX_VK_HALF_RATE_SHADING=1` with `RADV_FORCE_VRS_CONFIG_FILE` naming a `1x1` file | | Before: main @ aa95d35f (with other open PRs) | After: this PR @ 53c0be91 (in `vrs-verify` bfc69759) | |---|---|---| | a start that found its pipelines | 22,350 ms; no save | 12,481-20,670 ms over 4 starts; no save before the title (below 30 s) | | a start that compiled them (1,222 s), the cache | grew 4.3 MB, `written once it is 64 MB larger`: never written | not run yet | | the write's time before the title, Deck internal storage / Windows | none | not run yet / not run yet | | the start after that one | depends on the driver's folder | not run yet | ``` Before: vk: pipeline cache 4303 KB larger in play: written once it is 64 MB larger, or by a forced save After: vk: precompile: 27660 programs, 32438 graphics and 3623 compute pipelines, 2284 mesh fallback programs in 12481 ms (11974 ms of it the caches opened, the notes and programs read) ``` ### Flags and settings One new switch, on by default. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `RX_VK_PRECOMPILE_SAVE_S=<s>` | the precompile build time from which a start writes the pipeline cache before the title | none: written in play at 64 MB of growth | 30 (`0`: never) | after a long precompile, a write of the whole cache before the title, only if it grew | untested in the game (the run is queued) | 30 | | `LONGSHOT_PREPARE_ONLY` | the preparation's own process (set by `vk_main.cpp`, not a switch) | as is | also read here: no early save there, the preparation writes the cache anyway | none | n/a | leave as is | ## Testing **Tested:** - `tests/test_precompile_save.py`: fails at the test commit (49d11cab) and passes at the fix. - `tests/test_pipeline_cache_save.py`, which compiles the real `pipeline_cache_save`, passes. - `gpu/vk/precompile_save.h` compiled alone with g++ (libstdc++) and llvm-mingw clang (libc++, Windows target). - Steam Deck OLED: 4 starts with this PR (in `vrs-verify`) that found their pipelines; none saved before the title, as intended. - `gpu/vk/vk_draw.cpp` whole-file syntax check with the build's renderer flags, Linux (clang) and Windows (llvm-mingw): the same as main @ f637b42d (Linux 103 warnings and main's one narrowing error at an unchanged line; Windows 0 errors, 105 warnings). - The full `tests/run_all.py` on Linux: 529 passed, 18 failed, 79 skipped (tools this PC lacks), 8 at once, on this PR merged with its two siblings onto main @ f637b42d. The 18 fail the same way on main @ f637b42d run alone: `test_appimage_player_build`, `test_bake_portability`, `test_crash_dialog`, `test_crt_options`, `test_env_on`, `test_lift_cache`, `test_lift_cleanup_error`, `test_linux_preflight`, `test_linux_update`, `test_make_game_cache`, `test_missing_nup`, `test_mods_size_table`, `test_options_budget_once`, `test_pipeline_tour`, `test_readback_late`, `test_vk_occlusion_queries`, `test_vk_pgo`, `test_walk_key`. The checks that read this PR's files pass, among them `test_driver_env_choose`, `test_driver_cache_env`, `test_precompile_save`, `test_pipeline_cache_save`, `test_prepare_stall`, `test_display_startup_failure` and `test_tested_configurations`. **Not tested:** the save before the title in the game; its time on a Deck's internal storage, a microSD card and Windows; a full renderer build and run on Windows (the change is platform-neutral code). **Must be tested:** - steamdeck (SteamOS, RADV), by hand: with a backup of `cache/` restored before each start, set `RX_VK_HALF_RATE_SHADING=1` and `RADV_FORCE_VRS_CONFIG_FILE` naming a file holding `1x1`, start to the title, expecting `vk: precompile: its pipelines took ~1200 s (compiled, not found in the caches): the pipeline cache written, N MB, in T ms, before the title` with the start-up screen saying it saves; then start again the same way from that cache, not the backup, expecting the precompile in seconds. 2 runs. - Windows, any GPU, by hand: move `longshot_vk_pipelines.bin` out of the cache folder (or start after a driver update), start the game, expecting the same line when the precompile took 30 s or more and nothing otherwise, and report T. ## Risks **What could break:** a start that compiled for 30 s or more now writes the whole pipeline cache before the title, on every platform: seconds, more on a microSD card or a slow disk. A start that took 30 s while finding its pipelines (a slow disk) asks the driver for the cache's size and writes nothing when it didn't grow. **Turning it off:** `RX_VK_PRECOMPILE_SAVE_S=0`. **Compatibility:** nothing changes in the caches beyond `longshot_vk_pipelines.bin` being written earlier; saves, `longshot_options.ini` and the vk ABI are unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [ ] Measurements has a verdict, or one line saying why there is nothing to measure. (the save before the title not run yet in the game: the Deck run with `RX_VK_HALF_RATE_SHADING=1` is queued) - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (waits on the Deck run above) - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [ ] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`). (waits on the Deck run above) - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] Docs updated.

zeldaootr added 2 commits

2026-11-15 19:51:22 +00:00

test(vk): a check that a long precompile writes the pipeline cache before the title 49d11cabb7

`tests/test_precompile_save.py` fails on main: a start whose noted
pipelines were compiled, not found, leaves them to the play save, which
waits for 64 MB of growth, and the warm pass's line says it wrote the
cache whatever the save did.

Assisted-by: Claude

fix(vk): a start that compiled its noted pipelines writes the pipeline cache before the title

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

53c0be913a

A Steam Deck start from Steam compiled its 32,438 noted pipelines for
20 minutes and grew the pipeline cache by 4.3 MB: RADV keeps a pipeline
as its key and its shaders' hashes, and the code was already there. In
play the cache is written only at 64 MB of growth, so none of it was
written and the next start had only the driver's own folder.

- `gpu/vk/precompile_save.h`: when it is due (the build took
  `RX_VK_PRECOMPILE_SAVE_S`, 30 s; 0 never; not when stopped, gone on
  or in the preparation's own process), and what a save did, in words.
- `precompile_run` writes it then, with the start-up screen's saving
  state on as `precompile_warm` has it, and logs written N MB / nothing
  new / not written and why, and the time. `precompile_warm`'s line
  (`RX_VK_LOAD=create`) says the same instead of "written out".
- `pipeline_cache_save` records how each save ended
  (`pcache_save_outcome`, thread-local: a forced save runs on its
  caller's thread). `tests/pipeline_cache_save_test.cpp` checks it.

On every platform: after any precompile of 30 s or more (a driver
update on Windows too) the write now happens before the title.

Assisted-by: Claude