MagicBeanDev/Longshot

Watch

Fork 102

fix(vk): the start names the driver's shader cache and says when the pipelines were built in another environment #902

Open

zeldaootr wants to merge 2 commits from zeldaootr/Longshot:driver-cache-diagnostics into main

zeldaootr commented

2026-11-15 19:51:17 +00:00

Contributor

Type: fix · Risk: low · Tested on: Steam Deck OLED (SteamOS 3.9.5, RADV, Mesa 26.2.0-devel); Linux game-free checks · Needs: none

Why

When a Steam Deck start from Game Mode compiled for 25-30 minutes the pipelines a start from the desktop loaded in seconds, the log gave no reason: its driver cache line read 0 MB (no folder found) beside a full Mesa cache, and it never said which cache folder or which driver switches the start ran with. This PR makes the start say both, and warn when its pipelines were built under other keys, so the next problem of this kind shows in the first log a player sends. It changes nothing in what is compiled.

The cause of that compile (Game Mode's RADV_FORCE_VRS_CONFIG_FILE) is fixed in the sibling PR "fix(linux): a start from the Steam Deck's Game Mode no longer compiles every pipeline the preparation built". No issue on the tracker covers it (searched: VRS, RADV_FORCE_VRS, half rate shading, Game Mode, driver shader cache); See #741 and #802: their logs show the old line, which looked only in mesa_shader_cache_db.

How to reproduce

Seen on: SteamOS 3.9.5 (Steam Deck OLED), RADV VANGOGH, Mesa 26.2.0-devel
Settings and switches: defaults
Steps: start from a shell with Mesa's cache under XDG_CACHE_HOME in a layout other than mesa_shader_cache_db, or with RADV_FORCE_VRS_CONFIG_FILE naming a file, and read logs/longshot_log.txt.
Expected: the log names the driver cache in use with its size, and says when RADV's VRS is forced and the pipelines were built without it.
Actual: vk: driver shader cache at the start: 0 MB (no folder found), and nothing about the environment.

What changed

The start reports the driver's shader cache as the environment chooses it, and warns when no completed build of the pipelines had its keys. The size count covers every Mesa layout and is bounded.

renderer: gpu/vk/driver_cache_env.h (new, standard library only): Mesa's folder (MESA_SHADER_CACHE_DIR, MESA_GLSL_CACHE_DIR, XDG_CACHE_HOME, ~/.cache) and its three layouts, NVIDIA's folder, RADV's "VRS forced", the records of the environments that built the pipelines, and the verdict.
renderer: gpu/vk/vk_draw.cpp:
- driver_cache_bytes counts mesa_shader_cache, mesa_shader_cache_sf and mesa_shader_cache_db, only Mesa's own folders (Steam's folder for a game holds Fossilize's and DXVK's too). It stops after 50,000 files or 250 ms and then says "at least".
- driver_cache_log("at the start") adds vk: driver shader cache: Mesa's cache in <folder> (<what chose it>), layout ...; RADV's VRS .... A variable named in LONGSHOT_DRIVER_ENV_SET (which the sibling PR sets when it drops RADV_FORCE_VRS_CONFIG_FILE) is called Longshot's, not the player's or Steam's. Without the sibling PR that variable is never set and the line reports the environment as it is.
- driver_env_note(), after a completed precompile_run, precompile_warm or predict_run, adds that build's keys and folder to cache/longshot_driver_env.txt (newest first, at most 8).
- The start compares its keys and folder with that list. It warns only when no recorded build had its keys, so starts alternating between desktop and Game Mode warn once, not every time. When a build had its keys in another folder and predicted pipelines exist, it adds an info line: those are only in the driver's cache, so each is compiled when first drawn.
docs: docs/development.md, "The driver's own shader cache (Linux)": the two lines and the record.
tests: tests/test_driver_cache_env.py with tests/driver_cache_env_test.cpp. The stand-ins in tests/test_prepare_stall.py follow the new call.

RADV's rules for "VRS forced", and how sure each is:

Forced VRS gives every graphics pipeline another key: measured on the Deck.
RADV_FORCE_VRS_CONFIG_FILE counts only when it names a file that exists, whatever its content: measured (an empty file and 1x1 compiled; a name with no file didn't).
When that variable is set, RADV_FORCE_VRS is not read; without it, RADV_FORCE_VRS other than 1x1 forces VRS: from Mesa's source, not measured.
Only RADV reads them, and only on GPUs with VK_KHR_fragment_shading_rate (gfx10.3 and later): from Mesa's source; only the Deck (gfx10.3) was run.

Measurements

Verdict: works on the Deck: the start names Mesa's folder and what chose it, and says whether RADV's VRS is forced; with the sibling fix it names the drop as Longshot's. No warning where none is due. It adds no time a start can see (the precompile and the title within the run-to-run spread, below).

Baseline: the After column is the local integration vrs-verify (bfc69759): main @ aa95d35f with this PR and its two siblings, so b shows the sibling fix's drop as well. Before is the old line, as main writes it. Rebased since onto main @ f637b42d; main's changes in between don't touch these lines.
Hardware: Steam Deck OLED, AMD Custom GPU 0932 (RADV VANGOGH), Mesa 26.2.0-devel (git-5c65d4fc31), SteamOS 3.9.5 (20261023.100), 16 GB
Scene: the start-up, up to the title; scratch game folders, each start with a fresh copy of the same warm caches; headless, defaults
Method: the log's vk: driver shader cache lines and the warning's presence; 4 starts interleaved a, b, b, a (a: no variable; b: RADV_FORCE_VRS_CONFIG_FILE naming a 1x1 file)
How to reproduce: printf 1x1 > <scratch>/vrs/vrs-1x1.txt; arm b sets RADV_FORCE_VRS_CONFIG_FILE to it

	Before: main @ `aa95d35f` (the old line)	After: this PR @ `1443161f` (in `vrs-verify` bfc69759)
a: the folder and the switches	only `mesa_shader_cache_db` (under `XDG_CACHE_HOME` or `~/.cache`, or `MESA_SHADER_CACHE_DIR` itself), so `0 MB (no folder found)` on a Deck start from a shell whose Mesa cache was in another layout; nothing about the switches	`Mesa's cache in <scratch>/game/state/xdgcache (XDG_CACHE_HOME), layout Mesa's default; RADV's VRS not forced`, and a size (`at least 101 MB`)
b: with the sibling fix's drop	nothing about the environment	`RADV_FORCE_VRS_CONFIG_FILE dropped by Longshot, so RADV's VRS is not forced (RX_VK_HALF_RATE_SHADING=1 keeps it)`
the warning that no build had this start's keys	none	none in a or b (correct: both ran with the preparation's keys)
precompile, a / b	a: 22,350 ms (one start)	17,379, 20,670 / 18,923, 12,481 ms

a: vk: driver shader cache: Mesa's cache in <scratch>/game/state/xdgcache (XDG_CACHE_HOME), layout Mesa's default; RADV's VRS not forced
a: vk: driver shader cache at the start: at least 101 MB (...)
b: vk: driver shader cache: Mesa's cache in <scratch>/game/state/xdgcache (XDG_CACHE_HOME), layout Mesa's default; RADV_FORCE_VRS_CONFIG_FILE dropped by Longshot, so RADV's VRS is not forced (RX_VK_HALF_RATE_SHADING=1 keeps it)

Flags and settings

None for players. The renderer reads LONGSHOT_DRIVER_ENV_SET, which only the host sets (the sibling PR).

Flag	What it is	Before this PR	After this PR	Effect	Helps / doesn't help / untested	Recommendation
`LONGSHOT_DRIVER_ENV_SET`	the driver variables the host itself set or dropped, comma-separated (not for players)	not read	read	the driver cache line calls those Longshot's	measured on the Deck with the sibling PR (b above)	leave to the host

Testing

Tested:

Steam Deck OLED, the run above: the lines as quoted, no warning where none was due.
tests/test_driver_cache_env.py: fails at the test commit (309c2610) and passes at the fix.
gpu/vk/driver_cache_env.h compiled alone with g++ (libstdc++) and llvm-mingw clang (libc++, Windows target).
gpu/vk/vk_draw.cpp whole-file syntax check with the build's renderer flags, Linux (clang) and Windows (llvm-mingw): the same as main @ f637b42d (Linux 103 warnings and main's one narrowing error at an unchanged line; Windows 0 errors, 105 warnings).
The full tests/run_all.py on Linux: 529 passed, 18 failed, 79 skipped (tools this PC lacks), 8 at once, on this PR merged with its two siblings onto main @ f637b42d. The 18 fail the same way on main @ f637b42d run alone: test_appimage_player_build, test_bake_portability, test_crash_dialog, test_crt_options, test_env_on, test_lift_cache, test_lift_cleanup_error, test_linux_preflight, test_linux_update, test_make_game_cache, test_missing_nup, test_mods_size_table, test_options_budget_once, test_pipeline_tour, test_readback_late, test_vk_occlusion_queries, test_vk_pgo, test_walk_key. The checks that read this PR's files pass, among them test_driver_env_choose, test_driver_cache_env, test_precompile_save, test_pipeline_cache_save, test_prepare_stall, test_display_startup_failure and test_tested_configurations.

Not tested: the warning itself in the game (a start with RX_VK_HALF_RATE_SHADING=1 and the file, which keeps RADV's VRS forced; the game-free test covers the verdict); the folder info line after a prediction; NVIDIA on Linux; a full renderer build and run on Windows (its Windows branch changed only in moving a lambda inside #ifdef _WIN32).

Must be tested: None.

Risks

What could break: the start's folder walk reads three Mesa folders instead of one, bounded at 50,000 files or 250 ms. One small file is written after each completed build of the pipelines. AMD's own Linux driver (AMDVLK, VK_DRIVER_ID_AMD_OPEN_SOURCE) would be described as Mesa's; it only changes the wording of a log line.

Turning it off: revert the commit.

Compatibility: nothing changes in the caches, saves, longshot_options.ini or the vk ABI. One new file, cache/longshot_driver_env.txt.

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main.
python -I -B tests/run_all.py passes (any failure that also fails on main is named).
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
A game-free test added for the fix, failing on main and passing here.
Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
Docs updated.

**Type:** fix · **Risk:** low · **Tested on:** Steam Deck OLED (SteamOS 3.9.5, RADV, Mesa 26.2.0-devel); Linux game-free checks · **Needs:** none ## Why When a Steam Deck start from Game Mode compiled for 25-30 minutes the pipelines a start from the desktop loaded in seconds, the log gave no reason: its driver cache line read `0 MB (no folder found)` beside a full Mesa cache, and it never said which cache folder or which driver switches the start ran with. This PR makes the start say both, and warn when its pipelines were built under other keys, so the next problem of this kind shows in the first log a player sends. It changes nothing in what is compiled. The cause of that compile (Game Mode's `RADV_FORCE_VRS_CONFIG_FILE`) is fixed in the sibling PR "fix(linux): a start from the Steam Deck's Game Mode no longer compiles every pipeline the preparation built". No issue on the tracker covers it (searched: VRS, RADV_FORCE_VRS, half rate shading, Game Mode, driver shader cache); See https://git.magicbean.dev/MagicBeanDev/Longshot/issues/741 and https://git.magicbean.dev/MagicBeanDev/Longshot/issues/802: their logs show the old line, which looked only in `mesa_shader_cache_db`. ## How to reproduce **Seen on:** SteamOS 3.9.5 (Steam Deck OLED), RADV VANGOGH, Mesa 26.2.0-devel **Settings and switches:** defaults **Steps:** start from a shell with Mesa's cache under `XDG_CACHE_HOME` in a layout other than `mesa_shader_cache_db`, or with `RADV_FORCE_VRS_CONFIG_FILE` naming a file, and read `logs/longshot_log.txt`. **Expected:** the log names the driver cache in use with its size, and says when RADV's VRS is forced and the pipelines were built without it. **Actual:** `vk: driver shader cache at the start: 0 MB (no folder found)`, and nothing about the environment. ## What changed The start reports the driver's shader cache as the environment chooses it, and warns when no completed build of the pipelines had its keys. The size count covers every Mesa layout and is bounded. - renderer: `gpu/vk/driver_cache_env.h` (new, standard library only): Mesa's folder (`MESA_SHADER_CACHE_DIR`, `MESA_GLSL_CACHE_DIR`, `XDG_CACHE_HOME`, `~/.cache`) and its three layouts, NVIDIA's folder, RADV's "VRS forced", the records of the environments that built the pipelines, and the verdict. - renderer: `gpu/vk/vk_draw.cpp`: - `driver_cache_bytes` counts `mesa_shader_cache`, `mesa_shader_cache_sf` and `mesa_shader_cache_db`, only Mesa's own folders (Steam's folder for a game holds Fossilize's and DXVK's too). It stops after 50,000 files or 250 ms and then says "at least". - `driver_cache_log("at the start")` adds `vk: driver shader cache: Mesa's cache in <folder> (<what chose it>), layout ...; RADV's VRS ...`. A variable named in `LONGSHOT_DRIVER_ENV_SET` (which the sibling PR sets when it drops `RADV_FORCE_VRS_CONFIG_FILE`) is called Longshot's, not the player's or Steam's. Without the sibling PR that variable is never set and the line reports the environment as it is. - `driver_env_note()`, after a completed `precompile_run`, `precompile_warm` or `predict_run`, adds that build's keys and folder to `cache/longshot_driver_env.txt` (newest first, at most 8). - The start compares its keys and folder with that list. It warns only when no recorded build had its keys, so starts alternating between desktop and Game Mode warn once, not every time. When a build had its keys in another folder and predicted pipelines exist, it adds an info line: those are only in the driver's cache, so each is compiled when first drawn. - docs: `docs/development.md`, "The driver's own shader cache (Linux)": the two lines and the record. - tests: `tests/test_driver_cache_env.py` with `tests/driver_cache_env_test.cpp`. The stand-ins in `tests/test_prepare_stall.py` follow the new call. RADV's rules for "VRS forced", and how sure each is: - Forced VRS gives every graphics pipeline another key: measured on the Deck. - `RADV_FORCE_VRS_CONFIG_FILE` counts only when it names a file that exists, whatever its content: measured (an empty file and `1x1` compiled; a name with no file didn't). - When that variable is set, `RADV_FORCE_VRS` is not read; without it, `RADV_FORCE_VRS` other than `1x1` forces VRS: from Mesa's source, not measured. - Only RADV reads them, and only on GPUs with `VK_KHR_fragment_shading_rate` (gfx10.3 and later): from Mesa's source; only the Deck (gfx10.3) was run. ## Measurements **Verdict:** works on the Deck: the start names Mesa's folder and what chose it, and says whether RADV's VRS is forced; with the sibling fix it names the drop as Longshot's. No warning where none is due. It adds no time a start can see (the precompile and the title within the run-to-run spread, below). Baseline: the After column is the local integration `vrs-verify` (bfc69759): main @ aa95d35f with this PR and its two siblings, so b shows the sibling fix's drop as well. Before is the old line, as main writes it. Rebased since onto main @ f637b42d; main's changes in between don't touch these lines. Hardware: Steam Deck OLED, AMD Custom GPU 0932 (RADV VANGOGH), Mesa 26.2.0-devel (git-5c65d4fc31), SteamOS 3.9.5 (20261023.100), 16 GB Scene: the start-up, up to the title; scratch game folders, each start with a fresh copy of the same warm caches; headless, defaults Method: the log's `vk: driver shader cache` lines and the warning's presence; 4 starts interleaved a, b, b, a (a: no variable; b: `RADV_FORCE_VRS_CONFIG_FILE` naming a `1x1` file) How to reproduce: `printf 1x1 > <scratch>/vrs/vrs-1x1.txt`; arm b sets `RADV_FORCE_VRS_CONFIG_FILE` to it | | Before: main @ aa95d35f (the old line) | After: this PR @ 1443161f (in `vrs-verify` bfc69759) | |---|---|---| | a: the folder and the switches | only `mesa_shader_cache_db` (under `XDG_CACHE_HOME` or `~/.cache`, or `MESA_SHADER_CACHE_DIR` itself), so `0 MB (no folder found)` on a Deck start from a shell whose Mesa cache was in another layout; nothing about the switches | `Mesa's cache in <scratch>/game/state/xdgcache (XDG_CACHE_HOME), layout Mesa's default; RADV's VRS not forced`, and a size (`at least 101 MB`) | | b: with the sibling fix's drop | nothing about the environment | `RADV_FORCE_VRS_CONFIG_FILE dropped by Longshot, so RADV's VRS is not forced (RX_VK_HALF_RATE_SHADING=1 keeps it)` | | the warning that no build had this start's keys | none | none in a or b (correct: both ran with the preparation's keys) | | precompile, a / b | a: 22,350 ms (one start) | 17,379, 20,670 / 18,923, 12,481 ms | ``` a: vk: driver shader cache: Mesa's cache in <scratch>/game/state/xdgcache (XDG_CACHE_HOME), layout Mesa's default; RADV's VRS not forced a: vk: driver shader cache at the start: at least 101 MB (...) b: vk: driver shader cache: Mesa's cache in <scratch>/game/state/xdgcache (XDG_CACHE_HOME), layout Mesa's default; RADV_FORCE_VRS_CONFIG_FILE dropped by Longshot, so RADV's VRS is not forced (RX_VK_HALF_RATE_SHADING=1 keeps it) ``` ### Flags and settings None for players. The renderer reads `LONGSHOT_DRIVER_ENV_SET`, which only the host sets (the sibling PR). | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `LONGSHOT_DRIVER_ENV_SET` | the driver variables the host itself set or dropped, comma-separated (not for players) | not read | read | the driver cache line calls those Longshot's | measured on the Deck with the sibling PR (b above) | leave to the host | ## Testing **Tested:** - Steam Deck OLED, the run above: the lines as quoted, no warning where none was due. - `tests/test_driver_cache_env.py`: fails at the test commit (309c2610) and passes at the fix. - `gpu/vk/driver_cache_env.h` compiled alone with g++ (libstdc++) and llvm-mingw clang (libc++, Windows target). - `gpu/vk/vk_draw.cpp` whole-file syntax check with the build's renderer flags, Linux (clang) and Windows (llvm-mingw): the same as main @ f637b42d (Linux 103 warnings and main's one narrowing error at an unchanged line; Windows 0 errors, 105 warnings). - The full `tests/run_all.py` on Linux: 529 passed, 18 failed, 79 skipped (tools this PC lacks), 8 at once, on this PR merged with its two siblings onto main @ f637b42d. The 18 fail the same way on main @ f637b42d run alone: `test_appimage_player_build`, `test_bake_portability`, `test_crash_dialog`, `test_crt_options`, `test_env_on`, `test_lift_cache`, `test_lift_cleanup_error`, `test_linux_preflight`, `test_linux_update`, `test_make_game_cache`, `test_missing_nup`, `test_mods_size_table`, `test_options_budget_once`, `test_pipeline_tour`, `test_readback_late`, `test_vk_occlusion_queries`, `test_vk_pgo`, `test_walk_key`. The checks that read this PR's files pass, among them `test_driver_env_choose`, `test_driver_cache_env`, `test_precompile_save`, `test_pipeline_cache_save`, `test_prepare_stall`, `test_display_startup_failure` and `test_tested_configurations`. **Not tested:** the warning itself in the game (a start with `RX_VK_HALF_RATE_SHADING=1` and the file, which keeps RADV's VRS forced; the game-free test covers the verdict); the folder info line after a prediction; NVIDIA on Linux; a full renderer build and run on Windows (its Windows branch changed only in moving a lambda inside `#ifdef _WIN32`). **Must be tested:** None. ## Risks **What could break:** the start's folder walk reads three Mesa folders instead of one, bounded at 50,000 files or 250 ms. One small file is written after each completed build of the pipelines. AMD's own Linux driver (AMDVLK, `VK_DRIVER_ID_AMD_OPEN_SOURCE`) would be described as Mesa's; it only changes the wording of a log line. **Turning it off:** revert the commit. **Compatibility:** nothing changes in the caches, saves, `longshot_options.ini` or the vk ABI. One new file, `cache/longshot_driver_env.txt`. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] Docs updated.

zeldaootr added 2 commits

2026-11-15 19:51:17 +00:00

test(vk): a check that the start counts the driver's caches and compares its environment with the builds' 309c261001

`tests/test_driver_cache_env.py` fails on main: the start looks for
Mesa's `mesa_shader_cache_db` alone (a Steam Deck's shell start read
"0 MB (no folder found)"), walks the folders without a bound, and says
nothing of the environment the pipelines were built in.

Assisted-by: Claude

fix(vk): the start names the driver's shader cache and says when the pipelines were built in another environment

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

1443161f56

The Builder prepares in the desktop's environment and Steam starts the
game in its own. In the Steam Deck's Game Mode that includes
`RADV_FORCE_VRS_CONFIG_FILE`: measured on a Deck OLED, with that file
there RADV gave every graphics pipeline another key, and the 32,438
noted pipelines compiled at about 20 a second instead of loading in
22 s. The log said nothing, and its cache line read "0 MB (no folder
found)" beside a full cache.

- `gpu/vk/driver_cache_env.h`: Mesa's folder and layouts as Mesa picks
  them, NVIDIA's, RADV's VRS (RADV with `VK_KHR_fragment_shading_rate`,
  the config file naming a file that exists, else the rate), the
  records and the verdict.
- `driver_cache_bytes` counts Mesa's three layouts, only Mesa's folders
  inside Steam's, and stops at 50,000 files or 250 ms ("at least").
- The start logs the folder, what chose it, and RADV's VRS, naming what
  the host itself set or dropped (`LONGSHOT_DRIVER_ENV_SET`) as
  Longshot's. Completed builds record their keys and folder in
  `cache/longshot_driver_env.txt` (the last 8). The start warns when no
  build had its keys, and says as information when only the folder
  differs and predicted pipelines exist.

Assisted-by: Claude