fix(vk): keep impossible depth-integer compute variants out of the seed #894

Closed

easymod0 wants to merge 1 commit from fix/seed-compute-depth-integer-variants into main

easymod0 commented

2026-11-09 00:30:01 +00:00

(Migrated from github.com)

What this fixes

On a fresh install with Mesa 26.2.6 (RADV, RX 9070 XT, Linux), the first start crashes in the shader warm-up, and the next two crash in the precompile before it gives up for one start:

CRASH: exception 0xc0000005 (write 0x40) at libvulkan_radeon.so+0x15b67d

seed_compute tries every answer to a compute program's texture questions. When a texture that is depth-compared gets the answer "integer format", the SPIR-V samples a uint image for a float compare result. spirv-val rejects it, Mesa's spirv_to_nir returns NULL, and RADV dereferences that.

No bound texture can give that answer, because a depth compare needs a depth format. The variant was still saved to hookshot_programs.bin, so precompile_run compiled it again on later starts, and the warm-up and precompile markers only skip those steps temporarily.

Change

seed_compute now treats a variant with a texture that is both is_depth and is_integer as failed, so it is never compiled or saved.
Only the seed's trial variants are affected. Programs translated in play are unchanged.
vk_shader.cpp is part of the shader stamp, so caches that already hold these variants are rebuilt.

Testing

On the RX 9070 XT with Mesa 26.2.6, using the v2.0.5 game with this branch's hookshot_vk.so:

Start	Before	After
First, empty cache	Crash in the warm-up	Preparation finishes (38,809 programs, 132.9 s), 10,083 compute pipelines built, window opens, ~60 fps
Second	Crash in the precompile	Window opens, ~59 fps, nothing rebuilt

Mesa's MESA_SPIRV_FAIL_DUMP_PATH wrote no rejected shaders during either start. Before the fix it wrote the failing shader.
I fed the suspect compute shaders from a full v2.0.5 cache to the driver one at a time. All 216 with a depth compare on an integer texture were rejected by spirv_to_nir. The other 194 passed it, including 190 valid gathers on integer textures and 4 with a mismatched image format, so the check does not need to be broader.
tests/test_seed_compute_depth_integer.py is new and game-free. It fails without the change. tests/run_all.py: 100 passed, 0 failed, 12 skipped (missing headers on this machine).

Only the opening screens were run, about 15 seconds per start. I did not test gameplay, other GPUs or Windows.

#800 stops repeated precompile retries, which limits the damage from a bad variant. This change removes the bad variant.
#758 guards the same compile against driver faults, on Windows only.

## What this fixes On a fresh install with Mesa 26.2.6 (RADV, RX 9070 XT, Linux), the first start crashes in the shader warm-up, and the next two crash in the precompile before it gives up for one start: ``` CRASH: exception 0xc0000005 (write 0x40) at libvulkan_radeon.so+0x15b67d ``` `seed_compute` tries every answer to a compute program's texture questions. When a texture that is depth-compared gets the answer "integer format", the SPIR-V samples a `uint` image for a `float` compare result. `spirv-val` rejects it, Mesa's `spirv_to_nir` returns NULL, and RADV dereferences that. No bound texture can give that answer, because a depth compare needs a depth format. The variant was still saved to `hookshot_programs.bin`, so `precompile_run` compiled it again on later starts, and the warm-up and precompile markers only skip those steps temporarily. ## Change - `seed_compute` now treats a variant with a texture that is both `is_depth` and `is_integer` as failed, so it is never compiled or saved. - Only the seed's trial variants are affected. Programs translated in play are unchanged. - `vk_shader.cpp` is part of the shader stamp, so caches that already hold these variants are rebuilt. ## Testing On the RX 9070 XT with Mesa 26.2.6, using the v2.0.5 game with this branch's `hookshot_vk.so`: | Start | Before | After | |---|---|---| | First, empty cache | Crash in the warm-up | Preparation finishes (38,809 programs, 132.9 s), 10,083 compute pipelines built, window opens, ~60 fps | | Second | Crash in the precompile | Window opens, ~59 fps, nothing rebuilt | - Mesa's `MESA_SPIRV_FAIL_DUMP_PATH` wrote no rejected shaders during either start. Before the fix it wrote the failing shader. - I fed the suspect compute shaders from a full v2.0.5 cache to the driver one at a time. All 216 with a depth compare on an integer texture were rejected by `spirv_to_nir`. The other 194 passed it, including 190 valid gathers on integer textures and 4 with a mismatched image format, so the check does not need to be broader. - `tests/test_seed_compute_depth_integer.py` is new and game-free. It fails without the change. `tests/run_all.py`: 100 passed, 0 failed, 12 skipped (missing headers on this machine). Only the opening screens were run, about 15 seconds per start. I did not test gameplay, other GPUs or Windows. ## Related - #800 stops repeated precompile retries, which limits the damage from a bad variant. This change removes the bad variant. - #758 guards the same compile against driver faults, on Windows only.

tickernelz commented

2026-11-09 04:25:36 +00:00

(Migrated from github.com)

Confirmed on an RX 6800 XT (RADV NAVI21, Mesa 26.2.6, Fedora 44, kernel 7.2.15) with the v2.0.5 game and this branch's hookshot_vk.so from the Linux AppImage build of 0e396a7.

Fresh cache, no hookshot_programs.bin:

vk: prepared 38809 shader programs (46502 new, 165 failed, 0 stage libraries; 10083 compute variants, 752 programs with texture variants, 65 at the limit) in 59.4 s
vk: precompile: 10083 programs, 0 graphics and 10083 compute pipelines in 1295 ms
vk: 4200 presents, 60.0 fps; ... GPU-bound wait 0.0 ms

Ten minutes of frames afterwards, no CRASH. The same folder and procedure with the v2.0.5 hookshot_vk.so instead crashes in under a minute at libvulkan_radeon.so+0x14a7d2, which is the report's signature.

Cache validation, spirv-val --target-env vulkan1.3 over every unique module the cache holds:

cache	modules	rejected	depth-compare + integer sampled type
v2.0.5, as shipped	50,824	222	216
this branch, rebuilt from empty	50,800	6	0

The six that remain are two other classes and are not this patch's: four compute modules with an R32ui storage image declared with a float sampled type (the same class as #848), and two fragment modules where an SBlock value does not dominate its use. Neither crashes RADV in the runs above.

One caveat on my side: the run renders offscreen (no window, no display in the session I ran it from), so I can confirm the warm-up and the precompile that used to kill the process now complete and the game holds 60 fps, but not that the window opens on this card.

Confirmed on an **RX 6800 XT** (RADV NAVI21, Mesa 26.2.6, Fedora 44, kernel 7.2.15) with the v2.0.5 game and this branch's `hookshot_vk.so` from the Linux AppImage build of `0e396a7`. Fresh cache, no `hookshot_programs.bin`: ``` vk: prepared 38809 shader programs (46502 new, 165 failed, 0 stage libraries; 10083 compute variants, 752 programs with texture variants, 65 at the limit) in 59.4 s vk: precompile: 10083 programs, 0 graphics and 10083 compute pipelines in 1295 ms vk: 4200 presents, 60.0 fps; ... GPU-bound wait 0.0 ms ``` Ten minutes of frames afterwards, no `CRASH`. The same folder and procedure with the v2.0.5 `hookshot_vk.so` instead crashes in under a minute at `libvulkan_radeon.so+0x14a7d2`, which is the report's signature. Cache validation, `spirv-val --target-env vulkan1.3` over every unique module the cache holds: | cache | modules | rejected | depth-compare + integer sampled type | |---|---|---|---| | v2.0.5, as shipped | 50,824 | 222 | 216 | | this branch, rebuilt from empty | 50,800 | 6 | **0** | The six that remain are two other classes and are not this patch's: four compute modules with an `R32ui` storage image declared with a float sampled type (the same class as #848), and two fragment modules where an `SBlock` value does not dominate its use. Neither crashes RADV in the runs above. One caveat on my side: the run renders offscreen (`no window`, no display in the session I ran it from), so I can confirm the warm-up and the precompile that used to kill the process now complete and the game holds 60 fps, but not that the window opens on this card.

NotRexed commented

2026-11-09 05:30:57 +00:00

(Migrated from github.com)