fix(vk): RADV on GCN 1-4 skips the two mesh fallback programs its compiler never finishes #2525

Merged

hckrman101 merged 3 commits from fix/radv-gcn-mesh-fallback into main

2026-11-11 23:34:14 +00:00

hckrman101 commented

2026-11-11 22:01:18 +00:00

(Migrated from github.com)

Why

On an RX 580 (Polaris 10, Mesa 26.2.6 RADV), a cold --prepare-shaders got through the translation and the prediction, then ran out of memory in the precompile. It grew past 24 GB and was killed, on main too. The game's own first start does the same precompile, so a player with an RX 400/500-class card can't get through the first start: it pushes the desktop into swap or the OOM killer.

The cause is two mesh programs' fallback compute pipelines, 93e3e999f2fb9fd8 (stage 1) and ea4dd64df9c8f2b7 (stage 0). RADV never finishes compiling them on GCN 1-4 (GFX6-8), and its thread keeps the memory it grew to. A time limit wouldn't help for that reason.

What makes them different from the other mesh programs:

Size. Both mesh stages are 4.3 MB of SPIR-V: about 28,000 blocks and 80,000 phis. The native programs already have about 11,000 blocks each. The next biggest mesh programs have up to 29,400 phis.
Live values. About 400 values are live at once (≈300 32-bit, ≈100 booleans), far more than a wave's 256 VGPRs and ~100 SGPRs. ACO has to spill tens of thousands of them.
Not the 126 vertices. Their max vertices/primitives of 126 isn't what sets them apart: 246 of the first 400 mesh programs the prepare logs are 126/126, and they build normally.
Not the fallback's own code. The CFG comes from the native program's structurization, which the native mesh path shares.

I compiled the dumped modules with RADV itself, through Mesa's amdgpu drm-shim (no AMD GPU needed), under a 6 GB cap:

module	Polaris 10 / 12, Stoney, Bonaire, Pitcairn (GFX6-8)	Vega 10, Raven (GFX9)	Navi 10 / 21
`ea4dd64df9c8f2b7` mesh (79,947 phis)	not done after 45-90 s, killed at the 6 GB cap (the lab: 25 GB at 150 s)	14 s, 2.9 GB, 24,879 SGPR + 10,644 VGPR spills	13-14 s, 2.9 GB
`ecbbff63280b68e2` mesh (29,372 phis)	4.5 s, 480 MB
`cf987f56b971bdb3` mesh (28,506 phis)	5.5 s, 664 MB

RX_VK_NOOPT (DISABLE_OPTIMIZATION) doesn't change it. The spiller runs either way. GFX6-8 have no scratch instructions. ACO spills there through buffer instructions, and these programs' scratch is past their offset range, so ACO takes its overflow path for every spill and reload (aco_spill.cpp). With this many spills it doesn't finish. That's my reading of the source, not profiled.

I couldn't find a translation-side change that compiles these two normally. Their register pressure and block count come from the native program and the shared goto-elimination structurizer. Changing either would retranslate every shader for every GPU. So this PR leaves just these two out on the cards whose compiler can't build them.

What changes

gpu/vk/fallback_size_policy.h (new): spirv_phis() counts a module's OpPhi instructions. too_big() decides whether a mesh fallback compute stage should be skipped: the device is RADV on GCN 1-4, and the stage has more than 50,000 phis. 50,000 sits between the 29,400 that build in seconds and the 80,000 that never finish.
vk_progcache.cpp: each stage's phi count is recorded where its SPIR-V passes through the program cache. That is on save, which every translated program goes through, and on read. It's done there and not in vk_shader.cpp because vk_shader.cpp is part of the translator stamp, and touching it would retranslate every player's programs. The record format is unchanged.
vk_main.cpp / vk_internal.h: dev.radv_gcn is set when the driver is RADV and storageInputOutput16 is false. RADV sets that feature from packed 16-bit math, which means GFX9 and later. Checked through the drm-shim: false on Polaris and Pitcairn, true on Vega 10 and Navi 10.
vk_draw.cpp fallback_compute_pipelines: before anything is built, a program with a stage over the limit goes into the same "failed" set as a pipeline the driver refuses. It logs one line, and its draws are left out (as #2382 leaves out draws whose fallback isn't ready). The precompile and the in-play builds both go through this function.
RX_VK_FALLBACK_PHI_LIMIT=<n> sets another limit, and 0 turns it off. It's documented in docs/development.md.
Tests: tests/test_fallback_size_policy.py checks the counter on well-formed, cut-short and non-SPIR-V input, checks the limit, and checks that the source still calls the policy before any fallback pipeline is built. program_cache_cleanup_test.cpp now includes the header.

What's lost: on RADV with an RX 400/500, R9 2xx/3xx or Fury card, the draws of these two mesh programs. Neither was drawn in the lab's Kokiri Forest or Hyrule Field runs, so I don't know yet which objects they are. Nothing changes on Vega and later, on AMD's Windows driver, on NVIDIA, or on Intel. The SPIR-V of every program is unchanged.

Verified

Dumped the mesh fallback SPIR-V on Linux (RADV Polaris 10 through the drm-shim, RX_VK_DUMP_MESH_SPV). Both modules pass spirv-val --target-env vulkan1.3 --uniform-buffer-standard-layout --scalar-block-layout. This PR doesn't change any SPIR-V.
Ran the precompile on the drm-shim's Polaris 10, translating only these two and the two biggest mesh programs that do build (a local-only key filter), with a 6 GB cap:
- main: OOM-killed at the 6 GB cap after 12 s.
- this PR: completes in 16 s with the two log lines (79,947 phis, the most 50,000) and the other two fallback programs built.
Ran the phi count over all ~800 mesh fallback modules: only these two are over 50,000.
Ran python -I -B tests/run_all.py fallback_size_policy program_cache_cleanup mesh_fallback mesh_fallback_failed_once program_cache_reader driver_faults run_all_declarations (8 passed), plus the program cache, prediction and recorder tests (all pass). hookshot_vk.so builds on the current main, ABI check ok.

Not tested

A real RX 580: no full cold prepare + precompile with this build yet.
Windows, and AMD's own driver (the check is RADV only).
In play: which objects the two programs draw, and so what is missing on these cards.
A Mesa release with a fixed spiller would build them. The limit would then leave them out for no reason until RX_VK_FALLBACK_PHI_LIMIT=0 or a version check is added.

## Why On an RX 580 (Polaris 10, Mesa 26.2.6 RADV), a cold `--prepare-shaders` got through the translation and the prediction, then ran out of memory in the precompile. It grew past 24 GB and was killed, on main too. The game's own first start does the same precompile, so a player with an RX 400/500-class card can't get through the first start: it pushes the desktop into swap or the OOM killer. The cause is two mesh programs' fallback compute pipelines, `93e3e999f2fb9fd8` (stage 1) and `ea4dd64df9c8f2b7` (stage 0). RADV never finishes compiling them on GCN 1-4 (GFX6-8), and its thread keeps the memory it grew to. A time limit wouldn't help for that reason. What makes them different from the other mesh programs: - **Size.** Both mesh stages are 4.3 MB of SPIR-V: about 28,000 blocks and 80,000 phis. The native programs already have about 11,000 blocks each. The next biggest mesh programs have up to 29,400 phis. - **Live values.** About 400 values are live at once (≈300 32-bit, ≈100 booleans), far more than a wave's 256 VGPRs and ~100 SGPRs. ACO has to spill tens of thousands of them. - **Not the 126 vertices.** Their max vertices/primitives of 126 isn't what sets them apart: 246 of the first 400 mesh programs the prepare logs are 126/126, and they build normally. - **Not the fallback's own code.** The CFG comes from the native program's structurization, which the native mesh path shares. I compiled the dumped modules with RADV itself, through Mesa's amdgpu drm-shim (no AMD GPU needed), under a 6 GB cap: | module | Polaris 10 / 12, Stoney, Bonaire, Pitcairn (GFX6-8) | Vega 10, Raven (GFX9) | Navi 10 / 21 | |---|---|---|---| | `ea4dd64df9c8f2b7` mesh (79,947 phis) | not done after 45-90 s, killed at the 6 GB cap (the lab: 25 GB at 150 s) | 14 s, 2.9 GB, 24,879 SGPR + 10,644 VGPR spills | 13-14 s, 2.9 GB | | `ecbbff63280b68e2` mesh (29,372 phis) | 4.5 s, 480 MB | | | | `cf987f56b971bdb3` mesh (28,506 phis) | 5.5 s, 664 MB | | | `RX_VK_NOOPT` (DISABLE_OPTIMIZATION) doesn't change it. The spiller runs either way. GFX6-8 have no scratch instructions. ACO spills there through buffer instructions, and these programs' scratch is past their offset range, so ACO takes its overflow path for every spill and reload (`aco_spill.cpp`). With this many spills it doesn't finish. That's my reading of the source, not profiled. I couldn't find a translation-side change that compiles these two normally. Their register pressure and block count come from the native program and the shared goto-elimination structurizer. Changing either would retranslate every shader for every GPU. So this PR leaves just these two out on the cards whose compiler can't build them. ## What changes - `gpu/vk/fallback_size_policy.h` (new): `spirv_phis()` counts a module's OpPhi instructions. `too_big()` decides whether a mesh fallback compute stage should be skipped: the device is RADV on GCN 1-4, and the stage has more than 50,000 phis. 50,000 sits between the 29,400 that build in seconds and the 80,000 that never finish. - `vk_progcache.cpp`: each stage's phi count is recorded where its SPIR-V passes through the program cache. That is on save, which every translated program goes through, and on read. It's done there and not in `vk_shader.cpp` because `vk_shader.cpp` is part of the translator stamp, and touching it would retranslate every player's programs. The record format is unchanged. - `vk_main.cpp` / `vk_internal.h`: `dev.radv_gcn` is set when the driver is RADV and `storageInputOutput16` is false. RADV sets that feature from packed 16-bit math, which means GFX9 and later. Checked through the drm-shim: false on Polaris and Pitcairn, true on Vega 10 and Navi 10. - `vk_draw.cpp` `fallback_compute_pipelines`: before anything is built, a program with a stage over the limit goes into the same "failed" set as a pipeline the driver refuses. It logs one line, and its draws are left out (as #2382 leaves out draws whose fallback isn't ready). The precompile and the in-play builds both go through this function. - `RX_VK_FALLBACK_PHI_LIMIT=<n>` sets another limit, and `0` turns it off. It's documented in `docs/development.md`. - Tests: `tests/test_fallback_size_policy.py` checks the counter on well-formed, cut-short and non-SPIR-V input, checks the limit, and checks that the source still calls the policy before any fallback pipeline is built. `program_cache_cleanup_test.cpp` now includes the header. **What's lost:** on RADV with an RX 400/500, R9 2xx/3xx or Fury card, the draws of these two mesh programs. Neither was drawn in the lab's Kokiri Forest or Hyrule Field runs, so I don't know yet which objects they are. Nothing changes on Vega and later, on AMD's Windows driver, on NVIDIA, or on Intel. The SPIR-V of every program is unchanged. ## Verified - Dumped the mesh fallback SPIR-V on Linux (RADV Polaris 10 through the drm-shim, `RX_VK_DUMP_MESH_SPV`). Both modules pass `spirv-val --target-env vulkan1.3 --uniform-buffer-standard-layout --scalar-block-layout`. This PR doesn't change any SPIR-V. - Ran the precompile on the drm-shim's Polaris 10, translating only these two and the two biggest mesh programs that do build (a local-only key filter), with a 6 GB cap: - main: OOM-killed at the 6 GB cap after 12 s. - this PR: completes in 16 s with the two log lines (79,947 phis, the most 50,000) and the other two fallback programs built. - Ran the phi count over all ~800 mesh fallback modules: only these two are over 50,000. - Ran `python -I -B tests/run_all.py fallback_size_policy program_cache_cleanup mesh_fallback mesh_fallback_failed_once program_cache_reader driver_faults run_all_declarations` (8 passed), plus the program cache, prediction and recorder tests (all pass). `hookshot_vk.so` builds on the current main, ABI check ok. ## Not tested - A real RX 580: no full cold prepare + precompile with this build yet. - Windows, and AMD's own driver (the check is RADV only). - In play: which objects the two programs draw, and so what is missing on these cards. - A Mesa release with a fixed spiller would build them. The limit would then leave them out for no reason until `RX_VK_FALLBACK_PHI_LIMIT=0` or a version check is added.

hckrman101 commented

2026-11-11 23:05:53 +00:00

(Migrated from github.com)

Tested on the real card: RX 580 (Polaris 10), Mesa 26.2.6 RADV, i7-9700K, 15 GB. This PR (42d84fc9) built natively, with a cold --prepare-shaders (empty caches, the precompile on), capped at 12 GB RAM + 4 GB swap.

The prepare finishes. On main it was killed in the precompile every time, past 24 GB.

Step	Time
Translation	392.6 s (38,809 programs)
Prediction	1,175.9 s (55,252 pipelines made)
Precompile	393.3 s: 17,553 graphics + 9,683 compute pipelines, 1,096 mesh fallback programs
Total	1,966 s, exit code 0

Peak memory: 10.6 GB (8.4 GB RAM + 2.1 GB swap, at the start of the precompile).

The log has RADV on GCN 1-4: yes, and exactly the two expected skip lines:

vk: fallback: the compute pipeline of 93e3e999f2fb9fd8 (stage 1 of 4, 79947 phis, the most 50000) is not built: RADV on this GCN card would not finish it (RX_VK_FALLBACK_PHI_LIMIT); its draws are left out
vk: fallback: the compute pipeline of ea4dd64df9c8f2b7 (stage 0 of 3, 79947 phis, the most 50000) is not built: RADV on this GCN card would not finish it (RX_VK_FALLBACK_PHI_LIMIT); its draws are left out

A warm second start and a play run in Kokiri and Hyrule Field follow; I'll add them here.

Tested on the real card: RX 580 (Polaris 10), Mesa 26.2.6 RADV, i7-9700K, 15 GB. This PR (42d84fc9) built natively, with a cold `--prepare-shaders` (empty caches, the precompile on), capped at 12 GB RAM + 4 GB swap. **The prepare finishes.** On main it was killed in the precompile every time, past 24 GB. | Step | Time | |---|---| | Translation | 392.6 s (38,809 programs) | | Prediction | 1,175.9 s (55,252 pipelines made) | | Precompile | **393.3 s**: 17,553 graphics + 9,683 compute pipelines, 1,096 mesh fallback programs | | Total | 1,966 s, exit code 0 | **Peak memory: 10.6 GB** (8.4 GB RAM + 2.1 GB swap, at the start of the precompile). The log has `RADV on GCN 1-4: yes`, and exactly the two expected skip lines: ``` vk: fallback: the compute pipeline of 93e3e999f2fb9fd8 (stage 1 of 4, 79947 phis, the most 50000) is not built: RADV on this GCN card would not finish it (RX_VK_FALLBACK_PHI_LIMIT); its draws are left out vk: fallback: the compute pipeline of ea4dd64df9c8f2b7 (stage 0 of 3, 79947 phis, the most 50000) is not built: RADV on this GCN card would not finish it (RX_VK_FALLBACK_PHI_LIMIT); its draws are left out ``` A warm second start and a play run in Kokiri and Hyrule Field follow; I'll add them here.

hckrman101 commented

2026-11-11 23:11:52 +00:00

(Migrated from github.com)

The rest of the RX 580 check:

Warm second start: clean, with no "did not get through the precompile" line. The two skip lines appear again at once. The precompile takes 15.8 s, 80 s for the whole start, with a 9.4 GB peak (mostly page cache).
Play (about 70 s in Kokiri, then a teleport to Hyrule Field for 60 s):
- reached play in 122 s;
- 0 draws left out, and 0 programs translated during play;
- only 867 ms of pipeline builds during play. Main's runs with the precompile skipped spent 2–4 s;
- nothing visibly missing compared with main at the same spots: Kokiri's houses, foliage, characters and fog, and the Lost Woods bridge exit.

So whatever the two skipped programs draw, it isn't in those two areas.

The rest of the RX 580 check: - **Warm second start:** clean, with no "did not get through the precompile" line. The two skip lines appear again at once. The precompile takes 15.8 s, 80 s for the whole start, with a 9.4 GB peak (mostly page cache). - **Play** (about 70 s in Kokiri, then a teleport to Hyrule Field for 60 s): - reached play in 122 s; - 0 draws left out, and 0 programs translated during play; - only 867 ms of pipeline builds during play. Main's runs with the precompile skipped spent 2–4 s; - nothing visibly missing compared with main at the same spots: Kokiri's houses, foliage, characters and fog, and the Lost Woods bridge exit. So whatever the two skipped programs draw, it isn't in those two areas.