vk: RX_VK_MESH_NOCULL, the Steam Deck's task shader hang without leaving models out (#817) #1300

Merged

KewinBR merged 1 commit from deck-mesh-nocull into main

2026-11-09 23:14:49 +00:00

KewinBR commented

2026-11-09 21:16:48 +00:00

(Migrated from github.com)

What

A new renderer switch, RX_VK_MESH_NOCULL=<task shader code key prefix,...>, turns off the culling inside the task shaders it names. On a Steam Deck, RX_VK_MESH_NOCULL=2c56dc1006942f74 avoids the GPU hang of #817 with every model drawn, where RX_VK_SKIP_CODE=2c56dc1006942f74 (#1165) avoids it by leaving out the ~30 programs that use that task shader, and the houses, platforms and rupees they draw.

gpu/vk/vk_shader.cpp: mesh_nocull() runs on the translated task program, before dump_ir. It looks for INotEqual(BitwiseOr32(x, BitwiseAnd32(4, GetCbufU32(3, 2320))), 0), replaces its uses with true and runs DeadCodeEliminationPass. If the pattern is not found, the program is left as it is and the log says so. It logs the first 8 task shaders it met.
readme.md: the Deck troubleshooting row now points to this switch first, and the switch is described under Switches for testing, next to RX_VK_SKIP_CODE.

Why it works

Task shader 2c56dc1006942f74 projects each meshlet's bounding sphere onto a grid of c3[3088] x c3[3092] cells and, for every covered cell, reads four words from c3[3120 + cell * 16] by a dynamic index (two nested loops). On RDNA 2 some draws run for seconds, and past 10 s the kernel resets the ring. The shader's answer is:

visible = (result & 1) | (c3[2320] & 4)

Bit 4 of c3[2320] is the game's own "draw everything" switch. Making that comparison true lets dead code elimination take the grid walk with it: no loops are left, every meshlet goes to the mesh stage, and the rasterizer clips what is off screen. It costs some GPU time on cards that cull quickly, so it is off unless set. The name is under RX_VK_MESH, so the program cache starts again when it changes.

Tested

Steam Deck (RADV VANGOGH), SteamOS, Game Mode, HOS_HANDHELD=1, RX_VK_MESH_OPT=1, the memory caps from #817, with the same change on by default in a local build:

v2.0.7: 15 minutes driven through Kokiri Forest (scripted input), 0 stalls, 0 draws left out, about 30 fps. Before it, the same route hung within seconds. A player then went from the start to the Kokiri Sword in Game Mode: the houses, the platforms over the pond and the rupees that RX_VK_SKIP_CODE hides are all there.
v2.0.11: 12 minutes turning the camera and walking in Kokiri Forest, 0 stalls, 0 device lost.
This branch (main with the switch off by default) builds with the Linux AppImage toolchain, and tests/run_all.py gives the same results as main without the change. I did not run this exact build on the Deck: the runs above are the same function, on by default.

The flashing on integrated GPUs (#1186, #1126) is a separate problem and is not touched here.

Generated with Claude Code

**What** A new renderer switch, `RX_VK_MESH_NOCULL=<task shader code key prefix,...>`, turns off the culling inside the task shaders it names. On a Steam Deck, `RX_VK_MESH_NOCULL=2c56dc1006942f74` avoids the GPU hang of #817 with every model drawn, where `RX_VK_SKIP_CODE=2c56dc1006942f74` (#1165) avoids it by leaving out the ~30 programs that use that task shader, and the houses, platforms and rupees they draw. - `gpu/vk/vk_shader.cpp`: `mesh_nocull()` runs on the translated task program, before `dump_ir`. It looks for `INotEqual(BitwiseOr32(x, BitwiseAnd32(4, GetCbufU32(3, 2320))), 0)`, replaces its uses with `true` and runs `DeadCodeEliminationPass`. If the pattern is not found, the program is left as it is and the log says so. It logs the first 8 task shaders it met. - `readme.md`: the Deck troubleshooting row now points to this switch first, and the switch is described under *Switches for testing*, next to `RX_VK_SKIP_CODE`. **Why it works** Task shader `2c56dc1006942f74` projects each meshlet's bounding sphere onto a grid of `c3[3088]` x `c3[3092]` cells and, for every covered cell, reads four words from `c3[3120 + cell * 16]` by a dynamic index (two nested loops). On RDNA 2 some draws run for seconds, and past 10 s the kernel resets the ring. The shader's answer is: ``` visible = (result & 1) | (c3[2320] & 4) ``` Bit 4 of `c3[2320]` is the game's own "draw everything" switch. Making that comparison true lets dead code elimination take the grid walk with it: no loops are left, every meshlet goes to the mesh stage, and the rasterizer clips what is off screen. It costs some GPU time on cards that cull quickly, so it is off unless set. The name is under `RX_VK_MESH`, so the program cache starts again when it changes. **Tested** Steam Deck (RADV VANGOGH), SteamOS, Game Mode, `HOS_HANDHELD=1`, `RX_VK_MESH_OPT=1`, the memory caps from #817, with the same change on by default in a local build: - v2.0.7: 15 minutes driven through Kokiri Forest (scripted input), 0 stalls, 0 draws left out, about 30 fps. Before it, the same route hung within seconds. A player then went from the start to the Kokiri Sword in Game Mode: the houses, the platforms over the pond and the rupees that `RX_VK_SKIP_CODE` hides are all there. - v2.0.11: 12 minutes turning the camera and walking in Kokiri Forest, 0 stalls, 0 device lost. - This branch (main with the switch off by default) builds with the Linux AppImage toolchain, and `tests/run_all.py` gives the same results as main without the change. I did not run this exact build on the Deck: the runs above are the same function, on by default. The flashing on integrated GPUs (#1186, #1126) is a separate problem and is not touched here. Generated with [Claude Code](https://claude.com/claude-code)

hckrman101 (Migrated from github.com) approved these changes

2026-11-09 23:07:57 +00:00

hckrman101 (Migrated from github.com) left a comment

Looks good. I checked the pieces that could go wrong:

The pattern match is strict (INotEqual of BitwiseOr32 against 0, with a BitwiseAnd32 of 4 and GetCbufU32 c3[2320] inside), and it logs and leaves the program untouched when nothing matches. Other task shaders are not affected.
The switch is off unless set. The program cache key already hashes every RX_VK_MESH* env var (vk_shader.cpp, the prefix list in the cache hash), so RX_VK_MESH_NOCULL invalidates the cache as the comment says.
passes.h is included for DeadCodeEliminationPass, and the call site sits before dump_ir, so dumps show the rewritten IR.
The readme row and the Switches section match the code.

Nit: the description says "no loops are left". DCE does not remove structured control flow, so I would expect the loop skeleton to stay with an empty body, which is still cheap. It matches your Deck results either way, so just worth wording carefully.

No CI ran on this branch, and the PR itself notes the exact build was not run on the Deck (only the same function enabled by default), which is fine for an opt-in switch.

Looks good. I checked the pieces that could go wrong: - The pattern match is strict (INotEqual of BitwiseOr32 against 0, with a BitwiseAnd32 of 4 and GetCbufU32 c3[2320] inside), and it logs and leaves the program untouched when nothing matches. Other task shaders are not affected. - The switch is off unless set. The program cache key already hashes every `RX_VK_MESH*` env var (vk_shader.cpp, the prefix list in the cache hash), so `RX_VK_MESH_NOCULL` invalidates the cache as the comment says. - `passes.h` is included for `DeadCodeEliminationPass`, and the call site sits before `dump_ir`, so dumps show the rewritten IR. - The readme row and the Switches section match the code. Nit: the description says "no loops are left". DCE does not remove structured control flow, so I would expect the loop skeleton to stay with an empty body, which is still cheap. It matches your Deck results either way, so just worth wording carefully. No CI ran on this branch, and the PR itself notes the exact build was not run on the Deck (only the same function enabled by default), which is fine for an opt-in switch.