Reland the mesh shader fallback with its graphics stage filter limited to fallback programs #779

Merged

Nitnek merged 2 commits from fix/mesh-fallback-reland into main

2026-11-09 04:59:40 +00:00

Nitnek commented

2026-11-08 21:01:40 +00:00

(Migrated from github.com)

Relands the mesh shader fallback for GPUs without mesh shaders (#640, ea4eced). 806c566 reverted it because it lost the device on an RTX 4060 Ti, which has mesh shaders. This PR also fixes that cause, which I described in #640.

What changed

feat(vk): reland ...: reverts 806c566 and brings ea4eced back as it was (shayder's code). The CI step that b11058c removed is not restored, because tests/run_all.py now picks up tests/test_mesh_fallback.py by itself.
fix(vk): leave out task and mesh stages only for fallback programs: build_graphics() skipped every COMPUTE, TASK and MESH stage when it built a graphics pipeline:
```
if (st.vk_stage & (VK_SHADER_STAGE_COMPUTE_BIT | VK_SHADER_STAGE_TASK_BIT_EXT | VK_SHADER_STAGE_MESH_BIT_EXT))
    continue;
```
On a GPU with mesh shaders, the task and mesh stages are the pipeline's own, so every native mesh program got built with only its fragment stage. In the fallback, those stages are already VK_SHADER_STAGE_COMPUTE_BIT, so the TASK/MESH bits only ever matched native programs. The skip is now limited to p.fallback. tests/test_mesh_fallback.py checks it: it fails without the fix and passes with it.

Testing

RTX 3070 8 GB, Windows 11, NVIDIA driver with mesh shaders (mesh shaders yes in the log).

build	where	GPU time a frame	fps	grass / leaves	device lost
`2ff3cff` (#640 in, before the revert)	Kakariko gate, loaded soon after the title screen	11 ms, then 157–240 ms after ~30 s	6	missing	in several runs
`2ff3cff` + the fix	same	13 ms, steady over ~4 min	44	present	no
this branch (on `b179914`)	from the Hyrule Castle gate to Kakariko, played by hand (two sessions)	p50 23–24 ms, max 47 ms	p50 26–32	present	no

The last row's fps are lower because those are heavier areas played by hand, not the fixed Kakariko view.

tests/run_all.py on Windows, with llvm-mingw's clang on PATH: 96 passed, 2 failed, 10 skipped. The two failures, nx2/test_nup_paths.py (symlinks) and tests/test_fps_overlay_order.py (g++ -Werror), fail the same way on main without this branch on this PC.

Not tested: the fallback path itself, since this GPU has mesh shaders. The RX 5500 XT from #640 or another RDNA1 card would be the test for that part. RX_VK_NO_MESH_FALLBACK=1 still turns it off.

Fixes the device loss behind 806c566, refs #640.

🤖 Generated with Claude Code

Relands the mesh shader fallback for GPUs without mesh shaders (#640, `ea4eced`). `806c566` reverted it because it lost the device on an RTX 4060 Ti, which has mesh shaders. This PR also fixes that cause, which I described in https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/640. ## What changed 1. `feat(vk): reland ...`: reverts `806c566` and brings `ea4eced` back as it was (shayder's code). The CI step that `b11058c` removed is not restored, because `tests/run_all.py` now picks up `tests/test_mesh_fallback.py` by itself. 2. `fix(vk): leave out task and mesh stages only for fallback programs`: `build_graphics()` skipped every COMPUTE, TASK and MESH stage when it built a graphics pipeline: ```cpp if (st.vk_stage & (VK_SHADER_STAGE_COMPUTE_BIT | VK_SHADER_STAGE_TASK_BIT_EXT | VK_SHADER_STAGE_MESH_BIT_EXT)) continue; ``` On a GPU with mesh shaders, the task and mesh stages are the pipeline's own, so every native mesh program got built with only its fragment stage. In the fallback, those stages are already `VK_SHADER_STAGE_COMPUTE_BIT`, so the TASK/MESH bits only ever matched native programs. The skip is now limited to `p.fallback`. `tests/test_mesh_fallback.py` checks it: it fails without the fix and passes with it. ## Testing RTX 3070 8 GB, Windows 11, NVIDIA driver with mesh shaders (`mesh shaders yes` in the log). | build | where | GPU time a frame | fps | grass / leaves | device lost | |---|---|---|---|---|---| | `2ff3cff` (#640 in, before the revert) | Kakariko gate, loaded soon after the title screen | 11 ms, then 157–240 ms after ~30 s | 6 | missing | in several runs | | `2ff3cff` + the fix | same | 13 ms, steady over ~4 min | 44 | present | no | | this branch (on `b179914`) | from the Hyrule Castle gate to Kakariko, played by hand (two sessions) | p50 23–24 ms, max 47 ms | p50 26–32 | present | no | The last row's fps are lower because those are heavier areas played by hand, not the fixed Kakariko view. `tests/run_all.py` on Windows, with llvm-mingw's clang on PATH: 96 passed, 2 failed, 10 skipped. The two failures, `nx2/test_nup_paths.py` (symlinks) and `tests/test_fps_overlay_order.py` (`g++ -Werror`), fail the same way on `main` without this branch on this PC. **Not tested:** the fallback path itself, since this GPU has mesh shaders. The RX 5500 XT from #640 or another RDNA1 card would be the test for that part. `RX_VK_NO_MESH_FALLBACK=1` still turns it off. Fixes the device loss behind `806c566`, refs #640. 🤖 Generated with [Claude Code](https://claude.com/claude-code)