Work around AMD Vulkan task shader timeouts and restore cloth rendering #1747

Merged

lucasmsqt merged 9 commits from fix/amd-gpu-rx6700xt into main

2026-11-10 17:10:36 +00:00

lucasmsqt commented

2026-11-10 15:25:20 +00:00

(Migrated from github.com)

Changes

Use the compute fallback by default for task + mesh draws on Windows Navi 22. Mesh-only draws keep the native path.
Use main’s compact allocator from #1511, with a 64 MiB records buffer plus the task payload. Preserve the starting workgroup across direct-draw batches and GPU-driven indirect draws.
Keep main’s background pipeline compilation and graphics pipeline library cache. Retry eligible graphics pipelines without optimization after VK_ERROR_UNKNOWN.
Disable optimization before the first compile of the affected cloth program on AMD’s proprietary Vulkan driver on Windows. Frame captures showed that the depth pass completed while the color draw was still waiting for its pipeline.

Experimental fallback testing

Set RX_VK_TASK_MESH_FALLBACK=1 in hookshot_env.txt to opt in on Vulkan devices that expose task shaders, mesh shaders, and buffer device addresses. The override works across GPU vendors, drivers, and operating systems, including SteamOS with RADV. Set it to 0 to use the native path on those devices.

Validation

Windows, RX 6700 XT, AMD driver 26.10.1. Local test builds restored the affected pot’s cloth in Kokiri Forest and stopped the distant-tree flicker in Hyrule Field.
The final Windows Vulkan DLL built successfully. The focused mesh fallback and pipeline cache tests passed. Four failures in the full Windows suite also reproduced on the upstream base.
A controlled native-versus-fallback crash comparison and SteamOS/RADV hardware testing are still pending.

Follow-up to #1413. Related to #996 and #1540. Integrates #1510 and #1511.

The RX 6700 XT can still hit driver timeouts in the native task + mesh path on Windows. A separate issue makes the cloth on some Kokiri Forest pots glow white when its color pipeline gets stuck compiling. ## Changes - Use the compute fallback by default for task + mesh draws on Windows Navi 22. Mesh-only draws keep the native path. - Use main’s compact allocator from #1511, with a 64 MiB records buffer plus the task payload. Preserve the starting workgroup across direct-draw batches and GPU-driven indirect draws. - Keep main’s background pipeline compilation and graphics pipeline library cache. Retry eligible graphics pipelines without optimization after `VK_ERROR_UNKNOWN`. - Disable optimization before the first compile of the affected cloth program on AMD’s proprietary Vulkan driver on Windows. Frame captures showed that the depth pass completed while the color draw was still waiting for its pipeline. ## Experimental fallback testing Set `RX_VK_TASK_MESH_FALLBACK=1` in `hookshot_env.txt` to opt in on Vulkan devices that expose task shaders, mesh shaders, and buffer device addresses. The override works across GPU vendors, drivers, and operating systems, including SteamOS with RADV. Set it to `0` to use the native path on those devices. ## Validation - Windows, RX 6700 XT, AMD driver 26.10.1. Local test builds restored the affected pot’s cloth in Kokiri Forest and stopped the distant-tree flicker in Hyrule Field. - The final Windows Vulkan DLL built successfully. The focused mesh fallback and pipeline cache tests passed. Four failures in the full Windows suite also reproduced on the upstream base. - A controlled native-versus-fallback crash comparison and SteamOS/RADV hardware testing are still pending. Follow-up to #1413. Related to #996 and #1540. Integrates #1510 and #1511.

hckrman101 (Migrated from github.com) approved these changes

2026-11-10 16:42:21 +00:00

hckrman101 (Migrated from github.com) left a comment

A few non-blocking notes:

nit: fallback_indirect_arena is thread-local and allocates about 392 MB (384 MB records + 8 MB payload) per thread on the first GPU-counted task draw. That is fine on a 12 GB RX 6700 XT, but this path also runs on every GPU that lacks VK_EXT_mesh_shader (the dev.mesh_fallback case), including small-VRAM cards, and with the recorder thread now on main it can be one arena per recording thread. Might be worth sharing one arena or sizing it down if VRAM is tight, with the existing 'unavailable, using the small buffers' fallback covering allocation failure.
nit: the cloth workaround is keyed on the single program hash 0x5fa56485a6b07619. That is OK as a targeted fix, but it will silently stop applying if the program's key ever changes (e.g. a translation change that alters the hash). The VK_ERROR_UNKNOWN retry is the more general safety net.
nit: the RX_VK_NOOPT/task_mesh_fallback DISABLE_OPTIMIZATION on all fallback compute pipelines is a perf trade-off on those paths. Fine given the timeouts, just worth remembering when comparing against native.
The PR description says the native-vs-fallback crash comparison is still pending, so that is still unverified. No CI checks are reported on the branch, so I couldn't confirm the Python/C++ tests in CI; please make sure they run once checks start.

Overlap: #1510/#1511/#1731 also touch the mesh/task fallback, so expect some merge friction there.

Read through the whole diff and it hangs together. I checked the push-constant layout (capacity/vertices/indirect in/out at +16/+20/+24/+32, all under kPushSize, and the generated vertex stage gets the capacity member too), the indirect launch args (dispatch at byte 0, VkDrawIndirectCommand at byte 16 with the trailing words zeroed by the 32-byte fill), the shared per-task-group payload copy, the split of direct draws into batches with first_task carried through, and the runtime/seed/disk-stamp selection (`uses_mesh_fallback`, stamp bit 4). The test in test_fallback_buffer_cleanup matches the new payload sizing (capacity 4, 2 threads gives 32 bytes). Nothing I'd block on. A few non-blocking notes: - nit: `fallback_indirect_arena` is thread-local and allocates about 392 MB (384 MB records + 8 MB payload) per thread on the first GPU-counted task draw. That is fine on a 12 GB RX 6700 XT, but this path also runs on every GPU that lacks VK_EXT_mesh_shader (the `dev.mesh_fallback` case), including small-VRAM cards, and with the recorder thread now on main it can be one arena per recording thread. Might be worth sharing one arena or sizing it down if VRAM is tight, with the existing 'unavailable, using the small buffers' fallback covering allocation failure. - nit: the cloth workaround is keyed on the single program hash `0x5fa56485a6b07619`. That is OK as a targeted fix, but it will silently stop applying if the program's key ever changes (e.g. a translation change that alters the hash). The `VK_ERROR_UNKNOWN` retry is the more general safety net. - nit: the `RX_VK_NOOPT`/`task_mesh_fallback` DISABLE_OPTIMIZATION on all fallback compute pipelines is a perf trade-off on those paths. Fine given the timeouts, just worth remembering when comparing against native. - The PR description says the native-vs-fallback crash comparison is still pending, so that is still unverified. No CI checks are reported on the branch, so I couldn't confirm the Python/C++ tests in CI; please make sure they run once checks start. Overlap: #1510/#1511/#1731 also touch the mesh/task fallback, so expect some merge friction there.

Que20 commented

2026-11-10 16:59:07 +00:00

(Migrated from github.com)

Thanks for working on this!

As you might have seen on the open issues, there is some similar GPU hangs on AMD handhelds w/ SteamOS.

Currently, the only viable solution is RX_VK_NO_MESH=1 that prevents the GPU hang, but causes missing geometry and significantly degraded performance. Adding RX_VK_PER_QUEUE_FENCES=0 also fixes the flickering, making the game playable, although far from ideal.

Would it be possible to expose the task/mesh compute fallback introduced in this PR through an experimental environment flag, independently of the GPU vendor/driver or OS? Some RX_VK_TASK_MESH_FALLBACK or so would be great.

I'd be happy to test it on RADV and report back on stability, rendering quality, and performance. This could potentially help other AMD handheld users facing similar issues.

Thanks for working on this! As you might have seen on the open issues, there is some similar GPU hangs on AMD handhelds w/ SteamOS. Currently, the only viable solution is `RX_VK_NO_MESH=1` that prevents the GPU hang, but causes missing geometry and significantly degraded performance. Adding `RX_VK_PER_QUEUE_FENCES=0` also fixes the flickering, making the game playable, although far from ideal. Would it be possible to expose the task/mesh compute fallback introduced in this PR through an experimental environment flag, independently of the GPU vendor/driver or OS? Some RX_VK_TASK_MESH_FALLBACK or so would be great. I'd be happy to test it on RADV and report back on stability, rendering quality, and performance. This could potentially help other AMD handheld users facing similar issues.

lucasmsqt commented

2026-11-10 17:12:23 +00:00

(Migrated from github.com)

Thanks for the review and for offering to test on RADV! I updated the PR to 6c07058, and it has now been merged into main as b235c4d.

The updated code uses main’s compact mesh allocator from #1511. The old 392 MiB arena was replaced by a 64 MiB records buffer plus the task payload. The update preserves the starting workgroup for direct and indirect draws, keeps main’s graphics pipeline library cache, and retains the targeted workaround for the cloth shader on AMD’s Windows Vulkan driver. I verified that the distant trees in Hyrule Field stopped flickering on my RX 6700 XT.

For SteamOS and RADV, a build from main that includes b235c4d already supports RX_VK_TASK_MESH_FALLBACK=1. Add it to hookshot_env.txt next to image.bin and remove RX_VK_NO_MESH=1, which skips those draws. The startup log should say task + mesh programs use the compute fallback.

For an A/B comparison, use RX_VK_TASK_MESH_FALLBACK=0 for the native path, keep RX_VK_PER_QUEUE_FENCES at the same setting in both runs, and let each mode warm its shader cache before comparing FPS. A report with the handheld model, SteamOS and RADV versions, the scene tested, and hookshot_log.txt would help compare hangs, missing geometry, flicker, and performance.

Thanks for the review and for offering to test on RADV! I updated the PR to `6c07058`, and it has now been merged into `main` as `b235c4d`. The updated code uses main’s compact mesh allocator from #1511. The old 392 MiB arena was replaced by a 64 MiB records buffer plus the task payload. The update preserves the starting workgroup for direct and indirect draws, keeps main’s graphics pipeline library cache, and retains the targeted workaround for the cloth shader on AMD’s Windows Vulkan driver. I verified that the distant trees in Hyrule Field stopped flickering on my RX 6700 XT. For SteamOS and RADV, a build from `main` that includes `b235c4d` already supports `RX_VK_TASK_MESH_FALLBACK=1`. Add it to `hookshot_env.txt` next to `image.bin` and remove `RX_VK_NO_MESH=1`, which skips those draws. The startup log should say `task + mesh programs use the compute fallback`. For an A/B comparison, use `RX_VK_TASK_MESH_FALLBACK=0` for the native path, keep `RX_VK_PER_QUEUE_FENCES` at the same setting in both runs, and let each mode warm its shader cache before comparing FPS. A report with the handheld model, SteamOS and RADV versions, the scene tested, and `hookshot_log.txt` would help compare hangs, missing geometry, flicker, and performance.