MagicBeanDev/Hookshot

Archived

Watch

Fork 0

vk: land #792's four mesh fallback fixes (the fallback's compute pipelines crashed NVIDIA's Pascal driver) #1314

Merged

CiLiBi18 merged 4 commits from fix/mesh-fallback-spirv into main

2026-11-09 23:04:51 +00:00

CiLiBi18 commented

2026-11-09 21:33:51 +00:00

(Migrated from github.com)

Commit	What
`e6e816a`	The payload's push member index (it was the byte offset, 64), the task's started word stored as a u32, and `Aligned 4` on the fallback's PhysicalStorageBuffer accesses
`2511050`	SPIR-V that passes spirv-val (loops unrolled; `RX_VK_DUMP_MESH_SPV`)
`6be15ce`	A fallback draw after a dispatch rebuilds its descriptors
`b9d6c6f`	A fallback mesh workgroup's index is its meshlet's in the task workgroup

There was one conflict, in 6be15ce: dw is dw() on main (TLS_OBJECT). The resolution is the same change written for dw(), and the test's expected strings follow it. Every commit passes tests/test_mesh_fallback.py on its own.

The bug on Pascal

PushU64(ctx, MESH_FALLBACK_PAYLOAD_PUSH_OFFSET) passes a byte offset (64) where OpAccessChain takes a member index. The ResolutionInfo block has 5 or 6 members, so the access chain indexes past its end. NVIDIA's compiler segfaults on it. On Linux, guarded_compute catches the fault, but the replay then stops presenting for good: the picture freezes when the intro's forest scene starts, while the game's threads keep running. This matches the GTX 1060 / 1070 / 1070 Ti reports in #1001 of a black or frozen screen after the intro or at the Deku Tree.

Tested on

GTX 1070 8 GB, NVIDIA 580.178.04 (the last driver branch for Pascal), Linux (Ubuntu), v2.0.9 game folder. I only swapped bin/hookshot_vk.so: the tag v2.0.9 plus these four commits. emit_spirv_mesh.cpp is the same in v2.0.9, v2.0.11 and main.

Offline: every fallback stage, through the real driver

I wrote a small harness. It reads a stage's SPIR-V, builds a pipeline layout from its own descriptor decorations, enables every feature the GPU offers and calls vkCreateComputePipelines, each compile in a child process. It ran over every task and mesh compute stage in the program cache (1,300 fallback programs, 2,468 compute modules):

	stock v2.0.9	v2.0.9 + these four commits
`spirv-val --target-env vulkan1.3 --uniform-buffer-standard-layout`	0 / 2,468 valid	5,068 / 5,068 valid (compute, vertex and fragment stages, from `RX_VK_DUMP_MESH_SPV`)
`vkCreateComputePipelines` on the GTX 1070	2,336 SIGSEGV (every module with the index-64 access chain), 132 ok	2,468 / 2,468 ok

To check the cause on its own, I also edited the stock SPIR-V by hand, changing only the index 64 to the payload member. That made all 2,336 compile.

In game: same intro, headless, scripted A presses, 260 s

	stock v2.0.9	v2.0.9 + these four commits
`the driver faulted compiling a compute pipeline`	1 (`a5da9f84b72d1471`), then no more presents	0
where it ends	frozen on the first frame of the forest scene	reaches the Deku Tree's speech to Navi
draws a frame in the 3D scenes	5 (12 skipped) before the freeze	~700–1,060, 0 skipped, ~100 fallback dispatches
fps (1080p)	60 in the menus, then frozen	22–28 in the forest cutscenes, GPU-bound (~22 ms a frame)

The foliage, the trees and the Deku Tree are drawn (screenshots and both logs attached).

What this does not cover yet: the run ends at the Deku Tree's speech in the intro (260 s). I haven't played further or loaded a save, so the later GTX 1070 reports in #1001 are not checked: black after Link wakes up, and the hang while loading a save. Both runs log the same 16 view of texture ... (vk 139) as vk 95 ... not made lines (the BC4 view from #1101), so that view is not what froze the intro here.

Which build ran: only the v2.0.9-based hookshot_vk.so was run in game. The branch in this PR (main + the four commits, with the dw() conflict resolution) was built and passes the tests, but has not been run in game.

Overlap: #1023 (RDNA1) carries its own version of the payload member fix inside a larger change. This PR is the small, focused alternative, made of #792's commits as they were written.

Tests: python -I -B tests/run_all.py passes, except nx2/test_nup_read_name.py, which needs zstandard. That module is missing on my machine, and the test fails the same way on main. hookshot_vk.so builds on Linux (RX_VK_ONLY=1 nx2/build.py --standalone).

[hookshot_log_fixed.txt](https://github.com/user-attachments/files/34600960/hookshot_log_fixed.txt) [hookshot_log_stock_v2.0.9.txt](https://github.com/user-attachments/files/34600962/hookshot_log_stock_v2.0.9.txt)

This brings the four mesh fallback commits of the closed PR #792 (by hckrman101, found by AmiriisCustoms) onto main, as Crono141 suggested in #1001 after testing them on an RX 590 / RADV. They never came over with #779, and v2.0.11 still has the bug. This PR adds a test from a GTX 1070, the most common card in #1001's reports. Without these commits, the fallback can't run on it at all. | Commit | What | |---|---| | `e6e816a` | The payload's push member index (it was the byte offset, 64), the task's started word stored as a u32, and `Aligned 4` on the fallback's PhysicalStorageBuffer accesses | | `2511050` | SPIR-V that passes spirv-val (loops unrolled; `RX_VK_DUMP_MESH_SPV`) | | `6be15ce` | A fallback draw after a dispatch rebuilds its descriptors | | `b9d6c6f` | A fallback mesh workgroup's index is its meshlet's in the task workgroup | There was one conflict, in `6be15ce`: `dw` is `dw()` on main (TLS_OBJECT). The resolution is the same change written for `dw()`, and the test's expected strings follow it. Every commit passes `tests/test_mesh_fallback.py` on its own. ## The bug on Pascal `PushU64(ctx, MESH_FALLBACK_PAYLOAD_PUSH_OFFSET)` passes a byte offset (64) where `OpAccessChain` takes a member index. The `ResolutionInfo` block has 5 or 6 members, so the access chain indexes past its end. NVIDIA's compiler segfaults on it. On Linux, guarded_compute catches the fault, but the replay then stops presenting for good: the picture freezes when the intro's forest scene starts, while the game's threads keep running. This matches the GTX 1060 / 1070 / 1070 Ti reports in #1001 of a black or frozen screen after the intro or at the Deku Tree. ## Tested on GTX 1070 8 GB, NVIDIA 580.178.04 (the last driver branch for Pascal), Linux (Ubuntu), v2.0.9 game folder. I only swapped `bin/hookshot_vk.so`: the tag v2.0.9 plus these four commits. `emit_spirv_mesh.cpp` is the same in v2.0.9, v2.0.11 and main. ### Offline: every fallback stage, through the real driver I wrote a small harness. It reads a stage's SPIR-V, builds a pipeline layout from its own descriptor decorations, enables every feature the GPU offers and calls `vkCreateComputePipelines`, each compile in a child process. It ran over every task and mesh compute stage in the program cache (1,300 fallback programs, 2,468 compute modules): | | stock v2.0.9 | v2.0.9 + these four commits | |---|---|---| | `spirv-val --target-env vulkan1.3 --uniform-buffer-standard-layout` | 0 / 2,468 valid | 5,068 / 5,068 valid (compute, vertex and fragment stages, from `RX_VK_DUMP_MESH_SPV`) | | `vkCreateComputePipelines` on the GTX 1070 | **2,336 SIGSEGV** (every module with the index-64 access chain), 132 ok | **2,468 / 2,468 ok** | To check the cause on its own, I also edited the stock SPIR-V by hand, changing only the index 64 to the payload member. That made all 2,336 compile. ### In game: same intro, headless, scripted A presses, 260 s | | stock v2.0.9 | v2.0.9 + these four commits | |---|---|---| | `the driver faulted compiling a compute pipeline` | 1 (`a5da9f84b72d1471`), then no more presents | 0 | | where it ends | frozen on the first frame of the forest scene | reaches the Deku Tree's speech to Navi | | draws a frame in the 3D scenes | 5 (12 skipped) before the freeze | ~700–1,060, 0 skipped, ~100 fallback dispatches | | fps (1080p) | 60 in the menus, then frozen | 22–28 in the forest cutscenes, GPU-bound (~22 ms a frame) | The foliage, the trees and the Deku Tree are drawn (screenshots and both logs attached). **What this does not cover yet:** the run ends at the Deku Tree's speech in the intro (260 s). I haven't played further or loaded a save, so the later GTX 1070 reports in #1001 are not checked: black after Link wakes up, and the hang while loading a save. Both runs log the same 16 `view of texture ... (vk 139) as vk 95 ... not made` lines (the BC4 view from #1101), so that view is not what froze the intro here. **Which build ran:** only the v2.0.9-based `hookshot_vk.so` was run in game. The branch in this PR (main + the four commits, with the `dw()` conflict resolution) was built and passes the tests, but has not been run in game. **Overlap:** #1023 (RDNA1) carries its own version of the payload member fix inside a larger change. This PR is the small, focused alternative, made of #792's commits as they were written. Tests: `python -I -B tests/run_all.py` passes, except `nx2/test_nup_read_name.py`, which needs `zstandard`. That module is missing on my machine, and the test fails the same way on main. `hookshot_vk.so` builds on Linux (`RX_VK_ONLY=1 nx2/build.py --standalone`). <img width="960" height="540" alt="fixed_deku_tree" src="https://github.com/user-attachments/assets/2ea5a24d-16a3-4295-8182-0617da8e796b" /> <img width="960" height="540" alt="fixed_forest" src="https://github.com/user-attachments/assets/24c2402d-0b64-4c07-b262-beef162153c8" /> <img width="960" height="540" alt="stock_v2 0 9_frozen_forest" src="https://github.com/user-attachments/assets/903f8375-5375-4180-85e3-a3bb88efa9d5" /> [hookshot_log_fixed.txt](https://github.com/user-attachments/files/34600960/hookshot_log_fixed.txt) [hookshot_log_stock_v2.0.9.txt](https://github.com/user-attachments/files/34600962/hookshot_log_stock_v2.0.9.txt)

royalsaltmerchant commented

2026-11-09 22:28:48 +00:00

(Migrated from github.com)

Additional Windows save-load test for #1314 on a GTX 1070 8 GB (NVIDIA 581.63, Windows 11 build 26200, i7-7700).

Build: PR head 14e0865240f8a6d32677a655646dbeb92a5e6505 built on Windows with LLVM-MinGW (RX_VK_ONLY=1, nx2/build.py --standalone); tests/test_mesh_fallback.py passed. The PR head's renderer interface differs from v2.0.11, so for the game run I applied the PR's patch unchanged to the v2.0.11 tag, built that DLL, and ran it with the existing v2.0.11 executable and a copied save profile. The fallback test passed on that source as well.
Reproduction: Continue into the same imported save that black-screened on stock v2.0.11. A separate diagnostic build of the stock renderer caught a Windows exception in vkCreateComputePipelines for program 6ea0c171f1a7b6d8 (NVN task stage 6, Hookshot marker -77).
With the PR patch: The run reached that same program, and all four modules it dumped (task compute, mesh compute, vertex, fragment) passed SPIRV-Tools v2026.4 with --target-env vulkan1.3 --uniform-buffer-standard-layout. I saw no repeat of the pipeline fault or device loss in the observed log. With startup shader preparation skipped for this test, presentation slowed sharply around loading, then resumed at about 33 FPS in a subsequent five-second window.

This confirms frame presentation recovered past the previous failure on Windows. It does not establish visual correctness or broader stability. Build and shader analysis were assisted by Codex.

Additional Windows save-load test for #1314 on a GTX 1070 8 GB (NVIDIA 581.63, Windows 11 build 26200, i7-7700). - **Build:** PR head `14e0865240f8a6d32677a655646dbeb92a5e6505` built on Windows with LLVM-MinGW (`RX_VK_ONLY=1`, `nx2/build.py --standalone`); `tests/test_mesh_fallback.py` passed. The PR head's renderer interface differs from v2.0.11, so for the game run I applied the PR's patch unchanged to the v2.0.11 tag, built that DLL, and ran it with the existing v2.0.11 executable and a copied save profile. The fallback test passed on that source as well. - **Reproduction:** Continue into the same imported save that black-screened on stock v2.0.11. A separate diagnostic build of the stock renderer caught a Windows exception in `vkCreateComputePipelines` for program `6ea0c171f1a7b6d8` (NVN task stage 6, Hookshot marker `-77`). - **With the PR patch:** The run reached that same program, and all four modules it dumped (task compute, mesh compute, vertex, fragment) passed SPIRV-Tools v2026.4 with `--target-env vulkan1.3 --uniform-buffer-standard-layout`. I saw no repeat of the pipeline fault or device loss in the observed log. With startup shader preparation skipped for this test, presentation slowed sharply around loading, then resumed at about 33 FPS in a subsequent five-second window. This confirms frame presentation recovered past the previous failure on Windows. It does not establish visual correctness or broader stability. Build and shader analysis were assisted by Codex.

hckrman101 (Migrated from github.com) approved these changes

2026-11-09 22:45:44 +00:00

hckrman101 (Migrated from github.com) left a comment

Nice find, and the Pascal evidence (2,336 SIGSEGV -> 0 over the real driver, plus the hand-edited index-64 control) is convincing. I read all four commits against the emitter and they hold up.

Push member: PushU64(..., ctx.mesh_fallback_payload_member_index) is right. The member is set under wants_mesh_fallback for every fallback stage, and the byte offset stays only for the host-side memcpy in vk_draw.cpp.
Strided unrolled: same behaviour as the old loop (each index is still guarded by the < count If), and it drops the mid-function OpVariable. std::max(threads, 1U) also covers mesh_threads == 0. The test's AddLocalVariable check covers that.
Started word: 1/0 u32 on the store, and the mesh epilogue compares it with zero, so both sides agree.
Slot -> meshlet in EmitIsbeRead: slot is wg * task_threads + i, so % task_threads gives i as the native path does. The threads come from the task's workgroup size, not the mesh's, which is what you want, and it's guarded for 0.
Descriptors: dispatch() calls build_descriptors for the compute program, so the dw().prog != &p check does catch a draw after a dispatch. Forcing a rebuild in draw() plus the bail-out in fallback_record is the right pairing.
RX_VK_DUMP_MESH_SPV is env-gated and harmless. The shader cache stamp hashes eden/ and vk_shader.cpp, so cached programs with the old SPIR-V are invalidated.

Overlap: #1023 carries its own versions of the same fixes (LoadWord/StoreWord, UMod by mesh_fallback_task_threads, payload member, the OpVariable placement) inside a much larger rewrite of emit_spirv_mesh.cpp; it's currently conflicting, so whichever lands second will need a rebase and a quick check that nothing is doubled up. #1317 only touches vk_draw.cpp (remembering a failed compute pipeline) and is complementary. #1341's repair-at-module-creation layer is made redundant by this: the emitters are fixed at the source.

Nits, none blocking:

No CI runs are attached to this head, so the green status is from your local run_all.py and royalsaltmerchant's Windows build.
The fourth commit's mesh_fallback_task_threads isn't in the program-cache record, but that's fine since the stamp change invalidates old records anyway.
As you say, main + the four commits hasn't been run in game, only v2.0.9 + the patch (and royalsaltmerchant's v2.0.11 run). The dw() resolution is a straightforward translation, so I'm comfortable.

LGTM.

Nice find, and the Pascal evidence (2,336 SIGSEGV -> 0 over the real driver, plus the hand-edited index-64 control) is convincing. I read all four commits against the emitter and they hold up. - Push member: `PushU64(..., ctx.mesh_fallback_payload_member_index)` is right. The member is set under `wants_mesh_fallback` for every fallback stage, and the byte offset stays only for the host-side memcpy in vk_draw.cpp. - `Strided` unrolled: same behaviour as the old loop (each index is still guarded by the `< count` If), and it drops the mid-function OpVariable. `std::max(threads, 1U)` also covers `mesh_threads == 0`. The test's `AddLocalVariable` check covers that. - Started word: 1/0 u32 on the store, and the mesh epilogue compares it with zero, so both sides agree. - Slot -> meshlet in `EmitIsbeRead`: slot is `wg * task_threads + i`, so `% task_threads` gives `i` as the native path does. The threads come from the task's workgroup size, not the mesh's, which is what you want, and it's guarded for 0. - Descriptors: `dispatch()` calls `build_descriptors` for the compute program, so the `dw().prog != &p` check does catch a draw after a dispatch. Forcing a rebuild in `draw()` plus the bail-out in `fallback_record` is the right pairing. - `RX_VK_DUMP_MESH_SPV` is env-gated and harmless. The shader cache stamp hashes `eden/` and `vk_shader.cpp`, so cached programs with the old SPIR-V are invalidated. Overlap: #1023 carries its own versions of the same fixes (`LoadWord`/`StoreWord`, `UMod` by `mesh_fallback_task_threads`, payload member, the OpVariable placement) inside a much larger rewrite of `emit_spirv_mesh.cpp`; it's currently conflicting, so whichever lands second will need a rebase and a quick check that nothing is doubled up. #1317 only touches `vk_draw.cpp` (remembering a failed compute pipeline) and is complementary. #1341's repair-at-module-creation layer is made redundant by this: the emitters are fixed at the source. Nits, none blocking: - No CI runs are attached to this head, so the green status is from your local `run_all.py` and royalsaltmerchant's Windows build. - The fourth commit's `mesh_fallback_task_threads` isn't in the program-cache record, but that's fine since the stamp change invalidates old records anyway. - As you say, main + the four commits hasn't been run in game, only v2.0.9 + the patch (and royalsaltmerchant's v2.0.11 run). The `dw()` resolution is a straightforward translation, so I'm comfortable. LGTM.

CiLiBi18 commented

2026-11-09 23:45:40 +00:00

(Migrated from github.com)

Thanks for merging this and #1317. Here is a longer play test on the card it was meant for, as a reference for other GTX 10xx owners.

Setup: v2.0.11 + this PR + #1317, GTX 1070 8 GB / NVIDIA 580.178.04, i7-7700HQ (4 cores), Linux, output 1920x1080 at 75% resolution. The last session was 40 minutes through Kokiri Forest up to the Deku Tree, after four shorter ones.

Stable: 0 driver faulted, 0 failed fallback pipelines, no crash or freeze in any session. Stock v2.0.9–v2.0.11 froze at the intro on this card.
Frame rate: median 25 fps over the 475 five-second windows of the last session, 68% of them between 20 and 30 fps and 14% under 20. CPU replay and GPU cost about the same: submit 22 ms a frame (p90 36 ms) and GPU time 22 ms (p90 29 ms), with ~790 draws and ~100 fallback dispatches a frame.
Stutters come mostly from first-time pipelines: 905 graphics pipelines were compiled during that session (215 s of compile time in all). Windows with more than 1 s of compiling had a median of 21 fps, against 27 fps without. They are kept, though: the start-up precompile went 364 → 1,251 → 1,418 → 1,536 → 2,441 graphics pipelines over the five sessions, so places already visited get smoother. The other drops (10–15 fps) are frames with 2,500–4,000 draws.

Still wrong on this card, not from this PR:

Some mesh geometry isn't drawn but is still solid: Link's house door is missing, Link climbs invisible objects, and a large black faceted shape covers part of the house.
Update: #1023 fixes this on the GTX 1070. With it, the invisible walls and the push block inside the Deku Tree are drawn (details in #1023).
RX_VK_MIRROR=0 makes it much worse on NVIDIA, with white foliage silhouettes and large black triangles. Mirrors are needed here.

For GTX 10xx owners: #1023 is the next piece needed for a correct picture on this card.

Thanks for merging this and #1317. Here is a longer play test on the card it was meant for, as a reference for other GTX 10xx owners. **Setup:** v2.0.11 + this PR + #1317, GTX 1070 8 GB / NVIDIA 580.178.04, i7-7700HQ (4 cores), Linux, output 1920x1080 at 75% resolution. The last session was 40 minutes through Kokiri Forest up to the Deku Tree, after four shorter ones. - **Stable:** 0 `driver faulted`, 0 failed fallback pipelines, no crash or freeze in any session. Stock v2.0.9–v2.0.11 froze at the intro on this card. - **Frame rate:** median 25 fps over the 475 five-second windows of the last session, 68% of them between 20 and 30 fps and 14% under 20. CPU replay and GPU cost about the same: submit 22 ms a frame (p90 36 ms) and GPU time 22 ms (p90 29 ms), with ~790 draws and ~100 fallback dispatches a frame. - **Stutters come mostly from first-time pipelines:** 905 graphics pipelines were compiled during that session (215 s of compile time in all). Windows with more than 1 s of compiling had a median of 21 fps, against 27 fps without. They are kept, though: the start-up precompile went 364 → 1,251 → 1,418 → 1,536 → 2,441 graphics pipelines over the five sessions, so places already visited get smoother. The other drops (10–15 fps) are frames with 2,500–4,000 draws. **Still wrong on this card, not from this PR:** - Some mesh geometry isn't drawn but is still solid: Link's house door is missing, Link climbs invisible objects, and a large black faceted shape covers part of the house. - **Update:** #1023 fixes this on the GTX 1070. With it, the invisible walls and the push block inside the Deku Tree are drawn (details in #1023). - `RX_VK_MIRROR=0` makes it much worse on NVIDIA, with white foliage silhouettes and large black triangles. Mirrors are needed here. For GTX 10xx owners: #1023 is the next piece needed for a correct picture on this card.