AMD GPU (RX 9070 XT): translator and binding fixes, plus findings on the bare tree canopies #138

Closed

AyyBeeDee20738 wants to merge 1 commit from amd-fixes into main

AyyBeeDee20738 commented

2026-11-07 12:41:35 +00:00

(Migrated from github.com)

Tested on an AMD GPU only (Radeon RX 9070 XT, AMD's proprietary Windows driver). Nothing here was tested on NVIDIA or Intel. Where a fix is AMD-specific it says so, and the shader changes are written to leave NVIDIA's results unchanged.

Testing on an AMD card, as the README asks. This is a day of frame-dump debugging on an RX 9070 XT, most of it on lighting-fix (da030c5). The fixes that are not yet in main are in this PR, rebased onto 4756836 and build-checked there (the renderer alone, RX_VK_ONLY=1: 0 failures, links). Nothing from the game is included: no game files, keys, logs, frame dumps or shader dumps. Programs are named by their Hookshot keys only.

Setup: AMD Radeon RX 9070 XT 16 GB, AMD Software: Adrenalin Edition 26.10.1 (Windows driver 32.0.32015.7029; Vulkan driver version 2.0.416, 0x8001a0, Vulkan 1.4), Ryzen 5 7600, Windows 11 25H2. Resolution 100%, Upscaling Off, 1440p.

In this PR (all missing from `main` as of `4756836`)

#	Change	Symptom it fixes	Files
1	Indirect branch tables whose index is clamped with `ISETP` + `SEL` against a register a `MOV32I` set, not an `IMNMX`	30 mesh programs rejected with "Failed to track indirect branch" (the NVN2 lowering produces this form); failed programs 89 → 43	`indirect_branch_table_track.cpp`, `control_flow.cpp` (the error now lists the opcodes before the jump)
2	NaN-safe float min / max / clamp / saturate (`NMin`/`NMax`), Lichii #432 / #437	AMD returns the NaN where NVIDIA returns the other operand; it spreads through the bloom as a yellow-white haze	`emit_spirv_floating_point.cpp`, `spirv_emit_context.cpp` (the f16x2/f32x2 atomic min/max loops)
3	Colour write masks leave out the components the fragment shader does not write (Lichii #353)	AMD writes its export registers' garbage there (NaN, saturated values)	`collect_shader_info_pass.cpp`, `shader_info.h` (`frag_color_components`), `vk_draw.cpp` `target_write_mask`. `RX_VK_ALL_COMPONENTS=1`: off
4	`RED` (`0x98e`) added to the opcodes whose bits 32-39 are a data register for the high-GPR rename (Lichii #149)	A reduction's data operand in a high GPR read the plain register	`lower_to_maxwell.h`
5	Storage buffer descriptors read through `stage_cbuf_word`, not `driver_bank_word`	A descriptor in `c1` (the program's constants) read 0 from the driver banks and was bound to nothing	`vk_draw.cpp` (both binding loops)
6	A shader that samples with a depth reference always gets a comparing sampler (the game's function, else LEQUAL)	A Dref sample through a non-comparing sampler is undefined in Vulkan; NVIDIA compares anyway, AMD does not. The game does sample this way (logged: compare mode 0, function 0)	`vk_draw.cpp` `sampler_for`. `RX_VK_DREF_FROM_SAMPLER=1`: as before
7	A storage-table entry whose range does not fit its pool binds smaller (down to 1 MB) instead of nothing	Entries flipped bound/unbound between frames (the "table entry UNBOUND / bound again" log), so their reads alternated between data and 0: a bright/dark flicker	`vk_draw.cpp`
8	Dynamic-resolution network `9dc98871eb7fc101` kept by default (next to the TAAU)	With Resolution Auto on AMD the 3D scene is black under the HUD: this pass resolves the 2048x1152 scene, and the upscale stub can't stand in (input and output are the same size)	`vk_shader.cpp`
9	Diagnostics, all off by default	—	`vk_draw.cpp`: `RX_VK_LOG_MESH_PROGS=1` (each mesh program's draws and indirect arguments); in a frame dump, `dump-reach` lines (each dispatch's storage/uniform buffers and translated bindings) and `mesh indirect` buffers (indirect mesh arguments read back from the GPU)

Found independently, already fixed in `main` (no action needed)

Compute shaders reading through loaded pointers. On da030c5 the volume lighting pass ee7764a6b1ef459c got 0 from all 101 of its global loads (its inputs sit behind pointers in a 64-byte storage buffer). Result: no sky ambient, dark characters, black canopies, a white haze. I fixed it with a compute pointer table; ffcdb71 / 52cebb1 fix it more generally with the page table. Confirmed that this was the main cause of the dark characters.
Lichii #410 (loaded pointers competing with a cbuf pointer): also ported here, and in ffcdb71.
VRAM: reached ~15 GB on a 16 GB card at 1440p; 4756836 caps the texture cache.

Still open: findings for you

A. Tree canopies are missing everywhere (Kokiri Forest, the Deku Tree's crown)

Compared with Switch 2 footage of the same spot, every tree is bare branches; near ferns and grass draw. What I measured:

All 41 indirect mesh draws in the scene have zero groups on the GPU. These are mesh-only programs with indirect arguments. I read the arguments back from the GPU after each pass (diagnostic 9), so this isn't guest memory lagging. Their culling compute passes (f0fed2bbccae7bdc, cb2e06ddfb9d7d27, 44511d69b52282ad, 4247f1b4c3ce6cf3, …) have those argument buffers bound and store to memory.
In f0fed2bbccae7bdc the instance counts are atomic adds/max at c0[0x90] + c1[0] + small constants. c1[0] is 12 here. Storage tracking takes c1+0 as the buffer: the biased search fails because compute descriptors start at c0+0x40, and the unbiased search accepts the first cbuf read. So the atomics target a "buffer" at address 12.
Skipping c1 in Track (my experiment, not in this PR) makes the atomics target the right buffer (c0+0x90). The counts were still 0, and other programs started flashing white, so I backed it out. The tracking is still worth a look, but the culling also rejects everything for another reason: its inputs (frustum, visibility/HiZ, instance records).
The 30 programs fixed by #1 don't draw in this scene, so they aren't the canopies.
Likely related: with no canopy, nothing shades the fog, which is part of why it looks too thick (B).

B. Fog far too thick with `RX_VK_OOT_VISUAL=0` (on `da030c5`)

Once the sky ambient worked, turning the workaround off made the fog-apply pass (1a7482c0ec9991ea) raise the scene's mean from about 0.5 to 2.7. Lichii's fog at the same spot is much lighter. The missing canopies (A) probably account for part of it.

C. Brightness pulses only while the window has focus

With focus, the scene goes bright and dark every 5-10 s. Alt-tab out, and it settles.
Captured both ways at the same spot: sky, lighting volume, lit scene, fog and bloom all differ by one factor, x2.32. The final pass's exposure (c3+0x38) is constant 1.0 in both, so the factor comes in earlier (the pre-exposure / light constants).
Frame pacing differs too: with focus 51-54 fps with many 22-30 ms frames; without focus a steady 60. (Without focus only input pauses; the renderer keeps running.)
RX_VK_SUBMIT_DELAY_US=1000 helps a lot, which fits your notes on the race with the game's threads.
AMD Anti-Lag, Chill, Boost, Fluid Motion Frames and Enhanced Sync are all off: no change.

D. Small things

Shader preparation console freezes when clicked. Windows' QuickEdit selection blocks the console write in prepare_run's progress callback, so "Preparing shaders: 0 of 42537" looks hung. All workers had finished; the main thread was waiting in WriteFile. Esc resumes it. Turning QuickEdit off on that console would avoid it: SetConsoleMode(GetStdHandle(STD_INPUT_HANDLE), ENABLE_EXTENDED_FLAGS) without ENABLE_QUICK_EDIT_MODE.
9dc98871eb7fc101 left out made Auto resolution black on AMD (#8 above); 100% resolution worked around it.

Tried with no visible effect (kept here so nobody repeats them)

FP16 off on AMD (Lichii #378's workaround); main has RX_VK_NO_F16.
Read-only storage buffers bound past their size on AMD (AMD bounds-checks, NVIDIA reads on).
Running the volume pass 53509044401d78d2 that is skipped by default (RX_VK_SKIP_PROG=-).
More pointer-table reach for task/mesh (256 MB) and more entries for compute (8 regions, 1 KB scans, 64 pointers): no pointers were left out, and the canopies stayed bare.

Happy to test anything on this card. All of it can be reproduced with the switches above and the dump trigger (RX_VK_DUMP_FILE).

🤖 Generated with Claude Code

> **Tested on an AMD GPU only** (Radeon RX 9070 XT, AMD's proprietary Windows driver). Nothing here was tested on NVIDIA or Intel. Where a fix is AMD-specific it says so, and the shader changes are written to leave NVIDIA's results unchanged. Testing on an AMD card, as the README asks. This is a day of frame-dump debugging on an RX 9070 XT, most of it on `lighting-fix` (da030c5). The fixes that are not yet in `main` are in this PR, rebased onto `4756836` and build-checked there (the renderer alone, `RX_VK_ONLY=1`: 0 failures, links). Nothing from the game is included: no game files, keys, logs, frame dumps or shader dumps. Programs are named by their Hookshot keys only. **Setup:** AMD Radeon RX 9070 XT 16 GB, AMD Software: Adrenalin Edition 26.10.1 (Windows driver 32.0.32015.7029; Vulkan driver version 2.0.416, 0x8001a0, Vulkan 1.4), Ryzen 5 7600, Windows 11 25H2. Resolution 100%, Upscaling Off, 1440p. ## In this PR (all missing from `main` as of 4756836) | # | Change | Symptom it fixes | Files | |---|---|---|---| | 1 | Indirect branch tables whose index is clamped with `ISETP` + `SEL` against a register a `MOV32I` set, not an `IMNMX` | **30 mesh programs rejected** with "Failed to track indirect branch" (the NVN2 lowering produces this form); failed programs 89 → 43 | `indirect_branch_table_track.cpp`, `control_flow.cpp` (the error now lists the opcodes before the jump) | | 2 | NaN-safe float min / max / clamp / saturate (`NMin`/`NMax`), Lichii #432 / #437 | AMD returns the NaN where NVIDIA returns the other operand; it spreads through the bloom as a yellow-white haze | `emit_spirv_floating_point.cpp`, `spirv_emit_context.cpp` (the f16x2/f32x2 atomic min/max loops) | | 3 | Colour write masks leave out the components the fragment shader does not write (Lichii #353) | AMD writes its export registers' garbage there (NaN, saturated values) | `collect_shader_info_pass.cpp`, `shader_info.h` (`frag_color_components`), `vk_draw.cpp` `target_write_mask`. `RX_VK_ALL_COMPONENTS=1`: off | | 4 | `RED` (`0x98e`) added to the opcodes whose bits 32-39 are a data register for the high-GPR rename (Lichii #149) | A reduction's data operand in a high GPR read the plain register | `lower_to_maxwell.h` | | 5 | Storage buffer descriptors read through `stage_cbuf_word`, not `driver_bank_word` | A descriptor in `c1` (the program's constants) read 0 from the driver banks and was bound to nothing | `vk_draw.cpp` (both binding loops) | | 6 | A shader that samples with a depth reference always gets a comparing sampler (the game's function, else LEQUAL) | A Dref sample through a non-comparing sampler is undefined in Vulkan; NVIDIA compares anyway, AMD does not. The game does sample this way (logged: compare mode 0, function 0) | `vk_draw.cpp` `sampler_for`. `RX_VK_DREF_FROM_SAMPLER=1`: as before | | 7 | A storage-table entry whose range does not fit its pool binds smaller (down to 1 MB) instead of nothing | Entries flipped bound/unbound between frames (the "table entry UNBOUND / bound again" log), so their reads alternated between data and 0: a bright/dark flicker | `vk_draw.cpp` | | 8 | Dynamic-resolution network `9dc98871eb7fc101` kept by default (next to the TAAU) | With Resolution **Auto** on AMD the 3D scene is black under the HUD: this pass resolves the 2048x1152 scene, and the upscale stub can't stand in (input and output are the same size) | `vk_shader.cpp` | | 9 | Diagnostics, all off by default | — | `vk_draw.cpp`: `RX_VK_LOG_MESH_PROGS=1` (each mesh program's draws and indirect arguments); in a frame dump, `dump-reach` lines (each dispatch's storage/uniform buffers and translated bindings) and `mesh indirect` buffers (indirect mesh arguments read back **from the GPU**) | ## Found independently, already fixed in `main` (no action needed) - **Compute shaders reading through loaded pointers.** On da030c5 the volume lighting pass `ee7764a6b1ef459c` got 0 from all 101 of its global loads (its inputs sit behind pointers in a 64-byte storage buffer). Result: no sky ambient, dark characters, black canopies, a white haze. I fixed it with a compute pointer table; `ffcdb71` / `52cebb1` fix it more generally with the page table. Confirmed that this was the main cause of the dark characters. - **Lichii #410** (loaded pointers competing with a cbuf pointer): also ported here, and in `ffcdb71`. - **VRAM:** reached ~15 GB on a 16 GB card at 1440p; `4756836` caps the texture cache. ## Still open: findings for you ### A. Tree canopies are missing everywhere (Kokiri Forest, the Deku Tree's crown) Compared with Switch 2 footage of the same spot, every tree is bare branches; near ferns and grass draw. What I measured: - **All 41 indirect mesh draws in the scene have zero groups on the GPU.** These are mesh-only programs with indirect arguments. I read the arguments back from the GPU after each pass (diagnostic 9), so this isn't guest memory lagging. Their culling compute passes (`f0fed2bbccae7bdc`, `cb2e06ddfb9d7d27`, `44511d69b52282ad`, `4247f1b4c3ce6cf3`, …) have those argument buffers bound and store to memory. - In `f0fed2bbccae7bdc` the instance counts are atomic adds/max at **`c0[0x90]` + `c1[0]` + small constants**. `c1[0]` is 12 here. Storage tracking takes `c1+0` as the buffer: the biased search fails because compute descriptors start at c0+0x40, and the unbiased search accepts the first cbuf read. So the atomics target a "buffer" at address 12. - Skipping `c1` in `Track` (my experiment, not in this PR) makes the atomics target the right buffer (`c0+0x90`). **The counts were still 0**, and other programs started flashing white, so I backed it out. The tracking is still worth a look, but the culling also rejects everything for another reason: its inputs (frustum, visibility/HiZ, instance records). - The 30 programs fixed by #1 don't draw in this scene, so they aren't the canopies. - Likely related: with no canopy, nothing shades the fog, which is part of why it looks too thick (B). ### B. Fog far too thick with `RX_VK_OOT_VISUAL=0` (on da030c5) Once the sky ambient worked, turning the workaround off made the fog-apply pass (`1a7482c0ec9991ea`) raise the scene's mean from about 0.5 to 2.7. Lichii's fog at the same spot is much lighter. The missing canopies (A) probably account for part of it. ### C. Brightness pulses only while the window has focus - With focus, the scene goes bright and dark every 5-10 s. Alt-tab out, and it settles. - Captured both ways at the same spot: sky, lighting volume, lit scene, fog and bloom all differ by **one factor, x2.32**. The final pass's exposure (c3+0x38) is constant 1.0 in both, so the factor comes in earlier (the pre-exposure / light constants). - Frame pacing differs too: with focus 51-54 fps with many 22-30 ms frames; without focus a steady 60. (Without focus only input pauses; the renderer keeps running.) - **`RX_VK_SUBMIT_DELAY_US=1000` helps a lot**, which fits your notes on the race with the game's threads. - AMD Anti-Lag, Chill, Boost, Fluid Motion Frames and Enhanced Sync are all off: no change. ### D. Small things - **Shader preparation console freezes when clicked.** Windows' QuickEdit selection blocks the console write in `prepare_run`'s progress callback, so "Preparing shaders: 0 of 42537" looks hung. All workers had finished; the main thread was waiting in `WriteFile`. Esc resumes it. Turning QuickEdit off on that console would avoid it: `SetConsoleMode(GetStdHandle(STD_INPUT_HANDLE), ENABLE_EXTENDED_FLAGS)` without `ENABLE_QUICK_EDIT_MODE`. - `9dc98871eb7fc101` left out made Auto resolution black on AMD (#8 above); 100% resolution worked around it. ## Tried with no visible effect (kept here so nobody repeats them) - FP16 off on AMD (Lichii #378's workaround); `main` has `RX_VK_NO_F16`. - Read-only storage buffers bound past their size on AMD (AMD bounds-checks, NVIDIA reads on). - Running the volume pass `53509044401d78d2` that is skipped by default (`RX_VK_SKIP_PROG=-`). - More pointer-table reach for task/mesh (256 MB) and more entries for compute (8 regions, 1 KB scans, 64 pointers): no pointers were left out, and the canopies stayed bare. Happy to test anything on this card. All of it can be reproduced with the switches above and the dump trigger (`RX_VK_DUMP_FILE`). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

NotRexed commented

2026-11-07 15:39:06 +00:00

(Migrated from github.com)

Thank you! Applied by hand to the current code (it had moved on, so it couldn't merge directly), credited in the commit, and shipped in https://github.com/NotRexed/Hookshot/releases/tag/less-stutter.