Foliage culling (#35): three bugs that zero the tree counts, plus a trace of where the canopies are still lost #252

Closed

AyyBeeDee20738 wants to merge 1 commit from leaves-culling into main

AyyBeeDee20738 commented

2026-11-07 17:32:51 +00:00

(Migrated from github.com)

Tested on an AMD GPU only (RX 9070 XT, Adrenalin 26.10.1). None of these bugs is AMD-specific: #35 and #124 show the same bare trees on an RTX 4070 and an RTX 5070.

Follow-up to #138, on the missing leaves (#35; also #124, and the earlier #61, #78, #84). Based on main (ccdf22e), renderer build-checked there (RX_VK_ONLY=1: 0 failures, links), played on less-stutter with each change. Nothing from the game is included: programs are named by their Hookshot keys only.

This does not bring the canopies back yet. It removes three bugs that each forced the culling's counts to 0 (each confirmed in frame dumps), so the remaining cause can be found. The trace of what is left is below.

In this PR

#	Change	What it fixed (measured)	Where
1	Indirect dispatches see their real grid. `dispatch()` filled c0+0..0xc from guest memory at recording time, but the foliage chain writes the next pass's group counts on the GPU in the same frame. The program now gets a c0 upload of its own, and the 12 bytes of arguments are copied into it on the GPU just before the dispatch. `RX_VK_NO_GRID_PATCH=1`: as before	All 12 culling passes read a grid of 0 (dump: recorded `0 1 1`, GPU had `0x658 1 1` for `44511d69`, `0x22d` for `cb2e06dd`, `0xb1` for `521f8ef5`, …)	`vk_draw.cpp` (`uniform_range`, `dispatch`)
2	Compute storage buffers tracked to c0's descriptor area first. Compute descriptors start at c0+0x40, under `nvn_bias`, so the unbiased search took the first constant it met. The counters are at `c0[0x90] + c1[0] + k`, `c1[0] = 12`. Now c0 0x40–0xA00 (align 16) is tried before the unbiased search, compute only. `RX_VK_C1_FIRST=1`: as before	7 culling passes bound their written counters to a "buffer" at address 0xc (`ssbo=c1+0w->0xc` in `f0fed2bb`, `cb2e06dd`, `44511d69`, `6344c53d`, `102f5b0a`, `abccf81b`, `7650519f`); now `c0+0x90`/`c0+0xc0`. No white flashes seen or logged errors (the blanket "skip c1" from #138's notes caused flashes)	`global_memory_to_storage_buffer_pass.cpp`
3	Storage-table entries without a size word. `383a5669` (fills the instance lists the culling reads) writes through a table of bare pointers in c4: 16 bytes an entry, no size. Every entry read size 0 and was left unbound, so all its writes went nowhere. Such an entry now takes the size another compute descriptor gave the same address (e.g. the culling's own `ssbo1`, 0xa040); one never seen with a size stays unbound. `RX_VK_TABLE_NO_SIZE=1`: as before	`dump ssbo` showed all of `383a5669`'s c4 entries `size 0 bound NO`; now bound	`vk_draw.cpp` (binding loop)
4	Diagnostics, off by default	—	`RX_VK_PTR_SCAN=<prog>`: in a dump, pointer-like values in the game's storage buffers for that program, inside / outside the page table's windows. `vk_dump.cpp`: buffer dumps also count all-ones words (the culling's "no entry")

Why #3 takes a known size instead of a guess: binding such an entry to its pool's end (I tried 4 MB) made the write-back copy that whole range over guest memory at the submission's end. It overwrote the game's pool headers past the buffers, and the game crashed in ModuleSystemWorker2 right after loading a save. With known sizes: no crash, 60 fps.

Where the canopies are still lost

After 1–3, 12 of 41 indirect mesh draws in Kokiri Forest get work (the release: 10). The other 29 stay at 0, the canopies among them. Measured, so nobody repeats it:

44511d69 (1624 groups) sorts instances into three lists. One gets about 100 groups and the other two none. A lane's list index is -1 when the keep flag (%12450 in its SPIR-V) is false. The flag starts as ssbo1[i] > -1 (signed). ssbo1 (41 KB) is mostly 0 with no all-ones words, so that test passes. The rejection happens later in that chain.
Its inputs look right: the 512x512 texture is a wind field rewritten every frame (14058c1a), and the 1360x1000x8 one holds 0–1 data.
Ruled out, each with no change:
- RX_VK_PIPE_SKIP=0.
- RX_VK_MIRROR=0: the page table points at the imported pools, not the mirrors, but this made no difference.
- RX_VK_MESH_ARGS=1 together with 1–3. It must be tested with them: alone, every count was 0 anyway. With them it made no visible difference, at about 40 fps.
- Page-table coverage: every pointer in the culling's buffers lies in the 8 GB window.
Lichii's tree fix (6612817cb, d151e53e6: GPU-written command lists read before the GPU filled them) doesn't apply: Hookshot hands indirect draws to Vulkan directly.

Related open PRs:

#206 extends the c0 range task and mesh programs read.
#217 and #225 fix RX_VK_MESH_FIRST and RX_VK_FOLLOW_MESH.

Once the counts get through, those two switches are worth retesting with this PR.

Happy to test anything on this card.

🤖 Generated with Claude Code

> **Tested on an AMD GPU only** (RX 9070 XT, Adrenalin 26.10.1). None of these bugs is AMD-specific: #35 and #124 show the same bare trees on an RTX 4070 and an RTX 5070. Follow-up to #138, on the missing leaves (#35; also #124, and the earlier #61, #78, #84). Based on `main` (ccdf22e), renderer build-checked there (`RX_VK_ONLY=1`: 0 failures, links), played on `less-stutter` with each change. Nothing from the game is included: programs are named by their Hookshot keys only. **This does not bring the canopies back yet.** It removes three bugs that each forced the culling's counts to 0 (each confirmed in frame dumps), so the remaining cause can be found. The trace of what is left is below. ## In this PR | # | Change | What it fixed (measured) | Where | |---|---|---|---| | 1 | **Indirect dispatches see their real grid.** `dispatch()` filled c0+0..0xc from guest memory at recording time, but the foliage chain writes the next pass's group counts on the GPU in the same frame. The program now gets a c0 upload of its own, and the 12 bytes of arguments are copied into it on the GPU just before the dispatch. `RX_VK_NO_GRID_PATCH=1`: as before | All 12 culling passes read a grid of **0** (dump: recorded `0 1 1`, GPU had `0x658 1 1` for `44511d69`, `0x22d` for `cb2e06dd`, `0xb1` for `521f8ef5`, …) | `vk_draw.cpp` (`uniform_range`, `dispatch`) | | 2 | **Compute storage buffers tracked to c0's descriptor area first.** Compute descriptors start at c0+0x40, under `nvn_bias`, so the unbiased search took the first constant it met. The counters are at `c0[0x90] + c1[0] + k`, `c1[0] = 12`. Now c0 0x40–0xA00 (align 16) is tried before the unbiased search, compute only. `RX_VK_C1_FIRST=1`: as before | 7 culling passes bound their written counters to a "buffer" at address **0xc** (`ssbo=c1+0w->0xc` in `f0fed2bb`, `cb2e06dd`, `44511d69`, `6344c53d`, `102f5b0a`, `abccf81b`, `7650519f`); now `c0+0x90`/`c0+0xc0`. No white flashes seen or logged errors (the blanket "skip c1" from #138's notes caused flashes) | `global_memory_to_storage_buffer_pass.cpp` | | 3 | **Storage-table entries without a size word.** `383a5669` (fills the instance lists the culling reads) writes through a table of bare pointers in c4: 16 bytes an entry, no size. Every entry read size 0 and was left unbound, so all its writes went nowhere. Such an entry now takes the size another compute descriptor gave the same address (e.g. the culling's own `ssbo1`, 0xa040); one never seen with a size stays unbound. `RX_VK_TABLE_NO_SIZE=1`: as before | `dump ssbo` showed all of `383a5669`'s c4 entries `size 0 bound NO`; now bound | `vk_draw.cpp` (binding loop) | | 4 | Diagnostics, off by default | — | `RX_VK_PTR_SCAN=<prog>`: in a dump, pointer-like values in the game's storage buffers for that program, inside / outside the page table's windows. `vk_dump.cpp`: buffer dumps also count all-ones words (the culling's "no entry") | **Why #3 takes a known size instead of a guess:** binding such an entry to its pool's end (I tried 4 MB) made the write-back copy that whole range over guest memory at the submission's end. It overwrote the game's pool headers past the buffers, and the game crashed in `ModuleSystemWorker2` right after loading a save. With known sizes: no crash, 60 fps. ## Where the canopies are still lost After 1–3, 12 of 41 indirect mesh draws in Kokiri Forest get work (the release: 10). The other 29 stay at 0, the canopies among them. Measured, so nobody repeats it: - **`44511d69` (1624 groups) sorts instances into three lists.** One gets about 100 groups and the other two none. A lane's list index is -1 when the keep flag (`%12450` in its SPIR-V) is false. The flag starts as `ssbo1[i] > -1` (signed). `ssbo1` (41 KB) is mostly 0 with no all-ones words, so that test passes. The rejection happens later in that chain. - **Its inputs look right:** the 512x512 texture is a wind field rewritten every frame (`14058c1a`), and the 1360x1000x8 one holds 0–1 data. - **Ruled out, each with no change:** - `RX_VK_PIPE_SKIP=0`. - `RX_VK_MIRROR=0`: the page table points at the imported pools, not the mirrors, but this made no difference. - **`RX_VK_MESH_ARGS=1` together with 1–3.** It must be tested with them: alone, every count was 0 anyway. With them it made no visible difference, at about 40 fps. - Page-table coverage: every pointer in the culling's buffers lies in the 8 GB window. - **Lichii's tree fix** (`6612817cb`, `d151e53e6`: GPU-written command lists read before the GPU filled them) doesn't apply: Hookshot hands indirect draws to Vulkan directly. **Related open PRs:** - **#206** extends the c0 range task and mesh programs read. - **#217 and #225** fix `RX_VK_MESH_FIRST` and `RX_VK_FOLLOW_MESH`. Once the counts get through, those two switches are worth retesting with this PR. Happy to test anything on this card. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

NotRexed commented

2026-11-09 05:29:04 +00:00

(Migrated from github.com)

Closing to keep the queue clear — please rebase on current main and reopen; thanks! (Part 1, the indirect dispatch grids, is on main as 08b413e; main handles part 2 with its own compute bias, so only the sizeless pointer tables are still new.)

Closing to keep the queue clear — please rebase on current main and reopen; thanks! (Part 1, the indirect dispatch grids, is on main as 08b413e; main handles part 2 with its own compute bias, so only the sizeless pointer tables are still new.)

Commenting is not possible because the repository is archived.

No reviewers