fix(vk): dense grass cut off and far trees flickering without mesh shaders: fallback records from an allocator, 65535 slots #1511

Merged

rakefire-impala merged 3 commits from fix/mesh-fallback-dense-grass-record-allocator into main

2026-11-10 16:45:35 +00:00

rakefire-impala commented

2026-11-10 05:43:49 +00:00

(Migrated from github.com)

Builds on #1510 (it replaces that PR's slot count), so this PR also shows that commit; merge that one first. Part of splitting up #1023.

Problem

Each slot (one task thread's meshlet) gets records for max primitives × corners. With 13,900 slots that would be about 650 MB, and at 1024 slots only the first 32 task workgroups' meshlets get any.

Hyrule Field's far trees (the task/mesh pair 3cb76827/4f0c0b40, an indirect draw) start 721 to 849 task workgroups of 32 threads, measured with RX_VK_DRAW_STATS. At 16384 slots, the task workgroups past 512 started nothing. The culling list's order changes from frame to frame, so a different 29-40% of the trees was missing each frame, and the trees flickered continuously.

Change

Records from an allocator. A header (the vertex stage's indirect draw, an allocator, and the mesh stage's indirect dispatch), then one fixed 64 MB area every program shares. A mesh workgroup atomically takes count × corners records and adds them to the draw's vertex count. Records that don't fit aren't taken, so the draw always covers the first ones. The grass uses about 17 MB. MESH_FALLBACK_CAPACITY becomes 65535, Vulkan's minimum maxComputeWorkGroupCount[0], which the mesh dispatch can reach. Only started slots run, so a draw that fits costs the same. The payload buffer is about 6.5 MB per recording thread at the largest payload seen (800 words).
The GPU's own count. The task's started threads take an index in the header's dispatch word and write their slot to the payload buffer's map. The mesh stage then runs as vkCmdDispatchIndirect from that count, so the CPU never needs the task count, which DrawMeshTasksIndirect doesn't record. A mesh workgroup reads its slot from the map in its entry block; its workgroup id is the slot modulo the task's threads, as natively.
Payload once per task workgroup, its threads sharing the copy. One copy per slot was 32 times the writes. No payload clear is needed, since the mesh dispatch reads only entries its task wrote.
Indirect draw. The vertex stage reads its VertexIndex's record past the header, drawn with vkCmdDrawIndirect from the header.
A barrier before the header reset. The pair is shared per recording thread (#1383). These are the same lines as the flicker-barrier PR, so the two merge cleanly in either order.

Known trade-off: primitives land in the order the workgroups took them, so blended mesh geometry may draw out of order. Lichii's fallback has the same trade-off.

Tests

tests/test_mesh_fallback.py checks the allocator, the map dispatch, the header and the barrier.
tests/fallback_buffer_cleanup_test.cpp's mock gains the three names the buffer code now uses. Its cases are unchanged, and all 9 pass.

Tested on


Tested on	RX 5500M (RDNA1, 4 GB VRAM, 16 GB RAM), Windows 11, AMD driver; Hookshot main 0c412ad (v2.0.11-383) with these PRs, and main `17e8cd8` with these PRs for the 65535 slots, 2026-11-10
Result	Floor grass in Link's house drawn; Kokiri Forest's foliage, bushes and flowers drawn; Hyrule Field's far trees steady (849 task workgroups, all drawn)

On a GPU without mesh shaders, the fallback gives each meshlet slot records for its program's worst case, with only 1024 slots. The floor grass in Link's house starts about 13,900 meshlets, so almost none of it draws, and Hyrule Field's far trees need up to ~27,000. This takes records from a shared allocator instead, and raises the capacity to 65535 slots. **Builds on https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/1510** (it replaces that PR's slot count), so this PR also shows that commit; merge that one first. Part of splitting up #1023. ## Problem Each slot (one task thread's meshlet) gets records for `max primitives × corners`. With 13,900 slots that would be about 650 MB, and at 1024 slots only the first 32 task workgroups' meshlets get any. Hyrule Field's far trees (the task/mesh pair `3cb76827`/`4f0c0b40`, an indirect draw) start 721 to 849 task workgroups of 32 threads, measured with `RX_VK_DRAW_STATS`. At 16384 slots, the task workgroups past 512 started nothing. The culling list's order changes from frame to frame, so a different 29-40% of the trees was missing each frame, and the trees flickered continuously. ## Change - **Records from an allocator.** A header (the vertex stage's indirect draw, an allocator, and the mesh stage's indirect dispatch), then one fixed 64 MB area every program shares. A mesh workgroup atomically takes `count × corners` records and adds them to the draw's vertex count. Records that don't fit aren't taken, so the draw always covers the first ones. The grass uses about 17 MB. `MESH_FALLBACK_CAPACITY` becomes 65535, Vulkan's minimum `maxComputeWorkGroupCount[0]`, which the mesh dispatch can reach. Only started slots run, so a draw that fits costs the same. The payload buffer is about 6.5 MB per recording thread at the largest payload seen (800 words). - **The GPU's own count.** The task's started threads take an index in the header's dispatch word and write their slot to the payload buffer's map. The mesh stage then runs as `vkCmdDispatchIndirect` from that count, so the CPU never needs the task count, which `DrawMeshTasksIndirect` doesn't record. A mesh workgroup reads its slot from the map in its entry block; its workgroup id is the slot modulo the task's threads, as natively. - **Payload once per task workgroup**, its threads sharing the copy. One copy per slot was 32 times the writes. No payload clear is needed, since the mesh dispatch reads only entries its task wrote. - **Indirect draw.** The vertex stage reads its `VertexIndex`'s record past the header, drawn with `vkCmdDrawIndirect` from the header. - **A barrier before the header reset.** The pair is shared per recording thread (#1383). These are the same lines as the flicker-barrier PR, so the two merge cleanly in either order. Known trade-off: primitives land in the order the workgroups took them, so blended mesh geometry may draw out of order. Lichii's fallback has the same trade-off. ## Tests - `tests/test_mesh_fallback.py` checks the allocator, the map dispatch, the header and the barrier. - `tests/fallback_buffer_cleanup_test.cpp`'s mock gains the three names the buffer code now uses. Its cases are unchanged, and all 9 pass. ## Tested on | | | |---|---| | Tested on | RX 5500M (RDNA1, 4 GB VRAM, 16 GB RAM), Windows 11, AMD driver; Hookshot main 0c412ad (v2.0.11-383) with these PRs, and main 17e8cd8 with these PRs for the 65535 slots, 2026-11-10 | | Result | Floor grass in Link's house drawn; Kokiri Forest's foliage, bushes and flowers drawn; Hyrule Field's far trees steady (849 task workgroups, all drawn) |

hckrman101 (Migrated from github.com) approved these changes

2026-11-10 16:39:38 +00:00

hckrman101 (Migrated from github.com) left a comment

Read through the allocator, the map/indirect dispatch and the vertex-side indirect draw, and I couldn't find a problem.

Things I checked: thread 0 takes the records after a barrier that follows every thread's read of the primitive count, and the base <= fit && taken <= fit - base test is overflow-safe. The header reset sits behind a full barrier, and full_barrier covers ALL_COMMANDS with memory read/write, so the indirect-argument and dispatch reads are covered. Task threads only write a map entry when slot < MESH_FALLBACK_CAPACITY, so the dispatch count can't exceed 65535 and the payload sizing (map + ceil(cap/threads) payloads) holds. The mesh_fallback_slot load in the entry block dominates all its uses, and EmitWorkgroupId falls back to the raw id before the prologue has run. The disjoint header fills are right (no ordering between overlapping fills). CI is green.

Nit: the 64 MB records area is allocated per recording thread, plus ~6.5 MB of payload, so VRAM use no longer scales with the program but is a fixed cost whenever a fallback draw happens. Fine for the 128 MB per-thread cap, just worth knowing on 4 GB cards.

Heads-up: this stacks on #1510 (still open), so that one needs to merge first, and #1747 also touches the mesh/task fallback, so the two will likely need reconciling.

Read through the allocator, the map/indirect dispatch and the vertex-side indirect draw, and I couldn't find a problem. Things I checked: thread 0 takes the records after a barrier that follows every thread's read of the primitive count, and the `base <= fit && taken <= fit - base` test is overflow-safe. The header reset sits behind a full barrier, and `full_barrier` covers ALL_COMMANDS with memory read/write, so the indirect-argument and dispatch reads are covered. Task threads only write a map entry when `slot < MESH_FALLBACK_CAPACITY`, so the dispatch count can't exceed 65535 and the payload sizing (map + ceil(cap/threads) payloads) holds. The `mesh_fallback_slot` load in the entry block dominates all its uses, and `EmitWorkgroupId` falls back to the raw id before the prologue has run. The disjoint header fills are right (no ordering between overlapping fills). CI is green. Nit: the 64 MB records area is allocated per recording thread, plus ~6.5 MB of payload, so VRAM use no longer scales with the program but is a fixed cost whenever a fallback draw happens. Fine for the 128 MB per-thread cap, just worth knowing on 4 GB cards. Heads-up: this stacks on #1510 (still open), so that one needs to merge first, and #1747 also touches the mesh/task fallback, so the two will likely need reconciling.

Commenting is not possible because the repository is archived.

No reviewers

hckrman101