vk: share one mesh-fallback record/payload pair per recording thread #1383

Merged

nicholasgower merged 2 commits from fix/mesh-fallback-shared-buffers into main

2026-11-10 04:40:20 +00:00

nicholasgower commented

2026-11-10 00:00:57 +00:00

(Migrated from github.com)

Fixes #1380 (and the same report as #1001 / #1253).

Cause

The mesh-shader fallback, relanded in v2.0.6 for GPUs without VK_EXT_mesh_shader (RX 5000 and older, GTX 10 and older), allocated a records buffer and a payload buffer per fallback program, per recording thread, each sized for MESH_FALLBACK_CAPACITY = 1024 slots. With the ~95 fallback programs OoT draws, that is ~2.2 GB of VRAM on an RX 5500M, and more on a busy recording thread.

On an 8 GB RX 5700 XT already close to its texture budget, the driver then pages VRAM to system memory. The reporter's machine has 16 GB of RAM and the logs show the process committing ~30 GB private (the game reserves 9.8 GB and a 6.9 GB heap); the pagefile thrashes, audio keeps working, the picture falls to ~1 fps, and Windows can reset the display driver or become unresponsive. The log also shows fallback: the compute pipeline of a5da9f84b72d1471 could not be made: that fault is the invalid SPIR-V of #1314 (already on main) and #1317 stops it being retried on every draw.

Fix

Keep one grow-only FallbackBuffers pair per recording thread instead of an unordered_map<const Program*, .>. A draw already clears the slots it runs and full_barrier orders it after the previous draw reading the records, so every program can reuse the same buffers.
A program that needs more than the per-thread cap is left out rather than taking the whole budget.
Clear only the payload slots the draw can start, not a larger program's whole shared payload.
Grown buffers are retained: a command buffer recorded before a growth may still reference them.

Validation

python -I -B tests/test_mesh_fallback.py
python -I -B tests/test_tls_objects.py
python -I -B tests/test_mesh_stage_buffer_zero.py
python -I -B tests/test_mesh_fallback_failed_once.py

No AMD GPU here, so this is not an in-game A/B. It would be good to have the reporter retest on a build with #1314, #1317 and this change together.

Fixes #1380 (and the same report as #1001 / #1253). ## Cause The mesh-shader fallback, relanded in v2.0.6 for GPUs without `VK_EXT_mesh_shader` (RX 5000 and older, GTX 10 and older), allocated a records buffer **and** a payload buffer **per fallback program, per recording thread**, each sized for `MESH_FALLBACK_CAPACITY` = 1024 slots. With the ~95 fallback programs OoT draws, that is ~2.2 GB of VRAM on an RX 5500M, and more on a busy recording thread. On an 8 GB RX 5700 XT already close to its texture budget, the driver then pages VRAM to system memory. The reporter's machine has 16 GB of RAM and the logs show the process committing ~30 GB private (the game reserves 9.8 GB and a 6.9 GB heap); the pagefile thrashes, audio keeps working, the picture falls to ~1 fps, and Windows can reset the display driver or become unresponsive. The log also shows `fallback: the compute pipeline of a5da9f84b72d1471 could not be made`: that fault is the invalid SPIR-V of #1314 (already on `main`) and #1317 stops it being retried on every draw. ## Fix - Keep one grow-only `FallbackBuffers` pair per recording thread instead of an `unordered_map<const Program*, .>`. A draw already clears the slots it runs and `full_barrier` orders it after the previous draw reading the records, so every program can reuse the same buffers. - A program that needs more than the per-thread cap is left out rather than taking the whole budget. - Clear only the payload slots the draw can start, not a larger program's whole shared payload. - Grown buffers are retained: a command buffer recorded before a growth may still reference them. ## Validation - `python -I -B tests/test_mesh_fallback.py` - `python -I -B tests/test_tls_objects.py` - `python -I -B tests/test_mesh_stage_buffer_zero.py` - `python -I -B tests/test_mesh_fallback_failed_once.py` No AMD GPU here, so this is not an in-game A/B. It would be good to have the reporter retest on a build with #1314, #1317 and this change together.

NotRexed commented

2026-11-10 02:24:53 +00:00

(Migrated from github.com)

This landed in the v2.0.12 branch (merged there for testing; it reaches main with that release), so closing. Thanks!

This landed in the v2.0.12 branch (merged there for testing; it reaches `main` with that release), so closing. Thanks!

NotRexed commented

2026-11-10 03:54:35 +00:00

(Migrated from github.com)

Reopened: closing this was a mistake on my side. It was tested in the v2.0.12 branch, and it will be merged into main here, so it shows as merged. Sorry for the noise.

hckrman101 (Migrated from github.com) reviewed

2026-11-10 04:14:41 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks, the diagnosis makes sense and going from one pair per program to one shared, grow-only pair per thread is the right fix for the VRAM blowup. Narrowing the payload clear to slots checks out too: with a task stage the mesh dispatch is exactly slots workgroups, so nothing past the cleared range runs. The error paths that free a half-made pair are a nice cleanup.

One thing I think needs fixing before merge, plus one smaller point (inline):

WAR hazard on the shared records/payload. The vkCmdFillBuffer calls come before full_barrier, and end_pass doesn't record a barrier. So nothing orders the fill after the previous fallback draw's vertex shader reading records. The barrier after the fill orders the compute dispatches, but not the fill itself. Before this PR you only hit that when the same program drew twice in a row. Now any two fallback draws back to back (same pass, different programs, which is the common case) share the buffers, so the next draw's clear can race the previous draw's reads and wipe its flags. Calling full_barrier(cmd) once before the fills fixes it, and the dedupe drops it if it's redundant.
Growth policy. Each new pair is sized to exactly what the requesting program needs, and old pairs are never freed. If programs first show up in increasing size order, you end up with one pair per size step, which is the per-program allocation pattern again, only smaller. Sizing a new pair to at least the largest existing one (or rounding up to a power of two) keeps it at a few pairs. Freeing retired pairs once their command buffers have completed could come later.

CI: Python checks and Builder pass. SPIR-V lowering, the Windows back end and appimage were still pending when I looked.

Thanks, the diagnosis makes sense and going from one pair per program to one shared, grow-only pair per thread is the right fix for the VRAM blowup. Narrowing the payload clear to `slots` checks out too: with a task stage the mesh dispatch is exactly `slots` workgroups, so nothing past the cleared range runs. The error paths that free a half-made pair are a nice cleanup. One thing I think needs fixing before merge, plus one smaller point (inline): 1. **WAR hazard on the shared records/payload.** The `vkCmdFillBuffer` calls come before `full_barrier`, and `end_pass` doesn't record a barrier. So nothing orders the fill after the previous fallback draw's vertex shader reading `records`. The barrier after the fill orders the compute dispatches, but not the fill itself. Before this PR you only hit that when the same program drew twice in a row. Now any two fallback draws back to back (same pass, different programs, which is the common case) share the buffers, so the next draw's clear can race the previous draw's reads and wipe its flags. Calling `full_barrier(cmd)` once before the fills fixes it, and the dedupe drops it if it's redundant. 2. **Growth policy.** Each new pair is sized to exactly what the requesting program needs, and old pairs are never freed. If programs first show up in increasing size order, you end up with one pair per size step, which is the per-program allocation pattern again, only smaller. Sizing a new pair to at least the largest existing one (or rounding up to a power of two) keeps it at a few pairs. Freeing retired pairs once their command buffers have completed could come later. CI: Python checks and Builder pass. SPIR-V lowering, the Windows back end and appimage were still pending when I looked.

gpu/vk/vk_draw.cpp

Small correctness note on this comment: the barrier comes after the clear, so it orders the dispatches after the previous draw but not the clear itself (see the comment on the fill below).

gpu/vk/vk_draw.cpp

This new pair is sized only to this program's records/payload. If a slightly larger program comes along later, it gets yet another pair, and all of them are kept. Taking the max of the request and the largest existing pair's sizes (or rounding up) bounds the number of pairs. Also, a program with no payload creates a payload-less pair, and a later task program can't reuse it, so it allocates a second records buffer. Growing to cover both would avoid that.

This new pair is sized only to this program's `records`/`payload`. If a slightly larger program comes along later, it gets yet another pair, and all of them are kept. Taking the max of the request and the largest existing pair's sizes (or rounding up) bounds the number of pairs. Also, a program with no payload creates a payload-less pair, and a later task program can't reuse it, so it allocates a second records buffer. Growing to cover both would avoid that.

gpu/vk/vk_draw.cpp

Nothing orders this fill (or the payload fill below) after the previous fallback draw's vkCmdDraw, which reads records in its generated vertex stage. end_pass only ends rendering, and the full_barrier below comes after the fills. Now that every program shares the pair, consecutive fallback draws hit this all the time. Suggest a full_barrier(cmd); right after end_pass(cmd); (it gets deduped when one was already recorded).

Nothing orders this fill (or the payload fill below) after the previous fallback draw's `vkCmdDraw`, which reads `records` in its generated vertex stage. `end_pass` only ends rendering, and the `full_barrier` below comes after the fills. Now that every program shares the pair, consecutive fallback draws hit this all the time. Suggest a `full_barrier(cmd);` right after `end_pass(cmd);` (it gets deduped when one was already recorded).