MagicBeanDev/Hookshot

Archived

Watch

Fork 0

fix(vk): a pool whose mirror found no VRAM is asked for again, not refused for good #2618

Open

aquinas22 wants to merge 1 commit from aquinas22/fix/mirror-no-vram-retry into main

aquinas22 commented

2026-11-12 02:42:16 +00:00

(Migrated from github.com)

On cards with little VRAM, the scene's hottest memory pool can lose its VRAM mirror for the whole session: when mirror_make's allocation fails once (VRAM full of textures and retired pools' mirrors at that moment), the pool is MIRROR_REFUSED for good and the GPU reads it over PCIe from then on. This asks for it again 64 submissions later instead.

What I saw. RTX A1000 Laptop (4 GB), Windows, v2.1.0, Kokiri Forest. One 144 MB pool takes ~2.3 GB of the ~2.4 GB the GPU reads over PCIe in 600 submits (the bound in guest memory, by pool log line), state 3 (refused). The process is over what the driver allows (~4.2 of 3.3 GB), so VMA's WITHIN_BUDGET allocation fails at the pool's first binding; allocating without the budget flag fails too here (NVIDIA does not oversubscribe), so a "make it anyway" fallback is not an option. Whether the pool got its mirror depended on load timing, which made frame rates swing between launches of the same build.

The change. On an allocation failure mirror_make leaves the pool MIRROR_NONE and notes the submission (Pool::no_vram_epoch); gpu_buffer_bind neither makes nor asks for its mirror again until 64 submissions on, so a full VRAM costs one failed allocation per pool per 64 submissions, not one a binding. By then colder or retired pools' mirrors have usually been given back. Both outcomes are logged (8 lines each at most). RX_VK_MIRROR_NO_RETRY=1 keeps the old behaviour; documented in docs/development.md.

Measured (headless, HOS_ENV_FILE per run, same save, ~4240 draws a frame, 90 s after 45 s of play; with #1377's mirrors-by-heat and RX_VK_WB_MAX_KB=64, built on v2.1.0):

	fps	PCIe reads / 600 submits
refused for good (current)	18-22 (30 on a launch that happened to mirror it first)	~2.4 GB
asked for again (this PR)	30.2, 30.3	~0 MB

The pool was mirrored ~70 submissions after its first failure in every run. A third launch measured after only 20 s of play read 25.1 fps while mirrors were still arriving.

Checks. tests/run_all.py for every test that reads vk_memory.cpp or docs/development.md passes locally on Windows (llvm-mingw), except test_env_on (HOS_FPS_LOG in host/oot_high_fps.c, failing on main too) and test_vram_budget_row (fails on main locally, unrelated). test_parallel_mirror_make still finds !par on the first-binding line.

Independent of #1377, which makes the retry more likely to find room (it gives back colder mirrors for hot pools).

On cards with little VRAM, the scene's hottest memory pool can lose its VRAM mirror for the whole session: when `mirror_make`'s allocation fails once (VRAM full of textures and retired pools' mirrors at that moment), the pool is `MIRROR_REFUSED` for good and the GPU reads it over PCIe from then on. This asks for it again 64 submissions later instead. **What I saw.** RTX A1000 Laptop (4 GB), Windows, v2.1.0, Kokiri Forest. One 144 MB pool takes ~2.3 GB of the ~2.4 GB the GPU reads over PCIe in 600 submits (the `bound in guest memory, by pool` log line), state 3 (refused). The process is over what the driver allows (~4.2 of 3.3 GB), so VMA's `WITHIN_BUDGET` allocation fails at the pool's first binding; allocating without the budget flag fails too here (NVIDIA does not oversubscribe), so a "make it anyway" fallback is not an option. Whether the pool got its mirror depended on load timing, which made frame rates swing between launches of the same build. **The change.** On an allocation failure `mirror_make` leaves the pool `MIRROR_NONE` and notes the submission (`Pool::no_vram_epoch`); `gpu_buffer_bind` neither makes nor asks for its mirror again until 64 submissions on, so a full VRAM costs one failed allocation per pool per 64 submissions, not one a binding. By then colder or retired pools' mirrors have usually been given back. Both outcomes are logged (8 lines each at most). `RX_VK_MIRROR_NO_RETRY=1` keeps the old behaviour; documented in `docs/development.md`. **Measured** (headless, `HOS_ENV_FILE` per run, same save, ~4240 draws a frame, 90 s after 45 s of play; with #1377's mirrors-by-heat and `RX_VK_WB_MAX_KB=64`, built on v2.1.0): | | fps | PCIe reads / 600 submits | |---|---|---| | refused for good (current) | 18-22 (30 on a launch that happened to mirror it first) | ~2.4 GB | | asked for again (this PR) | 30.2, 30.3 | ~0 MB | The pool was mirrored ~70 submissions after its first failure in every run. A third launch measured after only 20 s of play read 25.1 fps while mirrors were still arriving. **Checks.** `tests/run_all.py` for every test that reads `vk_memory.cpp` or `docs/development.md` passes locally on Windows (llvm-mingw), except `test_env_on` (`HOS_FPS_LOG` in `host/oot_high_fps.c`, failing on main too) and `test_vram_budget_row` (fails on main locally, unrelated). `test_parallel_mirror_make` still finds `!par` on the first-binding line. Independent of #1377, which makes the retry more likely to find room (it gives back colder mirrors for hot pools).

github-sync referenced this pull request from a commit

2026-11-12 06:19:00 +00:00

fix(vk): capture switches follow the on/off rule (#2651)

github-sync commented

2026-11-12 12:37:18 +00:00

Owner

@sheepbun commented on GitHub · 2026-11-12 11:37 UTC · original

Measured on a 10 GB card too: the refusal is not only a 4 GB problem.

Setup. RTX 3080 10 GB, driver 615.78, Linux (GNOME on Wayland), i7-11700, 2560x1440 at 100% render scale, cap 170, frame generation off. Built from upstream main of 2026-11-11 with the renderer's profile and ThinLTO, with and without this PR's commit (58834871) cherry-picked. The renderer was swapped between runs and everything else held fixed. RX_VK_WB_MAX_KB was unset. Runs were interleaved, each started below 70 C, with other programs at 2.5-4.7% of the CPU.

scene_changes (the bench route that teleports through several maps; 20 s measured):

	fps	Pools refused for good	Readback waits (of 3,000)
with #2618	68.7	none	1,551 waited, 0.04-0.05 ms each
without	43.9	one (`bound in guest memory ... state 3`, 84 log lines)	339-349 waited, 14.7-16.2 ms each, all for their submission's last part
with #2618	61.4	none	1,561 waited, 0.04-0.06 ms each
without	45.5	one (96 log lines)	341-358 waited, 6.6-7.9 ms each, all for their submission's last part

With the PR, 8 pools a run (16-240 MB) hit "no VRAM for its mirror" during the scene loads. All of them got their mirror 64-87 submissions later; one 240 MB pool took 468. VRAM was at ~7.8 of 9.4 GB.

Without it, the refused pool's binds in guest memory raise the in-place part, so every readback waits for its whole submission. That wait is what costs the frame rate here, more than the PCIe reads themselves.

The same refusal turned up in a Hyrule Field run on this card. A 224 MB pool, merged from three imports during the teleport while VRAM was at 8.05 of 9.34 GB, stayed refused, and the run measured 52.2 fps. The runs that found room measured 62.9-71.5.

The 1% lows (the scene loads themselves) are unchanged. I saw no visual difference, but I did not check screenshots.

> [@sheepbun](https://github.com/sheepbun) commented on GitHub · 2026-11-12 11:37 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2618#issuecomment-6822535163) Measured on a 10 GB card too: the refusal is not only a 4 GB problem. **Setup.** RTX 3080 10 GB, driver 615.78, Linux (GNOME on Wayland), i7-11700, 2560x1440 at 100% render scale, cap 170, frame generation off. Built from upstream `main` of 2026-11-11 with the renderer's profile and ThinLTO, with and without this PR's commit (`58834871`) cherry-picked. The renderer was swapped between runs and everything else held fixed. `RX_VK_WB_MAX_KB` was unset. Runs were interleaved, each started below 70 C, with other programs at 2.5-4.7% of the CPU. **`scene_changes`** (the bench route that teleports through several maps; 20 s measured): | | fps | Pools refused for good | Readback waits (of 3,000) | |---|---:|---|---| | with #2618 | 68.7 | none | 1,551 waited, 0.04-0.05 ms each | | without | 43.9 | one (`bound in guest memory ... state 3`, 84 log lines) | 339-349 waited, 14.7-16.2 ms each, all for their submission's last part | | with #2618 | 61.4 | none | 1,561 waited, 0.04-0.06 ms each | | without | 45.5 | one (96 log lines) | 341-358 waited, 6.6-7.9 ms each, all for their submission's last part | With the PR, 8 pools a run (16-240 MB) hit "no VRAM for its mirror" during the scene loads. All of them got their mirror 64-87 submissions later; one 240 MB pool took 468. VRAM was at ~7.8 of 9.4 GB. Without it, the refused pool's binds in guest memory raise the in-place part, so every readback waits for its whole submission. That wait is what costs the frame rate here, more than the PCIe reads themselves. The same refusal turned up in a Hyrule Field run on this card. A 224 MB pool, merged from three imports during the teleport while VRAM was at 8.05 of 9.34 GB, stayed refused, and the run measured 52.2 fps. The runs that found room measured 62.9-71.5. The 1% lows (the scene loads themselves) are unchanged. I saw no visual difference, but I did not check screenshots.

github-sync referenced this pull request

2026-11-12 14:41:08 +00:00

fix(vk): give back the import of a retired pool whose mirror was refused #2795

github-sync commented

2026-11-12 14:43:28 +00:00

Owner

@Joey-Magnus reviewed on GitHub · 2026-11-12 14:33 UTC · original

I merged this into current main. The only conflict is in docs/development.md, where #2651 added the RX_VK_CAPTURE_KEEP_OLD row in the same spot; both rows can stay. The code merges cleanly. Every test that reads vk_memory.cpp or development.md passes on Linux, including test_import_buffer_binding, test_texel_buffer_ownership, test_parallel_mirror_make and test_mirror_inherit.
The logic is sound. A failed create_device_buffer leaves the pool MIRROR_NONE with no_vram_epoch set, and gpu_buffer_bind neither makes nor queues its mirror for 64 submissions. A success logs how long it took and clears the epoch. Two small notes, neither blocking:

While a pool waits it isn't REFUSED, so its read bindings miss the lock-free fast path (vk_memory.cpp, the MIRROR_REFUSED || MIRROR_WANTED test) and take pools_mu for those 64 submissions. That's the same as a pool over the mirror budget on main, so it's fine.
no_vram_epoch == 0 means "none", so a failure in the very first submission retries at the next binding. That's harmless.
Your A1000 numbers and sheepbun's RTX 3080 runs (43.9/45.5 against 68.7/61.4 fps in scene_changes) both come from the game. Please rebase; once it's rebased, I'd merge it.

> [@Joey-Magnus](https://github.com/Joey-Magnus) reviewed on GitHub · 2026-11-12 14:33 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2618#pullrequestreview-5876222552) I merged this into current main. The only conflict is in docs/development.md, where #2651 added the RX_VK_CAPTURE_KEEP_OLD row in the same spot; both rows can stay. The code merges cleanly. Every test that reads vk_memory.cpp or development.md passes on Linux, including test_import_buffer_binding, test_texel_buffer_ownership, test_parallel_mirror_make and test_mirror_inherit. The logic is sound. A failed create_device_buffer leaves the pool MIRROR_NONE with no_vram_epoch set, and gpu_buffer_bind neither makes nor queues its mirror for 64 submissions. A success logs how long it took and clears the epoch. Two small notes, neither blocking: - While a pool waits it isn't REFUSED, so its read bindings miss the lock-free fast path (vk_memory.cpp, the MIRROR_REFUSED || MIRROR_WANTED test) and take pools_mu for those 64 submissions. That's the same as a pool over the mirror budget on main, so it's fine. - no_vram_epoch == 0 means "none", so a failure in the very first submission retries at the next binding. That's harmless. Your A1000 numbers and sheepbun's RTX 3080 runs (43.9/45.5 against 68.7/61.4 fps in scene_changes) both come from the game. Please rebase; once it's rebased, I'd merge it.

github-sync referenced this pull request

2026-11-12 14:48:30 +00:00

perf(vk): sparse VRAM mirrors, memory only where a game pool is (RX_VK_MIRROR_SPARSE=1) #2794

github-sync referenced this pull request

2026-11-12 14:48:31 +00:00

perf(vk): sparse VRAM mirrors, memory only where a game pool is (RX_VK_MIRROR_SPARSE=1) #2794

github-sync referenced this pull request

2026-11-12 14:48:33 +00:00

fix(vk): give back the import of a retired pool whose mirror was refused #2795

github-sync referenced this pull request from a commit

2026-11-12 16:54:32 +00:00

fix(vk): give back the import of a retired pool whose mirror was refused (#2795)