fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation #2482

github-sync · 2026-11-11T20:44:32Z

Erfboom commented

2026-11-11 20:44:32 +00:00

(Migrated from github.com)

Problem

When /dev/udmabuf exists on the host system but fails to import guest memory (e.g. kernel/driver rejection with EINVAL, EBADFD, or EPERM, or when the driver reports no valid memory type), the Vulkan backend continuously attempts imports on every buffer pool allocation. This causes continuous ioctl failure logging and wasted kernel transitions instead of cleanly falling back to host-pointer imports (VK_EXT_external_memory_host).

In addition, pools that failed device mirror creation under transient VRAM pressure remained permanently marked as MIRROR_REFUSED. If memory was later freed or other mirrors were reclaimed, these pools would never be retried, leaving hot buffers in slower system memory permanently.

Solution

Add disable_dmabuf_imports(const char* why): if udmabuf creation or driver import is rejected before any successful import has occurred, close the udmabuf handle and permanently fallback to host-pointer imports for subsequent allocations.
In mirror_make, if device buffer creation fails due to VRAM pressure, trigger mirrors_give_back() and retry once.
If mirror allocation still fails due to transient pressure, set mirror_retry_epoch = mirror_epoch + 120 rather than permanently refusing mirrors for the pool. In gpu_buffer_bind, allow pools to exit MIRROR_REFUSED once the retry epoch is reached.

Impact

Eliminates repeated failing ioctl(..., kUdmabufCreate) syscalls and log spam on Linux configurations where /dev/udmabuf is present but unsupported or restricted.
Restores mirroring for pools that experienced temporary allocation failures during high VRAM spikes once pressure subsides.
Verified with tests/run_all.py (359 passed, 0 failed).

## Problem When `/dev/udmabuf` exists on the host system but fails to import guest memory (e.g. kernel/driver rejection with `EINVAL`, `EBADFD`, or `EPERM`, or when the driver reports no valid memory type), the Vulkan backend continuously attempts imports on every buffer pool allocation. This causes continuous ioctl failure logging and wasted kernel transitions instead of cleanly falling back to host-pointer imports (`VK_EXT_external_memory_host`). In addition, pools that failed device mirror creation under transient VRAM pressure remained permanently marked as `MIRROR_REFUSED`. If memory was later freed or other mirrors were reclaimed, these pools would never be retried, leaving hot buffers in slower system memory permanently. ## Solution - Add `disable_dmabuf_imports(const char* why)`: if udmabuf creation or driver import is rejected before any successful import has occurred, close the udmabuf handle and permanently fallback to host-pointer imports for subsequent allocations. - In `mirror_make`, if device buffer creation fails due to VRAM pressure, trigger `mirrors_give_back()` and retry once. - If mirror allocation still fails due to transient pressure, set `mirror_retry_epoch = mirror_epoch + 120` rather than permanently refusing mirrors for the pool. In `gpu_buffer_bind`, allow pools to exit `MIRROR_REFUSED` once the retry epoch is reached. ## Impact - Eliminates repeated failing `ioctl(..., kUdmabufCreate)` syscalls and log spam on Linux configurations where `/dev/udmabuf` is present but unsupported or restricted. - Restores mirroring for pools that experienced temporary allocation failures during high VRAM spikes once pressure subsides. - Verified with `tests/run_all.py` (359 passed, 0 failed).

hckrman101 (Migrated from github.com) reviewed

2026-11-11 20:44:32 +00:00

Joey-Magnus (Migrated from github.com) requested changes

2026-11-11 21:24:02 +00:00

Joey-Magnus (Migrated from github.com) left a comment

I ran the suite merged into main and it passes, but:

mirror_make now calls mirrors_give_back(n) (vk_memory.cpp:3060), which is reached from gpu_buffer_bind mid-recording (:2752). mirror_free then rewrites the page table and bumps pools_gen while the recording is still open. That used to happen only at the submission boundary.
With VRAM genuinely full, the retry at mirror_epoch + 120 (:3063) repeats free-and-retry every 120 submissions, freeing idle mirrors for an allocation that keeps failing. Low-VRAM cards churn.
Imports are switched off for the session on the first failed import before any success (:204), so one transient VK_ERROR_OUT_OF_DEVICE_MEMORY disables dma-buf imports for good. The log was already capped at 8 lines.

I'd split out or drop the retry, and attach a run from the game.

I ran the suite merged into main and it passes, but: - `mirror_make` now calls `mirrors_give_back(n)` (`vk_memory.cpp:3060`), which is reached from `gpu_buffer_bind` mid-recording (`:2752`). `mirror_free` then rewrites the page table and bumps `pools_gen` while the recording is still open. That used to happen only at the submission boundary. - With VRAM genuinely full, the retry at `mirror_epoch + 120` (`:3063`) repeats free-and-retry every 120 submissions, freeing idle mirrors for an allocation that keeps failing. Low-VRAM cards churn. - Imports are switched off for the session on the first failed import before any success (`:204`), so one transient `VK_ERROR_OUT_OF_DEVICE_MEMORY` disables dma-buf imports for good. The log was already capped at 8 lines. I'd split out or drop the retry, and attach a run from the game.

hckrman101 (Migrated from github.com) reviewed

2026-11-11 22:18:39 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks. The udmabuf half makes sense to me: EBADFD/EINVAL from udmabuf on a memfd it won't take are permanent, and so is "no memory type", so giving up on the first one is fine there. Two things I'd still change, plus I agree with @Joey-Magnus's review:

His third point stands: vkImportMemoryFdKHR/bind failing with VK_ERROR_OUT_OF_DEVICE_MEMORY is not proof the driver can't import, so I'd only disable on the ioctl errnos and the missing memory type, and keep the import-failure path as it was.
His first point (mirrors_give_back reached from gpu_buffer_bind mid-recording, page table rewritten and pools_gen bumped outside the submission boundary) is a real behaviour change and not mentioned in the description. More below on why the immediate retry rarely helps anyway, which makes it easy to just drop.
The 120-submission retry of MIRROR_REFUSED pools on its own is reasonable, but I'd split it from the udmabuf fix into its own PR with a run from the game on a low-VRAM card, as Joey asked.

Nit: the description says the failures meant "continuous ioctl failure logging"; the log was already capped at 8 lines, so this is about the wasted ioctls/imports, not log spam.

CI has no checks reported on this branch yet.

Thanks. The udmabuf half makes sense to me: `EBADFD`/`EINVAL` from udmabuf on a memfd it won't take are permanent, and so is "no memory type", so giving up on the first one is fine there. Two things I'd still change, plus I agree with @Joey-Magnus's review: - His third point stands: `vkImportMemoryFdKHR`/bind failing with `VK_ERROR_OUT_OF_DEVICE_MEMORY` is not proof the driver can't import, so I'd only disable on the ioctl errnos and the missing memory type, and keep the import-failure path as it was. - His first point (`mirrors_give_back` reached from `gpu_buffer_bind` mid-recording, page table rewritten and `pools_gen` bumped outside the submission boundary) is a real behaviour change and not mentioned in the description. More below on why the immediate retry rarely helps anyway, which makes it easy to just drop. - The 120-submission retry of `MIRROR_REFUSED` pools on its own is reasonable, but I'd split it from the udmabuf fix into its own PR with a run from the game on a low-VRAM card, as Joey asked. Nit: the description says the failures meant "continuous ioctl failure logging"; the log was already capped at 8 lines, so this is about the wasted ioctls/imports, not log spam. CI has no checks reported on this branch yet.

gpu/vk/vk_memory.cpp

					
				@ -158,0 +159,4 @@

				void disable_dmabuf_imports(const char* why) {

				    if (dmabuf_memfd >= 0) logf("vk: disabling udmabuf pool imports (%s); falling back to host-pointer imports", why);

				    dmabuf_memfd = -1;

Minor: dma_buf_memory can run on the background import thread (bg_main calls import_pool without pools_mu) as well as on the recording thread, so closing dmabuf_dev here can race the other thread's ioctl on the same fd (or a reused fd number). The background imports are opt-in (RX_VK_BG_IMPORT, RX_VK_ASYNC_IMPORT, prefetch), so not a default-path problem; setting dmabuf_memfd = -1 and leaving the fd open would avoid it.

Minor: `dma_buf_memory` can run on the background import thread (`bg_main` calls `import_pool` without `pools_mu`) as well as on the recording thread, so closing `dmabuf_dev` here can race the other thread's `ioctl` on the same fd (or a reused fd number). The background imports are opt-in (`RX_VK_BG_IMPORT`, `RX_VK_ASYNC_IMPORT`, prefetch), so not a default-path problem; setting `dmabuf_memfd = -1` and leaving the fd open would avoid it.

gpu/vk/vk_memory.cpp

We only get here after the budget check above passed, so inside mirrors_give_back the mirror_bytes + want_bytes > mirror_budget() clause is false: it frees only retired mirrors and ones idle past kLongIdle, which memory_begin_submission already does every 64 submissions. So the second create_device_buffer is usually the same allocation against the same VRAM, and the cost is the mid-recording mirror_free Joey pointed out. I'd drop the give-back-and-retry here and keep just the mirror_retry_epoch part (in its own PR).

We only get here after the budget check above passed, so inside `mirrors_give_back` the `mirror_bytes + want_bytes > mirror_budget()` clause is false: it frees only retired mirrors and ones idle past `kLongIdle`, which `memory_begin_submission` already does every 64 submissions. So the second `create_device_buffer` is usually the same allocation against the same VRAM, and the cost is the mid-recording `mirror_free` Joey pointed out. I'd drop the give-back-and-retry here and keep just the `mirror_retry_epoch` part (in its own PR).

felipefl142 commented

2026-11-12 00:41:01 +00:00

(Migrated from github.com)

I tested this PR in a real play session. It fixes the stuck refused mirrors I reported in #2412 (#2412).

Setup: RTX 3070 Laptop 8 GB, NVIDIA 620.70.06, CachyOS (Linux). I merged this PR into main at 57ed3aa (v2.1.0-rc.3-177-gf302c495). DLSS (built with the SDK), output 1920x1080, RX_VK_VRAM_MB=3072, RX_VK_COMPACT=1, default mirror budget (2048 MB). In the game: the whole Water Temple, then Lake Hylia, Hyrule Field, and burning Kakariko. Before the merge, tests/ scripts that read vk_memory.cpp passed: 26 run, one skipped because it needs Windows.

Timeline

Time	Where	Game fps	Draws a frame	Guest memory read over PCIe
0-54 min	Water Temple	38-46 (52-55 in light rooms)	800-1100	none
56-60 min	Leaving for Lake Hylia	30-44	600-1700	up to 474 MB per 600 submits
62-74 min	Hyrule Field, Kakariko	25-30	1500-2200	almost none

At Lake Hylia, VRAM ran out the same way as in my earlier report. The log showed image over budget (x10), 40 textures evicted, and pools left in system memory (0x7fb8000000 (176 MB) stays in system memory: the mirrors have 1920 MB of 2048). This time the pools went back to state 1 (MIRROR_WANTED), got mirrors again, and the PCIe reads dropped to almost nothing within about 3 minutes. Over the whole session, pools appeared in state 3 in only 4 summaries. My bad session on 7759a6f had 23,649.

Same scene size, four sessions (same PC and settings)

Draws a frame	`7759a6f`, good session	`7759a6f`, bad session (mirrors stuck refused)	`940eb9e`, `RX_VK_VRAM_MB=2560`	`57ed3aa` + #2482
600-1000	42.5 fps	36.6	33.7	42.0
1400-1800	34.4	28.8	27.3	34.5
1800+	32.3	25.1	22.0	27.9
`memory sync` a frame	1.1-2.0 ms	3.0-4.6 ms	1.4-2.3 ms	0.7-0.9 ms

(fps: mean of the vi: lines, grouped by the draw count in the nearest presents line.)

Lowering the texture budget to 2560 did not leave room for the mirrors. VRAM still filled up to 6.4-6.7 GB, textures were missing or blurry in some areas, and fps was worse than with 3072. With this PR and the budget back at 3072, the session at up to 1800 draws a frame matched my best earlier one, memory sync stayed under 1 ms the whole time, and nothing was missing in the temple.

The lower fps in Hyrule Field comes from the size of the scene, not from memory. At 1500-2200 draws a frame, both the replay (submit 13-17 ms) and GPU time (13-16 ms) grow with the draw count, while PCIe reads and memory sync stay low. That is #995's territory.

On NVIDIA the udmabuf part of the PR doesn't come into play: the log has no disabling udmabuf pool imports line, since #2340 already keeps dma-buf imports off there.

I tested this PR in a real play session. It fixes the stuck refused mirrors I reported in #2412 (https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2412). **Setup:** RTX 3070 Laptop 8 GB, NVIDIA 620.70.06, CachyOS (Linux). I merged this PR into main at 57ed3aa (`v2.1.0-rc.3-177-gf302c495`). DLSS (built with the SDK), output 1920x1080, `RX_VK_VRAM_MB=3072`, `RX_VK_COMPACT=1`, default mirror budget (2048 MB). In the game: the whole Water Temple, then Lake Hylia, Hyrule Field, and burning Kakariko. Before the merge, `tests/` scripts that read `vk_memory.cpp` passed: 26 run, one skipped because it needs Windows. ## Timeline | Time | Where | Game fps | Draws a frame | Guest memory read over PCIe | |---|---|---|---|---| | 0-54 min | Water Temple | 38-46 (52-55 in light rooms) | 800-1100 | none | | 56-60 min | Leaving for Lake Hylia | 30-44 | 600-1700 | up to 474 MB per 600 submits | | 62-74 min | Hyrule Field, Kakariko | 25-30 | 1500-2200 | almost none | At Lake Hylia, VRAM ran out the same way as in my earlier report. The log showed `image over budget` (x10), 40 textures evicted, and pools left in system memory (`0x7fb8000000 (176 MB) stays in system memory: the mirrors have 1920 MB of 2048`). This time the pools went back to state 1 (`MIRROR_WANTED`), got mirrors again, and the PCIe reads dropped to almost nothing within about 3 minutes. Over the whole session, pools appeared in **state 3 in only 4 summaries. My bad session on 7759a6f had 23,649.** ## Same scene size, four sessions (same PC and settings) | Draws a frame | 7759a6f, good session | 7759a6f, bad session (mirrors stuck refused) | 940eb9e, `RX_VK_VRAM_MB=2560` | **57ed3aa + #2482** | |---|---|---|---|---| | 600-1000 | 42.5 fps | 36.6 | 33.7 | **42.0** | | 1400-1800 | 34.4 | 28.8 | 27.3 | **34.5** | | 1800+ | 32.3 | 25.1 | 22.0 | **27.9** | | `memory sync` a frame | 1.1-2.0 ms | 3.0-4.6 ms | 1.4-2.3 ms | **0.7-0.9 ms** | (fps: mean of the `vi:` lines, grouped by the draw count in the nearest `presents` line.) Lowering the texture budget to 2560 did not leave room for the mirrors. VRAM still filled up to 6.4-6.7 GB, textures were missing or blurry in some areas, and fps was worse than with 3072. With this PR and the budget back at 3072, the session at up to 1800 draws a frame matched my best earlier one, `memory sync` stayed under 1 ms the whole time, and nothing was missing in the temple. The lower fps in Hyrule Field comes from the size of the scene, not from memory. At 1500-2200 draws a frame, both the replay (`submit` 13-17 ms) and GPU time (13-16 ms) grow with the draw count, while PCIe reads and memory sync stay low. That is #995's territory. On NVIDIA the udmabuf part of the PR doesn't come into play: the log has no `disabling udmabuf pool imports` line, since #2340 already keeps dma-buf imports off there.

YakuzaFanA commented

2026-11-12 01:13:02 +00:00

(Migrated from github.com)

Tested this on Windows with an 8 GB card and it fixes the slowdown for me.

Setup: RTX 5060 Ti 8 GB, driver 610.88, Core Ultra 7 265F, Windows 11, 1080p. Built on main at 497060f with this PR merged on top. My build also has a few other local changes (#1377 and some switches), but those were identical in both runs below. The only difference between them is this PR.

Same route both times: Graveyard, Kakariko, Hyrule Field, then into the Market from the field, then back out. Without the PR, the Market entry pushed VRAM over the limit and two pools (160 MB and 240 MB) ended up in state 3 for the rest of the session. The GPU read them over PCIe the whole time and the game never recovered, even after leaving. With the PR they got their mirrors back.

	without #2482	with #2482
mean fps	27.1	49.0
frames over 33 ms	47.5%	4.0%
"bound in guest memory", median MB per 600 submits	3,818	9.1
GPU time a frame	25.3 ms	10.4 ms
Hyrule Field after the Market	16-22 fps	46-56 fps

The Market itself still drops to around 22 fps, but that one looks CPU-bound (#995), not this.

(I used Claude to help read the logs; the playing and testing were mine.)

Tested this on Windows with an 8 GB card and it fixes the slowdown for me. Setup: RTX 5060 Ti 8 GB, driver 610.88, Core Ultra 7 265F, Windows 11, 1080p. Built on main at 497060f with this PR merged on top. My build also has a few other local changes (#1377 and some switches), but those were identical in both runs below. The only difference between them is this PR. Same route both times: Graveyard, Kakariko, Hyrule Field, then into the Market from the field, then back out. Without the PR, the Market entry pushed VRAM over the limit and two pools (160 MB and 240 MB) ended up in state 3 for the rest of the session. The GPU read them over PCIe the whole time and the game never recovered, even after leaving. With the PR they got their mirrors back. | | without #2482 | with #2482 | |---|---|---| | mean fps | 27.1 | 49.0 | | frames over 33 ms | 47.5% | 4.0% | | "bound in guest memory", median MB per 600 submits | 3,818 | 9.1 | | GPU time a frame | 25.3 ms | 10.4 ms | | Hyrule Field after the Market | 16-22 fps | 46-56 fps | The Market itself still drops to around 22 fps, but that one looks CPU-bound (#995), not this. (I used Claude to help read the logs; the playing and testing were mine.)

Erfboom commented

2026-11-12 04:01:52 +00:00

(Migrated from github.com)

Thanks @Joey-Magnus and @hckrman101 for the feedback, and @YakuzaFanA for the benchmark run on Windows!

Pushed an update addressing all points:

Transient OOM no longer disables udmabuf: In dma_buf_memory, VK_ERROR_OUT_OF_DEVICE_MEMORY is explicitly guarded so a temporary allocation failure during a VRAM spike will not disable udmabuf imports for the remainder of the session.
Removed mid-recording mirror eviction: Dropped mirrors_give_back(n) entirely from inside mirror_make. Evictions and page table rewrites continue to only happen safely at submission boundaries.
Budget-gated mirror retry: In gpu_buffer_bind, mirror retry is gated on budget headroom (mirror_bytes + (p->hi - p->lo) <= mirror_budget()). If VRAM is genuinely full and no room exists, the retry epoch is deferred by 120 submissions without evicting active mirrors or churning on low-VRAM GPUs. Pools > 1 GB remain permanently refused.

Tested locally on Linux (6 GB hardware):

Unit test suite passes cleanly.
udmabuf remains active across heavy transitions (80 live imports, 2.9 GB imported, clean lifecycle).
Scene transitions and Hyrule Field run smoothly with stable frame pacing (21–25 FPS under heavy scene load, 60 FPS in menus) with zero churn or mid-recording invalidations.

Thanks @Joey-Magnus and @hckrman101 for the feedback, and @YakuzaFanA for the benchmark run on Windows! Pushed an update addressing all points: 1. **Transient OOM no longer disables udmabuf**: In `dma_buf_memory`, `VK_ERROR_OUT_OF_DEVICE_MEMORY` is explicitly guarded so a temporary allocation failure during a VRAM spike will not disable `udmabuf` imports for the remainder of the session. 2. **Removed mid-recording mirror eviction**: Dropped `mirrors_give_back(n)` entirely from inside `mirror_make`. Evictions and page table rewrites continue to only happen safely at submission boundaries. 3. **Budget-gated mirror retry**: In `gpu_buffer_bind`, mirror retry is gated on budget headroom (`mirror_bytes + (p->hi - p->lo) <= mirror_budget()`). If VRAM is genuinely full and no room exists, the retry epoch is deferred by 120 submissions without evicting active mirrors or churning on low-VRAM GPUs. Pools > 1 GB remain permanently refused. Tested locally on Linux (6 GB hardware): - Unit test suite passes cleanly. - `udmabuf` remains active across heavy transitions (80 live imports, 2.9 GB imported, clean lifecycle). - Scene transitions and Hyrule Field run smoothly with stable frame pacing (21–25 FPS under heavy scene load, 60 FPS in menus) with zero churn or mid-recording invalidations.

github-sync referenced this pull request

2026-11-12 06:22:44 +00:00

[Tracking] Low FPS with low GPU usage (CPU-bound), FPS drops in Castle Town / Hyrule Field, shader stutter #995

github-sync referenced this pull request

2026-11-12 06:23:18 +00:00

fix(vk): sweep a pile of dead render targets every present, not every tenth #2484

github-sync commented

2026-11-12 14:42:48 +00:00

@Joey-Magnus requested changes on GitHub · 2026-11-12 14:33 UTC · original

The update addresses my first round: no mirrors_give_back mid-recording, the retry is budget-gated and frees nothing, and an out-of-device-memory import no longer turns dma-buf imports off (gpu/vk/vk_memory.cpp:213).

I merged it into current main and ran tests/run_all.py. One failure is this PR's: tests/test_direct_buffer_writes.py cuts gpu_buffer_bind out, and the retry block (:3042) uses mirror_bytes and mirror_budget(), which tests/direct_buffer_writes_test.cpp doesn't declare, so it no longer compiles. Please add the two stubs. The other CI reds are main's from early Nov 12.

Still open:

hckrman101's point on :163: disable_dmabuf_imports closes dmabuf_dev while the background import thread (RX_VK_BG_IMPORT, RX_VK_ASYNC_IMPORT) can be in ioctl on it. Setting dmabuf_memfd = -1 and leaving the fd open avoids that.
The description still describes the give-back retry and log spam.
felipefl142's and YakuzaFanA's runs were on 2fa7431b; a short re-run on 993f7f02 would confirm the budget-gated retry still recovers.

> [@Joey-Magnus](https://github.com/Joey-Magnus) requested changes on GitHub · 2026-11-12 14:33 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2482#pullrequestreview-5876213742) The update addresses my first round: no `mirrors_give_back` mid-recording, the retry is budget-gated and frees nothing, and an out-of-device-memory import no longer turns dma-buf imports off (`gpu/vk/vk_memory.cpp:213`). I merged it into current main and ran `tests/run_all.py`. One failure is this PR's: `tests/test_direct_buffer_writes.py` cuts `gpu_buffer_bind` out, and the retry block (`:3042`) uses `mirror_bytes` and `mirror_budget()`, which `tests/direct_buffer_writes_test.cpp` doesn't declare, so it no longer compiles. Please add the two stubs. The other CI reds are main's from early Nov 12. Still open: - hckrman101's point on `:163`: `disable_dmabuf_imports` closes `dmabuf_dev` while the background import thread (`RX_VK_BG_IMPORT`, `RX_VK_ASYNC_IMPORT`) can be in `ioctl` on it. Setting `dmabuf_memfd = -1` and leaving the fd open avoids that. - The description still describes the give-back retry and log spam. - felipefl142's and YakuzaFanA's runs were on 2fa7431b; a short re-run on 993f7f02 would confirm the budget-gated retry still recovers.

Commenting is not possible because the repository is archived.

No reviewers

hckrman101

Joey-Magnus