WIP: fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation #1164

Draft
TrashPanda wants to merge 1 commit from TrashPanda/Longshot:fix/udmabuf-fallback into main
Contributor

Type: fix · Risk: medium · Tested on: Linux, RTX 4070 Ti (game-free checks) · Needs: Linux, NVIDIA and AMD

Why

On Linux, when /dev/udmabuf exists but rejects the game's memory, every buffer pool kept trying it instead of falling back, and a pool whose GPU mirror failed once in a VRAM spike was never mirrored again, read over PCIe for the rest of the session.

Restored from Hookshot 2482, by Erfboom, and split out of #288 so it can be reviewed and merged on its own.

What changed

disable_dmabuf_imports turns udmabuf off after a rejection before any import worked, and mirror_make retries a refused mirror 120 submissions later when the budget has room.

Commits, each with its original author and message:

5267d142 fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation (Erfboom)

Problem

  1. When /dev/udmabuf exists on the host system but persistently
    rejects guest memory before any import has succeeded (e.g. kernel/driver
    rejection with EINVAL, EBADFD, or EPERM, or when the driver
    reports no valid memory type, or a non-VK_ERROR_OUT_OF_DEVICE_MEMORY
    import failure), the Vulkan backend continues attempting udmabuf
    creation on every buffer pool allocation instead of falling back to
    host-pointer imports (VK_EXT_external_memory_host).
  2. Pools that failed create_device_buffer in mirror_make under a
    transient VRAM spike remained permanently marked as MIRROR_REFUSED
    (state 3). Even after texture retirement or VMA compaction freed VRAM
    and mirror budget headroom was available, those pools were never
    retried, forcing the GPU to read them over PCIe for the rest of the
    session.

Solution

  • Add disable_dmabuf_imports(const char* why): if udmabuf creation or
    driver import is rejected before any successful import has occurred
    (excluding transient VK_ERROR_OUT_OF_DEVICE_MEMORY), set dmabuf_memfd = -1 (leaving dmabuf_dev open so background import threads cannot
    race on the fd) to fall back to host-pointer imports for subsequent
    allocations.
  • In mirror_make, if create_device_buffer fails (for a pool \le 1
    GB), set p->mirror_retry_epoch = mirror_epoch + 120 rather than
    refusing mirrors permanently.
  • In gpu_buffer_bind, once mirror_epoch >= p->mirror_retry_epoch,
    check if the pool fits within the mirror budget (`mirror_bytes + (p->hi
  • p->lo) <= mirror_budget()). If room exists, transition p->mstateback toMIRROR_NONEso its mirror is requested at the next submission boundary; otherwise, defermirror_retry_epoch` by another 120
    submissions without evicting active mirrors or churning on low-VRAM
    cards.
  • Add mirror_bytes and mirror_budget() stubs to
    tests/direct_buffer_writes_test.cpp.

Measurements & Verification

  • A/B in-game verification (OoT, Linux x86_64, RTX 4050 Laptop GPU 6
    GB, NVIDIA 615.78.08, 2048x1152, RX_VK_VRAM_MB=2240,
    RX_VK_MIRROR_MB=2048)
    — identical route: Hyrule Field \rightarrow
    Market Town \rightarrow Hyrule Castle (~2,680 frames):
  • Without https://github.com/NotRexed/Hookshot/pull/2482 (main at 20d50332): Upon entering Castle during
    a VRAM spike (image over budget), two pools (0x7ee5400000 172 MB and
    0x7fb2c00000 40 MB) entered state 3 (MIRROR_REFUSED). Even after
    VRAM compaction 1 over: ... 35 blocks (560 MB) given back to the driver and mirror usage dropped to 1,652 MB / 2,048 MB, both pools
    remained stuck in state 3 for all 7 remaining 600-submit summaries,
    reading 4,850.1 MB total over PCIe (474.4–1,219.8 MB per 600
    submits, e.g. vk: bound in guest memory, by pool ... 1219.8 MB in all ... 0x7ee5400000 (172 MB, state 3): 6657 binds, 1211 MB; 0x7fb2c00000 (40 MB, state 3): 170 binds, 9 MB;).
  • With https://github.com/NotRexed/Hookshot/pull/2482 (46497054): On the same Castle entry spike, two pools
    (0x7eec000000 56 MB and 0x7fb3400000 24 MB) entered state 3 for
    only 1 summary (270.1 MB in all: 0x7eec000000 (56 MB, state 3): 611 binds, 269 MB; 0x7fb3400000 (24 MB, state 3): 14 binds, 1 MB;). Once
    textures retired and budget room opened 120 submissions later, both
    pools exited state 3 and got their VRAM mirrors back (vk: a frame: mirrors 312 (1800 MB)). Across the next 5 consecutive 600-submit
    summaries (3,000 submissions through Castle), 0 pools were in state 3 and 0 MB was read over PCIe.
  • Also verified by @felipefl142 across a 2.5 h full-game completion run
    on RTX 3070 Laptop 8 GB (CachyOS) and @YakuzaFanA on RTX 5060 Ti 8 GB
    (Windows 11).
  • Test suite: python3 -I -B tests/test_direct_buffer_writes.py
    (12/12 pass) and tests/run_all.py.

Co-authored-by: erfboom [email protected]
Restored-from: https://github.com/NotRexed/Hookshot/pull/2482

Measurements

Verdict: not measured on Longshot: the Hookshot PR's A/B (RTX 4050 Laptop 6 GB, Linux) is in the commit below; a run on Longshot is needed before this is marked ready.

Flags and settings

None.

Testing

Tested: On Linux (kernel 7.0), RTX 4070 Ti, main @ 71b1edd8, this PR @ 5267d142: the checks that read the changed files (tests/run_all.py --changed origin/main) pass. The full suite (tests/run_all.py) has not been run on this branch yet; a comment will follow when it has.

Not tested: The game on Linux with a rejecting udmabuf. Opened as work in progress until it runs in the game.

Must be tested:

  • Linux, NVIDIA, 6-8 GB card (by hand): Hyrule Field to Castle Town, expecting mirrors to come back after a spike (RX_VK_MIRROR_LOG=1)

Risks

What could break: buffer imports and mirrors on Linux.

Turning it off: revert the commit.

Compatibility: Nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).

When it applies

  • A game-free test added for the fix, failing on main and passing here.
**Type:** fix · **Risk:** medium · **Tested on:** Linux, RTX 4070 Ti (game-free checks) · **Needs:** Linux, NVIDIA and AMD ## Why On Linux, when `/dev/udmabuf` exists but rejects the game's memory, every buffer pool kept trying it instead of falling back, and a pool whose GPU mirror failed once in a VRAM spike was never mirrored again, read over PCIe for the rest of the session. Restored from [Hookshot 2482](https://github.com/NotRexed/Hookshot/pull/2482), by Erfboom, and split out of [#288](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/288) so it can be reviewed and merged on its own. ## What changed `disable_dmabuf_imports` turns udmabuf off after a rejection before any import worked, and `mirror_make` retries a refused mirror 120 submissions later when the budget has room. Commits, each with its original author and message: <details><summary><code>5267d142</code> fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation (Erfboom)</summary> #### Problem 1. When `/dev/udmabuf` exists on the host system but persistently rejects guest memory before any import has succeeded (e.g. kernel/driver rejection with `EINVAL`, `EBADFD`, or `EPERM`, or when the driver reports no valid memory type, or a non-`VK_ERROR_OUT_OF_DEVICE_MEMORY` import failure), the Vulkan backend continues attempting udmabuf creation on every buffer pool allocation instead of falling back to host-pointer imports (`VK_EXT_external_memory_host`). 2. Pools that failed `create_device_buffer` in `mirror_make` under a transient VRAM spike remained permanently marked as `MIRROR_REFUSED` (`state 3`). Even after texture retirement or VMA compaction freed VRAM and mirror budget headroom was available, those pools were never retried, forcing the GPU to read them over PCIe for the rest of the session. #### Solution - Add `disable_dmabuf_imports(const char* why)`: if udmabuf creation or driver import is rejected before any successful import has occurred (excluding transient `VK_ERROR_OUT_OF_DEVICE_MEMORY`), set `dmabuf_memfd = -1` (leaving `dmabuf_dev` open so background import threads cannot race on the fd) to fall back to host-pointer imports for subsequent allocations. - In `mirror_make`, if `create_device_buffer` fails (for a pool $\le$ 1 GB), set `p->mirror_retry_epoch = mirror_epoch + 120` rather than refusing mirrors permanently. - In `gpu_buffer_bind`, once `mirror_epoch >= p->mirror_retry_epoch`, check if the pool fits within the mirror budget (`mirror_bytes + (p->hi - p->lo) <= mirror_budget()`). If room exists, transition `p->mstate` back to `MIRROR_NONE` so its mirror is requested at the next submission boundary; otherwise, defer `mirror_retry_epoch` by another 120 submissions without evicting active mirrors or churning on low-VRAM cards. - Add `mirror_bytes` and `mirror_budget()` stubs to `tests/direct_buffer_writes_test.cpp`. #### Measurements & Verification - **A/B in-game verification (OoT, Linux x86_64, RTX 4050 Laptop GPU 6 GB, NVIDIA 615.78.08, 2048x1152, `RX_VK_VRAM_MB=2240`, `RX_VK_MIRROR_MB=2048`)** — identical route: Hyrule Field $\rightarrow$ Market Town $\rightarrow$ Hyrule Castle (~2,680 frames): - **Without https://github.com/NotRexed/Hookshot/pull/2482 (`main` at `20d50332`)**: Upon entering Castle during a VRAM spike (`image over budget`), two pools (`0x7ee5400000` 172 MB and `0x7fb2c00000` 40 MB) entered `state 3` (`MIRROR_REFUSED`). Even after `VRAM compaction 1 over: ... 35 blocks (560 MB) given back to the driver` and mirror usage dropped to 1,652 MB / 2,048 MB, both pools remained stuck in `state 3` for all 7 remaining 600-submit summaries, reading **4,850.1 MB total over PCIe** (474.4–1,219.8 MB per 600 submits, e.g. `vk: bound in guest memory, by pool ... 1219.8 MB in all ... 0x7ee5400000 (172 MB, state 3): 6657 binds, 1211 MB; 0x7fb2c00000 (40 MB, state 3): 170 binds, 9 MB;`). - **With https://github.com/NotRexed/Hookshot/pull/2482 (`46497054`)**: On the same Castle entry spike, two pools (`0x7eec000000` 56 MB and `0x7fb3400000` 24 MB) entered `state 3` for **only 1 summary** (`270.1 MB in all: 0x7eec000000 (56 MB, state 3): 611 binds, 269 MB; 0x7fb3400000 (24 MB, state 3): 14 binds, 1 MB;`). Once textures retired and budget room opened 120 submissions later, both pools exited `state 3` and got their VRAM mirrors back (`vk: a frame: mirrors 312 (1800 MB)`). Across the next 5 consecutive 600-submit summaries (3,000 submissions through Castle), **0 pools were in `state 3` and 0 MB was read over PCIe**. - Also verified by @felipefl142 across a 2.5 h full-game completion run on RTX 3070 Laptop 8 GB (CachyOS) and @YakuzaFanA on RTX 5060 Ti 8 GB (Windows 11). - **Test suite**: `python3 -I -B tests/test_direct_buffer_writes.py` (12/12 pass) and `tests/run_all.py`. --------- Co-authored-by: erfboom <[email protected]> Restored-from: https://github.com/NotRexed/Hookshot/pull/2482 </details> ## Measurements **Verdict:** not measured on Longshot: the Hookshot PR's A/B (RTX 4050 Laptop 6 GB, Linux) is in the commit below; a run on Longshot is needed before this is marked ready. ### Flags and settings None. ## Testing **Tested:** On Linux (kernel 7.0), RTX 4070 Ti, `main` @ 71b1edd8, this PR @ 5267d142: `the checks that read the changed files (`tests/run_all.py --changed origin/main`)` pass. The full suite (`tests/run_all.py`) has not been run on this branch yet; a comment will follow when it has. **Not tested:** The game on Linux with a rejecting udmabuf. Opened as work in progress until it runs in the game. **Must be tested:** - Linux, NVIDIA, 6-8 GB card (by hand): Hyrule Field to Castle Town, expecting mirrors to come back after a spike (`RX_VK_MIRROR_LOG=1`) ## Risks **What could break:** buffer imports and mirrors on Linux. **Turning it off:** revert the commit. **Compatibility:** Nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). ### When it applies - [x] A game-free test added for the fix, failing on `main` and passing here.
fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 6s
Website / Website on Cloudflare (pull_request_target) Has been skipped
5267d142bb
## Problem

1. When `/dev/udmabuf` exists on the host system but persistently
rejects guest memory before any import has succeeded (e.g. kernel/driver
rejection with `EINVAL`, `EBADFD`, or `EPERM`, or when the driver
reports no valid memory type, or a non-`VK_ERROR_OUT_OF_DEVICE_MEMORY`
import failure), the Vulkan backend continues attempting udmabuf
creation on every buffer pool allocation instead of falling back to
host-pointer imports (`VK_EXT_external_memory_host`).
2. Pools that failed `create_device_buffer` in `mirror_make` under a
transient VRAM spike remained permanently marked as `MIRROR_REFUSED`
(`state 3`). Even after texture retirement or VMA compaction freed VRAM
and mirror budget headroom was available, those pools were never
retried, forcing the GPU to read them over PCIe for the rest of the
session.

## Solution

- Add `disable_dmabuf_imports(const char* why)`: if udmabuf creation or
driver import is rejected before any successful import has occurred
(excluding transient `VK_ERROR_OUT_OF_DEVICE_MEMORY`), set `dmabuf_memfd
= -1` (leaving `dmabuf_dev` open so background import threads cannot
race on the fd) to fall back to host-pointer imports for subsequent
allocations.
- In `mirror_make`, if `create_device_buffer` fails (for a pool $\le$ 1
GB), set `p->mirror_retry_epoch = mirror_epoch + 120` rather than
refusing mirrors permanently.
- In `gpu_buffer_bind`, once `mirror_epoch >= p->mirror_retry_epoch`,
check if the pool fits within the mirror budget (`mirror_bytes + (p->hi
- p->lo) <= mirror_budget()`). If room exists, transition `p->mstate`
back to `MIRROR_NONE` so its mirror is requested at the next submission
boundary; otherwise, defer `mirror_retry_epoch` by another 120
submissions without evicting active mirrors or churning on low-VRAM
cards.
- Add `mirror_bytes` and `mirror_budget()` stubs to
`tests/direct_buffer_writes_test.cpp`.

## Measurements & Verification

- **A/B in-game verification (OoT, Linux x86_64, RTX 4050 Laptop GPU 6
GB, NVIDIA 615.78.08, 2048x1152, `RX_VK_VRAM_MB=2240`,
`RX_VK_MIRROR_MB=2048`)** — identical route: Hyrule Field $\rightarrow$
Market Town $\rightarrow$ Hyrule Castle (~2,680 frames):
- **Without https://github.com/NotRexed/Hookshot/pull/2482 (`main` at `20d50332`)**: Upon entering Castle during
a VRAM spike (`image over budget`), two pools (`0x7ee5400000` 172 MB and
`0x7fb2c00000` 40 MB) entered `state 3` (`MIRROR_REFUSED`). Even after
`VRAM compaction 1 over: ... 35 blocks (560 MB) given back to the
driver` and mirror usage dropped to 1,652 MB / 2,048 MB, both pools
remained stuck in `state 3` for all 7 remaining 600-submit summaries,
reading **4,850.1 MB total over PCIe** (474.4–1,219.8 MB per 600
submits, e.g. `vk: bound in guest memory, by pool ... 1219.8 MB in all
... 0x7ee5400000 (172 MB, state 3): 6657 binds, 1211 MB; 0x7fb2c00000
(40 MB, state 3): 170 binds, 9 MB;`).
- **With https://github.com/NotRexed/Hookshot/pull/2482 (`46497054`)**: On the same Castle entry spike, two pools
(`0x7eec000000` 56 MB and `0x7fb3400000` 24 MB) entered `state 3` for
**only 1 summary** (`270.1 MB in all: 0x7eec000000 (56 MB, state 3): 611
binds, 269 MB; 0x7fb3400000 (24 MB, state 3): 14 binds, 1 MB;`). Once
textures retired and budget room opened 120 submissions later, both
pools exited `state 3` and got their VRAM mirrors back (`vk: a frame:
mirrors 312 (1800 MB)`). Across the next 5 consecutive 600-submit
summaries (3,000 submissions through Castle), **0 pools were in `state
3` and 0 MB was read over PCIe**.
- Also verified by @felipefl142 across a 2.5 h full-game completion run
on RTX 3070 Laptop 8 GB (CachyOS) and @YakuzaFanA on RTX 5060 Ti 8 GB
(Windows 11).
- **Test suite**: `python3 -I -B tests/test_direct_buffer_writes.py`
(12/12 pass) and `tests/run_all.py`.

---------

Co-authored-by: erfboom <[email protected]>
Restored-from: https://github.com/NotRexed/Hookshot/pull/2482
Author
Contributor

Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @ 71b1edd8 (make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. NVIDIA's driver refuses dma-bufs here and the game took host pointers (log: "guest memory as dma-bufs: no ... host pointers"), the fallback this changes.

Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @ 71b1edd8 (make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. NVIDIA's driver refuses dma-bufs here and the game took host pointers (log: "guest memory as dma-bufs: no ... host pointers"), the fallback this changes.
Collaborator

Possibly related, for context:

  • PR #1155 fix(linux): guest memory as dma-bufs under Steam and Flatpak, where udmabuf's size_limit_mb can't be read — Overlaps Linux udmabuf guest-memory import handling
Possibly related, for context: - PR #1155 fix(linux): guest memory as dma-bufs under Steam and Flatpak, where udmabuf's size_limit_mb can't be read — Overlaps Linux udmabuf guest-memory import handling
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 6s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/udmabuf-fallback:TrashPanda-fix/udmabuf-fallback'
git switch 'TrashPanda-fix/udmabuf-fallback'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1164
No description provided.