perf(linux): hand guest memory to the GPU as udmabuf dma-bufs, not host pointers

azlekayn commented

2026-11-08 02:46:28 +00:00

(Migrated from github.com)

What this fixes

On Linux/RADV, vkQueueSubmit costs grow with the guest memory imported: 15-45 ms per submit in Kokiri Forest, 3-5 fps with about 11 ms of GPU time (#312, #413).

On amdgpu a VK_EXT_external_memory_host import is a userptr buffer object. RADV lists every buffer object in every submission, and at every CS ioctl the kernel re-validates every page of every userptr BO. Batching (#413) cuts the number of submits; this change removes the per-page cost itself.

Change

The guest window becomes a memfd's (MAP_SHARED, size sealed) whenever /dev/udmabuf can be opened.

Each pool is then imported as a dma-buf of the window's own pages:

UDMABUF_CREATE over the memfd range;
VK_KHR_external_memory_fd + VK_EXT_external_memory_dma_buf, both optional extensions;
the memory type comes from vkGetMemoryFdPropertiesKHR.

These are the same pages the game reads and writes. Nothing is copied, the pages are pinned once, and there's no page walk at submit.

Supporting changes:

Freeing pages: done with MADV_REMOVE on the shared window (hos_posix_guest_discard), because MADV_DONTNEED does not zero shmem.
Ranges a dma-buf holds: zeroed in place, as FreeMemoryPages already does for imports.
RX_VK_LOG_ADDR: also logs dma-buf pools.

No setup needed. udmabuf refuses a dma-buf larger than its size_limit_mb (64 MB by default), and pools are up to 1 GB. A pool larger than the limit therefore becomes a sparse buffer (VK_BUFFER_CREATE_SPARSE_BINDING_BIT), with each limit-sized piece bound to its own dma-buf of the window:

one vkQueueBindSparse when the pool is made, under dev.present_mu, waited on a fence;
it's still one buffer with one device address, so nothing else changes;
the pieces are freed with the pool.

Raising the limit (as root) only makes the large pools single dma-bufs. Devices without sparse binding on the queue keep larger pools as host-pointer imports, which is logged.

A shared mapping cannot be a host-pointer import on RADV (VK_ERROR_INVALID_EXTERNAL_HANDLE), which is why the memfd is only used when /dev/udmabuf opens.

Switches:

HOS_GUEST_MEMFD=0/1 forces the window's kind.
RX_VK_NO_DMABUF=1 keeps host-pointer imports.
RX_VK_DMABUF_PIECE_MB=<n> caps the pieces, to try the sparse path with the limit raised.

This is complementary to #413's batching and independent of the retired-imports PR (#446), though the two combine. It is also unrelated to the THP change for #312: the dma-buf path doesn't use THP.

Tested

Hardware: RX 7800 XT, RADV (Mesa, Vulkan 1.4), Fedora 44, Linux 7.2.
Kokiri Forest fps:

	fps
host pointers	3.7
host pointers, retired imports given back	5.6
udmabuf, limit raised to 1024 MB	50-60, mostly locked at 60
udmabuf, 64 MB pieces in sparse buffers (the default limit)	steady 60

Sparse path: two runs of 4 and 7 minutes with RX_VK_DMABUF_PIECE_MB=64. Pools over 64 MB (80-160 MB among those logged) were bound from 64 MB pieces, and retired ones were given back. There were no bind failures, device loss or crashes, and the picture was correct.
Live imports: 144 MB at the title, about 1.5 GB in Kokiri.
Stability: no device loss or crash in about 15 runs of 4-8 minutes, including gameplay and scene loads.
Builds measured: the numbers come from the foliage-linux-updater release with this patch.
Against main: the patch is rebased onto current main, hookshot_vk builds there, and the host files compile with the release toolchain. I have not run a full main build.
Not tested: NVIDIA or Intel on Linux. Their drivers need the dma-buf import extensions; without them the code falls back to host pointers.

Analysis and patch made with Claude (Anthropic); I ran every test on my machine.

🤖 Generated with Claude Code

## What this fixes On Linux/RADV, `vkQueueSubmit` costs grow with the guest memory imported: 15-45 ms per submit in Kokiri Forest, 3-5 fps with about 11 ms of GPU time (#312, #413). On amdgpu a `VK_EXT_external_memory_host` import is a userptr buffer object. RADV lists every buffer object in every submission, and at every CS ioctl the kernel re-validates every page of every userptr BO. Batching (#413) cuts the number of submits; this change removes the per-page cost itself. ## Change The guest window becomes a memfd's (`MAP_SHARED`, size sealed) whenever `/dev/udmabuf` can be opened. Each pool is then imported as a dma-buf of the window's own pages: - `UDMABUF_CREATE` over the memfd range; - `VK_KHR_external_memory_fd` + `VK_EXT_external_memory_dma_buf`, both optional extensions; - the memory type comes from `vkGetMemoryFdPropertiesKHR`. These are the same pages the game reads and writes. Nothing is copied, the pages are pinned once, and there's no page walk at submit. Supporting changes: - **Freeing pages:** done with `MADV_REMOVE` on the shared window (`hos_posix_guest_discard`), because `MADV_DONTNEED` does not zero shmem. - **Ranges a dma-buf holds:** zeroed in place, as `FreeMemoryPages` already does for imports. - **`RX_VK_LOG_ADDR`:** also logs dma-buf pools. **No setup needed.** udmabuf refuses a dma-buf larger than its `size_limit_mb` (64 MB by default), and pools are up to 1 GB. A pool larger than the limit therefore becomes a sparse buffer (`VK_BUFFER_CREATE_SPARSE_BINDING_BIT`), with each limit-sized piece bound to its own dma-buf of the window: - one `vkQueueBindSparse` when the pool is made, under `dev.present_mu`, waited on a fence; - it's still one buffer with one device address, so nothing else changes; - the pieces are freed with the pool. Raising the limit (as root) only makes the large pools single dma-bufs. Devices without sparse binding on the queue keep larger pools as host-pointer imports, which is logged. A shared mapping cannot be a host-pointer import on RADV (`VK_ERROR_INVALID_EXTERNAL_HANDLE`), which is why the memfd is only used when `/dev/udmabuf` opens. Switches: - `HOS_GUEST_MEMFD=0/1` forces the window's kind. - `RX_VK_NO_DMABUF=1` keeps host-pointer imports. - `RX_VK_DMABUF_PIECE_MB=<n>` caps the pieces, to try the sparse path with the limit raised. This is complementary to #413's batching and independent of the retired-imports PR (#446), though the two combine. It is also unrelated to the THP change for #312: the dma-buf path doesn't use THP. ## Tested - **Hardware:** RX 7800 XT, RADV (Mesa, Vulkan 1.4), Fedora 44, Linux 7.2. - **Kokiri Forest fps:** | | fps | |---|---| | host pointers | 3.7 | | host pointers, retired imports given back | 5.6 | | udmabuf, limit raised to 1024 MB | 50-60, mostly locked at 60 | | udmabuf, 64 MB pieces in sparse buffers (the default limit) | steady 60 | - **Sparse path:** two runs of 4 and 7 minutes with `RX_VK_DMABUF_PIECE_MB=64`. Pools over 64 MB (80-160 MB among those logged) were bound from 64 MB pieces, and retired ones were given back. There were no bind failures, device loss or crashes, and the picture was correct. - **Live imports:** 144 MB at the title, about 1.5 GB in Kokiri. - **Stability:** no device loss or crash in about 15 runs of 4-8 minutes, including gameplay and scene loads. - **Builds measured:** the numbers come from the `foliage-linux-updater` release with this patch. - **Against `main`:** the patch is rebased onto current `main`, `hookshot_vk` builds there, and the host files compile with the release toolchain. I have not run a full `main` build. - **Not tested:** NVIDIA or Intel on Linux. Their drivers need the dma-buf import extensions; without them the code falls back to host pointers. Analysis and patch made with Claude (Anthropic); I ran every test on my machine. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

azlekayn commented

2026-11-08 14:01:16 +00:00

(Migrated from github.com)

Closing: on current main (7e9bf73) this no longer shows a benefit on the setup it was written for (RX 7800 XT, RADV). Two runs each in Kokiri: 33.7 and 35.1 fps without it, 32.3 and 30.6 fps with it, no flashes either way. The same build can vary by about 10 fps between runs, so that is no measurable gain. The 3-4 fps this was meant to fix no longer reproduces on main. The branch stays on my fork in case it turns out to be useful later.

Closing: on current main (7e9bf73) this no longer shows a benefit on the setup it was written for (RX 7800 XT, RADV). Two runs each in Kokiri: 33.7 and 35.1 fps without it, 32.3 and 30.6 fps with it, no flashes either way. The same build can vary by about 10 fps between runs, so that is no measurable gain. The 3-4 fps this was meant to fix no longer reproduces on main. The branch stays on my fork in case it turns out to be useful later.

azlekayn commented

2026-11-08 14:13:27 +00:00

(Migrated from github.com)

Reopening, and correcting my closing comment: its numbers came from a broken test setup, and its claim that the 3-4 fps no longer reproduces on main is wrong.

The "without it" runs used an executable that still had this PR's host side (the guest window as a memfd) with a renderer that cannot use it, a combination no real install has. Host-pointer imports cannot work on that shared mapping, so those runs had no pool imports at all and came out at about 34 fps.

Re-measured with HOS_GUEST_MEMFD=0 (anonymous memory, which is what main does), same main renderer (7e9bf73). RX 7800 XT, RADV, Kokiri, two runs each:

Build	fps	Notes
main, anonymous memory	about 6.5 in both runs	colour noise over the whole picture in the second run
main + this PR	32.3, 30.6	no colour noise

So this PR is still needed on this setup. The branch has also been updated: merged with main, every piece of a sparse pool is now freed (retired pools from #446 and dropped read-ahead pools), and the .orig backup files I committed by mistake are gone.

Reopening, and correcting my closing comment: its numbers came from a broken test setup, and its claim that the 3-4 fps no longer reproduces on main is wrong. The "without it" runs used an executable that still had this PR's host side (the guest window as a memfd) with a renderer that cannot use it, a combination no real install has. Host-pointer imports cannot work on that shared mapping, so those runs had no pool imports at all and came out at about 34 fps. Re-measured with `HOS_GUEST_MEMFD=0` (anonymous memory, which is what main does), same main renderer (7e9bf73). RX 7800 XT, RADV, Kokiri, two runs each: | Build | fps | Notes | |---|---|---| | main, anonymous memory | about 6.5 in both runs | colour noise over the whole picture in the second run | | main + this PR | 32.3, 30.6 | no colour noise | So this PR is still needed on this setup. The branch has also been updated: merged with main, every piece of a sparse pool is now freed (retired pools from #446 and dropped read-ahead pools), and the `.orig` backup files I committed by mistake are gone.

RufflezAU commented

2026-11-08 15:33:09 +00:00

(Migrated from github.com)

Heads-up for when this branch is rebased onto main: main now has #446 (f2203e4), and its giveback frees only p.buffer and p.memory. On this branch a pool can be imported as several dma-bufs (Pool::pieces), so every piece after the first is never freed — the giveback then frees less than it reports and leaves the rest pinned.

Two-line fix, tested here on a Steam Deck OLED (RADV VANGOGH) for ~3 minutes of world, no device lost:

struct DeadImport { VkBuffer buffer; VkDeviceMemory memory;
                    std::vector<VkDeviceMemory> pieces; uint64_t value; uint64_t bytes; };
// in reap_mirrors(), after vkFreeMemory(memory):
for (VkDeviceMemory m : dead_imports.front().pieces) vkFreeMemory(dev.device, m, nullptr);
// in retired_imports_give_back():
dead_imports.push_back({p.buffer, p.memory, std::move(p.pieces),
                        std::max(mem_rec_value, p.last_value), p.hi - p.lo});
p.buffer = VK_NULL_HANDLE, p.memory = VK_NULL_HANDLE, p.address = 0, p.pieces.clear();

With it the log reads imports: 49 live (1328 MB), 0 retired still imported (0 MB), 39 retired given back (1104 MB) on a 14 GB Deck, and a world scene that used to fail with radv/amdgpu: Not enough memory for command submission (5991 of 6037 MB allowed) runs without it.

Happy to test a rebased branch on the Deck if that helps.

Heads-up for when this branch is rebased onto main: main now has #446 (`f2203e4`), and its giveback frees only `p.buffer` and `p.memory`. On this branch a pool can be imported as several dma-bufs (`Pool::pieces`), so every piece after the first is never freed — the giveback then frees less than it reports and leaves the rest pinned. Two-line fix, tested here on a Steam Deck OLED (RADV VANGOGH) for ~3 minutes of world, no device lost: ```cpp struct DeadImport { VkBuffer buffer; VkDeviceMemory memory; std::vector<VkDeviceMemory> pieces; uint64_t value; uint64_t bytes; }; // in reap_mirrors(), after vkFreeMemory(memory): for (VkDeviceMemory m : dead_imports.front().pieces) vkFreeMemory(dev.device, m, nullptr); // in retired_imports_give_back(): dead_imports.push_back({p.buffer, p.memory, std::move(p.pieces), std::max(mem_rec_value, p.last_value), p.hi - p.lo}); p.buffer = VK_NULL_HANDLE, p.memory = VK_NULL_HANDLE, p.address = 0, p.pieces.clear(); ``` With it the log reads `imports: 49 live (1328 MB), 0 retired still imported (0 MB), 39 retired given back (1104 MB)` on a 14 GB Deck, and a world scene that used to fail with `radv/amdgpu: Not enough memory for command submission` (5991 of 6037 MB allowed) runs without it. Happy to test a rebased branch on the Deck if that helps.

azlekayn commented

2026-11-08 15:39:41 +00:00

(Migrated from github.com)

Thanks @RufflezAU. This is already on the branch: d23c572 ("free every piece of a sparse pool when it goes") makes the same change as your patch, with pieces in DeadImport, freed in reap_mirrors() and moved over in retired_imports_give_back(). It also frees the pieces on the dropped-prefetch path in bg_drain(). The branch head is e317563, which includes #446.

Your Deck result is useful, since it shows the leak causing a real Not enough memory for command submission failure. A run of the current head on the Deck would be welcome if you have time. It is merged up to 7e9bf73, not the very latest main.

Thanks @RufflezAU. This is already on the branch: d23c572 ("free every piece of a sparse pool when it goes") makes the same change as your patch, with `pieces` in `DeadImport`, freed in `reap_mirrors()` and moved over in `retired_imports_give_back()`. It also frees the pieces on the dropped-prefetch path in `bg_drain()`. The branch head is e317563, which includes #446. Your Deck result is useful, since it shows the leak causing a real `Not enough memory for command submission` failure. A run of the current head on the Deck would be welcome if you have time. It is merged up to 7e9bf73, not the very latest main.

t0masz407 commented

2026-11-08 16:44:09 +00:00

(Migrated from github.com)

Tested on RDNA 3 / RADV (RX 7900 XTX, Mesa 26.2.6, NixOS, Hyprland): a598b9f + this PR, Kokiri Forest, same spot in every run (scripted with HOS_INPUT_LIVE), RX_VK_BATCH_POLL_US=2000:

	fps (presents, frame generation 2x)	`submit` / frame	GPU busy
without this PR	27.4–27.9	25 ms	58 %
with this PR (`guest memory as dma-bufs: yes`)	42.6–43.5	12.7 ms	71 %

It also merges cleanly onto v2.0.1 (with #438 and #485), which is the build I will play on. Full numbers and the other settings in #528.

🤖 Measured and written with Claude Code

Tested on RDNA 3 / RADV (RX 7900 XTX, Mesa 26.2.6, NixOS, Hyprland): `a598b9f` + this PR, Kokiri Forest, same spot in every run (scripted with `HOS_INPUT_LIVE`), `RX_VK_BATCH_POLL_US=2000`: | | fps (presents, frame generation 2x) | `submit` / frame | GPU busy | |---|---|---|---| | without this PR | 27.4–27.9 | 25 ms | 58 % | | with this PR (`guest memory as dma-bufs: yes`) | **42.6–43.5** | **12.7 ms** | 71 % | It also merges cleanly onto v2.0.1 (with #438 and #485), which is the build I will play on. Full numbers and the other settings in https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/528. 🤖 Measured and written with [Claude Code](https://claude.com/claude-code)

Rows
Columns

perf(linux): hand guest memory to the GPU as udmabuf dma-bufs, not host pointers #448

What this fixes

Change

Tested