MagicBeanDev/Longshot

Watch

Fork 98

[Feature Request] Unified Memory / Zero-Copy Path for AMD APUs on Linux (Steam Deck,Other Handhelds, BC-250, RADV) #309

New issue

Open

opened 2026-11-13 21:12:32 +00:00 by Loris_Kujo · 2 comments

Loris_Kujo commented

2026-11-13 21:12:32 +00:00

The idea

Summary

On unified memory hardware running Linux + Mesa RADV (such as the Steam Deck, Ryzen Z1/Z2 handhelds, and AMD BC-250 systems), CPU and GPU share a single physical memory pool. Currently, Hookshot treats these devices like discrete-GPU PCs—allocating a guest memory arena in system RAM and duplicating assets into DEVICE_LOCAL Vulkan memory—which quickly exhausts 16 GB unified pools despite the target Switch 2 hardware operating within a 9 GB game budget.

Adding an opt-in unified memory path (e.g., RX_VK_UNIFIED_MEM=1) would drastically reduce memory duplication and prevent OOM stalls or crashes on Linux APUs.

Problem Description

Asset Duplication Inside One Physical Pool: Guest buffers and textures are allocated in host RAM and copied into DEVICE_LOCAL VRAM heaps, causing the same data to occupy two regions of the same physical LPDDR5/GDDR6 chips.
Static Heap Pressure: On systems with small BIOS UMA frame buffer carve-outs and dynamic GTT pools, discrete-style allocations force aggressive texture caps or trigger severe slowdowns when VRAM spills over during zone transitions.

Expected Benefit

Eliminating guest-to-VRAM duplication on unified APUs should bring total runtime memory usage much closer to native Switch 2 hardware, avoiding OOM kills on 16 GB devices without sacrificing texture quality.

Anything else

Proposed Technical Approach (RADV / Linux APU Zero-Copy)

Zero-Copy Buffer Mapping via VK_EXT_external_memory_host:
- When running on an integrated/unified GPU (VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU) supporting VK_EXT_external_memory_host, bind guest RAM allocations directly as host-imported Vulkan buffers for geometry, uniform buffers, and staging data instead of allocating separate DEVICE_LOCAL copies.
- Unlike discrete GPUs over PCIe, AMD APUs access GTT/host-mapped memory over the same unified memory controller with minimal bandwidth penalty.
Single-Copy Compressed Texture Pipeline:
- Keep transcoded BC7/BCn textures (from Hookshot's ASTC compute transcoder) in a single GTT-backed GPU allocation and discard the intermediate guest RAM staging copy immediately after upload/detiling.
GTT-Aware Heap Selection & Compaction:
- Pair dynamic GTT memory type selection (HOST_VISIBLE | HOST_COHERENT | DEVICE_LOCAL where exposed by RADV's resizable BAR / large-BAR heap) with the existing RX_VK_COMPACT=1 compaction logic so free memory returns cleanly to the shared OS pool.

### The idea ### Summary On unified memory hardware running Linux + Mesa RADV (such as the Steam Deck, Ryzen Z1/Z2 handhelds, and AMD BC-250 systems), CPU and GPU share a single physical memory pool. Currently, Hookshot treats these devices like discrete-GPU PCs—allocating a guest memory arena in system RAM and duplicating assets into `DEVICE_LOCAL` Vulkan memory—which quickly exhausts 16 GB unified pools despite the target Switch 2 hardware operating within a 9 GB game budget. Adding an opt-in unified memory path (e.g., `RX_VK_UNIFIED_MEM=1`) would drastically reduce memory duplication and prevent OOM stalls or crashes on Linux APUs. ### Problem Description 1. **Asset Duplication Inside One Physical Pool:** Guest buffers and textures are allocated in host RAM and copied into `DEVICE_LOCAL` VRAM heaps, causing the same data to occupy two regions of the same physical LPDDR5/GDDR6 chips. 2. **Static Heap Pressure:** On systems with small BIOS UMA frame buffer carve-outs and dynamic GTT pools, discrete-style allocations force aggressive texture caps or trigger severe slowdowns when VRAM spills over during zone transitions. ### Expected Benefit Eliminating guest-to-VRAM duplication on unified APUs should bring total runtime memory usage much closer to native Switch 2 hardware, avoiding OOM kills on 16 GB devices without sacrificing texture quality. ### Anything else ### Proposed Technical Approach (RADV / Linux APU Zero-Copy) 1. **Zero-Copy Buffer Mapping via `VK_EXT_external_memory_host`:** - When running on an integrated/unified GPU (`VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU`) supporting `VK_EXT_external_memory_host`, bind guest RAM allocations directly as host-imported Vulkan buffers for geometry, uniform buffers, and staging data instead of allocating separate `DEVICE_LOCAL` copies. - Unlike discrete GPUs over PCIe, AMD APUs access GTT/host-mapped memory over the same unified memory controller with minimal bandwidth penalty. 2. **Single-Copy Compressed Texture Pipeline:** - Keep transcoded BC7/BCn textures (from Hookshot's ASTC compute transcoder) in a single GTT-backed GPU allocation and discard the intermediate guest RAM staging copy immediately after upload/detiling. 3. **GTT-Aware Heap Selection & Compaction:** - Pair dynamic GTT memory type selection (`HOST_VISIBLE | HOST_COHERENT | DEVICE_LOCAL` where exposed by RADV's resizable BAR / large-BAR heap) with the existing `RX_VK_COMPACT=1` compaction logic so free memory returns cleanly to the shared OS pool.

Loris_Kujo added the

kind

feature

label

2026-11-13 21:12:32 +00:00

sake963 commented

2026-11-14 13:37:20 +00:00

Adding a data point from a consumer APU that is not in the list above:
RDNA 3.5 (Krackan) on Linux RADV.

Hardware

GPU: AMD Radeon 860M (RDNA 3.5, Krackan) [1002:1114] rev d2 —
Vulkan reports AMD Radeon 860M Graphics (RADV KRACKAN1)
CPU: AMD Ryzen AI 7 PRO 350, 16 threads
VRAM carve-out: 8192 MB · GTT: 27,934 MB
Mesa 26.2.6 (arch3.1), Vulkan 1.4.354, driverID MESA_RADV
OS: CachyOS, kernel 7.1.x
/dev/udmabuf available (ACL granted to the user)

Observed on Hookshot v2.1.0 (this tree's ancestor), 100% render at 1536x864:

textures 4614 MB against a default budget of 2560 MB — the game
evicted and re-uploaded across zone transitions
raising RX_VK_VRAM_MB to 6144 removed the churn; textures settled at
4614 MB with 1536 MB of live imports
11 shader programs still fail to translate · 26–33 real fps,
CPU-bound · device lost: 0

This matches the "static heap pressure" case described above: with an 8 GB
carve-out and a 27,934 MB GTT, a discrete-style budget does not fit the
pool, and the 2560 MB default is far below what the game's own textures need.

More precise numbers from the same run, in case they help size the problem.

Settings: Handheld profile On — this is what fixes the output at
1536x864, so "100% render" means 100% of 1536x864, not of 1920x1080.
Output resolution 1920x1080, Upscaler "Game's own", Frame generation Off.

Scene	draws/frame	real fps
Kokiri, first area	2772	27.4–32.6
Kokiri, heavier area	3400	25.3–26.0
after the budget raise	3243–3547	31.9–34.3

Memory after RX_VK_VRAM_MB=6144:

textures 4614 MB (budget 6144 MB)
imports: 44 live (1536 MB)

The symptom was specific: entering a zone for the first time stuttered,
the second pass was smooth — texture eviction and re-upload, not shader
compilation. At the default budget the log had textures above it
(2783 MB against 2560).

Still failing: 11 shader programs, which on this GPU are the game's own AA
networks — the log says matrix multiplies in software (the cooperative matrices lack 16x8x16 f16).

I will add numbers from a Longshot build once v3.0.0 is out.

Adding a data point from a consumer APU that is not in the list above: **RDNA 3.5 (Krackan) on Linux RADV**. **Hardware** - GPU: AMD Radeon 860M (RDNA 3.5, Krackan) `[1002:1114]` rev d2 — Vulkan reports `AMD Radeon 860M Graphics (RADV KRACKAN1)` - CPU: AMD Ryzen AI 7 PRO 350, 16 threads - VRAM carve-out: **8192 MB** · GTT: **27,934 MB** - Mesa 26.2.6 (arch3.1), Vulkan 1.4.354, driverID `MESA_RADV` - OS: CachyOS, kernel 7.1.x - `/dev/udmabuf` available (ACL granted to the user) **Observed on Hookshot v2.1.0** (this tree's ancestor), 100% render at 1536x864: - `textures 4614 MB` against a default budget of **2560 MB** — the game evicted and re-uploaded across zone transitions - raising `RX_VK_VRAM_MB` to 6144 removed the churn; textures settled at 4614 MB with 1536 MB of live imports - 11 shader programs still fail to translate · 26–33 real fps, CPU-bound · device lost: 0 This matches the "static heap pressure" case described above: with an 8 GB carve-out and a 27,934 MB GTT, a discrete-style budget does not fit the pool, and the 2560 MB default is far below what the game's own textures need. More precise numbers from the same run, in case they help size the problem. Settings: Handheld profile **On** — this is what fixes the output at 1536x864, so "100% render" means 100% of 1536x864, not of 1920x1080. Output resolution 1920x1080, Upscaler "Game's own", Frame generation Off. | Scene | draws/frame | real fps | |---|---|---| | Kokiri, first area | 2772 | 27.4–32.6 | | Kokiri, heavier area | 3400 | 25.3–26.0 | | after the budget raise | 3243–3547 | 31.9–34.3 | Memory after `RX_VK_VRAM_MB=6144`: - `textures 4614 MB (budget 6144 MB)` - `imports: 44 live (1536 MB)` The symptom was specific: **entering a zone for the first time stuttered, the second pass was smooth** — texture eviction and re-upload, not shader compilation. At the default budget the log had textures *above* it (2783 MB against 2560). Still failing: 11 shader programs, which on this GPU are the game's own AA networks — the log says `matrix multiplies in software (the cooperative matrices lack 16x8x16 f16)`. I will add numbers from a Longshot build once v3.0.0 is out.

dukenheimer commented

2026-11-15 17:37:49 +00:00

Contributor

Buffers are already zero-copy. The guest memory window is a memfd, handed to the GPU as dma-bufs through udmabuf. Draws read the game's own pages in place, and VRAM mirrors (the discrete-card path) are off, so there are no mirror syncs or write-back copies:

[hos host] mem: the guest window is a memfd's (16), shared with the GPU as dma-bufs
[hos sead::MainThread] vk: dma-bufs of at most 64 MB (udmabuf's limit 64 MB); larger pools are several, in a sparse buffer
[hos Presentation Thread] vk: imports: 33 live (1280 MB), 0 retired still imported (0 MB), 9 retired given back (464 MB)
[hos Presentation Thread] vk: a frame: mirrors 0 (0 MB): 0.00 MB synced in 0.0 copies, 0.00 MB written back

What is still copied, and why:

Captures. At each game submit, the pages the CPU wrote since the last one are copied, so the replay reads them as submitted while the game writes on. Per submit that's 110-410 pages (median ~177, ~0.7 MB) and 0.2-1.6 ms (two runs' capture: a submit lines).
Textures. The game's textures are ASTC in the Switch's block-linear layout, and RDNA2 can't sample ASTC, so each one is decoded into a GPU image. That's the one real duplicate: 2.4-2.8 GB of textures in those runs. Smaller texture caps cut it; p0lux's env cut the decoded ASTC from 999 to 732 MB on the Deck.
The texture shadow. Pages of textures the CPU writes are kept as submitted, so a texture can be uploaded again from them.

Why not host-pointer imports (VK_EXT_external_memory_host): under Proton the Windows build can only import guest memory as host pointers, and amdgpu checks each page of those again at every submission.

Measured on the Deck: vkQueueSubmit took 148-189 ms a frame under Proton, against 4 ms natively.
Real fps: 4.5-5.0 against 20.7-20.9. Settings: FSR 3.1 Quality, frame generation 2x, GPU at 1600 MHz.
On a Linux APU, use the native build.

Where the memory goes instead.

Current main (a00c485c) at default settings on the Deck reached an 11.2-11.4 GB RSS and was stopped for memory in all 3 starts.
With #225's Steam Deck profile it ran at 7.8-7.9 GB and finished both runs (#225 (comment)). Most of that difference is the shader and pipeline side: the profile keeps no graphics pipelines ready before the title. It isn't buffer duplication.
#223 gives memory back under pressure (zeldaootr's T-5P52, #223 (comment)).

Copying less isn't an fps win by itself here. With RX_VK_READBACK_EARLY=1, which is what makes the capture wait for late pages, taking that wait out (RX_VK_CAPTURE_WAIT=0) removed 8-19 ms of memory sync a frame. Fps didn't follow, because the game's readbacks then waited instead:

13.35 → 12.85 real fps at the player's settings (Handheld 1536x864, 100% render, frame generation 3x);
14.4 → 14.5 at 1280x800 with the upscaler off;
2 interleaved runs each.

What Longshot's native Linux build already does on the Steam Deck, from our Deck runs (Steam Deck OLED, SteamOS 3.9.5, RADV, stock clocks, docked; the player's rc.4 user build with #222 applied, Kokiri Forest route). **Buffers are already zero-copy.** The guest memory window is a memfd, handed to the GPU as dma-bufs through udmabuf. Draws read the game's own pages in place, and VRAM mirrors (the discrete-card path) are off, so there are no mirror syncs or write-back copies: ``` [hos host] mem: the guest window is a memfd's (16), shared with the GPU as dma-bufs [hos sead::MainThread] vk: dma-bufs of at most 64 MB (udmabuf's limit 64 MB); larger pools are several, in a sparse buffer [hos Presentation Thread] vk: imports: 33 live (1280 MB), 0 retired still imported (0 MB), 9 retired given back (464 MB) [hos Presentation Thread] vk: a frame: mirrors 0 (0 MB): 0.00 MB synced in 0.0 copies, 0.00 MB written back ``` **What is still copied, and why:** - **Captures.** At each game submit, the pages the CPU wrote since the last one are copied, so the replay reads them as submitted while the game writes on. Per submit that's 110-410 pages (median ~177, ~0.7 MB) and 0.2-1.6 ms (two runs' `capture: a submit` lines). - **Textures.** The game's textures are ASTC in the Switch's block-linear layout, and RDNA2 can't sample ASTC, so each one is decoded into a GPU image. That's the one real duplicate: 2.4-2.8 GB of textures in those runs. Smaller texture caps cut it; p0lux's env cut the decoded ASTC from 999 to 732 MB on the Deck. - **The texture shadow.** Pages of textures the CPU writes are kept as submitted, so a texture can be uploaded again from them. **Why not host-pointer imports (`VK_EXT_external_memory_host`):** under Proton the Windows build can only import guest memory as host pointers, and amdgpu checks each page of those again at every submission. - Measured on the Deck: `vkQueueSubmit` took 148-189 ms a frame under Proton, against 4 ms natively. - Real fps: 4.5-5.0 against 20.7-20.9. Settings: FSR 3.1 Quality, frame generation 2x, GPU at 1600 MHz. - On a Linux APU, use the native build. **Where the memory goes instead.** - Current main (a00c485c) at default settings on the Deck reached an 11.2-11.4 GB RSS and was stopped for memory in all 3 starts. - With #225's Steam Deck profile it ran at 7.8-7.9 GB and finished both runs (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/225#issuecomment-25982). Most of that difference is the shader and pipeline side: the profile keeps no graphics pipelines ready before the title. It isn't buffer duplication. - #223 gives memory back under pressure (zeldaootr's T-5P52, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/223#issuecomment-21604). **Copying less isn't an fps win by itself here.** With `RX_VK_READBACK_EARLY=1`, which is what makes the capture wait for late pages, taking that wait out (`RX_VK_CAPTURE_WAIT=0`) removed 8-19 ms of memory sync a frame. Fps didn't follow, because the game's readbacks then waited instead: - 13.35 → 12.85 real fps at the player's settings (Handheld 1536x864, 100% render, frame generation 3x); - 14.4 → 14.5 at 1280x800 with the upscaler off; - 2 interleaved runs each.