WIP: fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation #1164
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1164
Loading…
Reference in a new issue
No description provided.
Delete branch "TrashPanda/Longshot:fix/udmabuf-fallback"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: medium · Tested on: Linux, RTX 4070 Ti (game-free checks) · Needs: Linux, NVIDIA and AMD
Why
On Linux, when
/dev/udmabufexists but rejects the game's memory, every buffer pool kept trying it instead of falling back, and a pool whose GPU mirror failed once in a VRAM spike was never mirrored again, read over PCIe for the rest of the session.Restored from Hookshot 2482, by Erfboom, and split out of #288 so it can be reviewed and merged on its own.
What changed
disable_dmabuf_importsturns udmabuf off after a rejection before any import worked, andmirror_makeretries a refused mirror 120 submissions later when the budget has room.Commits, each with its original author and message:
5267d142fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation (Erfboom)Problem
/dev/udmabufexists on the host system but persistentlyrejects guest memory before any import has succeeded (e.g. kernel/driver
rejection with
EINVAL,EBADFD, orEPERM, or when the driverreports no valid memory type, or a non-
VK_ERROR_OUT_OF_DEVICE_MEMORYimport failure), the Vulkan backend continues attempting udmabuf
creation on every buffer pool allocation instead of falling back to
host-pointer imports (
VK_EXT_external_memory_host).create_device_bufferinmirror_makeunder atransient VRAM spike remained permanently marked as
MIRROR_REFUSED(
state 3). Even after texture retirement or VMA compaction freed VRAMand mirror budget headroom was available, those pools were never
retried, forcing the GPU to read them over PCIe for the rest of the
session.
Solution
disable_dmabuf_imports(const char* why): if udmabuf creation ordriver import is rejected before any successful import has occurred
(excluding transient
VK_ERROR_OUT_OF_DEVICE_MEMORY), setdmabuf_memfd = -1(leavingdmabuf_devopen so background import threads cannotrace on the fd) to fall back to host-pointer imports for subsequent
allocations.
mirror_make, ifcreate_device_bufferfails (for a pool\le1GB), set
p->mirror_retry_epoch = mirror_epoch + 120rather thanrefusing mirrors permanently.
gpu_buffer_bind, oncemirror_epoch >= p->mirror_retry_epoch,check if the pool fits within the mirror budget (`mirror_bytes + (p->hi
). If room exists, transitionp->mstateback toMIRROR_NONEso its mirror is requested at the next submission boundary; otherwise, defermirror_retry_epoch` by another 120submissions without evicting active mirrors or churning on low-VRAM
cards.
mirror_bytesandmirror_budget()stubs totests/direct_buffer_writes_test.cpp.Measurements & Verification
GB, NVIDIA 615.78.08, 2048x1152,
RX_VK_VRAM_MB=2240,RX_VK_MIRROR_MB=2048) — identical route: Hyrule Field\rightarrowMarket Town
\rightarrowHyrule Castle (~2,680 frames):mainat20d50332): Upon entering Castle duringa VRAM spike (
image over budget), two pools (0x7ee5400000172 MB and0x7fb2c0000040 MB) enteredstate 3(MIRROR_REFUSED). Even afterVRAM compaction 1 over: ... 35 blocks (560 MB) given back to the driverand mirror usage dropped to 1,652 MB / 2,048 MB, both poolsremained stuck in
state 3for all 7 remaining 600-submit summaries,reading 4,850.1 MB total over PCIe (474.4–1,219.8 MB per 600
submits, e.g.
vk: bound in guest memory, by pool ... 1219.8 MB in all ... 0x7ee5400000 (172 MB, state 3): 6657 binds, 1211 MB; 0x7fb2c00000 (40 MB, state 3): 170 binds, 9 MB;).46497054): On the same Castle entry spike, two pools(
0x7eec00000056 MB and0x7fb340000024 MB) enteredstate 3foronly 1 summary (
270.1 MB in all: 0x7eec000000 (56 MB, state 3): 611 binds, 269 MB; 0x7fb3400000 (24 MB, state 3): 14 binds, 1 MB;). Oncetextures retired and budget room opened 120 submissions later, both
pools exited
state 3and got their VRAM mirrors back (vk: a frame: mirrors 312 (1800 MB)). Across the next 5 consecutive 600-submitsummaries (3,000 submissions through Castle), 0 pools were in
state 3and 0 MB was read over PCIe.on RTX 3070 Laptop 8 GB (CachyOS) and @YakuzaFanA on RTX 5060 Ti 8 GB
(Windows 11).
python3 -I -B tests/test_direct_buffer_writes.py(12/12 pass) and
tests/run_all.py.Co-authored-by: erfboom [email protected]
Restored-from: https://github.com/NotRexed/Hookshot/pull/2482
Measurements
Verdict: not measured on Longshot: the Hookshot PR's A/B (RTX 4050 Laptop 6 GB, Linux) is in the commit below; a run on Longshot is needed before this is marked ready.
Flags and settings
None.
Testing
Tested: On Linux (kernel 7.0), RTX 4070 Ti,
main@71b1edd8, this PR @5267d142:the checks that read the changed files (tests/run_all.py --changed origin/main)pass. The full suite (tests/run_all.py) has not been run on this branch yet; a comment will follow when it has.Not tested: The game on Linux with a rejecting udmabuf. Opened as work in progress until it runs in the game.
Must be tested:
RX_VK_MIRROR_LOG=1)Risks
What could break: buffer imports and mirrors on Linux.
Turning it off: revert the commit.
Compatibility: Nothing changes.
Checklist
Everyone
type(scope): description) of the status line's type.main.third_party/untouched (a new folder with aPROVENANCE.mdinstead).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).When it applies
mainand passing here.Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @
71b1edd8(make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. NVIDIA's driver refuses dma-bufs here and the game took host pointers (log: "guest memory as dma-bufs: no ... host pointers"), the fallback this changes.Possibly related, for context:
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.