[Linux/RADV, RX 7600] 3-5 fps in Kokiri Forest: vkQueueSubmit ~250-295 ms per frame while GPU time is ~11 ms; textures garbled at random #312
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot#312
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hookshot version
Linux AppImage from the
Linux AppImageworkflow, main @170ce53c(Hookshot-foliage-linux-updater). Same results on main @4c8ff1e0(with a local stub forhos_renderer_wait, before60760695).What kind of problem
Low fps or stutter (plus garbled graphics and, with larger mirror budgets, device lost)
What happened
The opening cutscenes run at a locked 60 fps. Right after the first cutscene, when the game goes to the Great Deku Tree / Kokiri Forest area, it drops to 3-5 fps.
The log shows the time is not spent on the GPU but inside
vkQueueSubmit:In-scene numbers per run (last log windows before closing):
RX_VK_MIRROR_MB170ce53c)So: Resizable BAR on/off and the mirror budget do not change the submit time. My guess (not verified): the guest memory is imported with
VK_EXT_external_memory_host, which on amdgpu means userptr BOs, and the kernel re-validates them on every submit after the CPU writes to guest memory. That would explain a CPU-side submit cost that does not depend on GPU load.Garbled textures: the whole 3D scene becomes coloured noise (the shapes of the ground and sky are still visible). Runs 2 and 3 had the same settings and gave opposite results, so it looks random / timing-dependent rather than tied to a setting.
Device lost with larger mirror budgets: with the default budget (mirrors reached 1648 MB) and with
RX_VK_MIRROR_MB=1024, the GPU hangs every time when entering the Deku Tree area, right afterpool ... took in 2 pools, the same pattern as #150. RADV reports a GPUVM permission fault and the kernel resets the GPU (the second time a MODE1 reset with VRAM lost):With
RX_VK_MIRROR_MB=192there is no hang, but it runs at 3-5 fps as above.The networks are left out on this card as expected (
a network left out on this card, TAAUfd029a317a9a4b94stood in for), andmatrix multiplies in software (the cooperative matrices lack 16x8x16 f16)is logged.Where in the game
Right after the first opening cutscene, entering the Great Deku Tree / Kokiri Forest area. Opening cutscenes are fine (60 fps).
How often
Every time (low fps); garbled textures sometimes; device lost every time with mirror budget >= 1 GB.
Graphics card and driver version
AMD Radeon RX 7600 8 GB (RADV NAVI33), Mesa 25.3.6, Vulkan 1.4.328, Resizable BAR tested off and on
CPU, RAM and operating system
Ryzen 5 5600X, 40 GB RAM, KDE neon (Ubuntu 24.04 base), kernel 7.0.0-42, KDE Plasma Wayland session. Game files on an NTFS HDD (ntfs3).
Changes from the defaults
hookshot_env.txt:RX_VK_MIRROR_MB=192(runs 2-4),RX_VK_MIRROR_MB=1024(hang test). No Insert-menu changes.Other software hooked into the game
None
I can test any build or flag you suggest. I can't attach the full logs here, but I can paste any section you need.
Great report, thank you. 250-295 ms in
vkQueueSubmitwith ~11 ms of GPU time points at the submission itself on RADV (likely the imported host memory being re-validated per submit), not the GPU. I'll look into it; if you can, tryRX_VK_IMPORT_CHUNK_MB=64inhookshot_env.txtand say whether the submit time changes.Two more data points from the same session (build
170ce53c,RX_VK_MIRROR_MB=192, Resizable BAR on), after walking around Kokiri Forest for a while:1. The frame is serialized on the Presentation thread, the machine is mostly idle.
top -Hon the hookshot process:System-wide: CPU ~28%, GPU ~6% busy. So the ~250-295 ms
queue submitper frame looks like time blocked in the driver/kernel on that one thread, not CPU or GPU throughput.2. VRAM fills up over time and the frame rate gets worse (3-5 fps -> 1.4 fps). At scene entry the log showed ~1.5 GB used; later:
The kernel agrees (
/proc/<pid>/fdinfo):drm-resident-vram: 7190112 KiB,drm-resident-gtt: 3876968 KiB, and card-widemem_info_vram_used7754 of 8176 MB. With VRAM this full, amdgpu has to evict to GTT, which probably makes the submits slower still. But the ~250 ms submits were already there at ~1.5 GB VRAM, so this makes it worse rather than causing it.El rendimiento no es bueno en mi rx 6700 xt, ryzen 5 5600 24 gb ram, 8 fps como maximo en el bosque kokiri, linux cachy os hyperland (Wayland) puedo compartir mi hookshok_log
hookshot_log.txt
Tested
RX_VK_IMPORT_CHUNK_MBat 4, 16 (default) and 64, everything else the same (170ce53c,RX_VK_MIRROR_MB=192, Resizable BAR on), in Kokiri Forest / Link's house:RX_VK_IMPORT_CHUNK_MBslow presentqueue submit (avg of last 12)queue submit(in scene)So changing the chunk size does not fix it. 64 is clearly worse per present (fewer but larger imports, 46 vs 27 ms), and 4 is not better than 16. The default looks best, but the runs were not in exactly the same spot (the 16 MB run was outside in the forest, the others in Link's house), so treat small differences as noise. In every case the frame is still ~25-40x the GPU time.
Separate finding, the garbled 3D from my first post: it is the Insert menu's Resolution option, not random. With
Resolution: Autothe whole 3D scene is coloured noise (HUD and menu fine). With a fixed50%it renders correctly, and switching between the two in the same session flips it right away. That matches the "Resolution Auto black screen" AMD note in theless-stutterrelease, so on RADV / RX 7600 Auto still breaks the picture, just as noise instead of black. I'll add screenshots of both in a follow-up comment.I had the same problems than naneek-code, had steamos in my legión go amd z1 extreme 16gb ram. And the CPU goes up to. Gpu low in my opinion. Using the same setup but in windows. It freeze before the intro and blackscreen loading screen
Tested
40c3f851(the THP change). No difference on this machine: in-scenequeue submitis still ~285-640 ms per frame, ~2 fps, GPU time ~10-16 ms.THP was never involved here, so this is a second cause on top of the one
40c3f851fixes:/sys/kernel/mm/transparent_hugepage/enabledisalways [madvise] never(Ubuntu / KDE neon default)AnonHugePages: 0 kBin/proc/<pid>/smaps_rollupThat probably fixes it for distros that default to
always(CachyOS, Arch), like the RX 6700 XT report above, but not formadvisesystems.I also counted memory syscalls with
strace -c -f -e trace=madvise,mprotect,munmap,mmapattached for 15 s while standing in Link's house:So no
madviseat all (theVirtualFreedecommit path is not being hit), but ~132mmap+ ~132munmapper second, about 66 pairs per frame at ~2 fps. I don't know whether those touch the imported guest ranges or are driver-side BO maps, but if amunmapoverlaps a userptr import it would trigger the same re-pin. Happy to runstracewith arguments (-e trace=mmap,munmapfor a few seconds, to see sizes and addresses) if that helps.VRAM still fills quickly on this build (
VRAM: 7198 MB used of 7552 MB allowed, textures 3338 MB) withRX_VK_MIRROR_MB=192.Thanks for testing
40c3f851and for the strace counts; that points to a second cause onmadvisesystems. Yes, a few seconds ofstrace -e trace=mmap,munmapwith sizes and addresses would help.Here's the
strace -f -tt -e trace=mmap,munmapcapture, 5 s in Link's house (40c3f851, ~2-3 fps), compared against/proc/<pid>/mapstaken while the game was still running.None of them touch the guest window. Every call is a CPU mapping of a DRM buffer object on the render node:
/proc/<pid>/maps:0x100000000000-0x108000000000 rw-p(anonymous, 512 GB reserved).munmapaddresses are in0x79b9b7530000 .. 0x79b9ca68e000, next to libLLVM and the other driver mappings. 0 of 692 overlap the guest window, so they can't invalidate the userptr imports.sead::MainThread. That's ~140 map+unmap pairs per second, all 80 KB BOs, so it looks like a small buffer being created/mapped and destroyed per submit or per draw batch. It's cheap in itself (strace -c showed ~0.03 s of syscall time in 15 s), but it's churn on the same thread that sits invkQueueSubmit.So whatever re-pins the imports on
madvisesystems, it isn'tmadvise,munmapor THP. The remaining candidates for the userptr invalidation I can think of are the 95mprotectcalls in 15 s (if any of them land in the guest window) or page migration/compaction. I can stracemprotectwith addresses the same way if useful.One more checks:
foliage-linux-updater, nohookshot_env.txt):queue submitper framevi: frames presented, game frames)host memory import/guest memory imported in placeSame GPU work, ~400-1000x difference in the submit itself, so this really is the RADV/amdgpu submission path on Linux.
Full files for the comment above: strace capture, /proc//maps taken during it, and the Konsole run log.
strace-mmap-5s.txt
run8-konsole-log.txt
proc-pid-maps.txt