Slowdown after moving between areas: GPU memory spills into system RAM #2412
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot#2412
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hookshot version
2.1.0
What kind of problem
Low fps or stutter
What happened
Slowdown after moving between areas: GPU memory seems to spill into system RAM and write-back copies drop from ~120 GB/s to ~6 GB/s (v2.1.0, RTX 5070 12 GB)
Summary
Right after a fresh start the game runs at 60 fps. After a few minutes of moving between areas it drops to 10–35 fps. Restarting and staying in one area gives 60 fps again.
I returned to the same place with the same draw count and the same frame cost went up about 3x. The game's own GPU timing shows the write-back copies (mirror -> guest memory) running at VRAM speed when fast and at PCIe speed when slow. The game's shared GPU memory (system RAM used by the GPU) grows from 2 GB to 6 GB while VRAM is at the limit.
Environment
hookshot_env.txt: onlyRX_VK_PASS_TIMING=1(and SLOW_LOG/FRAME_LOG); no other switches. Note: PASS_TIMING waits on the GPU per span, so absolute GPU times in those blocks are inflated, but the pattern is the same as in the normal "GPU time" lines.Steps (clock times from my notes)
Evidence 1: same place, same draws, ~3x the cost
Evidence 2: write-back copy speed (GPU pass timing blocks)
"wb" = copies mirror -> guest memory. MB and ms are per frame as the game reports them.
When fast the copies run at 100+ GB/s (VRAM speed). When slow they run at 6–7 GB/s, which is what a PCIe 3.0 x8 link carries. The draw passes also get slower (19 µs -> up to 255 µs per span), which looks like the GPU reading textures/buffers from system memory.
Evidence 3: VRAM and memory (game log + nvidia-smi + Windows counters)
Ruled out
Hypotheses (not confirmed)
Questions
RX_VK_MIRROR_MB,RX_VK_VRAM_MB, the VRAM compaction from #1821)?Not done yet
RX_VK_MIRROR_MB/RX_VK_VRAM_MBcaps.Attachments
hookshot_issue_log_excerpt_v2.txt: trimmed log with clock annotations (full log available).hookshot_monitor.csv: nvidia-smi / memory / Windows counters every 5 s, including the size of hookshot_log.txt so log lines can be matched to clock time.hookshot_issue_log_excerpt_v2.txt
Where in the game
Kokiri Forest, Hyrule Field and Castle Town, walking back and forth between them
How often
Every time
Graphics card and driver version
RTX 5070 12GB, NVIDIA 617.14
CPU, RAM and operating system
Ryzen 7 5700X3D, 32 GB, Windows 11 Pro 26200
Changes from the defaults
quality_preset=0, output_res=2048x1152, dlss=2, dlss_model=3, anisotropic_filtering=4, and the env line RX_VK_PASS_TIMING=1
Other software hooked into the game
No, nothing hooked in
hookshot_log.txt
hookshot_log.txt
Screenshots or video
No response
Before you send it
Same for me. Intel Core Ultra 9 275HX, 32GB DDR5 RAM (6400MT/s), RTX 5070TI Laptop GPU (12GB)
Also my logs.
hookshot_log.3.txt
hookshot_log.4.txt
hookshot_log.txt
hookshot_log.1.txt
hookshot_log.2.txt
It's been happening to me for a couple of versions now i have the same laptop specs and I still have issues with the game overflowing into system ram and using up all 12 GB of VRAM.
Can confirm.
Same slowdown on an 8 GB card under Linux. The log also shows pools left in system memory for good after one short VRAM shortage, which #2482 addresses.
Setup: RTX 3070 Laptop 8 GB, NVIDIA 620.70.06, CachyOS (kernel 7.3), main at
7759a6f(v2.1.0-rc.3-86-g7759a6f). DLSS (built with the SDK), output 1920x1080,RX_VK_VRAM_MB=3072,RX_VK_IMAGE_BLOCK_MB=64,RX_VK_COMPACT=1,RX_VK_FSR_FG=0. Mirror budget is the default for 8 GB: 2048 MB.Two sessions of about an hour each on the same build:
vi:lines)submita framememory synca frameSession 1 reached the mirror budget only near its end. Session 2 reached it about 10 minutes in. VRAM was at its limit then, and image allocations started failing:
From then until the session ended (53,000 log lines later), the per-600-submit summaries show the same pools in state 3 (
MIRROR_REFUSED), with 200-560 MB read over PCIe in each window:None of these pools went back to state 0 or 2. That includes a stretch about 25 minutes in when VRAM dropped to 4.3 of 7.5 GB. In
mirror_make(gpu/vk/vk_memory.cpp), a failedcreate_device_buffersetsMIRROR_REFUSED, and onlymirror_freesets the state back, which a refused pool never reaches. So a short allocation failure leaves the pool in system memory until its import is given back. #2482 (retry after VRAM pressure) changes exactly this, so I've linked it here.The retry alone may not fix everything. In session 2 the mirrors stayed at 2016-2048 of the 2048 MB budget for the rest of the session, so a retried pool still has to compete for that budget (#1377 would choose which pools get it). For now I lowered
RX_VK_VRAM_MBfrom 3072 to 2560 to leave VRAM for the mirrors. I'll report back after I've played on that setting.Thanks for the report. Reports of the same problem are now collected in #993, which lists this one with your details and links back here, so one fix closes them together and the open list stays readable. Follow #993 for the fix, and add new details there (or here). If this turns out to be a separate problem it will be reopened.
Follow-up to my comment above. I tested #2482 (retry refused mirrors) merged into main at
57ed3aaon the same 8 GB card, and it fixes the part where pools stayed refused for good. VRAM still filled up when I left the Water Temple for Lake Hylia, but the pools got their mirrors back within about 3 minutes. Pools appeared in state 3 in 4 summaries, against 23,649 in the bad session.memory syncstayed at 0.7-0.9 ms the whole session. Numbers by scene size are in #2482.An update to what I wrote above: lowering
RX_VK_VRAM_MBto 2560 did not help. VRAM still filled up, textures were missing or blurry, and fps was lower. I went back to 3072.