Memory issue: Every area load decodes ~1 GB of ASTC again, revisits included: textures come back at new addresses and the address-keyed cache misses them #1069
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#1069
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Longshot version
v3.0.0-rc.4 (source
53f96d7805)Channel
Release candidate
What kind of problem
Low fps or stutter
What happened
Each area change has 200 to 700 ms frames, and going back to an area you just left is no smoother than the first visit.
Measured with a scripted teleport tour through the control bridge (
longshot_ctl.py launch --profile test, a copiedsave, headless), Kokiri Forest, Hyrule Field, Kakariko, Hyrule Field, Kokiri Forest, twice, 10 s between hops.
What the log shows:
vk: textures: decoded ASTCbefore and after each teleport), the same ona revisit as on the first visit, including loads where nothing had been evicted (
textures retired ... budget 0).(
vk: VRAM: ... N textures), so the same textures are being held more than once.textures retired ... budget N).program census (play): created 0), so this is not shader compilation. The slowframes' own breakdowns (
vk: slow submit) are in queue submit, dispatch, draw and descriptors.Checked with
RX_VK_LOG_UPLOADSaround a Hyrule Field revisit (Kakariko to Hyrule Field, second visit, nothingevicted for the budget on that load): every upload logged from the teleport on was
(new)(a cache entry made at thatload), none a re-upload of an existing entry; over the whole 200-line window, 192
(new), 7written, 1whole.The texture count went from 5,968 to 7,189 across that one revisit. So a revisit does not reuse or refresh the first
visit's entries: it makes new ones and the old ones stay.
Addresses checked with
RX_VK_WATCH_DIM=683x683over three Hyrule Field visits (with Kakariko between them, noevictions in the whole run): each visit made a new set of 683x683 textures, 54, 46 and 43 of them, every one at a
new address. Not one address is shared between any two visits. The sets have the same fingerprints each time (NVN
format, levels, layout, class and guest size: e.g. 26, 23 and 22 of
0x8c/0x30005/278016 B; 15, 14 and 13 of0x8b/0x30004/308224 B), and the earlier visits' copies were still alive when the later ones were made.Cause: in
gpu/vk/vk_texture.cpp,struct Key(around line 207) identifies a texture by its guest GPU address plustarget, size, levels, samples, layout and class, with no content hash. The game's reload puts the area's textures at
new addresses, so every revisit misses the cache, decodes and uploads them again, and the old copies stay until the
budget evicts them. (The texels themselves were not compared; the identical fingerprint sets per visit are the
evidence that they are the same textures.) On an AMD card (no ASTC sampling) that is about 1 GB of decode and BC3
re-encode per load.
Idea: a content hash (or a hash of the first block and the size) as a second lookup for decoded ASTC, or a disk cache of
the BC3 re-encodes, so a revisit reuses the decode.
Numbers from five runs (same route, 13 GB budget):
RX_VK_ASTC_BC_ALL=1RX_VK_ASTC_BC_ALL=1, repeatedRX_VK_UPLOAD_BUDGET_MS=16RX_VK_PREFETCH_CHUNKS=2RX_VK_ASTC_BC_ALL=1helps on a 24 GB card too (61% fewer evictions, worst frames down by about a fifth), but it onlyslows the growth; the re-decode on every load stays.
Related: #388 (memory grows with every area change), #939 (6 GB card over the allowance after a few area changes),
#1040 (APU budget thrashing, 100-450 ms re-decode stalls).
Where in the game
Any area change; measured between Kokiri Forest, Hyrule Field and Kakariko (child).
How often
Every time
Graphics card and driver version
AMD Radeon RX 7900 XTX (24 GB), AMD Software 26.10.1
CPU, RAM and operating system
Ryzen 7 5800X3D, 32 GB, Windows 11
Changes from the defaults
For the measurements:
RX_VK_VRAM_MB=13312, andRX_VK_ASTC_BC_ALL=1where the tablesays so. The build also has a co-op mod's patches applied (OoT Co-op v0.11.2, Longshot module); the tour ran solo, with
no session.
longshot_log.txt
Every area load decodes ~1 GB of ASTC again, revisits included: textures come back at new addresses and the address-keyed cache misses them
Longshot version: v3.0.0-rc.4 (source
53f96d7805), release candidateProblem: low fps / stutter, every time, on any area change (measured between Kokiri Forest, Hyrule Field and Kakariko, child)
GPU: AMD Radeon RX 7900 XTX (24 GB), AMD Software 26.10.1
System: Ryzen 7 5800X3D, 32 GB, Windows 11
Changes from defaults: for the measurements RX_VK_VRAM_MB=13312, and RX_VK_ASTC_BC_ALL=1 where the table says so. The build also has a co-op mod's patches applied (OoT Co-op v0.11.2, Longshot module); the tour ran solo, with no session.
Each area change has 200 to 700 ms frames, and going back to an area you just left is no smoother than the first visit. Measured with a scripted teleport tour through the control bridge (
longshot_ctl.py launch --profile test, a copied save, headless), Kokiri Forest, Hyrule Field, Kakariko, Hyrule Field, Kokiri Forest, twice, 10 s between hops.What the log shows:
vk: textures: decoded ASTCbefore and after each teleport), the same on a revisit as on the first visit, including loads where nothing had been evicted (textures retired ... budget 0).vk: VRAM: ... N textures), so the same textures are being held more than once.textures retired ... budget N).program census (play): created 0), so this is not shader compilation. The slow frames' own breakdowns (vk: slow submit) are in queue submit, dispatch, draw and descriptors.Checked with
RX_VK_LOG_UPLOADSaround a Hyrule Field revisit (Kakariko to Hyrule Field, second visit, nothing evicted for the budget on that load): every upload logged from the teleport on was(new)(a cache entry made at that load), none a re-upload of an existing entry; over the whole 200-line window, 192(new), 7written, 1whole. The texture count went from 5,968 to 7,189 across that one revisit. So a revisit does not reuse or refresh the first visit's entries: it makes new ones and the old ones stay.Addresses checked with
RX_VK_WATCH_DIM=683x683over three Hyrule Field visits (with Kakariko between them, no evictions in the whole run): each visit made a new set of 683x683 textures, 54, 46 and 43 of them, every one at a new address. Not one address is shared between any two visits. The sets have the same fingerprints each time (NVN format, levels, layout, class and guest size: e.g. 26, 23 and 22 of0x8c/0x30005/278016 B; 15, 14 and 13 of0x8b/0x30004/308224 B), and the earlier visits' copies were still alive when the later ones were made.Cause: in
gpu/vk/vk_texture.cpp,struct Key(around line 207) identifies a texture by its guest GPU address plus target, size, levels, samples, layout and class, with no content hash. The game's reload puts the area's textures at new addresses, so every revisit misses the cache, decodes and uploads them again, and the old copies stay until the budget evicts them. (The texels themselves were not compared; the identical fingerprint sets per visit are the evidence that they are the same textures.) On an AMD card (no ASTC sampling) that is about 1 GB of decode and BC3 re-encode per load.Idea: a content hash (or a hash of the first block and the size) as a second lookup for decoded ASTC, or a disk cache of the BC3 re-encodes, so a revisit reuses the decode.
Numbers from five runs (same route, 13 GB budget):
RX_VK_ASTC_BC_ALL=1RX_VK_ASTC_BC_ALL=1, repeatedRX_VK_UPLOAD_BUDGET_MS=16RX_VK_PREFETCH_CHUNKS=2RX_VK_ASTC_BC_ALL=1helps on a 24 GB card too (61% fewer evictions, worst frames down by about a fifth), but it only slows the growth; the re-decode on every load stays.Related: #388 (memory grows with every area change), #939 (6 GB card over the allowance after a few area changes), #1040 (APU budget thrashing, 100-450 ms re-decode stalls).
Screenshots or video
No response
Before you send it
Possibly related, for context:
Game: OOTRto Game: OOT remakeGame: OOT remaketo Memory issue: Every area load decodes ~1 GB of ASTC again, revisits included: textures come back at new addresses and the address-keyed cache misses themMeasurements from a Steam Deck OLED (RADV, 16 GB shared with the system,
RX_VK_VRAM_MB=2400) that show a second cause for the same symptom, in case it helps whoever picks this up.Route. A copied save in Kokiri Forest, then four teleports through the control bridge: Hyrule Field, Kokiri Forest, Hyrule Field, Kokiri Forest, about 145 s in each. Build: main
c167ec67with local changes on top, and for one tour a counter onnvn_texture_created/nvn_texture_releasedand on each cache entry's first upload, so read the numbers as one machine's.What the counters say on a return (two areas only):
textures retired ... budget Nat each area change read 921, 639, 595 + 316 and 783, with the cache at 1.9 to 2.1 GB of its 2,400 MB (Kokiri Forest is about 1.95 GB of images here, Hyrule Field about 1.3 GB).Effect, frames over 150 ms after each return:
The first visit to Hyrule Field is the same either way (1.5 to 2.0 s). At 4,000 MB the cache held 2.8 to 3.2 GB and still retired 572 textures for the budget over the tour: the two areas only just fit.
So there are two cases. With a budget smaller than two areas, a return decodes again because the renderer let go of the area just left while the game still holds its textures at the same addresses: a content hash would find nothing to reuse there, the images are gone. The case described above (the same textures back at new addresses, the old copies still alive) did not show on this two-area route; your route has a third area between the visits, which may be what makes the game free the first one's. I did not look at which textures the 100 to 150 made on each return are.
Possibly related for the small-budget case: #521 (opaque ASTC kept as BC1) would let more of two areas fit in the same budget; not tried here.