perf(host): the game's texture heap is compacted at most once a second, not at every frame (HOS_TEX_COMPACT_MS) #1478
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1478
Loading…
Reference in a new issue
No description provided.
Delete branch "GummiNaomi/Longshot:perf/tex-compact-gap"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: medium · Tested on: Windows 11, RTX 4070 SUPER (NVIDIA 617.42), Ryzen 9 9950X3D · Needs: a long played session; Linux; an AMD card; a Steam Deck or another CPU of four cores
TL;DR
In busy places like Castle Town's market the game often hitched every few frames, even standing still. The game was tidying its texture memory at every frame, and each texture it moved had to be set up again by Longshot. This PR lets the game start that tidying at most once a second. Standing in the market on my PC, frames too long for 60 fps fell from 5 to 13 % to 1 to 1.5 %, and six areas look the same as before; a new test covers the rule and the patch.
Breakdown
Why
Standing in Castle Town's market, some starts of the same build have 1 to 3 % of frames over 16.7 ms and others 10 to 14 %. In the rough ones the long frames come every 4, 10 or 14 frames, and each carries a batch of new textures: about 23 of them (10 MB) in one frame, 3 to 5 ms of uploads, descriptor builds and memory sync.
Those textures are not new to the game. The game keeps its textures' storage in one heap of 2,400 MB, which in the market stays at its line of 5 % free (read from the running game: 120.1 to 120.3 MB free, the line at 120.0). Its texture data manager's per-frame step (
f_80797240) asks the heap's Compaction thread for a round at every frame none is running (the manager's flag bit 1, which its setup sets once, unconditionally), and every texture a round moved is finalized and made again at its new address (f_80797da0). On the console that is a copy and a new texture object. Here each one is a new texture to the renderer: its storage is hashed to find the image it had (#1222), an image is made and that one's copied in, and the bytes the game moved are synced to the mirrors.How I found the caller: a temporary log of the game's call chain at
nvnTextureFinalize(not in this PR) named one pool B job for the asset textures; the manager's fields are from reading that job's code and from the running game. With the flag's bit cleared in a running test game by a write from outside, the batches stopped at once and the long frames went from 3.4 to 4.0 % to 0.3 to 0.5 % in each of the three runs measured (a fourth left no trace).The market's rough frame times are reported in #597, #1153 and #996. I have not shown that any of those is this; they are the same place.
What changed
The texture data manager asks for a compaction round at most once a second. The round itself is the game's, untouched: what it moves, how much, and how its textures are made again.
host/oot_tex_compact.hholds the rule (a gap of 0 is the game's own; below 0 never; else at least that long from one round's start to the next) and the switch's reading.host/oot_tex_compact.cholds the state, the default of 1000 ms and two log lines: what was chosen, and every 3,600 asks how many rounds were let through.nx2/oot_patches.pypatch_tex_compactmakes the request's second test ask the host too (!((flags >> 1) & 1) || !hos_oot_tex_compact_due()), inf_80797240only. A file patched before, or whose code differs (another game version), is left as it is, and the report warns when the place is not found exactly once. The manager only asks when no round is running, so a round that has started always finishes.HOS_TEX_COMPACT_MS=<ms>sets another gap;0is at every frame, as the game has it;offnever asks. Rows ingpu/vk/switches.defanddocs/development.md.tests/test_tex_compact.py.docs/tested/.Measurements
Verdict: standing in the market, frames over 16.7 ms fall from 5.4 and 12.6 % to 1.0 and 1.5 %, the 99th percentile from 18.5 and 23.5 ms to 16.7 and 17.0 ms, and the frame rate rises from 70.8 and 67.2 to 75.4 and 74.9. The heap is still compacted: 89 and 84 rounds were asked for in those runs.
Hardware: Windows 11 Pro 26200, RTX 4070 SUPER 12 GB (NVIDIA 617.42), Ryzen 9 9950X3D (16 cores, SMT off), 32 GB
Scene: the repository's
kokiri-startsave, the debug teleport toMarket_Child, then 55 s standing, no input; headless,scripts/longshot_ctl.py launch --profile test, uncapped (HOS_FPS_MAX=1000)Method: one build, the two sides in turn, twice; the last 2,600 frames of the frame trace (
RX_VK_FRAME_TRACE=1, about 35 s)Baseline: this PR's code with
HOS_TEX_COMPACT_MS=0, which is main's behaviour (the game's own request at every frame). The build is main50170f81with this change in the working tree; the PR's head,54a42715, is that change unchanged on main359131b4, and was not measured again after the rebaseHow to reproduce:
HOS_TEX_COMPACT_MS=0against no switch, standing in the market for a minute afterlongshot_ctl.py teleport Market_Child; the log'sdecoded ASTC used again: N texturesline counts the textures the game made again, andtex: the game's texture heap: N compaction rounds asked forthe roundsHOS_TEX_COMPACT_MS=0: this change on main @50170f81(two runs)50170f81(two runs)The choice of a second, from an earlier set on the same code with the default still 0 and the switch set by hand (the same scene and method, two runs each):
HOS_TEX_COMPACT_MS=0=1000=3000=offA second gives the frames of three seconds or of never, and keeps the most compaction of the three.
Area changes, with no compaction at all (the outside write, before the switch existed; one tour each, 8 minutes, 29 and 30 debug teleports round six areas): 24,327 texture uploads against 48,758 with the game's own; frames of 50 ms or more 125 (11.4 s) against 156 (14.5 s); none of 250 ms or more against 2. That is the
offend, not this PR's default, and one tour a side.The picture: one full-size shot in each of six areas (Kokiri Forest, the market, Hyrule Field, Kakariko, the Lost Woods, Zora's River) with
HOS_TEX_COMPACT_MS=0and with=1000, compared by eye: the same things are drawn on both, and what differs is what two runs differ by (the pose, Navi, fireflies, the camera still settling after the save's load).Flags and settings
One new switch.
HOS_TEX_COMPACT_MS1000(unset);0the game's own;offneverTesting
Tested: Windows 11, RTX 4070 SUPER, NVIDIA 617.42: the four runs above on this PR's code, the eight of the earlier set and the two tours, no error or device loss in any log; the six areas' shots by eye.
tests/test_tex_compact.pypasses (the rule and the switch's values compiled alone with clang, the patch over the step as lifted,patch_file's count);patch_tex_compactover this PC's liftedcode_004.cfinds the place once and leaves a patched file as it is.python -I -B tests/run_all.pywith clang here, this PR @54a42715: 399 passed, 113 failed, 229 skipped; main @359131b4in the same session: 399 passed, 112 failed, 229 skipped. The 112 are the same on both (this PC's toolchain: no gcc, Windows line endings, fixtures needing tools not installed;tests/test_switch_table.pyfails on main itself now,RX_VK_INVALIDATE_WAITlisted as gone and still read). The one more here istests/test_longshot_ctl.py, which passes run by itself on this PR (51 tests, 2 skipped) and does not touch this change.Not tested: a session longer than a few minutes with the default, and so what the heap does when it is compacted a sixtieth as often for hours (a texture that cannot be placed would not be in the log); walking and camera turns; adult Link, dungeons, cutscenes; Linux; AMD and Intel cards; a CPU with few cores; a build without the supported game version (the patch leaves other code as it is: by the test only).
Must be tested:
HOS_TEX_COMPACT_MS=0, expecting no crash, the same picture, and fewer frames over 16.7 ms with no switch (Test Farm: @Beanstalk linux)Risks
What could break: the game's heap is compacted far less often than the game asks for, so it can be more fragmented than on the console. If a large texture then cannot be placed, I do not know what the game does: I did not find that path, and nothing in these runs showed it (the heap moved between 120 MB and 1,073 MB free over a tour with no compaction at all, and the game kept dropping and loading textures). A round still runs every second, and a round that has started always finishes.
Turning it off:
HOS_TEX_COMPACT_MS=0.Compatibility: no saved data or file format changes. A build of another game version keeps the game's own behaviour (the patch does not apply; the report says so).
Issues
Not shown to fix any, the same place: #597, #1153, #996. Searched
compaction,texture heap,market,stutter,long frame: no issue or pull request on the game's own texture heap compaction. #1222 is the reuse these textures go through; #1281 (open) touches how they are hashed.Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each new measured run (format indocs/tested-configurations.md).tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
@Beanstalk linux and windows amd: stand in Castle Town's market for a minute (the debug teleport to
Market_Child, no input), once with this PR's default and once withHOS_TEX_COMPACT_MS=0(the game's own compaction request at every frame, as on main). Expecting: no crash; the same picture on both; with the default no more frames over 16.7 ms than with the switch at 0 (here, on a 9950X3D with an RTX 4070 SUPER on Windows: 1.0 and 1.5 % against 5.4 and 12.6 %); and in the log the linetex: the game's texture heap is compacted at most every 1000 mswith the default and no such line with the switch at 0. A build on Linux is the other thing this asks for: the checklist's Linux build box is still open.🤖 Generated with Claude Code
@GummiNaomi approval of
54a4271recorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.Beanstalk test farm
Commit
54a4271· tier double-reviewed · approved by GummiNaomi, futondestroyer · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)On Linux and Windows AMD, the farm builds both revisions and idles in the child Market for one minute per run, using the PR default and its zero-gap switch. Screenshots and p99 frame times are compared with main; crashes, substantial picture differences, or a p99 regression fail the plan. The farm cannot directly compare the two PR settings’ counts of frames over 16.7 ms or apply the requested setting-specific log checks, so p99 is a proxy. Gameplay takes about 24 minutes per PC, apart from timed lift builds; either hardware target waits if no matching PC is online.
0(PR only).kokiri-start(settings vsync=2, frame_rate_cap=6, frame_generation=0), then teleport to Market, measure market for 60 s (after 5 s warm-up), take the picture market-default to compare with main; 1 run on each side.kokiri-start(settings vsync=2, frame_rate_cap=6, frame_generation=0), then teleport to Market, measure market for 60 s (after 5 s warm-up), take the picture market-zero-gap to compare with main; 1 run on each side.Done (started 22:40 UTC). Run 2: re-planned by futondestroyer. Comment
@Beanstalkto run it again.This diff touches sensitive paths, so it needs two members of
BigBeans:nx2/oot_patches.py@Beanstalk
@Beanstalk linux and windows amd: stand in Castle Town's market for a minute (the debug teleport to Market_Child, no input), once with this PR's default and once with HOS_TEX_COMPACT_MS=0 (the game's own compaction request at every frame, as on main). Expecting: no crash; the same picture on both; with the default no more frames over 16.7 ms than with the switch at 0 (here, on a 9950X3D with an RTX 4070 SUPER on Windows: 1.0 and 1.5 % against 5.4 and 12.6 %); and in the log the line tex: the game's texture heap is compacted at most every 1000 ms with the default and no such line with the switch at 0. A build on Linux is the other thing this asks for: the checklist's Linux build box is still open.
WIP: perf(host): the game's texture heap is compacted at most once a second, not at every frame (HOS_TEX_COMPACT_MS)to perf(host): the game's texture heap is compacted at most once a second, not at every frame (HOS_TEX_COMPACT_MS)BeanIssues referenced this pull request2026-11-18 00:02:25 +00:00
Veix referenced this pull request2026-11-18 10:34:33 +00:00