Game: Kokiri Forest and the market run ~30% slower when the game files weren't unpacked by Longshot #1939
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#1939
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens
If your game files (
romfs) weren't unpacked by Longshot's own builder, the game keeps reloading a part of HyruleField in the background while you're in Kokiri Forest or Castle Town's market. It never finishes. On a Steam Deck this
uses about 1.3 CPU cores the whole time and costs about a third of the frame rate (Kokiri 30 fps instead of 44). Nothing
on screen or in the log says so. Hyrule Field itself isn't affected.
Who is affected
Anyone whose
romfscomes from an incomplete dump (about 23.3 GB instead of 23.8 GB) and wasn't unpacked by LongshotBuilder from a
.nup. For example:romfsunpacked with another tool and chosen in the launcher as folders;romfscopied in by hand.A
.nupthat Longshot Builder unpacked already gets this fixed: it writeslongshot_damaged_files.txtinto the game folder.How to tell: open a terminal in the folder that holds
romfsand read the first four bytes of one texture. A wholefile starts
28 b5 2f fd; a damaged one starts00 00 00 00.Linux or Steam Deck:
Windows PowerShell:
Steps to reproduce
main(or the newest release) with a game folder whoseromfsshows00 00 00 00above.HOS_THREAD_TIMES=5tolongshot_env.txtin the game folder. The log then shows each thread's CPU use every 5seconds.
python -I scripts/longshot_ctl.py launch <game folder> --repo-save kokiri-start.logs/longshot_log.txtat theprof: threads in 5 s:lines. They showOptimizeChangeBancat about 80% and twoParallelDecompThreadentries together at about 55%, for as long as youstand there.
OptimizeChangeBancandParallelDecompThreaddrop out of theline, and the fps goes up.
Expected: once Kokiri has loaded, the background loader goes idle, and standing still costs no CPU.
Actual: the loader and the decoders stay busy for as long as you stand in Kokiri Forest or the market.
A line from step 5 (Steam Deck, Kokiri):
Measured
These numbers are from a Steam Deck OLED (SteamOS 3.9.5, Mesa RADV, GPU at 1600 MHz), with the Minimum preset, Render 50%,
the frame rate uncapped and
HOS_NATIVE_ZSTD=1. Each run starts the game, then stands for 60 s. There are two starts oneach side, and the stand-ins are read through
HOS_FS_OVERLAYas in the workaround.With the stand-ins, the Hyrule Field preload finishes in about 10 s at boot (
ocb: loaded: HyruleField_Child in 10.0 s)instead of never.
Cause
Incomplete dumps have three damaged textures that begin with zeros instead of a zstd frame:
Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zsTex/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zsTex/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zsThe first is used by the Great Fairy Fountain model
Land_FairyFountain_Base_A_01. The game's preloader(OptimizeChangeBank) loads it for Hyrule Field while Link is in Kokiri or the market, fails on the texture, and asks for
it again about 15 times a second, decoding the 10.8 MiB model each time. Longshot Builder copies a similar texture over
each damaged one (
nx2/nup.pystand_in_damaged, called fromnx2/make_game.pystand_in_damaged_files). But it doesthis only in a
romfsit unpacked itself (the.longshot_romfsstamp).check_unpacked_romfsonly looks for missingfiles, and the game checks nothing when it starts.
Workaround (changes none of your game files)
Put copies of the three textures the builder would use somewhere the game reads first. Those three surfaces then show
a similar texture: the Great Fairy Fountain's floor, the Water Temple's gate walls and one Ganon stage texture. With the
damaged files, the game leaves those models out instead.
Option A, a mod folder (the game's normal mod support; I have not tested this one myself, only option B). In the folder that holds
romfs, on Linux or the Deck:Windows PowerShell:
The
modsfolder goes besidebin, as in Using mods.The log should list the mod at start. To undo, delete
mods/damaged-texture-stand-ins.Option B,
HOS_FS_OVERLAY(how the numbers above were measured). Copy the same three files into a folder of yourown, laid out as
<folder>/Tex/<damaged name>. Then start the game withHOS_FS_OVERLAY=<folder>set in theenvironment. On the Deck, that means Steam's launch options:
HOS_FS_OVERLAY=/home/deck/longshot-stand-ins %command%.This doesn't work from
longshot_env.txt, which takes no paths. The log then hasfs: OpenFile(content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs): from the overlay.The real fix is still to dump the game again: a complete dump has whole files there.
Proposed fix
When the game opens one of these three files and it doesn't start as a zstd frame, it reads the builder's stand-in in
its place. It logs one warning naming both and saying the dump is incomplete. This works for every game folder, however
its
romfswas unpacked, and writes nothing to the player's files.zeldaootr
Possibly related, for context:
Checked every game folder we have on a Deck LCD (VanGogh): 27 folders across NVMe and SD card.
find romfs -type f | wc -l= 186,583,du -sb romfs= 26,236,422,768 bytes, the probe texture reads28 b5 2f fd, a random 25-file Tex sample is intact, and there are no zero-size files. None of our folders carrylongshot_damaged_files.txt(nothing needed standing in).HOS_PROFILEarms standing in Kokiri (merged-main runtime, pinned clocks, LCD) show about 1.1 cores total on-CPU with wait-dominated stacks — noOptimizeChangeBancorParallelDecompThreadentry at all. So on a complete-dump LCD this issue does not fire and does not contaminate published numbers.@zeldaootr answering your note: our files are clean, so the 1860/1875 numbers we posted stand as measured; they don't stack for us because there's nothing broken on our side to fix.
One gap from the launcher side: LongshotLauncher only surfaces an incomplete dump when the builder says so (
Problems.cskeys on "dump is incomplete" / "damaged file" in build output and tells the player to redump). A game folder that never went through the builder produces no message anywhere — consistent with the cause section here (check_unpacked_romfsonly looks for missing files; the game checks nothing at start). The proposed read-time stand-in is the first thing that would cover every folder however it was unpacked. We support it, and we can run the LCD (VanGogh) arm of it — your table is OLED-only and the panels aren't interchangeable.For closing: PR #546 covers new folder-based builds; this issue's runtime proposal covers pre-existing folders. If the runtime stand-in gets its own PR, this can close against it with the check and fingerprint above as the user-facing docs.
Thanks for checking all of that, and good to know your numbers stand: complete dumps don't have this, so the two changes not stacking on your Deck has another cause. The runtime stand-in has its own PR now, #1943. On my OLED with the damaged files it took Kokiri from 32 to 44 fps and the market from 21.5 to 26, and logged its warning once. An LCD run would be very welcome. Since your folders are complete, the fix should simply do nothing there (no warning line, same fps), which is worth confirming too. The file-count and size fingerprint you gave is a better check than my four bytes; I'll point the PR's docs at it.