Game: Kokiri Forest and the market run ~30% slower when the game files weren't unpacked by Longshot #1939

Closed
opened 2026-11-20 23:38:16 +00:00 by zeldaootr · 3 comments
Contributor

What happens

If your game files (romfs) weren't unpacked by Longshot's own builder, the game keeps reloading a part of Hyrule
Field in the background while you're in Kokiri Forest or Castle Town's market. It never finishes. On a Steam Deck this
uses about 1.3 CPU cores the whole time and costs about a third of the frame rate (Kokiri 30 fps instead of 44). Nothing
on screen or in the log says so. Hyrule Field itself isn't affected.

Who is affected

Anyone whose romfs comes from an incomplete dump (about 23.3 GB instead of 23.8 GB) and wasn't unpacked by Longshot
Builder from a .nup. For example:

  • a game folder carried over from Hookshot;
  • a romfs unpacked with another tool and chosen in the launcher as folders;
  • a romfs copied in by hand.

A .nup that Longshot Builder unpacked already gets this fixed: it writes longshot_damaged_files.txt into the game folder.

How to tell: open a terminal in the folder that holds romfs and read the first four bytes of one texture. A whole
file starts 28 b5 2f fd; a damaged one starts 00 00 00 00.

Linux or Steam Deck:

head -c 4 romfs/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs | od -An -tx1

Windows PowerShell:

(([IO.File]::ReadAllBytes("$PWD\romfs\Tex\CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs"))[0..3] | ForEach-Object { '{0:x2}' -f $_ }) -join ' '

Steps to reproduce

  1. Use a build of current main (or the newest release) with a game folder whose romfs shows 00 00 00 00 above.
  2. Add HOS_THREAD_TIMES=5 to longshot_env.txt in the game folder. The log then shows each thread's CPU use every 5
    seconds.
  3. Start from the repository's Kokiri save, which leaves your own saves alone:
    python -I scripts/longshot_ctl.py launch <game folder> --repo-save kokiri-start.
  4. Load the save, then stand still in Kokiri Forest for a minute.
  5. Look at the fps counter (Insert menu), and in logs/longshot_log.txt at the prof: threads in 5 s: lines. They show
    OptimizeChangeBanc at about 80% and two ParallelDecompThread entries together at about 55%, for as long as you
    stand there.
  6. Do the same again with the workaround below in place. OptimizeChangeBanc and ParallelDecompThread drop out of the
    line, and the fps goes up.

Expected: once Kokiri has loaded, the background loader goes idle, and standing still costs no CPU.

Actual: the loader and the decoders stay busy for as long as you stand in Kokiri Forest or the market.

A line from step 5 (Steam Deck, Kokiri):

prof: threads in 5 s: sead::MainThread 28%; Presentation Thread 44%; Resource Memory 8%; ParallelDecompThread 33%; ParallelDecompThread 22%; ... OptimizeChangeBanc 80%;

Measured

These numbers are from a Steam Deck OLED (SteamOS 3.9.5, Mesa RADV, GPU at 1600 MHz), with the Minimum preset, Render 50%,
the frame rate uncapped and HOS_NATIVE_ZSTD=1. Each run starts the game, then stands for 60 s. There are two starts on
each side, and the stand-ins are read through HOS_FS_OVERLAY as in the workaround.

Area fps, as dumped fps, with the 3 stand-ins Loader + decode CPU, as dumped With the stand-ins
Kokiri Forest 30.5 / 29.5 44.2 / 44.4 ~79% + ~55% 0
Castle Town market 20.5 / 20.3 27.6 / 26.5 ~75% + ~45% 0
Hyrule Field (control) 47.8 / 47.9 47.8 / 47.3 0 0

With the stand-ins, the Hyrule Field preload finishes in about 10 s at boot (ocb: loaded: HyruleField_Child in 10.0 s)
instead of never.

Cause

Incomplete dumps have three damaged textures that begin with zeros instead of a zstd frame:

  • Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs
  • Tex/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs
  • Tex/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs

The first is used by the Great Fairy Fountain model Land_FairyFountain_Base_A_01. The game's preloader
(OptimizeChangeBank) loads it for Hyrule Field while Link is in Kokiri or the market, fails on the texture, and asks for
it again about 15 times a second, decoding the 10.8 MiB model each time. Longshot Builder copies a similar texture over
each damaged one (nx2/nup.py stand_in_damaged, called from nx2/make_game.py stand_in_damaged_files). But it does
this only in a romfs it unpacked itself (the .longshot_romfs stamp). check_unpacked_romfs only looks for missing
files, and the game checks nothing when it starts.

Workaround (changes none of your game files)

Put copies of the three textures the builder would use somewhere the game reads first. Those three surfaces then show
a similar texture: the Great Fairy Fountain's floor, the Water Temple's gate walls and one Ganon stage texture. With the
damaged files, the game leaves those models out instead.

Option A, a mod folder (the game's normal mod support; I have not tested this one myself, only option B). In the folder that holds romfs, on Linux or the Deck:

d=mods/damaged-texture-stand-ins/romfs/Tex && mkdir -p "$d"
cp romfs/Tex/CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs "$d/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs"
cp romfs/Tex/Rock_WaterTemple_Gate_P_01_Nrm.bntx.zs "$d/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs"
cp romfs/Tex/RBake_Scene_GanonStage_G_2_2_d0_t0.bntx.zs "$d/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs"

Windows PowerShell:

$d = "mods\damaged-texture-stand-ins\romfs\Tex"; New-Item -ItemType Directory -Force $d | Out-Null
Copy-Item romfs\Tex\CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs "$d\CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs"
Copy-Item romfs\Tex\Rock_WaterTemple_Gate_P_01_Nrm.bntx.zs "$d\Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs"
Copy-Item romfs\Tex\RBake_Scene_GanonStage_G_2_2_d0_t0.bntx.zs "$d\RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs"

The mods folder goes beside bin, as in Using mods.
The log should list the mod at start. To undo, delete mods/damaged-texture-stand-ins.

Option B, HOS_FS_OVERLAY (how the numbers above were measured). Copy the same three files into a folder of your
own, laid out as <folder>/Tex/<damaged name>. Then start the game with HOS_FS_OVERLAY=<folder> set in the
environment. On the Deck, that means Steam's launch options: HOS_FS_OVERLAY=/home/deck/longshot-stand-ins %command%.
This doesn't work from longshot_env.txt, which takes no paths. The log then has
fs: OpenFile(content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs): from the overlay.

The real fix is still to dump the game again: a complete dump has whole files there.

Proposed fix

When the game opens one of these three files and it doesn't start as a zstd frame, it reads the builder's stand-in in
its place. It logs one warning naming both and saying the dump is incomplete. This works for every game folder, however
its romfs was unpacked, and writes nothing to the player's files.

zeldaootr

<!-- Form fields (Game problem): Longshot version: v3.0.0-83 (main 2210931cf + a trace build; the same on current main 1d8e73acf) Channel: Built from source What kind of problem: Low fps or stutter Where in the game: Kokiri Forest (standing anywhere); Castle Town market How often: Every time Graphics card and driver version: Steam Deck OLED, Mesa RADV (SteamOS 3.9.5) CPU, RAM and operating system: Steam Deck OLED (Zen 2, 4 cores / 8 threads), 16 GB, SteamOS 3.9.5 Changes from the defaults: Minimum preset, Render 50%, frame rate uncapped; HOS_NATIVE_ZSTD=1 (the Deck default); HOS_THREAD_TIMES=5 for the thread line --> ## What happens If your game files (`romfs`) weren't unpacked by Longshot's own builder, the game keeps reloading a part of Hyrule Field in the background while you're in Kokiri Forest or Castle Town's market. It never finishes. On a Steam Deck this uses about 1.3 CPU cores the whole time and costs about a third of the frame rate (Kokiri 30 fps instead of 44). Nothing on screen or in the log says so. Hyrule Field itself isn't affected. ## Who is affected Anyone whose `romfs` comes from an incomplete dump (about 23.3 GB instead of 23.8 GB) and wasn't unpacked by Longshot Builder from a `.nup`. For example: - a game folder carried over from Hookshot; - a `romfs` unpacked with another tool and chosen in the launcher as folders; - a `romfs` copied in by hand. A `.nup` that Longshot Builder unpacked already gets this fixed: it writes `longshot_damaged_files.txt` into the game folder. **How to tell:** open a terminal in the folder that holds `romfs` and read the first four bytes of one texture. A whole file starts `28 b5 2f fd`; a damaged one starts `00 00 00 00`. Linux or Steam Deck: ```sh head -c 4 romfs/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs | od -An -tx1 ``` Windows PowerShell: ```powershell (([IO.File]::ReadAllBytes("$PWD\romfs\Tex\CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs"))[0..3] | ForEach-Object { '{0:x2}' -f $_ }) -join ' ' ``` ## Steps to reproduce 1. Use a build of current `main` (or the newest release) with a game folder whose `romfs` shows `00 00 00 00` above. 2. Add `HOS_THREAD_TIMES=5` to `longshot_env.txt` in the game folder. The log then shows each thread's CPU use every 5 seconds. 3. Start from the repository's Kokiri save, which leaves your own saves alone: `python -I scripts/longshot_ctl.py launch <game folder> --repo-save kokiri-start`. 4. Load the save, then stand still in Kokiri Forest for a minute. 5. Look at the fps counter (Insert menu), and in `logs/longshot_log.txt` at the `prof: threads in 5 s:` lines. They show `OptimizeChangeBanc` at about 80% and two `ParallelDecompThread` entries together at about 55%, for as long as you stand there. 6. Do the same again with the workaround below in place. `OptimizeChangeBanc` and `ParallelDecompThread` drop out of the line, and the fps goes up. **Expected:** once Kokiri has loaded, the background loader goes idle, and standing still costs no CPU. **Actual:** the loader and the decoders stay busy for as long as you stand in Kokiri Forest or the market. A line from step 5 (Steam Deck, Kokiri): ``` prof: threads in 5 s: sead::MainThread 28%; Presentation Thread 44%; Resource Memory 8%; ParallelDecompThread 33%; ParallelDecompThread 22%; ... OptimizeChangeBanc 80%; ``` ## Measured These numbers are from a Steam Deck OLED (SteamOS 3.9.5, Mesa RADV, GPU at 1600 MHz), with the Minimum preset, Render 50%, the frame rate uncapped and `HOS_NATIVE_ZSTD=1`. Each run starts the game, then stands for 60 s. There are two starts on each side, and the stand-ins are read through `HOS_FS_OVERLAY` as in the workaround. | Area | fps, as dumped | fps, with the 3 stand-ins | Loader + decode CPU, as dumped | With the stand-ins | |---|---|---|---|---| | Kokiri Forest | 30.5 / 29.5 | 44.2 / 44.4 | ~79% + ~55% | 0 | | Castle Town market | 20.5 / 20.3 | 27.6 / 26.5 | ~75% + ~45% | 0 | | Hyrule Field (control) | 47.8 / 47.9 | 47.8 / 47.3 | 0 | 0 | With the stand-ins, the Hyrule Field preload finishes in about 10 s at boot (`ocb: loaded: HyruleField_Child in 10.0 s`) instead of never. ## Cause Incomplete dumps have three damaged textures that begin with zeros instead of a zstd frame: - `Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs` - `Tex/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs` - `Tex/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs` The first is used by the Great Fairy Fountain model `Land_FairyFountain_Base_A_01`. The game's preloader (OptimizeChangeBank) loads it for Hyrule Field while Link is in Kokiri or the market, fails on the texture, and asks for it again about 15 times a second, decoding the 10.8 MiB model each time. Longshot Builder copies a similar texture over each damaged one (`nx2/nup.py` `stand_in_damaged`, called from `nx2/make_game.py` `stand_in_damaged_files`). But it does this only in a `romfs` it unpacked itself (the `.longshot_romfs` stamp). `check_unpacked_romfs` only looks for missing files, and the game checks nothing when it starts. ## Workaround (changes none of your game files) Put copies of the three textures the builder would use somewhere the game reads first. Those three surfaces then show a similar texture: the Great Fairy Fountain's floor, the Water Temple's gate walls and one Ganon stage texture. With the damaged files, the game leaves those models out instead. **Option A, a mod folder** (the game's normal mod support; I have not tested this one myself, only option B). In the folder that holds `romfs`, on Linux or the Deck: ```sh d=mods/damaged-texture-stand-ins/romfs/Tex && mkdir -p "$d" cp romfs/Tex/CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs "$d/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs" cp romfs/Tex/Rock_WaterTemple_Gate_P_01_Nrm.bntx.zs "$d/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs" cp romfs/Tex/RBake_Scene_GanonStage_G_2_2_d0_t0.bntx.zs "$d/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs" ``` Windows PowerShell: ```powershell $d = "mods\damaged-texture-stand-ins\romfs\Tex"; New-Item -ItemType Directory -Force $d | Out-Null Copy-Item romfs\Tex\CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs "$d\CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs" Copy-Item romfs\Tex\Rock_WaterTemple_Gate_P_01_Nrm.bntx.zs "$d\Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs" Copy-Item romfs\Tex\RBake_Scene_GanonStage_G_2_2_d0_t0.bntx.zs "$d\RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs" ``` The `mods` folder goes beside `bin`, as in [Using mods](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/docs/user/using-mods.md). The log should list the mod at start. To undo, delete `mods/damaged-texture-stand-ins`. **Option B, `HOS_FS_OVERLAY`** (how the numbers above were measured). Copy the same three files into a folder of your own, laid out as `<folder>/Tex/<damaged name>`. Then start the game with `HOS_FS_OVERLAY=<folder>` set in the environment. On the Deck, that means Steam's launch options: `HOS_FS_OVERLAY=/home/deck/longshot-stand-ins %command%`. This doesn't work from `longshot_env.txt`, which takes no paths. The log then has `fs: OpenFile(content:/Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs): from the overlay`. The real fix is still to dump the game again: a complete dump has whole files there. ## Proposed fix When the game opens one of these three files and it doesn't start as a zstd frame, it reads the builder's stand-in in its place. It logs one warning naming both and saying the dump is incomplete. This works for every game folder, however its `romfs` was unpacked, and writes nothing to the player's files. zeldaootr
Member

Possibly related, for context:

  • PR #546 WIP: fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does — Open PR adds stand-ins for damaged folder-based romfs builds; build-time related, but does not cover pre-existing folders
  • issue #996 Game: Huge performance hit just running short circles in specific spot of Hyrule Town Market — Market performance loss, but triggered by movement and a different reported cause
Possibly related, for context: - PR #546 WIP: fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does — Open PR adds stand-ins for damaged folder-based romfs builds; build-time related, but does not cover pre-existing folders - issue #996 Game: Huge performance hit just running short circles in specific spot of Hyrule Town Market — Market performance loss, but triggered by movement and a different reported cause
Member

Checked every game folder we have on a Deck LCD (VanGogh): 27 folders across NVMe and SD card.

  • Every full install matches the complete-unpack fingerprint exactly: find romfs -type f | wc -l = 186,583, du -sb romfs = 26,236,422,768 bytes, the probe texture reads 28 b5 2f fd, a random 25-file Tex sample is intact, and there are no zero-size files. None of our folders carry longshot_damaged_files.txt (nothing needed standing in).
  • Measured corroboration: our 60 s HOS_PROFILE arms standing in Kokiri (merged-main runtime, pinned clocks, LCD) show about 1.1 cores total on-CPU with wait-dominated stacks — no OptimizeChangeBanc or ParallelDecompThread entry at all. So on a complete-dump LCD this issue does not fire and does not contaminate published numbers.

@zeldaootr answering your note: our files are clean, so the 1860/1875 numbers we posted stand as measured; they don't stack for us because there's nothing broken on our side to fix.

One gap from the launcher side: LongshotLauncher only surfaces an incomplete dump when the builder says so (Problems.cs keys on "dump is incomplete" / "damaged file" in build output and tells the player to redump). A game folder that never went through the builder produces no message anywhere — consistent with the cause section here (check_unpacked_romfs only looks for missing files; the game checks nothing at start). The proposed read-time stand-in is the first thing that would cover every folder however it was unpacked. We support it, and we can run the LCD (VanGogh) arm of it — your table is OLED-only and the panels aren't interchangeable.

For closing: PR #546 covers new folder-based builds; this issue's runtime proposal covers pre-existing folders. If the runtime stand-in gets its own PR, this can close against it with the check and fingerprint above as the user-facing docs.

Checked every game folder we have on a Deck **LCD** (VanGogh): 27 folders across NVMe and SD card. - Every full install matches the complete-unpack fingerprint exactly: `find romfs -type f | wc -l` = **186,583**, `du -sb romfs` = **26,236,422,768 bytes**, the probe texture reads `28 b5 2f fd`, a random 25-file Tex sample is intact, and there are no zero-size files. None of our folders carry `longshot_damaged_files.txt` (nothing needed standing in). - Measured corroboration: our 60 s `HOS_PROFILE` arms standing in Kokiri (merged-main runtime, pinned clocks, LCD) show about 1.1 cores total on-CPU with wait-dominated stacks — no `OptimizeChangeBanc` or `ParallelDecompThread` entry at all. So on a complete-dump LCD this issue does not fire and does not contaminate published numbers. @zeldaootr answering your note: our files are clean, so the 1860/1875 numbers we posted stand as measured; they don't stack for us because there's nothing broken on our side to fix. One gap from the launcher side: LongshotLauncher only surfaces an incomplete dump when the *builder* says so (`Problems.cs` keys on "dump is incomplete" / "damaged file" in build output and tells the player to redump). A game folder that never went through the builder produces no message anywhere — consistent with the cause section here (`check_unpacked_romfs` only looks for missing files; the game checks nothing at start). The proposed read-time stand-in is the first thing that would cover every folder however it was unpacked. We support it, and we can run the LCD (VanGogh) arm of it — your table is OLED-only and the panels aren't interchangeable. For closing: PR #546 covers *new* folder-based builds; this issue's runtime proposal covers pre-existing folders. If the runtime stand-in gets its own PR, this can close against it with the check and fingerprint above as the user-facing docs.
Author
Contributor

Thanks for checking all of that, and good to know your numbers stand: complete dumps don't have this, so the two changes not stacking on your Deck has another cause. The runtime stand-in has its own PR now, #1943. On my OLED with the damaged files it took Kokiri from 32 to 44 fps and the market from 21.5 to 26, and logged its warning once. An LCD run would be very welcome. Since your folders are complete, the fix should simply do nothing there (no warning line, same fps), which is worth confirming too. The file-count and size fingerprint you gave is a better check than my four bytes; I'll point the PR's docs at it.

Thanks for checking all of that, and good to know your numbers stand: complete dumps don't have this, so the two changes not stacking on your Deck has another cause. The runtime stand-in has its own PR now, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1943. On my OLED with the damaged files it took Kokiri from 32 to 44 fps and the market from 21.5 to 26, and logged its warning once. An LCD run would be very welcome. Since your folders are complete, the fix should simply do nothing there (no warning line, same fps), which is worth confirming too. The file-count and size fingerprint you gave is a better check than my four bytes; I'll point the PR's docs at it.
Sign in to join this conversation.
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1939
No description provided.