Builder: pre-#456 unpacked romfs retains outer .nup Zstd frame on damaged .zs files, bypassing damaged_game_files() on --update (v2.0.12–v2.1.0) #2361

Closed

opened 2026-11-11 15:57:35 +00:00 by andreluisfc · 1 comment

andreluisfc commented

2026-11-11 15:57:35 +00:00

(Migrated from github.com)

Summary

Game directories whose romfs was unpacked from the incomplete 23.3 GB .nup (25,020,741,061 bytes) before PR #456 (2026-11-08) and later updated to v2.0.12–v2.1.0 via HookshotBuilder / make_game.py --update still exhibit invisible walls and floors in the Water Temple (#1259), Great Fairy Fountain (#867, #1700), and Ganon's Stage (#1002), even though stand_in_damaged_files() (#1308, #1373) ran and wrote .hookshot_romfs_checked.

Root Cause

Pre-#456 nup.py wrote the outer .nup Zstd frame directly to romfs/Tex/*.bntx.zs:
In the 23.3 GB .nup, the 3 damaged .zs members are stored with outer Zstd compression (codec == 1, stored < size), and their inner decompressed payloads begin with 0x4000 zero bytes (00 00 00 00) instead of a Zstd frame header:
- Tex/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs (size=452972, stored=444818)
- Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs (size=216015, stored=203753)
- Tex/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs (size=1901341, stored=1889157)
Before PR #456, Nup.is_packed(r, name) did not check r[5] == 1 (codec == 1); for .zs files where frame_content_size(head) == r[2], it fell back to:
```
return size == r[2] and self.inner_is_zstd(r)
```
Because the inner payload of those 3 damaged members begins with 00 00 00 00, inner_is_zstd(r) returned False. The old unpacker therefore treated all 3 members as uncompressed raw extents (packed = False) and copied size raw bytes from offset off in the .nup—consisting of the still-compressed outer .nup Zstd frame (stored bytes starting with 28 B5 2F FD) plus size - stored trailing bytes from the next extent—directly to romfs/Tex/*.bntx.zs on disk, and wrote .hookshot_romfs.
--update in v2.0.12–v2.1.0 skips re-unpacking romfs and damaged_game_files(romfs) only checks the first 4 bytes (28 B5 2F FD):
When an affected install is updated in v2.0.12–v2.1.0:
- make_game.py sees .hookshot_romfs matches unpacked by this builder: <nup_name> 25020741061, so it skips unpack_nup_dir(nup, "romfs", ...) (meaning neither PR #456's codec == 1 unpack nor PR #1885's HeadSink nor PR #2019's extraction digest check runs).
- stand_in_damaged_files() runs nup.damaged_game_files(romfs), which opens each .zs file on disk and checks only zstd_start(f.read(4)) (28 B5 2F FD or skippable frame magic).
- Because the 3 corrupted files on disk begin with the un-unpacked outer .nup Zstd frame (28 B5 2F FD), zstd_start() returns True for all 3:
  - Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs (452,972 bytes on disk, starts with 28 b5 2f fd a0 6c e9 06...): Zstd-decompresses to 452,972 bytes of 00 00 00 00... (unused_data=8154).
  - CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs (216,015 bytes on disk, starts with 28 b5 2f fd a0 cf 4b 03...): Zstd-decompresses to 216,015 bytes of 00 00 00 00... (unused_data=12262).
  - RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs (1,901,341 bytes on disk, starts with 28 b5 2f fd a0 1d 03 1d...): Zstd-decompresses to 1,901,341 bytes of 00 00 00 00... (unused_data=12184).
- Consequently, stand_in_damaged() replaces 0 files and writes .hookshot_romfs_checked, causing subsequent updates (such as v2.1.0) to skip stand_in_damaged_files() immediately (check_files: 0.0s).
- At runtime, hookshot.exe decompresses the outer .nup Zstd frame on disk into 00 00 00 00... instead of a BNTX texture, so the affected walls and floors fail to render.

Suggested Fix

Detect pre-#456 outer-frame-wrapped damaged .zs files in damaged_game_files(romfs) (or stand_in_damaged_files()):
- For .zs files on disk (or when checking an existing romfs without .hookshot_romfs_heads), either decompress the first few bytes of the Zstd stream to verify the inner header is not 00 00 00 00 (e.g. valid BNTX magic for .bntx.zs or non-zero header, and/or check for unused_data), or when --nup is provided, cross-check against the .nup TOC SHA-256 / codec == 1 damaged extents.
Bump the .hookshot_romfs_checked marker version/format:
- Because affected installs that updated to v2.0.12–v2.1.0 already have .hookshot_romfs_checked matching .hookshot_romfs, bumping the marker string (similar to the versioned marker proposed in #2096) will ensure already-updated game folders re-run the check once and replace the 3 files with their healthy stand-ins (Rock_WaterTemple_Gate_P_01_Nrm.bntx.zs, CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs, RBake_Scene_GanonStage_G_2_2_d0_t0.bntx.zs).

### Summary Game directories whose `romfs` was unpacked from the incomplete 23.3 GB `.nup` (`25,020,741,061` bytes) before PR #456 (2026-11-08) and later updated to v2.0.12–v2.1.0 via `HookshotBuilder` / `make_game.py --update` still exhibit invisible walls and floors in the Water Temple (#1259), Great Fairy Fountain (#867, #1700), and Ganon's Stage (#1002), even though `stand_in_damaged_files()` (#1308, #1373) ran and wrote `.hookshot_romfs_checked`. ### Root Cause 1. **Pre-#456 `nup.py` wrote the outer `.nup` Zstd frame directly to `romfs/Tex/*.bntx.zs`:** In the 23.3 GB `.nup`, the 3 damaged `.zs` members are stored with outer Zstd compression (`codec == 1`, `stored < size`), and their inner decompressed payloads begin with `0x4000` zero bytes (`00 00 00 00`) instead of a Zstd frame header: - `Tex/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs` (`size=452972`, `stored=444818`) - `Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs` (`size=216015`, `stored=203753`) - `Tex/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs` (`size=1901341`, `stored=1889157`) Before PR #456, `Nup.is_packed(r, name)` did not check `r[5] == 1` (`codec == 1`); for `.zs` files where `frame_content_size(head) == r[2]`, it fell back to: ```python return size == r[2] and self.inner_is_zstd(r) ``` Because the inner payload of those 3 damaged members begins with `00 00 00 00`, `inner_is_zstd(r)` returned `False`. The old unpacker therefore treated all 3 members as uncompressed raw extents (`packed = False`) and copied `size` raw bytes from offset `off` in the `.nup`—consisting of the **still-compressed outer `.nup` Zstd frame** (`stored` bytes starting with `28 B5 2F FD`) plus `size - stored` trailing bytes from the next extent—directly to `romfs/Tex/*.bntx.zs` on disk, and wrote `.hookshot_romfs`. 2. **`--update` in v2.0.12–v2.1.0 skips re-unpacking `romfs` and `damaged_game_files(romfs)` only checks the first 4 bytes (`28 B5 2F FD`):** When an affected install is updated in v2.0.12–v2.1.0: - `make_game.py` sees `.hookshot_romfs` matches `unpacked by this builder: <nup_name> 25020741061`, so it skips `unpack_nup_dir(nup, "romfs", ...)` (meaning neither PR #456's `codec == 1` unpack nor PR #1885's `HeadSink` nor PR #2019's extraction digest check runs). - `stand_in_damaged_files()` runs `nup.damaged_game_files(romfs)`, which opens each `.zs` file on disk and checks only `zstd_start(f.read(4))` (`28 B5 2F FD` or skippable frame magic). - Because the 3 corrupted files on disk begin with the un-unpacked outer `.nup` Zstd frame (`28 B5 2F FD`), `zstd_start()` returns `True` for all 3: - `Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs` (`452,972` bytes on disk, starts with `28 b5 2f fd a0 6c e9 06...`): Zstd-decompresses to `452,972` bytes of `00 00 00 00...` (`unused_data=8154`). - `CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs` (`216,015` bytes on disk, starts with `28 b5 2f fd a0 cf 4b 03...`): Zstd-decompresses to `216,015` bytes of `00 00 00 00...` (`unused_data=12262`). - `RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs` (`1,901,341` bytes on disk, starts with `28 b5 2f fd a0 1d 03 1d...`): Zstd-decompresses to `1,901,341` bytes of `00 00 00 00...` (`unused_data=12184`). - Consequently, `stand_in_damaged()` replaces **0 files** and writes `.hookshot_romfs_checked`, causing subsequent updates (such as v2.1.0) to skip `stand_in_damaged_files()` immediately (`check_files: 0.0s`). - At runtime, `hookshot.exe` decompresses the outer `.nup` Zstd frame on disk into `00 00 00 00...` instead of a `BNTX` texture, so the affected walls and floors fail to render. ### Suggested Fix 1. **Detect pre-#456 outer-frame-wrapped damaged `.zs` files in `damaged_game_files(romfs)` (or `stand_in_damaged_files()`):** - For `.zs` files on disk (or when checking an existing `romfs` without `.hookshot_romfs_heads`), either decompress the first few bytes of the Zstd stream to verify the inner header is not `00 00 00 00` (e.g. valid `BNTX` magic for `.bntx.zs` or non-zero header, and/or check for `unused_data`), or when `--nup` is provided, cross-check against the `.nup` TOC SHA-256 / `codec == 1` damaged extents. 2. **Bump the `.hookshot_romfs_checked` marker version/format:** - Because affected installs that updated to v2.0.12–v2.1.0 already have `.hookshot_romfs_checked` matching `.hookshot_romfs`, bumping the marker string (similar to the versioned marker proposed in #2096) will ensure already-updated game folders re-run the check once and replace the 3 files with their healthy stand-ins (`Rock_WaterTemple_Gate_P_01_Nrm.bntx.zs`, `CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm.bntx.zs`, `RBake_Scene_GanonStage_G_2_2_d0_t0.bntx.zs`).

NotRexed commented

2026-11-11 21:29:21 +00:00

(Migrated from github.com)

Thanks for the report. Reports of the same problem are now collected in #2505, which lists this one with your details and links back here, so one fix closes them together and the open list stays readable. Follow #2505 for the fix, and add new details there (or here). If this turns out to be a separate problem it will be reopened. Pull request #2391 works on this.