Builder: stand in for the damaged textures of a 23.3 GB dump (Water Temple walls, #1002) #1308

Merged

Joey-Magnus merged 1 commit from fix/damaged-dump-stand-ins into main

2026-11-09 23:11:07 +00:00

Joey-Magnus commented

2026-11-09 21:27:48 +00:00

(Migrated from github.com)

What

The incomplete 23.3 GB .nup (the complete one is about 23.8 GB) has three romfs .zs files whose first bytes are zeros. They are not zstd, and the game leaves out every model that uses one:

Damaged file	Where it shows
`Tex/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs`	most of the Water Temple's walls and floors (#1002, #1259, #718)
`Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs`	the Great Fairy Fountain's floor (#867, #960)
`Tex/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs`	part of Ganon's stage

The zeros are inside the stored data of the package's only title, so nothing in the .nup holds a whole copy.

After make_game.py unpacks a .nup (or reuses an earlier unpack, so folders that are already built get the fix on their next build), it:

finds the .zs files that do not start with the zstd magic (nup.damaged_game_files);
replaces each one with a whole file of the same kind next to it: the same last part of the name (_Nrm.bntx.zs for a normal map), then the longest common name start, then the nearest size. Other damaged files are never chosen. For the three files above, the picks are Rock_WaterTemple_Gate_P_01_Nrm, CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm and RBake_Scene_GanonStage_G_2_2_d0_t0;
prints that the dump is incomplete, lists each replacement, and says that dumping again restores the original look.

It writes only to the romfs that the builder unpacked. A --romfs folder belongs to the player and is not touched. This is the same idea as Lichii's zortos293/Lichii#740, done once at build time.

Tested

Linux, RTX 4070 Ti, the 23.3 GB dump, using the Water Temple save that was posted on Discord (slot 0, 1F). Before: the room's floor and walls are missing and you look straight into open water. With the stand-ins: the walls, floor, ceiling and door are all drawn and the room is walkable.
Read-only run of the picker against the full unpacked romfs: it finds exactly the three files above, and a cold-cache scan of the 102k files takes about 34 s.
nx2/test_nup_stand_in.py (new, synthetic, no game data) and python -I -B tests/run_all.py nup: all pass.

🤖 Generated with Claude Code

## What The incomplete 23.3 GB `.nup` (the complete one is about 23.8 GB) has three romfs `.zs` files whose first bytes are zeros. They are not zstd, and the game leaves out every model that uses one: | Damaged file | Where it shows | |---|---| | `Tex/Rock_WaterTemple_Gate_M_01_Nrm.bntx.zs` | most of the Water Temple's walls and floors (#1002, #1259, #718) | | `Tex/CmnTex_Map_Soil_FairyFountain_Floor_D_Nrm.bntx.zs` | the Great Fairy Fountain's floor (#867, #960) | | `Tex/RBake_Scene_GanonStage_G_2_1_d0_t0.bntx.zs` | part of Ganon's stage | The zeros are inside the stored data of the package's only title, so nothing in the `.nup` holds a whole copy. After `make_game.py` unpacks a `.nup` (or reuses an earlier unpack, so folders that are already built get the fix on their next build), it: 1. finds the `.zs` files that do not start with the zstd magic (`nup.damaged_game_files`); 2. replaces each one with a whole file of the same kind next to it: the same last part of the name (`_Nrm.bntx.zs` for a normal map), then the longest common name start, then the nearest size. Other damaged files are never chosen. For the three files above, the picks are `Rock_WaterTemple_Gate_P_01_Nrm`, `CmnTex_Map_Sand_FairyFountain_Sand_A_Nrm` and `RBake_Scene_GanonStage_G_2_2_d0_t0`; 3. prints that the dump is incomplete, lists each replacement, and says that dumping again restores the original look. It writes only to the romfs that the builder unpacked. A `--romfs` folder belongs to the player and is not touched. This is the same idea as Lichii's zortos293/Lichii#740, done once at build time. ## Tested - Linux, RTX 4070 Ti, the 23.3 GB dump, using the Water Temple save that was posted on Discord (slot 0, 1F). Before: the room's floor and walls are missing and you look straight into open water. With the stand-ins: the walls, floor, ceiling and door are all drawn and the room is walkable. - Read-only run of the picker against the full unpacked romfs: it finds exactly the three files above, and a cold-cache scan of the 102k files takes about 34 s. - `nx2/test_nup_stand_in.py` (new, synthetic, no game data) and `python -I -B tests/run_all.py nup`: all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hckrman101 (Migrated from github.com) approved these changes

2026-11-09 23:07:59 +00:00

hckrman101 (Migrated from github.com) left a comment

Looks good. The replacement goes through .part plus os.replace, --romfs is untouched, and the new test is discovered by run_all.

Nits:

the scan of every .zs runs on every --nup build, including the read-before path (about 34 s cold by your numbers);
--update without --nup doesn't apply the fix, as the description says;
detection is "doesn't start with the zstd magic", so a skippable frame would be flagged too, though on the real dump it found exactly the 3 expected files.

CI is green.

Looks good. The replacement goes through `.part` plus `os.replace`, `--romfs` is untouched, and the new test is discovered by run_all. Nits: - the scan of every .zs runs on every `--nup` build, including the read-before path (about 34 s cold by your numbers); - `--update` without `--nup` doesn't apply the fix, as the description says; - detection is "doesn't start with the zstd magic", so a skippable frame would be flagged too, though on the real dump it found exactly the 3 expected files. CI is green.

Joey-Magnus commented

2026-11-09 23:12:55 +00:00

(Migrated from github.com)

Thanks for the review. All three nits are addressed in 45f4900:

The scan on every build: it now runs once per unpack. .hookshot_romfs_checked stores the .hookshot_romfs stamp the check ran under, so a rebuild over the same .nup skips it. A fresh unpack deletes the note first, and the note goes through check_cache_paths like the other generated paths.
--update: it applies there too, but only to a romfs the builder unpacked itself (the .hookshot_romfs stamp must exist).
Skippable frames: nup.zstd_start accepts 0x184D2A50 to 0x184D2A5F, and the test has a case for it.

test_make_game_cache.test_same_source_reuses_the_cache now allows exactly that one added note, with the same stamp. The full tests/run_all.py run passes locally on Linux: 167 passed, 17 skipped for missing tools.

Thanks for the review. All three nits are addressed in 45f4900: - **The scan on every build:** it now runs once per unpack. `.hookshot_romfs_checked` stores the `.hookshot_romfs` stamp the check ran under, so a rebuild over the same `.nup` skips it. A fresh unpack deletes the note first, and the note goes through `check_cache_paths` like the other generated paths. - **`--update`:** it applies there too, but only to a romfs the builder unpacked itself (the `.hookshot_romfs` stamp must exist). - **Skippable frames:** `nup.zstd_start` accepts `0x184D2A50` to `0x184D2A5F`, and the test has a case for it. `test_make_game_cache.test_same_source_reuses_the_cache` now allows exactly that one added note, with the same stamp. The full `tests/run_all.py` run passes locally on Linux: 167 passed, 17 skipped for missing tools.

hckrman101 (Migrated from github.com) approved these changes

2026-11-09 23:14:35 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks for the quick follow-up, all three nits check out in 45f4900.

Once per unpack: the note holds the stamp, is removed before a fresh extract, and goes through check_cache_paths. An unpack from an older builder has the stamp but no note, so it is still checked once, then skipped. A romfs without the stamp (--romfs/--exefs) is left alone.
--update now runs the same step.
zstd_start is right: 0x184D2A50..5F is 5x 2A 4D 18 in LE bytes, and the len == 4 guard covers short files. Only the first four bytes are looked at, which is fine here.

The new tests cover the skippable frame, the range edges, and the stamp/once/re-check cases, and the cache test accounts for the extra note file. Looks good to me.

Thanks for the quick follow-up, all three nits check out in 45f4900. - Once per unpack: the note holds the stamp, is removed before a fresh extract, and goes through check_cache_paths. An unpack from an older builder has the stamp but no note, so it is still checked once, then skipped. A romfs without the stamp (--romfs/--exefs) is left alone. - --update now runs the same step. - zstd_start is right: 0x184D2A50..5F is 5x 2A 4D 18 in LE bytes, and the len == 4 guard covers short files. Only the first four bytes are looked at, which is fine here. The new tests cover the skippable frame, the range edges, and the stamp/once/re-check cases, and the cache test accounts for the extra note file. Looks good to me.