MagicBeanDev/Longshot

Watch

Fork 99

fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does #546

Open

jarynclouatre wants to merge 1 commit from jarynclouatre/Longshot:fix/romfs-folder-stand-ins into main

jarynclouatre commented

2026-11-14 19:03:35 +00:00

Contributor

Type: fix · Risk: medium · Tested on: Linux x86-64 (game-free checks only) · Needs: any OS, an incomplete dump extracted to folders

Why

An incomplete dump (about 23.3 GB instead of 23.8) has texture files whose start is zeros, and the game then leaves out every model that uses one: most of the Water Temple's walls and floors, the Great Fairy Fountain's floor (#1002). stand_in_damaged_files in nx2/make_game.py replaces them with whole files of the same kind and says so, but only in a romfs the builder unpacked from a .nup (its .longshot_romfs stamp) and on --update of such a folder. A game built from folders (--romfs with --exefs, or --game with a folder) has the damaged files linked in as they are, and nothing is said.

The folder way was left out so the player's own folder is never written to (bd527024: "--romfs folders are not touched"). The game folder's romfs is a set of hard links to that folder, and nup.stand_in_damaged writes each stand-in to <file>.part and then replaces the name with it, so the stand-in can go in the game folder without writing the player's file.

How to reproduce

Seen on: found reading the code; not seen in the game
Settings and switches: --romfs with --exefs, or --game with a folder; an incomplete dump of about 23.3 GB instead of 23.8.
Steps: Build from the incomplete dump's folders, check the build's messages, then look at the Water Temple's walls and floors or the Great Fairy Fountain's floor.
Expected: Damaged files are reported and stand-ins are used where available without writing the player's folder.
Actual: The damaged files are linked in as they are and nothing is said; the game leaves out every model that uses one.

What changed

Builds from folders now check for damaged texture files and use stand-ins in the game folder without writing the player's files. Refused replacements are listed and tried again on the next build; a romfs that is the player's folder itself, or holds it, is only reported.

build (nx2/make_game.py, nx2/nup.py): the folder way calls stand_in_damaged_files(out, romfs, a.romfs) after linking. For a folder its stamp is the folder's path and file count, kept in .longshot_romfs_checked as for an unpack, so the roughly 100,000 files are opened once per folder and not at every build. A stand-in replaces the game folder's name of the damaged file; the player's file and its other names keep their bytes. A file that link_or_copy puts in afresh (one deleted from the game folder, a stand-in among them) clears that stamp, so it is looked at again; a rescan of a folder build that finds nothing keeps the list of what was replaced. nup.stand_in_damaged removes an existing <file>.part before copying, so a .part linked in from the player's folder is not written through. A replacement Windows refuses (a read-only file: through a hard link the attribute is the player's file's own, so it is not cleared; or a file another program has open, such as a virus scanner right after the linking) leaves that file as linked, removes its .part, and lists it as "not replaced, Windows refused it (read-only in your folder, or open in another program); the next build tries again"; the build goes on, as it did before, and the check is not marked done, so the next build tries that file again. The .nup way's wording and its stop for a file with no stand-in are unchanged. When the game folder's romfs is the player's folder itself, or holds it, the damaged files are listed (printed and in longshot_damaged_files.txt) and not replaced, and the build goes on. The .nup and --update ways are unchanged; the scan's progress line says "once per unpack or folder".
tests (tests/test_make_game_cache.py): a chosen romfs with a .zs that starts with zeros beside a whole one of the same kind, and a second hard link to the damaged file: the game folder's file is the stand-in, the player's file and its other name still hold their bytes (link count 2), the note names it, and a second build from the same folder opens no file again. A leftover .part in the player's folder keeps its bytes, and the list stays after the next build. A stand-in deleted from the game folder is put back on the next build. A replacement refused with PermissionError (os.replace made to refuse one file) leaves that file as linked and listed, while the other damaged file still gets its stand-in; the note names both; the next build, with the refusal gone, puts the stand-in in. A romfs that is the game folder's own, or inside it, is only reported. The harness lets nx2/unpack.py through for a chosen exefs. tests/test_damaged_check_progress.py reads the scan call, which now has a second argument, and nx2/test_nup_stand_in.py's stand-in stub takes the new refused argument (that check runs only with zstandard installed).

Measurements

Verdict: the new checks fail on main and pass here; the suite has no new failures.

Hardware: none used (game-free checks on a Linux x86-64 PC, Python 3.14, gcc 15.2)
Scene: none, the game was not run
Method: 2026-11-14. python3 -I -B tests/run_all.py, two runs each, interleaved (main, PR, main, PR). The new checks also ran against main's nx2/make_game.py (LONGSHOT_CACHE_TEST_SOURCE, in a full checkout of main). Branch: fix/romfs-folder-stand-ins.
How to reproduce: the commands above in a checkout of each commit

	Before: main @ `543846eb`	After: this PR @ `602ad901`
`tests/run_all.py` (run 1, run 2)	485 passed, 2 failed, 84 skipped (both runs)	485 passed, 2 failed, 84 skipped (both runs)
failing checks	`test_bake_portability.py`, `test_walk_key.py`	the same two
`tests/run_all.py` in a venv with zstandard (one run)	490 passed, 2 failed, 79 skipped	the same
failing checks there	`test_lift_cache.py` (no numpy in that venv), `test_walk_key.py`	the same two
`tests/test_make_game_cache.py`, folder stand-in case	fails: the game folder's file is still the damaged one	passes
`tests/test_make_game_cache.py`, own folder case	fails: nothing is said	passes

test_bake_portability.py fails here for a missing Python module (zstandard). test_walk_key.py's fixture doesn't link on main (undefined reference to hos_oot_autoload_input).

Flags and settings

None.

Testing

Tested: the game-free suite on a Linux PC, as in Measurements.

Not tested: a real incomplete dump (none here), and the game's Water Temple after such a build. Windows: neither a real read-only attribute nor the copy a failed hard link falls back to (the stand-in replaces that copy the same way).

Must be tested:

Any OS, with an incomplete (23.3 GB) dump extracted to folders: build from the folders, expecting the build to report "Your game dump is incomplete" and use stand-ins where available (by hand: look at that message and the Water Temple's walls).

Risks

What could break: A build from folders now opens every .zs file once per folder, which takes minutes where a virus scanner looks at each one, as an unpack's check does. While a file stays refused, the check is not marked done, so every build from that folder opens every .zs again (minutes) until the file can be replaced. A check that runs again (after a refusal, or a file linked in afresh) and finds damaged files lists only those, so that build's note no longer names files an earlier build already replaced. #534 adds a check of the chosen romfs in main and the same unpack.py line in the test harness; both merge cleanly together and the cache tests pass on the merge. #360 (open) copies these marks between game folders and calls link_or_copy, which now returns how many files it put in: whichever lands second needs a small rebase.

Turning it off: Reverting the commit takes the check off the folder way.

Compatibility: A damaged file with no stand-in now stops a build from folders, as it does on the .nup way. As before, a game folder built again keeps the files it already holds (link_or_copy skips names that exist), so a whole dump put at the same path later does not replace stand-ins already in the game folder.

Issues

For #464: walls missing in the Water Temple, as with a damaged texture. The reporter's log does not say how their game folder was made or whether their dump is complete, so this may not be their case; they were not asked.

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main.
python -I -B tests/run_all.py passes (any failure that also fails on main is named).
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.

**Type:** fix · **Risk:** medium · **Tested on:** Linux x86-64 (game-free checks only) · **Needs:** any OS, an incomplete dump extracted to folders ## Why An incomplete dump (about 23.3 GB instead of 23.8) has texture files whose start is zeros, and the game then leaves out every model that uses one: most of the Water Temple's walls and floors, the Great Fairy Fountain's floor (#1002). `stand_in_damaged_files` in `nx2/make_game.py` replaces them with whole files of the same kind and says so, but only in a romfs the builder unpacked from a .nup (its `.longshot_romfs` stamp) and on `--update` of such a folder. A game built from folders (`--romfs` with `--exefs`, or `--game` with a folder) has the damaged files linked in as they are, and nothing is said. The folder way was left out so the player's own folder is never written to (bd527024: "--romfs folders are not touched"). The game folder's romfs is a set of hard links to that folder, and `nup.stand_in_damaged` writes each stand-in to `<file>.part` and then replaces the name with it, so the stand-in can go in the game folder without writing the player's file. ## How to reproduce **Seen on:** found reading the code; not seen in the game **Settings and switches:** `--romfs` with `--exefs`, or `--game` with a folder; an incomplete dump of about 23.3 GB instead of 23.8. **Steps:** Build from the incomplete dump's folders, check the build's messages, then look at the Water Temple's walls and floors or the Great Fairy Fountain's floor. **Expected:** Damaged files are reported and stand-ins are used where available without writing the player's folder. **Actual:** The damaged files are linked in as they are and nothing is said; the game leaves out every model that uses one. ## What changed Builds from folders now check for damaged texture files and use stand-ins in the game folder without writing the player's files. Refused replacements are listed and tried again on the next build; a romfs that is the player's folder itself, or holds it, is only reported. - build (`nx2/make_game.py`, `nx2/nup.py`): the folder way calls `stand_in_damaged_files(out, romfs, a.romfs)` after linking. For a folder its stamp is the folder's path and file count, kept in `.longshot_romfs_checked` as for an unpack, so the roughly 100,000 files are opened once per folder and not at every build. A stand-in replaces the game folder's name of the damaged file; the player's file and its other names keep their bytes. A file that `link_or_copy` puts in afresh (one deleted from the game folder, a stand-in among them) clears that stamp, so it is looked at again; a rescan of a folder build that finds nothing keeps the list of what was replaced. `nup.stand_in_damaged` removes an existing `<file>.part` before copying, so a `.part` linked in from the player's folder is not written through. A replacement Windows refuses (a read-only file: through a hard link the attribute is the player's file's own, so it is not cleared; or a file another program has open, such as a virus scanner right after the linking) leaves that file as linked, removes its `.part`, and lists it as "not replaced, Windows refused it (read-only in your folder, or open in another program); the next build tries again"; the build goes on, as it did before, and the check is not marked done, so the next build tries that file again. The .nup way's wording and its stop for a file with no stand-in are unchanged. When the game folder's romfs is the player's folder itself, or holds it, the damaged files are listed (printed and in `longshot_damaged_files.txt`) and not replaced, and the build goes on. The .nup and `--update` ways are unchanged; the scan's progress line says "once per unpack or folder". - tests (`tests/test_make_game_cache.py`): a chosen romfs with a .zs that starts with zeros beside a whole one of the same kind, and a second hard link to the damaged file: the game folder's file is the stand-in, the player's file and its other name still hold their bytes (link count 2), the note names it, and a second build from the same folder opens no file again. A leftover `.part` in the player's folder keeps its bytes, and the list stays after the next build. A stand-in deleted from the game folder is put back on the next build. A replacement refused with PermissionError (`os.replace` made to refuse one file) leaves that file as linked and listed, while the other damaged file still gets its stand-in; the note names both; the next build, with the refusal gone, puts the stand-in in. A romfs that is the game folder's own, or inside it, is only reported. The harness lets `nx2/unpack.py` through for a chosen exefs. `tests/test_damaged_check_progress.py` reads the scan call, which now has a second argument, and `nx2/test_nup_stand_in.py`'s stand-in stub takes the new `refused` argument (that check runs only with zstandard installed). ## Measurements **Verdict:** the new checks fail on main and pass here; the suite has no new failures. Hardware: none used (game-free checks on a Linux x86-64 PC, Python 3.14, gcc 15.2) Scene: none, the game was not run Method: 2026-11-14. `python3 -I -B tests/run_all.py`, two runs each, interleaved (main, PR, main, PR). The new checks also ran against main's `nx2/make_game.py` (`LONGSHOT_CACHE_TEST_SOURCE`, in a full checkout of main). Branch: `fix/romfs-folder-stand-ins`. How to reproduce: the commands above in a checkout of each commit | | Before: main @ 543846eb | After: this PR @ 602ad901 | |---|---|---| | `tests/run_all.py` (run 1, run 2) | 485 passed, 2 failed, 84 skipped (both runs) | 485 passed, 2 failed, 84 skipped (both runs) | | failing checks | `test_bake_portability.py`, `test_walk_key.py` | the same two | | `tests/run_all.py` in a venv with zstandard (one run) | 490 passed, 2 failed, 79 skipped | the same | | failing checks there | `test_lift_cache.py` (no numpy in that venv), `test_walk_key.py` | the same two | | `tests/test_make_game_cache.py`, folder stand-in case | fails: the game folder's file is still the damaged one | passes | | `tests/test_make_game_cache.py`, own folder case | fails: nothing is said | passes | `test_bake_portability.py` fails here for a missing Python module (zstandard). `test_walk_key.py`'s fixture doesn't link on main (`undefined reference to hos_oot_autoload_input`). ### Flags and settings None. ## Testing **Tested:** the game-free suite on a Linux PC, as in Measurements. **Not tested:** a real incomplete dump (none here), and the game's Water Temple after such a build. Windows: neither a real read-only attribute nor the copy a failed hard link falls back to (the stand-in replaces that copy the same way). **Must be tested:** - Any OS, with an incomplete (23.3 GB) dump extracted to folders: build from the folders, expecting the build to report "Your game dump is incomplete" and use stand-ins where available (by hand: look at that message and the Water Temple's walls). ## Risks **What could break:** A build from folders now opens every .zs file once per folder, which takes minutes where a virus scanner looks at each one, as an unpack's check does. While a file stays refused, the check is not marked done, so every build from that folder opens every .zs again (minutes) until the file can be replaced. A check that runs again (after a refusal, or a file linked in afresh) and finds damaged files lists only those, so that build's note no longer names files an earlier build already replaced. #534 adds a check of the chosen romfs in `main` and the same `unpack.py` line in the test harness; both merge cleanly together and the cache tests pass on the merge. #360 (open) copies these marks between game folders and calls `link_or_copy`, which now returns how many files it put in: whichever lands second needs a small rebase. **Turning it off:** Reverting the commit takes the check off the folder way. **Compatibility:** A damaged file with no stand-in now stops a build from folders, as it does on the .nup way. As before, a game folder built again keeps the files it already holds (`link_or_copy` skips names that exist), so a whole dump put at the same path later does not replace stand-ins already in the game folder. ## Issues For #464: walls missing in the Water Temple, as with a damaged texture. The reporter's log does not say how their game folder was made or whether their dump is complete, so this may not be their case; they were not asked. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.

jarynclouatre added 1 commit

2026-11-14 19:03:35 +00:00

fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does

beanstalk/approval awaiting maintainer approval

Details

PR description / Description follows the template (pull_request_target) Successful in 17s

Details

CI / Conventional Commit title (pull_request) Has been cancelled

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled

Details

CI / Workflow lint (actionlint) (pull_request) Has been cancelled

Details

Third-party parts / As their notes say (pull_request) Has been cancelled

Details

Third-party parts / The same as upstream (pull_request) Has been cancelled

Details

359b340f1f

An incomplete dump has textures whose start is zeros, and the game then
leaves out every model that uses one: most of the Water Temple's walls
and floors, the Great Fairy Fountain's floor (#1002). stand_in_damaged_files
replaced them only in a romfs the builder unpacked from a .nup; a game
built from --romfs with --exefs, or --game with folders, linked them in
as they were.

The folder way now runs the same check after linking. A stand-in is
written as a new file over the game folder's name of the damaged one
(nup.stand_in_damaged's .part and replace), so the player's own file
and its other names are not written. When the game folder's romfs is
the player's folder itself, the damaged files are only listed. The
check runs once per folder: .longshot_romfs_checked holds the folder's
path and file count.

nup.stand_in_damaged removes an existing <file>.part before copying, so
a .part linked in from the player's folder is not written through. A
folder build that links a file in afresh (one deleted from the game
folder, a stand-in among them) clears the checked stamp, so it is
looked at again.
A folder inside the game folder's romfs counts as the player's own too,
and a rescan of a folder build that finds nothing keeps the list of
what was replaced (the stand-ins are still there).
A replacement Windows refuses (a read-only file: through a hard link
the attribute is the player's file's own, so it is not cleared) leaves
that file as linked, listed as not replaced, and the build goes on.

jarynclouatre changed title from ~~WIP: fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does~~ to fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does

2026-11-14 19:06:29 +00:00

jarynclouatre force-pushed fix/romfs-folder-stand-ins from 359b340f1f

beanstalk/approval awaiting maintainer approval

Details

PR description / Description follows the template (pull_request_target) Successful in 17s

Details

CI / Conventional Commit title (pull_request) Has been cancelled

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled

Details

CI / Workflow lint (actionlint) (pull_request) Has been cancelled

Details

Third-party parts / As their notes say (pull_request) Has been cancelled

Details

Third-party parts / The same as upstream (pull_request) Has been cancelled

Details

to 602ad901c0

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 5s

Details

beanstalk/approval Test farm not requested

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

2026-11-14 19:06:31 +00:00

Compare

Chloe_The_Red commented

2026-11-14 21:47:20 +00:00

First-time contributor

Author of #464 here. I did use romfs+exefs to build the game, dumped folders total 24.4 GB, original nup file totaled 23.3 GB. Will attempt a complete dump and report back.

Chloe_The_Red commented

2026-11-14 23:20:13 +00:00

First-time contributor

My issue was fixed by using a complete dump.

llun referenced this pull request

2026-11-15 03:23:33 +00:00

fix(builder): a game dump Windows refuses to read says why and what to do, not "Permission denied" #641

mooki added the

labels

2026-11-15 13:55:26 +00:00

HumanBean reviewed

2026-11-15 19:03:57 +00:00

HumanBean left a comment

Code review only (no tests run, no game run). The approach holds up: os.replace of <file>.part swaps the game folder's directory entry, so the player's file and its other hard links keep their bytes; the .part pre-removal avoids writing through a linked leftover; own (same folder, or romfs holding the source) is only reported; a refused replace is listed and the stamp is not written, so it is retried.

Things I'd like answered before merge, none a clear bug:

Behaviour change: a folder build with a damaged file that has no stand-in now stops (fail(...)), where it used to build a game that left those models out. The PR says so under Compatibility; is that intended for the folder way, given the player can't re-extract as easily as re-picking a .nup? An error saying what to do is fine, I just want the decision to be explicit.
known (the .longshot_heads list) is read from out for the folder way too. If out holds a heads file left by an earlier .nup extraction, the folder check would use that list instead of opening files. Please confirm it is removed or cannot be stale there (the end of the function removes it, but only when the function gets that far, and fail(...) returns earlier).
For a folder, the stamp is the folder path plus file count. A same-count change (one damaged file swapped for another dump's) won't re-run the check; fine for the stated purpose, maybe worth a line in the docs.

The PR is not run on a real incomplete dump and lists that under Must be tested, so it should stay open for a person or tester with one. Commenting rather than approving for that reason and the questions above.

Code review only (no tests run, no game run). The approach holds up: `os.replace` of `<file>.part` swaps the game folder's directory entry, so the player's file and its other hard links keep their bytes; the `.part` pre-removal avoids writing through a linked leftover; `own` (same folder, or romfs holding the source) is only reported; a refused replace is listed and the stamp is not written, so it is retried. Things I'd like answered before merge, none a clear bug: 1. Behaviour change: a folder build with a damaged file that has no stand-in now stops (`fail(...)`), where it used to build a game that left those models out. The PR says so under Compatibility; is that intended for the folder way, given the player can't re-extract as easily as re-picking a .nup? An error saying what to do is fine, I just want the decision to be explicit. 2. `known` (the `.longshot_heads` list) is read from `out` for the folder way too. If `out` holds a heads file left by an earlier .nup extraction, the folder check would use that list instead of opening files. Please confirm it is removed or cannot be stale there (the end of the function removes it, but only when the function gets that far, and `fail(...)` returns earlier). 3. For a folder, the stamp is the folder path plus file count. A same-count change (one damaged file swapped for another dump's) won't re-run the check; fine for the stated purpose, maybe worth a line in the docs. The PR is not run on a real incomplete dump and lists that under Must be tested, so it should stay open for a person or tester with one. Commenting rather than approving for that reason and the questions above.