WIP: feat(builder): the two known damaged textures replaced by their pairs, sizes and names checked (LONGSHOT_STAND_INS=pairs, off) #1983

Draft
siantm wants to merge 5 commits from siantm/Longshot:feat/builder-texture-pairs into main
Contributor

Summary

An incomplete dump (about 23.3 GB instead of 23.8 GB) has damaged textures. Today the builder copies in a whole file
of the same kind with the nearest name and size (nup.stand_in_damaged, #1453). That file keeps its own embedded
texture name, and nothing checks its size against what the game allocates for the damaged one.

This adds a second mode beside it, off unless set: with LONGSHOT_STAND_INS=pairs, only the two textures known to be
damaged in those dumps are repaired, each from its pair (nx2/texture_repair.py):

  • both must have the same allocation in the game's resource size table (read through the dictionary pack), and the
    donor's payload must fill it;
  • the donor's layout is checked (a known block format, each mip level's blocks inside the image data, which overlaps
    no metadata), and its two name records, the BRTI's and the file name, are resolved and must hold its name; only those
    are rewritten to the damaged texture's name, and the result is recompressed and checked;
  • all or nothing: every replacement is made and checked in memory first; when any damaged file stays unresolved none
    is written, the build stops and says for each file why (no known pair, its pair damaged, unreadable or not fitting).
    Each is written through a temporary file of its own beside the target; a write that stops part-way stops the build,
    and the next build completes the set;
  • pairs mode covers exactly the two known textures: a dump with another damaged one cannot be built in this mode yet
    (the 23.3 GB dump also has a third, in Ganon's stage, with no known pair; the build stops at it and says to build
    without the switch);
  • the unpack's list of its damaged files (.longshot_romfs_heads, under the unpack's stamp) is its damage record: it
    is kept after the check in both modes, and in pairs mode written before anything is replaced, with the files a
    default-mode check of the same unpack replaced (its longshot_damaged_files.txt) and those the probe of the two known
    names finds added. A recorded file is replaced whatever its bytes now say; the probe uses the game's dictionary, and
    a healthy original is never replaced. A change of mode checks the recorded files again (.longshot_romfs_checked's
    version line) without opening every file; a new unpack removes the old check mark and note only once its package was
    read and checked, so a failed read keeps them with the old romfs.

Unset, the builder does what it did, except that it keeps .longshot_romfs_heads after its check instead of deleting
it (so a change of mode reads it); nup.stand_in_damaged is untouched, and its tests clear the switch.
nx2/nup_repair_fixture.py makes authored zstd frames, single-texture BNTX files, a dictionary SARC and a RESTBL: no
game bytes. texture_repair.py is adapted from ProjectReLink's tools/dump_repair.py (GPL-3.0, 90801cdf).

From the local branch exp/x86-asm-sync 5259070a, which replaced the similar-file search outright; here it sits beside
it behind the switch. Not in this PR, from the same commit: the float helpers (another PR), the hot pieces' compile
safeguard and scheduling cost (not ported).

Read first: repair_texture_pairs in nx2/nup.py; stand_in_damaged_files in nx2/make_game.py;
nx2/texture_repair.py; nx2/test_nup_texture_pairs.py (new).

Commits: the feature (9a94469c); review fixes as bug fixes, each failing game-free tests then the fix: 4c17ee20 and
eefcb329 (first review), df7d58e4 and 9fdcce1d (second review: the probe's finds recorded, the old marks kept until a new
unpack takes over).

Testing

Tested: Windows 11, i7-9700, RTX 4060, Python 3.14.7 (the Builder's); game-free checks only. No build from a dump
with the two textures damaged has run with this change on this PC yet, with the switch set or unset, and no game start
of such a build (Still to test). Each check ran in a hidden console of its own, in one job of the PC's job queue
(fb2-1983-3), on this branch rebased onto main @ 9dd41326 (head 9fdcce1d):

  • At the test commit df7d58e4 (the first review's fix with the second review's tests):
    python -I -B nx2/test_nup_texture_pairs.py exit 1, "Ran 19 tests", FAILED (failures=1: the no-note migration case,
    whose list missed the probe's find); python -I -B tests/test_make_game_cache.py exit 1, "Ran 49 tests", FAILED
    (failures=1, skipped=2: the old check mark and note gone after a failed new unpack).
  • At 9fdcce1d: nx2/test_nup_texture_pairs.py exit 0, "Ran 19 tests in 1.115s", OK; tests/test_make_game_cache.py
    exit 0, "Ran 49 tests in 2.549s", OK (skipped=2).
  • nx2/test_nup_stand_in.py exit 0, "stand-ins: ok"; nx2/test_nup_zstd_heads.py exit 0, "Ran 8 tests", OK;
    nx2/test_nup_verify.py exit 0, PASS; tests/test_damaged_check_progress.py exit 0, "Ran 4 tests", OK;
    tests/test_damaged_stand_in_fail_fast.py exit 0, "Ran 4 tests", OK; python -I -m py_compile of the four changed
    modules: exit 0.
  • Before this round, at a1990170 (fb2-1983-2), the first review's tests failed at e7951cff (failures=9, errors=7) and
    passed at a1990170. tests/test_make_game_cache.py was not run then: it still expected the list deleted after the
    check, which a1990170 changed; df7d58e4 updates that expectation.

Earlier, at b51ee688 on main @ 2e1f6c32 (batch 9), tests/run_all.py --strict --changed origin/main had 67 passed,
14 failed, 2 skipped, the 14 named as failing on main too (one of them, test_pipeline_states.py, had found no Vulkan headers in that run (RX_CPM unset) and compiled nothing on either side; run again with the headers it passes at a1990170 and on main @ f1f2dc5b); it was not run again on a1990170 or 9fdcce1d.

Still to test:

  • Windows 11, a PC with a .nup dump with the known textures damaged: fresh builds into separate game folders with
    the switch unset and with LONGSHOT_STAND_INS=pairs, expecting the pairs log to give each pair's decoded size and
    allocation (both real donors fitting the 280-byte allocation overhead) or, with the third texture damaged, the stop
    naming it; a game start of a pairs build with the Great Fairy Fountain's floor and the Water Temple's gate drawn; the
    check's one-time time on a kept folder after a change of mode (by hand)
  • Whether a pair with an equal allocation exists for the third damaged texture (by hand, on that PC)
  • python -I -B tests/run_all.py --strict --changed origin/main on 9fdcce1d, expecting no failure that main does not have (by hand)
  • Linux: python3 -I -B nx2/test_nup_texture_pairs.py, expecting it to pass (by hand)

Measurements

Nothing to measure: a builder mode that ships off and runs only on a damaged dump (the scan is the existing one; the repair reads two textures and the size table).

Risks

Off (unset), nothing changes but a small list kept in the game folder. Set, a damaged file other than the two known
textures stops the build, with no file replaced, where the default mode would have copied in a similar file: the player
is told why and to build without the switch or dump the game again. A romfs checked before in the default mode is
checked once more when the mode is set, and once more in the default mode when it is unset again (similar files go back
in place of the pairs' textures). A malformed size table, dictionary or donor leaves every file as it was and the build
stops (29 such inputs are tested). Players notice nothing unless the switch is set; a revert is enough.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A check under tests/ that reads changed source is updated, not weakened.
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary An incomplete dump (about 23.3 GB instead of 23.8 GB) has damaged textures. Today the builder copies in a whole file of the same kind with the nearest name and size (`nup.stand_in_damaged`, #1453). That file keeps its own embedded texture name, and nothing checks its size against what the game allocates for the damaged one. This adds a second mode beside it, off unless set: with `LONGSHOT_STAND_INS=pairs`, only the two textures known to be damaged in those dumps are repaired, each from its pair (`nx2/texture_repair.py`): - both must have the same allocation in the game's resource size table (read through the dictionary pack), and the donor's payload must fill it; - the donor's layout is checked (a known block format, each mip level's blocks inside the image data, which overlaps no metadata), and its two name records, the BRTI's and the file name, are resolved and must hold its name; only those are rewritten to the damaged texture's name, and the result is recompressed and checked; - all or nothing: every replacement is made and checked in memory first; when any damaged file stays unresolved none is written, the build stops and says for each file why (no known pair, its pair damaged, unreadable or not fitting). Each is written through a temporary file of its own beside the target; a write that stops part-way stops the build, and the next build completes the set; - pairs mode covers exactly the two known textures: a dump with another damaged one cannot be built in this mode yet (the 23.3 GB dump also has a third, in Ganon's stage, with no known pair; the build stops at it and says to build without the switch); - the unpack's list of its damaged files (`.longshot_romfs_heads`, under the unpack's stamp) is its damage record: it is kept after the check in both modes, and in pairs mode written before anything is replaced, with the files a default-mode check of the same unpack replaced (its `longshot_damaged_files.txt`) and those the probe of the two known names finds added. A recorded file is replaced whatever its bytes now say; the probe uses the game's dictionary, and a healthy original is never replaced. A change of mode checks the recorded files again (`.longshot_romfs_checked`'s version line) without opening every file; a new unpack removes the old check mark and note only once its package was read and checked, so a failed read keeps them with the old romfs. Unset, the builder does what it did, except that it keeps `.longshot_romfs_heads` after its check instead of deleting it (so a change of mode reads it); `nup.stand_in_damaged` is untouched, and its tests clear the switch. `nx2/nup_repair_fixture.py` makes authored zstd frames, single-texture BNTX files, a dictionary SARC and a RESTBL: no game bytes. `texture_repair.py` is adapted from ProjectReLink's `tools/dump_repair.py` (GPL-3.0, 90801cdf). From the local branch `exp/x86-asm-sync` 5259070a, which replaced the similar-file search outright; here it sits beside it behind the switch. Not in this PR, from the same commit: the float helpers (another PR), the hot pieces' compile safeguard and scheduling cost (not ported). Read first: `repair_texture_pairs` in `nx2/nup.py`; `stand_in_damaged_files` in `nx2/make_game.py`; `nx2/texture_repair.py`; `nx2/test_nup_texture_pairs.py` (new). Commits: the feature (9a94469c); review fixes as bug fixes, each failing game-free tests then the fix: 4c17ee20 and eefcb329 (first review), df7d58e4 and 9fdcce1d (second review: the probe's finds recorded, the old marks kept until a new unpack takes over). ## Testing **Tested:** Windows 11, i7-9700, RTX 4060, Python 3.14.7 (the Builder's); game-free checks only. No build from a dump with the two textures damaged has run with this change on this PC yet, with the switch set or unset, and no game start of such a build (Still to test). Each check ran in a hidden console of its own, in one job of the PC's job queue (fb2-1983-3), on this branch rebased onto `main` @ 9dd41326 (head 9fdcce1d): - At the test commit df7d58e4 (the first review's fix with the second review's tests): `python -I -B nx2/test_nup_texture_pairs.py` exit 1, "Ran 19 tests", FAILED (failures=1: the no-note migration case, whose list missed the probe's find); `python -I -B tests/test_make_game_cache.py` exit 1, "Ran 49 tests", FAILED (failures=1, skipped=2: the old check mark and note gone after a failed new unpack). - At 9fdcce1d: `nx2/test_nup_texture_pairs.py` exit 0, "Ran 19 tests in 1.115s", OK; `tests/test_make_game_cache.py` exit 0, "Ran 49 tests in 2.549s", OK (skipped=2). - `nx2/test_nup_stand_in.py` exit 0, "stand-ins: ok"; `nx2/test_nup_zstd_heads.py` exit 0, "Ran 8 tests", OK; `nx2/test_nup_verify.py` exit 0, PASS; `tests/test_damaged_check_progress.py` exit 0, "Ran 4 tests", OK; `tests/test_damaged_stand_in_fail_fast.py` exit 0, "Ran 4 tests", OK; `python -I -m py_compile` of the four changed modules: exit 0. - Before this round, at a1990170 (fb2-1983-2), the first review's tests failed at e7951cff (failures=9, errors=7) and passed at a1990170. `tests/test_make_game_cache.py` was not run then: it still expected the list deleted after the check, which a1990170 changed; df7d58e4 updates that expectation. Earlier, at b51ee688 on `main` @ 2e1f6c32 (batch 9), `tests/run_all.py --strict --changed origin/main` had 67 passed, 14 failed, 2 skipped, the 14 named as failing on `main` too (one of them, `test_pipeline_states.py`, had found no Vulkan headers in that run (`RX_CPM` unset) and compiled nothing on either side; run again with the headers it passes at a1990170 and on `main` @ f1f2dc5b); it was not run again on a1990170 or 9fdcce1d. **Still to test:** - Windows 11, a PC with a `.nup` dump with the known textures damaged: fresh builds into separate game folders with the switch unset and with `LONGSHOT_STAND_INS=pairs`, expecting the pairs log to give each pair's decoded size and allocation (both real donors fitting the 280-byte allocation overhead) or, with the third texture damaged, the stop naming it; a game start of a pairs build with the Great Fairy Fountain's floor and the Water Temple's gate drawn; the check's one-time time on a kept folder after a change of mode (by hand) - Whether a pair with an equal allocation exists for the third damaged texture (by hand, on that PC) - `python -I -B tests/run_all.py --strict --changed origin/main` on 9fdcce1d, expecting no failure that `main` does not have (by hand) - Linux: `python3 -I -B nx2/test_nup_texture_pairs.py`, expecting it to pass (by hand) ## Measurements Nothing to measure: a builder mode that ships off and runs only on a damaged dump (the scan is the existing one; the repair reads two textures and the size table). ## Risks Off (unset), nothing changes but a small list kept in the game folder. Set, a damaged file other than the two known textures stops the build, with no file replaced, where the default mode would have copied in a similar file: the player is told why and to build without the switch or dump the game again. A romfs checked before in the default mode is checked once more when the mode is set, and once more in the default mode when it is unset again (similar files go back in place of the pairs' textures). A malformed size table, dictionary or donor leaves every file as it was and the build stops (29 such inputs are tested). Players notice nothing unless the switch is set; a revert is enough. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. - [ ] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
feat(builder): the two known damaged textures replaced by their pairs, sizes and names checked (LONGSHOT_STAND_INS=pairs, off)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 5s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
0a83513e24
An incomplete dump (23.3 GB) has damaged textures; the builder copies in a whole file of the same kind with the
nearest name (nup.stand_in_damaged). That file keeps its own embedded texture name, and its size is not checked
against the game's resource size table. LONGSHOT_STAND_INS=pairs repairs only the two textures known to be damaged in
those dumps, each from its pair: both must have the same allocation in the RESTBL (read through the dictionary pack),
the donor's payload must fill it, and its BNTX and BRTI name records are rewritten to the damaged texture's before it
is recompressed and written atomically. Any other damaged file is left unresolved, so the build stops (as main does
when no similar file is found). In that mode the check runs again over a romfs an earlier check passed, taking the
similar files it copied in from its note, and never replaces a healthy original. Off unless set: unset, the builder
does what it did, and nup.stand_in_damaged is untouched.

nx2/texture_repair.py (adapted from ProjectReLink tools/dump_repair.py, GPL-3.0, 90801cdf), nup.repair_texture_pairs,
make_game.stand_in_damaged_files's mode, and nx2/nup_repair_fixture.py, authored files with no game bytes.
nx2/test_nup_texture_pairs.py: the repair, 22 malformed inputs that write nothing, no similar file in this mode, the
mode off unless set, and the builder's check (earlier similar files, healthy originals, unresolved retries).
tests/test_damaged_check_progress.py reads the check's call through the mode's choice now (nup.stand_in_damaged when
unset).

Ported from the local branch exp/x86-asm-sync 5259070a, which replaced the similar-file search outright; here it sits
beside main's, behind the switch.

Assisted-by: Claude Code
Member

Possibly related, for context:

  • PR #546 WIP: fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does — Open builder stand-in handling for damaged folder-based romfs; overlaps the builder repair path.
  • issue #1939 Game: Kokiri Forest and the market run ~30% slower when the game files weren't unpacked by Longshot — Open report on damaged textures and the effects this repair addresses.
  • PR #1943 WIP: fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's — Open host-side fallback for the same damaged textures; complementary runtime fix.
Possibly related, for context: - PR #546 WIP: fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does — Open builder stand-in handling for damaged folder-based romfs; overlaps the builder repair path. - issue #1939 Game: Kokiri Forest and the market run ~30% slower when the game files weren't unpacked by Longshot — Open report on damaged textures and the effects this repair addresses. - PR #1943 WIP: fix(host): an incomplete dump's damaged textures are read as their stand-ins in any romfs, not only the builder's — Open host-side fallback for the same damaged textures; complementary runtime fix.
BeanBot left a comment

Recommendation: Request changes. The documented 23.3 GB dump has a third damaged texture, so pairs mode stops a build the default mode can complete. The repair also accepts some damaged files as healthy and can lose track of older substitutions across builds.

Adds an opt-in builder mode intended to preserve the allocation and embedded names of two replacement textures. It keeps the existing stand-in search as the default for fresh builds. The narrower repair is useful if it works on the affected dump, but it adds a binary parser, persistent migration state, and a potentially minutes-long rescan of an existing install.

@fix will do (16)

  1. L1 (its part an edit can do) nx2/texture_repair.py:16: Add a reviewed repair for the third texture if one exists, or scope the mode and its documentation to a verified two-damage variant.
  2. L2 nx2/make_game.py:726: A fresh extraction replaces the romfs and clears the checked marker but leaves the old damage note in place. (more on the line)
  3. L3 nx2/make_game.py:753: The human-readable note is the only record of old substitutions after HEADS is deleted. (more on the line)
  4. L4 nx2/nup.py:863: A matching BRTI name overrides the extraction's damage verdict. (more on the line)
  5. L5 nx2/nup.py:865: A failed probe is treated as permission to replace a known target. (more on the line)
  6. L6 nx2/texture_repair.py:156: The BNTX filename field only adds an address to an allowlist. (more on the line)
  7. L7 nx2/texture_repair.py:127: The payload check accepts any nonzero dimensions and image size if the declared image range lies inside the file. (more on the line)
  8. L8 nx2/nup.py:879: The fixed .part file is shared by concurrent builders targeting the same output with different --work folders. (more on the line)
  9. L9 nx2/texture_repair.py:84: The 256 MiB decompression limit admits over 33 million hash entries. (more on the line)
  10. L10 (minor) nx2/nup.py:882: The donor comes back under the canonical extraction path, but the displayed donor path is rebased against the lexical root after replacement. (more on the line)
  11. L11 (minor) nx2/nup.py:876: The repair discards every expected error and tells the player to redump. (more on the line)
  12. L12 (minor) nx2/test_nup_texture_pairs.py:67: The positive test constructs its expected output with the production rename function, and the fixture gets its donor choices from the production map. (more on the line)
  13. L13 (minor) nx2/test_nup_stand_in.py:99: Existing builder tests now inherit the new switch. With LONGSHOT_STAND_INS=pairs exported, this test's fake nup module has no repair_texture_pairs, and the HEADS tests also enter the wrong mode. Explicitly clear the switch in tests that exercise the default, including the HEADS, fail-fast, and cache cases.
  14. L14 (minor) docs/dev/reference/build.md:54: Unsetting the switch changes the next search, but it cannot undo a published pair repair. (more on the line)
  15. L15 (its part an edit can do) nx2/make_game.py:744: Retain the unpack-scoped damage and repair record so a mode transition can recheck affected paths without reopening the whole romfs.
  16. L16 (its part an edit can do): Log the allocation and decoded size for each pair, along with the repair verdict and elapsed read/repair time.

For the author (3) (a run, a measurement or a decision)

  • L1 nx2/texture_repair.py:16: The documented 23.3 GB dump contains a third damaged texture in Ganon's Stage, recorded in the original stand-in commit bd5270246 and the legacy frame fix #2391. (more on the line)
  • L15 (minor) nx2/make_game.py:744: Changing modes on an already checked output whose HEADS was deleted causes a full walk and about 100,000 file opens. (more on the line)
  • L16 (minor): The authored fixture assumes the donors' compression and the exact 280-byte allocation overhead. Establish that both real donors satisfy those assumptions and that the resulting game renders both affected locations. Compare base and head with separate fresh outputs, record both switch modes on Windows and Linux, and cover affected Tier 1 configurations. Report build outcome and time in Measurements.

Checks: Still running: CI / Conventional Commit title (pull_request), CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request), CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request), CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request), CI / Workflow lint (actionlint) (pull_request), CI / Website build and checks (pull_request), Guide screenshots / Guide images and local UI source review (pull_request), Third-party parts / As their notes say (pull_request), Third-party parts / The same as upstream (pull_request), Website / Website on Cloudflare (pull_request).

More: does it help, other systems, switches, what was read

Does it help: Unproven. The author reports 10 game-free tests passing in 0.805 s on Windows at b51ee688. The base comparison stopped at a missing fixture import. There is no build or game run from the affected dump in either mode, and no Linux result or build-time comparison.

Other systems: Fresh extraction, cached updates, and --romfs-from builds can all reach this path. Both the Windows Builder package and Linux AppImage carry the module and already have zstandard. The output is shared across GPU vendors and Steam Deck. In-game rendering and the interaction with mods that replace textures or resource-size entries remain unverified. There are no changes to the renderer or host ABI, saves, audio, or input.

Switches: LONGSHOT_STAND_INS=pairs selects pair repair. Unset and all other values select the existing similar-file search. Keeping that default avoids imposing the experimental build refusal on existing players. The build-switch table has a row. The runtime switch table excludes build-only settings. The switch is one-way for files already rewritten, which prevents a clean A/B comparison in the same output folder.

Reviewed: 0a83513e24 against 6e1521a70a: Read the complete seven-file diff, commit message, new repair module, fixture and tests, updated progress test, and build reference. Read the changed functions in make_game.py and nup.py and traced their extraction, update, hard-link, lock, path, and stamp code. Read AGENTS.md, CONTRIBUTING.md, the PR template and developer standards. The checkout has no docs/review-checklist.md or adjacent Tooling checkout. Reviewed the supplied run summaries and CI status and consulted the existing BNTX, SARC, RESTBL and packaging code. No translations or generated files changed.; nothing run (a code review).

Checked and fine:

  • The branch has one topic and an Assisted-by trailer. The WIP prefix follows the repository workflow.
  • The fixtures construct their bytes in Python. No game files, build products, generated headers, or third-party edits are present. The adapted module names its source commit and GPL-3.0 license.
  • The switch is documented with the build-only settings, and the new test declares the existing zstandard dependency for discovery.
  • Both repair calls in the normal build flow and the call after background extraction execute after their relevant reading work has joined. No new thread-local allocations are introduced.
  • The texture writer uses replacement instead of truncating the final file, preserving other hard links to that texture when a repair succeeds.
  • No new network calls, subprocesses, installer actions, or renderer API changes.

Testing: Read-only review. No code was executed. The author reports Windows passes for the 10 pair tests, the default stand-in test, and four progress tests on the earlier rebase. The changed suite reports 67 passed, 14 failed on both revisions, and two skipped. Head CI is pending. Full lint and test results, Linux execution, and builds and game checks from the affected dump are outstanding.

Overlaps and merge order: No overlapping open pull request is identified in the supplied context or checkout. The prior stand-in work and legacy frame fix #2391 are already on the base.


Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The documented 23.3 GB dump has a third damaged texture, so pairs mode stops a build the default mode can complete. The repair also accepts some damaged files as healthy and can lose track of older substitutions across builds. Adds an opt-in builder mode intended to preserve the allocation and embedded names of two replacement textures. It keeps the existing stand-in search as the default for fresh builds. The narrower repair is useful if it works on the affected dump, but it adds a binary parser, persistent migration state, and a potentially minutes-long rescan of an existing install. **`@fix` will do (16)** 1. **L1** (its part an edit can do) `nx2/texture_repair.py:16`: Add a reviewed repair for the third texture if one exists, or scope the mode and its documentation to a verified two-damage variant. 2. **L2** `nx2/make_game.py:726`: A fresh extraction replaces the romfs and clears the checked marker but leaves the old damage note in place. (more on the line) 3. **L3** `nx2/make_game.py:753`: The human-readable note is the only record of old substitutions after HEADS is deleted. (more on the line) 4. **L4** `nx2/nup.py:863`: A matching BRTI name overrides the extraction's damage verdict. (more on the line) 5. **L5** `nx2/nup.py:865`: A failed probe is treated as permission to replace a known target. (more on the line) 6. **L6** `nx2/texture_repair.py:156`: The BNTX filename field only adds an address to an allowlist. (more on the line) 7. **L7** `nx2/texture_repair.py:127`: The payload check accepts any nonzero dimensions and image size if the declared image range lies inside the file. (more on the line) 8. **L8** `nx2/nup.py:879`: The fixed .part file is shared by concurrent builders targeting the same output with different --work folders. (more on the line) 9. **L9** `nx2/texture_repair.py:84`: The 256 MiB decompression limit admits over 33 million hash entries. (more on the line) 10. **L10** (minor) `nx2/nup.py:882`: The donor comes back under the canonical extraction path, but the displayed donor path is rebased against the lexical root after replacement. (more on the line) 11. **L11** (minor) `nx2/nup.py:876`: The repair discards every expected error and tells the player to redump. (more on the line) 12. **L12** (minor) `nx2/test_nup_texture_pairs.py:67`: The positive test constructs its expected output with the production rename function, and the fixture gets its donor choices from the production map. (more on the line) 13. **L13** (minor) `nx2/test_nup_stand_in.py:99`: Existing builder tests now inherit the new switch. With LONGSHOT_STAND_INS=pairs exported, this test's fake nup module has no repair_texture_pairs, and the HEADS tests also enter the wrong mode. Explicitly clear the switch in tests that exercise the default, including the HEADS, fail-fast, and cache cases. 14. **L14** (minor) `docs/dev/reference/build.md:54`: Unsetting the switch changes the next search, but it cannot undo a published pair repair. (more on the line) 15. **L15** (its part an edit can do) `nx2/make_game.py:744`: Retain the unpack-scoped damage and repair record so a mode transition can recheck affected paths without reopening the whole romfs. 16. **L16** (its part an edit can do): Log the allocation and decoded size for each pair, along with the repair verdict and elapsed read/repair time. **For the author (3)** (a run, a measurement or a decision) - **L1** `nx2/texture_repair.py:16`: The documented 23.3 GB dump contains a third damaged texture in Ganon's Stage, recorded in the original stand-in commit bd5270246 and the legacy frame fix #2391. (more on the line) - **L15** (minor) `nx2/make_game.py:744`: Changing modes on an already checked output whose HEADS was deleted causes a full walk and about 100,000 file opens. (more on the line) - **L16** (minor): The authored fixture assumes the donors' compression and the exact 280-byte allocation overhead. Establish that both real donors satisfy those assumptions and that the resulting game renders both affected locations. Compare base and head with separate fresh outputs, record both switch modes on Windows and Linux, and cover affected Tier 1 configurations. Report build outcome and time in Measurements. **Checks:** Still running: `CI / Conventional Commit title (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request)`, `CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request)`, `CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request)`, `CI / Workflow lint (actionlint) (pull_request)`, `CI / Website build and checks (pull_request)`, `Guide screenshots / Guide images and local UI source review (pull_request)`, `Third-party parts / As their notes say (pull_request)`, `Third-party parts / The same as upstream (pull_request)`, `Website / Website on Cloudflare (pull_request)`. <details> <summary>More: does it help, other systems, switches, what was read</summary> **Does it help:** Unproven. The author reports 10 game-free tests passing in 0.805 s on Windows at b51ee688. The base comparison stopped at a missing fixture import. There is no build or game run from the affected dump in either mode, and no Linux result or build-time comparison. **Other systems:** Fresh extraction, cached updates, and --romfs-from builds can all reach this path. Both the Windows Builder package and Linux AppImage carry the module and already have zstandard. The output is shared across GPU vendors and Steam Deck. In-game rendering and the interaction with mods that replace textures or resource-size entries remain unverified. There are no changes to the renderer or host ABI, saves, audio, or input. **Switches:** LONGSHOT_STAND_INS=pairs selects pair repair. Unset and all other values select the existing similar-file search. Keeping that default avoids imposing the experimental build refusal on existing players. The build-switch table has a row. The runtime switch table excludes build-only settings. The switch is one-way for files already rewritten, which prevents a clean A/B comparison in the same output folder. **Reviewed:** 0a83513e240b against 6e1521a70a32: Read the complete seven-file diff, commit message, new repair module, fixture and tests, updated progress test, and build reference. Read the changed functions in make_game.py and nup.py and traced their extraction, update, hard-link, lock, path, and stamp code. Read AGENTS.md, CONTRIBUTING.md, the PR template and developer standards. The checkout has no docs/review-checklist.md or adjacent Tooling checkout. Reviewed the supplied run summaries and CI status and consulted the existing BNTX, SARC, RESTBL and packaging code. No translations or generated files changed.; nothing run (a code review). **Checked and fine:** - The branch has one topic and an Assisted-by trailer. The WIP prefix follows the repository workflow. - The fixtures construct their bytes in Python. No game files, build products, generated headers, or third-party edits are present. The adapted module names its source commit and GPL-3.0 license. - The switch is documented with the build-only settings, and the new test declares the existing zstandard dependency for discovery. - Both repair calls in the normal build flow and the call after background extraction execute after their relevant reading work has joined. No new thread-local allocations are introduced. - The texture writer uses replacement instead of truncating the final file, preserving other hard links to that texture when a repair succeeds. - No new network calls, subprocesses, installer actions, or renderer API changes. **Testing:** Read-only review. No code was executed. The author reports Windows passes for the 10 pair tests, the default stand-in test, and four progress tests on the earlier rebase. The changed suite reports 67 passed, 14 failed on both revisions, and two skipped. Head CI is pending. Full lint and test results, Linux execution, and builds and game checks from the affected dump are outstanding. **Overlaps and merge order:** No overlapping open pull request is identified in the supplied context or checkout. The prior stand-in work and legacy frame fix #2391 are already on the base. </details> --- <sub>Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -51,6 +51,7 @@ Read by the build, not the game, so they are not in `switches.def`.
| `LONGSHOT_LIFT_STORE=<dir>` | Translations shared by every work folder with the same lift key |
| `LONGSHOT_NO_PREPARE=1` | Skip the shader preparation |
| `LONGSHOT_ROMFS_IN_PLACE=1` (or `--romfs-in-place`) | With `--romfs` and `--exefs`: read the romfs where it is; the game folder has no `romfs`, so start the game with `--romfs <folder>`. Passed over by a `.nup` or `--update` build (the switch with them is refused). `dev.py packs` sets it for packs |
| `LONGSHOT_STAND_INS=pairs` | An incomplete dump's damaged textures (a `.nup` the builder unpacked): only the two textures known to be damaged are replaced, each by its pair's texture after the game's resource size table shows both the same size, with the damaged one's name written into it (`nx2/texture_repair.py`); any other damaged file stops the build. Unset: a similar file of the same kind is copied in, as before. Off unless set: an experiment, not yet run in the game. A change of mode checks the game files again |
Member

L14 (minor): Unsetting the switch changes the next search, but it cannot undo a published pair repair. The default scan sees the new zstd frame as healthy. Document that a fresh extraction or separate output is needed for a clean comparison or rollback, and describe the mode's scope once the third known damaged texture is accounted for.

**L14** (minor): Unsetting the switch changes the next search, but it cannot undo a published pair repair. The default scan sees the new zstd frame as healthy. Document that a fresh extraction or separate output is needed for a clean comparison or rollback, and describe the mode's scope once the third known damaged texture is accounted for.
@ -733,3 +752,2 @@
done = nup.stand_in_damaged(romfs, damaged=[os.path.normpath(os.path.join(romfs, rel)) for rel in known])
note = os.path.join(out, DAMAGED_NOTE)
done = stand_in(romfs, damaged=[os.path.normpath(os.path.join(romfs, rel)) for rel in known])
if not done:
Member

L3: The human-readable note is the only record of old substitutions after HEADS is deleted. For example, default mode substitutes an unknown texture, pairs mode rejects it, then default mode sees a valid zstd prefix and deletes the note. The next pairs run accepts the same substitution. An unreadable note is also silently treated as empty. Keep an atomic, unpack-scoped record of substitutions and their mode, independent of the warning text. Use it on every mode transition and retain it through failures.

**L3**: The human-readable note is the only record of old substitutions after HEADS is deleted. For example, default mode substitutes an unknown texture, pairs mode rejects it, then default mode sees a valid zstd prefix and deletes the note. The next pairs run accepts the same substitution. An unreadable note is also silently treated as empty. Keep an atomic, unpack-scoped record of substitutions and their mode, independent of the warning text. Use it on every mode transition and retain it through failures.
nx2/make_game.py Outdated
@ -725,13 +744,12 @@ def stand_in_damaged_files(out, romfs):
if known is None:
Member

L15 (minor): Changing modes on an already checked output whose HEADS was deleted causes a full walk and about 100,000 file opens. The builder's own comment says that can take minutes with antivirus. Time the read phase on a kept output and report the one-time cost alongside fresh build results.

**L15** (minor): Changing modes on an already checked output whose HEADS was deleted causes a full walk and about 100,000 file opens. The builder's own comment says that can take minutes with antivirus. Time the read phase on a kept output and report the one-time cost alongside fresh build results.
nx2/make_game.py Outdated
@ -716,1 +724,3 @@
heads = os.path.join(out, HEADS)
previous = []
version, separator, prior_stamp = prior.partition("\n")
if pairs and (not prior or prior == stamp or (separator and version.removesuffix("\r") in ("resolved-v1", "resolved-v2")
Member

L2: A fresh extraction replaces the romfs and clears the checked marker but leaves the old damage note in place. Pairs mode interprets that missing marker as permission to reuse the note. If the old dump had a substitution for a file absent from the new complete dump, the new build reports that stale path as unresolved and stops. Bind repair metadata to the unpack and clear or replace it when publishing a new romfs.

**L2**: A fresh extraction replaces the romfs and clears the checked marker but leaves the old damage note in place. Pairs mode interprets that missing marker as permission to reuse the note. If the old dump had a substitution for a file absent from the new complete dump, the new build reports that stale path as unresolved and stops. Bind repair metadata to the unpack and clear or replace it when publishing a new romfs.
nx2/nup.py Outdated
@ -826,0 +860,4 @@
if rel.endswith(".bntx.zs"):
try:
data = texture_repair.read_frame(target, texture_repair.MAX_TEXTURE)
if texture_repair.texture_name(data)[0] == name.encode("utf-8"):
Member

L4: A matching BRTI name overrides the extraction's damage verdict. HEADS can contain a zstd member whose package digest failed even though it still decompresses and carries its original name. That file is skipped and can be stamped resolved, including for a texture outside the supported pairs. Keep fresh extraction failures authoritative and use a recorded repair identity to recognize bytes successfully written by an earlier attempt.

**L4**: A matching BRTI name overrides the extraction's damage verdict. HEADS can contain a zstd member whose package digest failed even though it still decompresses and carries its original name. That file is skipped and can be stamped resolved, including for a texture outside the supported pairs. Keep fresh extraction failures authoritative and use a recorded repair identity to recognize bytes successfully written by an earlier attempt.
nx2/nup.py Outdated
@ -826,0 +862,4 @@
data = texture_repair.read_frame(target, texture_repair.MAX_TEXTURE)
if texture_repair.texture_name(data)[0] == name.encode("utf-8"):
continue # (whole and its own: an original, or a repair done before while another file stays unresolved)
except errors:
Member

L5: A failed probe is treated as permission to replace a known target. A healthy dictionary-compressed BNTX fails the dictionary-free read, and an I/O failure takes the same path. If the donor is readable, the builder can overwrite the original. Use the archive dictionary where required, distinguish read or format failures from confirmed damage, and require positive evidence of damage or a prior substitution before writing.

**L5**: A failed probe is treated as permission to replace a known target. A healthy dictionary-compressed BNTX fails the dictionary-free read, and an I/O failure takes the same path. If the donor is readable, the builder can overwrite the original. Use the archive dictionary where required, distinguish read or format failures from confirmed damage, and require positive evidence of damage or a prior substitution before writing.
nx2/nup.py Outdated
@ -826,0 +873,4 @@
table = texture_repair.size_table(romfs, path_for)
if table is not None:
other, data = texture_repair.replacement(romfs, name, table, path_for)
except errors:
Member

L11 (minor): The repair discards every expected error and tells the player to redump. A donor held by another program and a donor with a mismatched allocation produce the same message. Preserve a short per-file reason and distinguish access failures from invalid or unsupported data.

**L11** (minor): The repair discards every expected error and tells the player to redump. A donor held by another program and a donor with a mismatched allocation produce the same message. Preserve a short per-file reason and distinguish access failures from invalid or unsupported data.
nx2/nup.py Outdated
@ -826,0 +876,4 @@
except errors:
other = None
if other:
with open(target + ".part", "wb") as f:
Member

L8: The fixed .part file is shared by concurrent builders targeting the same output with different --work folders. The existing lock covers only --work. On Linux one builder can rename the file while the other is still writing it. A failed write or rename also leaves the temporary behind. Serialize romfs and marker changes per output, stage in a unique file beside the target, and clean it up on failure.

**L8**: The fixed .part file is shared by concurrent builders targeting the same output with different --work folders. The existing lock covers only --work. On Linux one builder can rename the file while the other is still writing it. A failed write or rename also leaves the temporary behind. Serialize romfs and marker changes per output, stage in a unique file beside the target, and clean it up on failure.
nx2/nup.py Outdated
@ -826,0 +879,4 @@
with open(target + ".part", "wb") as f:
f.write(data)
os.replace(target + ".part", target)
other = os.path.normpath(os.path.join(romfs, os.path.relpath(other, long_path(root))))
Member

L10 (minor): The donor comes back under the canonical extraction path, but the displayed donor path is rebased against the lexical root after replacement. On Windows, an output below a junction into another drive can make relpath raise after the repair has already been published. Use the known romfs-relative donor path for reporting and resolve it before publishing.

**L10** (minor): The donor comes back under the canonical extraction path, but the displayed donor path is rebased against the lexical root after replacement. On Windows, an output below a junction into another drive can make relpath raise after the repair has already been published. Use the known romfs-relative donor path for reporting and resolve it before publishing.
@ -0,0 +64,4 @@
data = zstandard.ZstdDecompressor().decompress(Path(path).read_bytes())
self.assertEqual(consumer_name(data), bad)
self.assertEqual(len(data), len(fixture.bntx(good)))
self.assertEqual(data, texture_repair.rename_texture(fixture.bntx(good), good, bad))
Member

L12 (minor): The positive test constructs its expected output with the production rename function, and the fixture gets its donor choices from the production map. A wrong donor choice can pass. Author the expected mappings independently and compare the two name records, unchanged bytes, and image data against that fixture. In the bad-archive-bounds case, keep the dictionary name valid so the wrong name cannot mask a missing bounds check.

**L12** (minor): The positive test constructs its expected output with the production rename function, and the fixture gets its donor choices from the production map. A wrong donor choice can pass. Author the expected mappings independently and compare the two name records, unchanged bytes, and image data against that fixture. In the bad-archive-bounds case, keep the dictionary name valid so the wrong name cannot mask a missing bounds check.
@ -0,0 +13,4 @@
import zstandard
PAIRS = {
Member

L1: The documented 23.3 GB dump contains a third damaged texture in Ganon's Stage, recorded in the original stand-in commit bd5270246 and the legacy frame fix #2391. A fresh build from that dump repairs the two listed textures and then stops. Establish a verified treatment for the third texture or narrow the feature to a demonstrated dump variant with only these two failures. Update the description and build docs with that scope.

**L1**: The documented 23.3 GB dump contains a third damaged texture in Ganon's Stage, recorded in the original stand-in commit bd5270246 and the legacy frame fix #2391. A fresh build from that dump repairs the two listed textures and then stops. Establish a verified treatment for the third texture or narrow the feature to a demonstrated dump variant with only these two failures. Update the description and build docs with that scope.
@ -0,0 +81,4 @@
version, name_size, hashed, named = struct.unpack_from("<IIII", table, 6)
if version != 1 or not 0 < name_size <= 4096 or len(table) != 22 + 8 * hashed + (name_size + 4) * named:
raise ValueError("invalid RESTBL bounds")
sizes = {}
Member

L9: The 256 MiB decompression limit admits over 33 million hash entries. Materializing them as Python integers and dictionary entries can consume several GiB on a damaged dump before the builder reaches its normal error path. Keep only the four relevant keys while validating the table, and apply the same bounded approach to named entries.

**L9**: The 256 MiB decompression limit admits over 33 million hash entries. Materializing them as Python integers and dictionary entries can consume several GiB on a damaged dump before the builder reaches its normal error path. Keep only the four relevant keys while validating the table, and apply the same bounded approach to named entries.
@ -0,0 +124,4 @@
dimensions = struct.unpack_from("<IIII", data, brti + 0x24)
image_size = struct.unpack_from("<I", data, brti + 0x50)[0]
mip_table = struct.unpack_from("<Q", data, brti + 0x70)[0]
if not mip_count or not all(dimensions) or not image_size or not 0x40 <= mip_table <= len(data) - 8 * mip_count:
Member

L7: The payload check accepts any nonzero dimensions and image size if the declared image range lies inside the file. A donor with large dimensions but only one block of image data can pass, as can mip offsets into metadata. The allocation comparison only checks container length. Validate the expected format and layout and check each mip's required byte range against the image region before accepting a donor.

**L7**: The payload check accepts any nonzero dimensions and image size if the declared image range lies inside the file. A donor with large dimensions but only one block of image data can pass, as can mip offsets into metadata. The allocation comparison only checks container length. Validate the expected format and layout and check each mip's required byte range against the image region before accepting a donor.
@ -0,0 +153,4 @@
raise ValueError("texture names do not match the replacement pair")
# BRTI's texture name and the container's filename are the only supported name records. Even an apparent
# length-prefixed match elsewhere (for example in pixel data) must not be silently mutated.
allowed = {at, struct.unpack_from("<I", data, 0x10)[0]}
Member

L6: The BNTX filename field only adds an address to an allowlist. A donor with a valid BRTI name and a missing, unrelated, or out-of-range filename record still passes. The filename address is also never checked against the mip table or image bytes. Resolve and validate both records directly, require the expected donor name in each, and verify the exact spans before and after writing.

**L6**: The BNTX filename field only adds an address to an allowlist. A donor with a valid BRTI name and a missing, unrelated, or out-of-range filename record still passes. The filename address is also never checked against the mip table or image bytes. Resolve and validate both records directly, require the expected donor name in each, and verify the exact spans before and after writing.
futondestroyer left a comment

Recommendation: Request changes. Pair mode puts new textures into the existing game folder before it knows whether an unresolved candidate will stop the build. Validate the complete repair set before replacing any textures.

The patch also needs to resolve the earlier exact-head review's findings about the third damaged texture, the damage record, and the BNTX payload and name checks. Earlier review

Does it help? Unproven on an affected dump. The author reports ten authored Windows fixtures passing at b51ee688 and an unchanged patch after rebase. The base-side run stopped at a missing fixture import; it establishes neither a behavioral comparison nor compatibility with an affected dump. The updated documentation describes the opt-in mode. The Testing section still calls for the Windows player rebuild and manual scene verification, plus the Linux fixture.

Reviewed 0a83513e240b5978dcda10351320b7a8e47b8605 against actual merge base 6e1521a70a32603ed60a8263e4072625885669b7. Read the seven-file patch, the builder's unpack, update, stamp and hard-link paths, and the existing texture and resource-size readers. No PR code was run.

Blocking

  1. nx2/nup.py:881 publishes a valid repair immediately. A later unknown or invalid candidate is still returned as unresolved and stand_in_damaged_files stops the build. In an update, the previous executable can still use that folder, so the failed build leaves it using a partly repaired ROMFS. test_an_unresolved_retry_keeps_the_heads_and_never_rewrites_a_repair explicitly expects both replacements on the failing first attempt. Unsetting the switch will not restore those files. Validate and stage the complete set before replacing any files, then assert that unresolved input leaves every target byte-for-byte unchanged. Handle a publication failure so a partly committed set cannot silently become the next build's input.

Checked and fine

  • The fixtures construct their own bytes. The change reads donors from the builder's local unpacked ROMFS and writes the result there. The patch adds no asset payload, export, or network path. The source names the Project ReLink implementation and revision it adapts.
  • Fresh builds with LONGSHOT_STAND_INS unset use the previous repair function. The opt-in setting is documented as a build input. On --romfs-from, replacing a target entry protects the donor's original hard link from a write-through.

Testing and merge eligibility

CI was pending on the reviewed head. After fixing the source blockers, test a fresh build in both modes with independent output trees on a PC that already has the affected dump. Verify the resolved-file log and inspect both locations in game. Keep the dump, donors and repaired files on that PC. The current author request is a manual builder test, and an ordinary Beanstalk runtime comparison cannot verify the .nup extraction and repair. Keep the PR WIP pending source fixes and the affected-input result.

Overlaps

#546 covers a builder path for extracted-folder input. #1943 covers fallback at runtime. Keep their scopes explicit if either proceeds, since a runtime fallback can mask a failed builder repair in gameplay.

**Recommendation: Request changes.** Pair mode puts new textures into the existing game folder before it knows whether an unresolved candidate will stop the build. Validate the complete repair set before replacing any textures. The patch also needs to resolve the earlier exact-head review's findings about the third damaged texture, the damage record, and the BNTX payload and name checks. [Earlier review](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62612) **Does it help?** Unproven on an affected dump. The author reports ten authored Windows fixtures passing at `b51ee688` and an unchanged patch after rebase. The base-side run stopped at a missing fixture import; it establishes neither a behavioral comparison nor compatibility with an affected dump. The updated documentation describes the opt-in mode. The Testing section still calls for the Windows player rebuild and manual scene verification, plus the Linux fixture. **Reviewed** `0a83513e240b5978dcda10351320b7a8e47b8605` against actual merge base `6e1521a70a32603ed60a8263e4072625885669b7`. Read the seven-file patch, the builder's unpack, update, stamp and hard-link paths, and the existing texture and resource-size readers. No PR code was run. **Blocking** 1. `nx2/nup.py:881` publishes a valid repair immediately. A later unknown or invalid candidate is still returned as unresolved and `stand_in_damaged_files` stops the build. In an update, the previous executable can still use that folder, so the failed build leaves it using a partly repaired ROMFS. `test_an_unresolved_retry_keeps_the_heads_and_never_rewrites_a_repair` explicitly expects both replacements on the failing first attempt. Unsetting the switch will not restore those files. Validate and stage the complete set before replacing any files, then assert that unresolved input leaves every target byte-for-byte unchanged. Handle a publication failure so a partly committed set cannot silently become the next build's input. **Checked and fine** - The fixtures construct their own bytes. The change reads donors from the builder's local unpacked ROMFS and writes the result there. The patch adds no asset payload, export, or network path. The source names the Project ReLink implementation and revision it adapts. - Fresh builds with `LONGSHOT_STAND_INS` unset use the previous repair function. The opt-in setting is documented as a build input. On `--romfs-from`, replacing a target entry protects the donor's original hard link from a write-through. **Testing and merge eligibility** CI was pending on the reviewed head. After fixing the source blockers, test a fresh build in both modes with independent output trees on a PC that already has the affected dump. Verify the resolved-file log and inspect both locations in game. Keep the dump, donors and repaired files on that PC. The current author request is a manual builder test, and an ordinary Beanstalk runtime comparison cannot verify the `.nup` extraction and repair. Keep the PR WIP pending source fixes and the affected-input result. **Overlaps** [#546](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/546) covers a builder path for extracted-folder input. [#1943](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1943) covers fallback at runtime. Keep their scopes explicit if either proceeds, since a runtime fallback can mask a failed builder repair in gameplay.
nx2/nup.py Outdated
@ -826,0 +878,4 @@
if other:
with open(target + ".part", "wb") as f:
f.write(data)
os.replace(target + ".part", target)
Member

This commits a texture before a later unresolved file can stop the build. The existing game folder then keeps the experimental substitutions from the failed update, and the retry test currently expects those writes. Validate and stage every candidate before publishing repairs. Test that an unresolved candidate leaves both would-be targets unchanged.

This commits a texture before a later unresolved file can stop the build. The existing game folder then keeps the experimental substitutions from the failed update, and the retry test currently expects those writes. Validate and stage every candidate before publishing repairs. Test that an unresolved candidate leaves both would-be targets unchanged.
futondestroyer left a comment

Recommendation: Request changes. Pair mode puts new textures into the existing game folder before it knows whether an unresolved candidate will stop the build. Validate the complete repair set before replacing any textures.

The patch also needs to resolve the earlier exact-head review's findings about the third damaged texture, the damage record, and the BNTX payload and name checks. Earlier review

Does it help? Unproven on an affected dump. The author reports ten authored Windows fixtures passing at b51ee688 and an unchanged patch after rebase. The base-side run stopped at a missing fixture import; it establishes neither a behavioral comparison nor compatibility with an affected dump. The updated documentation describes the opt-in mode. The Testing section still calls for the Windows player rebuild and manual scene verification, plus the Linux fixture.

Reviewed 0a83513e240b5978dcda10351320b7a8e47b8605 against actual merge base 6e1521a70a32603ed60a8263e4072625885669b7. Read the seven-file patch, the builder's unpack, update, stamp and hard-link paths, and the existing texture and resource-size readers. No PR code was run.

Blocking

  1. nx2/nup.py:881 publishes a valid repair immediately. A later unknown or invalid candidate is still returned as unresolved and stand_in_damaged_files stops the build. In an update, the previous executable can still use that folder, so the failed build leaves it using a partly repaired ROMFS. test_an_unresolved_retry_keeps_the_heads_and_never_rewrites_a_repair explicitly expects both replacements on the failing first attempt. Unsetting the switch will not restore those files. Validate and stage the complete set before replacing any files, then assert that unresolved input leaves every target byte-for-byte unchanged. Handle a publication failure so a partly committed set cannot silently become the next build's input.

Checked and fine

  • The fixtures construct their own bytes. The change reads donors from the builder's local unpacked ROMFS and writes the result there. The patch adds no asset payload, export, or network path. The source names the Project ReLink implementation and revision it adapts.
  • Fresh builds with LONGSHOT_STAND_INS unset use the previous repair function. The opt-in setting is documented as a build input. On --romfs-from, replacing a target entry protects the donor's original hard link from a write-through.

Testing and merge eligibility

CI was pending on the reviewed head. After fixing the source blockers, test a fresh build in both modes with independent output trees on a PC that already has the affected dump. Verify the resolved-file log and inspect both locations in game. Keep the dump, donors and repaired files on that PC. The current author request is a manual builder test, and an ordinary Beanstalk runtime comparison cannot verify the .nup extraction and repair. Keep the PR WIP pending source fixes and the affected-input result.

Overlaps

#546 covers a builder path for extracted-folder input. #1943 covers fallback at runtime. Keep their scopes explicit if either proceeds, since a runtime fallback can mask a failed builder repair in gameplay.

**Recommendation: Request changes.** Pair mode puts new textures into the existing game folder before it knows whether an unresolved candidate will stop the build. Validate the complete repair set before replacing any textures. The patch also needs to resolve the earlier exact-head review's findings about the third damaged texture, the damage record, and the BNTX payload and name checks. [Earlier review](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62612) **Does it help?** Unproven on an affected dump. The author reports ten authored Windows fixtures passing at `b51ee688` and an unchanged patch after rebase. The base-side run stopped at a missing fixture import; it establishes neither a behavioral comparison nor compatibility with an affected dump. The updated documentation describes the opt-in mode. The Testing section still calls for the Windows player rebuild and manual scene verification, plus the Linux fixture. **Reviewed** `0a83513e240b5978dcda10351320b7a8e47b8605` against actual merge base `6e1521a70a32603ed60a8263e4072625885669b7`. Read the seven-file patch, the builder's unpack, update, stamp and hard-link paths, and the existing texture and resource-size readers. No PR code was run. **Blocking** 1. `nx2/nup.py:881` publishes a valid repair immediately. A later unknown or invalid candidate is still returned as unresolved and `stand_in_damaged_files` stops the build. In an update, the previous executable can still use that folder, so the failed build leaves it using a partly repaired ROMFS. `test_an_unresolved_retry_keeps_the_heads_and_never_rewrites_a_repair` explicitly expects both replacements on the failing first attempt. Unsetting the switch will not restore those files. Validate and stage the complete set before replacing any files, then assert that unresolved input leaves every target byte-for-byte unchanged. Handle a publication failure so a partly committed set cannot silently become the next build's input. **Checked and fine** - The fixtures construct their own bytes. The change reads donors from the builder's local unpacked ROMFS and writes the result there. The patch adds no asset payload, export, or network path. The source names the Project ReLink implementation and revision it adapts. - Fresh builds with `LONGSHOT_STAND_INS` unset use the previous repair function. The opt-in setting is documented as a build input. On `--romfs-from`, replacing a target entry protects the donor's original hard link from a write-through. **Testing and merge eligibility** CI was pending on the reviewed head. After fixing the source blockers, test a fresh build in both modes with independent output trees on a PC that already has the affected dump. Verify the resolved-file log and inspect both locations in game. Keep the dump, donors and repaired files on that PC. The current author request is a manual builder test, and an ordinary Beanstalk runtime comparison cannot verify the `.nup` extraction and repair. Keep the PR WIP pending source fixes and the affected-input result. **Overlaps** [#546](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/546) covers a builder path for extracted-folder input. [#1943](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1943) covers fallback at runtime. Keep their scopes explicit if either proceeds, since a runtime fallback can mask a failed builder repair in gameplay.
nx2/nup.py Outdated
@ -826,0 +878,4 @@
if other:
with open(target + ".part", "wb") as f:
f.write(data)
os.replace(target + ".part", target)
Member

This commits a texture before a later unresolved file can stop the build. The existing game folder then keeps the experimental substitutions from the failed update, and the retry test currently expects those writes. Validate and stage every candidate before publishing repairs. Test that an unresolved candidate leaves both would-be targets unchanged.

This commits a texture before a later unresolved file can stop the build. The existing game folder then keeps the experimental substitutions from the failed update, and the retry test currently expects those writes. Validate and stage every candidate before publishing repairs. Test that an unresolved candidate leaves both would-be targets unchanged.
siantm force-pushed feat/builder-texture-pairs from 0a83513e24
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 5s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to a1990170a1
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 28s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-21 03:42:04 +00:00
Compare
Author
Contributor

Pushed a1990170 (rebased onto main @ f1f2dc5b; the feature commit unchanged, git range-diff "="), answering futondestroyer's review 62700/62704 and BeanBot's 62612: the failing game-free tests first (e7951cff), then the fix (a1990170). Pair mode now writes nothing while any file stays unresolved and completes a set a failed write left part-way; the unpack's damage list is kept as its record; the donor's layout and both name records are checked; pairs mode's scope (exactly the two known textures) is stated. The description's Summary, Testing, Still to test and Risks are updated to match.

Pushed a1990170 (rebased onto `main` @ f1f2dc5b; the feature commit unchanged, `git range-diff` "="), answering futondestroyer's review 62700/62704 and BeanBot's 62612: the failing game-free tests first (e7951cff), then the fix (a1990170). Pair mode now writes nothing while any file stays unresolved and completes a set a failed write left part-way; the unpack's damage list is kept as its record; the donor's layout and both name records are checked; pairs mode's scope (exactly the two known textures) is stated. The description's Summary, Testing, Still to test and Risks are updated to match.
Author
Contributor

Re #1983 (comment): fixed in a1990170, with the failing game-free tests first in e7951cff (they fail on the PR's first commit and pass at a1990170, same queue job). Pair mode now makes and checks every replacement in memory and writes none while any file stays unresolved, so a failed build leaves the game folder's textures byte for byte as they were; a failure while publishing removes its temporary file, stops the build without marking the check done, and the next build completes the set from the kept, stamped damage list (test_a_replacement_that_stops_part_way_is_completed_next_time). BeanBot's 62612 findings (third texture, damage record, BNTX payload and name checks) are fixed or scoped in the same commit and answered on each line; the build from an affected dump in both modes, with both places looked at in game, is pending on a PC that has that dump.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62700: fixed in a1990170, with the failing game-free tests first in e7951cff (they fail on the PR's first commit and pass at a1990170, same queue job). Pair mode now makes and checks every replacement in memory and writes none while any file stays unresolved, so a failed build leaves the game folder's textures byte for byte as they were; a failure while publishing removes its temporary file, stops the build without marking the check done, and the next build completes the set from the kept, stamped damage list (`test_a_replacement_that_stops_part_way_is_completed_next_time`). BeanBot's 62612 findings (third texture, damage record, BNTX payload and name checks) are fixed or scoped in the same commit and answered on each line; the build from an affected dump in both modes, with both places looked at in game, is pending on a PC that has that dump.
Author
Contributor

Re #1983 (comment) (the same review as 62700): fixed in a1990170, test first in e7951cff; the details are in my reply to 62700. Pair mode writes nothing while any candidate stays unresolved, and a publication that stops part-way is completed by the next build.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62704 (the same review as 62700): fixed in a1990170, test first in e7951cff; the details are in my reply to 62700. Pair mode writes nothing while any candidate stays unresolved, and a publication that stops part-way is completed by the next build.
Author
Contributor

Re #1983 (comment): fixed in a1990170. nup.repair_texture_pairs stages every replacement in memory and returns before writing when any candidate is unresolved; test_an_unresolved_candidate_leaves_both_targets_unchanged and the retry test (now test_an_unresolved_retry_keeps_the_heads_and_writes_nothing) assert no texture write and both would-be targets byte for byte unchanged. Both fail on the PR's first commit (e7951cff) and pass at a1990170.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62699: fixed in a1990170. `nup.repair_texture_pairs` stages every replacement in memory and returns before writing when any candidate is unresolved; `test_an_unresolved_candidate_leaves_both_targets_unchanged` and the retry test (now `test_an_unresolved_retry_keeps_the_heads_and_writes_nothing`) assert no texture write and both would-be targets byte for byte unchanged. Both fail on the PR's first commit (e7951cff) and pass at a1990170.
Author
Contributor

Re #1983 (comment) (the same point as 62699): fixed in a1990170, test first in e7951cff. An unresolved candidate now leaves both would-be targets unchanged, and the retry test expects no writes on either attempt.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62703 (the same point as 62699): fixed in a1990170, test first in e7951cff. An unresolved candidate now leaves both would-be targets unchanged, and the retry test expects no writes on either attempt.
Author
Contributor

Re #1983 (comment) (L1): scoped in a1990170. Pairs mode covers exactly the two known textures: the third (Ganon's stage) has no known pair, so a build in this mode stops naming it, says nothing was replaced and to build without LONGSHOT_STAND_INS=pairs for the default similar file (test_a_third_damaged_texture_stops_the_build_and_says_what_to_do); the build reference and the description say so in one sentence. Whether a pair with an equal allocation exists for the third texture is to be checked on a PC with the affected dump: pending.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62598 (L1): scoped in a1990170. Pairs mode covers exactly the two known textures: the third (Ganon's stage) has no known pair, so a build in this mode stops naming it, says nothing was replaced and to build without `LONGSHOT_STAND_INS=pairs` for the default similar file (`test_a_third_damaged_texture_stops_the_build_and_says_what_to_do`); the build reference and the description say so in one sentence. Whether a pair with an equal allocation exists for the third texture is to be checked on a PC with the affected dump: pending.
Author
Contributor

Re #1983 (comment) (L2): fixed in a1990170. The note is read only after a default-mode check of this same unpack (its stamp in .longshot_romfs_checked), never when the mark is missing, and a fresh extraction now removes the note together with the mark; the damage list .longshot_romfs_heads carries the unpack's stamp and is replaced by the new unpack. test_a_note_from_another_unpack_is_not_taken fails at e7951cff and passes at a1990170.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62599 (L2): fixed in a1990170. The note is read only after a default-mode check of this same unpack (its stamp in `.longshot_romfs_checked`), never when the mark is missing, and a fresh extraction now removes the note together with the mark; the damage list `.longshot_romfs_heads` carries the unpack's stamp and is replaced by the new unpack. `test_a_note_from_another_unpack_is_not_taken` fails at e7951cff and passes at a1990170.
Author
Contributor

Re #1983 (comment) (L3): fixed in a1990170. .longshot_romfs_heads (the unpack's stamped damage list) now stays after the check in both modes and, in pairs mode, is written atomically before any replacement with the note's paths added, so default, pairs, default, pairs keeps the substituted file a damaged file (test_changes_of_mode_keep_the_damage_record); a note that can't be read stops the build instead of reading as empty. The mode is not stored per path: a change of mode re-checks every recorded path.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62600 (L3): fixed in a1990170. `.longshot_romfs_heads` (the unpack's stamped damage list) now stays after the check in both modes and, in pairs mode, is written atomically before any replacement with the note's paths added, so default, pairs, default, pairs keeps the substituted file a damaged file (`test_changes_of_mode_keep_the_damage_record`); a note that can't be read stops the build instead of reading as empty. The mode is not stored per path: a change of mode re-checks every recorded path.
Author
Contributor

Re #1983 (comment) (L4): fixed in a1990170. A recorded path is replaced whatever its bytes decode to; an earlier repair is recognised as the identical replacement (the repair is deterministic) and not written again. test_a_recorded_texture_is_repaired_whatever_its_bytes_say.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62601 (L4): fixed in a1990170. A recorded path is replaced whatever its bytes decode to; an earlier repair is recognised as the identical replacement (the repair is deterministic) and not written again. `test_a_recorded_texture_is_repaired_whatever_its_bytes_say`.
Author
Contributor

Re #1983 (comment) (L5): fixed in a1990170. Frames are decoded with the game's zs.zsdic where they name it (its id checked); a probe of a known name that can't be read or decoded is no evidence and leaves the file, and only a valid texture under another texture's name counts as an earlier substitution. test_a_healthy_texture_made_with_the_game_dictionary_is_kept (a frame that needs the dictionary).

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62602 (L5): fixed in a1990170. Frames are decoded with the game's `zs.zsdic` where they name it (its id checked); a probe of a known name that can't be read or decoded is no evidence and leaves the file, and only a valid texture under another texture's name counts as an earlier substitution. `test_a_healthy_texture_made_with_the_game_dictionary_is_kept` (a frame that needs the dictionary).
Author
Contributor

Re #1983 (comment) (L6): fixed in a1990170. texture_repair.layout resolves the header's file-name record and the BRTI's name record directly, each bounded and outside the header, BRTI, mip table and image data; both must hold the donor's name (the file name with or without .bntx), the name may occur nowhere else, and both are read back after the change. New malformed inputs: no, out-of-range and unrelated file-name records.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62603 (L6): fixed in a1990170. `texture_repair.layout` resolves the header's file-name record and the BRTI's name record directly, each bounded and outside the header, BRTI, mip table and image data; both must hold the donor's name (the file name with or without `.bntx`), the name may occur nowhere else, and both are read back after the change. New malformed inputs: no, out-of-range and unrelated file-name records.
Author
Contributor

Re #1983 (comment) (L7): fixed in a1990170. The donor's format must be one of the listed block formats (BC1-BC7, ASTC 4x4-12x12, RGBA8/BGRA8), each mip level must hold at least its blocks before the next level or the end of the image data, the levels times the array layers must fit the image size, and the image data must not overlap the header, BRTI or mip table. This is a lower bound (block-linear padding is not modelled); new inputs: 4096x4096 in one block, image data on the BRTI, a second mip past the data, an unknown format.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62604 (L7): fixed in a1990170. The donor's format must be one of the listed block formats (BC1-BC7, ASTC 4x4-12x12, RGBA8/BGRA8), each mip level must hold at least its blocks before the next level or the end of the image data, the levels times the array layers must fit the image size, and the image data must not overlap the header, BRTI or mip table. This is a lower bound (block-linear padding is not modelled); new inputs: 4096x4096 in one block, image data on the BRTI, a second mip past the data, an unknown format.
Author
Contributor

Re #1983 (comment) (L8): fixed in a1990170 for this mode's files: each replacement goes through a mkstemp file of its own beside the target, removed when the write or the rename fails (test_a_replacement_that_stops_part_way_is_completed_next_time checks none is left). A lock per output folder across builders with different --work folders is not added: on main the extraction, the marks and the default mode's stand-ins already share an output unlocked, so that is left for a separate change.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62605 (L8): fixed in a1990170 for this mode's files: each replacement goes through a `mkstemp` file of its own beside the target, removed when the write or the rename fails (`test_a_replacement_that_stops_part_way_is_completed_next_time` checks none is left). A lock per output folder across builders with different `--work` folders is not added: on `main` the extraction, the marks and the default mode's stand-ins already share an output unlocked, so that is left for a separate change.
Author
Contributor

Re #1983 (comment) (L9): fixed in a1990170. texture_repair.allocations walks the hashed entries with struct.iter_unpack, still checking their order, and keeps only the four pair entries; named entries likewise. The decoded table itself stays bounded at 256 MiB.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62606 (L9): fixed in a1990170. `texture_repair.allocations` walks the hashed entries with `struct.iter_unpack`, still checking their order, and keeps only the four pair entries; named entries likewise. The decoded table itself stays bounded at 256 MiB.
Author
Contributor

Re #1983 (comment) (L10): fixed in a1990170. The donor path reported is the romfs-relative one, built from the pair's name before anything is written; nothing is rebased against the canonical path any more.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62607 (L10): fixed in a1990170. The donor path reported is the romfs-relative one, built from the pair's name before anything is written; nothing is rebased against the canonical path any more.
Author
Contributor

Re #1983 (comment) (L11): fixed in a1990170. Each unresolved file gets a short reason in the log and the note: no known pair, its pair is damaged too, its pair could not be read (with the OS's message), or its pair does not fit (with the check that refused it). test_each_unresolved_file_says_why.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62608 (L11): fixed in a1990170. Each unresolved file gets a short reason in the log and the note: no known pair, its pair is damaged too, its pair could not be read (with the OS's message), or its pair does not fit (with the check that refused it). `test_each_unresolved_file_says_why`.
Author
Contributor

Re #1983 (comment) (L12): fixed in e7951cff. The fixture writes its two pairs out itself (a test compares texture_repair.PAIRS with them), the positive test expects the fixture's own texture under the damaged name byte for byte (both name records and every other byte), and bad-archive-bounds keeps the zs.zsdic name so only the bounds can refuse it.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62609 (L12): fixed in e7951cff. The fixture writes its two pairs out itself (a test compares `texture_repair.PAIRS` with them), the positive test expects the fixture's own texture under the damaged name byte for byte (both name records and every other byte), and bad-archive-bounds keeps the `zs.zsdic` name so only the bounds can refuse it.
Author
Contributor

Re #1983 (comment) (L14): fixed in a1990170. The damage list is now kept, so unsetting the switch re-checks the recorded paths and puts similar files back in place of the pairs' textures; docs/dev/reference/build.md says so, says a clean comparison needs a game folder per mode, and states that pairs mode covers exactly the two known textures (a dump with another damaged one cannot be built in this mode yet).

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62610 (L14): fixed in a1990170. The damage list is now kept, so unsetting the switch re-checks the recorded paths and puts similar files back in place of the pairs' textures; `docs/dev/reference/build.md` says so, says a clean comparison needs a game folder per mode, and states that pairs mode covers exactly the two known textures (a dump with another damaged one cannot be built in this mode yet).
Author
Contributor

Re #1983 (comment) (L15): the record is kept in a1990170, so a change of mode reads the unpack's stamped list and opens no other file (test_changes_of_mode_keep_the_damage_record fails if the scan runs). The one-time walk remains only for a folder whose list an older builder deleted after its check; timing it on a kept output needs the affected dump and is pending.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62611 (L15): the record is kept in a1990170, so a change of mode reads the unpack's stamped list and opens no other file (`test_changes_of_mode_keep_the_damage_record` fails if the scan runs). The one-time walk remains only for a folder whose list an older builder deleted after its check; timing it on a kept output needs the affected dump and is pending.
Author
Contributor

Re #1983 (comment), L13: fixed in a1990170. nx2/test_nup_stand_in.py, nx2/test_nup_zstd_heads.py and tests/test_damaged_stand_in_fail_fast.py (the default mode's, HEADS and fail-fast cases) clear LONGSHOT_STAND_INS before they run; the pairs tests set it per test, and the default-mode test there clears it explicitly.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62612, L13: fixed in a1990170. `nx2/test_nup_stand_in.py`, `nx2/test_nup_zstd_heads.py` and `tests/test_damaged_stand_in_fail_fast.py` (the default mode's, HEADS and fail-fast cases) clear `LONGSHOT_STAND_INS` before they run; the pairs tests set it per test, and the default-mode test there clears it explicitly.
Author
Contributor

Re #1983 (comment), L16: the edit part is in a1990170: each pair is logged with its decoded size, allocation and time, each file's verdict (with its reason) goes to the log and the note, and the check's total time is said. The author-side runs (both real donors' compression and the 280-byte allocation overhead, a fresh build in each mode into separate game folders, both places looked at in game, Windows and Linux, build time) need the affected dump: pending on the PC that has it, and Measurements will get them.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-62612, L16: the edit part is in a1990170: each pair is logged with its decoded size, allocation and time, each file's verdict (with its reason) goes to the log and the note, and the check's total time is said. The author-side runs (both real donors' compression and the 280-byte allocation overhead, a fresh build in each mode into separate game folders, both places looked at in game, Windows and Linux, build time) need the affected dump: pending on the PC that has it, and Measurements will get them.
Author
Contributor

Tested corrected: an earlier run on this PC had no Vulkan headers for the renderer checks (RX_CPM unset), so the ones it listed as also failing on main had compiled nothing on either side. They ran again with the headers at a1990170 and on main (queue job fb2-vk-1); the description now gives those results. No check fails at this head alone; nothing in the branch changed.

Tested corrected: an earlier run on this PC had no Vulkan headers for the renderer checks (`RX_CPM` unset), so the ones it listed as also failing on `main` had compiled nothing on either side. They ran again with the headers at a1990170 and on `main` (queue job fb2-vk-1); the description now gives those results. No check fails at this head alone; nothing in the branch changed.
futondestroyer left a comment

Recommendation: Request changes. The new validation pass fixes writes on unresolved input. Two migration paths can still lose the damage record, which prevents switching back to the previous mode and can hide an unresolved texture.

Reviewed a1990170a14bc8abe50813b37e0724563676fca4 against actual merge base f1f2dc5bd2821cef061798ba40a043d281ac1707, including the fixes since the previous review. No PR code was run.

Blocking

  1. nx2/nup.py:884 discovers earlier wrong-name substitutes after nx2/make_game.py:763 has persisted the damage list. For a legacy build with a missing damage note and one discoverable wrong-name substitute, the scan sees healthy Zstd headers and records an empty list. The probe then repairs the substitute under the correct name. When the switch is unset, default mode trusts the empty list, skips the target and removes the note. This silently preserves the experimental texture despite the documented switch-back behavior. If a later write fails, the corrected embedded name also prevents the next probe from rediscovering that completed path. Persist the effective candidate set before the first replacement. Extend the existing no-note migration fixture through switching back to default and check both the durable list and the restored file.
  2. nx2/make_game.py:3162 removes the old install's damage note and check marker before extraction starts. The existing ROMFS is kept when the new package fails validation or extraction. A legacy install with no .longshot_romfs_heads then loses the only list of its original stand-ins while still using them. On the next pairs-mode update, the scan can miss a previously substituted third texture because the replacement has a valid Zstd header and the probe covers only the two known names. Retire the old markers when the new unpack takes ownership of the output. Add a fixture where a failed extraction preserves the old ROMFS and both of its markers.

Checked and fine

  • The prior unresolved-candidate case now validates all planned replacements before writing any textures. The retry fixture covers an I/O failure partway through publication when the targets are already recorded. The doc and fixture now make clear that a third damaged texture stops pairs mode.
  • The patch builds fixtures in Python and uses donors from the builder's own extracted ROMFS. It includes no extracted assets or export path. The stronger layout, name and resource-allocation checks still need confirmation on the real donors.

Does it help? Unproven on the affected dump. The author reports 19 Windows fixtures passing at this head after failures on the test-only commit, plus passing default stand-in, Zstd-head, progress and fail-fast fixtures. Full changed checks, Linux fixtures and the affected Windows build and scene checks remain open. CI was pending in the current-head snapshot. Author evidence

Keep LONGSHOT_STAND_INS=pairs opt-in and the PR WIP. The author's manual Windows test requires an affected local .nup in separate output folders for the two modes, and scene verification only after the pairs build succeeds. The farm page has no plan or result. Its documented build variants cannot set this build-time flag, and timed builds use the base's ROMFS. Keep original and repaired assets on the tester's PC. Related builder and runtime fallback work remains separately scoped. Farm page · Builder folder input · Runtime fallback

**Recommendation: Request changes.** The new validation pass fixes writes on unresolved input. Two migration paths can still lose the damage record, which prevents switching back to the previous mode and can hide an unresolved texture. **Reviewed** `a1990170a14bc8abe50813b37e0724563676fca4` against actual merge base `f1f2dc5bd2821cef061798ba40a043d281ac1707`, including the fixes since the previous review. No PR code was run. **Blocking** 1. `nx2/nup.py:884` discovers earlier wrong-name substitutes after `nx2/make_game.py:763` has persisted the damage list. For a legacy build with a missing damage note and one discoverable wrong-name substitute, the scan sees healthy Zstd headers and records an empty list. The probe then repairs the substitute under the correct name. When the switch is unset, default mode trusts the empty list, skips the target and removes the note. This silently preserves the experimental texture despite the documented switch-back behavior. If a later write fails, the corrected embedded name also prevents the next probe from rediscovering that completed path. Persist the effective candidate set before the first replacement. Extend the existing no-note migration fixture through switching back to default and check both the durable list and the restored file. 2. `nx2/make_game.py:3162` removes the old install's damage note and check marker before extraction starts. The existing ROMFS is kept when the new package fails validation or extraction. A legacy install with no `.longshot_romfs_heads` then loses the only list of its original stand-ins while still using them. On the next pairs-mode update, the scan can miss a previously substituted third texture because the replacement has a valid Zstd header and the probe covers only the two known names. Retire the old markers when the new unpack takes ownership of the output. Add a fixture where a failed extraction preserves the old ROMFS and both of its markers. **Checked and fine** - The prior unresolved-candidate case now validates all planned replacements before writing any textures. The retry fixture covers an I/O failure partway through publication when the targets are already recorded. The doc and fixture now make clear that a third damaged texture stops pairs mode. - The patch builds fixtures in Python and uses donors from the builder's own extracted ROMFS. It includes no extracted assets or export path. The stronger layout, name and resource-allocation checks still need confirmation on the real donors. **Does it help?** Unproven on the affected dump. The author reports 19 Windows fixtures passing at this head after failures on the test-only commit, plus passing default stand-in, Zstd-head, progress and fail-fast fixtures. Full changed checks, Linux fixtures and the affected Windows build and scene checks remain open. CI was pending in the current-head snapshot. [Author evidence](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-63101) Keep `LONGSHOT_STAND_INS=pairs` opt-in and the PR WIP. The author's manual Windows test requires an affected local `.nup` in separate output folders for the two modes, and scene verification only after the pairs build succeeds. The farm page has no plan or result. Its documented build variants cannot set this build-time flag, and timed builds use the base's ROMFS. Keep original and repaired assets on the tester's PC. Related builder and runtime fallback work remains separately scoped. [Farm page](https://beanstalk.magicbean.dev/prs/1983) · [Builder folder input](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/546) · [Runtime fallback](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1943)
nx2/make_game.py Outdated
@ -3106,3 +3161,2 @@
shutil.rmtree(game, ignore_errors=True)
with contextlib.suppress(FileNotFoundError): # (a new unpack is checked again)
os.remove(os.path.join(out, ".longshot_romfs_checked"))
for name in (".longshot_romfs_checked", DAMAGED_NOTE): # (a new unpack is checked again; the old one's note goes)
Member

The old ROMFS survives a failed extraction, but these markers are erased before reading the new package. A legacy install with no HEADS then loses its only record of earlier stand-ins. A subsequent pairs run may miss an unsupported third texture hidden behind a valid Zstd header. Keep the old markers until the new unpack succeeds and owns the output.

The old ROMFS survives a failed extraction, but these markers are erased before reading the new package. A legacy install with no HEADS then loses its only record of earlier stand-ins. A subsequent pairs run may miss an unsupported third texture hidden behind a valid Zstd header. Keep the old markers until the new unpack succeeds and owns the output.
nx2/nup.py Outdated
@ -826,0 +881,4 @@
try:
data = texture_repair.read_frame(path_for(rel), texture_repair.MAX_TEXTURE, dictionary())
if texture_repair.layout(data)[0] != name.encode("utf-8"):
candidates.append(given(rel)) # (a whole texture under another's name: an earlier check's similar file)
Member

A recovered wrong-name substitute is added here after the builder has saved the damage list. With a missing legacy note, the repair can succeed while HEADS stays empty. Switching back to default then skips this target and keeps the experimental texture. Record probe-discovered paths before committing repairs and cover this with the no-note migration fixture.

A recovered wrong-name substitute is added here after the builder has saved the damage list. With a missing legacy note, the repair can succeed while HEADS stays empty. Switching back to default then skips this target and keeps the experimental texture. Record probe-discovered paths before committing repairs and cover this with the no-note migration fixture.
siantm force-pushed feat/builder-texture-pairs from a1990170a1
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 28s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 9fdcce1d36
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 7s
2026-11-21 09:32:21 +00:00
Compare
Author
Contributor

Pushed 9fdcce1d (rebased onto main @ 9dd41326; the earlier commits unchanged, git range-diff "=" for all three), answering futondestroyer's second review 65448: failing game-free tests first (df7d58e4), then the fix (9fdcce1d). The probe's finds now go into the damage list before anything is replaced, and a new unpack removes the old romfs's check mark and note only once it takes the folder over. tests/test_make_game_cache.py now runs with the PR's checks; it had not run at a1990170, where its expectation that the list is deleted after the check was out of date. Summary, Commits, Tested and Still to test are updated.

Pushed 9fdcce1d (rebased onto `main` @ 9dd41326; the earlier commits unchanged, `git range-diff` "=" for all three), answering futondestroyer's second review 65448: failing game-free tests first (df7d58e4), then the fix (9fdcce1d). The probe's finds now go into the damage list before anything is replaced, and a new unpack removes the old romfs's check mark and note only once it takes the folder over. `tests/test_make_game_cache.py` now runs with the PR's checks; it had not run at a1990170, where its expectation that the list is deleted after the check was out of date. Summary, Commits, Tested and Still to test are updated.
Author
Contributor

Re #1983 (comment): fixed in 9fdcce1d, test first in df7d58e4. nup.repair_texture_pairs now hands every path it takes as damaged, the probe's finds included, to a record callback before the first write, and make_game.py writes .longshot_romfs_heads from it. The no-note migration case of test_an_earlier_similar_file_is_repaired_and_healthy_originals_kept now goes on through unsetting the switch: the list must name the repaired path, and the default mode puts its similar file back there with the list kept; it fails at df7d58e4 and passes at 9fdcce1d.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-65446: fixed in 9fdcce1d, test first in df7d58e4. `nup.repair_texture_pairs` now hands every path it takes as damaged, the probe's finds included, to a `record` callback before the first write, and `make_game.py` writes `.longshot_romfs_heads` from it. The no-note migration case of `test_an_earlier_similar_file_is_repaired_and_healthy_originals_kept` now goes on through unsetting the switch: the list must name the repaired path, and the default mode puts its similar file back there with the list kept; it fails at df7d58e4 and passes at 9fdcce1d.
Author
Contributor

Re #1983 (comment): fixed in 9fdcce1d, test first in df7d58e4. The check mark and the damage note are now removed only once the new package was read and checked, just before the old romfs is replaced; until then a failed read keeps them with the old romfs. tests/test_make_game_cache.py test_a_failed_new_unpack_keeps_the_old_romfs_check_and_note covers a failed extraction, an unsupported build and a package with no romfs; it fails at df7d58e4 and passes at 9fdcce1d.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-65447: fixed in 9fdcce1d, test first in df7d58e4. The check mark and the damage note are now removed only once the new package was read and checked, just before the old romfs is replaced; until then a failed read keeps them with the old romfs. `tests/test_make_game_cache.py` `test_a_failed_new_unpack_keeps_the_old_romfs_check_and_note` covers a failed extraction, an unsupported build and a package with no romfs; it fails at df7d58e4 and passes at 9fdcce1d.
Author
Contributor

Re #1983 (comment): both migration paths are fixed in 9fdcce1d, with their failing tests first in df7d58e4, on main @ 9dd41326; replies on 65446 and 65447 give the details. The probe's finds now go into .longshot_romfs_heads before anything is replaced, and a new unpack retires the old romfs's check mark and note only when it takes the folder over. The build from the affected dump in both modes, with both places looked at in game, is still pending on a PC that has the dump.

Re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-65448: both migration paths are fixed in 9fdcce1d, with their failing tests first in df7d58e4, on `main` @ 9dd41326; replies on 65446 and 65447 give the details. The probe's finds now go into `.longshot_romfs_heads` before anything is replaced, and a new unpack retires the old romfs's check mark and note only when it takes the folder over. The build from the affected dump in both modes, with both places looked at in game, is still pending on a PC that has the dump.
futondestroyer left a comment

Recommendation: Wait for the affected-dump build and remaining checks. The two source blockers from my previous review are resolved at 9fdcce1d. The PR remains WIP. A repository admin can dismiss my earlier request for changes.

  • The replacement planner now calls record(candidates) after the probe adds any wrong-name substitutes and before the first texture write. The builder saves that full set under the unpack stamp. The no-note migration fixture continues through unsetting the switch and verifies both the durable damage list and the default-mode substitute.
  • The builder keeps the old check marker and damage note while it extracts and validates a replacement package. The new cache fixture covers extraction failure, an unsupported build, and a package with no ROMFS. In all three cases it checks that the old cache and markers survive.

Reviewed 9fdcce1d3636741e255f5f0e66b3289c17688f5a against actual merge base 9dd41326e71094d8b774459dac5154eb5a3306d9. The range-diff confirms the previous three commits are unchanged. I read the two new commits, their four changed files, and the surrounding call paths. No PR code was run.

Does it help? The author reports that the new migration and cache cases each fail at the test-only commit df7d58e4 and pass at this head. The Windows pair suite reports 19 tests passing. The cache suite reports 49 cases with 2 skips and no failures. Real donor compatibility remains unproven. Author's update

Keep LONGSHOT_STAND_INS=pairs opt-in. The remaining manual check needs an affected .nup already on the tester's PC, separate fresh outputs for both modes, and a successful pairs build before inspecting both textures in game. Record build outcome and time, confirm the unsupported third texture stops cleanly where present, and finish the Linux fixture, current-head checks and CI. The regular Farm path uses a prepared ROMFS and offers no build variant for this flag, so it cannot replace the manual test. Farm status

**Recommendation: Wait for the affected-dump build and remaining checks.** The two source blockers from my previous review are resolved at `9fdcce1d`. The PR remains WIP. A repository admin can dismiss my earlier request for changes. - The replacement planner now calls `record(candidates)` after the probe adds any wrong-name substitutes and before the first texture write. The builder saves that full set under the unpack stamp. The no-note migration fixture continues through unsetting the switch and verifies both the durable damage list and the default-mode substitute. - The builder keeps the old check marker and damage note while it extracts and validates a replacement package. The new cache fixture covers extraction failure, an unsupported build, and a package with no ROMFS. In all three cases it checks that the old cache and markers survive. **Reviewed** `9fdcce1d3636741e255f5f0e66b3289c17688f5a` against actual merge base `9dd41326e71094d8b774459dac5154eb5a3306d9`. The range-diff confirms the previous three commits are unchanged. I read the two new commits, their four changed files, and the surrounding call paths. No PR code was run. **Does it help?** The author reports that the new migration and cache cases each fail at the test-only commit `df7d58e4` and pass at this head. The Windows pair suite reports 19 tests passing. The cache suite reports 49 cases with 2 skips and no failures. Real donor compatibility remains unproven. [Author's update](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-65492) Keep `LONGSHOT_STAND_INS=pairs` opt-in. The remaining manual check needs an affected `.nup` already on the tester's PC, separate fresh outputs for both modes, and a successful pairs build before inspecting both textures in game. Record build outcome and time, confirm the unsupported third texture stops cleanly where present, and finish the Linux fixture, current-head checks and CI. The regular Farm path uses a prepared ROMFS and offers no build variant for this flag, so it cannot replace the manual test. [Farm status](https://beanstalk.magicbean.dev/prs/1983)
Author
Contributor

@futondestroyer, re #1983 (comment): thank you. The affected-dump checks are with the author's runs on this PC and have not run yet: fresh builds from the affected .nup into separate output folders with the switch unset and with pairs, the build outcome and time of each, both textures looked at in game after a successful pairs build, and the search for a pair for the third texture (which stops pairs mode cleanly as it is). The Linux fixture cannot run on this PC; the forge's CI on 9fdcce1d has not run yet. Results go into the description when they are in.

@futondestroyer, re https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1983#issuecomment-65630: thank you. The affected-dump checks are with the author's runs on this PC and have not run yet: fresh builds from the affected `.nup` into separate output folders with the switch unset and with `pairs`, the build outcome and time of each, both textures looked at in game after a successful pairs build, and the search for a pair for the third texture (which stops pairs mode cleanly as it is). The Linux fixture cannot run on this PC; the forge's CI on 9fdcce1d has not run yet. Results go into the description when they are in.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 7s
This pull request has changes conflicting with the target branch.
  • nx2/make_game.py
Some workflows are waiting to be reviewed.
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'feat/builder-texture-pairs:siantm-feat/builder-texture-pairs'
git switch 'siantm-feat/builder-texture-pairs'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1983
No description provided.