WIP: feat(builder): the two known damaged textures replaced by their pairs, sizes and names checked (LONGSHOT_STAND_INS=pairs, off) #1983
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1983
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:feat/builder-texture-pairs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
An incomplete dump (about 23.3 GB instead of 23.8 GB) has damaged textures. Today the builder copies in a whole file
of the same kind with the nearest name and size (
nup.stand_in_damaged, #1453). That file keeps its own embeddedtexture name, and nothing checks its size against what the game allocates for the damaged one.
This adds a second mode beside it, off unless set: with
LONGSHOT_STAND_INS=pairs, only the two textures known to bedamaged in those dumps are repaired, each from its pair (
nx2/texture_repair.py):donor's payload must fill it;
no metadata), and its two name records, the BRTI's and the file name, are resolved and must hold its name; only those
are rewritten to the damaged texture's name, and the result is recompressed and checked;
is written, the build stops and says for each file why (no known pair, its pair damaged, unreadable or not fitting).
Each is written through a temporary file of its own beside the target; a write that stops part-way stops the build,
and the next build completes the set;
(the 23.3 GB dump also has a third, in Ganon's stage, with no known pair; the build stops at it and says to build
without the switch);
.longshot_romfs_heads, under the unpack's stamp) is its damage record: itis kept after the check in both modes, and in pairs mode written before anything is replaced, with the files a
default-mode check of the same unpack replaced (its
longshot_damaged_files.txt) and those the probe of the two knownnames finds added. A recorded file is replaced whatever its bytes now say; the probe uses the game's dictionary, and
a healthy original is never replaced. A change of mode checks the recorded files again (
.longshot_romfs_checked'sversion line) without opening every file; a new unpack removes the old check mark and note only once its package was
read and checked, so a failed read keeps them with the old romfs.
Unset, the builder does what it did, except that it keeps
.longshot_romfs_headsafter its check instead of deletingit (so a change of mode reads it);
nup.stand_in_damagedis untouched, and its tests clear the switch.nx2/nup_repair_fixture.pymakes authored zstd frames, single-texture BNTX files, a dictionary SARC and a RESTBL: nogame bytes.
texture_repair.pyis adapted from ProjectReLink'stools/dump_repair.py(GPL-3.0, 90801cdf).From the local branch
exp/x86-asm-sync5259070a, which replaced the similar-file search outright; here it sits besideit behind the switch. Not in this PR, from the same commit: the float helpers (another PR), the hot pieces' compile
safeguard and scheduling cost (not ported).
Read first:
repair_texture_pairsinnx2/nup.py;stand_in_damaged_filesinnx2/make_game.py;nx2/texture_repair.py;nx2/test_nup_texture_pairs.py(new).Commits: the feature (
9a94469c); review fixes as bug fixes, each failing game-free tests then the fix:4c17ee20andeefcb329(first review),df7d58e4and9fdcce1d(second review: the probe's finds recorded, the old marks kept until a newunpack takes over).
Testing
Tested: Windows 11, i7-9700, RTX 4060, Python 3.14.7 (the Builder's); game-free checks only. No build from a dump
with the two textures damaged has run with this change on this PC yet, with the switch set or unset, and no game start
of such a build (Still to test). Each check ran in a hidden console of its own, in one job of the PC's job queue
(fb2-1983-3), on this branch rebased onto
main@9dd41326(head9fdcce1d):df7d58e4(the first review's fix with the second review's tests):python -I -B nx2/test_nup_texture_pairs.pyexit 1, "Ran 19 tests", FAILED (failures=1: the no-note migration case,whose list missed the probe's find);
python -I -B tests/test_make_game_cache.pyexit 1, "Ran 49 tests", FAILED(failures=1, skipped=2: the old check mark and note gone after a failed new unpack).
9fdcce1d:nx2/test_nup_texture_pairs.pyexit 0, "Ran 19 tests in 1.115s", OK;tests/test_make_game_cache.pyexit 0, "Ran 49 tests in 2.549s", OK (skipped=2).
nx2/test_nup_stand_in.pyexit 0, "stand-ins: ok";nx2/test_nup_zstd_heads.pyexit 0, "Ran 8 tests", OK;nx2/test_nup_verify.pyexit 0, PASS;tests/test_damaged_check_progress.pyexit 0, "Ran 4 tests", OK;tests/test_damaged_stand_in_fail_fast.pyexit 0, "Ran 4 tests", OK;python -I -m py_compileof the four changedmodules: exit 0.
a1990170(fb2-1983-2), the first review's tests failed ate7951cff(failures=9, errors=7) andpassed at
a1990170.tests/test_make_game_cache.pywas not run then: it still expected the list deleted after thecheck, which
a1990170changed;df7d58e4updates that expectation.Earlier, at b51ee688 on
main@2e1f6c32(batch 9),tests/run_all.py --strict --changed origin/mainhad 67 passed,14 failed, 2 skipped, the 14 named as failing on
maintoo (one of them,test_pipeline_states.py, had found no Vulkan headers in that run (RX_CPMunset) and compiled nothing on either side; run again with the headers it passes ata1990170and onmain@f1f2dc5b); it was not run again ona1990170or9fdcce1d.Still to test:
.nupdump with the known textures damaged: fresh builds into separate game folders withthe switch unset and with
LONGSHOT_STAND_INS=pairs, expecting the pairs log to give each pair's decoded size andallocation (both real donors fitting the 280-byte allocation overhead) or, with the third texture damaged, the stop
naming it; a game start of a pairs build with the Great Fairy Fountain's floor and the Water Temple's gate drawn; the
check's one-time time on a kept folder after a change of mode (by hand)
python -I -B tests/run_all.py --strict --changed origin/mainon9fdcce1d, expecting no failure thatmaindoes not have (by hand)python3 -I -B nx2/test_nup_texture_pairs.py, expecting it to pass (by hand)Measurements
Nothing to measure: a builder mode that ships off and runs only on a damaged dump (the scan is the existing one; the repair reads two textures and the size table).
Risks
Off (unset), nothing changes but a small list kept in the game folder. Set, a damaged file other than the two known
textures stops the build, with no file replaced, where the default mode would have copied in a similar file: the player
is told why and to build without the switch or dump the game again. A romfs checked before in the default mode is
checked once more when the mode is set, and once more in the default mode when it is unset again (similar files go back
in place of the pairs' textures). A malformed size table, dictionary or donor leaves every file as it was and the build
stops (29 such inputs are tested). Players notice nothing unless the switch is set; a revert is enough.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Possibly related, for context:
Recommendation: Request changes. The documented 23.3 GB dump has a third damaged texture, so pairs mode stops a build the default mode can complete. The repair also accepts some damaged files as healthy and can lose track of older substitutions across builds.
Adds an opt-in builder mode intended to preserve the allocation and embedded names of two replacement textures. It keeps the existing stand-in search as the default for fresh builds. The narrower repair is useful if it works on the affected dump, but it adds a binary parser, persistent migration state, and a potentially minutes-long rescan of an existing install.
@fixwill do (16)nx2/texture_repair.py:16: Add a reviewed repair for the third texture if one exists, or scope the mode and its documentation to a verified two-damage variant.nx2/make_game.py:726: A fresh extraction replaces the romfs and clears the checked marker but leaves the old damage note in place. (more on the line)nx2/make_game.py:753: The human-readable note is the only record of old substitutions after HEADS is deleted. (more on the line)nx2/nup.py:863: A matching BRTI name overrides the extraction's damage verdict. (more on the line)nx2/nup.py:865: A failed probe is treated as permission to replace a known target. (more on the line)nx2/texture_repair.py:156: The BNTX filename field only adds an address to an allowlist. (more on the line)nx2/texture_repair.py:127: The payload check accepts any nonzero dimensions and image size if the declared image range lies inside the file. (more on the line)nx2/nup.py:879: The fixed .part file is shared by concurrent builders targeting the same output with different --work folders. (more on the line)nx2/texture_repair.py:84: The 256 MiB decompression limit admits over 33 million hash entries. (more on the line)nx2/nup.py:882: The donor comes back under the canonical extraction path, but the displayed donor path is rebased against the lexical root after replacement. (more on the line)nx2/nup.py:876: The repair discards every expected error and tells the player to redump. (more on the line)nx2/test_nup_texture_pairs.py:67: The positive test constructs its expected output with the production rename function, and the fixture gets its donor choices from the production map. (more on the line)nx2/test_nup_stand_in.py:99: Existing builder tests now inherit the new switch. With LONGSHOT_STAND_INS=pairs exported, this test's fake nup module has no repair_texture_pairs, and the HEADS tests also enter the wrong mode. Explicitly clear the switch in tests that exercise the default, including the HEADS, fail-fast, and cache cases.docs/dev/reference/build.md:54: Unsetting the switch changes the next search, but it cannot undo a published pair repair. (more on the line)nx2/make_game.py:744: Retain the unpack-scoped damage and repair record so a mode transition can recheck affected paths without reopening the whole romfs.For the author (3) (a run, a measurement or a decision)
nx2/texture_repair.py:16: The documented 23.3 GB dump contains a third damaged texture in Ganon's Stage, recorded in the original stand-in commitbd5270246and the legacy frame fix #2391. (more on the line)nx2/make_game.py:744: Changing modes on an already checked output whose HEADS was deleted causes a full walk and about 100,000 file opens. (more on the line)Checks: Still running:
CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request),Website / Website on Cloudflare (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven. The author reports 10 game-free tests passing in 0.805 s on Windows at b51ee688. The base comparison stopped at a missing fixture import. There is no build or game run from the affected dump in either mode, and no Linux result or build-time comparison.
Other systems: Fresh extraction, cached updates, and --romfs-from builds can all reach this path. Both the Windows Builder package and Linux AppImage carry the module and already have zstandard. The output is shared across GPU vendors and Steam Deck. In-game rendering and the interaction with mods that replace textures or resource-size entries remain unverified. There are no changes to the renderer or host ABI, saves, audio, or input.
Switches: LONGSHOT_STAND_INS=pairs selects pair repair. Unset and all other values select the existing similar-file search. Keeping that default avoids imposing the experimental build refusal on existing players. The build-switch table has a row. The runtime switch table excludes build-only settings. The switch is one-way for files already rewritten, which prevents a clean A/B comparison in the same output folder.
Reviewed:
0a83513e24against6e1521a70a: Read the complete seven-file diff, commit message, new repair module, fixture and tests, updated progress test, and build reference. Read the changed functions in make_game.py and nup.py and traced their extraction, update, hard-link, lock, path, and stamp code. Read AGENTS.md, CONTRIBUTING.md, the PR template and developer standards. The checkout has no docs/review-checklist.md or adjacent Tooling checkout. Reviewed the supplied run summaries and CI status and consulted the existing BNTX, SARC, RESTBL and packaging code. No translations or generated files changed.; nothing run (a code review).Checked and fine:
Testing: Read-only review. No code was executed. The author reports Windows passes for the 10 pair tests, the default stand-in test, and four progress tests on the earlier rebase. The changed suite reports 67 passed, 14 failed on both revisions, and two skipped. Head CI is pending. Full lint and test results, Linux execution, and builds and game checks from the affected dump are outstanding.
Overlaps and merge order: No overlapping open pull request is identified in the supplied context or checkout. The prior stand-in work and legacy frame fix #2391 are already on the base.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -51,6 +51,7 @@ Read by the build, not the game, so they are not in `switches.def`.| `LONGSHOT_LIFT_STORE=<dir>` | Translations shared by every work folder with the same lift key || `LONGSHOT_NO_PREPARE=1` | Skip the shader preparation || `LONGSHOT_ROMFS_IN_PLACE=1` (or `--romfs-in-place`) | With `--romfs` and `--exefs`: read the romfs where it is; the game folder has no `romfs`, so start the game with `--romfs <folder>`. Passed over by a `.nup` or `--update` build (the switch with them is refused). `dev.py packs` sets it for packs || `LONGSHOT_STAND_INS=pairs` | An incomplete dump's damaged textures (a `.nup` the builder unpacked): only the two textures known to be damaged are replaced, each by its pair's texture after the game's resource size table shows both the same size, with the damaged one's name written into it (`nx2/texture_repair.py`); any other damaged file stops the build. Unset: a similar file of the same kind is copied in, as before. Off unless set: an experiment, not yet run in the game. A change of mode checks the game files again |L14 (minor): Unsetting the switch changes the next search, but it cannot undo a published pair repair. The default scan sees the new zstd frame as healthy. Document that a fresh extraction or separate output is needed for a clean comparison or rollback, and describe the mode's scope once the third known damaged texture is accounted for.
@ -733,3 +752,2 @@done = nup.stand_in_damaged(romfs, damaged=[os.path.normpath(os.path.join(romfs, rel)) for rel in known])note = os.path.join(out, DAMAGED_NOTE)done = stand_in(romfs, damaged=[os.path.normpath(os.path.join(romfs, rel)) for rel in known])if not done:L3: The human-readable note is the only record of old substitutions after HEADS is deleted. For example, default mode substitutes an unknown texture, pairs mode rejects it, then default mode sees a valid zstd prefix and deletes the note. The next pairs run accepts the same substitution. An unreadable note is also silently treated as empty. Keep an atomic, unpack-scoped record of substitutions and their mode, independent of the warning text. Use it on every mode transition and retain it through failures.
@ -725,13 +744,12 @@ def stand_in_damaged_files(out, romfs):if known is None:L15 (minor): Changing modes on an already checked output whose HEADS was deleted causes a full walk and about 100,000 file opens. The builder's own comment says that can take minutes with antivirus. Time the read phase on a kept output and report the one-time cost alongside fresh build results.
@ -716,1 +724,3 @@heads = os.path.join(out, HEADS)previous = []version, separator, prior_stamp = prior.partition("\n")if pairs and (not prior or prior == stamp or (separator and version.removesuffix("\r") in ("resolved-v1", "resolved-v2")L2: A fresh extraction replaces the romfs and clears the checked marker but leaves the old damage note in place. Pairs mode interprets that missing marker as permission to reuse the note. If the old dump had a substitution for a file absent from the new complete dump, the new build reports that stale path as unresolved and stops. Bind repair metadata to the unpack and clear or replace it when publishing a new romfs.
@ -826,0 +860,4 @@if rel.endswith(".bntx.zs"):try:data = texture_repair.read_frame(target, texture_repair.MAX_TEXTURE)if texture_repair.texture_name(data)[0] == name.encode("utf-8"):L4: A matching BRTI name overrides the extraction's damage verdict. HEADS can contain a zstd member whose package digest failed even though it still decompresses and carries its original name. That file is skipped and can be stamped resolved, including for a texture outside the supported pairs. Keep fresh extraction failures authoritative and use a recorded repair identity to recognize bytes successfully written by an earlier attempt.
@ -826,0 +862,4 @@data = texture_repair.read_frame(target, texture_repair.MAX_TEXTURE)if texture_repair.texture_name(data)[0] == name.encode("utf-8"):continue # (whole and its own: an original, or a repair done before while another file stays unresolved)except errors:L5: A failed probe is treated as permission to replace a known target. A healthy dictionary-compressed BNTX fails the dictionary-free read, and an I/O failure takes the same path. If the donor is readable, the builder can overwrite the original. Use the archive dictionary where required, distinguish read or format failures from confirmed damage, and require positive evidence of damage or a prior substitution before writing.
@ -826,0 +873,4 @@table = texture_repair.size_table(romfs, path_for)if table is not None:other, data = texture_repair.replacement(romfs, name, table, path_for)except errors:L11 (minor): The repair discards every expected error and tells the player to redump. A donor held by another program and a donor with a mismatched allocation produce the same message. Preserve a short per-file reason and distinguish access failures from invalid or unsupported data.
@ -826,0 +876,4 @@except errors:other = Noneif other:with open(target + ".part", "wb") as f:L8: The fixed .part file is shared by concurrent builders targeting the same output with different --work folders. The existing lock covers only --work. On Linux one builder can rename the file while the other is still writing it. A failed write or rename also leaves the temporary behind. Serialize romfs and marker changes per output, stage in a unique file beside the target, and clean it up on failure.
@ -826,0 +879,4 @@with open(target + ".part", "wb") as f:f.write(data)os.replace(target + ".part", target)other = os.path.normpath(os.path.join(romfs, os.path.relpath(other, long_path(root))))L10 (minor): The donor comes back under the canonical extraction path, but the displayed donor path is rebased against the lexical root after replacement. On Windows, an output below a junction into another drive can make relpath raise after the repair has already been published. Use the known romfs-relative donor path for reporting and resolve it before publishing.
@ -0,0 +64,4 @@data = zstandard.ZstdDecompressor().decompress(Path(path).read_bytes())self.assertEqual(consumer_name(data), bad)self.assertEqual(len(data), len(fixture.bntx(good)))self.assertEqual(data, texture_repair.rename_texture(fixture.bntx(good), good, bad))L12 (minor): The positive test constructs its expected output with the production rename function, and the fixture gets its donor choices from the production map. A wrong donor choice can pass. Author the expected mappings independently and compare the two name records, unchanged bytes, and image data against that fixture. In the bad-archive-bounds case, keep the dictionary name valid so the wrong name cannot mask a missing bounds check.
@ -0,0 +13,4 @@import zstandardPAIRS = {L1: The documented 23.3 GB dump contains a third damaged texture in Ganon's Stage, recorded in the original stand-in commit
bd5270246and the legacy frame fix #2391. A fresh build from that dump repairs the two listed textures and then stops. Establish a verified treatment for the third texture or narrow the feature to a demonstrated dump variant with only these two failures. Update the description and build docs with that scope.@ -0,0 +81,4 @@version, name_size, hashed, named = struct.unpack_from("<IIII", table, 6)if version != 1 or not 0 < name_size <= 4096 or len(table) != 22 + 8 * hashed + (name_size + 4) * named:raise ValueError("invalid RESTBL bounds")sizes = {}L9: The 256 MiB decompression limit admits over 33 million hash entries. Materializing them as Python integers and dictionary entries can consume several GiB on a damaged dump before the builder reaches its normal error path. Keep only the four relevant keys while validating the table, and apply the same bounded approach to named entries.
@ -0,0 +124,4 @@dimensions = struct.unpack_from("<IIII", data, brti + 0x24)image_size = struct.unpack_from("<I", data, brti + 0x50)[0]mip_table = struct.unpack_from("<Q", data, brti + 0x70)[0]if not mip_count or not all(dimensions) or not image_size or not 0x40 <= mip_table <= len(data) - 8 * mip_count:L7: The payload check accepts any nonzero dimensions and image size if the declared image range lies inside the file. A donor with large dimensions but only one block of image data can pass, as can mip offsets into metadata. The allocation comparison only checks container length. Validate the expected format and layout and check each mip's required byte range against the image region before accepting a donor.
@ -0,0 +153,4 @@raise ValueError("texture names do not match the replacement pair")# BRTI's texture name and the container's filename are the only supported name records. Even an apparent# length-prefixed match elsewhere (for example in pixel data) must not be silently mutated.allowed = {at, struct.unpack_from("<I", data, 0x10)[0]}L6: The BNTX filename field only adds an address to an allowlist. A donor with a valid BRTI name and a missing, unrelated, or out-of-range filename record still passes. The filename address is also never checked against the mip table or image bytes. Resolve and validate both records directly, require the expected donor name in each, and verify the exact spans before and after writing.
Recommendation: Request changes. Pair mode puts new textures into the existing game folder before it knows whether an unresolved candidate will stop the build. Validate the complete repair set before replacing any textures.
The patch also needs to resolve the earlier exact-head review's findings about the third damaged texture, the damage record, and the BNTX payload and name checks. Earlier review
Does it help? Unproven on an affected dump. The author reports ten authored Windows fixtures passing at
b51ee688and an unchanged patch after rebase. The base-side run stopped at a missing fixture import; it establishes neither a behavioral comparison nor compatibility with an affected dump. The updated documentation describes the opt-in mode. The Testing section still calls for the Windows player rebuild and manual scene verification, plus the Linux fixture.Reviewed
0a83513e240b5978dcda10351320b7a8e47b8605against actual merge base6e1521a70a32603ed60a8263e4072625885669b7. Read the seven-file patch, the builder's unpack, update, stamp and hard-link paths, and the existing texture and resource-size readers. No PR code was run.Blocking
nx2/nup.py:881publishes a valid repair immediately. A later unknown or invalid candidate is still returned as unresolved andstand_in_damaged_filesstops the build. In an update, the previous executable can still use that folder, so the failed build leaves it using a partly repaired ROMFS.test_an_unresolved_retry_keeps_the_heads_and_never_rewrites_a_repairexplicitly expects both replacements on the failing first attempt. Unsetting the switch will not restore those files. Validate and stage the complete set before replacing any files, then assert that unresolved input leaves every target byte-for-byte unchanged. Handle a publication failure so a partly committed set cannot silently become the next build's input.Checked and fine
LONGSHOT_STAND_INSunset use the previous repair function. The opt-in setting is documented as a build input. On--romfs-from, replacing a target entry protects the donor's original hard link from a write-through.Testing and merge eligibility
CI was pending on the reviewed head. After fixing the source blockers, test a fresh build in both modes with independent output trees on a PC that already has the affected dump. Verify the resolved-file log and inspect both locations in game. Keep the dump, donors and repaired files on that PC. The current author request is a manual builder test, and an ordinary Beanstalk runtime comparison cannot verify the
.nupextraction and repair. Keep the PR WIP pending source fixes and the affected-input result.Overlaps
#546 covers a builder path for extracted-folder input. #1943 covers fallback at runtime. Keep their scopes explicit if either proceeds, since a runtime fallback can mask a failed builder repair in gameplay.
@ -826,0 +878,4 @@if other:with open(target + ".part", "wb") as f:f.write(data)os.replace(target + ".part", target)This commits a texture before a later unresolved file can stop the build. The existing game folder then keeps the experimental substitutions from the failed update, and the retry test currently expects those writes. Validate and stage every candidate before publishing repairs. Test that an unresolved candidate leaves both would-be targets unchanged.
Recommendation: Request changes. Pair mode puts new textures into the existing game folder before it knows whether an unresolved candidate will stop the build. Validate the complete repair set before replacing any textures.
The patch also needs to resolve the earlier exact-head review's findings about the third damaged texture, the damage record, and the BNTX payload and name checks. Earlier review
Does it help? Unproven on an affected dump. The author reports ten authored Windows fixtures passing at
b51ee688and an unchanged patch after rebase. The base-side run stopped at a missing fixture import; it establishes neither a behavioral comparison nor compatibility with an affected dump. The updated documentation describes the opt-in mode. The Testing section still calls for the Windows player rebuild and manual scene verification, plus the Linux fixture.Reviewed
0a83513e240b5978dcda10351320b7a8e47b8605against actual merge base6e1521a70a32603ed60a8263e4072625885669b7. Read the seven-file patch, the builder's unpack, update, stamp and hard-link paths, and the existing texture and resource-size readers. No PR code was run.Blocking
nx2/nup.py:881publishes a valid repair immediately. A later unknown or invalid candidate is still returned as unresolved andstand_in_damaged_filesstops the build. In an update, the previous executable can still use that folder, so the failed build leaves it using a partly repaired ROMFS.test_an_unresolved_retry_keeps_the_heads_and_never_rewrites_a_repairexplicitly expects both replacements on the failing first attempt. Unsetting the switch will not restore those files. Validate and stage the complete set before replacing any files, then assert that unresolved input leaves every target byte-for-byte unchanged. Handle a publication failure so a partly committed set cannot silently become the next build's input.Checked and fine
LONGSHOT_STAND_INSunset use the previous repair function. The opt-in setting is documented as a build input. On--romfs-from, replacing a target entry protects the donor's original hard link from a write-through.Testing and merge eligibility
CI was pending on the reviewed head. After fixing the source blockers, test a fresh build in both modes with independent output trees on a PC that already has the affected dump. Verify the resolved-file log and inspect both locations in game. Keep the dump, donors and repaired files on that PC. The current author request is a manual builder test, and an ordinary Beanstalk runtime comparison cannot verify the
.nupextraction and repair. Keep the PR WIP pending source fixes and the affected-input result.Overlaps
#546 covers a builder path for extracted-folder input. #1943 covers fallback at runtime. Keep their scopes explicit if either proceeds, since a runtime fallback can mask a failed builder repair in gameplay.
@ -826,0 +878,4 @@if other:with open(target + ".part", "wb") as f:f.write(data)os.replace(target + ".part", target)This commits a texture before a later unresolved file can stop the build. The existing game folder then keeps the experimental substitutions from the failed update, and the retry test currently expects those writes. Validate and stage every candidate before publishing repairs. Test that an unresolved candidate leaves both would-be targets unchanged.
0a83513e24a1990170a1Pushed
a1990170(rebased ontomain@f1f2dc5b; the feature commit unchanged,git range-diff"="), answering futondestroyer's review 62700/62704 and BeanBot's 62612: the failing game-free tests first (e7951cff), then the fix (a1990170). Pair mode now writes nothing while any file stays unresolved and completes a set a failed write left part-way; the unpack's damage list is kept as its record; the donor's layout and both name records are checked; pairs mode's scope (exactly the two known textures) is stated. The description's Summary, Testing, Still to test and Risks are updated to match.Re #1983 (comment): fixed in
a1990170, with the failing game-free tests first ine7951cff(they fail on the PR's first commit and pass ata1990170, same queue job). Pair mode now makes and checks every replacement in memory and writes none while any file stays unresolved, so a failed build leaves the game folder's textures byte for byte as they were; a failure while publishing removes its temporary file, stops the build without marking the check done, and the next build completes the set from the kept, stamped damage list (test_a_replacement_that_stops_part_way_is_completed_next_time). BeanBot's 62612 findings (third texture, damage record, BNTX payload and name checks) are fixed or scoped in the same commit and answered on each line; the build from an affected dump in both modes, with both places looked at in game, is pending on a PC that has that dump.Re #1983 (comment) (the same review as 62700): fixed in
a1990170, test first ine7951cff; the details are in my reply to 62700. Pair mode writes nothing while any candidate stays unresolved, and a publication that stops part-way is completed by the next build.Re #1983 (comment): fixed in
a1990170.nup.repair_texture_pairsstages every replacement in memory and returns before writing when any candidate is unresolved;test_an_unresolved_candidate_leaves_both_targets_unchangedand the retry test (nowtest_an_unresolved_retry_keeps_the_heads_and_writes_nothing) assert no texture write and both would-be targets byte for byte unchanged. Both fail on the PR's first commit (e7951cff) and pass ata1990170.Re #1983 (comment) (the same point as 62699): fixed in
a1990170, test first ine7951cff. An unresolved candidate now leaves both would-be targets unchanged, and the retry test expects no writes on either attempt.Re #1983 (comment) (L1): scoped in
a1990170. Pairs mode covers exactly the two known textures: the third (Ganon's stage) has no known pair, so a build in this mode stops naming it, says nothing was replaced and to build withoutLONGSHOT_STAND_INS=pairsfor the default similar file (test_a_third_damaged_texture_stops_the_build_and_says_what_to_do); the build reference and the description say so in one sentence. Whether a pair with an equal allocation exists for the third texture is to be checked on a PC with the affected dump: pending.Re #1983 (comment) (L2): fixed in
a1990170. The note is read only after a default-mode check of this same unpack (its stamp in.longshot_romfs_checked), never when the mark is missing, and a fresh extraction now removes the note together with the mark; the damage list.longshot_romfs_headscarries the unpack's stamp and is replaced by the new unpack.test_a_note_from_another_unpack_is_not_takenfails ate7951cffand passes ata1990170.Re #1983 (comment) (L3): fixed in
a1990170..longshot_romfs_heads(the unpack's stamped damage list) now stays after the check in both modes and, in pairs mode, is written atomically before any replacement with the note's paths added, so default, pairs, default, pairs keeps the substituted file a damaged file (test_changes_of_mode_keep_the_damage_record); a note that can't be read stops the build instead of reading as empty. The mode is not stored per path: a change of mode re-checks every recorded path.Re #1983 (comment) (L4): fixed in
a1990170. A recorded path is replaced whatever its bytes decode to; an earlier repair is recognised as the identical replacement (the repair is deterministic) and not written again.test_a_recorded_texture_is_repaired_whatever_its_bytes_say.Re #1983 (comment) (L5): fixed in
a1990170. Frames are decoded with the game'szs.zsdicwhere they name it (its id checked); a probe of a known name that can't be read or decoded is no evidence and leaves the file, and only a valid texture under another texture's name counts as an earlier substitution.test_a_healthy_texture_made_with_the_game_dictionary_is_kept(a frame that needs the dictionary).Re #1983 (comment) (L6): fixed in
a1990170.texture_repair.layoutresolves the header's file-name record and the BRTI's name record directly, each bounded and outside the header, BRTI, mip table and image data; both must hold the donor's name (the file name with or without.bntx), the name may occur nowhere else, and both are read back after the change. New malformed inputs: no, out-of-range and unrelated file-name records.Re #1983 (comment) (L7): fixed in
a1990170. The donor's format must be one of the listed block formats (BC1-BC7, ASTC 4x4-12x12, RGBA8/BGRA8), each mip level must hold at least its blocks before the next level or the end of the image data, the levels times the array layers must fit the image size, and the image data must not overlap the header, BRTI or mip table. This is a lower bound (block-linear padding is not modelled); new inputs: 4096x4096 in one block, image data on the BRTI, a second mip past the data, an unknown format.Re #1983 (comment) (L8): fixed in
a1990170for this mode's files: each replacement goes through amkstempfile of its own beside the target, removed when the write or the rename fails (test_a_replacement_that_stops_part_way_is_completed_next_timechecks none is left). A lock per output folder across builders with different--workfolders is not added: onmainthe extraction, the marks and the default mode's stand-ins already share an output unlocked, so that is left for a separate change.Re #1983 (comment) (L9): fixed in
a1990170.texture_repair.allocationswalks the hashed entries withstruct.iter_unpack, still checking their order, and keeps only the four pair entries; named entries likewise. The decoded table itself stays bounded at 256 MiB.Re #1983 (comment) (L10): fixed in
a1990170. The donor path reported is the romfs-relative one, built from the pair's name before anything is written; nothing is rebased against the canonical path any more.Re #1983 (comment) (L11): fixed in
a1990170. Each unresolved file gets a short reason in the log and the note: no known pair, its pair is damaged too, its pair could not be read (with the OS's message), or its pair does not fit (with the check that refused it).test_each_unresolved_file_says_why.Re #1983 (comment) (L12): fixed in
e7951cff. The fixture writes its two pairs out itself (a test comparestexture_repair.PAIRSwith them), the positive test expects the fixture's own texture under the damaged name byte for byte (both name records and every other byte), and bad-archive-bounds keeps thezs.zsdicname so only the bounds can refuse it.Re #1983 (comment) (L14): fixed in
a1990170. The damage list is now kept, so unsetting the switch re-checks the recorded paths and puts similar files back in place of the pairs' textures;docs/dev/reference/build.mdsays so, says a clean comparison needs a game folder per mode, and states that pairs mode covers exactly the two known textures (a dump with another damaged one cannot be built in this mode yet).Re #1983 (comment) (L15): the record is kept in
a1990170, so a change of mode reads the unpack's stamped list and opens no other file (test_changes_of_mode_keep_the_damage_recordfails if the scan runs). The one-time walk remains only for a folder whose list an older builder deleted after its check; timing it on a kept output needs the affected dump and is pending.Re #1983 (comment), L13: fixed in
a1990170.nx2/test_nup_stand_in.py,nx2/test_nup_zstd_heads.pyandtests/test_damaged_stand_in_fail_fast.py(the default mode's, HEADS and fail-fast cases) clearLONGSHOT_STAND_INSbefore they run; the pairs tests set it per test, and the default-mode test there clears it explicitly.Re #1983 (comment), L16: the edit part is in
a1990170: each pair is logged with its decoded size, allocation and time, each file's verdict (with its reason) goes to the log and the note, and the check's total time is said. The author-side runs (both real donors' compression and the 280-byte allocation overhead, a fresh build in each mode into separate game folders, both places looked at in game, Windows and Linux, build time) need the affected dump: pending on the PC that has it, and Measurements will get them.Tested corrected: an earlier run on this PC had no Vulkan headers for the renderer checks (
RX_CPMunset), so the ones it listed as also failing onmainhad compiled nothing on either side. They ran again with the headers ata1990170and onmain(queue job fb2-vk-1); the description now gives those results. No check fails at this head alone; nothing in the branch changed.Recommendation: Request changes. The new validation pass fixes writes on unresolved input. Two migration paths can still lose the damage record, which prevents switching back to the previous mode and can hide an unresolved texture.
Reviewed
a1990170a14bc8abe50813b37e0724563676fca4against actual merge basef1f2dc5bd2821cef061798ba40a043d281ac1707, including the fixes since the previous review. No PR code was run.Blocking
nx2/nup.py:884discovers earlier wrong-name substitutes afternx2/make_game.py:763has persisted the damage list. For a legacy build with a missing damage note and one discoverable wrong-name substitute, the scan sees healthy Zstd headers and records an empty list. The probe then repairs the substitute under the correct name. When the switch is unset, default mode trusts the empty list, skips the target and removes the note. This silently preserves the experimental texture despite the documented switch-back behavior. If a later write fails, the corrected embedded name also prevents the next probe from rediscovering that completed path. Persist the effective candidate set before the first replacement. Extend the existing no-note migration fixture through switching back to default and check both the durable list and the restored file.nx2/make_game.py:3162removes the old install's damage note and check marker before extraction starts. The existing ROMFS is kept when the new package fails validation or extraction. A legacy install with no.longshot_romfs_headsthen loses the only list of its original stand-ins while still using them. On the next pairs-mode update, the scan can miss a previously substituted third texture because the replacement has a valid Zstd header and the probe covers only the two known names. Retire the old markers when the new unpack takes ownership of the output. Add a fixture where a failed extraction preserves the old ROMFS and both of its markers.Checked and fine
Does it help? Unproven on the affected dump. The author reports 19 Windows fixtures passing at this head after failures on the test-only commit, plus passing default stand-in, Zstd-head, progress and fail-fast fixtures. Full changed checks, Linux fixtures and the affected Windows build and scene checks remain open. CI was pending in the current-head snapshot. Author evidence
Keep
LONGSHOT_STAND_INS=pairsopt-in and the PR WIP. The author's manual Windows test requires an affected local.nupin separate output folders for the two modes, and scene verification only after the pairs build succeeds. The farm page has no plan or result. Its documented build variants cannot set this build-time flag, and timed builds use the base's ROMFS. Keep original and repaired assets on the tester's PC. Related builder and runtime fallback work remains separately scoped. Farm page · Builder folder input · Runtime fallback@ -3106,3 +3161,2 @@shutil.rmtree(game, ignore_errors=True)with contextlib.suppress(FileNotFoundError): # (a new unpack is checked again)os.remove(os.path.join(out, ".longshot_romfs_checked"))for name in (".longshot_romfs_checked", DAMAGED_NOTE): # (a new unpack is checked again; the old one's note goes)The old ROMFS survives a failed extraction, but these markers are erased before reading the new package. A legacy install with no HEADS then loses its only record of earlier stand-ins. A subsequent pairs run may miss an unsupported third texture hidden behind a valid Zstd header. Keep the old markers until the new unpack succeeds and owns the output.
@ -826,0 +881,4 @@try:data = texture_repair.read_frame(path_for(rel), texture_repair.MAX_TEXTURE, dictionary())if texture_repair.layout(data)[0] != name.encode("utf-8"):candidates.append(given(rel)) # (a whole texture under another's name: an earlier check's similar file)A recovered wrong-name substitute is added here after the builder has saved the damage list. With a missing legacy note, the repair can succeed while HEADS stays empty. Switching back to default then skips this target and keeps the experimental texture. Record probe-discovered paths before committing repairs and cover this with the no-note migration fixture.
a1990170a19fdcce1d36Pushed
9fdcce1d(rebased ontomain@9dd41326; the earlier commits unchanged,git range-diff"=" for all three), answering futondestroyer's second review 65448: failing game-free tests first (df7d58e4), then the fix (9fdcce1d). The probe's finds now go into the damage list before anything is replaced, and a new unpack removes the old romfs's check mark and note only once it takes the folder over.tests/test_make_game_cache.pynow runs with the PR's checks; it had not run ata1990170, where its expectation that the list is deleted after the check was out of date. Summary, Commits, Tested and Still to test are updated.Re #1983 (comment): fixed in
9fdcce1d, test first indf7d58e4.nup.repair_texture_pairsnow hands every path it takes as damaged, the probe's finds included, to arecordcallback before the first write, andmake_game.pywrites.longshot_romfs_headsfrom it. The no-note migration case oftest_an_earlier_similar_file_is_repaired_and_healthy_originals_keptnow goes on through unsetting the switch: the list must name the repaired path, and the default mode puts its similar file back there with the list kept; it fails atdf7d58e4and passes at9fdcce1d.Re #1983 (comment): fixed in
9fdcce1d, test first indf7d58e4. The check mark and the damage note are now removed only once the new package was read and checked, just before the old romfs is replaced; until then a failed read keeps them with the old romfs.tests/test_make_game_cache.pytest_a_failed_new_unpack_keeps_the_old_romfs_check_and_notecovers a failed extraction, an unsupported build and a package with no romfs; it fails atdf7d58e4and passes at9fdcce1d.Re #1983 (comment): both migration paths are fixed in
9fdcce1d, with their failing tests first indf7d58e4, onmain@9dd41326; replies on 65446 and 65447 give the details. The probe's finds now go into.longshot_romfs_headsbefore anything is replaced, and a new unpack retires the old romfs's check mark and note only when it takes the folder over. The build from the affected dump in both modes, with both places looked at in game, is still pending on a PC that has the dump.Recommendation: Wait for the affected-dump build and remaining checks. The two source blockers from my previous review are resolved at
9fdcce1d. The PR remains WIP. A repository admin can dismiss my earlier request for changes.record(candidates)after the probe adds any wrong-name substitutes and before the first texture write. The builder saves that full set under the unpack stamp. The no-note migration fixture continues through unsetting the switch and verifies both the durable damage list and the default-mode substitute.Reviewed
9fdcce1d3636741e255f5f0e66b3289c17688f5aagainst actual merge base9dd41326e71094d8b774459dac5154eb5a3306d9. The range-diff confirms the previous three commits are unchanged. I read the two new commits, their four changed files, and the surrounding call paths. No PR code was run.Does it help? The author reports that the new migration and cache cases each fail at the test-only commit
df7d58e4and pass at this head. The Windows pair suite reports 19 tests passing. The cache suite reports 49 cases with 2 skips and no failures. Real donor compatibility remains unproven. Author's updateKeep
LONGSHOT_STAND_INS=pairsopt-in. The remaining manual check needs an affected.nupalready on the tester's PC, separate fresh outputs for both modes, and a successful pairs build before inspecting both textures in game. Record build outcome and time, confirm the unsupported third texture stops cleanly where present, and finish the Linux fixture, current-head checks and CI. The regular Farm path uses a prepared ROMFS and offers no build variant for this flag, so it cannot replace the manual test. Farm status@futondestroyer, re #1983 (comment): thank you. The affected-dump checks are with the author's runs on this PC and have not run yet: fresh builds from the affected
.nupinto separate output folders with the switch unset and withpairs, the build outcome and time of each, both textures looked at in game after a successful pairs build, and the search for a pair for the third texture (which stops pairs mode cleanly as it is). The Linux fixture cannot run on this PC; the forge's CI on9fdcce1dhas not run yet. Results go into the description when they are in.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.