Builder: the damaged-texture check uses what the extraction saw (no ~100,000 files reopened) #1885

Merged

NotRexed merged 2 commits from fix/unpack-zstd-heads into main

2026-11-10 20:17:10 +00:00

NotRexed commented

2026-11-10 19:21:23 +00:00

(Migrated from github.com)

For #1876 and #1870 (closed into it). This should remove the long stall they hit, but I haven't reproduced their machine, so the issue stays open until a reporter confirms.

Problem: since v2.0.12, the first build after an unpack opens every romfs .zs file, about 100,000 of them, to find the few a 23.3 GB dump has damaged (#1002). Under a virus scanner every open is scanned, so the worker sits at 0% CPU for a long time:

after main: ZBIC -> plain NSO;
or, on an update, at "Reading your game files".

Players read that as a freeze. v2.0.11, which didn't have the check, built fine. Main's #6ad9f1cd now shows the check's progress, but it still opens every file.

Fix:

nx2/nup.py's extraction keeps each romfs .zs member's first 4 bytes as it writes them (HeadSink). When the whole romfs was walked, it leaves the list of the files that don't start as zstd beside it (romfs_zstd_heads.txt). A re-run opens only the files left by the run before.
nx2/make_game.py keeps that list in the game folder under the unpack's stamp (.hookshot_romfs_heads). The stand-in step takes it instead of reopening every file, then removes it.
A folder without the list, or with a list from another unpack, is checked as before.

Tests: new nx2/test_nup_zstd_heads.py checks that:

the damaged file is listed, including when written in pieces through a memoryview;
a re-run lists it again;
an exefs-only extraction leaves no list;
the build takes the list without opening anything (damaged_game_files patched to fail) and still puts the stand-in in;
a list from another unpack is ignored.

The nup and stand-in tests (test_nup_stand_in, test_damaged_check_progress, test_nup_paths, test_nup_codec, test_nup_read_name, test_missing_nup) pass. test_nup_out_of_space fails the same way without this change; #1793 fixes it.

🤖 Generated with Claude Code

For #1876 and #1870 (closed into it). This should remove the long stall they hit, but I haven't reproduced their machine, so the issue stays open until a reporter confirms. **Problem:** since v2.0.12, the first build after an unpack opens every romfs `.zs` file, about 100,000 of them, to find the few a 23.3 GB dump has damaged (#1002). Under a virus scanner every open is scanned, so the worker sits at 0% CPU for a long time: - after `main: ZBIC -> plain NSO`; - or, on an update, at "Reading your game files". Players read that as a freeze. v2.0.11, which didn't have the check, built fine. Main's #6ad9f1cd now shows the check's progress, but it still opens every file. **Fix:** - `nx2/nup.py`'s extraction keeps each romfs `.zs` member's first 4 bytes as it writes them (`HeadSink`). When the whole romfs was walked, it leaves the list of the files that don't start as zstd beside it (`romfs_zstd_heads.txt`). A re-run opens only the files left by the run before. - `nx2/make_game.py` keeps that list in the game folder under the unpack's stamp (`.hookshot_romfs_heads`). The stand-in step takes it instead of reopening every file, then removes it. - A folder without the list, or with a list from another unpack, is checked as before. **Tests:** new `nx2/test_nup_zstd_heads.py` checks that: - the damaged file is listed, including when written in pieces through a memoryview; - a re-run lists it again; - an exefs-only extraction leaves no list; - the build takes the list without opening anything (`damaged_game_files` patched to fail) and still puts the stand-in in; - a list from another unpack is ignored. The nup and stand-in tests (`test_nup_stand_in`, `test_damaged_check_progress`, `test_nup_paths`, `test_nup_codec`, `test_nup_read_name`, `test_missing_nup`) pass. `test_nup_out_of_space` fails the same way without this change; #1793 fixes it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)