fix(builder): a .nup name outside Windows' code page no longer stops the build (#2246) #2253

Merged

bronze-ff merged 3 commits from fix/romfs-stamp-utf8 into main

2026-11-11 16:36:47 +00:00

bronze-ff commented

2026-11-11 11:55:52 +00:00

(Migrated from github.com)

Fixes #2246.

The problem

The build stops with

UnicodeEncodeError: 'charmap' codec can't encode character '⁚' in position 46: character maps to <undefined>

after the whole .nup is extracted. The reporter's file is The Legend of Zelda™⁚ Ocarina of Time Remake.nup.

make_game.py writes the .nup's file name into the .hookshot_romfs stamp (unpacked by this builder: <name> <size>), and opened that file with the default encoding, which is the code page on Windows. U+205A is not in cp1252. The prefix is 26 characters and ⁚ is the 21st character of the name, which gives position 46 exactly. The same line reproduces the error on a cp1252 Windows:

>>> open(p, "w").write("unpacked by this builder: The Legend of Zelda™⁚ Ocarina of Time Remake.nup 1\n")
'charmap' codec can't encode character '⁚' in position 46: character maps to <undefined>

The change

.hookshot_romfs and its .hookshot_romfs_checked copy are written as UTF-8 (write_stamp).
They are read as UTF-8, and as the code page when the bytes are not UTF-8 (read_stamp). A stamp an earlier builder wrote for a name such as Zelda™.nup still matches, so that game is not unpacked a second time.
Line endings are kept as they are, so .hookshot_romfs_checked stays a byte copy of the stamp (tests/test_make_game_cache.py compares the bytes).

.hookshot_romfs_heads already used UTF-8.

Tests

New: test_a_stamp_keeps_a_name_outside_the_code_page in nx2/test_nup_zstd_heads.py.

Run on Windows 11 (code page cp1252), Python 3.14:

python -I -B nx2/test_nup_zstd_heads.py: 6 tests, OK
python -I -B nx2/test_nup_stand_in.py: ok
python -I -B tests/test_make_game_cache.py: 11 tests, OK (1 skipped)
python -I -B tests/test_missing_nup.py: 8 tests, OK

Not tested: a full build from a real .nup with such a name.

🤖 Generated with Claude Code

Fixes #2246. ## The problem The build stops with ``` UnicodeEncodeError: 'charmap' codec can't encode character '⁚' in position 46: character maps to <undefined> ``` after the whole `.nup` is extracted. The reporter's file is `The Legend of Zelda™⁚ Ocarina of Time Remake.nup`. `make_game.py` writes the `.nup`'s file name into the `.hookshot_romfs` stamp (`unpacked by this builder: <name> <size>`), and opened that file with the default encoding, which is the code page on Windows. U+205A is not in cp1252. The prefix is 26 characters and `⁚` is the 21st character of the name, which gives position 46 exactly. The same line reproduces the error on a cp1252 Windows: ``` >>> open(p, "w").write("unpacked by this builder: The Legend of Zelda™⁚ Ocarina of Time Remake.nup 1\n") 'charmap' codec can't encode character '⁚' in position 46: character maps to <undefined> ``` ## The change - `.hookshot_romfs` and its `.hookshot_romfs_checked` copy are written as UTF-8 (`write_stamp`). - They are read as UTF-8, and as the code page when the bytes are not UTF-8 (`read_stamp`). A stamp an earlier builder wrote for a name such as `Zelda™.nup` still matches, so that game is not unpacked a second time. - Line endings are kept as they are, so `.hookshot_romfs_checked` stays a byte copy of the stamp (`tests/test_make_game_cache.py` compares the bytes). `.hookshot_romfs_heads` already used UTF-8. ## Tests New: `test_a_stamp_keeps_a_name_outside_the_code_page` in `nx2/test_nup_zstd_heads.py`. Run on Windows 11 (code page cp1252), Python 3.14: - `python -I -B nx2/test_nup_zstd_heads.py`: 6 tests, OK - `python -I -B nx2/test_nup_stand_in.py`: ok - `python -I -B tests/test_make_game_cache.py`: 11 tests, OK (1 skipped) - `python -I -B tests/test_missing_nup.py`: 8 tests, OK Not tested: a full build from a real `.nup` with such a name. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Arkex-ai commented

2026-11-11 12:24:00 +00:00

(Migrated from github.com)

I found a cache-identity edge case at da0d302b. With Python UTF-8 mode enabled, locale.getpreferredencoding(False) returns UTF-8, so the fallback still decodes as UTF-8 and replaces invalid bytes. Two different names can then compare equal:

raw = b"unpacked by this builder: authored \xff.nup 22\n"
requested = "unpacked by this builder: authored \ufffd.nup 22\n"
assert raw.decode("utf-8", "replace") == requested

I exercised the actual build entry path with authored inputs and the preferred encoding set to UTF-8. This head made zero extraction calls and kept the old authored assets for the different selected name. A lossless comparison control extracted once and selected the new assets. No game data or real subprocesses were used; input/save sentinels stayed intact.

Could the legacy fallback use the actual Windows ANSI encoding, with strict decoding and a cache miss for an undecodable identity? locale.getencoding() ignores UTF-8 mode and returns the ANSI code page on Windows. The UTF-8-mode/replacement-character case would be useful regression coverage.

I held my #2246 PR publication after finding this one and will coordinate the remaining coverage here.

I found a cache-identity edge case at `da0d302b`. With Python UTF-8 mode enabled, `locale.getpreferredencoding(False)` returns UTF-8, so the fallback still decodes as UTF-8 and replaces invalid bytes. Two different names can then compare equal: ```python raw = b"unpacked by this builder: authored \xff.nup 22\n" requested = "unpacked by this builder: authored \ufffd.nup 22\n" assert raw.decode("utf-8", "replace") == requested ``` I exercised the actual build entry path with authored inputs and the preferred encoding set to UTF-8. This head made **zero extraction calls** and kept the old authored assets for the different selected name. A lossless comparison control extracted once and selected the new assets. No game data or real subprocesses were used; input/save sentinels stayed intact. Could the legacy fallback use the actual Windows ANSI encoding, with strict decoding and a cache miss for an undecodable identity? [`locale.getencoding()`](https://docs.python.org/3.14/library/locale.html#locale.getencoding) ignores UTF-8 mode and returns the ANSI code page on Windows. The UTF-8-mode/replacement-character case would be useful regression coverage. I held my #2246 PR publication after finding this one and will coordinate the remaining coverage here.

Arkex-ai commented

2026-11-11 13:35:48 +00:00

(Migrated from github.com)

Retested the authored entry-path collision at a261ecdc: it now extracts once and selects the new assets instead of reusing the previous name's cache. Input and save sentinels remain intact; no game data or real subprocesses were used. The getencoding() + strict fallback addresses the case I reported. Thanks for picking it up.

Retested the authored entry-path collision at `a261ecdc`: it now extracts once and selects the new assets instead of reusing the previous name's cache. Input and save sentinels remain intact; no game data or real subprocesses were used. The `getencoding()` + strict fallback addresses the case I reported. Thanks for picking it up.

hckrman101 (Migrated from github.com) approved these changes

2026-11-11 16:29:00 +00:00

hckrman101 (Migrated from github.com) left a comment

Looks good. The diagnosis lines up (prefix of 26 + ⁚ as the 21st letter = position 46), and the fix is the right shape: the stamp is written as UTF-8 with newline="", so .hookshot_romfs_checked stays a byte copy and test_make_game_cache.py's byte comparison keeps holding.

Checked against the UTF-8 path rules: nothing goes through the code page on the write side any more, and the code page is only used as a strict, read-only fallback for stamps an older builder wrote. .hookshot_romfs and _checked are only read and written in make_game.py (the tests write them as UTF-8 already), so there's no second reader left on the default encoding.

Both platforms: on Windows locale.getencoding() gives the ANSI code page even in UTF-8 mode, so an old Zelda™.nup stamp still matches and isn't unpacked again. On Linux the old stamps were already UTF-8 in practice, so the first decode takes them, and under a C locale the fallback is a strict ASCII decode, which can only miss. getencoding() needs 3.11+, which is fine with the bundled Python and CI's 3.12.

Arkex-ai's UTF-8-mode collision (two names becoming equal through U+FFFD) is fixed by the strict decode + "" on failure, and they confirmed it at a261ecdc. The second test covers exactly that case. One harmless corner: in stand_in_damaged_files, if both .hookshot_romfs and _checked are undecodable they compare equal as "", but _checked is a byte copy of the stamp, so that's the correct answer anyway.

CI: everything passes except appimage, which fails on main too (a Launchpad connection timeout in the container setup), so it isn't from this PR. The new tests are in nx2/test_*.py, which run_all.py already picks up.

Looks good. The diagnosis lines up (prefix of 26 + `⁚` as the 21st letter = position 46), and the fix is the right shape: the stamp is written as UTF-8 with `newline=""`, so `.hookshot_romfs_checked` stays a byte copy and `test_make_game_cache.py`'s byte comparison keeps holding. Checked against the UTF-8 path rules: nothing goes through the code page on the write side any more, and the code page is only used as a strict, read-only fallback for stamps an older builder wrote. `.hookshot_romfs` and `_checked` are only read and written in `make_game.py` (the tests write them as UTF-8 already), so there's no second reader left on the default encoding. Both platforms: on Windows `locale.getencoding()` gives the ANSI code page even in UTF-8 mode, so an old `Zelda™.nup` stamp still matches and isn't unpacked again. On Linux the old stamps were already UTF-8 in practice, so the first decode takes them, and under a C locale the fallback is a strict ASCII decode, which can only miss. `getencoding()` needs 3.11+, which is fine with the bundled Python and CI's 3.12. Arkex-ai's UTF-8-mode collision (two names becoming equal through U+FFFD) is fixed by the strict decode + `""` on failure, and they confirmed it at `a261ecdc`. The second test covers exactly that case. One harmless corner: in `stand_in_damaged_files`, if both `.hookshot_romfs` and `_checked` are undecodable they compare equal as `""`, but `_checked` is a byte copy of the stamp, so that's the correct answer anyway. CI: everything passes except `appimage`, which fails on main too (a Launchpad connection timeout in the container setup), so it isn't from this PR. The new tests are in `nx2/test_*.py`, which `run_all.py` already picks up.