fix(builder): a dump with many damaged files stops at once, not overnight, and a few find their stand-ins in one or two opens #1453

Merged
HumanBean merged 2 commits from HumanBean/Longshot:fix-1275-damaged-dump-fail-fast into main 2026-11-17 19:59:44 +00:00
Owner

Type: fix · Risk: low · Tested on: nothing run (code read on Linux; no checks run on this PC) · Needs: Windows, any GPU, a dump with damaged files (Forgejo CI for the game-free checks)

Why

A player's Windows build from a damaged dump sat after main: ZBIC -> plain NSO overnight, with nothing more in the
log, before the launcher finally said "Your game dump is incomplete" (1896 damaged files with no replacement). The dump
was bad, but the builder should say so at once, not after hours of silence.

The hours were spent in nup.stand_in_for (nx2/nup.py): for each damaged file it listed the folder and opened
every file of the same kind there before scoring them. PreComputedCulling/RuntimeId holds about 35,000
.bcul.zs, so 1896 damaged files there come to tens of millions of opens, and on Windows (Defender, a hard drive) that
takes hours. Nothing was printed between the extraction's last line and the result.

Fixes #1275

How to reproduce

Seen on: Windows 11, AMD Ryzen Z1 Extreme, Longshot Launcher v1.0.0-rc.9 building v3.0.0-rc.5 (#1275)
Settings and switches: defaults
Steps: build from a .nup whose extraction finds many damaged romfs .zs files (a hash mismatch or a zero start)
Expected: the build stops within seconds of the extraction with "The game dump still has N damaged files with no replacement"
Actual: the log ends at main: ZBIC -> plain NSO for hours (overnight in #1275) before that message

Game-free: python -I -B tests/test_damaged_stand_in_fail_fast.py (made-up files in a temporary folder; it counts the
files nup opens).

What changed

The stand-in search for damaged files no longer opens every file of their kind, and a dump with more damaged files than
an incomplete dump ever has stops straight away with the message the launcher already shows as "Your game dump is
incomplete".

  • nx2/nup.py, stand_in_damaged: past MOST_STAND_INS (100) damaged files, none is looked for; each comes back with
    no stand-in and nothing is written. An incomplete dump has a few (three textures in the 23.3 GB dumps).
  • nx2/nup.py, stand_in_for: the files of its kind are ranked by the same rule as before (longest common start of the
    name, then nearest size, ties to the first in the listing) from one scandir of the folder (sizes from the listing,
    kept for the pass in listings), then opened in that order until a whole one: usually one open per damaged file.
  • nx2/make_game.py, stand_in_damaged_files: says how many damaged files the extraction found before the stand-ins are
    looked for (the log no longer ends at the ZBIC line), says when there are too many to replace, names the first 50
    (SHOWN_DAMAGED) in the log and keeps all of them in longshot_damaged_files.txt. The stop is the same fail(...)
    as before ("still has N damaged files with no replacement"), which the launcher's Problems.cs maps to "Your game
    dump is incomplete".

Measurements

Verdict: not measured: no checks or builds were run on this PC (the person I work for asks for none here). The
reasoning: a romfs on this PC lists 35,313 .bcul.zs in PreComputedCulling/RuntimeId, 20,667 in Tex; on main
each damaged file opened every whole file of its kind, after this change one or two, and above 100 damaged none.

Flags and settings

None: MOST_STAND_INS and SHOWN_DAMAGED are constants, not switches.

Testing

Tested: nothing run here. The new check tests/test_damaged_stand_in_fail_fast.py was written but not run on this
PC: Forgejo CI runs it with the rest. It checks that 150 damaged files open no file and stop stand_in_damaged_files
with the incomplete-dump message and the full list in the note; that 3 damaged files among 400 .bcul.zs open at most
6 files and still get stand-ins; and that the choice (name, then size, never damaged, never a file that is not whole) is
as before.

Not tested: a real build from a damaged dump (none on this PC, and the dump is not to be damaged to make one); the
test run on main (it should fail there: main stands in all 150 and opens the 400 whole files for each).

Must be tested:

  • Forgejo CI: python -I -B tests/run_all.py passes, including tests/test_damaged_stand_in_fail_fast.py,
    nx2/test_nup_stand_in.py, nx2/test_nup_zstd_heads.py, nx2/test_nup_verify.py, tests/test_make_game_cache.py
    and tests/test_damaged_check_progress.py.
  • Windows, any GPU, a dump with damaged files (the #1275 reporter's, or any): build from it in the launcher, expecting
    the build to stop seconds after main: ZBIC -> plain NSO with "N damaged game files found while reading your game
    files" and then "Your game dump is incomplete" (by hand: the reporter or a tester with a bad dump).

Risks

What could break: a dump with more than 100 damaged files that used to get stand-ins for all of them now stops
instead. Such a game had that many assets replaced by others (with culling data among them) and was not playable as the
game; asking for a new dump is the better answer. The ranking is the same rule as before; a reviewer should check the
sort key against the old score > best loop.

Turning it off: revert the commit.

Compatibility: Nothing changes for a whole dump. A dump with 1 to 100 damaged files is treated as before, faster.

Issues

Fixes #1275

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). Not run on this PC; Forgejo CI runs it.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here. Added (test commit first, then the fix); not run either way on this PC.

🤖 Generated with Claude Code

**Type:** fix · **Risk:** low · **Tested on:** nothing run (code read on Linux; no checks run on this PC) · **Needs:** Windows, any GPU, a dump with damaged files (Forgejo CI for the game-free checks) ## Why A player's Windows build from a damaged dump sat after `main: ZBIC -> plain NSO` overnight, with nothing more in the log, before the launcher finally said "Your game dump is incomplete" (1896 damaged files with no replacement). The dump was bad, but the builder should say so at once, not after hours of silence. The hours were spent in `nup.stand_in_for` (`nx2/nup.py`): for **each** damaged file it listed the folder and opened **every** file of the same kind there before scoring them. `PreComputedCulling/RuntimeId` holds about 35,000 `.bcul.zs`, so 1896 damaged files there come to tens of millions of opens, and on Windows (Defender, a hard drive) that takes hours. Nothing was printed between the extraction's last line and the result. Fixes #1275 ## How to reproduce **Seen on:** Windows 11, AMD Ryzen Z1 Extreme, Longshot Launcher v1.0.0-rc.9 building v3.0.0-rc.5 (#1275) **Settings and switches:** defaults **Steps:** build from a `.nup` whose extraction finds many damaged romfs `.zs` files (a hash mismatch or a zero start) **Expected:** the build stops within seconds of the extraction with "The game dump still has N damaged files with no replacement" **Actual:** the log ends at `main: ZBIC -> plain NSO` for hours (overnight in #1275) before that message Game-free: `python -I -B tests/test_damaged_stand_in_fail_fast.py` (made-up files in a temporary folder; it counts the files `nup` opens). ## What changed The stand-in search for damaged files no longer opens every file of their kind, and a dump with more damaged files than an incomplete dump ever has stops straight away with the message the launcher already shows as "Your game dump is incomplete". - `nx2/nup.py`, `stand_in_damaged`: past `MOST_STAND_INS` (100) damaged files, none is looked for; each comes back with no stand-in and nothing is written. An incomplete dump has a few (three textures in the 23.3 GB dumps). - `nx2/nup.py`, `stand_in_for`: the files of its kind are ranked by the same rule as before (longest common start of the name, then nearest size, ties to the first in the listing) from one `scandir` of the folder (sizes from the listing, kept for the pass in `listings`), then opened in that order until a whole one: usually one open per damaged file. - `nx2/make_game.py`, `stand_in_damaged_files`: says how many damaged files the extraction found before the stand-ins are looked for (the log no longer ends at the ZBIC line), says when there are too many to replace, names the first 50 (`SHOWN_DAMAGED`) in the log and keeps all of them in `longshot_damaged_files.txt`. The stop is the same `fail(...)` as before ("still has N damaged files with no replacement"), which the launcher's `Problems.cs` maps to "Your game dump is incomplete". ## Measurements **Verdict:** not measured: no checks or builds were run on this PC (the person I work for asks for none here). The reasoning: a romfs on this PC lists 35,313 `.bcul.zs` in `PreComputedCulling/RuntimeId`, 20,667 in `Tex`; on `main` each damaged file opened every whole file of its kind, after this change one or two, and above 100 damaged none. ### Flags and settings None: `MOST_STAND_INS` and `SHOWN_DAMAGED` are constants, not switches. ## Testing **Tested:** nothing run here. The new check `tests/test_damaged_stand_in_fail_fast.py` was written but not run on this PC: Forgejo CI runs it with the rest. It checks that 150 damaged files open no file and stop `stand_in_damaged_files` with the incomplete-dump message and the full list in the note; that 3 damaged files among 400 `.bcul.zs` open at most 6 files and still get stand-ins; and that the choice (name, then size, never damaged, never a file that is not whole) is as before. **Not tested:** a real build from a damaged dump (none on this PC, and the dump is not to be damaged to make one); the test run on `main` (it should fail there: `main` stands in all 150 and opens the 400 whole files for each). **Must be tested:** - Forgejo CI: `python -I -B tests/run_all.py` passes, including `tests/test_damaged_stand_in_fail_fast.py`, `nx2/test_nup_stand_in.py`, `nx2/test_nup_zstd_heads.py`, `nx2/test_nup_verify.py`, `tests/test_make_game_cache.py` and `tests/test_damaged_check_progress.py`. - Windows, any GPU, a dump with damaged files (the #1275 reporter's, or any): build from it in the launcher, expecting the build to stop seconds after `main: ZBIC -> plain NSO` with "N damaged game files found while reading your game files" and then "Your game dump is incomplete" (by hand: the reporter or a tester with a bad dump). ## Risks **What could break:** a dump with more than 100 damaged files that used to get stand-ins for all of them now stops instead. Such a game had that many assets replaced by others (with culling data among them) and was not playable as the game; asking for a new dump is the better answer. The ranking is the same rule as before; a reviewer should check the sort key against the old `score > best` loop. **Turning it off:** revert the commit. **Compatibility:** Nothing changes for a whole dump. A dump with 1 to 100 damaged files is treated as before, faster. ## Issues Fixes #1275 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). Not run on this PC; Forgejo CI runs it. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] A game-free test added for the fix, failing on `main` and passing here. Added (test commit first, then the fix); not run either way on this PC. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
On main, nup.stand_in_for opens every whole file of the damaged one's kind in its folder for each damaged file: a dump
with 1896 damaged files sat after "main: ZBIC -> plain NSO" overnight on Windows before the build said it was
incomplete. Game-free: made-up files in a temporary folder, nup's open counted.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(builder): a dump with many damaged files stops at once, not overnight, and a few find their stand-ins in one or two opens (#1275)
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 17s
CI / Workflow lint (actionlint) (pull_request) Successful in 26s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 26s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 37s
PR description / Description follows the template (pull_request_target) Failing after 3s
b2359fde34
A Windows build from a dump with 1896 damaged files sat after "main: ZBIC -> plain NSO" overnight before it said the
dump was incomplete. nup.stand_in_for listed the damaged file's folder and opened every file of its kind there, for each
damaged file: PreComputedCulling/RuntimeId holds about 35,000 .bcul.zs, so tens of millions of opens on Windows, with
nothing in the log.

- nup.stand_in_damaged: past MOST_STAND_INS (100) damaged files none is looked for: an incomplete dump has a few (three
  textures in the 23.3 GB dumps), and the build stops at once with the same "still has N damaged files with no
  replacement" the launcher shows as "Your game dump is incomplete".
- nup.stand_in_for ranks the files of its kind by name and size from one listing of the folder (kept for the pass), then
  opens them in that order until a whole one: the same choice as before, ties to the first in the listing.
- make_game says how many damaged files the extraction found before the stand-ins are looked for, names the first 50 in
  the log and keeps them all in longshot_damaged_files.txt.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean changed title from WIP: fix(builder): a dump with many damaged files stops at once, not overnight, and a few find their stand-ins in one or two opens to fix(builder): a dump with many damaged files stops at once, not overnight, and a few find their stand-ins in one or two opens 2026-11-17 19:59:36 +00:00
HumanBean deleted branch fix-1275-damaged-dump-fail-fast 2026-11-17 19:59:45 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1453
No description provided.