fix(builder): a dump with many damaged files stops at once, not overnight, and a few find their stand-ins in one or two opens #1453
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1453
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:fix-1275-damaged-dump-fail-fast"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: nothing run (code read on Linux; no checks run on this PC) · Needs: Windows, any GPU, a dump with damaged files (Forgejo CI for the game-free checks)
Why
A player's Windows build from a damaged dump sat after
main: ZBIC -> plain NSOovernight, with nothing more in thelog, before the launcher finally said "Your game dump is incomplete" (1896 damaged files with no replacement). The dump
was bad, but the builder should say so at once, not after hours of silence.
The hours were spent in
nup.stand_in_for(nx2/nup.py): for each damaged file it listed the folder and openedevery file of the same kind there before scoring them.
PreComputedCulling/RuntimeIdholds about 35,000.bcul.zs, so 1896 damaged files there come to tens of millions of opens, and on Windows (Defender, a hard drive) thattakes hours. Nothing was printed between the extraction's last line and the result.
Fixes #1275
How to reproduce
Seen on: Windows 11, AMD Ryzen Z1 Extreme, Longshot Launcher v1.0.0-rc.9 building v3.0.0-rc.5 (#1275)
Settings and switches: defaults
Steps: build from a
.nupwhose extraction finds many damaged romfs.zsfiles (a hash mismatch or a zero start)Expected: the build stops within seconds of the extraction with "The game dump still has N damaged files with no replacement"
Actual: the log ends at
main: ZBIC -> plain NSOfor hours (overnight in #1275) before that messageGame-free:
python -I -B tests/test_damaged_stand_in_fail_fast.py(made-up files in a temporary folder; it counts thefiles
nupopens).What changed
The stand-in search for damaged files no longer opens every file of their kind, and a dump with more damaged files than
an incomplete dump ever has stops straight away with the message the launcher already shows as "Your game dump is
incomplete".
nx2/nup.py,stand_in_damaged: pastMOST_STAND_INS(100) damaged files, none is looked for; each comes back withno stand-in and nothing is written. An incomplete dump has a few (three textures in the 23.3 GB dumps).
nx2/nup.py,stand_in_for: the files of its kind are ranked by the same rule as before (longest common start of thename, then nearest size, ties to the first in the listing) from one
scandirof the folder (sizes from the listing,kept for the pass in
listings), then opened in that order until a whole one: usually one open per damaged file.nx2/make_game.py,stand_in_damaged_files: says how many damaged files the extraction found before the stand-ins arelooked for (the log no longer ends at the ZBIC line), says when there are too many to replace, names the first 50
(
SHOWN_DAMAGED) in the log and keeps all of them inlongshot_damaged_files.txt. The stop is the samefail(...)as before ("still has N damaged files with no replacement"), which the launcher's
Problems.csmaps to "Your gamedump is incomplete".
Measurements
Verdict: not measured: no checks or builds were run on this PC (the person I work for asks for none here). The
reasoning: a romfs on this PC lists 35,313
.bcul.zsinPreComputedCulling/RuntimeId, 20,667 inTex; onmaineach damaged file opened every whole file of its kind, after this change one or two, and above 100 damaged none.
Flags and settings
None:
MOST_STAND_INSandSHOWN_DAMAGEDare constants, not switches.Testing
Tested: nothing run here. The new check
tests/test_damaged_stand_in_fail_fast.pywas written but not run on thisPC: Forgejo CI runs it with the rest. It checks that 150 damaged files open no file and stop
stand_in_damaged_fileswith the incomplete-dump message and the full list in the note; that 3 damaged files among 400
.bcul.zsopen at most6 files and still get stand-ins; and that the choice (name, then size, never damaged, never a file that is not whole) is
as before.
Not tested: a real build from a damaged dump (none on this PC, and the dump is not to be damaged to make one); the
test run on
main(it should fail there:mainstands in all 150 and opens the 400 whole files for each).Must be tested:
python -I -B tests/run_all.pypasses, includingtests/test_damaged_stand_in_fail_fast.py,nx2/test_nup_stand_in.py,nx2/test_nup_zstd_heads.py,nx2/test_nup_verify.py,tests/test_make_game_cache.pyand
tests/test_damaged_check_progress.py.the build to stop seconds after
main: ZBIC -> plain NSOwith "N damaged game files found while reading your gamefiles" and then "Your game dump is incomplete" (by hand: the reporter or a tester with a bad dump).
Risks
What could break: a dump with more than 100 damaged files that used to get stand-ins for all of them now stops
instead. Such a game had that many assets replaced by others (with culling data among them) and was not playable as the
game; asking for a new dump is the better answer. The ranking is the same rule as before; a reviewer should check the
sort key against the old
score > bestloop.Turning it off: revert the commit.
Compatibility: Nothing changes for a whole dump. A dump with 1 to 100 damaged files is treated as before, faster.
Issues
Fixes #1275
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Not run on this PC; Forgejo CI runs it.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here. Added (test commit first, then the fix); not run either way on this PC.🤖 Generated with Claude Code
WIP: fix(builder): a dump with many damaged files stops at once, not overnight, and a few find their stand-ins in one or two opensto fix(builder): a dump with many damaged files stops at once, not overnight, and a few find their stand-ins in one or two opens