perf(build): a .nup's romfs extracts on a few workers, bounded by memory #2178
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2178
Loading…
Reference in a new issue
No description provided.
Delete branch "dazemc/Longshot:nup-extract-workers"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
nx2/nup.py --extractwalked a title's files and copied them one at a time. On this PC the 25.6 GB package is 186,588 files and 26.27 GB, and that took 54.754 s. The extraction now hands the members to a small thread pool: each worker opens the package for itself (the copy seeks, so the main reader is never shared), the output's folders are made once on the main thread before the workers start, and the progress counter, the completed-file count and the damaged-.zsheads list are kept whole by a lock. The serial path's writes are unchanged.The worker count is bounded by memory, never by the CPU count:
LONGSHOT_NUP_WORKERSwhen it is set, otherwise one per GiB of the memory the build may take — the smaller of the cgroup v2 memory limit andMemAvailableon Linux (the available physical memory on Windows) — at most 4 and at mostLONGSHOT_JOBSwhen that is given, and one for fewer than two members. A worker holds a zstd frame's window and its buffers, and this PC freezes when a build opens many at once. A reader that is not a package on disk (a check's stand-in) extracts on one thread. The archive format, the order and the bytes written are unchanged. A stale<member>.partleft by an interrupted extraction is now removed for every member on the main thread before the workers start, not only beside whichever member a worker happened to write first.A reviewer should read
extract,extract_one,extract_workersandExtractProgressinnx2/nup.py, andtests/test_nup_extract_parallel.py.Testing
Tested: Linux, 32 threads, 15 GB (build tooling, so no GPU or game run). On the real
.nup: serial (LONGSHOT_NUP_WORKERS=1) 54.754 s and four workers 35.049 s, both 186,588 files and 26.27 GB from fresh output folders. A full SHA-256 comparison of the two extracted trees (186,589 entries, the extra one isromfs_zstd_heads.txt) shows the same path set, no file only in one, and 0 differing:RESULT ok.tests/test_nup_extract_parallel.pyextracts a synthetic tree with one worker and with four and requires every member's SHA-256 equal, then extracts again over the same output and requires no write. It also drivesextract_onefrom two threads at once through a barrier and requires more than one thread seen, so a fall-back to one worker fails the check (onmain, which ignoresLONGSHOT_NUP_WORKERS, it fails on the missing worker path). It plants a.partbeside every member and requires all of them gone, and plants one beside a member already present on the second run. The default is checked too: 2 GiB free picks 2 workers, a 1 GiB cgroup limit picks 1,LONGSHOT_JOBS=2caps at 2, andLONGSHOT_NUP_WORKERSis the explicit override; in a realsystemd-run -p MemoryMax=2Gscope the reader sees 2048 MB and picks 2. The nup checks pass:test_nup_extract_parallel,test_nup_codec,test_nup_verify,test_nup_paths,test_nup_zstd_heads,test_nup_read_error,test_nup_stand_in,test_nup_out_of_space. The changed checks pass (16 of 17; the one failure,test_damaged_rom_files, fails the same way onmain, not from this change). The fullmise run test(--strict --jobs 5, 12 GiB) ran on06537e1f: 912 passed, 23 failed, 18 skipped in 406 s, and the 23 failed names are exactlymaina2dc5385's. In game: N/A, build tooling.Still to test:
Measurements
Verdict: 1.40× faster: median 50.734 s to 36.172 s, 14.6 s less, and the extracted tree identical.
Setup: Linux, 32 threads, 15 GB, f2fs on
/dev/md127p2; the 25.6 GB.nupof The Legend of Zelda: Ocarina of Time, title 0400486024dec000 v131072. Same package, same output layout, a fresh output folder each run; wall time fromdate +%s%Naroundpython -I nx2/nup.pywith--extract. Two interleaved pairs, the order reversed between them: pair one at16d0473fran serial then four workers (54.754 s, 35.049 s); pair two at06537e1f, with the per-member.partclean-up, ran four workers then serial (37.294 s, 46.714 s).Baseline: the before side is this PR's serial extraction (
LONGSHOT_NUP_WORKERS=1), whose copy loop ismain's unchanged; the full-tree comparison in Testing shows the serial and four-worker trees identical, so no separatemainbuild was needed.LONGSHOT_NUP_WORKERS=1: this PR @06537e1f06537e1fA cold full build on this PC is 1707 s (
build_times.json: lift 215 s, compile 1433 s), so the serial extraction is about 3.0% of an install and this saves 14.6 s of it, about 0.85% of a full install.Why the step measure stands in for a clean build: the extraction is about 3% of a 1707 s cold install, and the two extracted trees are hash-identical (186,589 entries, 0 differing), so a clean build before and after would spend about 97% of its time in the lift and compile this change does not touch; the extraction step is measured directly on the same package, same output layout, fresh output folders. The median is over two heads:
16d0473fand06537e1f, whose extraction code differs only by the per-member.partclean-up.Interleaving: the second pair reversed the order (four workers first, then serial) to balance a cache warm-up: its serial time is lower (46.714 s against 54.754 s) while its four-worker time is close (37.294 s against 35.049 s), so the median is the fairer number, and it is the one in the table.
Risks
A worker that read from a shared package handle or wrote another member's file would corrupt the tree; each worker opens its own reader, every member is created exclusively at its own path, the folders are made before any worker starts, and the game-free check plus the full real-package comparison above are the evidence that it does not. Memory is bounded on purpose: at most four workers and about 1 GiB of free memory budgeted for each, so a low-memory PC uses one or two;
LONGSHOT_NUP_WORKERS=1restores the serial extraction. The 1 GiB per-worker budget is a safety bound, not a measured peak: on the shipped.nupa worker's own reader holds 170 MB of file records (372,587 members) plus one 8 MiB read block, about 180 MB, so the budget is about 5.7× conservative there; a packed.nupadds the frame's window and one decompression call's output (a synthetic 8 MiB input expanded to 200 MB), and a reader's records grow with the member count, so a peak-RSS measurement on a packed or much larger package is what would settle those cases. The archive format and the output's bytes do not change, so nothing a player has is invalidated and a revert is enough. Windows was not run here; the shared code (os.openwithO_EXCL, long paths,GlobalMemoryStatusEx) is unchanged apart from running on several threads, and the Windows game-free checks run in CI.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named). 912 passed, 23 failed, 18 skipped in 406 s on06537e1f(--strict --jobs 5, 12 GiB); the 23 failed names are exactlymaina2dc5385's.main, or Measurements says why there is nothing to measure. Median of two interleaved pairs: 50.734 s serial, 36.172 s four workers (1.40×), 14.6 s saved; the extracted trees are identical/completed.third_party/untouched; nothing GPL-incompatible.When it applies
tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Tested in game: N/A: build tooling (nx2/nup.py's .nup extraction); no game run applies
Possibly related, for context:
Body: the Measurements before column is now this PR's serial extraction (
LONGSHOT_NUP_WORKERS=1), which is main's copy loop unchanged, with a Baseline line; Testing says the changed checks pass 16 of 17 (the failure is main's) and the fullmise run testwas not run, so its box is unticked and the PR goes back to WIP for the PC's next slot.perf(build): a .nup's romfs extracts on a few workers, bounded by memoryto WIP: perf(build): a .nup's romfs extracts on a few workers, bounded by memoryFull
mise run test(--strict --jobs 5, MemoryMax=12G, head16d0473f): 912 passed, 23 failed, 18 skipped in 396 s; the 23 failed names are exactly main a2dc5385's (diff empty). The suite box is ticked; PR back to ready.WIP: perf(build): a .nup's romfs extracts on a few workers, bounded by memoryto perf(build): a .nup's romfs extracts on a few workers, bounded by memoryDocs:
LONGSHOT_NUP_WORKERSis indocs/dev/reference/build.md's build switches table (620f2775), and Docs updated is ticked. Measurements: added why the extraction step stands in for a clean build (3.2% of a 1707 s install, trees hash-identical); the second interleaved serial/4-worker pair waits for the next machine slot (mem4 holds it now).Review fixes (
5ec282bctest,92f9098ffix): the extraction check now drivesextract_onefrom two threads through a barrier and requires more than one thread (it fails on main's missing worker path), and plants a.partbeside every member, requiring all gone; a stale.partis now removed for every member on the main thread before any worker starts, not only the first written member's. Targeted nup checks: 13 passed. The second interleaved extraction pair still waits for a slot; the pair above is at16d0473f.Cross-PR hazard with #2177: the extraction default now uses the same budget (the smallest of the cgroup v2 memory limit and MemAvailable) and is capped by
LONGSHOT_JOBS, withLONGSHOT_NUP_WORKERSstill the explicit override (cb77192e). Test: 2 GiB free -> 2, a 1 GiB cgroup -> 1,LONGSHOT_JOBS=2-> 2, override 3 -> 3; in a realMemoryMax=2Gscope the reader sees 2048 MB and picks 2. The per-worker budget is the existing 1 GiB safety bound, not a measured RSS; the change can only lower the count. Targeted checks: 13 passed.Per-worker RSS, derived: a raw copy holds one 8 MiB block; each worker's own Nup reader holds 170.2 MB of tables for the shipped package (372,587 records: files 79.2, file_details 71.8, kids 13.5, strings 5.5), so ~180 MB/worker, about 5.7x under the 1 GiB budget. The shipped .nup has zero outer-packed members (stored == logical for all 372,587), so only the raw path runs. The packed path can return a whole member in one decompress() call (a synthetic 8 MiB input expanded to 200 MB) and a reader's tables grow with member count, so a constant 'cannot exceed' bound is not sound and no budget assertion was added (Jev 0.04 / 0.10). Risks now states the derivation and that a peak-RSS measurement on a packed or much larger package would settle the remaining cases.
State: back to WIP. Head is
06537e1f(test reorder) and the last full suite is16d0473f, so the suite box is unticked with that reason; the measurement box is unticked because the second interleaved extraction pair is pending on06537e1f. The reordered test now fails on main at the worker path and on16d0473fat the stale .part, as Testing says.perf(build): a .nup's romfs extracts on a few workers, bounded by memoryto WIP: perf(build): a .nup's romfs extracts on a few workers, bounded by memorySuite at
06537e1f: 912 passed, 23 failed, 18 skipped in 406 s (--strict --jobs 5, MemoryMax=12G, pinned headers); the 23 failed names are exactly main a2dc5385's (diff empty). The reordered test is green underabrun. Suite box ticked; the second extraction pair is running on06537e1f, so the PR stays WIP until it lands.Second interleaved pair (order reversed: four workers then serial) on
06537e1f: 37.294 s and 46.714 s. Medians of the two pairs: 36.172 s four workers vs 50.734 s serial, 1.40x, 14.6 s saved (0.85% of a 1707 s install); both second-pair trees complete (186,589 files). Measurement box ticked.WIP: perf(build): a .nup's romfs extracts on a few workers, bounded by memoryto perf(build): a .nup's romfs extracts on a few workers, bounded by memoryView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.