perf(build): a .nup's romfs extracts on a few workers, bounded by memory #2178

Open
dazemc wants to merge 6 commits from dazemc/Longshot:nup-extract-workers into main
Contributor

Summary

nx2/nup.py --extract walked a title's files and copied them one at a time. On this PC the 25.6 GB package is 186,588 files and 26.27 GB, and that took 54.754 s. The extraction now hands the members to a small thread pool: each worker opens the package for itself (the copy seeks, so the main reader is never shared), the output's folders are made once on the main thread before the workers start, and the progress counter, the completed-file count and the damaged-.zs heads list are kept whole by a lock. The serial path's writes are unchanged.

The worker count is bounded by memory, never by the CPU count: LONGSHOT_NUP_WORKERS when it is set, otherwise one per GiB of the memory the build may take — the smaller of the cgroup v2 memory limit and MemAvailable on Linux (the available physical memory on Windows) — at most 4 and at most LONGSHOT_JOBS when that is given, and one for fewer than two members. A worker holds a zstd frame's window and its buffers, and this PC freezes when a build opens many at once. A reader that is not a package on disk (a check's stand-in) extracts on one thread. The archive format, the order and the bytes written are unchanged. A stale <member>.part left by an interrupted extraction is now removed for every member on the main thread before the workers start, not only beside whichever member a worker happened to write first.

A reviewer should read extract, extract_one, extract_workers and ExtractProgress in nx2/nup.py, and tests/test_nup_extract_parallel.py.

Testing

Tested: Linux, 32 threads, 15 GB (build tooling, so no GPU or game run). On the real .nup: serial (LONGSHOT_NUP_WORKERS=1) 54.754 s and four workers 35.049 s, both 186,588 files and 26.27 GB from fresh output folders. A full SHA-256 comparison of the two extracted trees (186,589 entries, the extra one is romfs_zstd_heads.txt) shows the same path set, no file only in one, and 0 differing: RESULT ok. tests/test_nup_extract_parallel.py extracts a synthetic tree with one worker and with four and requires every member's SHA-256 equal, then extracts again over the same output and requires no write. It also drives extract_one from two threads at once through a barrier and requires more than one thread seen, so a fall-back to one worker fails the check (on main, which ignores LONGSHOT_NUP_WORKERS, it fails on the missing worker path). It plants a .part beside every member and requires all of them gone, and plants one beside a member already present on the second run. The default is checked too: 2 GiB free picks 2 workers, a 1 GiB cgroup limit picks 1, LONGSHOT_JOBS=2 caps at 2, and LONGSHOT_NUP_WORKERS is the explicit override; in a real systemd-run -p MemoryMax=2G scope the reader sees 2048 MB and picks 2. The nup checks pass: test_nup_extract_parallel, test_nup_codec, test_nup_verify, test_nup_paths, test_nup_zstd_heads, test_nup_read_error, test_nup_stand_in, test_nup_out_of_space. The changed checks pass (16 of 17; the one failure, test_damaged_rom_files, fails the same way on main, not from this change). The full mise run test (--strict --jobs 5, 12 GiB) ran on 06537e1f: 912 passed, 23 failed, 18 skipped in 406 s, and the 23 failed names are exactly main a2dc5385's. In game: N/A, build tooling.

Still to test:

  • Windows: extract the same package into a fresh folder and compare the tree, by hand.

Measurements

Verdict: 1.40× faster: median 50.734 s to 36.172 s, 14.6 s less, and the extracted tree identical.

Setup: Linux, 32 threads, 15 GB, f2fs on /dev/md127p2; the 25.6 GB .nup of The Legend of Zelda: Ocarina of Time, title 0400486024dec000 v131072. Same package, same output layout, a fresh output folder each run; wall time from date +%s%N around python -I nx2/nup.py with --extract. Two interleaved pairs, the order reversed between them: pair one at 16d0473f ran serial then four workers (54.754 s, 35.049 s); pair two at 06537e1f, with the per-member .part clean-up, ran four workers then serial (37.294 s, 46.714 s).

Baseline: the before side is this PR's serial extraction (LONGSHOT_NUP_WORKERS=1), whose copy loop is main's unchanged; the full-tree comparison in Testing shows the serial and four-worker trees identical, so no separate main build was needed.

Before, LONGSHOT_NUP_WORKERS=1: this PR @ 06537e1f After: this PR @ 06537e1f
extraction, 186,588 files / 26.27 GB (median of two interleaved pairs) 50.734 s (54.754, 46.714) 36.172 s (35.049, 37.294)
extracted tree, full SHA-256 (pair one) 186,589 entries 186,589 entries, 0 differing; pair two wrote 186,589 files each

A cold full build on this PC is 1707 s (build_times.json: lift 215 s, compile 1433 s), so the serial extraction is about 3.0% of an install and this saves 14.6 s of it, about 0.85% of a full install.

Why the step measure stands in for a clean build: the extraction is about 3% of a 1707 s cold install, and the two extracted trees are hash-identical (186,589 entries, 0 differing), so a clean build before and after would spend about 97% of its time in the lift and compile this change does not touch; the extraction step is measured directly on the same package, same output layout, fresh output folders. The median is over two heads: 16d0473f and 06537e1f, whose extraction code differs only by the per-member .part clean-up.

Interleaving: the second pair reversed the order (four workers first, then serial) to balance a cache warm-up: its serial time is lower (46.714 s against 54.754 s) while its four-worker time is close (37.294 s against 35.049 s), so the median is the fairer number, and it is the one in the table.

Risks

A worker that read from a shared package handle or wrote another member's file would corrupt the tree; each worker opens its own reader, every member is created exclusively at its own path, the folders are made before any worker starts, and the game-free check plus the full real-package comparison above are the evidence that it does not. Memory is bounded on purpose: at most four workers and about 1 GiB of free memory budgeted for each, so a low-memory PC uses one or two; LONGSHOT_NUP_WORKERS=1 restores the serial extraction. The 1 GiB per-worker budget is a safety bound, not a measured peak: on the shipped .nup a worker's own reader holds 170 MB of file records (372,587 members) plus one 8 MiB read block, about 180 MB, so the budget is about 5.7× conservative there; a packed .nup adds the frame's window and one decompression call's output (a synthetic 8 MiB input expanded to 200 MB), and a reader's records grow with the member count, so a peak-RSS measurement on a packed or much larger package is what would settle those cases. The archive format and the output's bytes do not change, so nothing a player has is invalidated and a revert is enough. Windows was not run here; the shared code (os.open with O_EXCL, long paths, GlobalMemoryStatusEx) is unchanged apart from running on several threads, and the Windows game-free checks run in CI.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named). 912 passed, 23 failed, 18 skipped in 406 s on 06537e1f (--strict --jobs 5, 12 GiB); the 23 failed names are exactly main a2dc5385's.
  • Measured against main, or Measurements says why there is nothing to measure. Median of two interleaved pairs: 50.734 s serial, 36.172 s four workers (1.40×), 14.6 s saved; the extracted trees are identical/completed.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A check under tests/ that reads changed source is updated, not weakened.
  • Docs updated.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary `nx2/nup.py --extract` walked a title's files and copied them one at a time. On this PC the 25.6 GB package is 186,588 files and 26.27 GB, and that took 54.754 s. The extraction now hands the members to a small thread pool: each worker opens the package for itself (the copy seeks, so the main reader is never shared), the output's folders are made once on the main thread before the workers start, and the progress counter, the completed-file count and the damaged-`.zs` heads list are kept whole by a lock. The serial path's writes are unchanged. The worker count is bounded by memory, never by the CPU count: `LONGSHOT_NUP_WORKERS` when it is set, otherwise one per GiB of the memory the build may take — the smaller of the cgroup v2 memory limit and `MemAvailable` on Linux (the available physical memory on Windows) — at most 4 and at most `LONGSHOT_JOBS` when that is given, and one for fewer than two members. A worker holds a zstd frame's window and its buffers, and this PC freezes when a build opens many at once. A reader that is not a package on disk (a check's stand-in) extracts on one thread. The archive format, the order and the bytes written are unchanged. A stale `<member>.part` left by an interrupted extraction is now removed for every member on the main thread before the workers start, not only beside whichever member a worker happened to write first. A reviewer should read `extract`, `extract_one`, `extract_workers` and `ExtractProgress` in `nx2/nup.py`, and `tests/test_nup_extract_parallel.py`. ## Testing **Tested:** Linux, 32 threads, 15 GB (build tooling, so no GPU or game run). On the real `.nup`: serial (`LONGSHOT_NUP_WORKERS=1`) 54.754 s and four workers 35.049 s, both 186,588 files and 26.27 GB from fresh output folders. A full SHA-256 comparison of the two extracted trees (186,589 entries, the extra one is `romfs_zstd_heads.txt`) shows the same path set, no file only in one, and 0 differing: `RESULT ok`. `tests/test_nup_extract_parallel.py` extracts a synthetic tree with one worker and with four and requires every member's SHA-256 equal, then extracts again over the same output and requires no write. It also drives `extract_one` from two threads at once through a barrier and requires more than one thread seen, so a fall-back to one worker fails the check (on `main`, which ignores `LONGSHOT_NUP_WORKERS`, it fails on the missing worker path). It plants a `.part` beside every member and requires all of them gone, and plants one beside a member already present on the second run. The default is checked too: 2 GiB free picks 2 workers, a 1 GiB cgroup limit picks 1, `LONGSHOT_JOBS=2` caps at 2, and `LONGSHOT_NUP_WORKERS` is the explicit override; in a real `systemd-run -p MemoryMax=2G` scope the reader sees 2048 MB and picks 2. The nup checks pass: `test_nup_extract_parallel`, `test_nup_codec`, `test_nup_verify`, `test_nup_paths`, `test_nup_zstd_heads`, `test_nup_read_error`, `test_nup_stand_in`, `test_nup_out_of_space`. The changed checks pass (16 of 17; the one failure, `test_damaged_rom_files`, fails the same way on `main`, not from this change). The full `mise run test` (`--strict --jobs 5`, 12 GiB) ran on `06537e1f`: 912 passed, 23 failed, 18 skipped in 406 s, and the 23 failed names are exactly `main` a2dc5385's. In game: N/A, build tooling. **Still to test:** - Windows: extract the same package into a fresh folder and compare the tree, by hand. ## Measurements **Verdict:** 1.40× faster: median 50.734 s to 36.172 s, 14.6 s less, and the extracted tree identical. Setup: Linux, 32 threads, 15 GB, f2fs on `/dev/md127p2`; the 25.6 GB `.nup` of The Legend of Zelda: Ocarina of Time, title 0400486024dec000 v131072. Same package, same output layout, a fresh output folder each run; wall time from `date +%s%N` around `python -I nx2/nup.py` with `--extract`. Two interleaved pairs, the order reversed between them: pair one at `16d0473f` ran serial then four workers (54.754 s, 35.049 s); pair two at `06537e1f`, with the per-member `.part` clean-up, ran four workers then serial (37.294 s, 46.714 s). **Baseline:** the before side is this PR's serial extraction (`LONGSHOT_NUP_WORKERS=1`), whose copy loop is `main`'s unchanged; the full-tree comparison in Testing shows the serial and four-worker trees identical, so no separate `main` build was needed. | | Before, `LONGSHOT_NUP_WORKERS=1`: this PR @ 06537e1f | After: this PR @ 06537e1f | |---|---|---| | extraction, 186,588 files / 26.27 GB (median of two interleaved pairs) | 50.734 s (54.754, 46.714) | 36.172 s (35.049, 37.294) | | extracted tree, full SHA-256 (pair one) | 186,589 entries | 186,589 entries, 0 differing; pair two wrote 186,589 files each | A cold full build on this PC is 1707 s (`build_times.json`: lift 215 s, compile 1433 s), so the serial extraction is about 3.0% of an install and this saves 14.6 s of it, about 0.85% of a full install. **Why the step measure stands in for a clean build:** the extraction is about 3% of a 1707 s cold install, and the two extracted trees are hash-identical (186,589 entries, 0 differing), so a clean build before and after would spend about 97% of its time in the lift and compile this change does not touch; the extraction step is measured directly on the same package, same output layout, fresh output folders. The median is over two heads: `16d0473f` and `06537e1f`, whose extraction code differs only by the per-member `.part` clean-up. **Interleaving:** the second pair reversed the order (four workers first, then serial) to balance a cache warm-up: its serial time is lower (46.714 s against 54.754 s) while its four-worker time is close (37.294 s against 35.049 s), so the median is the fairer number, and it is the one in the table. ## Risks A worker that read from a shared package handle or wrote another member's file would corrupt the tree; each worker opens its own reader, every member is created exclusively at its own path, the folders are made before any worker starts, and the game-free check plus the full real-package comparison above are the evidence that it does not. Memory is bounded on purpose: at most four workers and about 1 GiB of free memory budgeted for each, so a low-memory PC uses one or two; `LONGSHOT_NUP_WORKERS=1` restores the serial extraction. The 1 GiB per-worker budget is a safety bound, not a measured peak: on the shipped `.nup` a worker's own reader holds 170 MB of file records (372,587 members) plus one 8 MiB read block, about 180 MB, so the budget is about 5.7× conservative there; a packed `.nup` adds the frame's window and one decompression call's output (a synthetic 8 MiB input expanded to 200 MB), and a reader's records grow with the member count, so a peak-RSS measurement on a packed or much larger package is what would settle those cases. The archive format and the output's bytes do not change, so nothing a player has is invalidated and a revert is enough. Windows was not run here; the shared code (`os.open` with `O_EXCL`, long paths, `GlobalMemoryStatusEx`) is unchanged apart from running on several threads, and the Windows game-free checks run in CI. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [x] `mise run test` passes (any failure that also fails on `main` is named). 912 passed, 23 failed, 18 skipped in 406 s on `06537e1f` (`--strict --jobs 5`, 12 GiB); the 23 failed names are exactly `main` a2dc5385's. - [x] Measured against `main`, or Measurements says why there is nothing to measure. Median of two interleaved pairs: 50.734 s serial, 36.172 s four workers (1.40×), 14.6 s saved; the extracted trees are identical/completed. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A check under `tests/` that reads changed source is updated, not weakened. - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
perf(build): a .nup's romfs extracts on a few workers, bounded by memory
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 5s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
16d0473f17
nup.py extract walked a title's files and copied them one at a time: 54.8 s for
186,588 files and 26.27 GB of the 25.6 GB package's tree on this PC (serial,
LONGSHOT_NUP_WORKERS=1). It now hands the members to a small thread pool: one
open package per worker (copy_to seeks, so the main reader is not shared), the
output's folders made on one thread before the workers start, and the progress
counter, the file count and the heads list kept whole by a lock. The serial path
is unchanged in what it writes.

The worker count is memory-bounded, never the CPU count: LONGSHOT_NUP_WORKERS
when set, else one per GB free (MemAvailable on Linux, the available physical
memory on Windows), 4 at most, and one for fewer than two members. A worker
holds a zstd frame's window and its buffers, and this PC freezes when a build
opens many at once. A reader that is not a package on disk (a check's stand-in)
extracts on one thread.

The archive format, the order and the bytes written are unchanged: the
game-free check extracts a synthetic tree with one worker and with four and
compares every member's SHA-256, then extracts again over the same output and
requires no write.

Assisted-by: DeepSeek V4.1 Flash 1017 (opencode-go)
Author
Contributor

Tested in game: N/A: build tooling (nx2/nup.py's .nup extraction); no game run applies

Tested in game: N/A: build tooling (nx2/nup.py's .nup extraction); no game run applies
Member

Possibly related, for context:

  • PR #999 test(build): the fake .nup extractions hold the files #863 requires of a romfs — Open PR touches fake .nup extraction test fixtures
Possibly related, for context: - PR #999 test(build): the fake .nup extractions hold the files #863 requires of a romfs — Open PR touches fake .nup extraction test fixtures
Author
Contributor

Body: the Measurements before column is now this PR's serial extraction (LONGSHOT_NUP_WORKERS=1), which is main's copy loop unchanged, with a Baseline line; Testing says the changed checks pass 16 of 17 (the failure is main's) and the full mise run test was not run, so its box is unticked and the PR goes back to WIP for the PC's next slot.

Body: the Measurements before column is now this PR's serial extraction (`LONGSHOT_NUP_WORKERS=1`), which is main's copy loop unchanged, with a Baseline line; Testing says the changed checks pass 16 of 17 (the failure is main's) and the full `mise run test` was not run, so its box is unticked and the PR goes back to WIP for the PC's next slot.
dazemc changed title from perf(build): a .nup's romfs extracts on a few workers, bounded by memory to WIP: perf(build): a .nup's romfs extracts on a few workers, bounded by memory 2026-11-22 04:06:40 +00:00
Author
Contributor

Full mise run test (--strict --jobs 5, MemoryMax=12G, head 16d0473f): 912 passed, 23 failed, 18 skipped in 396 s; the 23 failed names are exactly main a2dc5385's (diff empty). The suite box is ticked; PR back to ready.

Full `mise run test` (`--strict --jobs 5`, MemoryMax=12G, head 16d0473f): 912 passed, 23 failed, 18 skipped in 396 s; the 23 failed names are exactly main a2dc5385's (diff empty). The suite box is ticked; PR back to ready.
dazemc changed title from WIP: perf(build): a .nup's romfs extracts on a few workers, bounded by memory to perf(build): a .nup's romfs extracts on a few workers, bounded by memory 2026-11-22 05:11:49 +00:00
docs(build): the .nup extraction's worker switch is in the build switches
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
620f27755a
LONGSHOT_NUP_WORKERS (nx2/nup.py): members copied at once, romfs and exefs;
default one per GiB free, 4 at most, 1 for fewer than two files. The archive's
format and the bytes written do not change.

Assisted-by: DeepSeek V4.1 Flash 1017 (opencode-go)
Author
Contributor

Docs: LONGSHOT_NUP_WORKERS is in docs/dev/reference/build.md's build switches table (620f2775), and Docs updated is ticked. Measurements: added why the extraction step stands in for a clean build (3.2% of a 1707 s install, trees hash-identical); the second interleaved serial/4-worker pair waits for the next machine slot (mem4 holds it now).

Docs: `LONGSHOT_NUP_WORKERS` is in `docs/dev/reference/build.md`'s build switches table (`620f2775`), and Docs updated is ticked. Measurements: added why the extraction step stands in for a clean build (3.2% of a 1707 s install, trees hash-identical); the second interleaved serial/4-worker pair waits for the next machine slot (mem4 holds it now).
It drives nup.extract_one from two threads at once through a barrier and
requires more than one thread seen, so a fall-back to one worker fails here
(it passed on main, which ignores LONGSHOT_NUP_WORKERS). It plants a .part
beside every member and requires all of them gone, not just the one beside the
first member a worker writes, and plants one beside a member already present
on the second run.

Assisted-by: DeepSeek V4.1 Flash 1017 (opencode-go)
fix(build): every stale .part goes before the extraction's workers start
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 8s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
92f9098fa4
An interrupted extraction leaves <member>.part beside a member. Only the part
beside the first member a worker happened to write was removed, so with four
workers a stale romfs/...part whose member was written later survived. Every
member's .part is now removed on the main thread while the work is listed,
before any worker starts, whether that member will be written or is already
there. One unlink of a name that is usually absent per member.

Assisted-by: DeepSeek V4.1 Flash 1017 (opencode-go)
Author
Contributor

Review fixes (5ec282bc test, 92f9098f fix): the extraction check now drives extract_one from two threads through a barrier and requires more than one thread (it fails on main's missing worker path), and plants a .part beside every member, requiring all gone; a stale .part is now removed for every member on the main thread before any worker starts, not only the first written member's. Targeted nup checks: 13 passed. The second interleaved extraction pair still waits for a slot; the pair above is at 16d0473f.

Review fixes (`5ec282bc` test, `92f9098f` fix): the extraction check now drives `extract_one` from two threads through a barrier and requires more than one thread (it fails on main's missing worker path), and plants a `.part` beside every member, requiring all gone; a stale `.part` is now removed for every member on the main thread before any worker starts, not only the first written member's. Targeted nup checks: 13 passed. The second interleaved extraction pair still waits for a slot; the pair above is at `16d0473f`.
fix(build): the extraction's worker default keeps to the cgroup and LONGSHOT_JOBS
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
cb77192e64
MemAvailable is the whole PC's, so inside a systemd scope (MemoryMax) the default
could open four workers with only the scope's small headroom. The budget is now
the smallest of the cgroup v2 memory limit and MemAvailable, the same budget
tests/run_all.py's default check count reads (#2177), and the count is capped at
LONGSHOT_JOBS when the build was given one, as every step make_game.py runs side
by side is. LONGSHOT_NUP_WORKERS stays the explicit override.

The per-worker budget is the 1 GiB the default already used; it is a safety
bound, not a measured RSS, and the change can only lower the count.

Assisted-by: DeepSeek V4.1 Flash 1017 (opencode-go)
Author
Contributor

Cross-PR hazard with #2177: the extraction default now uses the same budget (the smallest of the cgroup v2 memory limit and MemAvailable) and is capped by LONGSHOT_JOBS, with LONGSHOT_NUP_WORKERS still the explicit override (cb77192e). Test: 2 GiB free -> 2, a 1 GiB cgroup -> 1, LONGSHOT_JOBS=2 -> 2, override 3 -> 3; in a real MemoryMax=2G scope the reader sees 2048 MB and picks 2. The per-worker budget is the existing 1 GiB safety bound, not a measured RSS; the change can only lower the count. Targeted checks: 13 passed.

Cross-PR hazard with #2177: the extraction default now uses the same budget (the smallest of the cgroup v2 memory limit and MemAvailable) and is capped by `LONGSHOT_JOBS`, with `LONGSHOT_NUP_WORKERS` still the explicit override (`cb77192e`). Test: 2 GiB free -> 2, a 1 GiB cgroup -> 1, `LONGSHOT_JOBS=2` -> 2, override 3 -> 3; in a real `MemoryMax=2G` scope the reader sees 2048 MB and picks 2. The per-worker budget is the existing 1 GiB safety bound, not a measured RSS; the change can only lower the count. Targeted checks: 13 passed.
Author
Contributor

Per-worker RSS, derived: a raw copy holds one 8 MiB block; each worker's own Nup reader holds 170.2 MB of tables for the shipped package (372,587 records: files 79.2, file_details 71.8, kids 13.5, strings 5.5), so ~180 MB/worker, about 5.7x under the 1 GiB budget. The shipped .nup has zero outer-packed members (stored == logical for all 372,587), so only the raw path runs. The packed path can return a whole member in one decompress() call (a synthetic 8 MiB input expanded to 200 MB) and a reader's tables grow with member count, so a constant 'cannot exceed' bound is not sound and no budget assertion was added (Jev 0.04 / 0.10). Risks now states the derivation and that a peak-RSS measurement on a packed or much larger package would settle the remaining cases.

Per-worker RSS, derived: a raw copy holds one 8 MiB block; each worker's own Nup reader holds 170.2 MB of tables for the shipped package (372,587 records: files 79.2, file_details 71.8, kids 13.5, strings 5.5), so ~180 MB/worker, about 5.7x under the 1 GiB budget. The shipped .nup has zero outer-packed members (stored == logical for all 372,587), so only the raw path runs. The packed path can return a whole member in one decompress() call (a synthetic 8 MiB input expanded to 200 MB) and a reader's tables grow with member count, so a constant 'cannot exceed' bound is not sound and no budget assertion was added (Jev 0.04 / 0.10). Risks now states the derivation and that a peak-RSS measurement on a packed or much larger package would settle the remaining cases.
test(build): the extraction's worker and .part checks run before the default's
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 5s
06537e1f51
test_worker_defaults reads nup.cgroup_limit_mb, which is newer than the worker
pool and the stale-.part fix; on main or an older head it raised AttributeError
before those checks ran, hiding the reason. The worker-path, stale-.part and
second-run checks now run first: main fails on the missing worker path, and
16d0473f fails on the leftover .part, as the PR says.

Assisted-by: DeepSeek V4.1 Flash 1017 (opencode-go)
Author
Contributor

State: back to WIP. Head is 06537e1f (test reorder) and the last full suite is 16d0473f, so the suite box is unticked with that reason; the measurement box is unticked because the second interleaved extraction pair is pending on 06537e1f. The reordered test now fails on main at the worker path and on 16d0473f at the stale .part, as Testing says.

State: back to WIP. Head is 06537e1f (test reorder) and the last full suite is 16d0473f, so the suite box is unticked with that reason; the measurement box is unticked because the second interleaved extraction pair is pending on 06537e1f. The reordered test now fails on main at the worker path and on 16d0473f at the stale .part, as Testing says.
dazemc changed title from perf(build): a .nup's romfs extracts on a few workers, bounded by memory to WIP: perf(build): a .nup's romfs extracts on a few workers, bounded by memory 2026-11-22 07:25:20 +00:00
Author
Contributor

Suite at 06537e1f: 912 passed, 23 failed, 18 skipped in 406 s (--strict --jobs 5, MemoryMax=12G, pinned headers); the 23 failed names are exactly main a2dc5385's (diff empty). The reordered test is green under abrun. Suite box ticked; the second extraction pair is running on 06537e1f, so the PR stays WIP until it lands.

Suite at `06537e1f`: 912 passed, 23 failed, 18 skipped in 406 s (`--strict --jobs 5`, MemoryMax=12G, pinned headers); the 23 failed names are exactly main a2dc5385's (diff empty). The reordered test is green under `abrun`. Suite box ticked; the second extraction pair is running on `06537e1f`, so the PR stays WIP until it lands.
Author
Contributor

Second interleaved pair (order reversed: four workers then serial) on 06537e1f: 37.294 s and 46.714 s. Medians of the two pairs: 36.172 s four workers vs 50.734 s serial, 1.40x, 14.6 s saved (0.85% of a 1707 s install); both second-pair trees complete (186,589 files). Measurement box ticked.

Second interleaved pair (order reversed: four workers then serial) on `06537e1f`: 37.294 s and 46.714 s. Medians of the two pairs: 36.172 s four workers vs 50.734 s serial, 1.40x, 14.6 s saved (0.85% of a 1707 s install); both second-pair trees complete (186,589 files). Measurement box ticked.
dazemc changed title from WIP: perf(build): a .nup's romfs extracts on a few workers, bounded by memory to perf(build): a .nup's romfs extracts on a few workers, bounded by memory 2026-11-22 07:34:15 +00:00
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 5s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'nup-extract-workers:dazemc-nup-extract-workers'
git switch 'dazemc-nup-extract-workers'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2178
No description provided.