perf(dev_packs): packs read the romfs in place, bases may too, and runtimes clone their files #1792

Merged
HumanBean merged 1 commit from HumanBean/Longshot:perf/dev-packs-romfs-in-place into main 2026-11-19 21:33:52 +00:00
Owner

Type: perf · Risk: low · Tested on: Linux (game-free tests only) · Needs: a dev_packs pack and base on Windows

TL;DR

On the Test Farm's Windows PCs, building a pull request took about 49 minutes against 10 on Linux. Most of that was copying the game's 26 GB of game files and then comparing every one of them, both copies, against the base's. With this PR a pack reads the game files where they are and never copies them, so neither step happens. Players' builds don't change. Only the game-free tests ran; nothing has run in the game yet.

Breakdown

  • Who it helps: anyone testing changes with scripts/dev_packs.py on Windows, above all the Test Farm (Beanstalk).
  • What was wrong: a sandbox that may only read the game files can't hard-link them, so make_game.py copied the romfs, and the pack's comparison with the base hashed both copies of every file.
  • What changes: a pack's build reads the romfs where it is; a base may too (--romfs-in-place); runtimes clone their files on btrfs and XFS.
  • What does not change: players' builds (--nup, --update, or a folder build that doesn't ask for it), and the game itself.
  • How it was checked: new game-free tests for the switch and for dev_packs' bases, packs, runtimes and preparation; the touched suites pass.
  • What is still open: a real pack and base built with it, and a runtime from an in-place base started in the game (Windows first).
  • Risk: low: a change that writes into the romfs fails to build in place, and LONGSHOT_PACK_ROMFS=copy builds it as before.

Why

The farm's reports from the last 14 days (128 jobs) put a Windows game-code pack at a median 49.3 minutes (Linux 10.0) and a Windows base rebuild at 15.3 (Linux 4.2). Two parts made most of it:

  • read, 8.8–8.9 min a build: the sandbox can read the game files but not hard-link them (a link needs write access to the source), so make_game.py's link_or_copy copies all 26 GB.
  • "finish", 34.5 min a pack: the agent's stage clock counts everything after make_game's last stage as finish, and that includes dev_packs.py's generated_changes. On Linux the romfs is a bind mount of the base's, so same_file matches at once; on Windows it is a copy, so same_file SHA-hashes both files of all ~186,000 romfs files, about 52 GB read.

Beanstalk's side (#104 there) builds Windows bases with --romfs-in-place and starts their games with --romfs; it does nothing until this is on main.

What changed

  • nx2/make_game.py: --romfs-in-place (or LONGSHOT_ROMFS_IN_PLACE=1) with --romfs/--exefs: the romfs is read where it is (OOT_ROMFS, the shader list, the texts and prompts read it there) and the game folder gets none. The variable is passed over by a .nup or --update build; the switch with them is refused. OOT_ROMFS is a path variable, so the lift key and the lift store's share key are the same as before.
  • scripts/dev_packs.py:
    • pack runs make_game with LONGSHOT_ROMFS_IN_PLACE=1 (pack_env); LONGSHOT_PACK_ROMFS=copy links as before. A tree from before the switch ignores the variable and links.
    • base --romfs-in-place: no romfs in the base's game folder; base.json records romfs_in_place, and the preparation passes --romfs.
    • rt copies the runtime's other files with clone_or_copy (FICLONE where the file system can, else copy2); from an in-place base it makes no romfs and says to start the game with --romfs.
  • docs/development.md: the variable, base --romfs-in-place, and the pack's and runtime's romfs.

Measurements

Verdict: nothing measured yet: the farm's numbers above are from before. Once this and Beanstalk #104 are in, the reports' new phases (Beanstalk #103) show each Windows pack's read and finish.

Testing

Tested: tests/test_make_game_romfs_in_place.py (new: which builds read in place, the refusal, the read stage links only without it) and tests/test_dev_packs.py (new test_romfs_in_place: the pack's environment, an in-place base's base.json, preparation and runtime, the clone's bytes and time) on Linux.

Not tested: a real build. tests/test_dev_packs.py has two checks that fail on main too and are unrelated (test_caches_named_in_renderer: longshot_fsr4_pipelines.bin unclassified; test_inputs_beside_the_objects: "a game patch shows in the objects"); I ran the rest of the file around them. tests/run_all.py was not run.

Must be tested:

  • Windows: dev_packs.py base <dir> --romfs-in-place, a game pack over it, rt, and the game started with --romfs reaches the title screen.
  • Linux: a game pack over an ordinary base builds and runs as before.

Risks

What could break: a pack whose build writes into the romfs fails in place (the romfs is read-only there); a runtime of an in-place base started without --romfs has no game files.
Turning it off: LONGSHOT_PACK_ROMFS=copy for packs; leave out --romfs-in-place for bases.
Compatibility: Nothing changes for a player after updating.

Issues

None filed: found from the Test Farm's reports.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says: opened without a run in the game, at the maintainer's request.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched.
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md).

🤖 Generated with Claude Code

**Type:** perf · **Risk:** low · **Tested on:** Linux (game-free tests only) · **Needs:** a dev_packs pack and base on Windows ## TL;DR On the Test Farm's Windows PCs, building a pull request took about 49 minutes against 10 on Linux. Most of that was copying the game's 26 GB of game files and then comparing every one of them, both copies, against the base's. With this PR a pack reads the game files where they are and never copies them, so neither step happens. Players' builds don't change. Only the game-free tests ran; nothing has run in the game yet. ## Breakdown - **Who it helps:** anyone testing changes with `scripts/dev_packs.py` on Windows, above all the Test Farm (Beanstalk). - **What was wrong:** a sandbox that may only read the game files can't hard-link them, so `make_game.py` copied the romfs, and the pack's comparison with the base hashed both copies of every file. - **What changes:** a pack's build reads the romfs where it is; a base may too (`--romfs-in-place`); runtimes clone their files on btrfs and XFS. - **What does not change:** players' builds (`--nup`, `--update`, or a folder build that doesn't ask for it), and the game itself. - **How it was checked:** new game-free tests for the switch and for dev_packs' bases, packs, runtimes and preparation; the touched suites pass. - **What is still open:** a real pack and base built with it, and a runtime from an in-place base started in the game (Windows first). - **Risk:** low: a change that writes into the romfs fails to build in place, and `LONGSHOT_PACK_ROMFS=copy` builds it as before. ## Why The farm's reports from the last 14 days (128 jobs) put a Windows game-code pack at a median 49.3 minutes (Linux 10.0) and a Windows base rebuild at 15.3 (Linux 4.2). Two parts made most of it: - **read, 8.8–8.9 min a build:** the sandbox can read the game files but not hard-link them (a link needs write access to the source), so `make_game.py`'s `link_or_copy` copies all 26 GB. - **"finish", 34.5 min a pack:** the agent's stage clock counts everything after make_game's last stage as finish, and that includes `dev_packs.py`'s `generated_changes`. On Linux the romfs is a bind mount of the base's, so `same_file` matches at once; on Windows it is a copy, so `same_file` SHA-hashes both files of all ~186,000 romfs files, about 52 GB read. Beanstalk's side (#104 there) builds Windows bases with `--romfs-in-place` and starts their games with `--romfs`; it does nothing until this is on `main`. ## What changed - `nx2/make_game.py`: `--romfs-in-place` (or `LONGSHOT_ROMFS_IN_PLACE=1`) with `--romfs`/`--exefs`: the romfs is read where it is (`OOT_ROMFS`, the shader list, the texts and prompts read it there) and the game folder gets none. The variable is passed over by a `.nup` or `--update` build; the switch with them is refused. `OOT_ROMFS` is a path variable, so the lift key and the lift store's share key are the same as before. - `scripts/dev_packs.py`: - `pack` runs make_game with `LONGSHOT_ROMFS_IN_PLACE=1` (`pack_env`); `LONGSHOT_PACK_ROMFS=copy` links as before. A tree from before the switch ignores the variable and links. - `base --romfs-in-place`: no romfs in the base's game folder; `base.json` records `romfs_in_place`, and the preparation passes `--romfs`. - `rt` copies the runtime's other files with `clone_or_copy` (FICLONE where the file system can, else `copy2`); from an in-place base it makes no romfs and says to start the game with `--romfs`. - `docs/development.md`: the variable, `base --romfs-in-place`, and the pack's and runtime's romfs. ## Measurements **Verdict:** nothing measured yet: the farm's numbers above are from before. Once this and Beanstalk #104 are in, the reports' new `phases` (Beanstalk #103) show each Windows pack's read and finish. ## Testing **Tested:** `tests/test_make_game_romfs_in_place.py` (new: which builds read in place, the refusal, the read stage links only without it) and `tests/test_dev_packs.py` (new `test_romfs_in_place`: the pack's environment, an in-place base's `base.json`, preparation and runtime, the clone's bytes and time) on Linux. **Not tested:** a real build. `tests/test_dev_packs.py` has two checks that fail on `main` too and are unrelated (`test_caches_named_in_renderer`: `longshot_fsr4_pipelines.bin` unclassified; `test_inputs_beside_the_objects`: "a game patch shows in the objects"); I ran the rest of the file around them. `tests/run_all.py` was not run. **Must be tested:** - [ ] Windows: `dev_packs.py base <dir> --romfs-in-place`, a game pack over it, `rt`, and the game started with `--romfs` reaches the title screen. - [ ] Linux: a game pack over an ordinary base builds and runs as before. ## Risks **What could break:** a pack whose build writes into the romfs fails in place (the romfs is read-only there); a runtime of an in-place base started without `--romfs` has no game files. **Turning it off:** `LONGSHOT_PACK_ROMFS=copy` for packs; leave out `--romfs-in-place` for bases. **Compatibility:** Nothing changes for a player after updating. ## Issues None filed: found from the Test Farm's reports. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [ ] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says: opened without a run in the game, at the maintainer's request. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [ ] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` passes. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched. - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
perf(dev_packs): packs read the romfs in place, bases may too, and runtimes clone their files
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Failing after 6s
73ef24c18f
On Windows a sandboxed pack build cannot hard-link the romfs it may only read, so make_game.py copied all 26 GB (the
read stage, about 9 min on the farm's Windows PCs), and the pack's comparison with the base then hashed both copies of
every romfs file (the 34 min after the finish stage). make_game.py --romfs-in-place (LONGSHOT_ROMFS_IN_PLACE=1) reads
the romfs where it is: dev_packs.py sets it for its packs (LONGSHOT_PACK_ROMFS=copy links as before), base
--romfs-in-place does the same for a base, whose preparation and runtimes then name it with --romfs, and rt clones the
runtime's files (FICLONE) where the file system can.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • PR #360 feat(build): a new version's game files hard-linked from another game folder of the same .nup — Earlier build path shares game files; related hard-linking optimization
Possibly related, for context: - PR #360 feat(build): a new version's game files hard-linked from another game folder of the same .nup — Earlier build path shares game files; related hard-linking optimization
HumanBean changed title from WIP: perf(dev_packs): packs read the romfs in place, bases may too, and runtimes clone their files to perf(dev_packs): packs read the romfs in place, bases may too, and runtimes clone their files 2026-11-19 21:33:46 +00:00
HumanBean deleted branch perf/dev-packs-romfs-in-place 2026-11-19 21:33:54 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1792
No description provided.