feat(build): a new version's game files hard-linked from another game folder of the same .nup #360

Open

madebyjordan wants to merge 3 commits from madebyjordan/Longshot:feat/romfs-from into main

pull from: madebyjordan/Longshot:feat/romfs-from

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 1 Commits 3 Files changed 3 +201 -19

madebyjordan commented 2026-11-14 04:23:13 +00:00

Contributor

• Copy link
• Copy Markdown

Why

Every new version and every update a launcher builds into its own folder unpacked the player's .nup again. That's 26 GB written and 26 GB more disk space each time, for game files the PC already has unpacked beside it. With this PR, a build can hard-link them from another game folder made from the same .nup: no disk space taken and nothing written. On an NVMe drive it's also about 10% quicker; on a hard disk, where writing 26 GB is slow, it should save more (not measured).

What changed

• build: romfs_from_build() in nx2/make_game.py handles --nup with --romfs-from <folder>, or LONGSHOT_ROMFS_FROM, which a launcher sets. It runs when that folder's .longshot_romfs stamp is this .nup's: its name and size, the check a folder's own unpack is already reused on. Then:

  • its romfs is hard-linked in by link_or_copy (copied where the disk can't link);
  • its kept exefs is copied;
  • its damaged-texture check is carried over (.longshot_romfs_checked, the heads list, the damaged files' note), so stand_in_damaged_files opens nothing;
  • links in it are refused, as in --romfs.

  A folder from another .nup, one without game files, or the build's own folder is passed over, and the .nup is read as before. A folder's own earlier unpack of the same .nup still comes first.

• build: refuse_dump_links() reads each entry's attributes from its folder's listing (os.scandir) instead of an lstat per file: 36-49 s down to 1.3-1.5 s on the romfs's ~186,000 files. A folder it can't list now stops the build with a plain message. Before, os.walk passed over it, and its files went missing from the game folder without a word.

• build: link_or_copy() makes each folder once. It lets a link that's already there (from a retry) fail with FileExistsError instead of looking each file up first; a failed link still falls back to the staged copy. A romfs's top-level files go to dst itself: dst\. in a \\?\ long path isn't tidied away, so they failed to link.

• docs: docs/development.md, the LONGSHOT_ROMFS_FROM row.

• tests: tests/test_make_game_cache.py checks:

  • linked, not read: the same files, as hard links;
  • the variable;
  • another .nup, no game files, and the folder itself are passed over;
  • the folder's own unpack still comes first;
  • a junction three folders deep is refused (Windows needs no right to make one, so this runs where the symlink check is skipped);
  • an unreadable folder is reported.

  The shared set-up is NupCacheHarness.

The launcher side, which picks the folder and sets LONGSHOT_ROMFS_FROM, is a separate PR on LongshotLauncher: MagicBeanDev/LongshotLauncher#58. Without it this does nothing: a build that isn't given the folder unpacks the .nup as before, and a launcher that sets the variable for an older builder gets the old behaviour.

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
LONGSHOT_ROMFS_FROM=<folder> / --romfs-from <folder>	With --nup: another game folder unpacked from the same .nup; its game files are hard-linked in instead of reading the .nup again	unset (the .nup is read)	Set by the launcher for a new version or update of a .nup it has built before	any PC building a second version from the same .nup	a first build; folders on another drive (copied, not linked)	Linux (ext4/btrfs links), FAT/exFAT drives (copy fallback)

Measurements

main: cf605c85 (2026-11-14)
this PR: feat/romfs-from @ bcf64463
Rebased since onto main 0caf0618 (now 7f96ae5f): the numbers still hold. #357 and #358 changed the lift (nx2/chunks.py) and the prompts (nx2/prompt_assets.py), not the romfs step, and this PR's code is the same; only a row of the switch table moved.
Hardware: Ryzen 5 5600, 16 GB, NVMe, Windows 11, NTFS
Scene: none (the build). The OoT .nup (25.62 GB, 186,583 files in v131072's romfs), building a second game folder from it.
Method: a second game folder's game files, main and this PR alternated (main / PR / main / PR), each into an empty folder on the same NVMe drive as the first:

• main: what make_game.py runs for a .nup it hasn't unpacked in that folder, nx2/nup.py <file.nup> --extract;
• this PR: romfs_from_build() from a game folder already built from that .nup.

Wall time, files counted, and for the PR, every file's link count plus 188 files sampled as the same file as the source's.
How to reproduce: build a game folder from the .nup, then python -I nx2/make_game.py --nup <file.nup> --out <second folder> --romfs-from <first folder>.

	main @ cf605c85 (unpacks the .nup)	this PR @ bcf64463 (links from the first folder)
run 1	149.3 s	134.2 s
run 2	144.2 s	128.5 s
median	146.8 s	131.4 s (-10%)
written to disk	26.27 GB (186,588 files)	none: 186,583 of 186,583 files hard links; 188 of 188 sampled the same file as the source's
refuse_dump_links on that romfs (2026-11-13)	36-49 s	1.3-1.5 s

NTFS takes the links one at a time, so threads don't help: 0.48 ms a link on 1 thread, 0.63 ms on 16.

	main @ cf605c85	this PR @ bcf64463
tests/test_make_game_cache.py	passes (fewer tests: none for --romfs-from)	25 tests pass (1 skipped: symbolic links need Developer Mode)
tests/run_all.py --jobs 4	483 passed, 6 failed, 70 skipped	486 passed, 6 failed (the same 6 as main), 70 skipped

Testing

Tested:

• The romfs step of a second build on Windows (NTFS, NVMe) from a folder built from the same .nup, alternated against main, as above.
• The game started from game folders whose romfs is hard-linked from another folder: tonight's headless test runs used such folders for 150-300 s each, from New Game into Kokiri, with no difference.
• tests/test_make_game_cache.py, and python -I -B tests/run_all.py --jobs 4.

Not tested:

• Linux and a drive that can't hard-link: the code takes the copy fallback there, which the tests cover with a stand-in.
• A full launcher run of a second version: that needs the launcher PR.

Needs other testers:

• Linux: a second game folder from the same .nup with --romfs-from, expecting its romfs as hard links (stat -c %h above 1) and the game starting from it (T-FWPE).

Screenshots

Nothing on screen.

Risk and rollback

• Shared files: two game folders share their romfs files. Nothing in the build or the game writes into a romfs in place. Damaged-file stand-ins are written to a .part file and swapped in with os.replace, which breaks the link rather than writing through it. A player who edits a romfs file by hand in one folder would change the other too; docs/development.md says so.
• Rollback: unset the variable (the launcher not setting it) and the .nup is read as before. Reverting the commit removes it.

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
• Tested in the game, or what wasn't is named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• Flags noted, and the switch reference updated (docs/development.md, the README).
• A game-free test added for the fix, failing on main and passing here.
• Docs updated.
• A check under tests/ that reads changed source updated in this PR, not weakened.

## Why Every new version and every update a launcher builds into its own folder unpacked the player's .nup again. That's 26 GB written and 26 GB more disk space each time, for game files the PC already has unpacked beside it. With this PR, a build can hard-link them from another game folder made from the same .nup: no disk space taken and nothing written. On an NVMe drive it's also about 10% quicker; on a hard disk, where writing 26 GB is slow, it should save more (not measured). ## What changed - build: `romfs_from_build()` in `nx2/make_game.py` handles `--nup` with `--romfs-from <folder>`, or `LONGSHOT_ROMFS_FROM`, which a launcher sets. It runs when that folder's `.longshot_romfs` stamp is this .nup's: its name and size, the check a folder's own unpack is already reused on. Then: - its romfs is hard-linked in by `link_or_copy` (copied where the disk can't link); - its kept exefs is copied; - its damaged-texture check is carried over (`.longshot_romfs_checked`, the heads list, the damaged files' note), so `stand_in_damaged_files` opens nothing; - links in it are refused, as in `--romfs`. A folder from another .nup, one without game files, or the build's own folder is passed over, and the .nup is read as before. A folder's own earlier unpack of the same .nup still comes first. - build: `refuse_dump_links()` reads each entry's attributes from its folder's listing (`os.scandir`) instead of an lstat per file: 36-49 s down to 1.3-1.5 s on the romfs's ~186,000 files. A folder it can't list now stops the build with a plain message. Before, `os.walk` passed over it, and its files went missing from the game folder without a word. - build: `link_or_copy()` makes each folder once. It lets a link that's already there (from a retry) fail with `FileExistsError` instead of looking each file up first; a failed link still falls back to the staged copy. A romfs's top-level files go to `dst` itself: `dst\.` in a `\\?\` long path isn't tidied away, so they failed to link. - docs: `docs/development.md`, the `LONGSHOT_ROMFS_FROM` row. - tests: `tests/test_make_game_cache.py` checks: - linked, not read: the same files, as hard links; - the variable; - another .nup, no game files, and the folder itself are passed over; - the folder's own unpack still comes first; - a junction three folders deep is refused (Windows needs no right to make one, so this runs where the symlink check is skipped); - an unreadable folder is reported. The shared set-up is `NupCacheHarness`. The launcher side, which picks the folder and sets `LONGSHOT_ROMFS_FROM`, is a separate PR on LongshotLauncher: https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/58. Without it this does nothing: a build that isn't given the folder unpacks the .nup as before, and a launcher that sets the variable for an older builder gets the old behaviour. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `LONGSHOT_ROMFS_FROM=<folder>` / `--romfs-from <folder>` | With `--nup`: another game folder unpacked from the same .nup; its game files are hard-linked in instead of reading the .nup again | unset (the .nup is read) | Set by the launcher for a new version or update of a .nup it has built before | any PC building a second version from the same .nup | a first build; folders on another drive (copied, not linked) | Linux (ext4/btrfs links), FAT/exFAT drives (copy fallback) | ## Measurements main: cf605c85 (2026-11-14) this PR: feat/romfs-from @ bcf64463 Rebased since onto main 0caf0618 (now 7f96ae5f): the numbers still hold. #357 and #358 changed the lift (`nx2/chunks.py`) and the prompts (`nx2/prompt_assets.py`), not the romfs step, and this PR's code is the same; only a row of the switch table moved. Hardware: Ryzen 5 5600, 16 GB, NVMe, Windows 11, NTFS Scene: none (the build). The OoT .nup (25.62 GB, 186,583 files in v131072's romfs), building a second game folder from it. Method: a second game folder's game files, main and this PR alternated (main / PR / main / PR), each into an empty folder on the same NVMe drive as the first: - main: what `make_game.py` runs for a .nup it hasn't unpacked in that folder, `nx2/nup.py <file.nup> --extract`; - this PR: `romfs_from_build()` from a game folder already built from that .nup. Wall time, files counted, and for the PR, every file's link count plus 188 files sampled as the same file as the source's. How to reproduce: build a game folder from the .nup, then `python -I nx2/make_game.py --nup <file.nup> --out <second folder> --romfs-from <first folder>`. | | main @ cf605c85 (unpacks the .nup) | this PR @ bcf64463 (links from the first folder) | |---|---|---| | run 1 | 149.3 s | 134.2 s | | run 2 | 144.2 s | 128.5 s | | median | **146.8 s** | **131.4 s** (-10%) | | written to disk | 26.27 GB (186,588 files) | none: 186,583 of 186,583 files hard links; 188 of 188 sampled the same file as the source's | | `refuse_dump_links` on that romfs (2026-11-13) | 36-49 s | 1.3-1.5 s | NTFS takes the links one at a time, so threads don't help: 0.48 ms a link on 1 thread, 0.63 ms on 16. | | main @ cf605c85 | this PR @ bcf64463 | |---|---|---| | `tests/test_make_game_cache.py` | passes (fewer tests: none for `--romfs-from`) | 25 tests pass (1 skipped: symbolic links need Developer Mode) | | `tests/run_all.py --jobs 4` | 483 passed, 6 failed, 70 skipped | 486 passed, 6 failed (the same 6 as main), 70 skipped | ## Testing **Tested:** - The romfs step of a second build on Windows (NTFS, NVMe) from a folder built from the same .nup, alternated against main, as above. - The game started from game folders whose romfs is hard-linked from another folder: tonight's headless test runs used such folders for 150-300 s each, from New Game into Kokiri, with no difference. - `tests/test_make_game_cache.py`, and `python -I -B tests/run_all.py --jobs 4`. **Not tested:** - Linux and a drive that can't hard-link: the code takes the copy fallback there, which the tests cover with a stand-in. - A full launcher run of a second version: that needs the launcher PR. **Needs other testers:** - Linux: a second game folder from the same .nup with `--romfs-from`, expecting its romfs as hard links (`stat -c %h` above 1) and the game starting from it (T-FWPE). ## Screenshots Nothing on screen. ## Risk and rollback - **Shared files:** two game folders share their romfs files. Nothing in the build or the game writes into a romfs in place. Damaged-file stand-ins are written to a `.part` file and swapped in with `os.replace`, which breaks the link rather than writing through it. A player who edits a romfs file by hand in one folder would change the other too; `docs/development.md` says so. - **Rollback:** unset the variable (the launcher not setting it) and the .nup is read as before. Reverting the commit removes it. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] Docs updated. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.

madebyjordan added 3 commits 2026-11-14 04:23:13 +00:00

feat(build): a new version's game files hard-linked from another game folder of the same .nup (LONGSHOT_ROMFS_FROM) ... e3310a8cca

Every new version and every update a launcher builds into its own folder read the whole .nup again: ~25 GB written once
more (2:45 on a Ryzen 5 5600 with an NVMe drive), and the disk space each time, for game files the PC already has
unpacked beside it.

- make_game.py: romfs_from_build, for --nup with --romfs-from <folder> (or LONGSHOT_ROMFS_FROM, which a launcher can set:
  a builder that does not know it unpacks the .nup as before). When that folder's .longshot_romfs stamp is this .nup's
  (its name and size, the check a folder's own unpack is already reused on), its romfs is hard-linked in (link_or_copy:
  copied where the disk can't link), its kept exefs copied, and its damaged-texture check carried over
  (.longshot_romfs_checked, the heads list, the damaged files' note), so stand_in_damaged_files opens nothing. Links in it
  are refused as in --romfs. A folder from another .nup, without game files, or the build's own folder: the .nup is read.
- docs/development.md: the switch.
- tests/test_make_game_cache.py: linked, not read (same files, hard links), the variable, another .nup, no game files, the
  folder itself, and the folder's own unpack still first. The shared set-up is NupCacheHarness.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

perf(build): a romfs's links checked from the folders' listings, and linked with no lookup per file ... e792505f4a

Linking a romfs's ~186,000 files (--romfs, and now --romfs-from) took longer than unpacking the .nup did: 188 s against
165 s on a Ryzen 5 5600 with an NVMe drive, Windows 11. Most of it was not the links.

- refuse_dump_links reads each entry's attributes from its folder's listing (os.scandir: on Windows the listing carries
  them) instead of an lstat per file: 36-49 s to 1.3-1.5 s on that romfs.
- link_or_copy makes each folder once and lets a link that is there already (a retry) fail with FileExistsError instead of
  looking each file up first; a failed link still falls back to the staged copy. A romfs's top-level files go to dst itself:
  dst\. in a \?\ long path is not tidied away, and they failed to link.
- The whole --romfs-from of that romfs: 188 s to 110-140 s (the game was preparing its shaders beside it), ~256 MB of disk
  against ~25 GB for an unpack. Threads don't help: NTFS takes the links one at a time (0.48 ms each on 1 thread, 0.63 on 16).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(build): a game-files folder that cannot be read is said, not passed over; a junction in a dump is tested ... bcf64463d2

refuse_dump_links lists the folders itself now (os.scandir). One it cannot list stopped the build with a traceback;
os.walk before passed over it, and its files were then missing from the game folder without a word. It now says which
folder could not be read and why. tests/test_make_game_cache.py: a junction three folders down is refused (Windows
needs no right to make one, so the check runs where the symbolic-link one is skipped), and an unreadable folder is said.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

madebyjordan referenced this pull request from MagicBeanDev/LongshotLauncher 2026-11-14 04:24:13 +00:00

feat(build): a new version's game files linked from a version built from the same .nup #58

madebyjordan changed title from WIP: feat(build): a new version's game files hard-linked from another game folder of the same .nup to feat(build): a new version's game files hard-linked from another game folder of the same .nup 2026-11-14 04:24:55 +00:00

madebyjordan referenced this pull request 2026-11-14 04:25:32 +00:00

perf(build): a first build in under 10 minutes: what was measured, and a proposal (a clang-free cold tier) #362

mooki added the

kind

feature

priority

low

labels 2026-11-14 04:31:55 +00:00

madebyjordan force-pushed feat/romfs-from from bcf64463d2 to 7f96ae5f5a 2026-11-14 07:15:58 +00:00 Compare

madebyjordan commented 2026-11-14 07:16:05 +00:00

Author

Contributor

• Copy link
• Copy Markdown

Rebased onto main 0caf0618 (#357 and #358 merged). The only conflict was the switch table in docs/development.md: both rows kept. The code is unchanged; test_make_game_cache (25 tests, 1 skipped for symbolic links), test_build_times, test_lift_cleanup_error and test_kept_translation pass on the new base.

Rebased onto main 0caf0618 (#357 and #358 merged). The only conflict was the switch table in docs/development.md: both rows kept. The code is unchanged; test_make_game_cache (25 tests, 1 skipped for symbolic links), test_build_times, test_lift_cleanup_error and test_kept_translation pass on the new base.

mooki added the

area/install

label 2026-11-14 16:27:44 +00:00

Some checks are pending

Hide checks

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Website / Website on Cloudflare (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.

Some workflows are waiting to be reviewed.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'feat/romfs-from:madebyjordan-feat/romfs-from'

git switch 'madebyjordan-feat/romfs-from'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Linux only

  

platform/steam-deck

Steam Deck

  

platform/windows

Windows only

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!360

No description provided.