perf(dev_packs): packs read the romfs in place, bases may too, and runtimes clone their files #1792
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1792
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:perf/dev-packs-romfs-in-place"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: low · Tested on: Linux (game-free tests only) · Needs: a dev_packs pack and base on Windows
TL;DR
On the Test Farm's Windows PCs, building a pull request took about 49 minutes against 10 on Linux. Most of that was copying the game's 26 GB of game files and then comparing every one of them, both copies, against the base's. With this PR a pack reads the game files where they are and never copies them, so neither step happens. Players' builds don't change. Only the game-free tests ran; nothing has run in the game yet.
Breakdown
scripts/dev_packs.pyon Windows, above all the Test Farm (Beanstalk).make_game.pycopied the romfs, and the pack's comparison with the base hashed both copies of every file.--romfs-in-place); runtimes clone their files on btrfs and XFS.--nup,--update, or a folder build that doesn't ask for it), and the game itself.LONGSHOT_PACK_ROMFS=copybuilds it as before.Why
The farm's reports from the last 14 days (128 jobs) put a Windows game-code pack at a median 49.3 minutes (Linux 10.0) and a Windows base rebuild at 15.3 (Linux 4.2). Two parts made most of it:
make_game.py'slink_or_copycopies all 26 GB.dev_packs.py'sgenerated_changes. On Linux the romfs is a bind mount of the base's, sosame_filematches at once; on Windows it is a copy, sosame_fileSHA-hashes both files of all ~186,000 romfs files, about 52 GB read.Beanstalk's side (#104 there) builds Windows bases with
--romfs-in-placeand starts their games with--romfs; it does nothing until this is onmain.What changed
nx2/make_game.py:--romfs-in-place(orLONGSHOT_ROMFS_IN_PLACE=1) with--romfs/--exefs: the romfs is read where it is (OOT_ROMFS, the shader list, the texts and prompts read it there) and the game folder gets none. The variable is passed over by a.nupor--updatebuild; the switch with them is refused.OOT_ROMFSis a path variable, so the lift key and the lift store's share key are the same as before.scripts/dev_packs.py:packruns make_game withLONGSHOT_ROMFS_IN_PLACE=1(pack_env);LONGSHOT_PACK_ROMFS=copylinks as before. A tree from before the switch ignores the variable and links.base --romfs-in-place: no romfs in the base's game folder;base.jsonrecordsromfs_in_place, and the preparation passes--romfs.rtcopies the runtime's other files withclone_or_copy(FICLONE where the file system can, elsecopy2); from an in-place base it makes no romfs and says to start the game with--romfs.docs/development.md: the variable,base --romfs-in-place, and the pack's and runtime's romfs.Measurements
Verdict: nothing measured yet: the farm's numbers above are from before. Once this and Beanstalk #104 are in, the reports' new
phases(Beanstalk #103) show each Windows pack's read and finish.Testing
Tested:
tests/test_make_game_romfs_in_place.py(new: which builds read in place, the refusal, the read stage links only without it) andtests/test_dev_packs.py(newtest_romfs_in_place: the pack's environment, an in-place base'sbase.json, preparation and runtime, the clone's bytes and time) on Linux.Not tested: a real build.
tests/test_dev_packs.pyhas two checks that fail onmaintoo and are unrelated (test_caches_named_in_renderer:longshot_fsr4_pipelines.binunclassified;test_inputs_beside_the_objects: "a game patch shows in the objects"); I ran the rest of the file around them.tests/run_all.pywas not run.Must be tested:
dev_packs.py base <dir> --romfs-in-place, a game pack over it,rt, and the game started with--romfsreaches the title screen.Risks
What could break: a pack whose build writes into the romfs fails in place (the romfs is read-only there); a runtime of an in-place base started without
--romfshas no game files.Turning it off:
LONGSHOT_PACK_ROMFS=copyfor packs; leave out--romfs-in-placefor bases.Compatibility: Nothing changes for a player after updating.
Issues
None filed: found from the Test Farm's reports.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses.third_party/untouched.When it applies
Assisted-by:/Co-Authored-By:trailers.🤖 Generated with Claude Code
Possibly related, for context:
WIP: perf(dev_packs): packs read the romfs in place, bases may too, and runtimes clone their filesto perf(dev_packs): packs read the romfs in place, bases may too, and runtimes clone their files