fix(scripts): the pipeline tour copies the game folder on ext4 too: romfs hard-linked, the rest copied #898
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8015c9a
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!898
Loading…
Reference in a new issue
No description provided.
Delete branch "jabbaxipil/Longshot:ext4-tour-copy"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux (Ubuntu 26.04.1, ext4), RTX 4070 · Needs: Linux on btrfs or XFS, any GPU
Why
pipeline_tour.py touronly runs where the game folder can be copied withcp -a --reflink=always, which means btrfs or XFS. On any other drive it stops before the first game. That covers ext4, the default on Ubuntu, Debian, Fedora Workstation and Mint, and it means most Linux PCs can't run the tour at all. mooki's review of #111 flagged this and suggested--reflink=auto. But that makes a full 36 GB copy for every tour, which the refusal message says is deliberately avoided.The tour needs a copy because it moves the pipeline lists aside, rewrites
longshot_options.ini, and plays games that writecache/in place. But 25 GB of the 36 GB isromfs/, which the game only reads. So a copy that sharesromfs/and really copies the rest keeps the game folder just as safe, without the full-copy cost.See #111. Related: #360 hard-links
romfs/between game folders made from the same .nup (the builder side).How to reproduce
Seen on: Linux (Ubuntu 26.04.1, ext4), RTX 4070, driver 615.71.09; also on a second Linux PC on ext4
Settings and switches: defaults
Steps: with the game folder on an ext4 drive, run
python -I scripts/pipeline_tour.py tour --game <game folder> --out <tour dir> --name play(mainf637b42d). Separately, runpython -I -B tests/test_pipeline_tour.py.Expected: the tour plays its route in a copy of the game folder, and the test passes.
Actual:
cpprintsfailed to clone '<game folder>-tour/...': Operation not supportedfor each file, then the tour exits 1 before the first game withpipeline_tour: the game folder could not be copied with --reflink=always (a copy-on-write copy, on btrfs or XFS): put it on such a drive, as a full copy of the game per tour is not made. The test fails on every run withpipeline tour: calm took a stale snapshot (game.frame 2635, reply 2637) for control.What changed
When the reflink copy fails, the tour now hard-links
romfs/and really copies everything else, about 12 GB. This is the layoutscripts/dev_packs.pyalready uses for its runtimes. After the tour, it checks that noromfs/file changed, and fails if one did. On btrfs and XFS nothing changes: the reflink copy is tried first, exactly as before.copy_game()inscripts/pipeline_tour.pyreplaces the inlinecp -a --reflink=always. When that copy fails:romfs/is a link or on another drive (a hard link can't cross drives, and copying 25 GB without saying so would be worse), or if the drive lacks room for the rest plusCOPY_SPARE(2 GB).cp -alforromfs/andcp -a --reflink=autofor each other entry.bin/andcache/are real files: the game finds its folder through/proc/self/exe, and the program cache is appended to in place (vk_progcache.cpp). It printstour: no reflinks on this drive: ... gets romfs/ hard-linked and N GB copied.romfs_stamp()records eachromfs/file's size, mtime and inode. Intour()'sfinally,romfs_changed()compares them and names any changed file, and the tour exits 1 if one changed.Nonefromcopy_game(), so there's nothing to check afterwards.game_copy()intests/test_pipeline_tour.pyforces the no-reflink path on a small made-up game folder. It checks thatromfs/is hard-linked, thatbin/longshot, the program cache and the options file are files of their own, that both refusals happen, and that a write into the sharedromfs/is found.STALE_FRAMES + 1frames old, not 1. The tour'sfresh()accepts a snapshot up toSTALE_FRAMES(10) behind, so a 1-frame-old snapshot was taken as fresh.calm()then returned at once, and the test failed on every run withcalm took a stale snapshot (game.frame 2635, reply 2637) for control.docs/development.mdand two docstrings describe the copy instead of "on btrfs or XFS, and refused".Measurements
Verdict: the tour now runs on ext4: the copy takes about 15 s and 12.3 GB instead of being refused, and the full tour passed with
romfs/unchanged.Hardware: Linux (Ubuntu 26.04.1, kernel 7.0), RTX 4070 12 GB, driver 615.71.09, Ryzen 7 5800X3D, 64 GB, NVMe, ext4
Scene: the tour's own route and settings (640x360, game upscaler at 50%)
Method: the tour's own output, and
find -newermton the source folder and itsromfs/afterwardsHow to reproduce: a game folder built from main on an ext4 drive, then
python -I scripts/pipeline_tour.py tour --game <game folder> --out <tour dir> --name playf637b42de8c1c19dromfs/hard-linked (186,583 files), 12.3 GB copiedromfs/files changed by the tourThe game runs were made with this change on main
a4ead95c(commit 3f9b3900), before the rebase ontof637b42d. The 48 commits between them don't touchscripts/pipeline_tour.pyortests/test_pipeline_tour.py.Flags and settings
None.
Testing
Tested: all on Linux (Ubuntu 26.04.1, ext4), RTX 4070, driver 615.71.09, Python 3.14.4.
python -I -B tests/test_pipeline_tour.py: passes 3 of 3 runs ate8c1c19d. It fails on main every time with the stale-snapshot message above.game_copycheck fails on two deliberately broken versions:cache/hard-linked instead of copied (bin/longshot in the copy is not a file of its own), and theromfs/check switched off (a write into the hard-linked romfs/ was not found).python -I -B tests/run_all.pyate8c1c19d: 529 passed, 16 failed, 78 skipped. The same 16 fail on mainf637b42dhere:test_appimage_player_build,test_bake_portability(nozstandardhere) andtest_crash_dialog(clang 21 rejects-x none)test_crt_options,test_env_on,test_lift_cleanup_error,test_linux_preflightandtest_linux_updatetest_make_game_cache,test_missing_nup,test_mods_size_table,test_options_budget_onceandtest_readback_latetest_vk_occlusion_queries,test_vk_pgoandtest_walk_keycopy_game()on a real 36 GB game folder took 15 s.romfs/files share one inode with the source (3 links), andbin/longshot,cache/longshot_programs.binandlongshot_options.inihave inodes of their own.romfs/(fs: mounted content: -> <game folder>-tour/romfs).romfs_changed()found 0 of 186,583 files changed. The game wrote 16 files, all in the copy'scache/, and none in the source folder.--parts opening,child-overworld --stops 2): exit 0, with 2,269 pipelines merged. The copy was removed, and 0 files changed in the source folder.romfs/check passed, the copy was removed, 0 files changed in the source folder, and there were no GPU faults.options: the settings in <game folder>-tour/longshot_options.ini were copied to <run>/data/user/longshot_options.ini. That file is identical to what the tour set.Not tested:
cp -a --reflink=alwaysas on main, but nowcopy_game()makes it, andromfs_changed()getsNone.cpthere either way, so it is Linux-only, as before.Must be tested:
python -I scripts/pipeline_tour.py tour --game GAME --out OUT --name t --parts opening --stops 1(GAME the game folder, OUT a new folder), expecting nono reflinks on this driveline, exit 0 andGAME-tourremoved at the end (by hand: a tester with such a drive, reading the tour's output)Risks
What could break: a tour on a drive without reflinks. A write into
romfs/during a tour would also land in the source folder, because the files are shared. The game mountsromfs/read-only and refuses writes to it (host/nn_fs.c), andromfs_changed()would name any file that changed anyway. On btrfs and XFS nothing changes.Turning it off: revert the commit.
Compatibility: Nothing changes. Only
scripts/pipeline_tour.pyand its test change; the game, the builds and the players' files are untouched.Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
WIP: fix(scripts): the pipeline tour copies the game folder on ext4 too: romfs hard-linked, the rest copiedto fix(scripts): the pipeline tour copies the game folder on ext4 too: romfs hard-linked, the rest copiede8c1c19d6a5b5b1403775b5b140377691d85a681View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.