fix(scripts): the pipeline tour copies the game folder on ext4 too: romfs hard-linked, the rest copied #898
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!898
Loading…
Reference in a new issue
No description provided.
Delete branch "jabbaxipil/Longshot:ext4-tour-copy"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux (Ubuntu 26.04.1, ext4), RTX 4070 · Needs: Linux on btrfs or XFS, any GPU
Why
pipeline_tour.py touronly runs where the game folder can be copied withcp -a --reflink=always, which means btrfs or XFS. On any other drive it stops before the first game. That covers ext4, the default on Ubuntu, Debian, Fedora Workstation and Mint, and it means most Linux PCs can't run the tour at all. mooki's review of #111 flagged this and suggested--reflink=auto. But that makes a full 36 GB copy for every tour, which the refusal message says is deliberately avoided.The tour needs a copy because it moves the pipeline lists aside, rewrites
longshot_options.ini, and plays games that writecache/in place. But 25 GB of the 36 GB isromfs/, which the game only reads. So a copy that sharesromfs/and really copies the rest keeps the game folder just as safe, without the full-copy cost.See #111. Related: #360 hard-links
romfs/between game folders made from the same .nup (the builder side).How to reproduce
Seen on: Linux (Ubuntu 26.04.1, ext4), RTX 4070, driver 615.71.09; also on a second Linux PC on ext4
Settings and switches: defaults
Steps: with the game folder on an ext4 drive, run
python -I scripts/pipeline_tour.py tour --game <game folder> --out <tour dir> --name play(main71b1edd8).Expected: the tour plays its route in a copy of the game folder.
Actual:
cpprintsfailed to clone '<game folder>-tour/...': Operation not supportedfor each file, then the tour exits 1 before the first game withpipeline_tour: the game folder could not be copied with --reflink=always (a copy-on-write copy, on btrfs or XFS): put it on such a drive, as a full copy of the game per tour is not made.What changed
When the reflink copy fails, the tour now hard-links
romfs/and really copies everything else, about 12 GB. This is the layoutscripts/dev_packs.pyalready uses for its runtimes. After the tour, it checks that noromfs/file changed, and fails if one did. On btrfs and XFS nothing changes: the reflink copy is tried first, exactly as before.copy_game()inscripts/pipeline_tour.pyreplaces the inlinecp -a --reflink=always. When that copy fails:romfs/is a link or on another drive (a hard link can't cross drives, and copying 25 GB without saying so would be worse), or if the drive lacks room for the rest plusCOPY_SPARE(2 GB).cp -alforromfs/andcp -a --reflink=autofor each other entry.bin/andcache/are real files: the game finds its folder through/proc/self/exe, and the program cache is appended to in place (vk_progcache.cpp). It printstour: no reflinks on this drive: ... gets romfs/ hard-linked and N GB copied.romfs_stamp()records eachromfs/file's size, mtime and inode. Intour()'sfinally,romfs_changed()compares them and names any changed file, and the tour exits 1 if one changed.Nonefromcopy_game(), so there's nothing to check afterwards.game_copy()intests/test_pipeline_tour.pyforces the no-reflink path on a small made-up game folder. It checks thatromfs/is hard-linked, thatbin/longshot, the program cache and the options file are files of their own, that both refusals happen, and that a write into the sharedromfs/is found.docs/development.mdand two docstrings describe the copy instead of "on btrfs or XFS, and refused".Measurements
Verdict: the tour now runs on ext4: the copy takes about 15 s and 12.3 GB instead of being refused, and the full tour passed with
romfs/unchanged.Hardware: Linux (Ubuntu 26.04.1, kernel 7.0), RTX 4070 12 GB, driver 615.71.09, Ryzen 7 5800X3D, 64 GB, NVMe, ext4
Scene: the tour's own route and settings (640x360, game upscaler at 50%)
Method: the tour's own output, and
find -newermton the source folder and itsromfs/afterwardsHow to reproduce: a game folder built from main on an ext4 drive, then
python -I scripts/pipeline_tour.py tour --game <game folder> --out <tour dir> --name play71b1edd8658c2c2cromfs/hard-linked (186,583 files), 12.3 GB copiedromfs/files changed by the tourThe game runs were made with this change at 3f9b3900 (on main
a4ead95c).658c2c2c, on main71b1edd8, has a byte-identicalscripts/pipeline_tour.py; main's only change to the tour's files since then is #627's test line.Flags and settings
None.
Testing
Tested: all on Linux (Ubuntu 26.04.1, ext4), RTX 4070, driver 615.71.09, Python 3.14.4.
python -I -B tests/test_pipeline_tour.py: passes 3 of 3 runs at658c2c2c.game_copycheck fails on two deliberately broken versions:cache/hard-linked instead of copied (bin/longshot in the copy is not a file of its own), and theromfs/check switched off (a write into the hard-linked romfs/ was not found).python -I -B tests/run_all.pyat658c2c2c: 575 passed, 17 failed, 85 skipped. The same 17 fail on main71b1edd8here:test_audio_output_devices,test_bug_report,test_bug_report_defaults,test_crt_optionsandtest_dlss_feature_sizetest_failed_compile_keeps_objects,test_guest_imported,test_hardware_report,test_linux_preflightandtest_linux_updatetest_make_game_cache,test_missing_nup,test_mods_size_tableandtest_pending_guest_importtest_run_all_changed,test_vk_occlusion_queriesandtest_walk_keycopy_game()on a real 36 GB game folder took 15 s.romfs/files share one inode with the source (3 links), andbin/longshot,cache/longshot_programs.binandlongshot_options.inihave inodes of their own.romfs/(fs: mounted content: -> <game folder>-tour/romfs).romfs_changed()found 0 of 186,583 files changed. The game wrote 16 files, all in the copy'scache/, and none in the source folder.--parts opening,child-overworld --stops 2): exit 0, with 2,269 pipelines merged. The copy was removed, and 0 files changed in the source folder.romfs/check passed, the copy was removed, 0 files changed in the source folder, and there were no GPU faults.options: the settings in <game folder>-tour/longshot_options.ini were copied to <run>/data/user/longshot_options.ini. That file is identical to what the tour set.Not tested:
cp -a --reflink=alwaysas on main, but nowcopy_game()makes it, andromfs_changed()getsNone.cpthere either way, so it is Linux-only, as before.Must be tested:
python -I scripts/pipeline_tour.py tour --game GAME --out OUT --name t --parts opening --stops 1(GAME the game folder, OUT a new folder), expecting nono reflinks on this driveline, exit 0 andGAME-tourremoved at the end (by hand: a tester with such a drive, reading the tour's output)Risks
What could break: a tour on a drive without reflinks. A write into
romfs/during a tour would also land in the source folder, because the files are shared. The game mountsromfs/read-only and refuses writes to it (host/nn_fs.c), andromfs_changed()would name any file that changed anyway. On btrfs and XFS nothing changes.Turning it off: revert the commit.
Compatibility: Nothing changes. Only
scripts/pipeline_tour.pyand its test change; the game, the builds and the players' files are untouched.Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
WIP: fix(scripts): the pipeline tour copies the game folder on ext4 too: romfs hard-linked, the rest copiedto fix(scripts): the pipeline tour copies the game folder on ext4 too: romfs hard-linked, the rest copiede8c1c19d6a5b5b1403775b5b140377691d85a681Code-only review of
691d85a681; informational findings, with final approval left to another reviewer.No new correctness defect found in the copy fallback. I checked the complete diff, the tour's cleanup and per-run file mutations, the read-only RomFS mount/write rejection, and the new behavioral test. For ordinary files, the fallback shares only RomFS and independently copies the mutable cache, executable and options. The stamp is a detection mechanism after the run, not write protection; the read-only guest mount is what prevents ordinary guest writes.
The author's ext4 full-tour and file-isolation results cover the new path. The explicitly required short tour on btrfs/XFS remains outstanding; the new test deliberately forces reflink failure and cannot cover that successful path. No tests, builds or game runs were performed for this review.
Forge currently reports this branch as not mergeable. During integration, preserve the stale-snapshot correction already merged through #627 (including its explanatory comment), alongside this PR's copy tests. No other substantive overlap was identified in the cached open-PR inventory.
691d85a681658c2c2c63Thanks for the review, @bibiburl. Rebased onto main
71b1edd8(head658c2c2c) to resolve the conflict: #627's stale-snapshot line and its comment are kept as merged, and this PR no longer changes that line, so it carries only the copy fallback, its test and the docs. scripts/pipeline_tour.py is byte-identical to the commit the game runs used. tests/test_pipeline_tour.py passes 3 of 3, and both deliberately broken versions still fail it; run_all's 17 failures here also fail on main71b1edd8. main's refusal on ext4 reproduced again at71b1edd8. The btrfs/XFS run is still open under Must be tested.658c2c2c630315746fd9Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.