fix(build): the build runs from, in and into folders past 260 characters #1132
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1132
Loading…
Reference in a new issue
No description provided.
Delete branch "enum/Longshot:fix/version-long-paths"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: medium · Tested on: Windows 11, RTX 4070 · Needs: none
Why
The Windows build stops at "Compiling" with
FileNotFoundErrorwhen the Builder sits deep in the file system, such asa portable Longshot Launcher in Downloads. At that depth the deepest sources pass Windows' 260-character path limit
(MAX_PATH), and the build's own Python reads them by plain paths. Most PCs have Windows' long-paths setting off, so the
build can't finish. A work folder or a game folder deep enough fails the same way further on: the objects are named
after their sources (up to ~90 characters), and their stamps, the object cache and the link record are read and written
by plain paths too. Fixes #1115.
How to reproduce
Seen on: reported on Windows (#1115: a portable Launcher, nightly-346-8a4aa282). Here: Windows 11 Pro (26300),
RTX 4070, driver 616.56, long paths off (
LongPathsEnabled=0, the default)Settings and switches: defaults
Steps: put the Builder's tools folder 176 characters deep, as the report's
~\Downloads\QuiverLauncher-win-Portable\...\LongshotBuilder\tools\is with an 8-letter user name, and build from it.Without the Launcher: unpack
git archive origin/maininto a folder that deep, thenpython -I <that folder>\nx2\make_game.py --nup <your .nup> --out <game> --work <work>Expected: the build goes through
Actual: it stops in
version.define()→fingerprint(), the report's traceback:FileNotFoundError: [Errno 2] No such file or directory: '...\gpu\third_party\suyu\shader_recompiler\backend\glsl\emit_glsl_bitwise_conversion.cpp'.With the work folder 180 characters deep as well, a build that gets past that stops in
compile_onewriting...\sa\build\obj\vk_eden_shader_recompiler_backend_spirv_emit_spirv_barriers.o.sha1.What changed
The build's Python reaches paths past 260 characters on Windows the way Python itself allows without Windows'
long-paths setting: in the extended-length form,
\\?\C:\.... A new module does it for the standard file calls, and eachscript the build runs installs it as it loads, so nothing in the build has to remember it. Only a path 248 characters or
more once absolute changes; for every path that worked before, the calls behave as they did. clang and the linker take
long paths already.
nx2/long_paths.py(new):install()wraps open (withos.open, and the copiestarfile/bz2and the like kept asthey loaded),
os.statand its kin,os.path.existsand its kin, the renames and links,shutil's copies andcopytree. Each wrapper changes only the path: no argument, keywords,dir_fdandos.supports_*stay as theywere, and what comes back is in the caller's spelling (scandir entries,
os.walkeither way round,shutil'sdestinations).
os.walk,makedirs,rmtree,tempfile,pathlib,glob,zipfileandtarfilefollow. OffWindows it does nothing.
make_game.py,build.py,lift.py(its spawned workers load it again, so they install too),nup.py,unpack.py,split_hot.py,shader_census.py,text_assets.py,prompt_assets.py,third_party_objects.pycalllong_paths.install()before anything reads a file.nx2/version.py(long_path(),walk(), used byfingerprint()) andnx2/translator_files.py(files(),stamp()) keep long paths of their own, becausescripts/dev_packs.py, the bake tools and the checks load themwithout the build. A folder skipped by the stamp's walk would change the shader stamp, and with it every player's
shader caches.
tests/test_prompt_skip.py: the prompt step importslong_pathswithout it in the prompt key, as withkit;neither changes what is drawn. The check still fails for any other unfingerprinted import.
docs/development.md(Layout) says every build script installs it, and a new one joins the per-script check;docs/worklist.mdhas a row for #1115 (S0, provisional reach 0.05, confidence 0.4), owner enum.I first fixed the tools folder site by site (10 sites). Making the work folder and the game folder deep showed about
100 more file calls across
build.py,make_game.pyand the scripts they start, so one module at every entry replacedthe per-site edits. Two cold reviews by a separate agent went over it. The second one found short-path regressions in
the first version of the module (
os.listdir()with no argument,open(file=...),os.supports_*membership) andcalls it missed (
os.openthroughtempfile,copytree's defaultcopy2,tarfile's andbz2's ownopen). Allof those are fixed here, and each has a case in
tests/test_long_paths.py.Measurements
Verdict: passes. The new checks fail on
mainand pass here. The whole game builds and plays with the Builder, itswork folder and its game folder 176-180 characters deep. For a tree that already worked, the fingerprint and the
shader stamp are unchanged.
Hardware: Windows 11 Pro (26300), RTX 4070 (616.56), Ryzen 9 7900X, 64 GB; llvm-mingw 23.1.5, Python 3.12.10; long
paths off
Scene: Kokiri Forest, the repository's
kokiri-startsave, headless (longshot_ctl.py launch --profile test)Method: the checks;
nx2/make_game.pyrun fromgit archiveof each commit, unpacked 176 characters deep, with--workand--outbeside it (180), throughscripts/job_queue.py --mode build; then the game folder started with thecontrol bridge, pressed into play, two screenshots, a stick walk, a stop
How to reproduce: the commands under How to reproduce, with this PR's commit in place of
origin/main71b1edd848432422nx2/test_version_long_paths.pytests/test_build_long_paths.pytests/test_long_paths.pyFileNotFoundErrorinfingerprint(); past it, incompile_onerun_all.py(Windows)Main's two extra suite failures (
test_bug_report,test_hardware_report) came from running it in a throwawayworktree; neither reads a file this PR changes.
Flags and settings
None. (
LONGSHOT_VERSIONappears only innx2/test_version_long_paths.py, which clears it so the check reads the release from the tree, asnx2/test_version.pydoes.)Testing
Tested:
version, stamp, prompt and build fixture checks pass; the per-script check skips there (the module does nothing off
Windows, and loading every script needs the build's packages).
(work and game folders short), and one built with the Builder, the work folder and the game folder 176-180
characters deep, with the first one's shader caches copied in. Both pressed through the title into Kokiri Forest at
about 60 fps, walked, and stopped cleanly, with no device lost and no error in
longshot_log.txt.script's case;
os.openunwrapped fails its case; each of the first version's 10 explicit sites reverted alonefailed a case.
Not tested: the real portable Launcher, and the reporter's own PC. PCs with
LongPathsEnabled=1in the registry: plain paths work there already, so this PR changesnothing for them. A current folder past ~258 characters (
subprocess'scwd=,os.chdir): Windows takes none in anyform, and the build runs at most in the game folder.
Must be tested: None
Risks
What could break: the build's file calls on Windows. The module patches
openand theos/shutilcallsprocess-wide in the build's own scripts. A path under 248 characters goes through unchanged, and the calls' conventions
(no argument, keywords,
dir_fd,os.supports_*) are kept and tested. Only the standard library's earlier copies ofopenare swept; code of ours keeps what it holds. Linux and macOS: nothing changes.Turning it off: revert the commit
Compatibility: Nothing changes for a player whose folders worked: the same fingerprint, shader stamp and object
keys, so no shader cache is dropped and nothing extra is compiled. The prompt key is unchanged.
Issues
Fixes #1115
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Tested in game: Windows 11 Pro (26300), RTX 4070, NVIDIA 616.56, long paths off: built with make_game.py from this commit unpacked 176 characters deep (the report's depth), with the work folder and the game folder 180 characters deep too (0:06:30, 0 failed); played that game folder headless on the repo's kokiri-start save: through the title into Kokiri Forest at ~60 fps, a stick walk, a clean stop, no device lost or file error in longshot_log.txt. Also played a game folder built by a deep Builder with short work and game folders, the same. Main stops on the same deep Builder with this issue's FileNotFoundError in fingerprint().
Possibly related, for context:
WIP: fix(build): the build runs from, in and into folders past 260 charactersto fix(build): the build runs from, in and into folders past 260 charactersNot a duplicate of #151, checked against its diff: #151 changed only the Windows setup (
builder/setup/setup.c,long_path.h), so the setup can copy the Builder out of a zip unpacked deeper than MAX_PATH. This PR is the next step: the Builder once installed there, its Python build (make_game.py,build.py,lift.pyand the steps they start) reading its sources and writing its work and game folders at that depth. Nothing here touchesbuilder/setup/, and #151 touched no Python.Prior work otherwise: none found in Longshot's open, merged and closed pull requests and issues (MAX_PATH, long path, 260 characters, extended-length, LongPathsEnabled), or in Hookshot's (MAX_PATH and LongPathsEnabled: none).
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.