fix(build): the build runs from, in and into folders past 260 characters #1132
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1132
Loading…
Reference in a new issue
No description provided.
Delete branch "enum/Longshot:fix/version-long-paths"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: medium · Tested on: Windows 11, RTX 4070 · Needs: none
Why
The Windows build stops at "Compiling" with
FileNotFoundErrorwhen the Builder sits deep in the file system, such asa portable Longshot Launcher in Downloads. At that depth the deepest sources pass Windows' 260-character path limit
(MAX_PATH), and the build's own Python reads them by plain paths. Most PCs have Windows' long-paths setting off, so the
build can't finish. A work folder or a game folder deep enough fails the same way further on: the objects are named
after their sources (up to ~90 characters), and their stamps, the object cache and the link record are read and written
by plain paths too. Fixes #1115.
How to reproduce
Seen on: reported on Windows (#1115: a portable Launcher, nightly-346-8a4aa282). Here: Windows 11 Pro (26300),
RTX 4070, driver 616.56, long paths off (
LongPathsEnabled=0, the default)Settings and switches: defaults
Steps: put the Builder's tools folder 176 characters deep, as the report's
~\Downloads\QuiverLauncher-win-Portable\...\LongshotBuilder\tools\is with an 8-letter user name, and build from it.Without the Launcher: unpack
git archive origin/maininto a folder that deep, thenpython -I <that folder>\nx2\make_game.py --nup <your .nup> --out <game> --work <work>Expected: the build goes through
Actual: it stops in
version.define()→fingerprint(), the report's traceback:FileNotFoundError: [Errno 2] No such file or directory: '...\gpu\third_party\suyu\shader_recompiler\backend\glsl\emit_glsl_bitwise_conversion.cpp'.With the work folder 180 characters deep as well, a build that gets past that stops in
compile_onewriting...\sa\build\obj\vk_eden_shader_recompiler_backend_spirv_emit_spirv_barriers.o.sha1.What changed
The build's Python reaches paths past 260 characters on Windows the way Python itself allows without Windows'
long-paths setting: in the extended-length form,
\\?\C:\.... A new module does it for the standard file calls, and eachscript the build runs installs it as it loads, so nothing in the build has to remember it. Only a path 248 characters or
more once absolute changes; for every path that worked before, the calls behave as they did. clang and the linker take
long paths already.
nx2/long_paths.py(new):install()wraps open (withos.open, and the copiestarfile/bz2and the like kept asthey loaded),
os.statand its kin,os.path.existsand its kin, the renames and links,shutil's copies andcopytree. Each wrapper changes only the path: no argument, keywords,dir_fdandos.supports_*stay as theywere, and what comes back is in the caller's spelling (scandir entries,
os.walkeither way round,shutil'sdestinations).
os.walk,makedirs,rmtree,tempfile,pathlib,glob,zipfileandtarfilefollow. OffWindows it does nothing.
make_game.py,build.py,lift.py(its spawned workers load it again, so they install too),nup.py,unpack.py,split_hot.py,shader_census.py,text_assets.py,prompt_assets.py,third_party_objects.pycalllong_paths.install()before anything reads a file.nx2/version.py(long_path(),walk(), used byfingerprint()) andnx2/translator_files.py(files(),stamp()) keep long paths of their own, becausescripts/dev_packs.py, the bake tools and the checks load themwithout the build. A folder skipped by the stamp's walk would change the shader stamp, and with it every player's
shader caches.
tests/test_prompt_skip.py: the prompt step importslong_pathswithout it in the prompt key, as withkit;neither changes what is drawn. The check still fails for any other unfingerprinted import.
docs/development.md(Layout) says every build script installs it, and a new one joins the per-script check;docs/worklist.mdhas a row for #1115 (S0, provisional reach 0.05, confidence 0.4), owner enum.I first fixed the tools folder site by site (10 sites). Making the work folder and the game folder deep showed about
100 more file calls across
build.py,make_game.pyand the scripts they start, so one module at every entry replacedthe per-site edits. Two cold reviews by a separate agent went over it. The second one found short-path regressions in
the first version of the module (
os.listdir()with no argument,open(file=...),os.supports_*membership) andcalls it missed (
os.openthroughtempfile,copytree's defaultcopy2,tarfile's andbz2's ownopen). Allof those are fixed here, and each has a case in
tests/test_long_paths.py.Measurements
Verdict: passes. The new checks fail on
mainand pass here. The whole game builds and plays with the Builder, itswork folder and its game folder 176-180 characters deep. For a tree that already worked, the fingerprint and the
shader stamp are unchanged.
Hardware: Windows 11 Pro (26300), RTX 4070 (616.56), Ryzen 9 7900X, 64 GB; llvm-mingw 23.1.5, Python 3.12.10; long
paths off
Scene: Kokiri Forest, the repository's
kokiri-startsave, headless (longshot_ctl.py launch --profile test)Method: the checks;
nx2/make_game.pyrun fromgit archiveof each commit, unpacked 176 characters deep, with--workand--outbeside it (180), throughscripts/job_queue.py --mode build; then the game folder started with thecontrol bridge, pressed into play, two screenshots, a stick walk, a stop
How to reproduce: the commands under How to reproduce, with this PR's commit in place of
origin/main71b1edd848432422nx2/test_version_long_paths.pytests/test_build_long_paths.pytests/test_long_paths.pyFileNotFoundErrorinfingerprint(); past it, incompile_onerun_all.py(Windows)Main's two extra suite failures (
test_bug_report,test_hardware_report) came from running it in a throwawayworktree; neither reads a file this PR changes.
Flags and settings
None. (
LONGSHOT_VERSIONappears only innx2/test_version_long_paths.py, which clears it so the check reads the release from the tree, asnx2/test_version.pydoes.)Testing
Tested:
version, stamp, prompt and build fixture checks pass; the per-script check skips there (the module does nothing off
Windows, and loading every script needs the build's packages).
(work and game folders short), and one built with the Builder, the work folder and the game folder 176-180
characters deep, with the first one's shader caches copied in. Both pressed through the title into Kokiri Forest at
about 60 fps, walked, and stopped cleanly, with no device lost and no error in
longshot_log.txt.script's case;
os.openunwrapped fails its case; each of the first version's 10 explicit sites reverted alonefailed a case.
Not tested: the real portable Launcher, and the reporter's own PC. PCs with
LongPathsEnabled=1in the registry: plain paths work there already, so this PR changesnothing for them. A current folder past ~258 characters (
subprocess'scwd=,os.chdir): Windows takes none in anyform, and the build runs at most in the game folder.
Must be tested: None
Risks
What could break: the build's file calls on Windows. The module patches
openand theos/shutilcallsprocess-wide in the build's own scripts. A path under 248 characters goes through unchanged, and the calls' conventions
(no argument, keywords,
dir_fd,os.supports_*) are kept and tested. Only the standard library's earlier copies ofopenare swept; code of ours keeps what it holds. Linux and macOS: nothing changes.Turning it off: revert the commit
Compatibility: Nothing changes for a player whose folders worked: the same fingerprint, shader stamp and object
keys, so no shader cache is dropped and nothing extra is compiled. The prompt key is unchanged.
Issues
Fixes #1115
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Tested in game: Windows 11 Pro (26300), RTX 4070, NVIDIA 616.56, long paths off: built with make_game.py from this commit unpacked 176 characters deep (the report's depth), with the work folder and the game folder 180 characters deep too (0:06:30, 0 failed); played that game folder headless on the repo's kokiri-start save: through the title into Kokiri Forest at ~60 fps, a stick walk, a clean stop, no device lost or file error in longshot_log.txt. Also played a game folder built by a deep Builder with short work and game folders, the same. Main stops on the same deep Builder with this issue's FileNotFoundError in fingerprint().
Possibly related, for context:
WIP: fix(build): the build runs from, in and into folders past 260 charactersto fix(build): the build runs from, in and into folders past 260 charactersNot a duplicate of #151, checked against its diff: #151 changed only the Windows setup (
builder/setup/setup.c,long_path.h), so the setup can copy the Builder out of a zip unpacked deeper than MAX_PATH. This PR is the next step: the Builder once installed there, its Python build (make_game.py,build.py,lift.pyand the steps they start) reading its sources and writing its work and game folders at that depth. Nothing here touchesbuilder/setup/, and #151 touched no Python.Prior work otherwise: none found in Longshot's open, merged and closed pull requests and issues (MAX_PATH, long path, 260 characters, extended-length, LongPathsEnabled), or in Hookshot's (MAX_PATH and LongPathsEnabled: none).
Code-only review of
4843242235; informational findings, with final approval left to another reviewer.Two corrections in the new process-wide path adapter:
nx2/long_paths.py:50.len(a)counts Python Unicode code points, while Windows file paths are sequences of WCHARs. A path containing supplementary characters (for example emoji folder names) can be fewer than 248 Python characters but already over 260 UTF-16 units.long_form()then passes it through unchanged, so build reads/writes still fail with long paths disabled. Count UTF-16 units (preserving surrogate handling), and add a case with each individual component within its limit but the total past MAX_PATH. The existing deep-path fixtures use ASCII and cannot catch this. Windows path rules.nx2/long_paths.py:184. The generic two-path wrapper convertsos.symlink'ssrcwithabspath, resolving it against the process's current folder. A relative link target is instead interpreted relative to the link's containing folder. For example, from a valid 240-character current folder,os.symlink("assets/file.txt", r"C:\short\link")converts the target to that unrelated current folder once its absolute length crosses 248. A call that previously pointed intoC:\short\assetsnow points elsewhere. Give symlink its own wrapper: preserve relative target text and convert the destination; only extend absolute targets. Add a Windows relative-link test with current folder different from link parent. This is an adapter API regression identified from source, not a reported broken game build; I found no direct productionos.symlinkcall innx2. Windows relative-link semantics.One smaller API-coverage gap:
_KEYWORDat line 160 mapsos.opentofile, but its keyword ispath. Consequentlyos.open(path=deep_path, flags=...)bypasses conversion. The build's current direct callers and the new fixture use positional paths, so I have not found a current build failure from this case. Usepathfor theos.openwrapper and include that keyword form in the fixture. Python signature.The reported Windows 11/Python 3.12 deep build and Kokiri run are useful coverage, and the tests exercise cache keys, header invalidation, stamps and entry-script installation. The remaining portable-Launcher check is disclosed. I read the full patch, related build callers, new tests and prior comments; no tests, builds or PR code were run here. This is separate from the already merged setup-copy fix: that handles installation, while this handles Python build I/O.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.