build(linux): the Linux builder gets its build tools from tools.lock, as Windows does #113
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!113
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:linux-tools-lock"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
Every Linux release shipped its whole compiler, sysroot and libraries again inside a 245 MB AppImage, and each installed version kept its own copy. The Linux builder now gets its build tools from
tools.lock, as Windows does: each one is published once and shared by every version that uses it.What changed
tools.lock: five
linux-x64lines (emsdk,ffmpeg,sdl2,sysroot,cpm). Eachdestis where the AppImage used to carry the tool (usr/lib/longshot/...,usr/share/longshot/cpm), so an unpacked builder looks the same as before. Windows also getscpmwith dav1d andthird-party.Where the tools live: Packages. Each archive is a file of a generic package of the MagicBeanDev organization:
<name>-<system>/<version>/<sha256>-<name>.zip, athttps://git.magicbean.dev/api/packages/MagicBeanDev/generic/.... Upstream URLs stay as fallbacks. The package is per system because a tool's Windows and Linux builds have versions of their own (emsdk ishookshot-v2.1.0on Windows and23.1.5on Linux). All 11 files are uploaded, and anyone can download them without an account.scripts/tools.py:store()is the one upload path, aPUTto the package registry. A 409 is accepted only when the existing file has the same size and SHA-256.check --requirechecks that every tool each builder needs has a line (REQUIREDper system), and that each line's package file exists with its size and SHA-256 (via the package API).profileis no longer required.PROFILEandassemble profilestay.--systemonfetch,installandmirror.pack <folder>: a deterministic archive that keeps file modes and relative links.unpackrestores them and refuses links that lead outside the folder.installwrites.tools/<name>stamps and unpacks a tool again when its line changes.Shared tool entry? Not done, and the format is unchanged. CPM is the only tool that could share an entry, and it can't: the Windows CPM archive also carries Windows FFmpeg and dav1d, and the two systems unpack to different dests.
scripts/linux/deps.sh: the default run builds only what the AppImage carries (Python with Tk, appimagetool, the runtime).deps.sh toolsbuilds LLVM, FFmpeg+dav1d, SDL2, the headers and the sysroot (moved here fromappimage.sh), puts each tool's licenses in its own folder, and packs each one intobuild-linux/deps/tools/<name>.zipwith anadd.shholding thetools.py addlines.docker.sh toolsruns it in the container.scripts/linux/appimage.sh: installs thelinux-x64tools intobuild-linux/toolsand builds the Vulkan back end and the sysroot check with them. It copies none of them into the AppDir, and copiestools.lockintousr/share/longshot.AppImage contents: the builder, its Python with Tk, Longshot's sources and the prebuilt
longshot_vk.so.make_game.pyall run on it, before any tool exists.-I, so packages outside its tree would need a path hook.requirements.txt, which changes with each version, so sharing them across versions would gain little.builder/linux/linux_tools.py(new) andbuilder/linux/AppRun: before the window or--clistarts, AppRun getsEMSDK,LONGSHOT_SYSROOT,LONGSHOT_FFMPEG,LONGSHOT_SDL2andRX_CPMfromlinux_tools.py env --install. Each tool is looked for in this order:toolchain.jsonnames the SHA-256);$XDG_DATA_HOME/longshot/builder/toolchains/<version>/, which uses the launcher's layout.Whatever is missing is downloaded there on the first start, checked by its SHA-256, with a progress window (on stderr for
--cli). A tool another version already has is hard-linked, not downloaded. Once the tools are there, no connection is needed.--clistops if a tool is missing.--playneeds no tools.RecompLauncher:
--cli --prepare --gamegoes through the same--clipath, with progress on stderr.CI:
build-linux.ymlinstalls the tools from Packages, cached bytools.lock's hash, and the object cache key namestools.lock.check_player_build.shrefuses an image that carries a tool, and runs the build from the image with the tools given as the builder's toolchains folder, so nothing is downloaded.release.yml's first job (check --require) now also covers the Linux lines and the package files.Launcher: MagicBeanDev/LongshotLauncher#4. After unpacking the AppImage, it installs the
linux-x64tools of thetools.lockinside it, with reuse across versions. It must be merged and released before the first AppImage made with this PR: an older launcher would runmake_game.pywithout the tools.Docs:
docs/releases.md(Linux no longer bundles; Packages),RELEASING.md(bumping a Linux tool, Packages, one-time setup step 6 done), the README (the standalone AppImage's size and first-start download), anddocs/development.md.docs/releases.mdandRELEASING.mdare byte-identical with the launcher PR.Flags and settings
LONGSHOT_BUILDER_HOME(existing)$XDG_DATA_HOME/longshot/builderTOOLS,TOOLS_CACHE(appimage.sh,check_player_build.sh)build-linux/tools,build-linux/tools-cacheMeasurements
main:
b9cdc432(2026-11-12)this PR: linux-tools-lock @
78d14610Hardware: Ryzen desktop, 24 threads, 30 GB, CachyOS (Linux 7.3); the build in
longshot-linux-build(Ubuntu 22.04, as CI)Method:
deps.sh+appimage.shin the container,LONGSHOT_VERSION=v3.0.0-measure. The size is the AppImage file's. The per-tool share is each folder of main's AppDir compressed with zstd -15, which approximates the AppImage's squashfs zstd. The package size is the uploaded zip. The AppImage is deterministic in size: main was built once, this PR twice (both 94,575,096 bytes).How to reproduce:
scripts/linux/docker.shon each commit (with--network hostwhere the container's DNS fails), thenstat -c %s build-linux/*.AppImage.b9cdc43278d14610Longshot-<v>-x86_64.AppImageA first start downloads 170.4 MB once (516 MB unpacked). A later version downloads only the tools whose line changed.
Testing
Tested:
LONGSHOT_VULKAN_HEADERS=/usr/include python -I -B tests/run_all.py, interleaved: main 492 passed, 2 failed, 36 skipped; this PR 493 passed, 2 failed, 36 skipped (the newtest_linux_tools.py); main again 492/2/36. The 2 failures are the known ones and fail on main too:test_bake_portability,test_builder_log_read_only.tests/test_linux_tools.py: the lookup order, the first-start download with progress, offline later starts, hard links across versions, the launcher's copy used as is, a changed download installing nothing, a changed stamp, and AppRun itself (--cli --preparewith tools found offline,--clistopping without them,--playneeding none). It uses fakes and no network.test_tools_lock: Linux lines, systems kept apart,pack/unpackmodes and links, links leading outside refused, the package store,check --requireper system and with the store, and the profile not required.test_appimage_player_build(the tools at their tools.lock places, DLSS withRX_CPM),test_linux_update(the sysroot loop, now in deps.sh, also round-tripped throughtools.py pack/unpack),test_make_game_ngx.check_player_build.shpassed:--cli --help, the pipeline list,build.py --standalone --dry-runwith 0 missing, nothing written into the image.Longshot-v3.0.0-measure-x86_64.AppImage --cli --helpdownloaded the 5 tools from the package URLs, unpacked 516 MB and ran make_game. The second start took 0.15 s and downloaded nothing.tools.py check --requirepasses against the uploaded packages.Not tested:
tools.py.cpm/third-partyon Windows. dav1d was built withscripts/windows/dav1d.sh(llvm-mingw 20261104 for Linux, nasm 2.16.01, libdav1d.a sha256 e6f1adec…), not run under Wine here.Needs other testers:
Longshot-*.AppImagewith no~/.local/share/longshot/builder/toolchains, build and play. Expect a download window (about 170 MB), then a normal build (request to open).LongshotBuilder.exewith this tools.lock. Expect the setup to installthird-party. This may fail, see Reviewer notes (request to open; blocked on the setup.c follow-up).Screenshots
Nothing on screen (the first-start progress window is new, but there was no display here to capture it).
Risk and rollback
make_game.pyfrom an unpacked AppImage that has no tools, so its Linux builds fail. Merge and release the launcher PR first. An AppImage started by itself is unaffected.Reviewer notes
third-partyon Windows is likely broken bysetup.c. The archive is uploaded and in tools.lock. Butbuilder/setup/setup.cmoves a tool's prefix folder onto itsdestwithMoveFileW, which fails when the dest exists, andLongshotBuilder/tools/third_partyis already in the builder zip. Its reuse also moves an older version's whole folder. The launcher merges files and is fine. This needs a follow-up insetup.c(merge a tool into an existing dest), or another dest.third-partyarchive comes from Hookshot v2.1.0's part (c361af47…, 29,574,904 B, the same as in v2.1.0'shookshot-setup.txt), keeping only its 8 prebuilt files. Itsa64emit.dll.sha1stamp doesn't match Longshot'sa64emit.cpp+arm64_to_c.h(whose bytes are unchanged since v2.1.0), so the player's PC would rebuild a64emit.dll anyway.scripts/release.pystill says "the tools release" in two comments. It is shared with the launcher, so it is left for a follow-up.scripts/linux/make_parts.pyandlinux_update.py's parts-list path (dev only) still describe the bundled layout.Issues
See MagicBeanDev/LongshotLauncher#4
Checklist
For everyone
type(scope): description).main.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above).third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
mainand passing here.tests/that reads changed source updated in this PR, not weakened.builder/setup/setup.c: one comment changed, no code.)WIP: build(linux): the Linux builder gets its build tools from tools.lock, as Windows doesto build(linux): the Linux builder gets its build tools from tools.lock, as Windows does