MagicBeanDev/Longshot

Watch

Fork 99

perf(build): the release ships the third-party code compiled, so a player's build links it instead of compiling it #629

Merged

HumanBean merged 2 commits from HumanBean/Longshot:perf/third-party-objects into main

2026-11-15 03:14:40 +00:00

HumanBean commented

2026-11-15 01:14:01 +00:00

Owner

Type: perf · Risk: medium · Tested on: Linux, i7-9700K lab (headless) · Needs: Windows: a release build's log, and a player's first build with it

Why

A player's first build, and every build after a change that empties the work folder's object cache, compiles about 315 third-party files (Eden's shader recompiler, sirit, Dear ImGui, zstd, libopus; on Windows also suyu's code under the WebGPU renderer). These are the same on every PC with the same compiler, so the release can compile them once and the player's build can link them. On the lab it cut a first build's compile count from 488 to 172 files. See #362 (a faster first build) and #182 (-march=native, which this keeps for everything but the third-party code).

What changed

The release makes a set of compiled third-party objects with the tools a player's build uses. A player's build copies each object whose object key it needs into its object cache, checking the object's SHA-256, and compiles anything it can't find, as before. These objects' keys also hold a stamp: the compiler's version, its driver defaults, and what the system headers, the C++ library and the Vulkan/SPIR-V/VMA headers preprocess to. A set made with another compiler or other headers is never used.

build: new nx2/third_party_objects.py: stamp(), fetch() (copies and checks the SHA-256 and size), write(), matching(), is_third_party() (the SOURCES folders), and make <out>. make runs build.py --standalone --third-party-objects <out> at RX_MARCH=x86-64-v3 with no game.
build: nx2/build.py: THIRD_PARTY_CPU builds the third-party code at -march=x86-64-v3 on any CPU that has v3, not native, so its keys don't name the CPU. Below v3, or with an RX_MARCH, it builds like the rest. The third-party jobs' deps end with the stamp, and they are built without -DLONGSHOT_RELEASE, which none of them reads. The set is fetched before the compile, and the dry run reports the third-party objects: N of M. The renderer's own gpu/vk/*.cpp (DLSS, FSR 4, the play profile), host/ and the lifted game are never in the set.
release: scripts/linux/appimage.sh makes the set into usr/share/longshot/third_party_objects. scripts/linux/check_player_build.sh fails unless the player's dry run finds N of N. .forgejo/workflows/build-windows.yml makes the Windows set (the same llvm-mingw release's Linux build, tools.lock's cpm and wgpu-native). scripts/builder_package.py package --third-party-objects puts it in tools/third_party_objects.
docs: docs/development.md (LONGSHOT_THIRD_PARTY_OBJECTS, and the RX_MARCH row), docs/releases.md (what the release files hold).
tests: new tests/test_third_party_objects.py; tests/test_shader_stamp_cache.py updated for the new flags and keys.

Measurements

Verdict: Helps a little on Linux. A first build compiled 172 of 488 files instead of 488, and took 848 s against 908 s (−7 %, one run each way). A renderer-only build took 16 s against 55 s. The lifted game is about 94 % of the compile's CPU time, so the gain is capped at that. On Windows it is not measured.

Hardware: Linux, i7-9700K (8 cores), 6 jobs, SCHED_IDLE under the lab's build lock; tools.lock's linux-x64 clang, sysroot and CPM headers
Scene: none (the build)
Method: a full first build in fresh work and game folders, one run each way. The Vulkan back end alone (RX_VK_ONLY=1) in a fresh work folder, twice each way. Each figure is build.py's own compiled in time and make_game.py's total.
How to reproduce: python -I nx2/third_party_objects.py make <set>, then a first build with LONGSHOT_THIRD_PARTY_OBJECTS=<set> against one with =0

	Before: main @ `2b1d38c6`	After: this PR @ 97d867be (same tree as `5902f062`, rebased)
Files compiled, first build	488 of 488	172 of 488 (316 from the set)
Compile, first build	722 s	679 s
Whole first build	908 s	848 s
Vulkan back end only, fresh work folder	55 s, 55 s	16 s, 16 s
Making the set (release side)	—	316 objects, 17 MB, 45 s (211 CPU-s)

Third-party CPU time from a lab first build's per-file lines: 315 compiles, 230.6 CPU-s. Lifted code took 3,939 CPU-s and everything else 103.
Same output: with the set turned off, the after tree compiles objects byte-identical to the set's (169 compared, 0 differ), and longshot_vk.so is identical either way.
The after build's game starts headless: it reached frame 60 and the intro, at 52.7 fps.
Mismatches compile instead of linking:

Mismatch	Taken from the set
No sysroot (other system headers)	0 of 169 ("made with another compiler or other headers")
`RX_MARCH=x86-64-v2`	0 of 169
imgui_widgets.cpp's object overwritten with random bytes of the same size	168 of 169 (imgui_widgets.cpp compiled)
`LONGSHOT_THIRD_PARTY_OBJECTS=0`	none

Flags and settings

One new development switch, no player setting.

Flag	What it is	Before this PR	After this PR	Effect	Helps / doesn't help / untested	Recommendation
`LONGSHOT_THIRD_PARTY_OBJECTS`	The set of third-party objects a build takes from	—	the tools' own `third_party_objects/` (a checkout has none)	`0` compiles them all; `<dir>` uses that set	Helps on Linux (above); Windows untested	Leave unset

Testing

Tested: on the lab (Linux, i7-9700K): making the set; a first build and a renderer-only build each way; byte-identical objects; the four mismatches above; the after game's headless start. tests/run_all.py with the before and after trees on the lab shows the same 15 failures in both (many because the lab copies have no .git). The after tree has 501 passing against 500. test_appimage_player_build.py's test_no_prebuilt_renderer already fails on main. Nothing was run on the author's PC.

Not tested: Windows, entirely: the CI step that makes the set with llvm-mingw, cpm and wgpu-native, and whether the Linux-hosted and Windows-hosted llvm-mingw 20261104 give the same stamp. If they don't, Windows players compile as before and the build log says "0 of N". The WebGPU suyu objects. appimage.sh and check_player_build.sh end to end (the lab used the same make with tools.lock's tools).

Must be tested:

Linux CI: a nightly AppImage. check_player_build.sh must pass (it requires N of N), and a player's first build log must say the third-party objects: N of N from ….
Windows: a nightly zip's player first build. Its log line the third-party objects: N of M should say N = M. 0 means the stamps differ.

Risks

The stamp is only as complete as its probe. A compiler that gives the same version line, driver defaults and preprocessed headers but generates different code would be trusted. Players all get tools.lock's one compiler, so this mostly concerns developers' own machines.
Eden's and sirit's keys never held the host/posix headers or the -I order beyond MKWVK_INCLUDES. A set made from the same release tree is not affected.
Every build computes the stamp: about 0.6 s warm, a few seconds on a cold disk.
Developers compile the third-party code once more, since its flags and keys changed.
About 17 MB more in the AppImage and the Windows zip.
Compiled GPL and BSD objects now ship; their sources are in the same packages.
Reverting the commit restores the old behaviour. LONGSHOT_THIRD_PARTY_OBJECTS=0 turns it off without a revert.

Issues

See #362

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main.
python -I -B tests/run_all.py passes (any failure that also fails on main is named). (run on the lab only, same 15 failures as main; left to CI)
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
A check under tests/ that reads changed source updated in this PR, not weakened.
Docs updated.
A row in tested configurations for each run measured. (not yet)

🤖 Generated with Claude Code

https://claude.ai/code/session_017vw8e3MfaADZobPGhWJk77

**Type:** perf · **Risk:** medium · **Tested on:** Linux, i7-9700K lab (headless) · **Needs:** Windows: a release build's log, and a player's first build with it ## Why A player's first build, and every build after a change that empties the work folder's object cache, compiles about 315 third-party files (Eden's shader recompiler, sirit, Dear ImGui, zstd, libopus; on Windows also suyu's code under the WebGPU renderer). These are the same on every PC with the same compiler, so the release can compile them once and the player's build can link them. On the lab it cut a first build's compile count from 488 to 172 files. See https://git.magicbean.dev/MagicBeanDev/Longshot/issues/362 (a faster first build) and https://git.magicbean.dev/MagicBeanDev/Longshot/issues/182 (`-march=native`, which this keeps for everything but the third-party code). ## What changed The release makes a set of compiled third-party objects with the tools a player's build uses. A player's build copies each object whose object key it needs into its object cache, checking the object's SHA-256, and compiles anything it can't find, as before. These objects' keys also hold a stamp: the compiler's version, its driver defaults, and what the system headers, the C++ library and the Vulkan/SPIR-V/VMA headers preprocess to. A set made with another compiler or other headers is never used. - build: new `nx2/third_party_objects.py`: `stamp()`, `fetch()` (copies and checks the SHA-256 and size), `write()`, `matching()`, `is_third_party()` (the `SOURCES` folders), and `make <out>`. `make` runs `build.py --standalone --third-party-objects <out>` at `RX_MARCH=x86-64-v3` with no game. - build: `nx2/build.py`: `THIRD_PARTY_CPU` builds the third-party code at `-march=x86-64-v3` on any CPU that has v3, not `native`, so its keys don't name the CPU. Below v3, or with an `RX_MARCH`, it builds like the rest. The third-party jobs' deps end with the stamp, and they are built without `-DLONGSHOT_RELEASE`, which none of them reads. The set is fetched before the compile, and the dry run reports `the third-party objects: N of M`. The renderer's own `gpu/vk/*.cpp` (DLSS, FSR 4, the play profile), `host/` and the lifted game are never in the set. - release: `scripts/linux/appimage.sh` makes the set into `usr/share/longshot/third_party_objects`. `scripts/linux/check_player_build.sh` fails unless the player's dry run finds N of N. `.forgejo/workflows/build-windows.yml` makes the Windows set (the same llvm-mingw release's Linux build, tools.lock's cpm and wgpu-native). `scripts/builder_package.py package --third-party-objects` puts it in `tools/third_party_objects`. - docs: `docs/development.md` (`LONGSHOT_THIRD_PARTY_OBJECTS`, and the `RX_MARCH` row), `docs/releases.md` (what the release files hold). - tests: new `tests/test_third_party_objects.py`; `tests/test_shader_stamp_cache.py` updated for the new flags and keys. ## Measurements **Verdict:** Helps a little on Linux. A first build compiled 172 of 488 files instead of 488, and took 848 s against 908 s (−7 %, one run each way). A renderer-only build took 16 s against 55 s. The lifted game is about 94 % of the compile's CPU time, so the gain is capped at that. On Windows it is not measured. Hardware: Linux, i7-9700K (8 cores), 6 jobs, SCHED_IDLE under the lab's build lock; tools.lock's linux-x64 clang, sysroot and CPM headers Scene: none (the build) Method: a full first build in fresh work and game folders, one run each way. The Vulkan back end alone (`RX_VK_ONLY=1`) in a fresh work folder, twice each way. Each figure is build.py's own `compiled in` time and make_game.py's total. How to reproduce: `python -I nx2/third_party_objects.py make <set>`, then a first build with `LONGSHOT_THIRD_PARTY_OBJECTS=<set>` against one with `=0` | | Before: main @ 2b1d38c6 | After: this PR @ 97d867be (same tree as 5902f062, rebased) | |---|---|---| | Files compiled, first build | 488 of 488 | 172 of 488 (316 from the set) | | Compile, first build | 722 s | 679 s | | Whole first build | 908 s | 848 s | | Vulkan back end only, fresh work folder | 55 s, 55 s | 16 s, 16 s | | Making the set (release side) | — | 316 objects, 17 MB, 45 s (211 CPU-s) | - Third-party CPU time from a lab first build's per-file lines: 315 compiles, 230.6 CPU-s. Lifted code took 3,939 CPU-s and everything else 103. - Same output: with the set turned off, the after tree compiles objects byte-identical to the set's (169 compared, 0 differ), and `longshot_vk.so` is identical either way. - The after build's game starts headless: it reached frame 60 and the intro, at 52.7 fps. - Mismatches compile instead of linking: | Mismatch | Taken from the set | |---|---| | No sysroot (other system headers) | 0 of 169 ("made with another compiler or other headers") | | `RX_MARCH=x86-64-v2` | 0 of 169 | | imgui_widgets.cpp's object overwritten with random bytes of the same size | 168 of 169 (imgui_widgets.cpp compiled) | | `LONGSHOT_THIRD_PARTY_OBJECTS=0` | none | ### Flags and settings One new development switch, no player setting. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `LONGSHOT_THIRD_PARTY_OBJECTS` | The set of third-party objects a build takes from | — | the tools' own `third_party_objects/` (a checkout has none) | `0` compiles them all; `<dir>` uses that set | Helps on Linux (above); Windows untested | Leave unset | ## Testing **Tested:** on the lab (Linux, i7-9700K): making the set; a first build and a renderer-only build each way; byte-identical objects; the four mismatches above; the after game's headless start. `tests/run_all.py` with the before and after trees on the lab shows the same 15 failures in both (many because the lab copies have no `.git`). The after tree has 501 passing against 500. `test_appimage_player_build.py`'s `test_no_prebuilt_renderer` already fails on main. Nothing was run on the author's PC. **Not tested:** Windows, entirely: the CI step that makes the set with llvm-mingw, cpm and wgpu-native, and whether the Linux-hosted and Windows-hosted llvm-mingw 20261104 give the same stamp. If they don't, Windows players compile as before and the build log says "0 of N". The WebGPU suyu objects. `appimage.sh` and `check_player_build.sh` end to end (the lab used the same `make` with tools.lock's tools). **Must be tested:** - Linux CI: a nightly AppImage. `check_player_build.sh` must pass (it requires N of N), and a player's first build log must say `the third-party objects: N of N from …`. - Windows: a nightly zip's player first build. Its log line `the third-party objects: N of M` should say N = M. 0 means the stamps differ. ## Risks - The stamp is only as complete as its probe. A compiler that gives the same version line, driver defaults and preprocessed headers but generates different code would be trusted. Players all get tools.lock's one compiler, so this mostly concerns developers' own machines. - Eden's and sirit's keys never held the `host/posix` headers or the `-I` order beyond `MKWVK_INCLUDES`. A set made from the same release tree is not affected. - Every build computes the stamp: about 0.6 s warm, a few seconds on a cold disk. - Developers compile the third-party code once more, since its flags and keys changed. - About 17 MB more in the AppImage and the Windows zip. - Compiled GPL and BSD objects now ship; their sources are in the same packages. - Reverting the commit restores the old behaviour. `LONGSHOT_THIRD_PARTY_OBJECTS=0` turns it off without a revert. ## Issues See https://git.magicbean.dev/MagicBeanDev/Longshot/issues/362 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (run on the lab only, same 15 failures as main; left to CI) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. - [ ] A row in [tested configurations](docs/tested-configurations.md) for each run measured. (not yet) 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017vw8e3MfaADZobPGhWJk77

HumanBean added 1 commit

2026-11-15 01:14:01 +00:00

perf(build): the release ships the third-party code compiled, so a player's build links it instead of compiling it

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 4s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Has been cancelled

Details

5902f0627f

A player's first build compiled ~230 CPU-seconds of third-party code (Eden's shader recompiler, sirit, Dear ImGui, zstd,
libopus; on Windows also suyu's code under the WebGPU renderer) that is the same on every PC. The release now compiles it
with the tools a player's build uses (nx2/third_party_objects.py make: scripts/linux/appimage.sh, build-windows.yml) and
ships the objects with their manifest (usr/share/longshot/third_party_objects, LongshotBuilder/tools/third_party_objects).

nx2/build.py takes one only by its object key, copied into the work folder's object cache and checked by its SHA-256:
- the third-party parts are built for x86-64-v3 on every CPU that has it (THIRD_PARTY_CPU), not -march=native, so their
  keys name no CPU; a CPU below v3 or an RX_MARCH builds them as the rest;
- their keys also hold a stamp: the compiler's version, the defaults its driver adds (-###, paths left out), and what the
  system headers, the C++ library and the Vulkan, SPIR-V and VMA headers preprocess to;
- any mismatch (another compiler or headers, other flags, a damaged object) compiles as before.
The renderer's own files (DLSS, FSR 4, the play profile), host/ and the lifted game are never in it.
LONGSHOT_THIRD_PARTY_OBJECTS=0 turns it off (docs/development.md). check_player_build.sh requires the AppImage's set to
cover every third-party object a player's dry run would compile.

Measured on the AMD lab (i7 Skylake, 6 jobs, tools.lock's Linux tools): a first build in new folders compiled 172 of 488
files instead of 488 (316 of 316 objects from the set), compile 722 -> 679 s, the whole build 908 -> 848 s; the Vulkan
back end alone 55 -> 16 s, twice each way. The set: 316 objects, 17 MB.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017vw8e3MfaADZobPGhWJk77

HumanBean changed title from ~~WIP: perf(build): the release ships the third-party code compiled, so a player's build links it instead of compiling it~~ to perf(build): the release ships the third-party code compiled, so a player's build links it instead of compiling it

2026-11-15 03:14:26 +00:00

HumanBean added 1 commit

2026-11-15 03:14:28 +00:00

Merge branch 'main' into perf/third-party-objects

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details