ci: cache the AppImage's back-end objects for pull requests, and a lighter deps cache #2079

Merged

hckrman101 merged 1 commit from ci-appimage-faster into main

2026-11-11 01:54:55 +00:00

hckrman101 commented

2026-11-11 01:53:57 +00:00

(Migrated from github.com)

Where the time went (per-step times from 14 recent runs, timestamps from the logs of 3):

Step	Time
System packages (apt, toolchain PPA)	39-66 s
Restore the deps cache (2,252 MB)	16-31 s
AppImage: the back end, 209 files, 4 jobs	77-136 s
AppImage: native helpers, sysroot, checks	about 10 s
AppImage: `make_parts.py`, 6 zips one after another	24-36 s
AppImage: mksquashfs (zstd)	12-36 s
Upload, player's-build check, FSR 4 check	about 15 s

What changed:

Back-end objects are cached for pull requests. nx2/build.py already keeps an object cache, keyed by each file's source, flags and headers (build-linux/vk-work/objcache). It is now kept with actions/cache, pruned to the objects the build used (as prune_object_cache in nx2/make_game.py does). Pull requests restore it: their own cache, else main's. Pushes to main and releases never restore it and still compile every object, so the nightly and the releases are built exactly as before. The key also names the pinned LLVM (deps.sh) and the container's libc and libstdc++ header packages, because the objects' own keys leave those out. A pull request that changes gpu/vk/*.cpp recompiles only those files; one that changes a gpu/vk header recompiles the back end's ~40 files, but not Eden's or sirit's ~165.
The built-deps cache leaves out build-linux/deps/downloads (about 1.4 of its 2.3 GB). deps.sh downloads a part only when it has not built it yet. The release's downloads-only cache is unchanged.
scripts/linux/make_parts.py zips the six update parts at once, one thread each. The zips have the same bytes and the list keeps its order (checked against the old script on a test AppDir).
dpkg runs with force-unsafe-io in the container, which is thrown away after the job.

The cache key names changed, so the first run of this rebuilds the dependencies once.

Before/after (this PR's runs):

Step	Before	Cold cache	Warm cache
System packages	39-66 s
Deps cache restore	16-31 s
AppImage	123-229 s
Job	215-335 s

The Linux AppImage job is one of the slowest checks on every pull request (215-335 s for the job in the last 14 runs), and most of that time goes on work that is the same every run: it recompiles the whole Vulkan back end and restores a 2.3 GB cache that is mostly downloads. Where the time went (per-step times from 14 recent runs, timestamps from the logs of 3): | Step | Time | |---|---| | System packages (apt, toolchain PPA) | 39-66 s | | Restore the deps cache (2,252 MB) | 16-31 s | | AppImage: the back end, 209 files, 4 jobs | 77-136 s | | AppImage: native helpers, sysroot, checks | about 10 s | | AppImage: `make_parts.py`, 6 zips one after another | 24-36 s | | AppImage: mksquashfs (zstd) | 12-36 s | | Upload, player's-build check, FSR 4 check | about 15 s | What changed: - **Back-end objects are cached for pull requests.** `nx2/build.py` already keeps an object cache, keyed by each file's source, flags and headers (`build-linux/vk-work/objcache`). It is now kept with `actions/cache`, pruned to the objects the build used (as `prune_object_cache` in `nx2/make_game.py` does). Pull requests restore it: their own cache, else main's. Pushes to main and releases never restore it and still compile every object, so **the nightly and the releases are built exactly as before**. The key also names the pinned LLVM (`deps.sh`) and the container's libc and libstdc++ header packages, because the objects' own keys leave those out. A pull request that changes `gpu/vk/*.cpp` recompiles only those files; one that changes a `gpu/vk` header recompiles the back end's ~40 files, but not Eden's or sirit's ~165. - **The built-deps cache leaves out `build-linux/deps/downloads`** (about 1.4 of its 2.3 GB). `deps.sh` downloads a part only when it has not built it yet. The release's downloads-only cache is unchanged. - **`scripts/linux/make_parts.py` zips the six update parts at once**, one thread each. The zips have the same bytes and the list keeps its order (checked against the old script on a test AppDir). - **dpkg runs with `force-unsafe-io`** in the container, which is thrown away after the job. The cache key names changed, so the first run of this rebuilds the dependencies once. Before/after (this PR's runs): | Step | Before | Cold cache | Warm cache | |---|---|---|---| | System packages | 39-66 s | | | | Deps cache restore | 16-31 s | | | | AppImage | 123-229 s | | | | Job | 215-335 s | | |