ci(release): no dav1d build per release, and a tag skips the build checks its commit's release passed #143

Merged
HumanBean merged 1 commit from HumanBean/Longshot:release-speed into main 2026-11-13 06:34:11 +00:00
Owner

Why

Every release builds dav1d from source in a job whose output nothing uses (the builder gets dav1d from tools.lock's cpm package), and a tag on a commit CI already released (its nightly, or a stable's release candidate) runs every build check again. Both cost runner time on a 4-core runner shared with a VM, and the dav1d job held one of its 3 slots for 4 minutes per release.

What changed

  • dav1d (build-windows.yml): the dav1d job (meson, nasm, Wine with apt; dav1d 1.5.4 built from source; an artifact nobody downloaded) is gone. The package job, which already has Wine and llvm-mingw, now installs tools.lock's cpm package (scripts/tools.py install --only cpm, checked by its SHA-256) and runs tests/test_dav1d_av1.py under Wine against that package's dav1d: the check now covers the library players actually get. scripts/windows/dav1d.sh stays, as the way a maintainer makes that package; its header and RELEASING.md's Bump a tool say so (assemble cpm --add <cpm folder>; the old line pointed at the removed artifact).
  • A tag on a commit already released (release.yml, scripts/release.py released-as): on a tag, The version asks for the published release of the same commit (a nightly-* whose tag is that commit, or a stable's release candidate), downloads its SHA256SUMS and .sig (through RELEASE_UPLOAD_API's host when it is set), and checks that a key in scripts/release_keys.json signed it and that it names exactly the release's files. A release of the commit that fails any of this stops the run; with none, the build is as today. When one is found, its tag goes to both build workflows as checked, and the build checks that ship no file and passed in its run are skipped: the Windows Vulkan back end job (every step gated on RELEASE_CHECKED) and the AppImage's AMD FSR 4 build check.
  • Files are not reused, on purpose: every release file carries its version in its content, not only its name. The AppImage has usr/share/longshot/VERSION (read by linux_tools.py/linux_update.py for the tools folder, and by nx2/version.py) and longshot_vk.so, whose vk_menu.cpp is compiled with -DLONGSHOT_VERSION="<tag> (source …)" (the menu, the bug report). The zip has VERSION (the setup installs into versions\<version>) and the builder published with -p:Version=<X.Y.Z-rc.N> (a nightly's is 0.0.0). A renamed nightly file would claim the nightly's version, so both files are always built again. Carrying the commit instead and reading the version from the release would change the setup, the AppImage, the game's version line and RecompLauncher's hand installs; rebuilding only the parts that change would still recompile the whole Vulkan back end for the one object that has the version. Neither is simpler than building the file again.
  • Tests: tests/test_release.py (shared) gets ReleasedAs: the lookup (by the tag's commit, not by the 8 hex in the name; drafts, other commits, the tag itself and non-product tags ignored; a stable finds its rc), the verification (untrusted key, corrupt or non-base64 signature, a file more or less than SHA256SUMS names, a missing .sig, a file that doesn't answer: each stops it), the download through the address without Cloudflare, and the command. tests/test_windows_on_linux.py checks that no dav1d is built in CI, that the package job checks tools.lock's dav1d under Wine, and that checked gates every step of the Vulkan job and only the FSR check on Linux.
  • Shared files: scripts/release.py, tests/test_release.py, docs/releases.md and RELEASING.md change the same way in MagicBeanDev/LongshotLauncher (MagicBeanDev/LongshotLauncher#11); the four files are byte-identical (cmp).

Flags and settings

None. (checked is a new optional input of build-linux.yml and build-windows.yml, empty by default: a run by hand behaves as before.)

Measurements

main: e28236b4 (2026-11-13)
this PR: release-speed @ 6a9895ee
Hardware: the Unraid runner (Ryzen 5 3400G, 4C/8T, shared with a VM), capacity 3, ghcr.io/catthehacker/ubuntu:act-24.04 in Docker-in-Docker
Scene: nightly run 717 (nightly-34-545618fc); its build workflows are main's (545618fc..e28236b4 changes no build workflow)
Method: per-step durations from Forgejo's action_task_step (the stored logs keep no per-step markers), queue time from action_run_job; one run (a release run can't be repeated on demand without making releases; the test suite below ran interleaved, twice each). This PR's own numbers wait for its first nightly and tag: the build workflows have no pull-request trigger and the fork has no runner
How to reproduce: sqlite3 -readonly gitea.db "select task_id, \"index\", stopped-started, name from action_task_step where task_id in (446,448,449,458,497)"

main @ e28236b4 (run 717) this PR @ 6a9895ee (expected)
dav1d job (runner time) 238 s (apt 86 s, build 114 s, Wine test 14 s), queued 3.7 min for a slot gone
Zip job (package) 197 s about 197 + 25 s (the 20 MB cpm download, the test compile and its Wine run)
Build jobs per release 4 on 3 slots 3 on 3 slots
A tag on a released commit: Windows Vulkan job 300 s (compile 259 s) skipped (a container start)
A tag on a released commit: AppImage's FSR check 23 s skipped
Critical path (appimage) 736 s unchanged (it carries the version)
run 717 (05:56:05 -> 06:17:32 UTC, 21.5 min)
appimage 736 s: System packages 154, Node 6, checkout 7, deps (Python with Tk) 108, tools.lock tools 61, AppImage 339 (Vulkan back end 226), FSR check 23, upload 17, player's build 9
package 197 s: Python 15, .NET 18, Wine 38, llvm-mingw 23, setup 3, dotnet publish 30, package 7, zip 13, PE check 3, Wine start 25, upload 14
vulkan 300 s: Python 16, llvm-mingw 14, longshot_vk.dll 259
dav1d 238 s (started 06:00:16, 3.7 min after the others): apt 86, llvm-mingw 16, dav1d 114, Wine test 14
publish 104 s, queued 06:09:02 -> 06:15:48: artifact downloads 45 + 44, sign 1, notes 1, upload 2

Testing

Tested: LONGSHOT_VULKAN_HEADERS=/usr/include python -I -B tests/run_all.py on main @ e28236b4 and on this branch, interleaved, twice each (Linux, CachyOS): 500 passed, 2 failed, 36 skipped on all four runs (main 92 s, 92 s; this branch 106 s, 86 s); the 2 failures are the known test_bake_portability and test_builder_log_read_only, on main too. tests/test_release.py passes in the LongshotLauncher copy as well. released-as against git.magicbean.dev: for 545618fc (tagged v3.0.0-rc.1), --not v3.0.0-rc.1 prints nightly-34-545618fc, its SHA256SUMS checked with the real release key. tests/test_dav1d_av1.py --cross with llvm-mingw 20261104 compiles and links against the cpm package's libdav1d.a installed by tools.py install --only cpm. actionlint 1.7.12 reports nothing new against main.

Not tested: the workflows themselves, because the fork's branch has no runner: the Wine run of the dav1d test in the zip's job (there's no Wine here) and the checked path of a tag. The first nightly after merge runs the dav1d check ("host/dav1d_av1.c on it, under Wine"). The first tag on a nightly's commit runs released-as: its The version prints a ::notice:: naming the nightly, and the Vulkan job's steps are skipped. The maintainer who merges and tags sees both.

Needs other testers:
None

Screenshots

Nothing on screen

Risk and rollback

  • If released-as finds a release of the commit whose SHA256SUMS does not check, the tag's run stops in The version: by design, nothing is built or published. Delete the bad release (or fix the key) and push the tag again.
  • If Forgejo did not apply the step if:s as expected, the Vulkan job would simply run as today.
  • If the cpm package's dav1d failed under Wine, the zip's job fails and so does the release: that is the library players get. Reverting the commit brings the old job back.

Reviewer notes

Start with released_as() in scripts/release.py and the version step in release.yml. Left for follow-ups (report in the first comment): publish waited 6.8 min for a runner in run 717 (it could run on quick: 104 s); release runs read no cache by design, though tools.py checks every cached archive's SHA-256, so a cache of tools.lock's archives would be safe for releases too.

Issues

None.

Checklist

For everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description).
  • Rebased on current main.
  • python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
  • Tested in the game, or what wasn't is named under Testing.
  • Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
  • Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.
## Why Every release builds dav1d from source in a job whose output nothing uses (the builder gets dav1d from `tools.lock`'s `cpm` package), and a tag on a commit CI already released (its nightly, or a stable's release candidate) runs every build check again. Both cost runner time on a 4-core runner shared with a VM, and the dav1d job held one of its 3 slots for 4 minutes per release. ## What changed - **dav1d (`build-windows.yml`)**: the `dav1d` job (meson, nasm, Wine with apt; dav1d 1.5.4 built from source; an artifact nobody downloaded) is gone. The `package` job, which already has Wine and llvm-mingw, now installs `tools.lock`'s `cpm` package (`scripts/tools.py install --only cpm`, checked by its SHA-256) and runs `tests/test_dav1d_av1.py` under Wine against that package's dav1d: the check now covers the library players actually get. `scripts/windows/dav1d.sh` stays, as the way a maintainer makes that package; its header and RELEASING.md's *Bump a tool* say so (`assemble cpm --add <cpm folder>`; the old line pointed at the removed artifact). - **A tag on a commit already released (`release.yml`, `scripts/release.py released-as`)**: on a tag, *The version* asks for the published release of the same commit (a `nightly-*` whose tag is that commit, or a stable's release candidate), downloads its `SHA256SUMS` and `.sig` (through `RELEASE_UPLOAD_API`'s host when it is set), and checks that a key in `scripts/release_keys.json` signed it and that it names exactly the release's files. A release of the commit that fails any of this stops the run; with none, the build is as today. When one is found, its tag goes to both build workflows as `checked`, and the build checks that ship no file and passed in its run are skipped: the Windows Vulkan back end job (every step gated on `RELEASE_CHECKED`) and the AppImage's AMD FSR 4 build check. - **Files are not reused, on purpose**: every release file carries its version in its content, not only its name. The AppImage has `usr/share/longshot/VERSION` (read by `linux_tools.py`/`linux_update.py` for the tools folder, and by `nx2/version.py`) and `longshot_vk.so`, whose `vk_menu.cpp` is compiled with `-DLONGSHOT_VERSION="<tag> (source …)"` (the menu, the bug report). The zip has `VERSION` (the setup installs into `versions\<version>`) and the builder published with `-p:Version=<X.Y.Z-rc.N>` (a nightly's is 0.0.0). A renamed nightly file would claim the nightly's version, so both files are always built again. Carrying the commit instead and reading the version from the release would change the setup, the AppImage, the game's version line and RecompLauncher's hand installs; rebuilding only the parts that change would still recompile the whole Vulkan back end for the one object that has the version. Neither is simpler than building the file again. - **Tests**: `tests/test_release.py` (shared) gets `ReleasedAs`: the lookup (by the tag's commit, not by the 8 hex in the name; drafts, other commits, the tag itself and non-product tags ignored; a stable finds its rc), the verification (untrusted key, corrupt or non-base64 signature, a file more or less than `SHA256SUMS` names, a missing `.sig`, a file that doesn't answer: each stops it), the download through the address without Cloudflare, and the command. `tests/test_windows_on_linux.py` checks that no dav1d is built in CI, that the `package` job checks `tools.lock`'s dav1d under Wine, and that `checked` gates every step of the Vulkan job and only the FSR check on Linux. - **Shared files**: `scripts/release.py`, `tests/test_release.py`, `docs/releases.md` and `RELEASING.md` change the same way in MagicBeanDev/LongshotLauncher (https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/11); the four files are byte-identical (`cmp`). ## Flags and settings None. (`checked` is a new optional input of `build-linux.yml` and `build-windows.yml`, empty by default: a run by hand behaves as before.) ## Measurements main: e28236b4 (2026-11-13) this PR: release-speed @ 6a9895ee Hardware: the Unraid runner (Ryzen 5 3400G, 4C/8T, shared with a VM), capacity 3, `ghcr.io/catthehacker/ubuntu:act-24.04` in Docker-in-Docker Scene: nightly run 717 (`nightly-34-545618fc`); its build workflows are main's (545618fc..e28236b4 changes no build workflow) Method: per-step durations from Forgejo's `action_task_step` (the stored logs keep no per-step markers), queue time from `action_run_job`; one run (a release run can't be repeated on demand without making releases; the test suite below ran interleaved, twice each). This PR's own numbers wait for its first nightly and tag: the build workflows have no pull-request trigger and the fork has no runner How to reproduce: `sqlite3 -readonly gitea.db "select task_id, \"index\", stopped-started, name from action_task_step where task_id in (446,448,449,458,497)"` | | main @ e28236b4 (run 717) | this PR @ 6a9895ee (expected) | |---|---|---| | `dav1d` job (runner time) | 238 s (apt 86 s, build 114 s, Wine test 14 s), queued 3.7 min for a slot | gone | | Zip job (`package`) | 197 s | about 197 + 25 s (the 20 MB `cpm` download, the test compile and its Wine run) | | Build jobs per release | 4 on 3 slots | 3 on 3 slots | | A tag on a released commit: Windows Vulkan job | 300 s (compile 259 s) | skipped (a container start) | | A tag on a released commit: AppImage's FSR check | 23 s | skipped | | Critical path (`appimage`) | 736 s | unchanged (it carries the version) | ``` run 717 (05:56:05 -> 06:17:32 UTC, 21.5 min) appimage 736 s: System packages 154, Node 6, checkout 7, deps (Python with Tk) 108, tools.lock tools 61, AppImage 339 (Vulkan back end 226), FSR check 23, upload 17, player's build 9 package 197 s: Python 15, .NET 18, Wine 38, llvm-mingw 23, setup 3, dotnet publish 30, package 7, zip 13, PE check 3, Wine start 25, upload 14 vulkan 300 s: Python 16, llvm-mingw 14, longshot_vk.dll 259 dav1d 238 s (started 06:00:16, 3.7 min after the others): apt 86, llvm-mingw 16, dav1d 114, Wine test 14 publish 104 s, queued 06:09:02 -> 06:15:48: artifact downloads 45 + 44, sign 1, notes 1, upload 2 ``` ## Testing **Tested:** `LONGSHOT_VULKAN_HEADERS=/usr/include python -I -B tests/run_all.py` on main @ e28236b4 and on this branch, interleaved, twice each (Linux, CachyOS): 500 passed, 2 failed, 36 skipped on all four runs (main 92 s, 92 s; this branch 106 s, 86 s); the 2 failures are the known `test_bake_portability` and `test_builder_log_read_only`, on main too. `tests/test_release.py` passes in the LongshotLauncher copy as well. `released-as` against git.magicbean.dev: for 545618fc (tagged `v3.0.0-rc.1`), `--not v3.0.0-rc.1` prints `nightly-34-545618fc`, its `SHA256SUMS` checked with the real release key. `tests/test_dav1d_av1.py --cross` with llvm-mingw 20261104 compiles and links against the `cpm` package's `libdav1d.a` installed by `tools.py install --only cpm`. actionlint 1.7.12 reports nothing new against main. **Not tested:** the workflows themselves, because the fork's branch has no runner: the Wine run of the dav1d test in the zip's job (there's no Wine here) and the `checked` path of a tag. The first nightly after merge runs the dav1d check ("host/dav1d_av1.c on it, under Wine"). The first tag on a nightly's commit runs `released-as`: its *The version* prints a `::notice::` naming the nightly, and the Vulkan job's steps are skipped. The maintainer who merges and tags sees both. **Needs other testers:** None ## Screenshots Nothing on screen ## Risk and rollback - If `released-as` finds a release of the commit whose `SHA256SUMS` does not check, the tag's run stops in *The version*: by design, nothing is built or published. Delete the bad release (or fix the key) and push the tag again. - If Forgejo did not apply the step `if:`s as expected, the Vulkan job would simply run as today. - If the `cpm` package's dav1d failed under Wine, the zip's job fails and so does the release: that is the library players get. Reverting the commit brings the old job back. ## Reviewer notes Start with `released_as()` in `scripts/release.py` and the version step in `release.yml`. Left for follow-ups (report in the first comment): `publish` waited 6.8 min for a runner in run 717 (it could run on `quick`: 104 s); release runs read no cache by design, though `tools.py` checks every cached archive's SHA-256, so a cache of `tools.lock`'s archives would be safe for releases too. ## Issues None. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.
ci(release): no dav1d build per release, and a tag skips the build checks its commit's release passed
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Failing after 7s
Third-party parts / As their notes say (pull_request) Successful in 12s
PR description / Description follows the template (pull_request_target) Successful in 8s
6a9895eede
The dav1d job built dav1d 1.5.4 from source on every release, for an artifact
nothing used: the Windows builder gets dav1d from tools.lock's cpm package.
The Longshot Builder zip's job now checks that package's dav1d under Wine with
tests/test_dav1d_av1.py; scripts/windows/dav1d.sh stays for making the package.

On a tag, release.py released-as finds the published release of the same
commit (its nightly, or a stable's release candidate), its SHA256SUMS signed
by a trusted key and naming its every file; one that is not stops the run.
The build checks that ship no file (the Windows Vulkan back end, AMD FSR 4)
are then not run again. The files are always built again: each carries its
version (VERSION, longshot_vk.so's version string, the builder's version).
HumanBean changed title from WIP: ci(release): no dav1d build per release, and a tag skips the build checks its commit's release passed to ci(release): no dav1d build per release, and a tag skips the build checks its commit's release passed 2026-11-13 06:31:03 +00:00
Author
Owner

Where the time goes in run 717 (nightly-34-545618fc, 21.5 min overall), and what caching would gain. These are follow-ups, not part of this PR.

  • Critical path: appimage, 736 s. The steps: apt 154 s, deps (Python with Tk) 108 s, tools.lock tools 61 s, AppImage 339 s (Vulkan back end compile 226 s, squashfs 55 s), FSR check 23 s.
  • Publish: publish waited 6.8 min for a runner slot, then ran in 104 s (90 s of that is the two artifact downloads through Cloudflare). Moving it to runs-on: quick is a one-line change and the biggest single gain.
  • (a) The cache already works. It is runner-local, under /data/.cache/actcache, and the quick runner has its own. The evidence: llvm-mingw-… was saved by task 278 and restored by task 279, and linux-deps-downloads-… was saved in run 717. No cache: setting needs to change. Release runs read no cache by design, though. Since tools.py fetch checks every cached archive's size and SHA-256, caching tools-cache and llvm-mingw would be safe on releases too.
    • Effort: about 1 h.
    • Gain: about 60 s on appimage and 20 s on package.
  • (b) Objects on releases (ccache or nx2's objcache). The objcache already exists, but only pull requests read it. Letting main and tags restore it would cut the AppImage's 226 s compile to about 20–40 s, and do the same for the Windows check's 259 s. The cost is trusting objects written by other runs: the cache isn't scoped per ref, so a pull request's run could write it.
    • Effort: about 2–4 h, plus a decision on that trust.
    • Gain: about 3 min on the critical path.
  • (c) Prebuilt job images in the forge's registry. These would cover ubuntu:22.04 with the AppImage's apt packages and Node, and the act image with Wine and .NET.
    • Effort: about 1 day (Dockerfiles, a build and push workflow, digests pinned, test_linux_docker_packages kept in step).
    • Gain: about 2.5 min on the critical path (apt 154 s), and about 1 min on package (Wine 38 s, .NET and Python setup 33 s).
Where the time goes in run 717 (nightly-34-545618fc, 21.5 min overall), and what caching would gain. These are follow-ups, not part of this PR. - **Critical path:** `appimage`, 736 s. The steps: apt 154 s, deps (Python with Tk) 108 s, tools.lock tools 61 s, AppImage 339 s (Vulkan back end compile 226 s, squashfs 55 s), FSR check 23 s. - **Publish:** `publish` waited 6.8 min for a runner slot, then ran in 104 s (90 s of that is the two artifact downloads through Cloudflare). Moving it to `runs-on: quick` is a one-line change and the biggest single gain. - **(a) The cache already works.** It is runner-local, under `/data/.cache/actcache`, and the quick runner has its own. The evidence: `llvm-mingw-…` was saved by task 278 and restored by task 279, and `linux-deps-downloads-…` was saved in run 717. No `cache:` setting needs to change. Release runs read no cache by design, though. Since `tools.py fetch` checks every cached archive's size and SHA-256, caching `tools-cache` and llvm-mingw would be safe on releases too. - Effort: about 1 h. - Gain: about 60 s on `appimage` and 20 s on `package`. - **(b) Objects on releases (ccache or nx2's objcache).** The objcache already exists, but only pull requests read it. Letting main and tags restore it would cut the AppImage's 226 s compile to about 20–40 s, and do the same for the Windows check's 259 s. The cost is trusting objects written by other runs: the cache isn't scoped per ref, so a pull request's run could write it. - Effort: about 2–4 h, plus a decision on that trust. - Gain: about 3 min on the critical path. - **(c) Prebuilt job images in the forge's registry.** These would cover ubuntu:22.04 with the AppImage's apt packages and Node, and the act image with Wine and .NET. - Effort: about 1 day (Dockerfiles, a build and push workflow, digests pinned, `test_linux_docker_packages` kept in step). - Gain: about 2.5 min on the critical path (apt 154 s), and about 1 min on `package` (Wine 38 s, .NET and Python setup 33 s).
HumanBean deleted branch release-speed 2026-11-13 06:34:11 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!143
No description provided.