ci(release): Sign and publish takes the builders from Forgejo's own address, and Wine installs beside the Windows build #1110

Merged
HumanBean merged 2 commits from HumanBean/Longshot:ci/release-faster into main 2026-11-16 11:31:46 +00:00
Owner

Type: ci · Risk: low · Tested on: Linux (the new command against a real run's artifacts, over the public address) · Needs: the first nightly after merge

Why

A nightly or a release spends minutes on steps that wait on the network rather than build anything. In nightly-325's run (#28520), Sign and publish took 2 min 55 s. About 2 min of that went on downloading the two builders (94 MB and 47 MB) through Forgejo's public address behind Cloudflare at ~1.2 MB/s: the server sends them out over the home uplink and takes them straight back in. The Windows builder's zip job also waits ~30 s for Wine's apt before it starts building. And every nightly and release recompiles the whole Vulkan back end from scratch (~44 s on Linux, ~26 s on Windows) and downloads tools.lock's tools again, though the back end is only a build check that no release file carries.

What changed

Sign and publish now fetches the build jobs' artifacts from Forgejo's own address (RELEASE_UPLOAD_API, which this job already uses for uploads). Artifacts are kept on Forgejo's disk, not in R2, so this is a local transfer. The zip job also installs Wine in the background while the builder compiles. A second commit lets the nightly and releases use the runner's caches for what they don't ship.

  • scripts/release.py artifacts <folder> --run <run id> <name>...: lists the run's artifacts (/repos/{repo}/actions/runs/{id}/artifacts), fetches each archive_download_url with the host swapped for FORGEJO_UPLOAD_API's (the existing download()), and unpacks it. It refuses a name the run lacks or has twice, and any zip entry that is not a plain file name. Nothing is written until every artifact is fetched.
  • release.yml, Sign and publish: a new first step artifacts (continue-on-error). The four download-artifact steps now run only when it failed, so they stay as the fallback, with their retry.
  • build-windows.yml, Longshot Builder zip: Wine starts the apt install detached (setsid nohup, all descriptors redirected, status written last by mv). A new Wine, installed step just before Start them under Wine waits for it (10 min cap), prints its log, and fails on a non-zero status. The packages are unchanged.
  • Caches (second commit), for what a release doesn't ship:
    • build-linux.yml, build-windows.yml (vulkan): the Vulkan back end's object cache is now read and written by every build of it, not only by pull requests. The back end is a build check: the AppImage carries none, and the Windows longshot_vk.dll is an artifact that is never released. Keys are as before (source, flags and headers SHA-1, plus the compiler and the system headers). A commit that's already released (RELEASE_CHECKED) still skips the build and leaves the cache as it is. The prune step lost its pull-request-only name and condition, since it does nothing when there are no stamps.
    • tools.lock's downloads ($RUNNER_TEMP/tools-cache) are kept for every run on both systems. tools.py fetch checks a kept archive's size and SHA-256 before using it.
    • Longshot Builder zip: a release keeps llvm-mingw's archive (llvm/.cache, checked the same way), not its unpacked folder, because that llvm-mingw compiles the shipped setup and third-party objects. Other runs keep the folder as before.
    • Unchanged: what a release ships is still made only from checked downloads. That covers the AppImage's Python and deps (linux-deps-built stays non-release) and the zip's third-party objects, which nx2/third_party_objects.py make compiles in a fresh work folder with no object store.
  • tests/test_release.py: Artifacts, against a fake Forgejo on 127.0.0.1. It checks that the files come from the FORGEJO_UPLOAD_API host though the list links a public one, that the token is sent, and that a missing name or a ../ entry writes nothing.

Measurements

Verdict: expected to save ~3.5 min per Longshot nightly or release: ~2 min in Sign and publish, ~30 s in the zip job, and ~1 min from the object caches and kept downloads (most of the ~44 s Linux and ~26 s Windows back-end compiles, plus ~10 s of tool downloads). The internal address can't be reached from my PC, so the real number comes from the first nightly after merge.

Where the time went in nightly-325 (run #28520, from the job logs' timestamps):

Job Total Spent waiting
Sign and publish 2m55s 76 s + 45 s downloading the AppImage and the zip (download-artifact, through Cloudflare)
Longshot Builder zip 2m21s ~29 s of Wine's apt before the build
appimage 3m34s (not changed here: ~88 s of apt, ~30 s of it software-properties-common for add-apt-repository; a follow-up, since two checks run that step)

Fetching the same two artifacts with the new command through the public address from my PC: 141 MB in 36.5 s. From the quick runner over RELEASE_UPLOAD_API it never leaves the server.

Flags and settings

None.

Testing

Tested:

  • release.py artifacts against run 30420's real artifacts (Linux, public address, my sign-in): both builders unpacked under their names, byte sizes as listed. A name the run lacks stops it: release.py: run 30420 has 0 artifacts named nope, not one.
  • The background-install pattern in a local shell, with separate shells standing in for steps: the first returns at once (its stdout piped, so no descriptor is held), and the waiting step passes on status 0 and fails on 1.
  • The edited workflows parse (PyYAML), and the new steps' if: conditions are as intended.

Not tested:

  • python -I -B tests/run_all.py, including the new Artifacts test: left to CI.
  • The job token (forgejo.token) reading the artifacts API. If it can't, the step fails and download-artifact runs as before.

Must be tested:

  • The first nightly after merge: The builders, from Forgejo's own address passes and the download-artifact steps are skipped; Wine, installed passes and Start them under Wine behaves as before (by hand: the run's log).

Risks

What could break: a cached object whose key misses something its compile depends on would turn the back end's build check falsely green. It can't reach a player: no release file carries the back end, and pull requests have used the same cache all along. The cache is the release runner's own, written only by release builds and Run workflow.

Also: if the artifacts API answers differently or refuses the job token, the step fails, the fallback fetches through Cloudflare, and the run is as slow as today. If the background apt fails, Wine, installed stops the job with the apt log, as Wine did before.

Turning it off: revert the commit.

Compatibility: nothing changes for players. The release files and SHA256SUMS are made from the same bytes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). Left to CI.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

🤖 Generated with Claude Code

https://claude.ai/code/session_017vw8dzkpXRZ2cBAf7vu5YQ

**Type:** ci · **Risk:** low · **Tested on:** Linux (the new command against a real run's artifacts, over the public address) · **Needs:** the first nightly after merge ## Why A nightly or a release spends minutes on steps that wait on the network rather than build anything. In nightly-325's run ([#28520](https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/28520)), *Sign and publish* took 2 min 55 s. About 2 min of that went on downloading the two builders (94 MB and 47 MB) through Forgejo's public address behind Cloudflare at ~1.2 MB/s: the server sends them out over the home uplink and takes them straight back in. The Windows builder's zip job also waits ~30 s for Wine's apt before it starts building. And every nightly and release recompiles the whole Vulkan back end from scratch (~44 s on Linux, ~26 s on Windows) and downloads tools.lock's tools again, though the back end is only a build check that no release file carries. ## What changed *Sign and publish* now fetches the build jobs' artifacts from Forgejo's own address (`RELEASE_UPLOAD_API`, which this job already uses for uploads). Artifacts are kept on Forgejo's disk, not in R2, so this is a local transfer. The zip job also installs Wine in the background while the builder compiles. A second commit lets the nightly and releases use the runner's caches for what they don't ship. - `scripts/release.py artifacts <folder> --run <run id> <name>...`: lists the run's artifacts (`/repos/{repo}/actions/runs/{id}/artifacts`), fetches each `archive_download_url` with the host swapped for `FORGEJO_UPLOAD_API`'s (the existing `download()`), and unpacks it. It refuses a name the run lacks or has twice, and any zip entry that is not a plain file name. Nothing is written until every artifact is fetched. - `release.yml`, *Sign and publish*: a new first step `artifacts` (`continue-on-error`). The four `download-artifact` steps now run only when it failed, so they stay as the fallback, with their retry. - `build-windows.yml`, *Longshot Builder zip*: *Wine* starts the apt install detached (`setsid nohup`, all descriptors redirected, status written last by `mv`). A new *Wine, installed* step just before *Start them under Wine* waits for it (10 min cap), prints its log, and fails on a non-zero status. The packages are unchanged. - Caches (second commit), for what a release doesn't ship: - `build-linux.yml`, `build-windows.yml` (*vulkan*): the Vulkan back end's object cache is now read and written by every build of it, not only by pull requests. The back end is a build check: the AppImage carries none, and the Windows `longshot_vk.dll` is an artifact that is never released. Keys are as before (source, flags and headers SHA-1, plus the compiler and the system headers). A commit that's already released (`RELEASE_CHECKED`) still skips the build and leaves the cache as it is. The prune step lost its pull-request-only name and condition, since it does nothing when there are no stamps. - tools.lock's downloads (`$RUNNER_TEMP/tools-cache`) are kept for every run on both systems. `tools.py fetch` checks a kept archive's size and SHA-256 before using it. - *Longshot Builder zip*: a release keeps llvm-mingw's *archive* (`llvm/.cache`, checked the same way), not its unpacked folder, because that llvm-mingw compiles the shipped setup and third-party objects. Other runs keep the folder as before. - Unchanged: what a release ships is still made only from checked downloads. That covers the AppImage's Python and deps (`linux-deps-built` stays non-release) and the zip's third-party objects, which `nx2/third_party_objects.py make` compiles in a fresh work folder with no object store. - `tests/test_release.py`: `Artifacts`, against a fake Forgejo on 127.0.0.1. It checks that the files come from the `FORGEJO_UPLOAD_API` host though the list links a public one, that the token is sent, and that a missing name or a `../` entry writes nothing. ## Measurements **Verdict:** expected to save ~3.5 min per Longshot nightly or release: ~2 min in *Sign and publish*, ~30 s in the zip job, and ~1 min from the object caches and kept downloads (most of the ~44 s Linux and ~26 s Windows back-end compiles, plus ~10 s of tool downloads). The internal address can't be reached from my PC, so the real number comes from the first nightly after merge. Where the time went in nightly-325 (run [#28520](https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/28520), from the job logs' timestamps): | Job | Total | Spent waiting | |---|---|---| | Sign and publish | 2m55s | 76 s + 45 s downloading the AppImage and the zip (download-artifact, through Cloudflare) | | Longshot Builder zip | 2m21s | ~29 s of Wine's apt before the build | | appimage | 3m34s | (not changed here: ~88 s of apt, ~30 s of it `software-properties-common` for `add-apt-repository`; a follow-up, since two checks run that step) | Fetching the same two artifacts with the new command through the *public* address from my PC: 141 MB in 36.5 s. From the quick runner over `RELEASE_UPLOAD_API` it never leaves the server. ### Flags and settings None. ## Testing **Tested:** - `release.py artifacts` against run 30420's real artifacts (Linux, public address, my sign-in): both builders unpacked under their names, byte sizes as listed. A name the run lacks stops it: `release.py: run 30420 has 0 artifacts named nope, not one`. - The background-install pattern in a local shell, with separate shells standing in for steps: the first returns at once (its stdout piped, so no descriptor is held), and the waiting step passes on status 0 and fails on 1. - The edited workflows parse (PyYAML), and the new steps' `if:` conditions are as intended. **Not tested:** - `python -I -B tests/run_all.py`, including the new `Artifacts` test: left to CI. - The job token (`forgejo.token`) reading the artifacts API. If it can't, the step fails and download-artifact runs as before. **Must be tested:** - The first nightly after merge: *The builders, from Forgejo's own address* passes and the download-artifact steps are skipped; *Wine, installed* passes and *Start them under Wine* behaves as before (by hand: the run's log). ## Risks **What could break:** a cached object whose key misses something its compile depends on would turn the back end's build check falsely green. It can't reach a player: no release file carries the back end, and pull requests have used the same cache all along. The cache is the release runner's own, written only by release builds and Run workflow. Also: if the artifacts API answers differently or refuses the job token, the step fails, the fallback fetches through Cloudflare, and the run is as slow as today. If the background apt fails, *Wine, installed* stops the job with the apt log, as *Wine* did before. **Turning it off:** revert the commit. **Compatibility:** nothing changes for players. The release files and SHA256SUMS are made from the same bytes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). Left to CI. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017vw8dzkpXRZ2cBAf7vu5YQ
ci(release): Sign and publish takes the builders from Forgejo's own address, and Wine installs beside the Windows build
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 14s
cb7dbf233f
Sign and publish fetched the two builders through Forgejo's public address behind Cloudflare, at about 1 MB/s: two of
its three minutes (nightly-325). scripts/release.py artifacts fetches a run's named artifacts from RELEASE_UPLOAD_API,
the address on the quick runner's network, where they are on Forgejo's disk; download-artifact stays as the fallback.

The Windows builder's zip job waited about 30 s for Wine's apt before building: it now runs in the background and
"Wine, installed" waits for it before the first step that starts Wine.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017vw8dzkpXRZ2cBAf7vu5YQ
ci(release): the nightly and the releases take the runner's caches for what they do not ship
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 14s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 21s
CI / Workflow lint (actionlint) (pull_request) Successful in 21s
Third-party parts / As their notes say (pull_request) Successful in 30s
PR description / Description follows the template (pull_request_target) Failing after 5s
28c5f1c7b9
The Vulkan back end, built on Linux and Windows, is a build check: no release file carries it (every player's build
compiles its own). Its object cache, read until now by pull requests only, is read and written by every build, so a
release compiles only the objects its changes reach (about 40 s of the Linux build and 30 s of the Windows one).
tools.lock's downloads are kept for every run (scripts/tools.py checks a kept archive's size and SHA-256 before it takes
it), and a release keeps llvm-mingw's archive (checked the same way), not its unpacked folder: what a release ships, the
AppImage's Python and the zip's setup and third-party objects, is still made from checked downloads only.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017vw8dzkpXRZ2cBAf7vu5YQ
Author
Owner

Pushed 28c5f1c7: the nightly and releases now use the runner's caches for what they don't ship. The Vulkan back end's object cache (a build check that no release file carries) is used by every build, not only by pull requests. tools.lock's downloads are kept for every run, and checked by SHA-256 before use. A release keeps llvm-mingw's archive, not its unpacked folder. Shipped files are still made only from checked downloads. The description is updated (What changed, Measurements, Risks).

Pushed 28c5f1c7: the nightly and releases now use the runner's caches for what they don't ship. The Vulkan back end's object cache (a build check that no release file carries) is used by every build, not only by pull requests. tools.lock's downloads are kept for every run, and checked by SHA-256 before use. A release keeps llvm-mingw's archive, not its unpacked folder. Shipped files are still made only from checked downloads. The description is updated (What changed, Measurements, Risks).
HumanBean changed title from WIP: ci(release): Sign and publish takes the builders from Forgejo's own address, and Wine installs beside the Windows build to ci(release): Sign and publish takes the builders from Forgejo's own address, and Wine installs beside the Windows build 2026-11-16 11:31:40 +00:00
HumanBean deleted branch ci/release-faster 2026-11-16 11:31:47 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1110
No description provided.