ci(release): Sign and publish takes the builders from Forgejo's own address, and Wine installs beside the Windows build #1110
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1110
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:ci/release-faster"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: ci · Risk: low · Tested on: Linux (the new command against a real run's artifacts, over the public address) · Needs: the first nightly after merge
Why
A nightly or a release spends minutes on steps that wait on the network rather than build anything. In nightly-325's run (#28520), Sign and publish took 2 min 55 s. About 2 min of that went on downloading the two builders (94 MB and 47 MB) through Forgejo's public address behind Cloudflare at ~1.2 MB/s: the server sends them out over the home uplink and takes them straight back in. The Windows builder's zip job also waits ~30 s for Wine's apt before it starts building. And every nightly and release recompiles the whole Vulkan back end from scratch (~44 s on Linux, ~26 s on Windows) and downloads tools.lock's tools again, though the back end is only a build check that no release file carries.
What changed
Sign and publish now fetches the build jobs' artifacts from Forgejo's own address (
RELEASE_UPLOAD_API, which this job already uses for uploads). Artifacts are kept on Forgejo's disk, not in R2, so this is a local transfer. The zip job also installs Wine in the background while the builder compiles. A second commit lets the nightly and releases use the runner's caches for what they don't ship.scripts/release.py artifacts <folder> --run <run id> <name>...: lists the run's artifacts (/repos/{repo}/actions/runs/{id}/artifacts), fetches eacharchive_download_urlwith the host swapped forFORGEJO_UPLOAD_API's (the existingdownload()), and unpacks it. It refuses a name the run lacks or has twice, and any zip entry that is not a plain file name. Nothing is written until every artifact is fetched.release.yml, Sign and publish: a new first stepartifacts(continue-on-error). The fourdownload-artifactsteps now run only when it failed, so they stay as the fallback, with their retry.build-windows.yml, Longshot Builder zip: Wine starts the apt install detached (setsid nohup, all descriptors redirected, status written last bymv). A new Wine, installed step just before Start them under Wine waits for it (10 min cap), prints its log, and fails on a non-zero status. The packages are unchanged.build-linux.yml,build-windows.yml(vulkan): the Vulkan back end's object cache is now read and written by every build of it, not only by pull requests. The back end is a build check: the AppImage carries none, and the Windowslongshot_vk.dllis an artifact that is never released. Keys are as before (source, flags and headers SHA-1, plus the compiler and the system headers). A commit that's already released (RELEASE_CHECKED) still skips the build and leaves the cache as it is. The prune step lost its pull-request-only name and condition, since it does nothing when there are no stamps.$RUNNER_TEMP/tools-cache) are kept for every run on both systems.tools.py fetchchecks a kept archive's size and SHA-256 before using it.llvm/.cache, checked the same way), not its unpacked folder, because that llvm-mingw compiles the shipped setup and third-party objects. Other runs keep the folder as before.linux-deps-builtstays non-release) and the zip's third-party objects, whichnx2/third_party_objects.py makecompiles in a fresh work folder with no object store.tests/test_release.py:Artifacts, against a fake Forgejo on 127.0.0.1. It checks that the files come from theFORGEJO_UPLOAD_APIhost though the list links a public one, that the token is sent, and that a missing name or a../entry writes nothing.Measurements
Verdict: expected to save ~3.5 min per Longshot nightly or release: ~2 min in Sign and publish, ~30 s in the zip job, and ~1 min from the object caches and kept downloads (most of the ~44 s Linux and ~26 s Windows back-end compiles, plus ~10 s of tool downloads). The internal address can't be reached from my PC, so the real number comes from the first nightly after merge.
Where the time went in nightly-325 (run #28520, from the job logs' timestamps):
software-properties-commonforadd-apt-repository; a follow-up, since two checks run that step)Fetching the same two artifacts with the new command through the public address from my PC: 141 MB in 36.5 s. From the quick runner over
RELEASE_UPLOAD_APIit never leaves the server.Flags and settings
None.
Testing
Tested:
release.py artifactsagainst run 30420's real artifacts (Linux, public address, my sign-in): both builders unpacked under their names, byte sizes as listed. A name the run lacks stops it:release.py: run 30420 has 0 artifacts named nope, not one.if:conditions are as intended.Not tested:
python -I -B tests/run_all.py, including the newArtifactstest: left to CI.forgejo.token) reading the artifacts API. If it can't, the step fails and download-artifact runs as before.Must be tested:
Risks
What could break: a cached object whose key misses something its compile depends on would turn the back end's build check falsely green. It can't reach a player: no release file carries the back end, and pull requests have used the same cache all along. The cache is the release runner's own, written only by release builds and Run workflow.
Also: if the artifacts API answers differently or refuses the job token, the step fails, the fallback fetches through Cloudflare, and the run is as slow as today. If the background apt fails, Wine, installed stops the job with the apt log, as Wine did before.
Turning it off: revert the commit.
Compatibility: nothing changes for players. The release files and SHA256SUMS are made from the same bytes.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Left to CI.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.🤖 Generated with Claude Code
https://claude.ai/code/session_017vw8dzkpXRZ2cBAf7vu5YQ
Pushed
28c5f1c7: the nightly and releases now use the runner's caches for what they don't ship. The Vulkan back end's object cache (a build check that no release file carries) is used by every build, not only by pull requests. tools.lock's downloads are kept for every run, and checked by SHA-256 before use. A release keeps llvm-mingw's archive, not its unpacked folder. Shipped files are still made only from checked downloads. The description is updated (What changed, Measurements, Risks).WIP: ci(release): Sign and publish takes the builders from Forgejo's own address, and Wine installs beside the Windows buildto ci(release): Sign and publish takes the builders from Forgejo's own address, and Wine installs beside the Windows build