ci(nightly): a rolling nightly prerelease of main, linked from the readme #1062

Closed

hckrman101 wants to merge 9 commits from ci-nightly-prerelease into main

hckrman101 commented

2026-11-09 08:49:28 +00:00

(Migrated from github.com)

Depends on #1059: this branch is built on it, because both change the release jobs of the two build workflows and scripts/release_upload.sh. Until #1059 is merged, the diff here also shows that PR's commits. Only ci(nightly): a rolling nightly prerelease of main, linked from the readme, the merge of #1059's branch into this one, and ci(nightly): the nightly's AppImage updates to the next nightly, and its files keep SHA256SUMS current belong to this PR.

What this adds

Today the only way to get a build of main is a workflow run's artifacts, and GitHub only lets signed-in users download those. This adds a rolling nightly prerelease that anyone can download, and a readme section that links to it.

How it works

On every push to main, once the Linux AppImage and Windows builder workflows have built that commit, their existing release jobs run as they do for a release. They pass nightly to scripts/release_upload.sh, so a release and the nightly share one upload step, and no build step is duplicated.
For nightly, release_upload.sh moves the nightly tag to the commit through the git refs API. The move uses GITHUB_TOKEN, so it starts no workflow. The script then writes the commit SHA and its date in the notes, keeps the release a prerelease and never "latest", and replaces the files with --clobber. Once the new AppImage and .zsync are up, it removes the previous commit's, so neither is ever missing. Then it makes SHA256SUMS again (scripts/release_files.sh finish). The Linux and Windows jobs both do this, and each one checks the uploaded SHA256SUMS against the files afterwards, so it ends up listing every file.
The nightly carries the same set of files as a release, with the same names: Hookshot-<git describe>-x86_64.AppImage and its .zsync, hookshot-linux.txt, HookshotBuilder.exe, hookshot-setup.txt and SHA256SUMS. Every file has a build provenance attestation, as for a release (#1059), so gh attestation verify <file> -R NotRexed/Hookshot works on a nightly too. The notes end with the same "Which file do I download?" table. On a push to main, the Linux workflow now also keeps its update-parts artifact, which it used to keep only for releases. New parts go to linux-parts and builder-parts as they do for a release, so both lists point at files that exist.
A run that finishes after a newer push to main leaves the nightly alone, so an older commit never overwrites a newer one. If one of the two builds fails, its files stay at the last commit that built.
The merge of a release PR runs both a release and a nightly of the same commit, and both can upload the same new part at once. The parts uploads now treat a part that the other run has just uploaded as done, instead of failing.
The nightly AppImage's update information names the nightly: gh-releases-zsync|NotRexed|Hookshot|nightly|Hookshot-*-x86_64.AppImage.zsync. The Linux build sets HOOKSHOT_UPDATE_RELEASE=nightly on a push to main that is not a release. AppImageUpdate reads releases/tags/nightly for it, so a nightly AppImage updates to the next nightly, not to a release. Release AppImages keep latest. Gear Lever ignores the release name unless it is latest-pre or latest-all, and always reads releases/latest. For a nightly AppImage, it offers the latest release instead. docs/versioning.md notes this.
The tag-triggered jobs skip a pushed nightly tag, the same way they skip builder-parts and linux-parts.
The nightly is invisible to everything that looks for releases. The builder's update check and the setup use releases/latest, release_parts.py previous skips prereleases, and nightly is not a v* tag, so release-please and git describe ignore it.
The backstop from #1059 still applies: release_upload.sh refuses file names that look like game data, dumps or NVIDIA runtime files.
The readme gets a "Latest development build" section after "Updating". It links to https://github.com/NotRexed/Hookshot/releases/tag/nightly and warns that the build is untested and may break saves or settings. On Windows, it says to start the setup as HookshotBuilder.exe --tag nightly: builder/setup/setup.c reads --tag to pick that release's hookshot-setup.txt, and without it the setup installs the latest release. docs/versioning.md gets a short nightly section.

How it was tested

actionlint 1.7.7 with shellcheck 0.10.0 passes on all workflows, and shellcheck passes on scripts/release_upload.sh, scripts/release_files.sh and scripts/linux/publish_parts.sh. python -I -B tests/run_all.py passes.
On the fork (hckrman101/Hookshot), a test branch nightly-test stood in for main. The only test-only changes were the branch name in the two build workflows and in release_upload.sh's check for main's newest commit.
- First push (https://github.com/hckrman101/Hookshot/actions/runs/42653679399 and https://github.com/hckrman101/Hookshot/actions/runs/42653679049): it made the nightly prerelease at that commit, with Hookshot-v1.0.0-4-gc8e8161-x86_64.AppImage, its .zsync, hookshot-linux.txt, HookshotBuilder.exe, hookshot-setup.txt and SHA256SUMS, and the table in the notes. It was not "latest". sha256sum -c SHA256SUMS passed for all five files, and gh attestation verify passed for all six. The AppImage's update information is gh-releases-zsync|hckrman101|Hookshot|nightly|Hookshot-*-x86_64.AppImage.zsync, and its .zsync matched the file (length, SHA-1, name, relative URL, and a sample of block sums).
- Second push (https://github.com/hckrman101/Hookshot/actions/runs/42654955199 and https://github.com/hckrman101/Hookshot/actions/runs/42654955237): the tag moved, Hookshot-v1.0.0-5-g4c81b23-x86_64.AppImage and its .zsync replaced the first push's, which were removed, and SHA256SUMS was made again. It listed exactly the new files, and sha256sum -c and gh attestation verify passed again. I followed AppImageUpdate's code with the first push's update information, and it resolved to the second push's .zsync, whose AppImage was on the release.
The release path from #1059 was tested at the same time on another branch (see #1059). Afterwards I deleted the fork's test releases, tags and branches.

What a maintainer has to do

Nothing to configure: the release jobs already have contents: write, and the nightly needs nothing more.
Each push to main now uploads the AppImage (about 215 MB) and the parts that changed, usually app zips of a few MB each. As with releases, old parts stay on linux-parts and builder-parts, so prune those now and then if they grow too large.
The first push to main after this merges creates the nightly tag and release. Until then, the readme link goes nowhere.

Depends on #1059: this branch is built on it, because both change the release jobs of the two build workflows and `scripts/release_upload.sh`. Until #1059 is merged, the diff here also shows that PR's commits. Only `ci(nightly): a rolling nightly prerelease of main, linked from the readme`, the merge of #1059's branch into this one, and `ci(nightly): the nightly's AppImage updates to the next nightly, and its files keep SHA256SUMS current` belong to this PR. ## What this adds Today the only way to get a build of main is a workflow run's artifacts, and GitHub only lets signed-in users download those. This adds a rolling `nightly` prerelease that anyone can download, and a readme section that links to it. ## How it works - On every push to main, once the Linux AppImage and Windows builder workflows have built that commit, their existing release jobs run as they do for a release. They pass `nightly` to `scripts/release_upload.sh`, so a release and the nightly share one upload step, and no build step is duplicated. - For `nightly`, `release_upload.sh` moves the `nightly` tag to the commit through the git refs API. The move uses `GITHUB_TOKEN`, so it starts no workflow. The script then writes the commit SHA and its date in the notes, keeps the release a prerelease and never "latest", and replaces the files with `--clobber`. Once the new AppImage and `.zsync` are up, it removes the previous commit's, so neither is ever missing. Then it makes `SHA256SUMS` again (`scripts/release_files.sh finish`). The Linux and Windows jobs both do this, and each one checks the uploaded `SHA256SUMS` against the files afterwards, so it ends up listing every file. - The nightly carries the same set of files as a release, with the same names: `Hookshot-<git describe>-x86_64.AppImage` and its `.zsync`, `hookshot-linux.txt`, `HookshotBuilder.exe`, `hookshot-setup.txt` and `SHA256SUMS`. Every file has a build provenance attestation, as for a release (#1059), so `gh attestation verify <file> -R NotRexed/Hookshot` works on a nightly too. The notes end with the same "Which file do I download?" table. On a push to main, the Linux workflow now also keeps its update-parts artifact, which it used to keep only for releases. New parts go to `linux-parts` and `builder-parts` as they do for a release, so both lists point at files that exist. - A run that finishes after a newer push to main leaves the nightly alone, so an older commit never overwrites a newer one. If one of the two builds fails, its files stay at the last commit that built. - The merge of a release PR runs both a release and a nightly of the same commit, and both can upload the same new part at once. The parts uploads now treat a part that the other run has just uploaded as done, instead of failing. - The nightly AppImage's update information names the nightly: `gh-releases-zsync|NotRexed|Hookshot|nightly|Hookshot-*-x86_64.AppImage.zsync`. The Linux build sets `HOOKSHOT_UPDATE_RELEASE=nightly` on a push to main that is not a release. AppImageUpdate reads `releases/tags/nightly` for it, so a nightly AppImage updates to the next nightly, not to a release. Release AppImages keep `latest`. Gear Lever ignores the release name unless it is `latest-pre` or `latest-all`, and always reads `releases/latest`. For a nightly AppImage, it offers the latest release instead. docs/versioning.md notes this. - The tag-triggered jobs skip a pushed `nightly` tag, the same way they skip `builder-parts` and `linux-parts`. - The nightly is invisible to everything that looks for releases. The builder's update check and the setup use `releases/latest`, `release_parts.py previous` skips prereleases, and `nightly` is not a `v*` tag, so release-please and `git describe` ignore it. - The backstop from #1059 still applies: `release_upload.sh` refuses file names that look like game data, dumps or NVIDIA runtime files. - The readme gets a "Latest development build" section after "Updating". It links to https://github.com/NotRexed/Hookshot/releases/tag/nightly and warns that the build is untested and may break saves or settings. On Windows, it says to start the setup as `HookshotBuilder.exe --tag nightly`: `builder/setup/setup.c` reads `--tag` to pick that release's `hookshot-setup.txt`, and without it the setup installs the latest release. docs/versioning.md gets a short nightly section. ## How it was tested - actionlint 1.7.7 with shellcheck 0.10.0 passes on all workflows, and shellcheck passes on `scripts/release_upload.sh`, `scripts/release_files.sh` and `scripts/linux/publish_parts.sh`. `python -I -B tests/run_all.py` passes. - On the fork (hckrman101/Hookshot), a test branch `nightly-test` stood in for main. The only test-only changes were the branch name in the two build workflows and in `release_upload.sh`'s check for main's newest commit. - First push (https://github.com/hckrman101/Hookshot/actions/runs/42653679399 and https://github.com/hckrman101/Hookshot/actions/runs/42653679049): it made the `nightly` prerelease at that commit, with `Hookshot-v1.0.0-4-gc8e8161-x86_64.AppImage`, its `.zsync`, `hookshot-linux.txt`, `HookshotBuilder.exe`, `hookshot-setup.txt` and `SHA256SUMS`, and the table in the notes. It was not "latest". `sha256sum -c SHA256SUMS` passed for all five files, and `gh attestation verify` passed for all six. The AppImage's update information is `gh-releases-zsync|hckrman101|Hookshot|nightly|Hookshot-*-x86_64.AppImage.zsync`, and its `.zsync` matched the file (length, SHA-1, name, relative URL, and a sample of block sums). - Second push (https://github.com/hckrman101/Hookshot/actions/runs/42654955199 and https://github.com/hckrman101/Hookshot/actions/runs/42654955237): the tag moved, `Hookshot-v1.0.0-5-g4c81b23-x86_64.AppImage` and its `.zsync` replaced the first push's, which were removed, and `SHA256SUMS` was made again. It listed exactly the new files, and `sha256sum -c` and `gh attestation verify` passed again. I followed AppImageUpdate's code with the first push's update information, and it resolved to the second push's `.zsync`, whose AppImage was on the release. - The release path from #1059 was tested at the same time on another branch (see #1059). Afterwards I deleted the fork's test releases, tags and branches. ## What a maintainer has to do - Nothing to configure: the release jobs already have `contents: write`, and the nightly needs nothing more. - Each push to main now uploads the AppImage (about 215 MB) and the parts that changed, usually app zips of a few MB each. As with releases, old parts stay on `linux-parts` and `builder-parts`, so prune those now and then if they grow too large. - The first push to main after this merges creates the `nightly` tag and release. Until then, the readme link goes nowhere.

hckrman101 (Migrated from github.com) reviewed

2026-11-10 18:13:11 +00:00

hckrman101 (Migrated from github.com) left a comment

Read through the nightly part (tag move through the refs API, the newest-commit check, --clobber plus removal of the previous AppImage/.zsync only after the new ones are up, the SHA256SUMS retry loop, the parts-upload race handling, the nightly tag skips, all action refs SHA-pinned). I found one real problem, inline, and otherwise it looks right.

CI: the red jobs are not from this PR. Both Python source checks fail on nx2/test_draw_parameters (no member named 'mesh_fallback' in 'Ctx', the fixture breakage), test_draw_stream_visibility, test_sampler_admission, test_lift_cleanup_error, test_recomplauncher_json, and (Windows) nx2/test_nup_out_of_space. Those fail on main too. actionlint, AppImage, Windows builder and Vulkan jobs pass. Merging main should clear the fixture ones.

Order: this branch carries #1059's commits (still open), so #1059 has to merge first.

Nits, no action needed: the notes and README say "after every change" while a push that lands before the previous one's builds finish simply skips (documented in the PR text, fine).

Read through the nightly part (tag move through the refs API, the newest-commit check, `--clobber` plus removal of the previous AppImage/.zsync only after the new ones are up, the SHA256SUMS retry loop, the parts-upload race handling, the `nightly` tag skips, all action refs SHA-pinned). I found one real problem, inline, and otherwise it looks right. **CI:** the red jobs are not from this PR. Both Python source checks fail on `nx2/test_draw_parameters` (`no member named 'mesh_fallback' in 'Ctx'`, the fixture breakage), `test_draw_stream_visibility`, `test_sampler_admission`, `test_lift_cleanup_error`, `test_recomplauncher_json`, and (Windows) `nx2/test_nup_out_of_space`. Those fail on main too. actionlint, AppImage, Windows builder and Vulkan jobs pass. Merging main should clear the fixture ones. **Order:** this branch carries #1059's commits (still open), so #1059 has to merge first. Nits, no action needed: the notes and README say "after every change" while a push that lands before the previous one's builds finish simply skips (documented in the PR text, fine).

.github/workflows/windows-builder.yml

Verified problem: the new || (github.event_name == 'push' && github.ref == 'refs/heads/main') clause also matches the Release workflow's workflow_call, because github.event_name and github.ref are the caller's (a push to main). So for a release made by release-please, this job runs even with vars.CI_RELEASE_WINDOWS == 'false', and uploads HookshotBuilder.exe / hookshot-setup.txt to the draft, which defeats the by-hand opt-out (the publish job's WINDOWS_BY_HAND then keeps the draft but the files are already attached). Suggest scoping the nightly clause with env.RELEASE_TAG == '' (or inputs.tag == ''), as the HOOKSHOT_UPDATE_RELEASE expression in linux-appimage.yml already does. The Linux job has no such switch, so it is unaffected.

Verified problem: the new `|| (github.event_name == 'push' && github.ref == 'refs/heads/main')` clause also matches the Release workflow's `workflow_call`, because `github.event_name` and `github.ref` are the caller's (a push to main). So for a release made by release-please, this job runs even with `vars.CI_RELEASE_WINDOWS == 'false'`, and uploads `HookshotBuilder.exe` / `hookshot-setup.txt` to the draft, which defeats the by-hand opt-out (the publish job's `WINDOWS_BY_HAND` then keeps the draft but the files are already attached). Suggest scoping the nightly clause with `env.RELEASE_TAG == ''` (or `inputs.tag == ''`), as the `HOOKSHOT_UPDATE_RELEASE` expression in linux-appimage.yml already does. The Linux job has no such switch, so it is unaffected.

hckrman101 commented

2026-11-10 23:42:29 +00:00

(Migrated from github.com)

Folded into #2018, which carries this nightly prerelease, with the fix from the review (the Windows nightly clause is scoped with !inputs.tag), alongside the release-candidate and Promote flow on top of #1059. Closing in favor of it.

Folded into https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/2018, which carries this nightly prerelease, with the fix from the review (the Windows nightly clause is scoped with `!inputs.tag`), alongside the release-candidate and Promote flow on top of #1059. Closing in favor of it.

Commenting is not possible because the repository is archived.

No reviewers