Proposal: use the release structure we already have (nightly for testing, rc that means frozen) #394

New issue

Open

opened 2026-11-14 08:19:18 +00:00 by madebyjordan · 0 comments

madebyjordan commented

2026-11-14 08:19:18 +00:00

Contributor

This doesn't argue against pull request builds. It argues that they shouldn't be our main testing path, and that most of the need they answer goes away once the structure we have works.

Where we are

Testing before merge mostly doesn't happen, and nothing automated stops a broken merge. Figures are from Longshot's Forgejo, the release list and the Matrix rooms, as of 14 Nov. The chat counts are parsed from messages, so treat them as approximate.

What	The number
Tests that run on a PR or on main	None automatically. CI checks the title, the description, workflow lint and third-party notes; a maintainer starts the full suite by hand (CONTRIBUTING.md)
main after the 14 Nov merge wave	Red: "main has two failing tests since #252" (03:32)
Test requests since the fork	141, on 95 PRs; 15 got a result, from about 12 testers
Merged PRs that had a test request	43, of which 36 merged with no result posted
Who merges	One account, HumanBean (formerly hckrman101): all 130 merged Longshot PRs and all 47 launcher PRs. Nothing merged for 5.6 hours (13 Nov 18:58 to 14 Nov 00:33 UTC) while it was idle
Merges by time of day	61 of 130 landed between 00:00 and 06:00 UTC (American evening). Most UK-daytime merges came on 13 Nov, when the maintainer was up around the clock. At 08:25 UTC on 14 Nov the last merge was at 05:42, with 66 PRs out of WIP waiting
Branch protection on main	Requires 1 approval and no CI checks, in both repositories. 111 of 130 merged Longshot PRs had no approving review (launcher: 46 of 47), so the approval rule is bypassed on most merges
How reviews and merges are done	Very likely by that maintainer's AI agents, from his account, in sessions he runs: 9 reviews posted within 2 minutes (13 Nov 16:17), 12 merges under 30 seconds apart, and his chat updates saying "Merged on Longshot" are agent posts. Inferred from timing and wording; the forge doesn't record it
Time from opening to merge (median)	7 minutes for maintainers' own PRs; 6.5 hours for everyone else's
Open PRs	134
Release candidates	v3.0.0-rc.1, rc.2, rc.3 within about 7 hours on 13 Nov; none tested as an rc; none promoted
Stable releases	None. main is 62 commits past rc.3; the launcher's default channel is stable, so a first start finds nothing (launcher PR 10)
Nightly reach	nightly-80: about 6,700 downloads. rc.3: about 12,000
Nightly reliability	Held for 9 hours on 14 Nov by one failed release run; the night's merges then shipped as one 47-commit jump

This is a habit carried over from Hookshot, not something new. Hookshot tagged v2.0.8-rc.1 to rc.5 in six hours, and v2.1.0 stable 37 minutes after its rc.1.

The channels: designed right, run wrong

Our own docs already describe the standard model, the one Firefox, Chrome and Dolphin use. Everything lands on the main branch, every merge is an installable build, and volunteers run it. Release candidates are cut deliberately, held while they're tested, then promoted unchanged.

Channel	What docs/releases.md and RELEASING.md say	What happens now
nightly	"every change, untested"; every merge builds one	Works, with a big audience. Testers aren't pointed at it: test requests ask people to build a PR from source instead
rc	Tag a commit, testers on the rc channel try it, a Windows tester runs the checklist; fixes make rc.2; then tag the same commit stable	Tagged hours apart on a moving main, with features landing in between. No test round, no checklist run, no promotion. Used mostly as a fixed point to compare against
stable	The tested rc's commit, re-tagged	Empty, so rc acts as stable

None of this needs new ideas. It needs us to run what we wrote.

Three asks

Each ask depends on the one before it, so the order matters.

Make CI gate merges. Run the full test suite automatically on every pull request and on main: both builders' compiles and the game-free checks (tests/run_all.py).
- Without it, "green" means "the title parses", and any faster merging just breaks main faster.
- If the runners can't take the full suite on every push, run it when a PR leaves WIP and again before merge.
Merging stops waiting on one person. Automate the merge, keep one human approval.
- The rule already exists but isn't enforced: main requires one approval, yet most merges have none, and no CI check is required. Make it real: require the CI checks, and an approval from someone other than the author, maintainers' own PRs included.
- A bot merges a PR as soon as it has one approval, CI is green and nothing requests changes. Merges go one at a time, re-run on current main, so a merge wave can't turn main red.
- More people can approve: an owner or two per area (renderer, launcher, input and menu, docs and scripts). Release and trust files (release.yml, signing, release_keys.json, tools.lock, sign-in scripts) still need a maintainer.
- Approvers spread across timezones keep nightly moving around the clock, without anyone losing sleep. Today, when the one merger is offline, nightly stops: contributors in Europe and Asia wait 5–8 hours or more for a merge, and players testing in those hours only get what was merged before, even when fixes for what they're seeing are ready.
- Later, if wanted: contributors with a record of clean merges skip the approval.
- An agent can do the reading, as it seems to now. We keep an approval from an accountable person because whatever lands is signed with the release key and installed on thousands of PCs within the hour. Reverting a bad merge doesn't undo it having run there. Rust, Kubernetes and Chromium automate the merge, not the approval.
Nightly is where testing happens; rc means frozen.
- A testing wanted request stays open after merge and names the first nightly that contains the change. Testers switch the launcher to nightly; nobody builds from source.
- Each nightly is posted to #test-requests with the PRs it contains.
- Revert first: a regression found on nightly gets its PR reverted within hours, then re-landed with a fix.
- release.yml gets a timeout or cancel guard, so one failed run can't hold nightlies for hours again.
- Keep nightlies for a few weeks, not the newest 5, so a tester can find which nightly broke something. They're 42 MB (Windows) and 99 MB (Linux) each, and the tools are shared.
- An rc is a nightly that has run clean for a couple of days, tagged on purpose. While it's tested, only fixes go to main for 48 hours; the Windows checklist runs once; then the same commit is tagged stable. One rc per release, a second only if something blocks it.
- First step: do this now for v3.0.0, so the stable channel stops being empty.

Where pull request builds fit

Issue 337 and its PRs (Longshot 353, LongshotLauncher 59) start from a real problem: testing a PR means building it from source. Once merges reach nightly within hours of approval, most of that problem is gone. The rest is better served by building on nightly.

Testers' attention is the scarce thing, not builds. About a dozen people have ever posted a result. One nightly puts them all on the same build. Dozens of PR builds would likely spread them thinner, though the data can't prove that.
The cost is real. A second signing key; signed builds of code nobody has reviewed (PR 353's own review ranked that its top risk); both builders built on every push of about 60 PRs, on runners that already stalled once.
A better home for the launcher work in PR 59: a picker for older nightlies, installed side by side, plus the "how did it go?" report from issue 337. Then a tester can say "broke in nightly-140, worked in nightly-136", which narrows it to a handful of PRs. Expect a few game builds of about 15–25 minutes each to do that.
Keep PR builds as a narrow exception: a maintainer labels a PR that's too risky to land first (a default-on renderer change, a fix for hardware the author doesn't have), and only those get a signed build. Most such changes can land on nightly behind an opt-in flag instead.

Open questions

Can our runners take the full suite on every PR push, or only on leaving WIP and before merge?
Can a scheduled workflow's own token merge PRs on our Forgejo, or does the merge bot need an account?
Who are the area owners for approvals?
Do unclaimed test requests go unclaimed because people can't build from source, or because they don't want to test? A week of nightly-first requests would tell us.

Sources

CONTRIBUTING.md: what CI runs on PRs and main
RELEASING.md and docs/releases.md: the channels as designed
Longshot releases: rc dates, download counts, file sizes
Issue 337, PR 353, LongshotLauncher PR 59
Longshot's PR list and main's history (merges, merge times)
Matrix #test-requests and #agent-updates, 12–14 Nov (test requests and results, the red main, the nightly stall)

We already built the right release structure: a signed nightly for every merge, installable from the launcher, plus rc and stable channels. We are not running it as designed. Our tests don't gate merges, every merge waits on one maintainer, and our release candidates are snapshots rather than frozen builds. Fix those three things and nightly becomes the testing channel we keep trying to invent. The urge to ship fast is healthy; nightly is where it belongs. This doesn't argue against pull request builds. It argues that they shouldn't be our main testing path, and that most of the need they answer goes away once the structure we have works. ## Where we are Testing before merge mostly doesn't happen, and nothing automated stops a broken merge. Figures are from Longshot's Forgejo, the release list and the Matrix rooms, as of 14 Nov. The chat counts are parsed from messages, so treat them as approximate. | What | The number | | --- | --- | | Tests that run on a PR or on main | None automatically. CI checks the title, the description, workflow lint and third-party notes; a maintainer starts the full suite by hand (CONTRIBUTING.md) | | main after the 14 Nov merge wave | Red: "main has two failing tests since #252" (03:32) | | Test requests since the fork | 141, on 95 PRs; 15 got a result, from about 12 testers | | Merged PRs that had a test request | 43, of which 36 merged with no result posted | | Who merges | One account, HumanBean (formerly hckrman101): all 130 merged Longshot PRs and all 47 launcher PRs. Nothing merged for 5.6 hours (13 Nov 18:58 to 14 Nov 00:33 UTC) while it was idle | | Merges by time of day | 61 of 130 landed between 00:00 and 06:00 UTC (American evening). Most UK-daytime merges came on 13 Nov, when the maintainer was up around the clock. At 08:25 UTC on 14 Nov the last merge was at 05:42, with 66 PRs out of WIP waiting | | Branch protection on main | Requires 1 approval and no CI checks, in both repositories. 111 of 130 merged Longshot PRs had no approving review (launcher: 46 of 47), so the approval rule is bypassed on most merges | | How reviews and merges are done | Very likely by that maintainer's AI agents, from his account, in sessions he runs: 9 reviews posted within 2 minutes (13 Nov 16:17), 12 merges under 30 seconds apart, and his chat updates saying "Merged on Longshot" are agent posts. Inferred from timing and wording; the forge doesn't record it | | Time from opening to merge (median) | 7 minutes for maintainers' own PRs; 6.5 hours for everyone else's | | Open PRs | 134 | | Release candidates | v3.0.0-rc.1, rc.2, rc.3 within about 7 hours on 13 Nov; none tested as an rc; none promoted | | Stable releases | None. main is 62 commits past rc.3; the launcher's default channel is stable, so a first start finds nothing (launcher PR 10) | | Nightly reach | nightly-80: about 6,700 downloads. rc.3: about 12,000 | | Nightly reliability | Held for 9 hours on 14 Nov by one failed release run; the night's merges then shipped as one 47-commit jump | This is a habit carried over from Hookshot, not something new. Hookshot tagged v2.0.8-rc.1 to rc.5 in six hours, and v2.1.0 stable 37 minutes after its rc.1. ## The channels: designed right, run wrong Our own docs already describe the standard model, the one Firefox, Chrome and Dolphin use. Everything lands on the main branch, every merge is an installable build, and volunteers run it. Release candidates are cut deliberately, held while they're tested, then promoted unchanged. | Channel | What docs/releases.md and RELEASING.md say | What happens now | | --- | --- | --- | | nightly | "every change, untested"; every merge builds one | Works, with a big audience. Testers aren't pointed at it: test requests ask people to build a PR from source instead | | rc | Tag a commit, testers on the rc channel try it, a Windows tester runs the checklist; fixes make rc.2; then tag the same commit stable | Tagged hours apart on a moving main, with features landing in between. No test round, no checklist run, no promotion. Used mostly as a fixed point to compare against | | stable | The tested rc's commit, re-tagged | Empty, so rc acts as stable | None of this needs new ideas. It needs us to run what we wrote. ## Three asks Each ask depends on the one before it, so the order matters. 1. **Make CI gate merges.** Run the full test suite automatically on every pull request and on main: both builders' compiles and the game-free checks (`tests/run_all.py`). - Without it, "green" means "the title parses", and any faster merging just breaks main faster. - If the runners can't take the full suite on every push, run it when a PR leaves WIP and again before merge. 2. **Merging stops waiting on one person.** Automate the merge, keep one human approval. - The rule already exists but isn't enforced: main requires one approval, yet most merges have none, and no CI check is required. Make it real: require the CI checks, and an approval from someone other than the author, maintainers' own PRs included. - A bot merges a PR as soon as it has one approval, CI is green and nothing requests changes. Merges go one at a time, re-run on current main, so a merge wave can't turn main red. - More people can approve: an owner or two per area (renderer, launcher, input and menu, docs and scripts). Release and trust files (`release.yml`, signing, `release_keys.json`, `tools.lock`, sign-in scripts) still need a maintainer. - Approvers spread across timezones keep nightly moving around the clock, without anyone losing sleep. Today, when the one merger is offline, nightly stops: contributors in Europe and Asia wait 5–8 hours or more for a merge, and players testing in those hours only get what was merged before, even when fixes for what they're seeing are ready. - Later, if wanted: contributors with a record of clean merges skip the approval. - An agent can do the reading, as it seems to now. We keep an approval from an accountable person because whatever lands is signed with the release key and installed on thousands of PCs within the hour. Reverting a bad merge doesn't undo it having run there. Rust, Kubernetes and Chromium automate the merge, not the approval. 3. **Nightly is where testing happens; rc means frozen.** - A `testing wanted` request stays open after merge and names the first nightly that contains the change. Testers switch the launcher to nightly; nobody builds from source. - Each nightly is posted to #test-requests with the PRs it contains. - Revert first: a regression found on nightly gets its PR reverted within hours, then re-landed with a fix. - `release.yml` gets a timeout or cancel guard, so one failed run can't hold nightlies for hours again. - Keep nightlies for a few weeks, not the newest 5, so a tester can find which nightly broke something. They're 42 MB (Windows) and 99 MB (Linux) each, and the tools are shared. - An rc is a nightly that has run clean for a couple of days, tagged on purpose. While it's tested, only fixes go to main for 48 hours; the Windows checklist runs once; then the same commit is tagged stable. One rc per release, a second only if something blocks it. - First step: do this now for v3.0.0, so the stable channel stops being empty. ## Where pull request builds fit Issue 337 and its PRs (Longshot 353, LongshotLauncher 59) start from a real problem: testing a PR means building it from source. Once merges reach nightly within hours of approval, most of that problem is gone. The rest is better served by building on nightly. - **Testers' attention is the scarce thing, not builds.** About a dozen people have ever posted a result. One nightly puts them all on the same build. Dozens of PR builds would likely spread them thinner, though the data can't prove that. - **The cost is real.** A second signing key; signed builds of code nobody has reviewed (PR 353's own review ranked that its top risk); both builders built on every push of about 60 PRs, on runners that already stalled once. - **A better home for the launcher work in PR 59:** a picker for older nightlies, installed side by side, plus the "how did it go?" report from issue 337. Then a tester can say "broke in nightly-140, worked in nightly-136", which narrows it to a handful of PRs. Expect a few game builds of about 15–25 minutes each to do that. - **Keep PR builds as a narrow exception:** a maintainer labels a PR that's too risky to land first (a default-on renderer change, a fix for hardware the author doesn't have), and only those get a signed build. Most such changes can land on nightly behind an opt-in flag instead. ## Open questions - Can our runners take the full suite on every PR push, or only on leaving WIP and before merge? - Can a scheduled workflow's own token merge PRs on our Forgejo, or does the merge bot need an account? - Who are the area owners for approvals? - Do unclaimed test requests go unclaimed because people can't build from source, or because they don't want to test? A week of nightly-first requests would tell us. ## Sources - [CONTRIBUTING.md](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/CONTRIBUTING.md): what CI runs on PRs and main - [RELEASING.md](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/RELEASING.md) and [docs/releases.md](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/docs/releases.md): the channels as designed - [Longshot releases](https://git.magicbean.dev/MagicBeanDev/Longshot/releases): rc dates, download counts, file sizes - [Issue 337](https://git.magicbean.dev/MagicBeanDev/Longshot/issues/337), [PR 353](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/353), [LongshotLauncher PR 59](https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/59) - Longshot's PR list and main's history (merges, merge times) - Matrix #test-requests and #agent-updates, 12–14 Nov (test requests and results, the red main, the nightly stall)