MagicBeanDev/Longshot

Watch

Fork 98

WIP: ci: a pull request's test on a tester's own hardware, started by a hardware label (fork-local pilot) #213

Draft

TrashPanda wants to merge 1 commit from TrashPanda/Longshot:ci/hardware-test-label into main

TrashPanda commented

2026-11-13 14:28:09 +00:00

Contributor

Why

Testers with a Steam Deck, or an AMD, Intel or 8 GB card, only find out about a PR when someone picks it from the test-request queue by hand. A tester's always-on PC could instead run a PR's test by itself once a maintainer labels it hardware: <name>, without the PR's code ever running outside that tester's sandbox. This was Hookshot PR #2807 (idea in Hookshot issue 2797), rewritten here for Forgejo Actions.

What changed

ci: .forgejo/workflows/hardware-test-label.yml (new) runs on pull_request_target (labeled, synchronize):
- It dispatches hardware-test.yml from the default branch for the labelled head, through POST /actions/workflows/hardware-test.yml/dispatches.
- New commits take the hardware: labels off, reading when each was put on from the issue timeline. It leaves alone a label put on after the push, and does nothing for an older push.
- It never checks out the PR.
ci: .forgejo/workflows/hardware-test.yml (new) has three jobs:
- check: started only by a dispatch from the default branch. The commit must be the PR's head and still carry the label.
- run: on [self-hosted, linux, longshot-hardware, <name>] with permissions: {}, no checkout and no token. It checks the PR again after waiting in the queue, then calls the owner's LONGSHOT_HW_SANDBOX.
- report: posts results.txt as quoted text, escaped so it can't close the code fence and @mentions don't ping anyone, and capped after escaping.
What changed from the GitHub version: gh became curl against Forgejo's API, with LONGSHOT_HW_TOKEN if set and the job's token otherwise (as website.yml does). JSON is read with python3, since the runners have no jq. Labels come off by ID, and the artifact steps use Forgejo's upload-artifact/download-artifact v4. The jobs that don't use hardware run on quick.
.forgejo/actionlint.yaml: the longshot-hardware runner label.
docs: docs/hardware-runners.md (new) covers what a runner owner must set up and why runners go on a tester's fork, not on this repository. docs/test-requests.md links it.
tests: tests/test_hardware_test_workflow.py (new) has 32 checks. They cover the safety rules (triggers, no self-hosted runner elsewhere, no token or checkout on the runner, pinned actions, no ${{ }} inside scripts) and run each shell step against a stand-in curl that answers as Forgejo's API.

Flags and settings

Flag	What it does	Default now	Recommendation	Untested on
`LONGSHOT_HW_RUNNERS` (repository variable)	The hardware names a repository dispatches for	Unset: a label only leaves a notice	Set only on a tester's fork	Forgejo
`LONGSHOT_HW_TOKEN` (repository secret)	Token for the dispatch, the label removal and the comment	Unset: the job's own token	Set it if the job's token can't dispatch	Forgejo
`LONGSHOT_HW_SANDBOX` (runner `.env`)	The owner's sandbox program; without it the job fails before any PR code runs	Unset	Required on a hardware runner

Measurements

CI only, so there's no game measurement.

main: 0c74253d (2026-11-13)
this PR: ci/hardware-test-label @ ed8d1409
Hardware: Linux (CachyOS), no runner
Scene: none
Method: python -I -B tests/test_hardware_test_workflow.py; actionlint 1.7.12 with .forgejo/actionlint.yaml
How to reproduce: the same two commands

	main @ `0c74253d`	this PR @ ed8d1409
`test_hardware_test_workflow.py`	not there	32 passed
actionlint on the two workflows	not there	clean

Testing

Tested: the 32 checks above and actionlint. On Hookshot, the GitHub version ran end to end on a fork with a test PR.

Not tested: any of it on Forgejo Actions:

whether the job's own token may call the workflow dispatch API (otherwise LONGSHOT_HW_TOKEN is needed);
whether a dispatch made with that token starts the workflow;
the timeline's label entries (type: label, body: "1" when a label is put on), which I took from Forgejo's API and haven't seen on a live run;
a run on a real runner with a sandbox.

Needs other testers:

A tester's fork with a registered longshot-hardware runner: label a test PR hardware: <name>, expecting the dispatch, the run in the sandbox and the comment (request to open)

Screenshots

Nothing on screen

Risk and rollback

Nothing runs until a repository sets LONGSHOT_HW_RUNNERS and has a runner with those labels. Without that, a label only leaves a notice. Deleting the two workflow files turns it all off.

Reviewer notes

Read docs/hardware-runners.md first, especially "Register runners on your own fork, not on this repository". It explains why this is a fork-only pilot.

Issues

See #

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). (Test request to open, after a fork run.)
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (CI only: the checks above.)
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Docs updated.

## Why Testers with a Steam Deck, or an AMD, Intel or 8 GB card, only find out about a PR when someone picks it from the test-request queue by hand. A tester's always-on PC could instead run a PR's test by itself once a maintainer labels it `hardware: <name>`, without the PR's code ever running outside that tester's sandbox. This was Hookshot PR #2807 (idea in Hookshot issue 2797), rewritten here for Forgejo Actions. ## What changed - ci: `.forgejo/workflows/hardware-test-label.yml` (new) runs on `pull_request_target` (`labeled`, `synchronize`): - It dispatches `hardware-test.yml` from the default branch for the labelled head, through `POST /actions/workflows/hardware-test.yml/dispatches`. - New commits take the `hardware:` labels off, reading when each was put on from the issue timeline. It leaves alone a label put on after the push, and does nothing for an older push. - It never checks out the PR. - ci: `.forgejo/workflows/hardware-test.yml` (new) has three jobs: - `check`: started only by a dispatch from the default branch. The commit must be the PR's head and still carry the label. - `run`: on `[self-hosted, linux, longshot-hardware, <name>]` with `permissions: {}`, no checkout and no token. It checks the PR again after waiting in the queue, then calls the owner's `LONGSHOT_HW_SANDBOX`. - `report`: posts `results.txt` as quoted text, escaped so it can't close the code fence and @mentions don't ping anyone, and capped after escaping. - What changed from the GitHub version: `gh` became `curl` against Forgejo's API, with `LONGSHOT_HW_TOKEN` if set and the job's token otherwise (as `website.yml` does). JSON is read with `python3`, since the runners have no `jq`. Labels come off by ID, and the artifact steps use Forgejo's `upload-artifact`/`download-artifact` v4. The jobs that don't use hardware run on `quick`. - `.forgejo/actionlint.yaml`: the `longshot-hardware` runner label. - docs: `docs/hardware-runners.md` (new) covers what a runner owner must set up and why runners go on a tester's fork, not on this repository. `docs/test-requests.md` links it. - tests: `tests/test_hardware_test_workflow.py` (new) has 32 checks. They cover the safety rules (triggers, no self-hosted runner elsewhere, no token or checkout on the runner, pinned actions, no `${{ }}` inside scripts) and run each shell step against a stand-in `curl` that answers as Forgejo's API. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `LONGSHOT_HW_RUNNERS` (repository variable) | The hardware names a repository dispatches for | Unset: a label only leaves a notice | Set only on a tester's fork | | | Forgejo | | `LONGSHOT_HW_TOKEN` (repository secret) | Token for the dispatch, the label removal and the comment | Unset: the job's own token | Set it if the job's token can't dispatch | | | Forgejo | | `LONGSHOT_HW_SANDBOX` (runner `.env`) | The owner's sandbox program; without it the job fails before any PR code runs | Unset | Required on a hardware runner | | | | ## Measurements CI only, so there's no game measurement. main: 0c74253d (2026-11-13) this PR: ci/hardware-test-label @ ed8d1409 Hardware: Linux (CachyOS), no runner Scene: none Method: python -I -B tests/test_hardware_test_workflow.py; actionlint 1.7.12 with .forgejo/actionlint.yaml How to reproduce: the same two commands | | main @ 0c74253d | this PR @ ed8d1409 | |---|---|---| | `test_hardware_test_workflow.py` | not there | 32 passed | | actionlint on the two workflows | not there | clean | ## Testing **Tested:** the 32 checks above and actionlint. On Hookshot, the GitHub version ran end to end on a fork with a test PR. **Not tested:** any of it on Forgejo Actions: - whether the job's own token may call the workflow dispatch API (otherwise `LONGSHOT_HW_TOKEN` is needed); - whether a dispatch made with that token starts the workflow; - the timeline's label entries (`type: label`, `body: "1"` when a label is put on), which I took from Forgejo's API and haven't seen on a live run; - a run on a real runner with a sandbox. **Needs other testers:** - A tester's fork with a registered `longshot-hardware` runner: label a test PR `hardware: <name>`, expecting the dispatch, the run in the sandbox and the comment (request to open) ## Screenshots Nothing on screen ## Risk and rollback Nothing runs until a repository sets `LONGSHOT_HW_RUNNERS` and has a runner with those labels. Without that, a label only leaves a notice. Deleting the two workflow files turns it all off. ## Reviewer notes Read `docs/hardware-runners.md` first, especially "Register runners on your own fork, not on this repository". It explains why this is a fork-only pilot. ## Issues See # ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [ ] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). (Test request to open, after a fork run.) - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (CI only: the checks above.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Docs updated.

TrashPanda force-pushed ci/hardware-test-label from ed8d140956

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

CI / Conventional Commit title (pull_request) Has been cancelled

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled

Details

CI / Workflow lint (actionlint) (pull_request) Has been cancelled

Details

Third-party parts / As their notes say (pull_request) Has been cancelled

Details

Third-party parts / The same as upstream (pull_request) Has been cancelled

Details

to 9c1c6e31b4

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 5s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

2026-11-13 14:45:22 +00:00

Compare

mooki added the

kind

chore

platform/linux

labels

2026-11-14 01:45:09 +00:00

mooki added the

priority

low

label

2026-11-14 04:18:11 +00:00

gab9281 referenced this pull request from MagicBeanDev/LongshotLauncher

2026-11-14 18:50:52 +00:00

Launcher redesign: who's playing, a home for each kind of player, and a splash for RecompLauncher and the Deck #68

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 5s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

This pull request has changes conflicting with the target branch.

.forgejo/actionlint.yaml

Some workflows are waiting to be reviewed.

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

WIP: ci: a pull request's test on a tester's own hardware, started by a hardware label (fork-local pilot) (#213)

Merge commit body

Reviewed-on: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/213

WIP: ci: a pull request's test on a tester's own hardware, started by a hardware label (fork-local pilot) (#213)

Reviewed-on: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/213