MagicBeanDev/Hookshot

Archived

Watch

Fork 0

Idea: hardware test runners, so an approved PR test runs on a Steam Deck, AMD, Intel or 8 GB card without waiting for a person #2797

Open

opened 2026-11-12 14:41:10 +00:00 by github-sync · 3 comments

github-sync commented

2026-11-12 14:41:10 +00:00

Owner

@mooki000 opened on GitHub · 2026-11-12 13:45 UTC · original

Status (2026-11-12): research is in this comment; what the Discord discussion added (a security review's five conditions, the label trigger, five further ideas, next steps) is in this one. Nothing is registered anywhere; the first step is a pilot on a fork.

The idea

Suggested by Dr.Crow in Discord this morning, written down here so it is not lost: register testers' machines as self-hosted GitHub Actions runners, each labelled with its hardware (steam-deck, amd-rdna4, intel-arc, nvidia-8gb, linux, ...), and let a workflow send a pull request's test to the hardware it needs. Today that step is a person or their agent picking a card from the test-request queue (#2734, docs/test-requests.md); a runner would do the same steps on a machine whose owner has opted in, whenever that machine is on.

What a hardware job would do is what a tester does now: check out the approved head, build it on that machine with that machine's own dump, run a scripted windowless scene (the control bridge and scripts/run-vk already drive this), and report hardware, driver, the commit, numbers and log lines on the pull request.

Why it would help: testing on other hardware is the slowest step (the reason #2734 exists). The queue made it visible; runners would take the waiting out for the machines that are always on.

What stands in the way, and what would answer it:

Untrusted code on someone's home PC. GitHub's own guidance is that self-hosted runners "should almost never be used for public repositories ... because any user can open pull requests against the repository and compromise the environment" (Secure use reference, Hardening for self-hosted runners). So: never on pull_request; only workflow_dispatch (or a label) by a maintainer for one exact commit, which is the gate #2746 already set for test requests (testing wanted); just-in-time runners that take one job and are removed; the runner in its own OS user or container with no credentials and a read-only token; results posted by a separate job that never runs the pull request's code. The sandboxing hckrman101 mentioned for agents is the same problem.
Game files. A hardware run needs a dump, and "No game in CI" (AGENTS.md) is a rule. The dump would stay on the owner's machine and each runner would build locally (a built game folder is made from the game and cannot travel as an artifact). Logs and artifacts would have to be limited to what CONTRIBUTING.md already allows in a comment: numbers, hookshot_log.txt lines, screenshots. These jobs would be opt-in test jobs, not checks a merge waits for.
Machines that are not always on. A job waits in GitHub's queue until its runner is online, so a card in the request queue would still be the fallback.

A small first step that risks nothing upstream: one workflow file with workflow_dispatch only (inputs: the pull request, the commit, the hardware label, the scene), tried on a fork with one Linux desktop as the runner, to learn what the sandbox needs and how long a job takes. I can do that on my fork with my RTX 4090 / Intel UHD 770 PC and report here before anything is proposed for this repository.

Anything else

The second half of the suggestion, CircleCI (offered as extra capacity, with GPUs, macOS and arm64): its GPU machines are on the paid Scale plan and are data-centre NVIDIA cards (Tesla T4, P4, V100, A10G: CircleCI, Using the GPU execution environment), so no AMD, Intel or Steam Deck, and no dump could live there. It could speed up the game-free checks if GitHub's free minutes or queue times become the limit; I have not seen that they are, and GitHub's hosted runners already cover Windows, Linux, macOS and arm64 for a public repository.
Projects that run tests on their own hardware farms behind a maintainer's approval (Mesa's CI, Dolphin's FifoCI) would be worth reading before designing this; I will add what I find as a comment.
Questions for the maintainers: is an exception to "No game in CI" acceptable for opt-in hardware jobs that publish only numbers, log lines and screenshots? Would you want runners registered on this repository at all, or kept on testers' forks with results posted to the pull request?

> [@mooki000](https://github.com/mooki000) opened on GitHub · 2026-11-12 13:45 UTC · [original](https://github.com/NotRexed/Hookshot/issues/2797) **Status (2026-11-12):** research is in [this comment](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2797); what the Discord discussion added (a security review's five conditions, the label trigger, five further ideas, next steps) is in [this one](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2797). Nothing is registered anywhere; the first step is a pilot on a fork. ### The idea Suggested by Dr.Crow in Discord this morning, written down here so it is not lost: register testers' machines as **self-hosted GitHub Actions runners**, each labelled with its hardware (`steam-deck`, `amd-rdna4`, `intel-arc`, `nvidia-8gb`, `linux`, ...), and let a workflow send a pull request's test to the hardware it needs. Today that step is a person or their agent picking a card from the test-request queue (#2734, `docs/test-requests.md`); a runner would do the same steps on a machine whose owner has opted in, whenever that machine is on. What a hardware job would do is what a tester does now: check out the approved head, build it on that machine with that machine's own dump, run a scripted windowless scene (the control bridge and `scripts/run-vk` already drive this), and report hardware, driver, the commit, numbers and log lines on the pull request. **Why it would help:** testing on other hardware is the slowest step (the reason #2734 exists). The queue made it visible; runners would take the waiting out for the machines that are always on. **What stands in the way, and what would answer it:** 1. **Untrusted code on someone's home PC.** GitHub's own guidance is that self-hosted runners "should almost never be used for public repositories ... because any user can open pull requests against the repository and compromise the environment" ([Secure use reference, Hardening for self-hosted runners](https://docs.github.com/en/actions/reference/security/secure-use#hardening-for-self-hosted-runners)). So: never on `pull_request`; only `workflow_dispatch` (or a label) by a maintainer for one exact commit, which is the gate #2746 already set for test requests (`testing wanted`); just-in-time runners that take one job and are removed; the runner in its own OS user or container with no credentials and a read-only token; results posted by a separate job that never runs the pull request's code. The sandboxing hckrman101 mentioned for agents is the same problem. 2. **Game files.** A hardware run needs a dump, and "No game in CI" (AGENTS.md) is a rule. The dump would stay on the owner's machine and each runner would build locally (a built game folder is made from the game and cannot travel as an artifact). Logs and artifacts would have to be limited to what `CONTRIBUTING.md` already allows in a comment: numbers, `hookshot_log.txt` lines, screenshots. These jobs would be opt-in test jobs, not checks a merge waits for. 3. **Machines that are not always on.** A job waits in GitHub's queue until its runner is online, so a card in the request queue would still be the fallback. **A small first step that risks nothing upstream:** one workflow file with `workflow_dispatch` only (inputs: the pull request, the commit, the hardware label, the scene), tried on a fork with one Linux desktop as the runner, to learn what the sandbox needs and how long a job takes. I can do that on my fork with my RTX 4090 / Intel UHD 770 PC and report here before anything is proposed for this repository. ### Anything else - **The second half of the suggestion, CircleCI** (offered as extra capacity, with GPUs, macOS and arm64): its GPU machines are on the paid Scale plan and are data-centre NVIDIA cards (Tesla T4, P4, V100, A10G: [CircleCI, Using the GPU execution environment](https://circleci.com/docs/guides/execute/using-gpu)), so no AMD, Intel or Steam Deck, and no dump could live there. It could speed up the game-free checks if GitHub's free minutes or queue times become the limit; I have not seen that they are, and GitHub's hosted runners already cover Windows, Linux, macOS and arm64 for a public repository. - Projects that run tests on their own hardware farms behind a maintainer's approval (Mesa's CI, Dolphin's FifoCI) would be worth reading before designing this; I will add what I find as a comment. - Questions for the maintainers: is an exception to "No game in CI" acceptable for opt-in hardware jobs that publish only numbers, log lines and screenshots? Would you want runners registered on this repository at all, or kept on testers' forks with results posted to the pull request?

github-sync commented

2026-11-12 14:48:35 +00:00

Author

Owner

@mooki000 commented on GitHub · 2026-11-12 14:10 UTC · original

Research findings (primary sources read 2026-11-12): this is feasible, but I recommend starting with an owner-controlled local consumer of the existing test-request queue, not registering home PCs upstream yet.

The important corrections to my initial sketch:

Dispatch and labels are not isolation. Another PR-controlled workflow can select the same repository runner; first-time-contributor approval is not a security boundary. GitHub explicitly warns against public-repository self-hosted runners (security guidance, fork-approval rules).
A just-in-time (JIT) runner means at most one job, not a wiped or sandboxed PC. Cleanup/isolation are ours to provide. An offline/missing runner leaves a job queued for up to 24 hours, not indefinitely (runner reference).
#2746 approves a PR, not a particular revision. Automation needs approval of the full head SHA and must refuse a changed head; a retained testing wanted label must not authorize every later push (current client).

The least-exposed design separates the networked controller/publisher from the build and game: fresh restricted account/environment, private home and disposable saves, read-only dump, no network in the PR-code process, bounded resources, and a trusted fixed-format result filter. A separate GitHub-hosted job can comment without ever executing the PR's code. GPU-driver faults/escapes and false results remain possible; read-only access does not prevent reading and leaking a dump.

No raw logs or directory artifacts. Public Actions logs are readable by other GitHub users; upload only small, validated observations. Normal log excerpts and reviewed screenshots are allowed by our rules, but a malicious PR can change what they contain (project rule, log access).

Steam Deck is a separate proof, not a promised supported runner. Windows 11 is on GitHub's supported list; Arch/SteamOS are not. A home-folder installation avoids modifying SteamOS's read-only image, but dependencies/service persistence still need testing across updates. Each tester must build locally from their own dump; the guide's “about 40 minutes” is not a Deck benchmark (supported systems, Valve guidance, build guide).

Useful precedents: Dolphin's current build controller rejects untrusted automatic requests and accepts a trusted developer's rebuild request. Mesa uses Marge/manual farm testing, but its Docker-farm warning says in its own words that this lets "anyone on the internet run code on your device". Neither proves a volunteer's personal files are safe. ANGLE demonstrates permissioned routing by GPU and driver.

Extra hosted capacity is not an observed problem now. In 60 recent runs (13:04–13:42 UTC), 125 executed jobs from completed runs queued a median 3 seconds, maximum 73 seconds. Nine completed PR-head cohorts took 4.47–6.00 minutes wall time; summed job-minutes are a different measure. Example: source checks. This is a short sample, not a monthly usage audit.

GitHub's standard public-repository runners are free and unlimited in minutes. CircleCI offers 400,000 Linux OSS credits/month on Free, but its GPU classes require Scale and are NVIDIA. Neither cloud can hold a dump or supply the missing AMD/Intel/Deck coverage.

Suggested stages: local queue consumer first; then one confined just-in-time experiment on my own fork and PC, with results posted only to a test pull request there; upstream only after that proof and an explicit policy decision. I have a two-job workflow draft and the confinement/publication acceptance criteria to offer. Nothing has been installed, registered or run for this research.

Two decisions for maintainers:

Does “No game in CI” permit this narrow opt-in local-hardware exception, without transferring game files or derived artifacts?
Should execution stay under testers' local/fork control, or become an upstream-managed service—and who may approve each exact commit and own that service?

> [@mooki000](https://github.com/mooki000) commented on GitHub · 2026-11-12 14:10 UTC · [original](https://github.com/NotRexed/Hookshot/issues/2797#issuecomment-6825904272) Research findings (primary sources read 2026-11-12): **this is feasible, but I recommend starting with an owner-controlled local consumer of the existing test-request queue, not registering home PCs upstream yet.** The important corrections to my initial sketch: - **Dispatch and labels are not isolation.** Another PR-controlled workflow can select the same repository runner; first-time-contributor approval is not a security boundary. GitHub explicitly warns against public-repository self-hosted runners ([security guidance](https://docs.github.com/en/actions/reference/security/secure-use), [fork-approval rules](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository)). - **A just-in-time (JIT) runner means at most one job, not a wiped or sandboxed PC.** Cleanup/isolation are ours to provide. An offline/missing runner leaves a job queued for up to **24 hours**, not indefinitely ([runner reference](https://docs.github.com/en/actions/reference/runners/self-hosted-runners)). - **#2746 approves a PR, not a particular revision.** Automation needs approval of the full head SHA and must refuse a changed head; a retained `testing wanted` label must not authorize every later push ([current client](https://git.magicbean.dev/MagicBeanDev/Hookshot/src/commit/51da0464b4cf0c7f56b81ebd6bb309325835e175/scripts/test_requests.py)). The least-exposed design separates the networked controller/publisher from the build and game: fresh restricted account/environment, private home and disposable saves, read-only dump, no network in the PR-code process, bounded resources, and a trusted fixed-format result filter. A separate GitHub-hosted job can comment without ever executing the PR's code. GPU-driver faults/escapes and false results remain possible; read-only access does not prevent reading and leaking a dump. **No raw logs or directory artifacts.** Public Actions logs are readable by other GitHub users; upload only small, validated observations. Normal log excerpts and reviewed screenshots are allowed by our rules, but a malicious PR can change what they contain ([project rule](https://git.magicbean.dev/MagicBeanDev/Hookshot/src/commit/51da0464b4cf0c7f56b81ebd6bb309325835e175/CONTRIBUTING.md#the-rule-with-no-exceptions), [log access](https://docs.github.com/en/actions/how-tos/monitor-workflows/use-workflow-run-logs)). **Steam Deck is a separate proof, not a promised supported runner.** Windows 11 is on GitHub's supported list; Arch/SteamOS are not. A home-folder installation avoids modifying SteamOS's read-only image, but dependencies/service persistence still need testing across updates. Each tester must build locally from their own dump; the guide's “about 40 minutes” is not a Deck benchmark ([supported systems](https://docs.github.com/en/actions/reference/runners/self-hosted-runners), [Valve guidance](https://help.steampowered.com/en/faqs/view/671A-4453-E8D2-323C), [build guide](https://git.magicbean.dev/MagicBeanDev/Hookshot/src/commit/51da0464b4cf0c7f56b81ebd6bb309325835e175/CONTRIBUTING.md#building)). Useful precedents: [Dolphin's current build controller](https://github.com/dolphin-emu/central/blob/master/central/buildbot.py) rejects untrusted automatic requests and accepts a trusted developer's rebuild request. [Mesa](https://docs.mesa3d.org/ci/index.html) uses Marge/manual farm testing, but its [Docker-farm warning](https://docs.mesa3d.org/ci/docker.html) says in its own words that this lets "anyone on the internet run code on your device". Neither proves a volunteer's personal files are safe. [ANGLE](https://github.com/google/angle/blob/main/doc/TestingOnBots.md) demonstrates permissioned routing by GPU **and driver**. **Extra hosted capacity is not an observed problem now.** In 60 recent runs (13:04–13:42 UTC), 125 executed jobs from completed runs queued a median **3 seconds**, maximum **73 seconds**. Nine completed PR-head cohorts took **4.47–6.00 minutes wall time**; summed job-minutes are a different measure. Example: [source checks](https://github.com/NotRexed/Hookshot/actions/runs/43228154071). This is a short sample, not a monthly usage audit. GitHub's [standard public-repository runners are free and unlimited in minutes](https://docs.github.com/en/actions/reference/runners/github-hosted-runners). CircleCI offers [400,000 Linux OSS credits/month on Free](https://circleci.com/docs/guides/plans-pricing/credits/), but its [GPU classes require Scale and are NVIDIA](https://circleci.com/docs/guides/execute/using-gpu/). Neither cloud can hold a dump or supply the missing AMD/Intel/Deck coverage. **Suggested stages:** local queue consumer first; then one confined just-in-time experiment on my own fork and PC, with results posted only to a test pull request there; upstream only after that proof and an explicit policy decision. I have a two-job workflow draft and the confinement/publication acceptance criteria to offer. Nothing has been installed, registered or run for this research. Two decisions for maintainers: 1. Does **“No game in CI”** permit this narrow opt-in local-hardware exception, without transferring game files or derived artifacts? 2. Should execution stay under testers' local/fork control, or become an upstream-managed service—and who may approve each exact commit and own that service?

github-sync commented

2026-11-12 14:48:35 +00:00

Author

Owner

@mooki000 commented on GitHub · 2026-11-12 14:46 UTC · original

What the Discord discussion added (#dev-work and its thread, 2026-11-12), written down here so the issue stays the record. Corrections welcome from anyone I have summarised.

A security review, from _trash.panda ("AppSec take: good idea"), who welcomed a pilot on a fork and offered to review its workflow. Before anything is registered on this repository:

Dispatch only from main with the pull request's commit as input, never on pull_request, so a pull request cannot edit the workflow that runs it.
Approval has to cover the build-time code (nx2/, scripts/, tests/, workflows), since that runs on the tester's machine.
A throwaway VM per job: no access to the owner's files or keys, the dump mounted read-only, no network after checkout, so the dump cannot leave.
No credentials on the runner. Results are posted by a separate GitHub-hosted job, and only numbers, log lines and screenshots go out.
Opt-in only, never a merge check.

From Dr.Crow, who made the original suggestion:

Runners on open-source projects are fine if hardened: never pass secrets to them, treat them as compromised.
Be selective. Do not run hardware tests on every pull request; start them by adding a label such as run-ci-on-hardware, optional and never blocking a merge, "to give folks a better idea of the impact of the PR on performance across the different sets of hardware".
Game files never travel through CI. The workflow checks that the dump is present on the machine (a hash check) and only then runs.
CircleCI as a second platform only if GitHub's minutes or queues become the limit. A plan for an open-source project should be obtainable, but most of its GPUs are not consumer cards. He also offered help with CI speed in general: the faster the checks, the sooner an agent knows whether its change is right.

NitnekLeBorgne: in favour; "the major work will be about security".

Joe Staff asked what would run on a runner, given how long the game takes to build. The same steps a tester does by hand: build the approved commit from that machine's own dump, run a scripted scene without a window for a minute or three, report. A complete source build measured today: 14 min 55 s on an i9-13900K on Linux (#2793's description), 20.5 min on an i7-13700F on Windows (the latest cycle on #2595). So a cold job is roughly 20 to 40 minutes; rebuilding only what a pull request changed is much shorter, and that depends on how much of a previous build a runner may keep (see idea 1).

Ideas added in the thread

Split the build from the run. The build is where most pull-request-controlled code runs (the Python build scripts), and it needs no GPU: it can sit in a throwaway VM or container with the dump read-only and no network (condition 3 in full). Only the built binary needs the real GPU, and that can run in a tight sandbox. A sandbox is weaker than a VM and the pilot will say exactly what it gives; handing a GPU to a VM is not practical on a one-card desktop.
Dedicated devices first. A Steam Deck that holds nothing but the dump is its own throwaway machine. PCs in daily use are better off working through the test-request queue locally under their owner's control.
The label can be made exact: it starts main's workflow file, not the pull request's, with the head commit at label time as the input, and is dropped on the next push. That would give Dr.Crow's trigger together with condition 1; what that event exposes is being checked before I claim it.
Test requests should pin a commit. A card names a pull request today. On #2731 this morning the head moved (a fix for a defect a tester found) while its card was being tested, and a later "pass" was for the earlier commit. I am preparing a pull request for this: the card records the head, shows when it has moved, and a result names the commit tested. It needs no runner and any automation needs it.
Game-free GPU tests need no dump at all. A Vulkan validation job on a software driver in the normal CI costs nothing, and the same tests could later run on real AMD, Intel or Deck runners that hold no game files, which removes the hardest part of the security problem. It is also the only kind of test a cloud GPU could run for this project. Whether the renderer can be driven without the game is being looked at.

Where this stands

Now: the commit-pinning pull request (idea 4).
Next: a pilot on my fork with my own PC, written to the five conditions above, with _trash.panda as reviewer of its workflow. Nothing is registered anywhere yet.
Being scoped: the game-free GPU job (idea 5), and what a CircleCI plan could add for it.
Unchanged: nothing on this repository until the pilot has been shown and the maintainers have answered the two questions in the research comment above ("No game in CI" exception; who runs and approves).

> [@mooki000](https://github.com/mooki000) commented on GitHub · 2026-11-12 14:46 UTC · [original](https://github.com/NotRexed/Hookshot/issues/2797#issuecomment-6826819619) **What the Discord discussion added** (`#dev-work` and its thread, 2026-11-12), written down here so the issue stays the record. Corrections welcome from anyone I have summarised. **A security review, from _trash.panda** ("AppSec take: good idea"), who welcomed a pilot on a fork and offered to review its workflow. Before anything is registered on this repository: 1. Dispatch only from main with the pull request's commit as input, never on `pull_request`, so a pull request cannot edit the workflow that runs it. 2. Approval has to cover the build-time code (`nx2/`, `scripts/`, `tests/`, workflows), since that runs on the tester's machine. 3. A throwaway VM per job: no access to the owner's files or keys, the dump mounted read-only, no network after checkout, so the dump cannot leave. 4. No credentials on the runner. Results are posted by a separate GitHub-hosted job, and only numbers, log lines and screenshots go out. 5. Opt-in only, never a merge check. **From Dr.Crow**, who made the original suggestion: - Runners on open-source projects are fine if hardened: never pass secrets to them, treat them as compromised. - Be selective. Do not run hardware tests on every pull request; start them by adding a label such as `run-ci-on-hardware`, optional and never blocking a merge, "to give folks a better idea of the impact of the PR on performance across the different sets of hardware". - Game files never travel through CI. The workflow checks that the dump is present on the machine (a hash check) and only then runs. - CircleCI as a second platform only if GitHub's minutes or queues become the limit. A plan for an open-source project should be obtainable, but most of its GPUs are not consumer cards. He also offered help with CI speed in general: the faster the checks, the sooner an agent knows whether its change is right. **NitnekLeBorgne:** in favour; "the major work will be about security". **Joe Staff asked what would run on a runner, given how long the game takes to build.** The same steps a tester does by hand: build the approved commit from that machine's own dump, run a scripted scene without a window for a minute or three, report. A complete source build measured today: 14 min 55 s on an i9-13900K on Linux (#2793's description), 20.5 min on an i7-13700F on Windows (the latest cycle on #2595). So a cold job is roughly 20 to 40 minutes; rebuilding only what a pull request changed is much shorter, and that depends on how much of a previous build a runner may keep (see idea 1). **Ideas added in the thread** 1. **Split the build from the run.** The build is where most pull-request-controlled code runs (the Python build scripts), and it needs no GPU: it can sit in a throwaway VM or container with the dump read-only and no network (condition 3 in full). Only the built binary needs the real GPU, and that can run in a tight sandbox. A sandbox is weaker than a VM and the pilot will say exactly what it gives; handing a GPU to a VM is not practical on a one-card desktop. 2. **Dedicated devices first.** A Steam Deck that holds nothing but the dump is its own throwaway machine. PCs in daily use are better off working through the test-request queue locally under their owner's control. 3. **The label can be made exact:** it starts main's workflow file, not the pull request's, with the head commit at label time as the input, and is dropped on the next push. That would give Dr.Crow's trigger together with condition 1; what that event exposes is being checked before I claim it. 4. **Test requests should pin a commit.** A card names a pull request today. On #2731 this morning the head moved (a fix for a defect a tester found) while its card was being tested, and a later "pass" was for the earlier commit. I am preparing a pull request for this: the card records the head, shows when it has moved, and a result names the commit tested. It needs no runner and any automation needs it. 5. **Game-free GPU tests need no dump at all.** A Vulkan validation job on a software driver in the normal CI costs nothing, and the same tests could later run on real AMD, Intel or Deck runners that hold no game files, which removes the hardest part of the security problem. It is also the only kind of test a cloud GPU could run for this project. Whether the renderer can be driven without the game is being looked at. **Where this stands** - Now: the commit-pinning pull request (idea 4). - Next: a pilot on my fork with my own PC, written to the five conditions above, with _trash.panda as reviewer of its workflow. Nothing is registered anywhere yet. - Being scoped: the game-free GPU job (idea 5), and what a CircleCI plan could add for it. - Unchanged: nothing on this repository until the pilot has been shown and the maintainers have answered the two questions in the research comment above ("No game in CI" exception; who runs and approves).

github-sync referenced this issue

2026-11-12 15:48:58 +00:00

ci: label-gated hardware tests on testers' self-hosted runners, through the owner's sandbox #2807

github-sync referenced this issue from a commit

2026-11-12 15:49:00 +00:00

ci: label-gated hardware tests on testers' self-hosted runners, through the owner's sandbox (#2797)

github-sync referenced this issue

2026-11-12 15:49:14 +00:00

feat(agents): pin test requests and results to commits #2830

github-sync commented

2026-11-12 15:53:36 +00:00

Author

Owner

@Joey-Magnus commented on GitHub · 2026-11-12 14:51 UTC · original

Thanks for writing the Discord discussion up, mooki. That covers the AppSec side. Two additions so the issue stays the full record:

Your idea 3, the exact label, is now a PR: #2807.

A hardware: <name> label starts main's copy of the workflow for the head commit at label time.
Any new push removes the labels.
The job on the tester's machine gets no token and no checkout, and only calls the owner's sandbox program.
A separate job on GitHub's own runners posts the results as quoted text.

It's inert until someone registers a runner, and a game-free test keeps those rules in place.

One correction to my own conditions, prompted by your research comment: a runner registered on this repository isn't protected by dispatch, labels or approvals. A PR can add its own pull_request workflow that asks for the runner's labels and runs straight on it, past the gate and the sandbox. So:

Runners go on testers' own forks only, where only the owner pushes workflows. #2807's doc now says so.
Upstream runners would need runner groups limited to one workflow. That needs an organization, and it's part of your second question to the maintainers.

Your thread ideas, from the AppSec side:

Split the build from the run: agreed. It's the best way to avoid needing GPU passthrough, and #2807's sandbox requirements now include it.
Dedicated devices first: agreed. A Deck holding only the dump is the easiest machine to make safe.
Pin test requests to a commit: agreed, and any automation needs the same, which #2807 enforces.
Game-free GPU tests: the best risk reduction of all, because a runner with no dump has nothing to leak.

Still happy to review the fork pilot's workflow. Tag me when it's up.

> [@Joey-Magnus](https://github.com/Joey-Magnus) commented on GitHub · 2026-11-12 14:51 UTC · [original](https://github.com/NotRexed/Hookshot/issues/2797#issuecomment-6826930483) Thanks for writing the Discord discussion up, mooki. That covers the AppSec side. Two additions so the issue stays the full record: **Your idea 3, the exact label, is now a PR: [#2807](https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/2807).** - A `hardware: <name>` label starts `main`'s copy of the workflow for the head commit at label time. - Any new push removes the labels. - The job on the tester's machine gets no token and no checkout, and only calls the owner's sandbox program. - A separate job on GitHub's own runners posts the results as quoted text. It's inert until someone registers a runner, and a game-free test keeps those rules in place. **One correction to my own conditions, prompted by your research comment:** a runner registered on *this* repository isn't protected by dispatch, labels or approvals. A PR can add its own `pull_request` workflow that asks for the runner's labels and runs straight on it, past the gate and the sandbox. So: - Runners go on testers' own forks only, where only the owner pushes workflows. #2807's doc now says so. - Upstream runners would need runner groups limited to one workflow. That needs an organization, and it's part of your second question to the maintainers. **Your thread ideas, from the AppSec side:** - **Split the build from the run:** agreed. It's the best way to avoid needing GPU passthrough, and #2807's sandbox requirements now include it. - **Dedicated devices first:** agreed. A Deck holding only the dump is the easiest machine to make safe. - **Pin test requests to a commit:** agreed, and any automation needs the same, which #2807 enforces. - **Game-free GPU tests:** the best risk reduction of all, because a runner with no dump has nothing to leak. Still happy to review the fork pilot's workflow. Tag me when it's up.

github-sync referenced this issue

2026-11-12 17:57:29 +00:00

ci: execute authored Vulkan fixtures on lavapipe #2866

github-sync referenced this issue from a commit

2026-11-12 17:57:55 +00:00

ci: execute authored Vulkan fixtures on lavapipe

github-sync referenced this issue

2026-11-12 17:59:31 +00:00

ci: label-gated hardware tests on testers' self-hosted runners, through the owner's sandbox #2807

github-sync referenced this issue from a commit

2026-11-12 19:01:05 +00:00

ci: execute authored Vulkan fixtures on lavapipe