MagicBeanDev/Longshot

Watch

Fork 10

Idea: opt-in hardware tests on self-hosted Forgejo runners #43

New issue

Open

opened 2026-11-12 23:57:12 +00:00 by mooki · 1 comment

mooki commented

2026-11-12 23:57:12 +00:00

The idea

Let an approved test run on a volunteer's Steam Deck, AMD or Intel GPU, or smaller NVIDIA card without waiting for someone to perform every step by hand. Dr.Crow suggested the hardware queue; the proposal is now for self-hosted Forgejo runners, not GitHub Actions. Route by GPU, driver, OS and memory, and keep the existing test-request queue as the human-controlled fallback.

A job would build one approved commit using that machine's own dump, run a bounded scripted scene without a window, and return the exact commit, hardware, driver and reviewed observations. Nothing made from the game would be transferred between machines. A game-free GPU fixture would be a safer first workload: no dump, save or game build to protect.

What already exists, and what does not follow from it: the open release migration moves ordinary checks and releases to Forgejo Actions. Its runner guide asks for ubuntu-latest, ubuntu-24.04 and windows-2022; these are workflow requirements, not a hardware inventory. The launcher has Actions enabled and a checks job waiting for ubuntu-latest. Neither establishes a working AMD, Intel or Deck lane. The runner inventory is not public, and Longshot's current contribution guide still says to run the checks locally. This proposal is an additional opt-in hardware service, not a claim that the release runners already provide it.

The security conditions from the earlier discussion still apply:

An owner-controlled workflow, one exact revision. Dispatch the trusted workflow with the full PR head SHA, never run a PR-controlled workflow on the hardware machine. A label may request a run, but approval must bind that SHA and become invalid on the next push. Recheck it before execution and name the tested SHA in the result. Forgejo's author/workflow approval is not this per-commit test approval.
Review everything that executes. That includes nx2/, scripts/, tests/, workflows and dependencies, not only the proposed game fix. The build is untrusted code too.
A disposable, confined environment. No access to the owner's files or keys; private home and disposable saves; the dump read-only; no network for PR-controlled build or game code after the trusted fetch; bounded time, memory and disk. Splitting the no-GPU build VM from the GPU run may be practical, but a GPU sandbox is not equivalent to a throwaway VM. The pilot must show the actual isolation, cleanup and remaining GPU-driver risk, not quietly weaken this condition.
Keep credentials and publication outside the payload. No forge token, signing key or publishing credential in the build/game environment. Isolate the runner's own registration credential in its controller. A separate trusted publisher must never execute PR code and must accept only a tightly bounded result format. No raw job output or directory uploads: even logs and screenshots can be made to contain private data by malicious code, so review them before publication. A read-only dump is still readable and can still leak through a result.
Opt-in, never a required merge check. The owner chooses when the machine is available. Offline, refused or timed-out work falls back to the test request; it is not a pass.

Runner placement is a separate boundary. Forgejo's security guide says runners can be selected from the instance, organization or repository, and host-mode runners have no real isolation. Trusted PR authors can supply workflow changes. A hardware label or a dispatch-only file therefore does not stop another eligible workflow selecting that runner. Start with an owner's fork or dedicated private-control repository, used only for trusted workflows; do not register a personal PC at the shared organization or upstream repository merely because the intended workflow is gated. Any later upstream service needs a demonstrated allocation boundary. Release/signing jobs must not share a potentially compromised PR-test environment.

Anything else

A small first step: an owner-controlled consumer of the existing test queue, then one game-free Forgejo pilot on my fork and PC. I can offer Linux on an RTX 4090 and Intel UHD 770. Only after its confinement and publication path have been reviewed would I propose a dump-using pilot. Dedicated devices come before daily-use PCs; SteamOS, Windows, GPU access and persistence through updates need their own proof. There is no Deck build-time promise here.

Reuse other work rather than add parallel infrastructure: the sanitizer proposal is game-free, while the live/restart A/B proposal could supply a reviewed workload if it lands. Neither is a hardware-runner security gate. Extra cloud CI capacity is a different question; the old GitHub minutes and queue measurements do not establish a Forgejo bottleneck.

Questions for the maintainers:

Does the rule that game runs stay local permit this narrowly confined, owner-approved hardware automation? Until that is decided, keep the pilot game-free and all game files local.
Should execution remain under each tester's local/fork control, or become an upstream-managed service? Who approves each exact commit, owns the machines and reviews what leaves them?

Moved from NotRexed/Hookshot#2797.

### The idea Let an approved test run on a volunteer's Steam Deck, AMD or Intel GPU, or smaller NVIDIA card without waiting for someone to perform every step by hand. Dr.Crow suggested the hardware queue; the proposal is now for **self-hosted Forgejo runners**, not GitHub Actions. Route by GPU, driver, OS and memory, and keep the existing [test-request queue](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/5) as the human-controlled fallback. A job would build one approved commit using that machine's own dump, run a bounded scripted scene without a window, and return the exact commit, hardware, driver and reviewed observations. Nothing made from the game would be transferred between machines. A game-free GPU fixture would be a safer first workload: no dump, save or game build to protect. **What already exists, and what does not follow from it:** [the open release migration](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/13) moves ordinary checks and releases to Forgejo Actions. Its runner guide asks for `ubuntu-latest`, `ubuntu-24.04` and `windows-2022`; these are workflow requirements, not a hardware inventory. The launcher has Actions enabled and [a checks job waiting for `ubuntu-latest`](https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/actions/runs/4/jobs/0/attempt/1). Neither establishes a working AMD, Intel or Deck lane. The runner inventory is not public, and Longshot's [current contribution guide](https://git.magicbean.dev/MagicBeanDev/Longshot/src/commit/d4a69456c9de5131589af1fcf9a09adba4c403d1/CONTRIBUTING.md#testing) still says to run the checks locally. This proposal is an additional opt-in hardware service, not a claim that the release runners already provide it. **The security conditions from the earlier discussion still apply:** 1. **An owner-controlled workflow, one exact revision.** Dispatch the trusted workflow with the full PR head SHA, never run a PR-controlled workflow on the hardware machine. A label may request a run, but approval must bind that SHA and become invalid on the next push. Recheck it before execution and name the tested SHA in the result. Forgejo's author/workflow approval is not this per-commit test approval. 2. **Review everything that executes.** That includes `nx2/`, `scripts/`, `tests/`, workflows and dependencies, not only the proposed game fix. The build is untrusted code too. 3. **A disposable, confined environment.** No access to the owner's files or keys; private home and disposable saves; the dump read-only; no network for PR-controlled build or game code after the trusted fetch; bounded time, memory and disk. Splitting the no-GPU build VM from the GPU run may be practical, but a GPU sandbox is not equivalent to a throwaway VM. The pilot must show the actual isolation, cleanup and remaining GPU-driver risk, not quietly weaken this condition. 4. **Keep credentials and publication outside the payload.** No forge token, signing key or publishing credential in the build/game environment. Isolate the runner's own registration credential in its controller. A separate trusted publisher must never execute PR code and must accept only a tightly bounded result format. No raw job output or directory uploads: even logs and screenshots can be made to contain private data by malicious code, so review them before publication. A read-only dump is still readable and can still leak through a result. 5. **Opt-in, never a required merge check.** The owner chooses when the machine is available. Offline, refused or timed-out work falls back to the test request; it is not a pass. **Runner placement is a separate boundary.** [Forgejo's security guide](https://forgejo.org/docs/v17.0/user/actions/security/) says runners can be selected from the instance, organization or repository, and host-mode runners have no real isolation. [Trusted PR authors can supply workflow changes](https://forgejo.org/docs/v17.0/user/actions/security-pull-request/). A hardware label or a dispatch-only file therefore does not stop another eligible workflow selecting that runner. Start with an owner's fork or dedicated private-control repository, used only for trusted workflows; do not register a personal PC at the shared organization or upstream repository merely because the intended workflow is gated. Any later upstream service needs a demonstrated allocation boundary. Release/signing jobs must not share a potentially compromised PR-test environment. ### Anything else **A small first step:** an owner-controlled consumer of the existing test queue, then one game-free Forgejo pilot on my fork and PC. I can offer Linux on an RTX 4090 and Intel UHD 770. Only after its confinement and publication path have been reviewed would I propose a dump-using pilot. Dedicated devices come before daily-use PCs; SteamOS, Windows, GPU access and persistence through updates need their own proof. There is no Deck build-time promise here. Reuse other work rather than add parallel infrastructure: [the sanitizer proposal](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/29) is game-free, while [the live/restart A/B proposal](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/36) could supply a reviewed workload if it lands. Neither is a hardware-runner security gate. Extra cloud CI capacity is a different question; the old GitHub minutes and queue measurements do not establish a Forgejo bottleneck. Questions for the maintainers: 1. Does the rule that game runs stay local permit this narrowly confined, owner-approved hardware automation? Until that is decided, keep the pilot game-free and all game files local. 2. Should execution remain under each tester's local/fork control, or become an upstream-managed service? Who approves each exact commit, owns the machines and reviews what leaves them? --- Moved from NotRexed/Hookshot#2797.

mooki commented

2026-11-12 23:57:12 +00:00

Author

Discussion carried over from NotRexed/Hookshot#2797

Every comment that was readable on the original thread when it was copied on 2026-11-12, in order, unchanged except that issue and pull request numbers are written as links to the old repository and names carry no @. Comments by an account that has since been closed there could no longer be read and are missing; where a reply quotes them, the quote remains.

mooki000, 2026-11-12 14:10 UTC (comment, original):

Research findings (primary sources read 2026-11-12): this is feasible, but I recommend starting with an owner-controlled local consumer of the existing test-request queue, not registering home PCs upstream yet.

The important corrections to my initial sketch:

Dispatch and labels are not isolation. Another PR-controlled workflow can select the same repository runner; first-time-contributor approval is not a security boundary. GitHub explicitly warns against public-repository self-hosted runners (security guidance, fork-approval rules).
A just-in-time (JIT) runner means at most one job, not a wiped or sandboxed PC. Cleanup/isolation are ours to provide. An offline/missing runner leaves a job queued for up to 24 hours, not indefinitely (runner reference).
NotRexed/Hookshot#2746 approves a PR, not a particular revision. Automation needs approval of the full head SHA and must refuse a changed head; a retained testing wanted label must not authorize every later push (current client).

The least-exposed design separates the networked controller/publisher from the build and game: fresh restricted account/environment, private home and disposable saves, read-only dump, no network in the PR-code process, bounded resources, and a trusted fixed-format result filter. A separate GitHub-hosted job can comment without ever executing the PR's code. GPU-driver faults/escapes and false results remain possible; read-only access does not prevent reading and leaking a dump.

No raw logs or directory artifacts. Public Actions logs are readable by other GitHub users; upload only small, validated observations. Normal log excerpts and reviewed screenshots are allowed by our rules, but a malicious PR can change what they contain (project rule, log access).

Steam Deck is a separate proof, not a promised supported runner. Windows 11 is on GitHub's supported list; Arch/SteamOS are not. A home-folder installation avoids modifying SteamOS's read-only image, but dependencies/service persistence still need testing across updates. Each tester must build locally from their own dump; the guide's “about 40 minutes” is not a Deck benchmark (supported systems, Valve guidance, build guide).

Useful precedents: Dolphin's current build controller rejects untrusted automatic requests and accepts a trusted developer's rebuild request. Mesa uses Marge/manual farm testing, but its Docker-farm warning says in its own words that this lets "anyone on the internet run code on your device". Neither proves a volunteer's personal files are safe. ANGLE demonstrates permissioned routing by GPU and driver.

Extra hosted capacity is not an observed problem now. In 60 recent runs (13:04–13:42 UTC), 125 executed jobs from completed runs queued a median 3 seconds, maximum 73 seconds. Nine completed PR-head cohorts took 4.47–6.00 minutes wall time; summed job-minutes are a different measure. Example: source checks. This is a short sample, not a monthly usage audit.

GitHub's standard public-repository runners are free and unlimited in minutes. CircleCI offers 400,000 Linux OSS credits/month on Free, but its GPU classes require Scale and are NVIDIA. Neither cloud can hold a dump or supply the missing AMD/Intel/Deck coverage.

Suggested stages: local queue consumer first; then one confined just-in-time experiment on my own fork and PC, with results posted only to a test pull request there; upstream only after that proof and an explicit policy decision. I have a two-job workflow draft and the confinement/publication acceptance criteria to offer. Nothing has been installed, registered or run for this research.

Two decisions for maintainers:

Does “No game in CI” permit this narrow opt-in local-hardware exception, without transferring game files or derived artifacts?
Should execution stay under testers' local/fork control, or become an upstream-managed service—and who may approve each exact commit and own that service?

mooki000, 2026-11-12 14:46 UTC (comment, original):

What the Discord discussion added (#dev-work and its thread, 2026-11-12), written down here so the issue stays the record. Corrections welcome from anyone I have summarised.

A security review, from _trash.panda ("AppSec take: good idea"), who welcomed a pilot on a fork and offered to review its workflow. Before anything is registered on this repository:

Dispatch only from main with the pull request's commit as input, never on pull_request, so a pull request cannot edit the workflow that runs it.
Approval has to cover the build-time code (nx2/, scripts/, tests/, workflows), since that runs on the tester's machine.
A throwaway VM per job: no access to the owner's files or keys, the dump mounted read-only, no network after checkout, so the dump cannot leave.
No credentials on the runner. Results are posted by a separate GitHub-hosted job, and only numbers, log lines and screenshots go out.
Opt-in only, never a merge check.

From Dr.Crow, who made the original suggestion:

Runners on open-source projects are fine if hardened: never pass secrets to them, treat them as compromised.
Be selective. Do not run hardware tests on every pull request; start them by adding a label such as run-ci-on-hardware, optional and never blocking a merge, "to give folks a better idea of the impact of the PR on performance across the different sets of hardware".
Game files never travel through CI. The workflow checks that the dump is present on the machine (a hash check) and only then runs.
CircleCI as a second platform only if GitHub's minutes or queues become the limit. A plan for an open-source project should be obtainable, but most of its GPUs are not consumer cards. He also offered help with CI speed in general: the faster the checks, the sooner an agent knows whether its change is right.

NitnekLeBorgne: in favour; "the major work will be about security".

Joe Staff asked what would run on a runner, given how long the game takes to build. The same steps a tester does by hand: build the approved commit from that machine's own dump, run a scripted scene without a window for a minute or three, report. A complete source build measured today: 14 min 55 s on an i9-13900K on Linux (NotRexed/Hookshot#2793's description), 20.5 min on an i7-13700F on Windows (the latest cycle on NotRexed/Hookshot#2595). So a cold job is roughly 20 to 40 minutes; rebuilding only what a pull request changed is much shorter, and that depends on how much of a previous build a runner may keep (see idea 1).

Ideas added in the thread

Split the build from the run. The build is where most pull-request-controlled code runs (the Python build scripts), and it needs no GPU: it can sit in a throwaway VM or container with the dump read-only and no network (condition 3 in full). Only the built binary needs the real GPU, and that can run in a tight sandbox. A sandbox is weaker than a VM and the pilot will say exactly what it gives; handing a GPU to a VM is not practical on a one-card desktop.
Dedicated devices first. A Steam Deck that holds nothing but the dump is its own throwaway machine. PCs in daily use are better off working through the test-request queue locally under their owner's control.
The label can be made exact: it starts main's workflow file, not the pull request's, with the head commit at label time as the input, and is dropped on the next push. That would give Dr.Crow's trigger together with condition 1; what that event exposes is being checked before I claim it.
Test requests should pin a commit. A card names a pull request today. On NotRexed/Hookshot#2731 this morning the head moved (a fix for a defect a tester found) while its card was being tested, and a later "pass" was for the earlier commit. I am preparing a pull request for this: the card records the head, shows when it has moved, and a result names the commit tested. It needs no runner and any automation needs it.
Game-free GPU tests need no dump at all. A Vulkan validation job on a software driver in the normal CI costs nothing, and the same tests could later run on real AMD, Intel or Deck runners that hold no game files, which removes the hardest part of the security problem. It is also the only kind of test a cloud GPU could run for this project. Whether the renderer can be driven without the game is being looked at.

Where this stands

Now: the commit-pinning pull request (idea 4).
Next: a pilot on my fork with my own PC, written to the five conditions above, with _trash.panda as reviewer of its workflow. Nothing is registered anywhere yet.
Being scoped: the game-free GPU job (idea 5), and what a CircleCI plan could add for it.
Unchanged: nothing on this repository until the pilot has been shown and the maintainers have answered the two questions in the research comment above ("No game in CI" exception; who runs and approves).

Joey-Magnus, 2026-11-12 14:51 UTC (comment, original):

Thanks for writing the Discord discussion up, mooki. That covers the AppSec side. Two additions so the issue stays the full record:

Your idea 3, the exact label, is now a PR: #2807.

A hardware: <name> label starts main's copy of the workflow for the head commit at label time.
Any new push removes the labels.
The job on the tester's machine gets no token and no checkout, and only calls the owner's sandbox program.
A separate job on GitHub's own runners posts the results as quoted text.

It's inert until someone registers a runner, and a game-free test keeps those rules in place.

One correction to my own conditions, prompted by your research comment: a runner registered on this repository isn't protected by dispatch, labels or approvals. A PR can add its own pull_request workflow that asks for the runner's labels and runs straight on it, past the gate and the sandbox. So:

Runners go on testers' own forks only, where only the owner pushes workflows. NotRexed/Hookshot#2807's doc now says so.
Upstream runners would need runner groups limited to one workflow. That needs an organization, and it's part of your second question to the maintainers.

Your thread ideas, from the AppSec side:

Split the build from the run: agreed. It's the best way to avoid needing GPU passthrough, and NotRexed/Hookshot#2807's sandbox requirements now include it.
Dedicated devices first: agreed. A Deck holding only the dump is the easiest machine to make safe.
Pin test requests to a commit: agreed, and any automation needs the same, which NotRexed/Hookshot#2807 enforces.
Game-free GPU tests: the best risk reduction of all, because a runner with no dump has nothing to leak.

Still happy to review the fork pilot's workflow. Tag me when it's up.

## Discussion carried over from NotRexed/Hookshot#2797 Every comment that was readable on [the original thread](https://github.com/NotRexed/Hookshot/issues/2797) when it was copied on 2026-11-12, in order, unchanged except that issue and pull request numbers are written as links to the old repository and names carry no @. Comments by an account that has since been closed there could no longer be read and are missing; where a reply quotes them, the quote remains. --- **mooki000**, 2026-11-12 14:10 UTC (comment, [original](https://github.com/NotRexed/Hookshot/issues/2797#issuecomment-6825904272)): Research findings (primary sources read 2026-11-12): **this is feasible, but I recommend starting with an owner-controlled local consumer of the existing test-request queue, not registering home PCs upstream yet.** The important corrections to my initial sketch: - **Dispatch and labels are not isolation.** Another PR-controlled workflow can select the same repository runner; first-time-contributor approval is not a security boundary. GitHub explicitly warns against public-repository self-hosted runners ([security guidance](https://docs.github.com/en/actions/reference/security/secure-use), [fork-approval rules](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository)). - **A just-in-time (JIT) runner means at most one job, not a wiped or sandboxed PC.** Cleanup/isolation are ours to provide. An offline/missing runner leaves a job queued for up to **24 hours**, not indefinitely ([runner reference](https://docs.github.com/en/actions/reference/runners/self-hosted-runners)). - **[NotRexed/Hookshot#2746](https://github.com/NotRexed/Hookshot/issues/2746) approves a PR, not a particular revision.** Automation needs approval of the full head SHA and must refuse a changed head; a retained `testing wanted` label must not authorize every later push ([current client](https://github.com/NotRexed/Hookshot/blob/51da0464b4cf0c7f56b81ebd6bb309325835e175/scripts/test_requests.py)). The least-exposed design separates the networked controller/publisher from the build and game: fresh restricted account/environment, private home and disposable saves, read-only dump, no network in the PR-code process, bounded resources, and a trusted fixed-format result filter. A separate GitHub-hosted job can comment without ever executing the PR's code. GPU-driver faults/escapes and false results remain possible; read-only access does not prevent reading and leaking a dump. **No raw logs or directory artifacts.** Public Actions logs are readable by other GitHub users; upload only small, validated observations. Normal log excerpts and reviewed screenshots are allowed by our rules, but a malicious PR can change what they contain ([project rule](https://github.com/NotRexed/Hookshot/blob/51da0464b4cf0c7f56b81ebd6bb309325835e175/CONTRIBUTING.md#the-rule-with-no-exceptions), [log access](https://docs.github.com/en/actions/how-tos/monitor-workflows/use-workflow-run-logs)). **Steam Deck is a separate proof, not a promised supported runner.** Windows 11 is on GitHub's supported list; Arch/SteamOS are not. A home-folder installation avoids modifying SteamOS's read-only image, but dependencies/service persistence still need testing across updates. Each tester must build locally from their own dump; the guide's “about 40 minutes” is not a Deck benchmark ([supported systems](https://docs.github.com/en/actions/reference/runners/self-hosted-runners), [Valve guidance](https://help.steampowered.com/en/faqs/view/671A-4453-E8D2-323C), [build guide](https://github.com/NotRexed/Hookshot/blob/51da0464b4cf0c7f56b81ebd6bb309325835e175/CONTRIBUTING.md#building)). Useful precedents: [Dolphin's current build controller](https://github.com/dolphin-emu/central/blob/master/central/buildbot.py) rejects untrusted automatic requests and accepts a trusted developer's rebuild request. [Mesa](https://docs.mesa3d.org/ci/index.html) uses Marge/manual farm testing, but its [Docker-farm warning](https://docs.mesa3d.org/ci/docker.html) says in its own words that this lets "anyone on the internet run code on your device". Neither proves a volunteer's personal files are safe. [ANGLE](https://github.com/google/angle/blob/main/doc/TestingOnBots.md) demonstrates permissioned routing by GPU **and driver**. **Extra hosted capacity is not an observed problem now.** In 60 recent runs (13:04–13:42 UTC), 125 executed jobs from completed runs queued a median **3 seconds**, maximum **73 seconds**. Nine completed PR-head cohorts took **4.47–6.00 minutes wall time**; summed job-minutes are a different measure. Example: [source checks](https://github.com/NotRexed/Hookshot/actions/runs/43228154071). This is a short sample, not a monthly usage audit. GitHub's [standard public-repository runners are free and unlimited in minutes](https://docs.github.com/en/actions/reference/runners/github-hosted-runners). CircleCI offers [400,000 Linux OSS credits/month on Free](https://circleci.com/docs/guides/plans-pricing/credits/), but its [GPU classes require Scale and are NVIDIA](https://circleci.com/docs/guides/execute/using-gpu/). Neither cloud can hold a dump or supply the missing AMD/Intel/Deck coverage. **Suggested stages:** local queue consumer first; then one confined just-in-time experiment on my own fork and PC, with results posted only to a test pull request there; upstream only after that proof and an explicit policy decision. I have a two-job workflow draft and the confinement/publication acceptance criteria to offer. Nothing has been installed, registered or run for this research. Two decisions for maintainers: 1. Does **“No game in CI”** permit this narrow opt-in local-hardware exception, without transferring game files or derived artifacts? 2. Should execution stay under testers' local/fork control, or become an upstream-managed service—and who may approve each exact commit and own that service? --- **mooki000**, 2026-11-12 14:46 UTC (comment, [original](https://github.com/NotRexed/Hookshot/issues/2797#issuecomment-6826819619)): **What the Discord discussion added** (`#dev-work` and its thread, 2026-11-12), written down here so the issue stays the record. Corrections welcome from anyone I have summarised. **A security review, from _trash.panda** ("AppSec take: good idea"), who welcomed a pilot on a fork and offered to review its workflow. Before anything is registered on this repository: 1. Dispatch only from main with the pull request's commit as input, never on `pull_request`, so a pull request cannot edit the workflow that runs it. 2. Approval has to cover the build-time code (`nx2/`, `scripts/`, `tests/`, workflows), since that runs on the tester's machine. 3. A throwaway VM per job: no access to the owner's files or keys, the dump mounted read-only, no network after checkout, so the dump cannot leave. 4. No credentials on the runner. Results are posted by a separate GitHub-hosted job, and only numbers, log lines and screenshots go out. 5. Opt-in only, never a merge check. **From Dr.Crow**, who made the original suggestion: - Runners on open-source projects are fine if hardened: never pass secrets to them, treat them as compromised. - Be selective. Do not run hardware tests on every pull request; start them by adding a label such as `run-ci-on-hardware`, optional and never blocking a merge, "to give folks a better idea of the impact of the PR on performance across the different sets of hardware". - Game files never travel through CI. The workflow checks that the dump is present on the machine (a hash check) and only then runs. - CircleCI as a second platform only if GitHub's minutes or queues become the limit. A plan for an open-source project should be obtainable, but most of its GPUs are not consumer cards. He also offered help with CI speed in general: the faster the checks, the sooner an agent knows whether its change is right. **NitnekLeBorgne:** in favour; "the major work will be about security". **Joe Staff asked what would run on a runner, given how long the game takes to build.** The same steps a tester does by hand: build the approved commit from that machine's own dump, run a scripted scene without a window for a minute or three, report. A complete source build measured today: 14 min 55 s on an i9-13900K on Linux ([NotRexed/Hookshot#2793](https://github.com/NotRexed/Hookshot/issues/2793)'s description), 20.5 min on an i7-13700F on Windows (the latest cycle on [NotRexed/Hookshot#2595](https://github.com/NotRexed/Hookshot/issues/2595)). So a cold job is roughly 20 to 40 minutes; rebuilding only what a pull request changed is much shorter, and that depends on how much of a previous build a runner may keep (see idea 1). **Ideas added in the thread** 1. **Split the build from the run.** The build is where most pull-request-controlled code runs (the Python build scripts), and it needs no GPU: it can sit in a throwaway VM or container with the dump read-only and no network (condition 3 in full). Only the built binary needs the real GPU, and that can run in a tight sandbox. A sandbox is weaker than a VM and the pilot will say exactly what it gives; handing a GPU to a VM is not practical on a one-card desktop. 2. **Dedicated devices first.** A Steam Deck that holds nothing but the dump is its own throwaway machine. PCs in daily use are better off working through the test-request queue locally under their owner's control. 3. **The label can be made exact:** it starts main's workflow file, not the pull request's, with the head commit at label time as the input, and is dropped on the next push. That would give Dr.Crow's trigger together with condition 1; what that event exposes is being checked before I claim it. 4. **Test requests should pin a commit.** A card names a pull request today. On [NotRexed/Hookshot#2731](https://github.com/NotRexed/Hookshot/issues/2731) this morning the head moved (a fix for a defect a tester found) while its card was being tested, and a later "pass" was for the earlier commit. I am preparing a pull request for this: the card records the head, shows when it has moved, and a result names the commit tested. It needs no runner and any automation needs it. 5. **Game-free GPU tests need no dump at all.** A Vulkan validation job on a software driver in the normal CI costs nothing, and the same tests could later run on real AMD, Intel or Deck runners that hold no game files, which removes the hardest part of the security problem. It is also the only kind of test a cloud GPU could run for this project. Whether the renderer can be driven without the game is being looked at. **Where this stands** - Now: the commit-pinning pull request (idea 4). - Next: a pilot on my fork with my own PC, written to the five conditions above, with _trash.panda as reviewer of its workflow. Nothing is registered anywhere yet. - Being scoped: the game-free GPU job (idea 5), and what a CircleCI plan could add for it. - Unchanged: nothing on this repository until the pilot has been shown and the maintainers have answered the two questions in the research comment above ("No game in CI" exception; who runs and approves). --- **Joey-Magnus**, 2026-11-12 14:51 UTC (comment, [original](https://github.com/NotRexed/Hookshot/issues/2797#issuecomment-6826930483)): Thanks for writing the Discord discussion up, mooki. That covers the AppSec side. Two additions so the issue stays the full record: **Your idea 3, the exact label, is now a PR: [#2807](https://github.com/NotRexed/Hookshot/pull/2807).** - A `hardware: <name>` label starts `main`'s copy of the workflow for the head commit at label time. - Any new push removes the labels. - The job on the tester's machine gets no token and no checkout, and only calls the owner's sandbox program. - A separate job on GitHub's own runners posts the results as quoted text. It's inert until someone registers a runner, and a game-free test keeps those rules in place. **One correction to my own conditions, prompted by your research comment:** a runner registered on *this* repository isn't protected by dispatch, labels or approvals. A PR can add its own `pull_request` workflow that asks for the runner's labels and runs straight on it, past the gate and the sandbox. So: - Runners go on testers' own forks only, where only the owner pushes workflows. [NotRexed/Hookshot#2807](https://github.com/NotRexed/Hookshot/issues/2807)'s doc now says so. - Upstream runners would need runner groups limited to one workflow. That needs an organization, and it's part of your second question to the maintainers. **Your thread ideas, from the AppSec side:** - **Split the build from the run:** agreed. It's the best way to avoid needing GPU passthrough, and [NotRexed/Hookshot#2807](https://github.com/NotRexed/Hookshot/issues/2807)'s sandbox requirements now include it. - **Dedicated devices first:** agreed. A Deck holding only the dump is the easiest machine to make safe. - **Pin test requests to a commit:** agreed, and any automation needs the same, which [NotRexed/Hookshot#2807](https://github.com/NotRexed/Hookshot/issues/2807) enforces. - **Game-free GPU tests:** the best risk reduction of all, because a runner with no dump has nothing to leak. Still happy to review the fork pilot's workflow. Tag me when it's up.

mooki referenced this issue

2026-11-13 01:24:41 +00:00

WIP: ci: run authored Vulkan fixtures locally and prepare Forgejo execution #67