feat(agents): test requests: ask the team to test a PR on other hardware, claim it, report the results #2734

Closed

github-sync wants to merge 1 commit from github/pr/2734 into main

github-sync commented

2026-11-12 10:34:15 +00:00

Owner

@hckrman101 opened this pull request on GitHub · 2026-11-12 09:51 UTC · original
Merged on GitHub in 6dcc15c72e

Testing a PR on hardware the author doesn't have (another GPU maker, Linux, an 8 GB card, the Steam Deck) is our slowest step, and today it is asked for by hand in Discord with no way to see who took it or whether it was done. This adds a shared queue for it that agents and people can use.

The flow

The author's agent runs /request-testing: scripts/test_requests.py request --pr 2711 --summary "<the change in one line>" --needs "AMD RX 6000 or newer, Windows" --notes "..." posts a request with an ID (T-7KQ2) in #test-requests:magicbean.dev.
Testers' agents run /testing (or /loop 30m /testing). list sorts what most needs a tester first (nobody tested or testing, then being tested with no result, then with results), and the agent takes the first request this PC fits. It claims it (several people can test one request), keeps the claim alive with renew (no ID: all of yours) while it works, builds and tests the PR, posts the results as a comment on the PR, then runs done --result pass|fail|mixed --comment <link>. A claim nobody renews clears by itself after 90 minutes.
The author's agent reads each tester's result back with list --mine --all and the comment links, and closes the request (cancel) when it has what it needs.

Every message is one line led by the ID and the action, easy to parse and to type by hand:

🧪 T-7KQ2 request: Keep 8 GB cards inside their memory budget
Needs: AMD RX 6000 or newer, Windows                (then the PR link, then the notes)
🧪 T-7KQ2 claim: until 2026-11-12 16:40Z
🧪 T-7KQ2 renew: until 2026-11-12 18:10Z
🧪 T-7KQ2 done pass: https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/2711
🧪 T-7KQ2 cancel

The room's messages are the only record: the script replays them with fixed rules (claims are per person and clear when they run out, only the claimer renews or releases, done must link a comment on the same PR and replaces that person's earlier result, only the requester closes) and refuses with exit code 3 what the replay would ignore (a closed request, claiming one you already reported on without --again).

The server's relay copies the room to Discord with the same rules: each request is one card (an embed), never more, edited as things happen. The title is the change and opens the PR; the card says who is testing it now and each tester's result with a link to their comment, then what the test needs and the PR with its state on GitHub (open, draft, merged, closed; checked every 15 minutes); and a small last line with, while it is open, a link to the testers' guide and /testing, and the ID. Its colour follows the state (blue nobody yet, yellow being tested, green passed, red a failure, orange mixed, grey closed), with a 👀 reaction while someone tests and ✅ once results are in. Renewing a claim changes nothing on the card.

For testers: docs/test-requests.md is the guide: a one-time setup (build from source, gh auth login, team_chat.py login), then /testing or /loop 30m /testing with an agent, or the commands by hand, a report template and the rules. CONTRIBUTING.md's Test on other hardware row, AGENTS.md and docs/workflow.md point to it.

Also: team_chat.py post no longer shows the robot twice when an agent starts its text with one (it happened in the room); the post keeps exactly one.

It uses the team chat sign-in from #2606 (no second login). Standard library only.

Tested

tests/test_team_chat.py: a post that starts with robots carries exactly one.
tests/test_test_requests.py (new): line parsing; the replay rules (two testers at once, claims clearing when they run out and their cap, renew/release only from the claimer, done from anyone with a comment on the same PR and replacing an earlier result, cancel only from the requester and nothing after it); the pick order; each command's refusals and posted lines. python -I -B tests/run_all.py test_requests team_chat: both pass.
End to end on the live server: request, claim, renew, cancel from the script; the relay posted one card per request to both Discord channels, edited it on the claim and the close (read back), and left it alone on the renew.

> [@hckrman101](https://github.com/hckrman101) opened this pull request on GitHub · 2026-11-12 09:51 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2734) > **Merged on GitHub** in https://git.magicbean.dev/MagicBeanDev/Hookshot/commit/6dcc15c72e745cf464e8ee0851cec16861f46e3c Testing a PR on hardware the author doesn't have (another GPU maker, Linux, an 8 GB card, the Steam Deck) is our slowest step, and today it is asked for by hand in Discord with no way to see who took it or whether it was done. This adds a shared queue for it that agents and people can use. **The flow** 1. The author's agent runs `/request-testing`: `scripts/test_requests.py request --pr 2711 --summary "<the change in one line>" --needs "AMD RX 6000 or newer, Windows" --notes "..."` posts a request with an ID (`T-7KQ2`) in `#test-requests:magicbean.dev`. 2. Testers' agents run `/testing` (or `/loop 30m /testing`). `list` sorts what most needs a tester first (nobody tested or testing, then being tested with no result, then with results), and the agent takes the first request this PC fits. It `claim`s it (several people can test one request), keeps the claim alive with `renew` (no ID: all of yours) while it works, builds and tests the PR, posts the results as a comment on the PR, then runs `done --result pass|fail|mixed --comment <link>`. A claim nobody renews clears by itself after 90 minutes. 3. The author's agent reads each tester's result back with `list --mine --all` and the comment links, and closes the request (`cancel`) when it has what it needs. Every message is one line led by the ID and the action, easy to parse and to type by hand: ``` 🧪 T-7KQ2 request: Keep 8 GB cards inside their memory budget Needs: AMD RX 6000 or newer, Windows (then the PR link, then the notes) 🧪 T-7KQ2 claim: until 2026-11-12 16:40Z 🧪 T-7KQ2 renew: until 2026-11-12 18:10Z 🧪 T-7KQ2 done pass: https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/2711 🧪 T-7KQ2 cancel ``` The room's messages are the only record: the script replays them with fixed rules (claims are per person and clear when they run out, only the claimer renews or releases, `done` must link a comment on the same PR and replaces that person's earlier result, only the requester closes) and refuses with exit code 3 what the replay would ignore (a closed request, claiming one you already reported on without `--again`). The server's relay copies the room to Discord with the same rules: each request is one card (an embed), never more, edited as things happen. The title is the change and opens the PR; the card says who is testing it now and each tester's result with a link to their comment, then what the test needs and the PR with its state on GitHub (open, draft, merged, closed; checked every 15 minutes); and a small last line with, while it is open, a link to the testers' guide and `/testing`, and the ID. Its colour follows the state (blue nobody yet, yellow being tested, green passed, red a failure, orange mixed, grey closed), with a 👀 reaction while someone tests and ✅ once results are in. Renewing a claim changes nothing on the card. **For testers:** [docs/test-requests.md](docs/test-requests.md) is the guide: a one-time setup (build from source, `gh auth login`, `team_chat.py login`), then `/testing` or `/loop 30m /testing` with an agent, or the commands by hand, a report template and the rules. CONTRIBUTING.md's *Test on other hardware* row, AGENTS.md and docs/workflow.md point to it. **Also:** `team_chat.py post` no longer shows the robot twice when an agent starts its text with one (it happened in the room); the post keeps exactly one. It uses the team chat sign-in from #2606 (no second login). Standard library only. **Tested** - `tests/test_team_chat.py`: a post that starts with robots carries exactly one. - `tests/test_test_requests.py` (new): line parsing; the replay rules (two testers at once, claims clearing when they run out and their cap, renew/release only from the claimer, done from anyone with a comment on the same PR and replacing an earlier result, cancel only from the requester and nothing after it); the pick order; each command's refusals and posted lines. `python -I -B tests/run_all.py test_requests team_chat`: both pass. - End to end on the live server: request, claim, renew, cancel from the script; the relay posted one card per request to both Discord channels, edited it on the claim and the close (read back), and left it alone on the renew.

github-sync added 1 commit

2026-11-12 10:34:15 +00:00

feat(agents): test requests: ask the team to test a PR on other hardware, claim and report results f8bd183277

scripts/test_requests.py posts and replays one-line messages (ID + action) in #test-requests:magicbean.dev:
request (summary, needs, PR, notes), claim (several testers at once; claims clear unless renewed), renew,
release, done with a link to the PR comment, cancel. list puts untested requests first. Two skills:
/request-testing (ask and read the results) and /testing (pick, claim, test, report; a loop), and a guide for
testers, docs/test-requests.md. team_chat.py: a post that starts with the robot no longer shows two.

~~github-sync referenced this pull request 2026-11-12 10:34:29 +00:00~~

chore(main): release 2.2.0-rc.1 #2071

github-sync closed this pull request

2026-11-12 10:35:17 +00:00

github-sync referenced this pull request

2026-11-12 11:35:39 +00:00

feat(agents): test requests need a collaborator's PR or the "testing wanted" label #2746

github-sync referenced this pull request from a commit

2026-11-12 11:35:41 +00:00

feat(agents): test requests need a collaborator's PR or the "testing wanted" label (#2746)

github-sync referenced this pull request

2026-11-12 14:41:10 +00:00