MagicBeanDev/Longshot
MagicBeanDev/Longshot
Watch
76
99
Fork
You've already forked Longshot
 104
Code Issues 259 Pull requests 232 Projects Releases 9 Packages Wiki Activity Actions 754

docs: the Test Farm replaces test requests; the site links to Beanstalk's How it works #953

Merged
HumanBean merged 1 commit from HumanBean/Longshot:docs/test-farm-beanstalk into main 2026-11-15 22:43:48 +00:00
Conversation 0 Commits 1 Files changed 13 +23 -1123
HumanBean commented 2026-11-15 22:38:44 +00:00
Owner

Type: docs · Risk: low · Tested on: Linux (checks, website build) · Needs: none

Why

Hardware tests now run on the Test Farm (Beanstalk). Approved pull requests are built and played on volunteers' PCs, and
the results are posted back. The manual test requests (a Matrix room, scripts/test_requests.py and two skills) are
the older way of doing the same thing, and keeping both confuses contributors and agents. How the farm works now lives on
Beanstalk's own dashboard, beanstalk.magicbean.dev/how
(MagicBeanDev/Beanstalk#21), so the site links there instead of keeping its own copy. This replaces #542, which was
closed for being out of date.

What changed

The site's sidebar links to Beanstalk's How it works page. Test requests are removed, and every place that sent people
to them now sends them to the Test Farm.

  • Removed: scripts/test_requests.py and tests/test_test_requests.py, docs/test-requests.md, and the
    request-testing and testing skills.
  • website/astro.config.ts: "Help test (Test Farm) ↗" under Development, linking to
    https://beanstalk.magicbean.dev/how.
  • AGENTS.md: test requests leave the tooling section and the skills list; "Getting a change tested on hardware you
    don't have" now means @Beanstalk <what to test> on the pull request.
  • CONTRIBUTING.md, docs/workflow.md, docs/reporting-and-contributing.md ("Help with testing"),
    docs/agent-pull-requests.md: the Test Farm in place of test requests. The #test-requests room is dropped from the
    channels table.
  • scripts/tooling.py and the team page: test requests are no longer mentioned.
  • Unchanged on purpose: the past runs in docs/tested/ and docs/tested-configurations.md keep their test request IDs.

Screenshot of the built site (npm run build, then astro preview), with the new sidebar entry under Development:

After: this PR @ 518baba0, the Contributing page with Help test (Test Farm) under Development, 1280 px, dark

Measurements

Nothing to measure: docs, a sidebar link and the removal of a script with its test.

Flags and settings

None

Testing

Tested:

  • python -I -B tests/run_all.py: 546 passed, 18 failed, 76 skipped on Linux. The 17 failures that also happen on
    main without this change are test_appimage_player_build, test_bake_portability, test_crash_dialog,
    test_crt_options, test_env_on, test_lift_cleanup_error, test_linux_preflight, test_linux_update,
    test_make_game_cache, test_missing_nup, test_mods_size_table, test_options_budget_once, test_pipeline_tour,
    test_readback_late, test_vk_occlusion_queries, test_vk_pgo and test_walk_key. The 18th,
    byte_compile_tracked, failed only because the deletions weren't committed yet when it ran (it lists git's tracked
    files). Once they were committed it passed.
  • website: npm run build passes. npm run check has one failure, graphics-settings.test.ts ("HDR Native condition
    accepts current prose but remains required"). It reads docs/graphics-and-controls.md, which this PR doesn't touch.

Not tested: nothing else is affected: no game code changes.

Must be tested: None

Risks

What could break: anyone still running scripts/test_requests.py loses it. Requests already open in the Matrix room
stay there but get no new ones.

Turning it off: revert the commit

Compatibility: Nothing changes

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Screenshots of the same spot, captioned like the Measurements columns, for a change that shows on screen.
  • Docs updated.
**Type:** docs · **Risk:** low · **Tested on:** Linux (checks, website build) · **Needs:** none ## Why Hardware tests now run on the Test Farm (Beanstalk). Approved pull requests are built and played on volunteers' PCs, and the results are posted back. The manual test requests (a Matrix room, `scripts/test_requests.py` and two skills) are the older way of doing the same thing, and keeping both confuses contributors and agents. How the farm works now lives on Beanstalk's own dashboard, [beanstalk.magicbean.dev/how](https://beanstalk.magicbean.dev/how) (MagicBeanDev/Beanstalk#21), so the site links there instead of keeping its own copy. This replaces #542, which was closed for being out of date. ## What changed The site's sidebar links to Beanstalk's How it works page. Test requests are removed, and every place that sent people to them now sends them to the Test Farm. - Removed: `scripts/test_requests.py` and `tests/test_test_requests.py`, `docs/test-requests.md`, and the `request-testing` and `testing` skills. - `website/astro.config.ts`: "Help test (Test Farm) ↗" under Development, linking to `https://beanstalk.magicbean.dev/how`. - `AGENTS.md`: test requests leave the tooling section and the skills list; "Getting a change tested on hardware you don't have" now means `@Beanstalk <what to test>` on the pull request. - `CONTRIBUTING.md`, `docs/workflow.md`, `docs/reporting-and-contributing.md` ("Help with testing"), `docs/agent-pull-requests.md`: the Test Farm in place of test requests. The `#test-requests` room is dropped from the channels table. - `scripts/tooling.py` and the team page: test requests are no longer mentioned. - Unchanged on purpose: the past runs in `docs/tested/` and `docs/tested-configurations.md` keep their test request IDs. Screenshot of the built site (`npm run build`, then `astro preview`), with the new sidebar entry under Development: ![After: this PR @ 518baba0, the Contributing page with Help test (Test Farm) under Development, 1280 px, dark](https://git.magicbean.dev/attachments/34833292-4bc8-453e-82b9-c0d91771105a) ## Measurements Nothing to measure: docs, a sidebar link and the removal of a script with its test. ### Flags and settings None ## Testing **Tested:** - `python -I -B tests/run_all.py`: 546 passed, 18 failed, 76 skipped on Linux. The 17 failures that also happen on `main` without this change are `test_appimage_player_build`, `test_bake_portability`, `test_crash_dialog`, `test_crt_options`, `test_env_on`, `test_lift_cleanup_error`, `test_linux_preflight`, `test_linux_update`, `test_make_game_cache`, `test_missing_nup`, `test_mods_size_table`, `test_options_budget_once`, `test_pipeline_tour`, `test_readback_late`, `test_vk_occlusion_queries`, `test_vk_pgo` and `test_walk_key`. The 18th, `byte_compile_tracked`, failed only because the deletions weren't committed yet when it ran (it lists git's tracked files). Once they were committed it passed. - `website`: `npm run build` passes. `npm run check` has one failure, `graphics-settings.test.ts` ("HDR Native condition accepts current prose but remains required"). It reads `docs/graphics-and-controls.md`, which this PR doesn't touch. **Not tested:** nothing else is affected: no game code changes. **Must be tested:** None ## Risks **What could break:** anyone still running `scripts/test_requests.py` loses it. Requests already open in the Matrix room stay there but get no new ones. **Turning it off:** revert the commit **Compatibility:** Nothing changes ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Screenshots of the same spot, captioned like the Measurements columns, for a change that shows on screen. - [x] Docs updated.
site-sidebar.png
186 KiB
docs: the Test Farm replaces test requests; the site links to Beanstalk's How it works
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Details
Third-party parts / The same as upstream (pull_request) Has been skipped
Details
Website / Website on Cloudflare (pull_request) Has been skipped
Details
CI / Conventional Commit title (pull_request) Successful in 3s
Details
CI / Workflow lint (actionlint) (pull_request) Successful in 9s
Details
Website / Website on Cloudflare (pull_request_target) Has been skipped
Details
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s
Details
Third-party parts / As their notes say (pull_request) Successful in 9s
Details
PR description / Description follows the template (pull_request_target) Successful in 4s
Details
518baba01a 
How the Test Farm works now lives on Beanstalk's dashboard
(https://beanstalk.magicbean.dev/how); the site's sidebar links to it
under Development, and the docs that sent people to test requests send
them to the farm (@Beanstalk on the pull request) instead.

Test requests are removed: scripts/test_requests.py and its test,
docs/test-requests.md, the request-testing and testing skills, and their
mentions in AGENTS.md, CONTRIBUTING.md, docs/workflow.md,
docs/reporting-and-contributing.md, docs/agent-pull-requests.md,
scripts/tooling.py and the team page. Past runs in docs/tested/ and
tested-configurations.md keep their test request IDs.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean changed title from WIP: docs: the Test Farm replaces test requests; the site links to Beanstalk's How it works to docs: the Test Farm replaces test requests; the site links to Beanstalk's How it works 2026-11-15 22:43:44 +00:00
HumanBean deleted branch docs/test-farm-beanstalk 2026-11-15 22:43:49 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area/accessibility
Barriers for people with disabilities

  
area/audio
Sound and music

  
area/input
Controllers, keyboard, mouse

  
area/install
Installing, updating, building from your game

  
area/menu
In-game menu and settings

  
area/renderer
Vulkan renderer, shaders, effects

  
area/saves
Save files

  
area/tooling
Developer tools: scripts, CI, build and release tooling

  
area/upscaling
DLSS, FSR, frame generation

  
beanstalk: approved
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: needs approval
Set by Beanstalk for show; the farm's database is what counts

  
beanstalk: running
Set by Beanstalk for show; the farm's database is what counts

  
ci: approved
A maintainer approved CircleCI for this fork PR (sticky; removed after 14 quiet days)

  
good first issue
A good start for a new contributor

  
gpu/amd
AMD GPUs

  
gpu/igpu
Integrated GPUs

  
gpu/intel
Intel GPUs

  
gpu/nvidia
NVIDIA GPUs

  
help wanted
Extra hands or knowledge wanted

  
kind
bug
 Something doesn't work as it should

  
kind
chore
 Build, CI, dependencies, tidying

  
kind
docs
 Documentation

  
kind
feature
 A new feature or an improvement

  
kind
performance
 Frame rate, frame time, loading, memory

  
kind
question
 A question rather than a change

  
needs measurements
Waiting for before/after measurements

  
platform/linux
Affects Linux (a change for every system carries both Windows and Linux)

  
platform/macos
macOS (Apple silicon or Intel Macs)

  
platform/steam-deck
Steam Deck

  
platform/windows
Affects Windows (a change for every system carries both Windows and Linux)

  
preview
Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  
priority
critical
 Crashes, data loss, or blocks most players

  
priority
high
 Important: next in line

  
priority
low
 Nice to have

  
regression
Worked in an earlier build, broken in a newer one: highest priority, above other bugs

  
risk
high
 Broad or core change: the renderer, defaults on many systems, saves, installs, security

  
risk
low
 Small, contained change: little can break

  
risk
medium
 Changes behaviour in one area, or a default on some systems

  
status
blocked
 Waits on something else first

  
status
confirmed
 Reproduced or agreed on

  
status
duplicate
 Already reported or proposed

  
status
invalid
 Not a problem in this project

  
status
needs-info
 Waiting for more information from the reporter

  
status
wontfix
 Will not be worked on

  
testing wanted
Wanted: a test on other hardware (posts a test request in #test-requests)

  
ci: approved
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: needs approval
beanstalk: running
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!953
No description provided.