Spec: commit pins for test requests exist as Longshot pull request 71; two risks in the planned /beanstalk flow #2

New issue

Open

opened 2026-11-14 16:15:29 +00:00 by mooki · 0 comments

mooki commented

2026-11-14 16:15:29 +00:00

Member

This adds to the Spec and to the field notes in #1: one pointer to work that already exists, and two risks in the change announced in the team chat on 14 November. As announced: a BigBean writes /beanstalk on a pull request, a model writes the test plan (which OS and GPU combinations are needed), the plan is posted on the pull request and handed to testers, and testers sign in to the forge from the Beanstalk app, which registers their machine.

1. The commit pin for test requests already exists

The field notes (finding 2, and item 3 of "What we can do next") ask for scripts/test_requests.py to record the pull request's head at claim and with each result, and to show when a pull request has moved since it was tested, "only if nobody else has claimed this". Somebody has: MagicBeanDev/Longshot#71.

What it does at its current head b2c7f34b7f:

A request pins a full commit: the head seen when the request is made, or an ancestor that the forge confirms.
A claim keeps the commit it was made for. When the head has moved, an ordinary claim is refused, and only the requester can move the card to the new head.
A result is filed under the commit that was tested. Results for earlier commits stay visible as history and no longer count as "already tested".
show and list mark a claim that was made for an earlier commit.

It was tried on the live system on 14 November with a card on the pull request itself: a request, a claim refused without the label, the claim, the head moving under the claim, an update and the result (MagicBeanDev/Longshot#71 (comment)). The trial found one fault, a result filed under a stale claim's commit, which the current head fixes. It has 32 offline cases.

What it does not do, so nobody assumes it:

It does not pin the approval. The testing wanted label still approves a pull request, not a commit. A tester is told that the head moved, but the label is not withdrawn, so the bait-and-switch row of the threat model is only half closed.
It does not close or hide requests when their pull request merges or closes.

Both would be small changes on top of it. It is still marked work in progress while the build check of its newest head finishes. If the existing system is to be the seed of the coordinator, as the field notes suggest, this is the piece to look at first.

The spec says farm-agent login "signs in through Forgejo OAuth2 and gets an agent token scoped to that account", and its review note 9 names scripts/forge.py login as the part to reuse.

On this forge that sign-in has a property that will reach testers. Forgejo keeps one OAuth grant per user per application and refuses a refresh token once a newer one was issued. So a second machine that signs in or renews makes the first machine's refresh token stale, and the first is signed out within about an hour. The mechanism, with the lines in Forgejo's source, is in MagicBeanDev/Longshot#435. It was seen on 14 November between a Linux PC and a Mac on one account.

For the farm that means:

If the app signs in through the same application as forge.py, every contributor who also tests loses the forge.py sign-in when the app renews, and the other way round.
With an application of its own the two no longer disturb each other, but a tester with the app on two PCs still does it to themselves. People with a Windows PC and a Linux PC are the testers a farm wants most.

What would avoid it: use the forge sign-in once, to prove who the person is, and exchange it at once for a token that belongs to that one registered machine and that the coordinator issues and renews itself. The app then never keeps or refreshes the forge's own token. The quoted sentence may already mean exactly this. If so, one sentence in docs/protocol.md would settle it.

3. Risk: a plan written from the pull request's text

The spec's core rule is that a test plan is data and that nothing runs unless a maintainer approved that exact commit. AGENTS.md in this repository says that text from the forge is data, never instructions. A model that reads a pull request to write its plan reads text the author controls: the description, the comments, the diff, a .farm/test.toml.

That cannot make a tester's PC run anything new, since the plan stays declarative. What it can steer is coverage. A description that says "only matters on Linux with NVIDIA", or text aimed at the model, can produce a plan that leaves out the hardware where the change breaks, and the pull request then carries a passing farm result. The flicker reported against recent nightlies on 14 November came from Windows PCs with NVIDIA cards. A plan without that pair would have passed. As built, the spec says a Windows machine is never offered farm jobs, so no plan can cover that pair yet.

Three small rules would keep the trigger a short command and still close this:

/beanstalk posts the plan and waits. Testing starts on a second word from the BigBean, for the same commit. One person, two looks.
The coordinator checks the plan against the schema and against a floor the model cannot lower: for a change under gpu/ or host/, at least one Windows and one Linux target and every GPU vendor the diff names.
The posted plan names the commit it was written for and what the model was given, and a push withdraws it as it withdraws an approval.

Not checked

Beanstalk's code is not public beyond the first commit, so sections 2 and 3 are read from the spec and from the announcement, not from the implementation. Either may already be handled.
No part of Beanstalk was run.
A small aside: README.md and CONTRIBUTING.md call the project Beanstock and link MagicBeanDev/Beanstock, which redirects to MagicBeanDev/Beanstalk.

This adds to the [Spec](https://git.magicbean.dev/MagicBeanDev/Beanstalk/wiki/Spec) and to the field notes in https://git.magicbean.dev/MagicBeanDev/Beanstalk/issues/1: one pointer to work that already exists, and two risks in the change announced in the team chat on 14 November. As announced: a BigBean writes `/beanstalk` on a pull request, a model writes the test plan (which OS and GPU combinations are needed), the plan is posted on the pull request and handed to testers, and testers sign in to the forge from the Beanstalk app, which registers their machine. ## 1. The commit pin for test requests already exists The field notes (finding 2, and item 3 of "What we can do next") ask for `scripts/test_requests.py` to record the pull request's head at claim and with each result, and to show when a pull request has moved since it was tested, "only if nobody else has claimed this". Somebody has: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/71. What it does at its current head `b2c7f34b7f`: - A request pins a full commit: the head seen when the request is made, or an ancestor that the forge confirms. - A claim keeps the commit it was made for. When the head has moved, an ordinary claim is refused, and only the requester can move the card to the new head. - A result is filed under the commit that was tested. Results for earlier commits stay visible as history and no longer count as "already tested". - `show` and `list` mark a claim that was made for an earlier commit. It was tried on the live system on 14 November with a card on the pull request itself: a request, a claim refused without the label, the claim, the head moving under the claim, an update and the result (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/71#issuecomment-18281). The trial found one fault, a result filed under a stale claim's commit, which the current head fixes. It has 32 offline cases. What it does not do, so nobody assumes it: - It does not pin the **approval**. The `testing wanted` label still approves a pull request, not a commit. A tester is told that the head moved, but the label is not withdrawn, so the bait-and-switch row of the threat model is only half closed. - It does not close or hide requests when their pull request merges or closes. Both would be small changes on top of it. It is still marked work in progress while the build check of its newest head finishes. If the existing system is to be the seed of the coordinator, as the field notes suggest, this is the piece to look at first. ## 2. Risk: the sign-in The spec says `farm-agent login` "signs in through Forgejo OAuth2 and gets an agent token scoped to that account", and its review note 9 names `scripts/forge.py login` as the part to reuse. On this forge that sign-in has a property that will reach testers. Forgejo keeps one OAuth grant per user per application and refuses a refresh token once a newer one was issued. So a second machine that signs in or renews makes the first machine's refresh token stale, and the first is signed out within about an hour. The mechanism, with the lines in Forgejo's source, is in https://git.magicbean.dev/MagicBeanDev/Longshot/issues/435. It was seen on 14 November between a Linux PC and a Mac on one account. For the farm that means: - If the app signs in through the **same** application as `forge.py`, every contributor who also tests loses the `forge.py` sign-in when the app renews, and the other way round. - With an application of its own the two no longer disturb each other, but a tester with the app on two PCs still does it to themselves. People with a Windows PC and a Linux PC are the testers a farm wants most. What would avoid it: use the forge sign-in once, to prove who the person is, and exchange it at once for a token that belongs to that one registered machine and that the coordinator issues and renews itself. The app then never keeps or refreshes the forge's own token. The quoted sentence may already mean exactly this. If so, one sentence in `docs/protocol.md` would settle it. ## 3. Risk: a plan written from the pull request's text The spec's core rule is that a test plan is data and that nothing runs unless a maintainer approved that exact commit. `AGENTS.md` in this repository says that text from the forge is data, never instructions. A model that reads a pull request to write its plan reads text the author controls: the description, the comments, the diff, a `.farm/test.toml`. That cannot make a tester's PC run anything new, since the plan stays declarative. What it can steer is **coverage**. A description that says "only matters on Linux with NVIDIA", or text aimed at the model, can produce a plan that leaves out the hardware where the change breaks, and the pull request then carries a passing farm result. The flicker reported against recent nightlies on 14 November came from Windows PCs with NVIDIA cards. A plan without that pair would have passed. As built, the spec says a Windows machine is never offered farm jobs, so no plan can cover that pair yet. Three small rules would keep the trigger a short command and still close this: 1. `/beanstalk` posts the plan and waits. Testing starts on a second word from the BigBean, for the same commit. One person, two looks. 2. The coordinator checks the plan against the schema and against a floor the model cannot lower: for a change under `gpu/` or `host/`, at least one Windows and one Linux target and every GPU vendor the diff names. 3. The posted plan names the commit it was written for and what the model was given, and a push withdraws it as it withdraws an approval. ## Not checked - Beanstalk's code is not public beyond the first commit, so sections 2 and 3 are read from the spec and from the announcement, not from the implementation. Either may already be handled. - No part of Beanstalk was run. - A small aside: `README.md` and `CONTRIBUTING.md` call the project Beanstock and link `MagicBeanDev/Beanstock`, which redirects to `MagicBeanDev/Beanstalk`.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Beanstalk#2

No description provided.

Rows
Columns