feat(planner): a security and sanity check of the diff before the plan #50
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk!50
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Beanstalk:feat/plan-security-check"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
Today, once a BigBean approves a commit, the planner writes a plan and testers' PCs build and run the code. Before that, the farm should take one quick look at the diff, so a commit that is plainly malicious or broken never reaches a tester's PC, even when a BigBean missed it.
What changed
planner.screen: before each PR plan, one call to a quick model checks the commit. The default isgpt-6-lunaat low effort;BEANSTALK_SCREEN_MODELandBEANSTALK_SCREEN_EFFORTchange it.{"safe": bool, "reason": str}, through the backend's structured output: a strict JSON schema namedbeanstalk_screen(complete()now takes the format's name; plans staybeanstalk_plan). The parser still accepts fenced JSON in case the backend ever drops the schema, and an unreadable answer counts as no verdict.What it holds: only what is plainly bad. That means:
Style, ordinary bugs and slow code all pass, and so does anything it isn't sure about.
A held commit: gets no plan and no jobs.
plan_errorstarts withsecurity check:, andbeanstalk/approvalfails with "held by the security check: see the farm's comment". The comment says:The reason has
@defused, backticks replaced and a 600-character cap.Checking again: a new push or
@Beanstalk plan <words>starts a new round, and the check runs again, with those words as the maintainer's.When the check can't answer (an error, 2 minutes with no answer, or no verdict), the farm plans anyway, since a BigBean approved the commit already. With AI planning off there is no check.
Setup:
create_app(..., screen_llm=)takes the check's model; in production it comes from the same ChatGPT sign-in as the planner.Farm.screen_llmis new.Docs:
docs/using.mdhas a section on the check; the setting is indocs/deploy.mdanddeploy/compose.example.yml.Testing
@Beanstalk plan <words>passes those words to the check; it then answers safe, and the commit is planned and started.json_schemaformat namedbeanstalk_screenfor the check andbeanstalk_planfor plans.gpt-6-lunacall. Nothing shows yet how often it holds ordinary PRs. After deploying, the coordinator log line "the security check held …" is worth watching for false holds. Manual runs (#48) don't get the check yet.Deploy
No migration. With the sign-in present, the check is on by default; set
BEANSTALK_SCREEN_MODELto another model to change it. Each PR plan costs one more quick call.Before a PR commit's plan is written, one call to a quick model (gpt-6-luna at low effort by default; BEANSTALK_SCREEN_MODEL/_EFFORT) reads the diff, changed files, title, body, latest comments and the maintainer's words, all as fenced untrusted data, and answers {safe, reason}. It holds only what is plainly bad: exfiltration, network access, sandbox probing, miners, destruction outside the build, obfuscated or prebuilt payloads, tampering with the farm's results or scripts, garbage diffs. Held: no plan, no jobs; plan_error starts with "security check: "; the comment says "Held by the security check" with the reason (no @mentions, no backticks, capped) and how to go on; beanstalk/approval fails with "held by the security check". A new push, or a maintainer's @Beanstalk plan <why it is fine>, checks again with those words. When the check can't answer (error, 2-minute timeout, no verdict), the farm plans anyway: a BigBean approved the commit already. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]>