feat(scheduler): a tester tries a failed job again after a wait, and after its app updates #58

Merged
HumanBean merged 1 commit from HumanBean/Beanstalk:feat/retry-failed-jobs into main 2026-11-16 11:07:45 +00:00
Owner

Why

A tester that failed a job, other than a passing failure, was never offered that job again. When the failure was the agent's own setup bug (#56 fixed four of them), the job stayed stuck after the fix reached that PC. Each such job needed a re-plan by hand. Six queued jobs are stuck like this now.

What changed

  • scheduler.failed_out: a tester that failed a job is offered it again after RETRY_WAIT (15 minutes), RETRY_TRIES (2) times in all. After that, it isn't offered the job again until its app is another version, which starts its tries again.
  • requeue records the tries, the app version and the time per tester in jobs.agent_fails (migration 0015). failed_agents stays as it was, and 3 different testers failing a job still make it an error.
  • Migration 0015 gives each existing failure all its tries at the tester's present app version. Those jobs are offered again once that tester's app updates (the nightly from #56), and not before. The migration is safe to run again.
  • The tester's page (done_reason) now says when the PC tries again: "it failed this job: it tries again from 11:14 UTC", or "… 2 times: it tries again once its app updates".
  • docs/protocol.md and docs/architecture.md describe the new rule.

Testing

  • Coordinator suite: 516 passed, 1 skipped.
  • New tests:
    • test_a_failed_pc_tries_again_after_a_wait_and_after_its_app_updates: two tries with the wait between them, then left out, then offered again after a version change with its count starting at 1.
    • test_failed_out_reads_old_and_odd_records: no record, a bad time, a bad count.
    • test_agent_fails_from_failed_agents: the migration over a 0014 database, run twice.
## Why A tester that failed a job, other than a passing failure, was never offered that job again. When the failure was the agent's own setup bug (#56 fixed four of them), the job stayed stuck after the fix reached that PC. Each such job needed a re-plan by hand. Six queued jobs are stuck like this now. ## What changed - `scheduler.failed_out`: a tester that failed a job is offered it again after `RETRY_WAIT` (15 minutes), `RETRY_TRIES` (2) times in all. After that, it isn't offered the job again until its app is another version, which starts its tries again. - `requeue` records the tries, the app version and the time per tester in `jobs.agent_fails` (migration 0015). `failed_agents` stays as it was, and 3 different testers failing a job still make it an error. - Migration 0015 gives each existing failure all its tries at the tester's present app version. Those jobs are offered again once that tester's app updates (the nightly from #56), and not before. The migration is safe to run again. - The tester's page (`done_reason`) now says when the PC tries again: "it failed this job: it tries again from 11:14 UTC", or "… 2 times: it tries again once its app updates". - `docs/protocol.md` and `docs/architecture.md` describe the new rule. ## Testing - Coordinator suite: 516 passed, 1 skipped. - New tests: - `test_a_failed_pc_tries_again_after_a_wait_and_after_its_app_updates`: two tries with the wait between them, then left out, then offered again after a version change with its count starting at 1. - `test_failed_out_reads_old_and_odd_records`: no record, a bad time, a bad count. - `test_agent_fails_from_failed_agents`: the migration over a 0014 database, run twice.
feat(scheduler): a tester tries a failed job again after a wait, and after its app updates
Some checks are pending
CI / Coordinator tests (pull_request) Waiting to run
CI / Agent build and checks (pull_request) Waiting to run
CI / Coordinator image (pull_request) Waiting to run
CI / Conventional Commit title (pull_request) Successful in 8s
CI / Workflow lint (actionlint) (pull_request) Successful in 8s
083b856fa7
A tester that failed a job (not passing) was never offered it again, so a
job stuck on a setup bug stayed stuck after the bug was fixed. Now each
tester gets 2 tries at a job, 15 minutes apart, and its tries start again
once its app is another version (jobs.agent_fails: tries, app version and
time per tester, migration 0015). Failures from before count as all their
tries at the tester's present version, so those jobs are offered again
once that tester's app updates. 3 different testers failing a job still
make it an error. The tester's page says when it tries again.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean deleted branch feat/retry-failed-jobs 2026-11-16 11:07:45 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Beanstalk!58
No description provided.