feat(scheduler): a tester tries a failed job again after a wait, and after its app updates #58
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk!58
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Beanstalk:feat/retry-failed-jobs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
A tester that failed a job, other than a passing failure, was never offered that job again. When the failure was the agent's own setup bug (#56 fixed four of them), the job stayed stuck after the fix reached that PC. Each such job needed a re-plan by hand. Six queued jobs are stuck like this now.
What changed
scheduler.failed_out: a tester that failed a job is offered it again afterRETRY_WAIT(15 minutes),RETRY_TRIES(2) times in all. After that, it isn't offered the job again until its app is another version, which starts its tries again.requeuerecords the tries, the app version and the time per tester injobs.agent_fails(migration 0015).failed_agentsstays as it was, and 3 different testers failing a job still make it an error.done_reason) now says when the PC tries again: "it failed this job: it tries again from 11:14 UTC", or "… 2 times: it tries again once its app updates".docs/protocol.mdanddocs/architecture.mddescribe the new rule.Testing
test_a_failed_pc_tries_again_after_a_wait_and_after_its_app_updates: two tries with the wait between them, then left out, then offered again after a version change with its count starting at 1.test_failed_out_reads_old_and_odd_records: no record, a bad time, a bad count.test_agent_fails_from_failed_agents: the migration over a 0014 database, run twice.