feat: semi-live logs of running jobs #77
No reviewers
Labels
No labels
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Beanstalk!77
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Beanstalk:feat/live-logs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
A job that seemed stuck (#1152 on AstraLol's PC: at build-pr for 45+ minutes, where its PR builds usually take 2-3) could not be looked into from the farm. Nothing of the tester's log reached the farm until the report.
What changed
LiveLogcollects the job's log lines (the agent's own, plus the build's and the game's, asagent.loghas them). Each heartbeat (every few seconds) carries the newest ones inlog, at most 100 lines of 300 characters, scrubbed as a report is (Scrubber.ForThisPc). It holds at most 1000 lines; if heartbeats fail for a while, the oldest go and a line says how many.scheduler.log_lineschecks them andappend_logadds them tojobs.live_logwhile that lease holds, keeping the last 64 KB, cut at a line. Migration 0018 addsjobs.live_logandlive_log_at. The log is emptied on a new lease and kept after the job ends./jobs/<id>/logshows the lines and reloads every 15 s while the job runs. PR and run pages link it from each job that has started.log).Tests
test_heartbeat_lines_make_the_live_log(append, another agent's lines refused, the 64 KB cut, reset on a new lease, bad bodies). The e2e flow sends a heartbeat with a line, reads the page (escaped, reloading) and the PR page's link, and gets 400 and 404 where it should. Suite: 541 passed, 1 skipped.