Idea: keep the field data usable once players send it: label limits that don't swallow releases, counters that survive a restart, alerts on failed pushes, and a longer retention #1266
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#1266
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The idea
Longshot-api (main 486d6da5050d1b63b3efbce8dc29b3b4000c343d) and its deploy config are safe against a flood, but four things will make the data worse than it looks once real players send reports. All four are read from the code and the live metrics (30 days, read 16 Nov 2026; everything counted so far is the developers' own test and dirty builds).
other. Each label keeps a fixed number of distinct values and everything after that is counted asother(pkg/labels/labels.go). The limits: performance labels 30 each by default,gpuandcpu80,driver60,kernel40,map250 (var maxValues = map[string]int{"map": 250, "gpu": 80, "cpu": 80, "driver": 60, "kernel": 40}signature,versionandwhere, and 1,000 series in all (MaxSeries = 1000versionvalues (one is0.0.0-api-test, three are-dirty) and the crash data 10 (0.0.0-api-test,0.0.0-livetest,0.0.0-windowtestand dirty ones). Every nightly is a newversion, so 30 is reached within weeks, and then the version of a new release would beother: the one dimension "did the fix work" needs. The state is in memory, so a restart happens to reset it (the metrics show 11 different pod names in 30 days), but that is luck, not design.release=trueversion, and do not letrelease=falseor-dirtyversions use up the slots (a separate small cap for them); a window instead of a lifetime for the limiter (the retention is 30 days); in the crash series use the same classed labels the performance series have (gpu_vendor,driveras major.minor,ram_class,osfamily and kernel major.minor) instead of the rawos,gpuandcpustrings, which carry the exact kernel, desktop session, driver and RAM and are one value per machine.crashes --all-buildsshows 13 reports. The crash-bot comments on issue 81 say "seen" 18 times in total, and the relay's own counters say 18 (new10 plusrepeat8). By signature: the live test 4 against 1, the API test 4 against 2. The crash series are counters in memory, pushed to Prometheus as OTLP sums, and a restart starts them at 0; andlongshot_crash_reports_total{result="prometheus_error"}is 40 (the push to Prometheus failed that often) againstnew10. The comment's signed tally is right (the relay reloads it after a restart,telemetry/crash/relay.goload), but the series are not rebuilt from it.deploy/grafana/12-alerting.yamlis "Magic Bean service down". A rule onincrease(longshot_crash_reports_total{result=~"error|prometheus_error|refused_new"}[15m]) > 0would have said so.MAX_NEW_CRASHES_PER_HOUR,telemetry/crash/relay.goline 25) and answers 429 "too many new crashes this hour, try again later" (line 129), after parsing the report. The report is dropped:refused_newcounts it, the series do not. The game's window shows "the crash server is busy ... try again in a minute" (host/hos_crash_dialog.hline 1062) for a limit that can last an hour. A bad release (newwhereandversionfor each build) is exactly when more than 10 new signatures arrive in an hour. Suggested: still count a refused report into the series (it is parsed already), and say "try again later" in the window.--storage.tsdb.retention.time=30d(deploy/prometheus/20-deployment.yamlline 34, in the deploy repo). Releases are weeks apart; a regression that shows between two of them is out of range by the time the second has players. The data is small: about 57 series per performance label set and at most 1,000 sets, so roughly 80,000 series at the cap (an estimate: today's 11 sets make 627 series), which should fit in the 1 GiB limit. Suggested: 180 days or a year, keeping the 8 GB size cap.Anything else
deploy/telemetry/40-ingress.yamllines 37 and 39: 6 a minute, burst 6, for the performance route) are lower than what the sender posts at once (12 a start); that part is in the game-side issue for the sender.Possibly related, for context: