Idea: keep the field data usable once players send it: label limits that don't swallow releases, counters that survive a restart, alerts on failed pushes, and a longer retention #1266

Open
opened 2026-11-16 23:27:17 +00:00 by mooki · 1 comment
Member

The idea

Longshot-api (main 486d6da5050d1b63b3efbce8dc29b3b4000c343d) and its deploy config are safe against a flood, but four things will make the data worse than it looks once real players send reports. All four are read from the code and the live metrics (30 days, read 16 Nov 2026; everything counted so far is the developers' own test and dirty builds).

  1. A release's version can be folded into other. Each label keeps a fixed number of distinct values and everything after that is counted as other (pkg/labels/labels.go). The limits: performance labels 30 each by default, gpu and cpu 80, driver 60, kernel 40, map 250 (
    var maxValues = map[string]int{"map": 250, "gpu": 80, "cpu": 80, "driver": 60, "kernel": 40}
    , line 24); crash labels 50 each, including signature, version and where, and 1,000 series in all (, line 21). The first values to arrive win. Today they are test and dirty builds: the performance data already has 6 version values (one is 0.0.0-api-test, three are -dirty) and the crash data 10 (0.0.0-api-test, 0.0.0-livetest, 0.0.0-windowtest and dirty ones). Every nightly is a new version, so 30 is reached within weeks, and then the version of a new release would be other: the one dimension "did the fix work" needs. The state is in memory, so a restart happens to reset it (the metrics show 11 different pod names in 30 days), but that is luck, not design.
    • Suggested: never fold a release=true version, and do not let release=false or -dirty versions use up the slots (a separate small cap for them); a window instead of a lifetime for the limiter (the retention is 30 days); in the crash series use the same classed labels the performance series have (gpu_vendor, driver as major.minor, ram_class, os family and kernel major.minor) instead of the raw os, gpu and cpu strings, which carry the exact kernel, desktop session, driver and RAM and are one value per machine.
  2. The crash counts in Prometheus are lower than what the relay accepted. The field-data tool's crashes --all-builds shows 13 reports. The crash-bot comments on issue 81 say "seen" 18 times in total, and the relay's own counters say 18 (new 10 plus repeat 8). By signature: the live test 4 against 1, the API test 4 against 2. The crash series are counters in memory, pushed to Prometheus as OTLP sums, and a restart starts them at 0; and longshot_crash_reports_total{result="prometheus_error"} is 40 (the push to Prometheus failed that often) against new 10. The comment's signed tally is right (the relay reloads it after a restart, telemetry/crash/relay.go load), but the series are not rebuilt from it.
    • Suggested: on start, seed the series from the tallies the relay already reloads; and say in the docs (the field-data skill already does) that the comment is the reference.
    • Nothing alerts on it: the only alert rule in deploy/grafana/12-alerting.yaml is "Magic Bean service down". A rule on increase(longshot_crash_reports_total{result=~"error|prometheus_error|refused_new"}[15m]) > 0 would have said so.
  3. A busy day costs reports, and the player is told "a minute". The relay posts at most 10 new crash comments an hour (MAX_NEW_CRASHES_PER_HOUR, telemetry/crash/relay.go line 25) and answers 429 "too many new crashes this hour, try again later" (line 129), after parsing the report. The report is dropped: refused_new counts it, the series do not. The game's window shows "the crash server is busy ... try again in a minute" (host/hos_crash_dialog.h line 1062) for a limit that can last an hour. A bad release (new where and version for each build) is exactly when more than 10 new signatures arrive in an hour. Suggested: still count a refused report into the series (it is parsed already), and say "try again later" in the window.
  4. 30 days of retention is short for comparing releases. --storage.tsdb.retention.time=30d (deploy/prometheus/20-deployment.yaml line 34, in the deploy repo). Releases are weeks apart; a regression that shows between two of them is out of range by the time the second has players. The data is small: about 57 series per performance label set and at most 1,000 sets, so roughly 80,000 series at the cap (an estimate: today's 11 sets make 627 series), which should fit in the 1 GiB limit. Suggested: 180 days or a year, keeping the 8 GB size cap.

Anything else

  • The ingress limits (deploy/telemetry/40-ingress.yaml lines 37 and 39: 6 a minute, burst 6, for the performance route) are lower than what the sender posts at once (12 a start); that part is in the game-side issue for the sender.
  • The privacy page for the statistics says "A summary carries no ID". True for the payload; the relay still sees an IP address at the ingress, and the crash comment is public. A sentence on who runs the service, what the access logs hold and for how long would complete it (this was asked in the review of #206).
  • None of this is urgent before players send data; all of it is cheaper to do before the first release that sends.
### The idea Longshot-api (main 486d6da5050d1b63b3efbce8dc29b3b4000c343d) and its deploy config are safe against a flood, but four things will make the data worse than it looks once real players send reports. All four are read from the code and the live metrics (30 days, read 16 Nov 2026; everything counted so far is the developers' own test and dirty builds). 1. **A release's version can be folded into `other`.** Each label keeps a fixed number of distinct values and everything after that is counted as `other` (`pkg/labels/labels.go`). The limits: performance labels 30 each by default, `gpu` and `cpu` 80, `driver` 60, `kernel` 40, `map` 250 (https://git.magicbean.dev/MagicBeanDev/DataServicesAPI/src/commit/486d6da5050d1b63b3efbce8dc29b3b4000c343d/telemetry/perf/totals.go#L22, line 24); crash labels 50 each, including `signature`, `version` and `where`, and 1,000 series in all (https://git.magicbean.dev/MagicBeanDev/DataServicesAPI/src/commit/486d6da5050d1b63b3efbce8dc29b3b4000c343d/telemetry/crash/series.go#L19, line 21). The first values to arrive win. Today they are test and dirty builds: the performance data already has 6 `version` values (one is `0.0.0-api-test`, three are `-dirty`) and the crash data 10 (`0.0.0-api-test`, `0.0.0-livetest`, `0.0.0-windowtest` and dirty ones). Every nightly is a new `version`, so 30 is reached within weeks, and then the version of a new release would be `other`: the one dimension "did the fix work" needs. The state is in memory, so a restart happens to reset it (the metrics show 11 different pod names in 30 days), but that is luck, not design. - Suggested: never fold a `release=true` version, and do not let `release=false` or `-dirty` versions use up the slots (a separate small cap for them); a window instead of a lifetime for the limiter (the retention is 30 days); in the crash series use the same classed labels the performance series have (`gpu_vendor`, `driver` as major.minor, `ram_class`, `os` family and kernel major.minor) instead of the raw `os`, `gpu` and `cpu` strings, which carry the exact kernel, desktop session, driver and RAM and are one value per machine. 2. **The crash counts in Prometheus are lower than what the relay accepted.** The field-data tool's `crashes --all-builds` shows 13 reports. The crash-bot comments on issue 81 say "seen" 18 times in total, and the relay's own counters say 18 (`new` 10 plus `repeat` 8). By signature: the live test 4 against 1, the API test 4 against 2. The crash series are counters in memory, pushed to Prometheus as OTLP sums, and a restart starts them at 0; and `longshot_crash_reports_total{result="prometheus_error"}` is 40 (the push to Prometheus failed that often) against `new` 10. The comment's signed tally is right (the relay reloads it after a restart, `telemetry/crash/relay.go` `load`), but the series are not rebuilt from it. - Suggested: on start, seed the series from the tallies the relay already reloads; and say in the docs (the field-data skill already does) that the comment is the reference. - Nothing alerts on it: the only alert rule in `deploy/grafana/12-alerting.yaml` is "Magic Bean service down". A rule on `increase(longshot_crash_reports_total{result=~"error|prometheus_error|refused_new"}[15m]) > 0` would have said so. 3. **A busy day costs reports, and the player is told "a minute".** The relay posts at most 10 new crash comments an hour (`MAX_NEW_CRASHES_PER_HOUR`, `telemetry/crash/relay.go` line 25) and answers 429 "too many new crashes this hour, try again later" (line 129), after parsing the report. The report is dropped: `refused_new` counts it, the series do not. The game's window shows "the crash server is busy ... try again in a minute" (`host/hos_crash_dialog.h` line 1062) for a limit that can last an hour. A bad release (new `where` and `version` for each build) is exactly when more than 10 new signatures arrive in an hour. Suggested: still count a refused report into the series (it is parsed already), and say "try again later" in the window. 4. **30 days of retention is short for comparing releases.** `--storage.tsdb.retention.time=30d` (`deploy/prometheus/20-deployment.yaml` line 34, in the deploy repo). Releases are weeks apart; a regression that shows between two of them is out of range by the time the second has players. The data is small: about 57 series per performance label set and at most 1,000 sets, so roughly 80,000 series at the cap (an estimate: today's 11 sets make 627 series), which should fit in the 1 GiB limit. Suggested: 180 days or a year, keeping the 8 GB size cap. ### Anything else - The ingress limits (`deploy/telemetry/40-ingress.yaml` lines 37 and 39: 6 a minute, burst 6, for the performance route) are lower than what the sender posts at once (12 a start); that part is in the game-side issue for the sender. - The privacy page for the statistics says "A summary carries no ID". True for the payload; the relay still sees an IP address at the ingress, and the crash comment is public. A sentence on who runs the service, what the access logs hold and for how long would complete it (this was asked in the review of https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/206). - None of this is urgent before players send data; all of it is cheaper to do before the first release that sends.
Collaborator

Possibly related, for context:

  • #81 feat: crash reports sent from the game to a bot (with consent) — Related crash-report telemetry feature and relay; this proposal concerns reliability and metrics around its data.
Possibly related, for context: - #81 feat: crash reports sent from the game to a bot (with consent) — Related crash-report telemetry feature and relay; this proposal concerns reliability and metrics around its data.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1266
No description provided.