feat(telemetry): performance statistics and failed builds, sent to the team #128
No reviewers
Labels
No labels
area/build
area/install
area/saves
area/tooling
area/ui
area/updates
ci: approved
good first issue
gpu/amd
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
platform/linux
platform/steam-deck
platform/windows
priority
critical
priority
high
priority
low
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/LongshotLauncher!128
Loading…
Reference in a new issue
No description provided.
Delete branch "gab9281/LongshotLauncher:feat/telemetry"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
When a build fails, is stopped or takes an hour, or players opt in to performance statistics, the team only hears about it if someone files an issue by hand. This lets the launcher send both, with the player's consent, to Longshot-api. Crash reports are no longer here: the game now builds and sends its own from its own crash window (MagicBeanDev/Longshot#983), so the launcher's
--crash-*commands are gone. This replaces #74 (performance statistics), which it contains, and #46 (the crash window drawn by the launcher), which the game's window supersedes.What changes
Send performance statistics (#74's two commits, rebased): the Settings tick, and Keep a copy of what was sent. Its conflicts with main were the settings next to it (Download updates early) and the translations; both sides are kept.
Send to the team on a failed build's page. The problem page a stopped build ends on gets a card with Send to the team, Show what is sent (the report exactly as it goes, as JSON) and, once sent, See it. It posts to
https://data-services.magicbean.dev/telemetry/build, where Longshot-api writes one comment per failure on MagicBeanDev/Longshot#987 and counts its repeats. Nothing is sent until the player presses it.~(BugReport.Scrub). No game file and no save.Core/BuildReports.cs), not through the version's tools as perf does, since a failed build may have failed in those tools.How every build ended, with the statistics ticked (
4821a4c). When a build ends, the launcher sends a summary on its own (auto): how it ended, each stage's progress and time, and the CPU. It covers completed, failed, the player's Stop (cancelled) and SIGTERM (terminated).make_game.py(exit 143): the build ends asterminated, with "make_game.py was stopped by a signal (SIGTERM) from outside the launcher." on its page, translated in the 12 catalogs./proc/cpuinfoon Linux, the registry on Windows.Send crash reports automatically (
71daab7): a Settings tick under the statistics, off by default. When it's on, the game sends a crash's report without showing its crash window.longshot_crash.ini(send=1) in its folder (Core/CrashReports.cs), only when the answer changed. The performance tick works the same way.The server side is live. Longshot-api's build statistics are deployed, and I validated them against
data-services.magicbean.dev:longshot_builds_total,longshot_build_stage_*_total);What is sent
Everything goes to
https://data-services.magicbean.dev/telemetry/buildwith no account, ID or path. Longshot-api caps each field and refuses a report that doesn't fit.auto)/telemetry/crash)/telemetry/perf)nx2/perf_report.pyat the next startBuild report (
Core/BuildReports.cs, at most 128 KiB)vlauncherversionchannelkindfirst build,update)outcomefailed,terminatedcompleted,failed,cancelled(Stop) orterminated(SIGTERM)autooscpupc15 GB, 16 threads, NVIDIA and AMD GPUsettingsmemory automatic, workers automatic, DLSS off, FSR 4 off, XeSS offwhyerrorstageslog~Stages (
stages, keyed by id)downloadreadliftcompilefinishshadersprogresssecondskeptWhat the team sees from summaries
longshot_builds_totaloutcome,stage(where it ended;donewhen completed),version,launcher,channel,os,cpulongshot_build_stage_seconds_total,_progress_total,_runs_totalstage,outcome,version,launcher,os,cpu,keptExample: a build stopped after 25 minutes, sent on its own
Checks
python -I tests/test_launcher_core.py: 1430 checks passed, withCrashReportChecks(longshot_crash.iniread by the helper's rule, written only when the answer changes),BuildReportChecks(what a report holds, the home folder taken out, the log's last lines, each server answer from a fake server) andBuildStatisticsChecks(stage times and progress, the CPU's name, the outcome and auto fields, an auto summary without log, why or error). Built with SDK 10.0.114, not the pinned 10.0.404, which isn't installed on this PC.python -I tests/test_release.py: OK.tools/screenshots'launcher-settingsscene, above. There's still no screenshot of the failed build's card.🤖 Generated with Claude Code
Possibly related, for context:
6eb4efefcd777ae2679efeat(telemetry): crash window, performance statistics and failed builds, sent to the teamto feat(telemetry): performance statistics and failed builds, sent to the team777ae2679e70afec7e4eA build report now says how the build ended (outcome: completed, failed, cancelled when the player pressed Stop, terminated when make_game.py got SIGTERM, exit code 143, or the launcher did while it built), this PC's CPU (/proc/cpuinfo's model name, Windows' ProcessorNameString) and each stage by its id with its progress (0..1), its seconds and whether it kept an earlier build's work: {"read": {"progress": 1, "seconds": 212.4, "kept": false}, ...}. BuildProgress times each stage from its start to the next one's, stopping where the build stopped. Send on the problem page sends it with the log, as before. With Settings' "Send performance statistics" ticked, every build's end is also sent on its own (auto: true), without the log, why or error line, so cancels and SIGTERM reach the team too; Longshot-api only counts those. The setting's text says so, in every catalog. Co-Authored-By: Claude Opus 5.5 <[email protected]>Reviewed the diff at
4821a4c, read against main (bfa2448), HumanBean's review of #74, and what Longshot-api already posted on Longshot#987.Run here (Windows 11, SDK 10.0.404):
dotnet build App/LongshotLauncher.csproj: 0 warnings, 0 errors.PerfChecksandBuildReportChecks, run early (right afterReleaseNotesChecks): they pass, including the home-folder scrub on a Windows path inside the JSON (BugReport.Scrubmatches the doubled\). The full core run stops atShellLinkChecks, the same as on main.translations.py: the same 26 missing texts per catalog as main.CI has not run the tests on this PR. Forgejo's "Compile and test" skips pull requests, and
4821a4chas no CircleCI status, so your Linux run is the only full one so far.Two things to fix before merge (inline):
BeforePlayreturns before--tick offruns, so the folder keepssend=1and the game keeps writing summaries. If the player ticks it again later, the summaries from the time it was off get sent.Also worth a look (inline):
BeforePlaystill blocks the window's thread for up to 5 s when Play is pressed (HumanBean's point 2 on #74).kindsends the button's verb:Build,Update to,Go back to(#119),Rebuild. It does not sendfirst build/update, as the description's table and example say.make_game.pyeven with the statistics unticked. Before, the build carried on. That's probably right, but it isn't a statistics change, so it's worth a line in the description. When the system stops the whole session,make_game.pygets the same SIGTERM, soExited(143)on the window's thread andTerminated()on the signal's thread can both pass thesummarizedcheck.make_gameruns again keeps timing from its first start, so stage seconds can add up to more than the build took.Merges: #59, #55 and #49 merge cleanly with this PR. #130 conflicts with it, in the 12 catalogs only, because both PRs append at the end. Putting the new texts after a related key in the middle of each file would avoid that.
@ -243,2 +246,4 @@if (OperatingSystem.IsWindows())foreach (var line in CarryDriverSettings(folder)) said = said == null ? line : said + " " + line;// (the performance statistics: the folder's tick made the player's answer, and what the last sessions left sent)PerfReports.BeforePlay(folder, Settings.SendPerformanceStatistics, Settings.KeepSentPerformanceStatistics, Tools);This runs on the window's thread: HomeViewModel.Play and the Ready page's Play call
Service.Playdirectly. So the first Play after the tick changes can freeze the window for up to 5 s while Python starts (longer on Windows with Defender scanning). HumanBean's point 2 on #74 is still open.@ -508,0 +514,4 @@/// The report Send on the problem page sends (BuildReports), with the build's last error line; auto: the statistics' summarypublic BuildReports.Failure Failure(string why, string outcome = BuildReports.Failed, bool auto = false) =>new(Job.Service.OwnTag, Job.Version, Job.Channel, Job.Verb.Trim(), BugReport.System(), PcLine(), SettingsLine(), why,Job.Verb.Trim()is the start button's English verb:Build,Update to,Go back to(#119) orRebuild. The description's table and example sayfirst build/update, and the comments on #987 so far came from the manual test. Sending an id made fromJob.Kind(plusJob.Older) would keep the label the same if the button's words change.@ -508,0 +544,4 @@}/// The launcher was sent SIGTERM while this build ran: its summary sent (waited for, briefly), then the build stoppedpublic void Terminated()Two things here.
This cancels the build even with the statistics unticked (Summarize returns early, then
cancel.Cancel()kills make_game's tree). Before, SIGTERM to the launcher leftmake_game.pyrunning. Probably the better behaviour, but worth saying in the description.This runs on the signal's thread. When the system stops the whole session,
make_game.pygets the same SIGTERM, andExited(143)runs on the window's thread at the same time.summarizedis a plain bool, so both can send.Interlocked.Exchangeon it would make it once only.@ -705,0 +786,4 @@/// What became of Send, under it ("" before it is pressed)public string SendSaid => answer switch{null => Tr.T("Sends the report below to the Longshot team, who see it on git.magicbean.dev. Nothing is sent until you press it."),Longshot#987 is public: it can be read without signing in. This sentence reads as "only the team sees it". Something like "Posts the report below publicly on Longshot's build-failures issue at git.magicbean.dev, for the team. Nothing is sent until you press it." would say where it goes, as #46's window did. The 12 catalogs would need it too.
@ -0,0 +41,4 @@/// The report as it is sent, and as the page shows it: JSON, the home folder as ~public static string Json(Failure f, string? home){IEnumerable<string> log = f.Auto ? [] : f.Log.Count > LogLines ? f.Log.Skip(f.Log.Count - LogLines) : f.Log;The lines aren't cut to the server's 300 characters here. One long line (a compiler command line, a Python traceback's repr) can push the report past 128 KiB. The player then gets "The server refused the report: HTTP 413" and has no way to send it. If each line (and
why/error) is cut to the server's limits here, Show what is sent also matches what the server keeps.@ -168,2 +177,4 @@current = stage;stage.State = StageState.Running;stage.Began ??= now;stage.Ended = null; // (make_game may run a stage again: its time runs on from its first start)A stage that is run again times from its first start, so its seconds include the stages between. Stage seconds can then add up to more than the build took, which
longshot_build_stage_seconds_totalwould add up. Minor, if make_game rarely does it.@ -0,0 +78,4 @@{var plan = Plan(on, keep, Ticked(folder), KeptSent(folder), Waiting(folder));if (plan.Tick == null && !plan.Send) return;if (ToolsFor(folder, tools) is not { } mine) return;With the tick turned off (
plan.Tick == false) and this version's tools gone (ToolsForis null), this returns, so the folder keepssend=1. The game keeps recording intoperf_pending/, and if the player ticks it again, what it recorded while off gets sent. The off case doesn't need perf_report.py: the launcher could writesend=0intolongshot_perf.iniitself and deleteperf_pending/(HumanBean's point 1 on #74).Re-review at
71daab7. I read the code only: no tests or builds run. I read it against madebyjordan's review at4821a4c, HumanBean's review of #74, and the game side in MagicBeanDev/Longshot#983 (host/hos_crash_dialog.c/.h,nx2/perf_report.py).For a player, this adds three things: a Send to the team card on a failed build's page; a summary of how each build ended, sent when performance statistics are ticked; and a new Send crash reports automatically tick, off by default.
The earlier review: nothing has changed
71daab7adds only the crash tick, its file and its texts. It changes none of the code madebyjordan commented on, so all of those points still stand. I won't repeat them:Core/PerfReports.cs:81, stillreturnbefore--tick off). This is also HumanBean's point 1 on #74.71daab7shows the fix is easy:CrashReports.BeforePlaywrites its ini itself, with no Python. The off case for perf can do the same: removelongshot_perf.iniand emptyperf_pending/andperf_sent/, asset_sending(folder, False)in 983 does.BuildStepViewModel.cs:789, "who see it on git.magicbean.dev").BeforePlaycan block the window's thread for up to 5 s.kindstill sends the button's verb, notfirst build/updateas the description says. Log lines,whyanderrorare not cut to 300 characters, so a report can go over 128 KiB and get a 413.summarizedis a plain bool, so the signal thread and the window's thread can both send. A stage that runs again keeps the time from its first start. The launcher's SIGTERM now stopsmake_game.pyeven with statistics off, and the description doesn't say so.New findings
Blocking: the crash tick has the same problem as Send, and it's worse here. The Settings text (
SettingsView.axaml:114) says the game "sends its report to the team". With the tick on, the helper posts the report to Longshot's issue 81 (MagicBeanDev/Longshot#81), which anyone can read without signing in, and the player never sees the report first. Send at least shows the report on the page before it goes. For consent, this text has to say where the report goes, for example "...posts its report publicly on Longshot's crash issue at git.magicbean.dev, for the team: ...". The 12 catalogs would need it too. Whatever wording is chosen for Send should be used here as well.The signal thread reads the window's data (
BuildStepViewModel.cs:547).Terminated()runs on the signal's thread. It callsFailure(), andFailure()enumerateslog.Last(200)andprogress.Times()whileLine/ErrorLineare still being added throughUi.Post. The JSON then drops the log anyway, becauseautois true. IfBuildLogthrows "collection was modified", the exception goes unhandled in the signal handler: the summary isn't sent andcancel.Cancel()never runs. Building the auto summary without reading the log, or postingTerminatedto the window's thread and waiting for it there, would avoid this. This goes with thesummarizedrace above.The description gets the
--crash-*commands wrong. It says the launcher's--crash-*commands "are gone". They were never on the launcher'smain(I greppedmain@origin), and the game on Longshot'smaindoesn't call the launcher. They existed only in #46. Nothing is removed, so there's no compatibility to keep. The description should say the commands aren't added.71daab7, Send crash reports automaticallyThe code is correct:
BeforePlaywrites before the game starts, and only when the answer differs.send=0. It doesn't delete the file, and the helper readssend=0as "ask". Off and never chosen, it writes no file.PlayFolder'sfolder), the same folder the helper reads (hos_crash_record_dir(), the game folder).Trim() == "send=1"per line. The helper skips spaces and tabs beforesend=1and spaces, tabs and\rafter it. The launcher only ever writessend=0orsend=1.--play,--launch, and so the shortcuts) goes throughPlayFolder.The check in
PerfChecks.cscovers all of this. PuttingCrashReportChecksinPerfChecks.csis odd, but that's a nit.Contract with Longshot 983
The contract is OK:
longshot_crash.iniwithsend=1in the game folder.perf_report.py <folder> --tick on|off [--keep-sent off]and<folder> --send.longshot_perf.iniwithsend=1/keep_sent=0, plusperf_pending/andperf_sent/.Both sides use the same names, keys, folder and arguments.
perf_report.pyis new in 983 and isn't on Longshot'smain. Until 983 ships in a version, no game folder has it, so the performance tick sends no game summaries. The build summaries and Send don't depend on 983.Merge order: they can merge in either order without breaking anything. 983 first is better: until a game version has 983, the two ticks promise something that doesn't happen yet.
Overlaps
Testing
The author ran the core checks on Linux. madebyjordan built it on Windows and ran PerfChecks and BuildReportChecks. Nobody has posted a run of the crash tick with a 983 game that actually crashes. There's still no screenshot of the failed build's card.
@ -508,0 +544,4 @@}/// The launcher was sent SIGTERM while this build ran: its summary sent (waited for, briefly), then the build stoppedpublic void Terminated()This runs on the signal's thread.
Summarize->Failure()enumerateslog.Last(200)andprogress.Times()while the window's thread is still adding lines (Ui.Post(() => Line(line))). The auto JSON drops the log anyway. If that enumeration throws, the exception goes unhandled here, so the summary isn't sent andcancel.Cancel()is skipped. Build the auto summary without the log, or post this to the window's thread and wait for it (which also settles thesummarizedrace).@ -102,0 +111,4 @@<StackPanel Spacing="4"><CheckBox Classes="tick" Content="{c:T 'Send crash reports automatically'}" IsChecked="{Binding SendCrashReportsAutomatically}"/><TextBlock FontSize="12" Foreground="{StaticResource InkSoft}" TextWrapping="Wrap" Margin="28,0,0,0"Text="{c:T 'After a crash, the game sends its report to the team without showing its crash window: the version, your GPU, CPU, OS and settings, and the end of the log, with no paths or names. If it cannot be sent, the window shows as usual.'}"/>With this tick on, the report is posted without the player seeing it, to Longshot's issue 81 (MagicBeanDev/Longshot#81), which anyone can read. "Sends its report to the team" reads as private. Say where it goes, e.g. "...posts its report publicly on Longshot's crash issue at git.magicbean.dev, for the team: ...", in the 12 catalogs too, with the same wording as Send's card.
@ -0,0 +78,4 @@{var plan = Plan(on, keep, Ticked(folder), KeptSent(folder), Waiting(folder));if (plan.Tick == null && !plan.Send) return;if (ToolsFor(folder, tools) is not { } mine) return;Still the blocker from the last review.
CrashReports.BeforePlayin this PR now shows the way: for the off case, the launcher can removelongshot_perf.iniand emptyperf_pending/andperf_sent/itself (what 983'sset_sending(folder, False)does), without needing the version's tools.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.