feat(perf): opt-in performance statistics, sent from the launcher to the team's Grafana #206
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!206
Loading…
Reference in a new issue
No description provided.
Delete branch "gab9281/Longshot:feat/perf-metrics"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Linux, RTX 3060 Laptop · Needs: Windows, AMD and Intel GPUs, 32 GB of RAM
Why
Players report stutter, cutscenes that play badly and uneven frame pacing, and the team can't tell which GPUs, drivers, CPUs, systems, renderers or maps they come from. With this PR, a player who ticks Send performance statistics in the Longshot Launcher (off by default; LongshotLauncher#74) has the game count how its frames went on each map, and the launcher sends those counts to the team's Grafana, where each of these can be filtered on its own.
What changed
With
send=1in the game folder'slongshot_perf.ini, which the launcher's tick writes, the game keeps a short summary of each map's frames and writes it toperf_pending/every 5 minutes and when the window closes. Before the next start, the launcher sends what is waiting withnx2/perf_report.pyto Longshot-api's telemetry service (/telemetry/perf), which counts it for Grafana. With the tick off, nothing is measured, written or sent.gpu/vk/perf_report.handperf_report.cpp. For each map, the summary holds:RX_VK_PERF_INTERVAL, 60 to 840 s) and when the window closes (mkwvk_close_askedinvk_main.cpp, sinceExitProcessis_exiton Linux andatexitnever runs); once on a lost device (vk_replay.cppdevice_lost). The game reads the tick again before each summary and stops when it was taken off during play. At most 48 summaries wait on disk.vk_fps.cpp: while the summary is on, the FPS counter takes its Breakdown's timings and passes each frame toperf_frame()(add_frame: 6-16 ns). With the tick off, the only cost is one load ofperf_measureper frame.MkwVkHost::perf_info(MKWVK_API_VERSION23):hos_main.chos_perf_info;nn_video.ccounts each picture as the game takes it, so a movie still playing at a summary is counted;dav1d_av1.ccounts a decoder that didn't start.perf_report.py, run by the launcher:--tick on|off [--keep-sent off]: writessend=1(andkeep_sent=0when the launcher's Keep a copy of what was sent is off, which also deletes the copies), or deletes the file, what is waiting and the copies of what was sent;--send: oldest first, with the version and the release flag filled in (as the crash report has them). A summary the service took or refused is deleted; one it couldn't reach, or a 429/5xx, stays for next time. Each one taken is kept, exactly as sent, inperf_sent/(the newest 48), unless the copies are off, so a player or tester can read what went to the team;Relay: one TLS handshake for them all; through a system proxy, urllib's one request each, as before), at most 12 a start, and a service that answers busy (429, 5xx) left alone until itsRetry-After, else an hour, doubled each time it is busy again up to a day (perf_pending/.next). A kept connection the service let go is tried once more on a new one: the summary never arrived, so none is sent twice;LONGSHOT_PERF_MAX_SENDS,LONGSHOT_PERF_TIMEOUT,LONGSHOT_PERF_BACKOFF,LONGSHOT_PERF_KEEP_SENT,LONGSHOT_PERF_URL(Flags and settings);logs/, as main keeps them.addresses.ini/addresses.py: the service's address (perf_reports = https://api.magicbean.gmatte.xyz/telemetry/perf), out of the code. The file is the same as #141's, crash addresses included, so whichever merges second has no conflict.docs/performance-statistics.md(what is sent and never sent, how the parts fit, the switches, the cost), a README line,longshot_perf.iniandRX_VK_PERF_INTERVALindocs/development.md, and the measured runs indocs/tested/.tests/test_perf_report.cpp(the summary's functions, the interval's rule);tests/test_perf_report.py(the launcher side against a stand-in service over HTTP/1.1:--tick,--keep-sent, one connection, a dropped one, the share a start,Retry-Afterand the doubling, the switches; and the wiring in the game, the pre-map names included); the video tests' fixtures get the new counter.The telemetry service and the Grafana dashboard ("Longshot performance", with a perf index by map and graphics card) are deployed (Longshot-api, Bean-Ops).
Measurements
Verdict: no cost that stands out from the noise on this PC. With the tick off, this PR was 2% behind main in the medians (124.4 against 127.0 fps), but main's own runs spread over 15 fps (121.5 to 136.7) as the PC slowed from round to round, and in the last round the tick off was ahead of main. The tick on cost about as much as the tick off (122.9 fps, 6.75 against 6.78 ms). The earlier A/Bs on main
fbd42024and6c9edffe, with the same method, found the same: within a percent or two, either way.Hardware: NVIDIA GeForce RTX 3060 Laptop GPU (6 GB), NVIDIA 620.70.06, AMD Ryzen 7 6800H, 15 GB, CachyOS (Linux 7.3.4), KDE Wayland
Scene: the game's start, the intro and title (no save on this PC), output 2048x1152, preset Auto (DLSS), no frame generation,
HOS_FPS_MAX=1000(uncapped),HOS_AUDIO=0Method: 75 s a run, started through CachyOS's
game-performance, withRX_VK_FRAME_LOG, ended with the control bridge's quit (the window's close path). Frames from 20 s on. Main, then this PR with the tick off, then on, 3 rounds interleaved; each value is the median of the 3 runs.How to reproduce: two runtimes on one base (
scripts/dev_packs.py rt), one with main'slongshotandlongshot_vk.so, one with this PR's;HOS_FPS_MAX=1000;longshot_perf.iniwithsend=1for "on"; quit withpython -I scripts/longshot_ctl.py quit.e090cc489981a39a9981a39aReading: each round ran main first, and every side got slower each round (main 136.7, 127.0, 121.5), with the PC busy (another app using most of a core) and swapping, so the order favours main. The per-frame work this PR adds is
add_frame(6-16 ns,tests/test_perf_report.cpp) with the tick on and one load with it off: far below what these runs can resolve. The earlier sets: main6c9edffe135.6, off 136.3, on 134.4 fps; mainfbd42024150.9, off 149.1, on 149.1 (docs/tested/). A quieter PC (test requests below) can settle a 1-2% difference.Since the measured
9981a39a,a401573bchanges nothing a frame does but compare against the interval read once at the start (period) instead of a constant; the rest is the launcher's sending, after the game. The numbers above still hold.Flags and settings
Two ini keys, written by the launcher's settings, and six switches with polite defaults that need no setting (
docs/performance-statistics.md, "Switches"). The sender's are read from the launcher's environment, the game's from its environment orlongshot_env.txt.longshot_perf.inisend=1(the launcher's Send performance statistics)longshot_perf.inikeep_sent=0(the launcher's Keep a copy of what was sent)perf_sent/RX_VK_PERF_INTERVAL(game)LONGSHOT_PERF_MAX_SENDSLONGSHOT_PERF_TIMEOUTLONGSHOT_PERF_BACKOFFLONGSHOT_PERF_KEEP_SENTkeep_sentLONGSHOT_PERF_URLaddresses.ini)Testing
Tested:
python -I -B tests/run_all.pyon Linux, ata401573bon maine090cc48(v3.0.0-rc.4): 564 passed, 14 failed, 16 skipped. 13 of the failures fail on main on this PC too (appimage_player_build, cpm_discovery, env_on, finish_output, lift_cleanup_error, linux_preflight, linux_update, make_game_cache, pipeline_tour, readback_late, vk_pgo, vram_budget_row, walk_key); the 14th,test_crash_dialog, fails on main too and is fixed by #141.tests/test_perf_report.pyandtests/test_perf_report.cpppass, with g++ and clang++.a401573b(RX_VK_ONLY=1 nx2/build.py --standalone): Linux compiled and linkedlongshot_vk.so, Windows with llvm-mingw (20261104,RX_TARGET=windows)longshot_vk.dll, 0 failed each; the changed host files (hos_main.c,nn_video.c,dav1d_av1.c,rx_backend.c) compile with no error and the same warnings as main's.9981a39awith LongshotLauncher#74 @ f7518c6, on the hardware above, a stand-in launcher home with the tick on, against the deployed service:--playwrotelongshot_perf.inisend=1, and the game logged "performance statistics on (the launcher's Send performance statistics)";perf_pending/("Title and intro", 82 s, the GPU, driver, CPU, OS and settings right);--playsent it before the game started: it moved toperf_sent/, and Grafana'slongshot_perf_reports_total{result="accepted"}went 3 → 4, withlongshot_sessions_totalforv3.0.0-rc.4-14-g9981a39alabelled RTX 3060 Laptop, NVIDIA 620.70, Ryzen 7 6800H, CachyOS, kernel 7.3, DLSS, preset Auto, cap 60, "Title and intro";a401573b, against the deployed service (https://api.magicbean.gmatte.xyz/telemetry/perf, behind Traefik):HTTPSConnection.connect), all taken;acceptedwent 5 → 12 over this PR's sends today, one for each summary sent, none twice.--playwrotesend=1andkeep_sent=0, the 3 copies kept before were deleted, the game still logged "performance statistics on"; quit through the control bridge (scripts/longshot_ctl.py quit), its summary was sent at the next--play, andperf_sent/stayed empty.RX_VK_PERF_INTERVAL=60, in the game built froma401573b(a pack over the shared base): logged "a summary every 60 s", wrote a summary of "Title and intro" 60.0 s, then one of 34.9 s at the quit.fbd42024and6c9edffe), in the game on the same PC:Not tested:
source=yet, so every summary is "not a release" for now, as in the crash report.Must be tested:
send=1in the game folder'slongshot_perf.ini(or tick it in LongshotLauncher#74), play 10 minutes on 2 maps, close the window, then runpython -I nx2/perf_report.py <folder> --send, expecting a summary inperf_pending\with those maps and the right GPU, driver, CPU and OS, thenperf_pending\empty and the same summary inperf_sent\(by hand: request T-ULGD; paste the summary's top lines)gpuanddriverto name the card and its driver (for exampleradv Mesa 25.2) and its fps to match the game's FPS counter (by hand: request T-WNBH)perf_pending/every 5 minutes, each with the maps since the one before, and those maps on the dashboard's map × graphics card panels after sending (by hand: request T-AAC7)Risks
What could break: with the tick on, each frame does a little more work (Measurements); a summary is written every 5 minutes, a few kB. With it off, the game reads one small file at its first frame and one load a frame. Privacy: a summary holds no paths, user name or ID, the log, saves, or anything from the game files;
docs/performance-statistics.mdlists what is sent.Turning it off: untick it (the launcher deletes
longshot_perf.ini, what is waiting and the copies), or deletelongshot_perf.ini; to stop all sending, removeperf_reportsfromnx2/addresses.ini; or revert.Compatibility:
MKWVK_API_VERSION22 → 23 (the renderer and the game are built together). Saves,longshot_options.iniand the caches don't change. A game folder gainsperf_pending/andperf_sent/only with the tick on.Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each new measured run (format indocs/tested-configurations.md).MKWVK_API_VERSIONbumped (gpu/vk/vk_api.h'sMkwVkHostorMkwVkRendererchanged).tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
https://claude.ai/code/session_015KgSKmeEEMir1hCDFZ8BmR
88404d20bb6702528faf910b654999ac41919db9ac41919db92c952130619e6c4c61ffedda0942aeWIP: feat(perf): opt-in performance statistics, sent from the launcher to the team's Grafanato feat(perf): opt-in performance statistics, sent from the launcher to the team's Grafanaedda0942ae65c63546ca65c63546cab271368d0be5d64b9d12efa3bd239fefa3bd239f6a00da2a2e6c9edffein tested configurationsfeat(perf): opt-in performance statistics, sent from the launcher to the team's Grafanato WIP: feat(perf): opt-in performance statistics, sent from the launcher to the team's GrafanaBack to WIP until it is tested end to end with the launcher: the tick in LongshotLauncher#74, a game run that writes the summaries, and the launcher sending them to Longshot-api's telemetry (now deployed at https://api.magicbean.gmatte.xyz/telemetry/perf; 3 summaries from this PR's A/B runs were taken and counted there). The new A/B on main
6c9edffeis in docs/tested-configurations.md (7fe3e981); the description follows.7fe3e981709981a39acfe090cc48Rebased on main
e090cc48(v3.0.0-rc.4) and tested end to end with LongshotLauncher#74 @ f7518c6, against the deployed telemetry service:send=1; the game wrote a summary at each window close; the next start sent it (perf_sent/). Grafana:longshot_perf_reports_total{result="accepted"}3 → 5,longshot_sessions_totalforv3.0.0-rc.4-14-g9981a39a2, with every hardware label right;e090cc48(Measurements): no cost that stands out from this PC's drift;docs/tested/(c70c0f88), as main now asks;The description is rewritten to the current template.
Description: Must be tested items now say what to expect, with the test request beside each.
WIP: feat(perf): opt-in performance statistics, sent from the launcher to the team's Grafanato feat(perf): opt-in performance statistics, sent from the launcher to the team's Grafanaa401573b: polite sending. One kept connection for every waiting summary (5 sent over 1 TLS connection to the deployed service), at most 12 a start, and a busy service left alone until its Retry-After or an hour doubling to a day. Switches with those defaults: LONGSHOT_PERF_MAX_SENDS, _TIMEOUT, _BACKOFF, _KEEP_SENT, _URL, and the game's RX_VK_PERF_INTERVAL (60 to 840 s, checked in the game at 60). Copies of what was sent are optional: --keep-sent off / keep_sent=0, the launcher's Keep a copy of what was sent (LongshotLauncher#74 d807edd, tested end to end). run_all as before (564 passed, the same 14 that fail on main); the renderer builds for Linux and Windows.a401573b2b962c5aad37962c5aad3713e858af3aCode review only (no tests run). Read the whole diff and every PR comment, including the end-to-end and test-request results. The game side is careful: API version bumped (22 to 23),
perf_infoappended last and null-checked, no ID or path in the summary, off by default, opt-in read fromlongshot_perf.ini, Windows paths wide viagame_dir_path(), locking is sound (muis never held across a call that can reachdevice_lost,at_exitonly try-locks,summarize(true)on close/device-lost). Nothing I'd block on; a few things worth deciding before merge:Sender deletes summaries on responses that aren't the service saying no (
nx2/perf_report.pyanswer()): every non-2xx other than 429/5xx is "refused" and the file is deleted. A 404 from Traefik while Longshot-api is redeployed or its route is missing, a 3xx (the Relay does not follow redirects, urllib does only in the proxied path), 408 or 425 would silently drop up to 12 summaries per start. Suggest treating 3xx, 404, 408, 425 as "later" (keep, no backoff or the backoff path), and keeping "refused" for 400/413/415/422.Two senders can overlap (
start_sending): each launcher start spawns a detached--send; if the previous one is still on its 10 s timeout (or a slow link) a second reads the sameperf_pending/and posts the same summaries, which the service then counts twice (the docs rely on 'deleted after'). A lock file (O_EXCLperf_pending/.sending, stale after a few minutes) would make that safe; the relay has no dedupe because summaries carry no ID.Hot-path cost is described as smaller than it is (docs/performance-statistics.md 'a few array increments', PR 'add_frame 6-16 ns'). Per frame with the tick on,
perf_framealso takesmu, copiesprof_ns, reads two atomics, and callsmenu_target_ms()->fps_target_ms()->menu_vsync()+display_refresh_mhz(); andhost->fps_timing(1)makes the host clock its sleeps. All probably sub-microsecond, but the A/B on that PC can't resolve it (main alone spread 15 fps, PR 'off' 2% behind main, 'on' about equal to 'off'). That the PR is honest about the noise is good; I'd either say 'sub-microsecond, not resolved by the runs' instead of 'a few array increments', or cache the target (it only changes with menu/display changes) inperf_frame. The test-request runs on a quieter PC will settle it.Minor: the 5-minute
summarize(false)runs on the presenting thread (readslongshot_perf.ini,where_now,vram_usage_now,/proc/self/statm, builds the JSON, spawns a thread). One slightly longer frame per period, and that frame's time lands in the next frame's histogram, possibly as one hitch (other). Harmless, but the JSON build could move into the writer thread by copyingmapsinstead.Nit: in
vk_menu.cppthe newfps_cap_now()and its comment were inserted betweenfps_target_ms()'s doc comment and the function, sofps_target_ms()lost its doc andfps_cap_now()has two stacked ones.Question:
perf_reportsinnx2/addresses.inipoints atapi.magicbean.gmatte.xyz, the same host as the crash reports (agreed in #141, I assume). Players' summaries go there with their IP visible to whoever runs it; worth the privacy page saying who operates it (it says 'Bean-Ops') and that the server logs/retention are the team's, since 'no ID' is claimed.Otherwise: the 'Must be tested' items (Windows, AMD/Intel, 32 GB RAM) are requested with test IDs, which is right for what this PC can't run.
Nit ( near line 2828):
fps_cap_now()now sits betweenfps_target_ms()'s doc comment (above, 2824) and the function: the target lost its doc and this has two. Move the new function above that comment.@ -0,0 +47,4 @@- **The game** (`gpu/vk/perf_report.cpp`, `perf_report.h`):- `perf_report_init()` reads `longshot_perf.ini` at the first frame. Off, nothing else runs.- On, the FPS counter (`vk_fps.cpp`) times every frame's parts as its Breakdown does and hands each frame to`perf_frame()`, which adds it to the current map's totals (a few array increments).'a few array increments' understates it (mutex, prof_ns copy, target lookup, the host timing its sleeps): say 'sub-microsecond a frame, not resolved by the A/B runs'.
@ -0,0 +283,4 @@// average, before this frame: a frame twice the usual, never every frame of a game the GPU holds under its cap)const double usual = avg_ms ? avg_ms : f.ms;avg_ms = usual * 0.98 + f.ms * 0.02;const double cap_ms = menu_target_ms();Per frame this goes through
fps_target_ms()(menu_vsync, display_refresh_mhz) and themulock above, so the docs' 'a few array increments' is generous. Likely negligible, but the target only changes with the menu or display: could be cached and refreshed withsince_map.@ -0,0 +288,4 @@pr::add_frame(*current, f, w, target, last_ms, usual);last_ms = f.ms;since_map += f.ms / 1000.0, since_summary += f.ms / 1000.0;if (since_summary >= period) summarize(false), since_summary = 0;The summary (ticked() file read, where_now, vram_usage_now, statm, JSON build) is made on the presenting thread inside
mu. Fine once per 5 minutes, but copyingmapsand building the JSON in the writer thread would keep that frame clean.@ -0,0 +406,4 @@passdef start_sending(folder, python=None, python_args=()):Nothing stops a second
--sendfrom starting while the first is still running (a 10 s timeout, or a slow link, outlasts a quick relaunch): both read the sameperf_pending/and post the same files, which the service counts twice. An O_EXCL lock file inperf_pending/(ignored once a few minutes old) would do.@ -0,0 +200,4 @@return "sent", Noneif code == 429 or code >= 500:return "busy", wait_of(retry_after)return "refused", NoneEverything that is not 2xx/429/5xx comes back 'refused' and the summary is deleted. A 404 during a redeploy (no Traefik route), a 3xx, 408 or 425 would drop it for good. Keep those for next time ('later'), delete only on 400/413/415/422-style answers.
View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.