feat(perf): the game sends its own v2 summaries with replay phases #1530
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1530
Loading…
Reference in a new issue
No description provided.
Delete branch "gab9281/Longshot:fix/perf-send-from-game"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Linux, CachyOS, RTX 3060 Laptop, NVIDIA 620.70 · Needs: Windows, any GPU (the send path uses the system's curl, as the crash helper does)
Why
With Send performance statistics ticked, the game writes summaries of how its frames went but the launcher was the only thing that sent them — at its start or at game exit. A long session showed nothing at the relay while it was playing, and a game folder played once and never opened again sent nothing at all. The field dashboard also could not say where the render thread spent its time, or whether frame generation, V-Sync and present mode were in play. See #1265 for the sender's related data-loss bugs.
The rule here is the one crash reports already follow: whoever does the work reports it. The game measures the frames and now sends them; the launcher only configures the tick and the address. Build reports stay with the launcher, which does the build.
What changed
The game now posts each summary itself with the system's curl, on the summary's own thread, and v2 summaries carry the missing replay phases and presentation labels; the launcher only writes the tick and stamps the relay address.
gpu/vk/perf_report.cpp: after each summary is written (every 5 minutes, at exit, on device loss),send_summary()POSTs it toperf_reports=with curl —find_curl/CreateProcessWon Windows,posix_spawnon Linux, UTF-8 paths. Curl is spawned and waited for off the render thread, with an 8 s / 10 s cap; one sender at a time (send_mu). What the relay took is kept inperf_sent/; only a definite refusal (400/413/415/422) deletes a file, while a 404, a redirect, a busy or unreachable answer leaves it for the next send. At start it retries leftovers then clears.tmp/curl files; at exit it sends the last summary then cleans.releaseis computed in the game with the crash helper's version/source rule.gpu/vk/perf_report.h: v2 summaries addphases_ms(the 13PROF_*replay phases),fine_ms(18 frame-time bucket edges with extra resolution around 60 Hz, so a 1% low no longer reads 49-50 fps on perfectly paced 60 fps play) andpresent_mode,vsync,framegen.parts_msstays strict; the new fields are separate. The relay (MagicBean-Mirror/Longshot-api@f30cd7e, already deployed) requires all v2 fields and still accepts v1 from older builds.nx2/make_game.py: stampsperf_reports=fromaddresses.ini, ascrash_reports=already is.gpu/vk/perf_report_schema.h: generated by Longshot-api'scmd/gen-game-schemafromtelemetry/perf/schema.go, so bucket edges and field names are no longer copied by hand between the game and the relay.waits_ms: the five waits already computed each frame for hitch classification (shader translation, pipeline, upload, memsync, readback), summed per map. Acausesobject adds the detail behind them: main thread CPU/blocked/vcore, late page waits, texture frame time, upload MB/count, pipe waits and textures destroyed. All of it is read and summed insideperf_frame, which never runs with the tick off. The relay exposeslongshot_frame_wait_seconds_totaland the dashboard has a per-cause bar gauge. Present-to-shown bookkeeping is gated on the tick or the frame trace, so with both off it is strictly zero.gpu/vk/vk_replay.cpp: the present-wait confirmation is now recorded per frame asshown_msinRX_VK_FRAME_TRACE's CSV, a single clock at the moment the frame reaches the screen (-1when the driver gives none), plusshown_at_ns, the absolute host-clock timestamp of that confirmation, so frame records can be aligned on the presentation moment. Previously only a 600-frame average was logged. The confirmations are also accumulated into the perf summary asshown_ms {sum, count}per map, so the relay exposeslongshot_present_shown_seconds_total/_count_totaland the dashboard shows present-to-shown latency in the field.gpu/vk/perf_os.h+perf_os_win.cpp/perf_os_posix.cppare a generic interface for running an external tool on either OS;perf_curl.cpp/howns curl discovery, arguments and the answer's status.perf_report.cppkeeps only the keep/drop/retry policy, with the duplicated per-frame gate merged into one block. A source test keeps the split.gpu/vk/switches.def,docs/performance-statistics.mdand the game-free checks updated. The dashboard change isMagicBean-Mirror/Bean-Ops@fa50471.gpu/vk/perf_report.cpp: a busy or unreachable relay ends the startup retry pass, as the launcher's sender did, so one busy answer cannot post a whole backlog at once. At exit or device loss the last summary is left inperf_pending/when another curl still holds the sender lock, so closing never queues a second wait behind it.Measurements
Verdict: the send path works in the game: six summaries were posted one per minute while the game was still running, accepted by a v2 relay, with real phase values and the new labels.
Hardware: CachyOS (Linux 7.3), RTX 3060 Laptop GPU 6 GB, NVIDIA 620.70, Ryzen 7 6800H, 15 GB.
Scene: a short unscripted session from a fresh runtime, starting at the title and moving into Kokiri Forest / Forest Temple; 1280x720, preset Auto, frame cap 60 via the menu,
RX_VK_PERF_INTERVAL=60.Method: built
rt-perf-v2bas a pack over the shared base, ran the game against a local v2 relay over HTTPS (the deployed relay image is still pre-v2), and readperf_sent/*.jsonand the relay's/metrics. Not an fps measurement: this changes no per-frame work except one thread creation every 5 minutes; the existing summary cost is unchanged.How to reproduce:
RX_VK_PERF_INTERVAL=60 CURL_CA_BUNDLE=<local cert> ./rt-perf-v2b/longshotwithsend=1andperf_reports=https://127.0.0.1:8443/telemetry/perfin the folder, play a few minutes, quit; inspectperf_sent/and the relay's/metrics.257c54916c653d2f(pre-rebase head)perf_pending/emptylongshot_perf_reports_total{result="accepted"}submit18.4 s,draw10.1 s,descriptors4.2 s over 60 s in Forest TempleThese numbers were measured at the pre-rebase head. The branch has since been rebased onto
main @ c507b98cand the sender policy fixes added; the game-side send has not been re-measured on that base (see Must be tested). The 11 original commits are unchanged in content: the rebased tree matched the clean three-way merge exactly.Flags and settings
No new shipping switch: the setting is the launcher's existing Send performance statistics tick and
send=1inlongshot_perf.ini. One test-only switch is added.RX_VK_PERF_TEST_CURLRX_VK_FRAME_TRACElogs/longshot_frames.csvshown_mscolumn from present-waitTesting
Tested: built
rt-perf-finalfrom this branch and ran it on Linux/CachyOS with the RTX 3060 for several minutes atRX_VK_PERF_INTERVAL=60, pointed at the real deployed relay (data-services.magicbean.dev). Five v2 summaries went out one a minute intoperf_sent/; the live relay's/metricsshowed this build'slongshot_frame_phase_seconds_total,longshot_frame_wait_seconds_total,longshot_frame_cause_*_total,longshot_frame_time_fine_secondsandlongshot_present_shown_*series, with the new labels and 13/13 non-zero phases. The exit summary stayed inperf_pending/for the next-start retry, as designed.shown_mswas zero because V-Sync was off (immediate), so present-wait had nothing to confirm — the farm matrix covers FIFO. Game-free at the current head3884d62: the newtest_perf_send(a busy relay gets one POST, not one per queued summary; the exit send does not queue behind an in-flight curl),test_perf_report,test_perf_schema,test_frame_trace_waitsandtest_doc_anchorspass.python -I -B tests/run_all.py: 661 passed, 32 failed, 97 skipped — exactly the 32 failures ofmain @ c507b98con this PC (Vulkan/SPIR-V headers and other local prerequisites are missing), so no new failure.test_switch_tablefails onmaintoo (missing rows for switches added there). The new test passes with bothg++andclang++. The renderer standalone build could not run here: no Vulkan, SPIR-V or VMA headers are installed; it is not counted as a pass.Pre-existing CI failure (not introduced here):
Guide screenshots / Guide images and local UI source reviewfails onmainwith 7 flagged source files; this branch adds one (gpu/vk/vk_fps.h, an implementation-only signature change). I captured the Graphics view with the repo's own UI-only helper on this branch and compared it withdocs/images/guide/settings-graphics.png: same 1280x720, but 60,048 pixels differ across the whole frame, so the guide image is genuinely stale against currentmain— this needs a proper recapture and human review, not a hash refresh. Perdocs/images/guide/README.mdI did not touch the provenance or hashes. The other CI checks pass or are skipped on this forge.Not tested: Windows (no mingw toolchain on this PC); the Windows branch mirrors the crash helper's proven
CreateProcessW/quoting code.Must be tested:
perf_sent/and no stuck files inperf_pending/. The Test Farm cannot do this one: its sandbox has no network and its security check holds commits that reach it, so a relay POST would never leave the agent.POST /telemetry/perfwith"v":2before this ships, expecting200 {"ok":true}(the old relay answers 400not a version 1 summaryand the game drops those summaries). Already verified here against the deployed relay.3884d62againstmain, expecting six v2 summaries inperf_sent/and the relay accepting them, so the numbers describe the branch that would merge.perf_pending/for the next start. The game-free test covers the policy, not real close timing.Risks
What could break: a v2 summary posted before the new relay is deployed is refused and deleted — deployment order matters. An unreachable relay leaves files in
perf_pending/up to the 48-summary cap, and a busy or unreachable answer stops the startup retry pass, so the rest wait for a later start. At exit or device loss, when another curl is still in flight, the final send is skipped and its summary stays inperf_pending/for the next start rather than adding a second wait to close; the in-flight child is ended with the process. Windows close timing with a send in flight is still to be checked by hand.Turning it off: untick Send performance statistics (the game deletes what is waiting, as before), or revert the commit.
Compatibility: v1 is still accepted by the relay and read as zeros/
not known; no save, options, cache or ABI change. Summaries carry no ID, paths or user name.Issues
See #1265
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.Tested in game: RTX 3060 Laptop, NVIDIA 620.70, CachyOS: ran rt-perf-v2b for several minutes with a local v2 relay; 6 summaries posted one per minute into perf_sent/, perf_pending/ empty, 12 accepted at the relay, non-zero phase metrics and present-mode/vsync/framegen labels in the summary JSON.
Possibly related, for context:
WIP: feat(perf): the game sends its own v2 summaries with replay phasesto feat(perf): the game sends its own v2 summaries with replay phasesThe presenter's VK_KHR_present_wait confirmations are accumulated and drained into the next summary as shown_ms {sum, count} per map, so the relay and Grafana can show how long presents waited to reach the screen. v1 has no such field. Assisted-by: OpenCode@Beanstalk windowed V-Sync and frame generation matrix on AMD and NVIDIA
This PR changes how performance data is collected and sent; the local run was Linux + RTX 3060 in a windowless lane, so the swapchain cases need real hardware. Please run a matrix on windowed Windows targets (windows+amd and windows+nvidia) varying V-Sync On/Off and frame generation On/Off, against main on the same PC, and report frame p99 and VRAM per cell. The v2 summary now labels each report with present_mode, vsync and framegen, so the report should name the settings each cell actually used.
Note: perf summaries and automatic crash reports cannot be enabled on farm agents — the sandbox has no network by design (bubblewrap / AppContainer / firewall) and the security check holds commits that reach it. That path is tested by hand on a normal Windows PC; the farm's job here is only the performance comparison under the different presentation settings.
@gab9281 approval of
265285frecorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.Beanstalk test farm
Commit
3884d62· tier untrusted · approved by nobody yet · dashboardThis diff touches sensitive paths, so it needs two members of
BigBeans:nx2/make_game.pyA member of
BigBeanscan review this commit and comment@Beanstalkto test it.Independent Windows sender check at
6c653d2f5f603c75189934abd02124ae0fec608d(llvm-mingw, native Windows, synthetic summaries).Startup retry on 429: three queued files produced three more-or-less immediate attempts in the same startup pass even though each returned 429. The files were retained. Please consider stopping the startup pass on transient failure and carrying forward any applicable retry delay, so a busy relay does not receive the rest of the backlog in that pass.
Shutdown bound: the 10-second Win32 wait is per child. The final synchronous send takes
send_mu, so it can wait for an earlier sender before starting its own wait. Please verify close with a send in flight and either document the observed behavior or enforce the intended total deadline while preserving the pending summary.All test inputs and logs stayed local.
Follow-up synthetic check at the same head: with one 5.5 s test child already active and another 5.5 s response on the final-send path, that final synchronous call took 11.125 s. Both 503 summaries were retained. This confirms that the shutdown send path can exceed 10 s under contention. Full game-close timing remains to be checked.
WIP #1625 exercises state tagging in a local sidecar with the existing field producer. The Windows run found same-map menus and a map load with only
unknownevidence in the middle. For a separate telemetry follow-up, propose optional bounded per-map state histograms using the v2 bucket edges. Keep the combined histogram available for queries and track the classified duration so unknown or transition-heavy samples have visible coverage. This would need relay acceptance and shared producer/relay fixtures before enabling emission.6c653d2f5f3884d62ed2Rebased onto
main @ c507b98cand addressed the sender findings; branch head is now3884d62(force-pushed tofix/perf-send-from-game). All 11 original commits are preserved in order — the rebased tree matched the clean three-way merge exactly.busy relay (429):
send_pending()used to callsend_summary()for every queued file even after a busy answer, so 48 pending summaries meant 48 immediate POSTs. It now stops at the firstBusy/None, as the launcher'snx2/perf_report.pysender always did; the rest stay inperf_pending/.close timing: the exit/device-loss path used to block on
send_muand then wait up to another 10 s for its own curl, which is how futondestroyer measured 11.125 s. It now usestry_lock: if a curl is in flight, the last summary stays inperf_pending/for the next start, andclean_pending()leaves that curl's scratch files alone.test: new game-free
tests/test_perf_send.pycuts the realsend_summary/send_pendingout ofperf_report.cpp, compiles them with a stand-in curl, and checks the one-POST-on-429 rule and the non-blocking exit send. It fails on the pre-fix head (3 POSTs) and passes now, with both g++ and clang++. The PR description records the full check results and what could not be run here (standalone renderer build: no Vulkan headers; in-game close timing and a re-measure on the new base are still listed under Must be tested).The description's measurement table still shows the pre-rebase numbers, labelled as such — the game has not been run on the new base here.
Format the new Must-be-tested items so the PR description check passes; no code change.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.