fix(perf): the sender keeps what the relay did not refuse, sends one at a time, and stays under the ingress #1368
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1368
Loading…
Reference in a new issue
No description provided.
Delete branch "omgitsgela/Longshot:fix/perf-sender-loss"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux x86-64 (the sender is the same Python on Windows and Linux; game-free checks here) · Needs: nothing beyond CI: the relay answers are reproduced by the local relay in the tests
Why
nx2/perf_report.py: the sender the launcher starts to post the game's opt-in performance summaries: can silently lose or double data three ways, named by their author in #1265 with the fix list.Fixes #1265.
How to reproduce
Seen on: main at
17394b71(the #983 merge);nx2/perf_report.pyunchanged on main to544229a2.Steps: run
python -I nx2/perf_report.py <game folder> --sendagainst a relay answering 404 (a redeployed service with its route missing), or start the launcher twice while a first detached sender is still waiting on its timeout, or let more than 6 summaries wait while the perf route's ingress (6 a minute, burst 6 per address) is in force.Expected: a summary is deleted only when the relay says it will never take it; one sender at a time; a start never posts past the ingress's burst.
Actual: any 4xx except 429 deleted the summary (up to 12 a start lost on a 404); two detached senders could post the same summaries (no id to dedupe on); 12 a start tripped a 429 on the 7th and an hour-to-a-day backoff for the rest.
What changed
The perf sender stops losing, doubling, and tripping its own rate limit: a summary is deleted only when the relay says it will never take it, one sender runs at a time under a lock file, and a start's share sits under the ingress's burst.
answer():' only 400, 413, 415, 422 are "refused" (delete the summary: the relay named why and will not take it). Every other status: 3xx a redirecthttp.clientdoes not follow, 404 while the service is redeployed, 408, 425: is "later": the summary is kept for the next start, and no busy-backoff note is written (a missing route is not the relay asking to be left alone).send_waitingholdsperf_pending/.sending, a lease: its body names the holder's pid and a lease end, and a heartbeat thread renews the lease every 30 s while the send runs. A second sender is refused while the lease runs. A live slow sender (a relay that trickles) is never taken over — wall-clock age is not liveness; a killed holder is taken over at once by a pid check (os.kill(pid, 0)), no waiting for the clock and no wedge after a clock jump backwards. A takeover claims the expired lock by rename and installs the new one with anO_CREAT | os.O_EXCLcreate — atomic on every file system, including the vfat/exfat SD cards a Deck game folder may live on (hard links, which an earlier draft used, do not exist there) — so two racing takers cannot both reclaim one stale lock and post the same summaries twice. Release removes only a lock whose body names the releaser, so a holder that finishes after being taken over never deletes its successor's lock. A folder that cannot hold a lock at all (read-only, gone,.sendingleft as a directory by an older form): sends unsent, as before; a half-written lock body (a full disk) is removed, not handed out unreadable.MAX_SENDS), under the ingress's burst of 6; the sender's number and the ingress limit are written next to each other in one comment so the next change moves both.send_these's "later" branch already returned without deleting; it needed no change beyondanswer()producing it.Measurements
Verdict: no fps to measure: a telemetry-sender fix; the checks are named under Testing.
Flags and settings
One existing switch's default moves; no flag is added, renamed, or removed.
LONGSHOT_PERF_MAX_SENDSNo new switch, menu row, or ini key. The consent tick and
keep_sentare untouched.Testing
Tested:
tests/test_perf_report.pyon this branch: all passed (88 checks, green standalone three runs in a row and under run_all's 12-way load), including new pins: a 404 relay keeps both summaries and writes no.nextnote; 400/413/415/422 each delete;answer()'s table (301/302/307/308/404/408/425 later; 400/413/415/422 refused; 204 sent; 500 busy; 429+Retry-After busy); a held lease refuses a second sender; an expired lease whose holder's pid is gone is taken over at once; a live holder past its lease is still refused (pid wall); release removes only the body's own holder's lock; the claim rendezvous (taker paused between claim and install, a fresh lock appearing meanwhile) makes the install fail rather than overwrite; four forked senders race one folder: every summary posted exactly once; a blocking relay outliving one lease: the heartbeat renews the lock's mtime mid-send and a second sender stays refused; a proxied 302 keeps the summary and never follows it into a GET; ENOSPC at the lock body leaves no lock and the next send works;.sendingas a directory is cleared; the MAX_SENDS pin is the literal 5 plus the docs/constant pairing text on both sides (no self-reference). Plus the review's own adversarial harness, all 15 cases (stale race, ownership between two real holder processes, live slow sender, killed holder, clock jump backwards, directory, write failure, read-only, permissions, proxy redirect, later lifecycle, cap order, direct statuses, untick in flight): 15/15. Mutation probes: removing the pid wall fails 4 pins and 3 harness repros; release by pathname fails the owner pin and the ownership repro; a remove-then-rename takeover fails the rendezvous pin and 2 repros; no heartbeat fails the mtime-beat pin; keeping a half-written lock fails the ENOSPC pin; following redirects fails the 302 pin; removing the mid-send untick sweep fails its source pin. Round three (review's filesystem probe):os.linkdoesn't exist on vfat/exfat, so the takeover install became anO_CREAT | os.O_EXCLcreate (pin updated to require it and forbidlink()); the re-pointed mutation (an install accepting an existing lock) fails exactly the rendezvous pin; harness stale-race/ownership/directory/write-failure/clock-backward/killed and the race probe all pass at33457e88.tests/run_all.py: this branch @81926bf5576 passed / 38 failed / 107 skipped vsmain544229a2573 / 41 / 107: the branch's failing set is a strict subset of main's (set-compared; main's extra three are flaky-order cases). Skips lackclang++/vulkan_headers/spirv_headershere.Not tested: a real relay deploy with a missing route (the local relay answers the same statuses); two launchers on Windows starting senders at once — the lock's file primitives (O_EXCL, rename, link, replace) are the same ones
set_sendingalready uses there, but the pid wall falls back to lease+LEGACY mtime on Windows, which the round-one review's clock caveats apply to; the Test Farm should run the two-launcher case.Must be tested: by hand, no hardware needed: with the perf tick on and summaries waiting, run
python -I nx2/perf_report.py <game folder> --sendtwice at once against any relay; expecting the second to printleftfor everything and the relay to receive each summary once (before this change both could post).Risks
What could break: summaries a genuinely-dead route would once have deleted now wait (that is the fix; they drain when the route returns, oldest first, 5 a start). A sender killed mid-send leaves
.sendingbehind only until the next sender's pid check says its holder is gone — immediately on Linux, after the lease on Windows. A player notices nothing. Nothing touches the consent tick,perf_sent/copies, the game's own perf writes, or saves.Turning it off: revert the one commit (three hunks in
nx2/perf_report.pyplus their tests).Compatibility: none: no protocol change, no new files read, one new dot-file inside
perf_pending/that older senders simply list past (waiting()matches*.json).Issues
Fixes #1265.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).Tested in game: no game run involved: the sender is game-free Python; the local relay in tests/test_perf_report.py answers every status named; all 74 checks pass with the mutation pins
A summary is deleted only when the relay says it will never take it, one sender runs at a time, and a start posts five, under the route's burst of six. I read head
cae65abe. It applies onto current main. This needs a change before it can merge.answertreats 400, 413, 415 and 422 as refused. A redirect, a 404, a 408 and a 425 keep the summary and do not write the busy-backoff note.send_thesereturns on that answer without deleting the file..sendingis taken withO_EXCL. A second sender leaves everything. A lock older than 15 minutes is taken over. A folder that cannot hold a lock still sends. The check holds a lock and posts nothing, and it pins those statuses. It also merges with #1299, the reconnect.docs/performance-statistics.mdstill says a start sends at most 12, and its switch table still givesLONGSHOT_PERF_MAX_SENDSthe default 12. The code's default is 5. That page should say 5, and that a start stays under the burst of 6.Round two (adversarial review, verdict request-changes — every finding reproduced and fixed on head @
d392492b):P1, ownership-safe and atomic lock:
POST bodies=[0,0](stale-race) now passes: one POST.successor lock deleted=True, third sender entered=True(ownership, run between two real holder processes) now passes.P2, liveness is not wall-clock age:
Tests no longer self-referential: the cap pin asserts MAX_SENDS == 5 as a literal, and pins the pairing text on both sides — the constant's own comment names the ingress's 6 a minute burst 6, and the docs' line must carry the same two numbers. docs/performance-statistics.md now says 5 a start + the 6/burst pairing + one-sender lease; the stale 12 is gone (docs pin fails if it returns).
Also from the review's notes: a half-written lock body (ENOSPC) is removed instead of handed out unreadable (write-failure repro); .sending left as a directory is cleared, not swallowed forever (directory repro); a mid-send set_sending(False) re-sweeps so an in-flight 503 can't leave a .next behind (untick-inflight repro); the proxied post() never follows a redirect — the 3xx itself is the answer, opener built per call so a proxy that starts/stops mid-run is seen (proxy-redirect repro: summary kept, no GET).
Mutation probes: no-pid-wall fails 4 suite pins + 3 adversarial; release-by-path fails owner-pin + ownership repro; unatomic-take fails the claim-rendezvous pin (rename-then-install caught deterministically) + stale-race/ownership repros; no-heartbeat fails the lease-renewal pin; keep-half-lock fails the ENOSPC pin; follow-redirect fails the 302 pin; no-untick-sweep fails the source pin.
Evidence: tests/test_perf_report.py 9x checks green (suite count grew from 74); the review harness (perf_adversarial.py, LOCK_STALE fixtures rewritten to lease bodies, ownership case now two real holder processes, clock-backward case pinned to the pid wall) 15/15; my own race probe (four senders, killed holder, ENOSPC, directory, proxied redirect, slow live holder) 7/7; full run_all comparison posted below. Windows/relay deployment notes unchanged under Testing.
Round three:
The exFAT/vfat finding (question 5) is REAL and fixed — head
33457e88. The takeover installed its claimed lock with os.link(); vfat/exfat carry no hard links, and a Deck game folder — where perf_pending/ and its lock live — is commonly on a microSD formatted exFAT. A failed install there fell to "send unlocked": exactly the double post this lock prevents, on a common Deck layout. The claim stays a rename (one winner); the install is now the same O_CREAT|O_EXCL create the ordinary path uses — atomic on every filesystem the launcher runs on. Measured on the real Deck: the dce318f1 microSD mount reports ext2/ext3 (link works there too — noted for honesty), but SD cards ship exFAT and the primitive must not assume the card's format. The source pin now requires O_CREAT|O_EXCL and forbids link(); the rendezvous pin still makes a taker's install fail against a fresh lock; the re-pointed mutation (an install that accepts an existing lock) fails exactly that pin and re-greens on restore. Suite 88 green, harness 7/7 on the lock cases, race probe 7/7, run_all 576/38/107 strict-subset.The consent-flip P1 (a lock left live, perf_sent/ made after set_sending(False)) does not reproduce on the pushed heads. Executed on
d392492b(before this fix, after round two): flip consent off from inside Relay.post, before the reply; result {'sent': 0, 'left': 2}, lock-left: False, .next-left: False, perf_sent exists: False, POSTs: 1. send_these checks sending(folder) at the top of every loop pass and re-sweeps; keep_sent returns without touching perf_sent/ when the tick is off. The untick-inflight harness case passes too, and keep_sent has its own sending(folder) guard. If that repro can be produced againstd392492bor33457e88, I'll take it immediately.The heartbeat-vs-release and pid-reuse notes are acknowledged as designed behavior: the heartbeat checks _lock_read(path)[0] == getpid() before each replace (and returns when our lock is gone), and a recycled pid is refused at most until the lease + STALL window (15.5 min) — capped by design, and the next sender after that proceeds. Accepted as-is with the comment on _heartbeat spelling the stall-window race out.
Body line corrected to match head
33457e88: the takeover install is an O_EXCL create (vfat/exfat-safe), not a hard link.Mechanism line and Testing now state head
81926bf5(the O_EXCL takeover install + the heartbeat window comment): suite 88 green, run_all 576/38/107, branch-only failures [].Round four: I reproduced all four of your findings against
81926bf5before changing anything (my own deterministic probes, not the test suite: A paused after judging the lock stale and before its claim rename; an O_EXCL-created name with the body not yet written; a beat parked between its ownership check and its replace; a release paused between its body read and its remove). 4/4 reproduced. Fixed onfc5fa3fa:send_waitingnow joins the heartbeat thread BEFORE releasing the lock (bounded at 5 beats: a beat can only still be in flight on a file system that is itself wedged). Source-pinned that the join precedes the release.The liveness rule is the load-bearing part, and your PID-reuse question is why it is not a bare "live pid = wall": a CERTAINLY-gone pid holds nothing (lease or no lease), and a holder whose lease ran out is judged by whether it is still working — the heartbeat touches the lock every BEAT, so a working sender's mtime walks with its beats and holds however far the lease ran or the clock jumped, while a quiet holder (dead, or its pid recycled by an unrelated process) is claimable once 2*BEAT pass without a touch. Pinned both directions: a wedged-but-beating holder holds; a silent live-pid holder is claimable.
Test honesty, per your note: the four-sender fork test is unchanged (it never claimed to pin the takeover); the claim rendezvous pins your exact window; the heartbeat test pins beats-during-send; the pid-wall pins now include lease-expired-live-holder (holds) and quiet-live-holder (claimable).
tests/test_perf_report.pygreen (88 checks), 4/4 of my probes NOT-REPRO atfc5fa3fa.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.