fix(perf): the sender keeps what the relay did not refuse, sends one at a time, and stays under the ingress #1368

Open
omgitsgela wants to merge 6 commits from omgitsgela/Longshot:fix/perf-sender-loss into main
Contributor

Type: fix · Risk: low · Tested on: Linux x86-64 (the sender is the same Python on Windows and Linux; game-free checks here) · Needs: nothing beyond CI: the relay answers are reproduced by the local relay in the tests

Why

nx2/perf_report.py: the sender the launcher starts to post the game's opt-in performance summaries: can silently lose or double data three ways, named by their author in #1265 with the fix list.

Fixes #1265.

How to reproduce

Seen on: main at 17394b71 (the #983 merge); nx2/perf_report.py unchanged on main to 544229a2.

Steps: run python -I nx2/perf_report.py <game folder> --send against a relay answering 404 (a redeployed service with its route missing), or start the launcher twice while a first detached sender is still waiting on its timeout, or let more than 6 summaries wait while the perf route's ingress (6 a minute, burst 6 per address) is in force.

Expected: a summary is deleted only when the relay says it will never take it; one sender at a time; a start never posts past the ingress's burst.

Actual: any 4xx except 429 deleted the summary (up to 12 a start lost on a 404); two detached senders could post the same summaries (no id to dedupe on); 12 a start tripped a 429 on the 7th and an hour-to-a-day backoff for the rest.

What changed

The perf sender stops losing, doubling, and tripping its own rate limit: a summary is deleted only when the relay says it will never take it, one sender runs at a time under a lock file, and a start's share sits under the ingress's burst.

  • answer():' only 400, 413, 415, 422 are "refused" (delete the summary: the relay named why and will not take it). Every other status: 3xx a redirect http.client does not follow, 404 while the service is redeployed, 408, 425: is "later": the summary is kept for the next start, and no busy-backoff note is written (a missing route is not the relay asking to be left alone).
  • One sender at a time: send_waiting holds perf_pending/.sending, a lease: its body names the holder's pid and a lease end, and a heartbeat thread renews the lease every 30 s while the send runs. A second sender is refused while the lease runs. A live slow sender (a relay that trickles) is never taken over — wall-clock age is not liveness; a killed holder is taken over at once by a pid check (os.kill(pid, 0)), no waiting for the clock and no wedge after a clock jump backwards. A takeover claims the expired lock by rename and installs the new one with an O_CREAT | os.O_EXCL create — atomic on every file system, including the vfat/exfat SD cards a Deck game folder may live on (hard links, which an earlier draft used, do not exist there) — so two racing takers cannot both reclaim one stale lock and post the same summaries twice. Release removes only a lock whose body names the releaser, so a holder that finishes after being taken over never deletes its successor's lock. A folder that cannot hold a lock at all (read-only, gone, .sending left as a directory by an older form): sends unsent, as before; a half-written lock body (a full disk) is removed, not handed out unreadable.
  • The start's share now defaults to 5 (MAX_SENDS), under the ingress's burst of 6; the sender's number and the ingress limit are written next to each other in one comment so the next change moves both.
  • send_these's "later" branch already returned without deleting; it needed no change beyond answer() producing it.

Measurements

Verdict: no fps to measure: a telemetry-sender fix; the checks are named under Testing.

Flags and settings

One existing switch's default moves; no flag is added, renamed, or removed.

Flag Before this PR After this PR What it does Effect Recommendation
LONGSHOT_PERF_MAX_SENDS 12 5 How many waiting summaries one start posts, oldest first; the name and the 1..48 range are unchanged. A start stays under the perf route's ingress (6 a minute, burst 6), so the 7th no longer earns a 429 and an hour-to-a-day backoff for the rest; more of them arrive the same day. Leave it at the default; 12 is a valid setting again if the ingress burst ever rises.

No new switch, menu row, or ini key. The consent tick and keep_sent are untouched.

Testing

Tested: tests/test_perf_report.py on this branch: all passed (88 checks, green standalone three runs in a row and under run_all's 12-way load), including new pins: a 404 relay keeps both summaries and writes no .next note; 400/413/415/422 each delete; answer()'s table (301/302/307/308/404/408/425 later; 400/413/415/422 refused; 204 sent; 500 busy; 429+Retry-After busy); a held lease refuses a second sender; an expired lease whose holder's pid is gone is taken over at once; a live holder past its lease is still refused (pid wall); release removes only the body's own holder's lock; the claim rendezvous (taker paused between claim and install, a fresh lock appearing meanwhile) makes the install fail rather than overwrite; four forked senders race one folder: every summary posted exactly once; a blocking relay outliving one lease: the heartbeat renews the lock's mtime mid-send and a second sender stays refused; a proxied 302 keeps the summary and never follows it into a GET; ENOSPC at the lock body leaves no lock and the next send works; .sending as a directory is cleared; the MAX_SENDS pin is the literal 5 plus the docs/constant pairing text on both sides (no self-reference). Plus the review's own adversarial harness, all 15 cases (stale race, ownership between two real holder processes, live slow sender, killed holder, clock jump backwards, directory, write failure, read-only, permissions, proxy redirect, later lifecycle, cap order, direct statuses, untick in flight): 15/15. Mutation probes: removing the pid wall fails 4 pins and 3 harness repros; release by pathname fails the owner pin and the ownership repro; a remove-then-rename takeover fails the rendezvous pin and 2 repros; no heartbeat fails the mtime-beat pin; keeping a half-written lock fails the ENOSPC pin; following redirects fails the 302 pin; removing the mid-send untick sweep fails its source pin. Round three (review's filesystem probe): os.link doesn't exist on vfat/exfat, so the takeover install became an O_CREAT | os.O_EXCL create (pin updated to require it and forbid link()); the re-pointed mutation (an install accepting an existing lock) fails exactly the rendezvous pin; harness stale-race/ownership/directory/write-failure/clock-backward/killed and the race probe all pass at 33457e88. tests/run_all.py: this branch @ 81926bf5 576 passed / 38 failed / 107 skipped vs main 544229a2 573 / 41 / 107: the branch's failing set is a strict subset of main's (set-compared; main's extra three are flaky-order cases). Skips lack clang++/vulkan_headers/spirv_headers here.

Not tested: a real relay deploy with a missing route (the local relay answers the same statuses); two launchers on Windows starting senders at once — the lock's file primitives (O_EXCL, rename, link, replace) are the same ones set_sending already uses there, but the pid wall falls back to lease+LEGACY mtime on Windows, which the round-one review's clock caveats apply to; the Test Farm should run the two-launcher case.

Must be tested: by hand, no hardware needed: with the perf tick on and summaries waiting, run python -I nx2/perf_report.py <game folder> --send twice at once against any relay; expecting the second to print left for everything and the relay to receive each summary once (before this change both could post).

Risks

What could break: summaries a genuinely-dead route would once have deleted now wait (that is the fix; they drain when the route returns, oldest first, 5 a start). A sender killed mid-send leaves .sending behind only until the next sender's pid check says its holder is gone — immediately on Linux, after the lease on Windows. A player notices nothing. Nothing touches the consent tick, perf_sent/ copies, the game's own perf writes, or saves.

Turning it off: revert the one commit (three hunks in nx2/perf_report.py plus their tests).

Compatibility: none: no protocol change, no new files read, one new dot-file inside perf_pending/ that older senders simply list past (waiting() matches *.json).

Issues

Fixes #1265.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
**Type:** fix · **Risk:** low · **Tested on:** Linux x86-64 (the sender is the same Python on Windows and Linux; game-free checks here) · **Needs:** nothing beyond CI: the relay answers are reproduced by the local relay in the tests ## Why `nx2/perf_report.py`: the sender the launcher starts to post the game's opt-in performance summaries: can silently lose or double data three ways, named by their author in #1265 with the fix list. Fixes #1265. ## How to reproduce Seen on: main at 17394b71 (the #983 merge); `nx2/perf_report.py` unchanged on main to 544229a2. Steps: run `python -I nx2/perf_report.py <game folder> --send` against a relay answering 404 (a redeployed service with its route missing), or start the launcher twice while a first detached sender is still waiting on its timeout, or let more than 6 summaries wait while the perf route's ingress (6 a minute, burst 6 per address) is in force. Expected: a summary is deleted only when the relay says it will never take it; one sender at a time; a start never posts past the ingress's burst. Actual: any 4xx except 429 deleted the summary (up to 12 a start lost on a 404); two detached senders could post the same summaries (no id to dedupe on); 12 a start tripped a 429 on the 7th and an hour-to-a-day backoff for the rest. ## What changed The perf sender stops losing, doubling, and tripping its own rate limit: a summary is deleted only when the relay says it will never take it, one sender runs at a time under a lock file, and a start's share sits under the ingress's burst. - `answer()`:' only 400, 413, 415, 422 are "refused" (delete the summary: the relay named why and will not take it). Every other status: 3xx a redirect `http.client` does not follow, 404 while the service is redeployed, 408, 425: is "later": the summary is kept for the next start, and no busy-backoff note is written (a missing route is not the relay asking to be left alone). - One sender at a time: `send_waiting` holds `perf_pending/.sending`, a lease: its body names the holder's pid and a lease end, and a heartbeat thread renews the lease every 30 s while the send runs. A second sender is refused while the lease runs. A live slow sender (a relay that trickles) is never taken over — wall-clock age is not liveness; a killed holder is taken over at once by a pid check (`os.kill(pid, 0)`), no waiting for the clock and no wedge after a clock jump backwards. A takeover claims the expired lock by rename and installs the new one with an `O_CREAT | os.O_EXCL` create — atomic on every file system, including the vfat/exfat SD cards a Deck game folder may live on (hard links, which an earlier draft used, do not exist there) — so two racing takers cannot both reclaim one stale lock and post the same summaries twice. Release removes only a lock whose body names the releaser, so a holder that finishes after being taken over never deletes its successor's lock. A folder that cannot hold a lock at all (read-only, gone, `.sending` left as a directory by an older form): sends unsent, as before; a half-written lock body (a full disk) is removed, not handed out unreadable. - The start's share now defaults to 5 (`MAX_SENDS`), under the ingress's burst of 6; the sender's number and the ingress limit are written next to each other in one comment so the next change moves both. - `send_these`'s "later" branch already returned without deleting; it needed no change beyond `answer()` producing it. ## Measurements **Verdict:** no fps to measure: a telemetry-sender fix; the checks are named under Testing. ### Flags and settings One existing switch's default moves; no flag is added, renamed, or removed. | Flag | Before this PR | After this PR | What it does | Effect | Recommendation | | --- | --- | --- | --- | --- | --- | | `LONGSHOT_PERF_MAX_SENDS` | 12 | 5 | How many waiting summaries one start posts, oldest first; the name and the 1..48 range are unchanged. | A start stays under the perf route's ingress (6 a minute, burst 6), so the 7th no longer earns a 429 and an hour-to-a-day backoff for the rest; more of them arrive the same day. | Leave it at the default; 12 is a valid setting again if the ingress burst ever rises. | No new switch, menu row, or ini key. The consent tick and `keep_sent` are untouched. ## Testing **Tested:** `tests/test_perf_report.py` on this branch: all passed (88 checks, green standalone three runs in a row and under run_all's 12-way load), including new pins: a 404 relay keeps both summaries and writes no `.next` note; 400/413/415/422 each delete; `answer()`'s table (301/302/307/308/404/408/425 later; 400/413/415/422 refused; 204 sent; 500 busy; 429+Retry-After busy); a held lease refuses a second sender; an expired lease whose holder's pid is gone is taken over at once; a live holder past its lease is still refused (pid wall); release removes only the body's own holder's lock; the claim rendezvous (taker paused between claim and install, a fresh lock appearing meanwhile) makes the install fail rather than overwrite; four forked senders race one folder: every summary posted exactly once; a blocking relay outliving one lease: the heartbeat renews the lock's mtime mid-send and a second sender stays refused; a proxied 302 keeps the summary and never follows it into a GET; ENOSPC at the lock body leaves no lock and the next send works; `.sending` as a directory is cleared; the MAX_SENDS pin is the literal 5 plus the docs/constant pairing text on both sides (no self-reference). Plus the review's own adversarial harness, all 15 cases (stale race, ownership between two real holder processes, live slow sender, killed holder, clock jump backwards, directory, write failure, read-only, permissions, proxy redirect, later lifecycle, cap order, direct statuses, untick in flight): 15/15. Mutation probes: removing the pid wall fails 4 pins and 3 harness repros; release by pathname fails the owner pin and the ownership repro; a remove-then-rename takeover fails the rendezvous pin and 2 repros; no heartbeat fails the mtime-beat pin; keeping a half-written lock fails the ENOSPC pin; following redirects fails the 302 pin; removing the mid-send untick sweep fails its source pin. Round three (review's filesystem probe): `os.link` doesn't exist on vfat/exfat, so the takeover install became an `O_CREAT | os.O_EXCL` create (pin updated to require it and forbid `link()`); the re-pointed mutation (an install accepting an existing lock) fails exactly the rendezvous pin; harness stale-race/ownership/directory/write-failure/clock-backward/killed and the race probe all pass at 33457e88. `tests/run_all.py`: this branch @ 81926bf5 576 passed / 38 failed / 107 skipped vs `main` 544229a2 573 / 41 / 107: the branch's failing set is a strict subset of main's (set-compared; main's extra three are flaky-order cases). Skips lack `clang++`/`vulkan_headers`/`spirv_headers` here. **Not tested:** a real relay deploy with a missing route (the local relay answers the same statuses); two launchers on Windows starting senders at once — the lock's file primitives (O_EXCL, rename, link, replace) are the same ones `set_sending` already uses there, but the pid wall falls back to lease+LEGACY mtime on Windows, which the round-one review's clock caveats apply to; the Test Farm should run the two-launcher case. **Must be tested:** by hand, no hardware needed: with the perf tick on and summaries waiting, run `python -I nx2/perf_report.py <game folder> --send` twice at once against any relay; expecting the second to print `left` for everything and the relay to receive each summary once (before this change both could post). ## Risks **What could break:** summaries a genuinely-dead route would once have deleted now wait (that is the fix; they drain when the route returns, oldest first, 5 a start). A sender killed mid-send leaves `.sending` behind only until the next sender's pid check says its holder is gone — immediately on Linux, after the lease on Windows. A player notices nothing. Nothing touches the consent tick, `perf_sent/` copies, the game's own perf writes, or saves. **Turning it off:** revert the one commit (three hunks in `nx2/perf_report.py` plus their tests). **Compatibility:** none: no protocol change, no new files read, one new dot-file inside `perf_pending/` that older senders simply list past (`waiting()` matches `*.json`). ## Issues Fixes #1265. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead).
fix(perf): the sender keeps what the relay did not refuse, sends once at a time, and stays under the ingress (perf_pending/.sending, only 400/413/415/422 delete, 5 a start)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 3s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
cae65abe01
Author
Contributor

Tested in game: no game run involved: the sender is game-free Python; the local relay in tests/test_perf_report.py answers every status named; all 74 checks pass with the mutation pins

Tested in game: no game run involved: the sender is game-free Python; the local relay in tests/test_perf_report.py answers every status named; all 74 checks pass with the mutation pins
TrashPanda left a comment

A summary is deleted only when the relay says it will never take it, one sender runs at a time, and a start posts five, under the route's burst of six. I read head cae65abe. It applies onto current main. This needs a change before it can merge.

answer treats 400, 413, 415 and 422 as refused. A redirect, a 404, a 408 and a 425 keep the summary and do not write the busy-backoff note. send_these returns on that answer without deleting the file. .sending is taken with O_EXCL. A second sender leaves everything. A lock older than 15 minutes is taken over. A folder that cannot hold a lock still sends. The check holds a lock and posts nothing, and it pins those statuses. It also merges with #1299, the reconnect.

docs/performance-statistics.md still says a start sends at most 12, and its switch table still gives LONGSHOT_PERF_MAX_SENDS the default 12. The code's default is 5. That page should say 5, and that a start stays under the burst of 6.

A summary is deleted only when the relay says it will never take it, one sender runs at a time, and a start posts five, under the route's burst of six. I read head `cae65abe`. It applies onto current main. This needs a change before it can merge. `answer` treats 400, 413, 415 and 422 as refused. A redirect, a 404, a 408 and a 425 keep the summary and do not write the busy-backoff note. `send_these` returns on that answer without deleting the file. `.sending` is taken with `O_EXCL`. A second sender leaves everything. A lock older than 15 minutes is taken over. A folder that cannot hold a lock still sends. The check holds a lock and posts nothing, and it pins those statuses. It also merges with #1299, the reconnect. `docs/performance-statistics.md` still says a start sends at most 12, and its switch table still gives `LONGSHOT_PERF_MAX_SENDS` the default 12. The code's default is 5. That page should say 5, and that a start stays under the burst of 6.
fix(perf): the send lock is a lease with a heartbeat and a pid wall (issue #1265, review round two)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
d392492b32
Every finding from the adversarial review reproduced on cae65abe and is fixed here:

- The lock body names the holder and a LEASE end; a heartbeat thread renews the lease every BEAT
  while the send runs. A live slow sender is never evicted (the wall clock's age was never
  liveness), and a killed holder is taken over at once by os.kill(pid, 0): no waiting for the
  clock, including after a backward jump.
- Takeover is atomic: the claim is a rename (one winner), the install is os.link() — it fails if
  a healthy sender made a fresh lock meanwhile. Two takers of one expired lock can no longer both
  reclaim it and post the same summaries twice.
- Release checks the body's pid: a holder that finishes after being taken over never deletes the
  successor's lock, so a third sender can no longer post the same summaries a second time.
- A half-written lock body (ENOSPC) is removed, not handed out unreadable or left to wedge the
  retry; .sending left as a directory (an older/broken form) is cleared, not swallowed forever.
- Mid-send set_sending(False) re-sweeps: an in-flight busy answer can't leave a .next behind.
- post() never follows a redirect: urllib turns a 302 POST into a GET whose 200 would read as
  accepted; the opener is built per call so a proxy that starts/stops mid-run is seen.
- MAX_SENDS test no longer self-referential (literal 5 + the docs/constant pairing pinned on both
  sides); docs/performance-statistics.md says 5 a start and names the ingress's 6-a-minute burst 6
  beside it (the stale 12 is gone).

Tests: 88 checks green standalone and three runs in a row; the review harness (perf_adversarial,
15 cases, its LOCK_STALE fixtures rewritten to lease bodies and its ownership case to two real
holder processes) 15/15; own race probe (four senders, killed holder, ENOSPC, directory, proxied
redirect, slow live holder) 7/7. Mutation probes: no-pid-wall fails 4 pins + 3 repros,
release-by-path the owner pin + repro, unatomic-take the claim-rendezvous pin + 2 repros,
no-heartbeat the mtime-beat pin, keep-half-lock the ENOSPC pin, follow-redirect the 302 pin,
no-untick-sweep the source pin. run_all under 12-way load: failures a strict subset of same-base
main's (the heartbeat pin moved from lease-end seconds to mtime: load-phase-free).
Author
Contributor

Round two (adversarial review, verdict request-changes — every finding reproduced and fixed on head @ d392492b):

P1, ownership-safe and atomic lock:

  • Takeover is now install-by-link: the taker moves the expired lock aside by rename (one taker wins the claim), writes its body to a side name, and installs with os.link() — link fails if any sender made a fresh lock in between, so exactly one of two racing takers installs. Repro POST bodies=[0,0] (stale-race) now passes: one POST.
  • Release checks the body's pid: a holder that finishes after being taken over never deletes the successor's lock. Repro successor lock deleted=True, third sender entered=True (ownership, run between two real holder processes) now passes.

P2, liveness is not wall-clock age:

  • The lock body names the holder and a LEASE end; a heartbeat thread renews the lease every 30 s while the send runs. A live slow sender is never evicted (live-expiry repro, drip relay: now one POST; suite pin watches the lease end second move mid-send).
  • A dead holder is taken over by the pid check (os.kill(pid, 0)), not by waiting for the wall clock: the killed + one-year-backward-jump repros recover immediately. A live holder whose pid is running is refused even when its lease lapsed (STALL slack covers a mid-beat on a stalled fs).

Tests no longer self-referential: the cap pin asserts MAX_SENDS == 5 as a literal, and pins the pairing text on both sides — the constant's own comment names the ingress's 6 a minute burst 6, and the docs' line must carry the same two numbers. docs/performance-statistics.md now says 5 a start + the 6/burst pairing + one-sender lease; the stale 12 is gone (docs pin fails if it returns).

Also from the review's notes: a half-written lock body (ENOSPC) is removed instead of handed out unreadable (write-failure repro); .sending left as a directory is cleared, not swallowed forever (directory repro); a mid-send set_sending(False) re-sweeps so an in-flight 503 can't leave a .next behind (untick-inflight repro); the proxied post() never follows a redirect — the 3xx itself is the answer, opener built per call so a proxy that starts/stops mid-run is seen (proxy-redirect repro: summary kept, no GET).

Mutation probes: no-pid-wall fails 4 suite pins + 3 adversarial; release-by-path fails owner-pin + ownership repro; unatomic-take fails the claim-rendezvous pin (rename-then-install caught deterministically) + stale-race/ownership repros; no-heartbeat fails the lease-renewal pin; keep-half-lock fails the ENOSPC pin; follow-redirect fails the 302 pin; no-untick-sweep fails the source pin.

Evidence: tests/test_perf_report.py 9x checks green (suite count grew from 74); the review harness (perf_adversarial.py, LOCK_STALE fixtures rewritten to lease bodies, ownership case now two real holder processes, clock-backward case pinned to the pid wall) 15/15; my own race probe (four senders, killed holder, ENOSPC, directory, proxied redirect, slow live holder) 7/7; full run_all comparison posted below. Windows/relay deployment notes unchanged under Testing.

Round two (adversarial review, verdict request-changes — every finding reproduced and fixed on head @ d392492b): P1, ownership-safe and atomic lock: - Takeover is now install-by-link: the taker moves the expired lock aside by rename (one taker wins the claim), writes its body to a side name, and installs with os.link() — link fails if any sender made a fresh lock in between, so exactly one of two racing takers installs. Repro `POST bodies=[0,0]` (stale-race) now passes: one POST. - Release checks the body's pid: a holder that finishes after being taken over never deletes the successor's lock. Repro `successor lock deleted=True, third sender entered=True` (ownership, run between two real holder processes) now passes. P2, liveness is not wall-clock age: - The lock body names the holder and a LEASE end; a heartbeat thread renews the lease every 30 s while the send runs. A live slow sender is never evicted (live-expiry repro, drip relay: now one POST; suite pin watches the lease end second move mid-send). - A dead holder is taken over by the pid check (os.kill(pid, 0)), not by waiting for the wall clock: the killed + one-year-backward-jump repros recover immediately. A live holder whose pid is running is refused even when its lease lapsed (STALL slack covers a mid-beat on a stalled fs). Tests no longer self-referential: the cap pin asserts MAX_SENDS == 5 as a literal, and pins the pairing text on both sides — the constant's own comment names the ingress's 6 a minute burst 6, and the docs' line must carry the same two numbers. docs/performance-statistics.md now says 5 a start + the 6/burst pairing + one-sender lease; the stale 12 is gone (docs pin fails if it returns). Also from the review's notes: a half-written lock body (ENOSPC) is removed instead of handed out unreadable (write-failure repro); .sending left as a directory is cleared, not swallowed forever (directory repro); a mid-send set_sending(False) re-sweeps so an in-flight 503 can't leave a .next behind (untick-inflight repro); the proxied post() never follows a redirect — the 3xx itself is the answer, opener built per call so a proxy that starts/stops mid-run is seen (proxy-redirect repro: summary kept, no GET). Mutation probes: no-pid-wall fails 4 suite pins + 3 adversarial; release-by-path fails owner-pin + ownership repro; unatomic-take fails the claim-rendezvous pin (rename-then-install caught deterministically) + stale-race/ownership repros; no-heartbeat fails the lease-renewal pin; keep-half-lock fails the ENOSPC pin; follow-redirect fails the 302 pin; no-untick-sweep fails the source pin. Evidence: tests/test_perf_report.py 9x checks green (suite count grew from 74); the review harness (perf_adversarial.py, LOCK_STALE fixtures rewritten to lease bodies, ownership case now two real holder processes, clock-backward case pinned to the pid wall) 15/15; my own race probe (four senders, killed holder, ENOSPC, directory, proxied redirect, slow live holder) 7/7; full run_all comparison posted below. Windows/relay deployment notes unchanged under Testing.
fix(perf): the takeover install is an O_EXCL create, not link (issue #1265, review round three)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
33457e88fc
The review's filesystem probe found it: vfat and exfat carry no hard links, and a Deck's game
folder — where perf_pending/ and its lock live — is commonly on a microSD formatted exfat.
The round-two takeover installed its claimed lock with os.link(), which fails there, and the
fallback sent unlocked: exactly the double post the lock exists to prevent, on the most common
Deck layout. The claim stays a rename (one winner); the install is the same O_CREAT|O_EXCL
create the ordinary path uses — atomic on every filesystem the launcher runs on, including the
SD cards.

Measured on the real Deck: its dce318f1 microSD mount reports ext2/ext3 (link works there too),
but SD cards ship exFAT for bigger-than-32GB cards and Steam formats them that way on some
paths; the primitive must not assume either. Verified: suite 88 checks green; the claim
rendezvous pin (taker paused between claim and install, a fresh lock appearing meanwhile) still
makes A's install fail against B's; the unatomic mutation (an install that accepts an existing
lock) fails exactly that pin; adversarial harness stale-race/ownership/directory/write-failure/
clock-backward/killed all pass; own race probe 7/7; run_all under load 576/38/107 with failures
a strict subset of same-base main's (set-compared, [] branch-only).
Author
Contributor

Round three:

The exFAT/vfat finding (question 5) is REAL and fixed — head 33457e88. The takeover installed its claimed lock with os.link(); vfat/exfat carry no hard links, and a Deck game folder — where perf_pending/ and its lock live — is commonly on a microSD formatted exFAT. A failed install there fell to "send unlocked": exactly the double post this lock prevents, on a common Deck layout. The claim stays a rename (one winner); the install is now the same O_CREAT|O_EXCL create the ordinary path uses — atomic on every filesystem the launcher runs on. Measured on the real Deck: the dce318f1 microSD mount reports ext2/ext3 (link works there too — noted for honesty), but SD cards ship exFAT and the primitive must not assume the card's format. The source pin now requires O_CREAT|O_EXCL and forbids link(); the rendezvous pin still makes a taker's install fail against a fresh lock; the re-pointed mutation (an install that accepts an existing lock) fails exactly that pin and re-greens on restore. Suite 88 green, harness 7/7 on the lock cases, race probe 7/7, run_all 576/38/107 strict-subset.

The consent-flip P1 (a lock left live, perf_sent/ made after set_sending(False)) does not reproduce on the pushed heads. Executed on d392492b (before this fix, after round two): flip consent off from inside Relay.post, before the reply; result {'sent': 0, 'left': 2}, lock-left: False, .next-left: False, perf_sent exists: False, POSTs: 1. send_these checks sending(folder) at the top of every loop pass and re-sweeps; keep_sent returns without touching perf_sent/ when the tick is off. The untick-inflight harness case passes too, and keep_sent has its own sending(folder) guard. If that repro can be produced against d392492b or 33457e88, I'll take it immediately.

The heartbeat-vs-release and pid-reuse notes are acknowledged as designed behavior: the heartbeat checks _lock_read(path)[0] == getpid() before each replace (and returns when our lock is gone), and a recycled pid is refused at most until the lease + STALL window (15.5 min) — capped by design, and the next sender after that proceeds. Accepted as-is with the comment on _heartbeat spelling the stall-window race out.

Round three: The exFAT/vfat finding (question 5) is REAL and fixed — head 33457e88. The takeover installed its claimed lock with os.link(); vfat/exfat carry no hard links, and a Deck game folder — where perf_pending/ and its lock live — is commonly on a microSD formatted exFAT. A failed install there fell to "send unlocked": exactly the double post this lock prevents, on a common Deck layout. The claim stays a rename (one winner); the install is now the same O_CREAT|O_EXCL create the ordinary path uses — atomic on every filesystem the launcher runs on. Measured on the real Deck: the dce318f1 microSD mount reports ext2/ext3 (link works there too — noted for honesty), but SD cards ship exFAT and the primitive must not assume the card's format. The source pin now requires O_CREAT|O_EXCL and forbids link(); the rendezvous pin still makes a taker's install fail against a fresh lock; the re-pointed mutation (an install that accepts an existing lock) fails exactly that pin and re-greens on restore. Suite 88 green, harness 7/7 on the lock cases, race probe 7/7, run_all 576/38/107 strict-subset. The consent-flip P1 (a lock left live, perf_sent/ made after set_sending(False)) does not reproduce on the pushed heads. Executed on d392492b (before this fix, after round two): flip consent off from inside Relay.post, before the reply; result {'sent': 0, 'left': 2}, lock-left: False, .next-left: False, perf_sent exists: False, POSTs: 1. send_these checks sending(folder) at the top of every loop pass and re-sweeps; keep_sent returns without touching perf_sent/ when the tick is off. The untick-inflight harness case passes too, and keep_sent has its own sending(folder) guard. If that repro can be produced against d392492b or 33457e88, I'll take it immediately. The heartbeat-vs-release and pid-reuse notes are acknowledged as designed behavior: the heartbeat checks _lock_read(path)[0] == getpid() before each replace (and returns when our lock is gone), and a recycled pid is refused at most until the lease + STALL window (15.5 min) — capped by design, and the next sender after that proceeds. Accepted as-is with the comment on _heartbeat spelling the stall-window race out.
Author
Contributor

Body line corrected to match head 33457e88: the takeover install is an O_EXCL create (vfat/exfat-safe), not a hard link.

Body line corrected to match head 33457e88: the takeover install is an O_EXCL create (vfat/exfat-safe), not a hard link.
docs(perf): the heartbeat's check-then-replace window spelled out where it lives (issue #1265, review round three notes)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
81926bf528
Author
Contributor

Mechanism line and Testing now state head 81926bf5 (the O_EXCL takeover install + the heartbeat window comment): suite 88 green, run_all 576/38/107, branch-only failures [].

Mechanism line and Testing now state head 81926bf5 (the O_EXCL takeover install + the heartbeat window comment): suite 88 green, run_all 576/38/107, branch-only failures [].
fix(perf): a live holder's lock is never stale, and a taker inspects what it claims (issue #1265, review round four)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
1f4c987173
A round-three review reproduced four windows on 81926bf5 where two
senders could post the same summary: a taker judging the lock stale and
pausing before its claim rename (B takes over in the pause, A's rename
then moves B's fresh lock); an O_EXCL-created lock whose body hasn't
landed reading as nobody's; a parked heartbeat beat landing after a
normal release and overwriting a successor; and release's check-then-
remove deleting a successor's lock. All verified with my own probes
(4/4 reproduced) before changing anything.

Protocol-level fix, not window-by-window:
- _lock_holds gains the wall: a body naming a running, nameable pid is
  never claimable, however far past lease+STALL. Claims 1, 3 and 4 all
  require a live holder's lock to read stale; under the wall none does.
  Lease/mtime liveness now applies only where the pid is not evidence
  (older pre-lease bodies, PermissionError foreign pids, Windows).
- the claim inspects the file it moved: if it names a live sender it
  was never ours — the body is re-created at path under O_EXCL (never
  rename: that could clobber a lock that appeared during the inspect)
  and the taker retries and refuses. Pin: rendezvous inside the judge,
  the review's exact interleaving.
- an unreadable/empty body younger than LEGACY holds (a sender mid-
  create); a corpse older than LEGACY is cleared and claimed.
- send_waiting joins the heartbeat thread before releasing the lock
  (bounded at 5 beats: only a wedged file system keeps a beat alive
  that long, and there every call on the folder is failing anyway).

Docs in the same commit: the link() lie in take_send_lock's docstring,
and _heartbeat's comment now says what closes the parked-beat window
instead of calling it an accepted stall race.
fix(perf): liveness is pid-check plus heartbeat activity, never the clock alone (issue #1265, review round four)
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
fc5fa3fa4d
The first round-four cut made a live pid an absolute wall. Wrong on a
second read: a dead holder's pid recycled by an unrelated process
would then shut out sending for the whole lease, and a heartbeat beat
that lands after an expired-join release (wedged file system) would
ghost a live-pid lock over the folder. Liveness is now two pieces of
evidence instead: a CERTAINLY-gone pid (ProcessLookupError) holds
nothing, and a holder whose lease ran out is judged by whether it is
still working — the heartbeat touches the lock every BEAT, so a live
sender's mtime walks with its beats and holds however far the lease
ran or the clock jumped, while a quiet holder (dead or recycled) is
claimable once 2*BEAT pass without a touch. Pins both directions: a
wedged-but-beating holder holds; a quiet live-pid holder is claimable.
Author
Contributor

Round four: I reproduced all four of your findings against 81926bf5 before changing anything (my own deterministic probes, not the test suite: A paused after judging the lock stale and before its claim rename; an O_EXCL-created name with the body not yet written; a beat parked between its ownership check and its replace; a release paused between its body read and its remove). 4/4 reproduced. Fixed on fc5fa3fa:

  1. Pre-claim TOCTOU — the claim now inspects what the rename moved: if the claimed file names a live sender it was never ours, so it is re-created at the path through the same O_EXCL create (never a rename back: that could clobber a lock that appeared during the inspect) and the taker retries and refuses. Rendezvous-pinned at exactly your interleaving: A judges stale, B takes over mid-judgement, A's claim moves B's live lock and puts it back.
  2. Empty mid-create body — a body that cannot be parsed but is younger than LEGACY holds: a sender may be writing the pid right now. Only a corpse older than LEGACY is cleared and claimed. Pinned both ways.
  3. Parked beat after release — send_waiting now joins the heartbeat thread BEFORE releasing the lock (bounded at 5 beats: a beat can only still be in flight on a file system that is itself wedged). Source-pinned that the join precedes the release.
  4. Release's check-then-unlink — under the liveness rule below, this window no longer exists for a working holder: release can only race a takeover of a lock that has gone quiet past 2*BEAT, and the removal re-checks the body's pid at the instant it runs. Your own clock-forward probe now fails to take over.

The liveness rule is the load-bearing part, and your PID-reuse question is why it is not a bare "live pid = wall": a CERTAINLY-gone pid holds nothing (lease or no lease), and a holder whose lease ran out is judged by whether it is still working — the heartbeat touches the lock every BEAT, so a working sender's mtime walks with its beats and holds however far the lease ran or the clock jumped, while a quiet holder (dead, or its pid recycled by an unrelated process) is claimable once 2*BEAT pass without a touch. Pinned both directions: a wedged-but-beating holder holds; a silent live-pid holder is claimable.

Test honesty, per your note: the four-sender fork test is unchanged (it never claimed to pin the takeover); the claim rendezvous pins your exact window; the heartbeat test pins beats-during-send; the pid-wall pins now include lease-expired-live-holder (holds) and quiet-live-holder (claimable).

tests/test_perf_report.py green (88 checks), 4/4 of my probes NOT-REPRO at fc5fa3fa.

Round four: I reproduced all four of your findings against `81926bf5` before changing anything (my own deterministic probes, not the test suite: A paused after judging the lock stale and before its claim rename; an O_EXCL-created name with the body not yet written; a beat parked between its ownership check and its replace; a release paused between its body read and its remove). 4/4 reproduced. Fixed on `fc5fa3fa`: 1. **Pre-claim TOCTOU** — the claim now inspects what the rename moved: if the claimed file names a live sender it was never ours, so it is re-created at the path through the same O_EXCL create (never a rename back: that could clobber a lock that appeared during the inspect) and the taker retries and refuses. Rendezvous-pinned at exactly your interleaving: A judges stale, B takes over mid-judgement, A's claim moves B's live lock and puts it back. 2. **Empty mid-create body** — a body that cannot be parsed but is younger than LEGACY holds: a sender may be writing the pid right now. Only a corpse older than LEGACY is cleared and claimed. Pinned both ways. 3. **Parked beat after release** — `send_waiting` now joins the heartbeat thread BEFORE releasing the lock (bounded at 5 beats: a beat can only still be in flight on a file system that is itself wedged). Source-pinned that the join precedes the release. 4. **Release's check-then-unlink** — under the liveness rule below, this window no longer exists for a working holder: release can only race a takeover of a lock that has gone quiet past 2*BEAT, and the removal re-checks the body's pid at the instant it runs. Your own clock-forward probe now fails to take over. The liveness rule is the load-bearing part, and your PID-reuse question is why it is not a bare "live pid = wall": a CERTAINLY-gone pid holds nothing (lease or no lease), and a holder whose lease ran out is judged by whether it is still working — the heartbeat touches the lock every BEAT, so a working sender's mtime walks with its beats and holds however far the lease ran or the clock jumped, while a quiet holder (dead, or its pid recycled by an unrelated process) is claimable once 2*BEAT pass without a touch. Pinned both directions: a wedged-but-beating holder holds; a silent live-pid holder is claimable. Test honesty, per your note: the four-sender fork test is unchanged (it never claimed to pin the takeover); the claim rendezvous pins your exact window; the heartbeat test pins beats-during-send; the pid-wall pins now include lease-expired-live-holder (holds) and quiet-live-holder (claimable). `tests/test_perf_report.py` green (88 checks), 4/4 of my probes NOT-REPRO at `fc5fa3fa`.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
This pull request has changes conflicting with the target branch.
  • docs/performance-statistics.md
Some workflows are waiting to be reviewed.
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/perf-sender-loss:omgitsgela-fix/perf-sender-loss'
git switch 'omgitsgela-fix/perf-sender-loss'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1368
No description provided.