WIP: fix(os): HOS_CHAIN's flag is an atomic, and the recorder writes a record only once its note has filled it #2019
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2019
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:fix/os-chain-flag-atomic"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
HOS_CHAIN=<path>(host/hos_chain.c, a diagnostic forscripts/perf.py chain) records the guest threads' waits andwake-ups between
<path>.goappearing and going away. Its on/off flag,hos_chain_on, was avolatile int: therecorder thread set and cleared it, a note at the buffer's capacity cleared it, and every wait and signal read it, with
no synchronisation, and nothing published the record buffer or the records to the thread that writes the file. At the
stop the recorder cleared the flag, slept 50 ms and wrote every record a note had taken, whether or not that note had
filled it in yet: a note whose thread the system had not run again for 50 ms left a record of zeros in the file.
This makes the flag an atomic: set with release once the buffer is allocated, read with acquire in
hos_chain_notebefore it uses the buffer, read relaxed by the
HOS_CHAINmacro and the six waits in host/nn_os_sync.c that decidewhether to call it. A note stores its record's kind last, with release, and the recorder waits (acquire) for the kind of
every record taken before the stop, at most 5 s in all, instead of sleeping 50 ms; the thread table it names threads
from is stored and read atomically too. The result: the file holds only whole records. With
HOS_CHAINunset nothingchanges but the flag's load, which is the same instruction on x86-64.
How to see it on
main:python -I -B tests/test_chain_recorder.pywith the test commit: a note held after it hastaken its record while the recording stops leaves a record of zeros in the file (see Testing).
Found by BeanBot's review of #1953 (Blocking 4,
volatile hos_chain_on),which #1953 left on
mainfor a separate change.Read first:
hos_chain_note,chain_writeandchain_threadinhost/hos_chain.c;tests/test_chain_recorder.py.Overlap: #1953 (
perf/mutex-lock-wait-cold@8443bba1) moves one of the six reads (const int chain = hos_chain_on;in
mutex_lock) into its newmutex_lock_wait; the two conflict on that one line (git merge-tree), and #1957, whichcarries #1953's commit, does too. Proposed order: this first (#1953 waits for its in-game measurements); #1953 and #1957
are then rebased, the moved line reading the flag as this PR does. If #1953 lands first, this PR is rebased instead.
Testing
Tested: Windows 11, i7-9700, RTX 4060 (NVIDIA 610.88), llvm-mingw clang 23.1.5; game-free checks only. This has
NOT run in the game on this PC (Still to test). Each check ran in a hidden console of its own, in build jobs of the PC's
job queue (fb3-sep-1, then fb3-sep-2 after the fixture fix below), on this branch on
main@f1f2dc5b, head 2aed1932. Then rebased ontomain@40bfceb0without a conflict, its twocommits unchanged (
git range-diff), nowf74e2760(test) anda6fffab8; the checks have not run again on that base (of the files this changes,mainchanged onlyspin_begininhost/nn_os_sync.csince (#2012), away from these lines):python -I -B tests/test_chain_recorder.py: "PASS: 4001 records of 5 threads, each whole, the note held across thestop among them; nothing before the recording". At the test commit 47c61625 (
main's recorder) it fails: "1 record(s)not whole, the first record 4000: time 0, key 0x0, thread 0, kind 0 (written while its note was still filling it);
the held note's record is not in the file whole". (fb3-sep-1 ran the test's first form, whose stub did not declare
hos_chain_start; added.)host/hos_chain.c,host/nn_os_sync.c,host/hos_thread.c,host/nvn_native.candhost/oot_barrier.c(theHOS_CHAINusers) compiled (-fsyntax-only) with the host build's own flags, with and without Tracy: no error.python -I -B tests/run_all.py --strict --changed origin/main(43 checks read the changed files): 36 passed, 7 failed,1 skipped (
test_tracy_funcs.py: Linux only). The 7 failing (test_crash_record.py,test_dev_packs.py,test_host_guest_writes.py,test_libc_hooks.py,test_mods_list.py,test_slow_frames.py,test_vcore_escape.py): they fail onmain@f1f2dc5btoo, run by name from amainworktree in the same job, with the same messages.Still to test:
HOS_CHAIN=chain.bin,chain.bin.gomade for about 20 s and removed, then
python scripts/perf.py chain chain.bin, expecting the "chain: N records of Mthreads written" line, a report like
main's and no record of kind 0; and two minutes withHOS_CHAINunset, thisPR against
main, expecting frame times within the runs' spread (by hand)python3 -I -B tests/test_chain_recorder.py, expecting it to pass (by hand)mainwithHOS_CHAINunset, expecting no stall and frame times within the runs' spread (the Test Farm: play Castle Town's market for two
minutes on Linux with an AMD CPU and the Steam Deck, this PR against main, comparing frame times)
Measurements
Nothing to measure: with
HOS_CHAINunset each wait still loads one flag and tests it (an atomic relaxed load, the same instruction on x86-64); the recorder's own cost only shows while a diagnostic recording runs.Risks
Only
HOS_CHAINrecordings change: the stop now waits for notes under way (normally microseconds; at most 5 s, then thefile ends before the first unfinished record and the log says so) instead of sleeping 50 ms, and a recording's file
can no longer hold a record of zeros. Off (no
HOS_CHAIN), the waits read the flag as before. No switch of its own; arevert gives back
main's code. Players notice nothing after updating.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Recommendation: Request changes. The readiness flag fixes the zero-filled record reproduced by the Windows fixture, but the recorder can still dereference a destroyed guest thread while writing its name. Stop and slot reservation also need a common cutoff so a late producer cannot delay or alter the capture after it closes.
This makes the diagnostic flag atomic and publishes each record after its fields are ready. That improves capture integrity without adding a lock to the waits or presents that check the flag. When recording is enabled, the new stop path scans the capture and can wait up to five seconds, then discard completed records after the first unfinished slot.
@fixwill do (20)host/hos_chain.c:85: The atomic load protects the pointer slot but leaves the pointed-to thread unowned. (more on the line)host/hos_chain.c:86: A live thread name can still race with SetThreadName. (more on the line)host/hos_chain.c:43: Checking the flag and reserving a slot are separate operations. (more on the line)host/hos_chain.c:111: Removing .go from an active capture with zero reservations leaves the flag set. (more on the line)host/hos_chain.c:73: Keep the original cutoff and inspect the remaining slots after the deadline. Report the omitted count and preserve completed records if partial captures are supported, or report a failed capture.host/hos_chain.c:47: The 1,025th distinct producer shares a thread ID with the 1,024th. (more on the line)host/hos_chain.c:88: An empty guest thread name produces a two-field header that the real reader rejects. (more on the line)host/hos_chain.c:92: The writer ignores header, payload and close errors and logs success even if a full disk leaves a truncated file. (more on the line)host/hos_chain.c:100: The marker path silently truncates output paths longer than 1,020 bytes. A valid path in a deep Linux directory can therefore fail to start recording or poll a different marker. Size the buffer from the path and suffix, or reject an overlong path with a clear diagnostic before polling.host/hos_chain.c:104: Any marker open failure is treated as removal. (more on the line)host/hos_chain.c:115: The one-shot recorder keeps its buffer and continues waking every 100 ms to open the marker for the rest of the process, even after a failed write. (more on the line)host/hos_chain.c:123: CreateThread's result is ignored. If thread creation fails, a requested capture silently does nothing. If it succeeds, the handle is leaked on Windows and retains the shim's handle reference on Linux. Log failure and close the successful handle if the thread is detached.tests/test_chain_recorder.py:76: The fixture gives the recorder pointers into each worker's TLS, then joins the workers before the recorder reads their names. (more on the line)tests/test_chain_recorder.py:111: The held note releases itself after two seconds even if the recorder has never observed the stop. (more on the line)tests/test_chain_recorder.py:223: The Windows fixture omits the UTF-8 manifest that Longshot uses for narrow argv, getenv and fopen. (more on the line)tests/test_chain_recorder.py:41: The fixture treats CC as a single executable. (more on the line)tests/test_chain_recorder.py:145: On Linux a failed pthread_create leaves join with an uninitialized thread handle. (more on the line)tests/test_chain_recorder.py:233: The fixture checks the held record's identity but accepts duplicate worker records, any valid kind for worker events and any caller address. (more on the line)docs/dev/reference/profiling.md:120: The reference says the capture is written when the marker goes away or the buffer fills, without describing delayed or partial output. A developer who removes the marker and immediately quits can miss the file while it waits for a note. Document the final timeout and output policy and the completion log to wait for before exiting.For the author (3) (a run, a measurement or a decision)
host/hos_chain.c:73: The deadline throws away every later record, including whole records from unrelated threads. (more on the line)Checks: Still running:
CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request).More: does it help, other systems, switches, what was read
Does it help: Yes for the reported Windows reproduction. On the earlier base, the test-only commit wrote one zero-filled record out of 4,001. The earlier fixed head wrote all 4,001 whole records from five threads. The current base and head have no reported rerun, and in-game behavior and frame pacing remain unverified.
Other systems: The flag is read by guest synchronization, virtual-core scheduling, sleep, the job barrier and NVN presents. Their lock and wake paths are unchanged in the diff. Windows and Linux share the recorder, with Linux providing the Win32 thread and sleep shims. The binary layout and consumer are unchanged, but the consumer still depends on valid thread headers and a complete file. The change has no direct GPU-vendor, renderer ABI, audio, input, save, cache or Launcher changes. Windows syntax checks with and without Tracy were reported on the earlier head. Linux, Steam Deck and in-game frame pacing are pending.
Switches: No switch was added or removed. HOS_CHAIN remains an opt-in developer diagnostic and defaults to off, which is appropriate. Its enabled stop behavior changes. The existing switch-table row still describes the default correctly, and the profiling reference needs the new stop behavior.
Reviewed:
a6fffab8d7against40bfceb055: Read all four changed files and both commits, plus AGENTS.md, CONTRIBUTING.md, the Forgejo template and developer standards. No docs/review-checklist.md was present. Read the relevant startup, thread lifetime and naming, macro caller, POSIX shim, capture reader and test-runner code. Skimmed the reported Windows runs, CI status and the earlier chain-reader fix. No generated or translated files changed.; nothing run (a code review).Checked and fine:
Testing: Read-only review. The author reported a Windows game-free failure on the earlier base and a pass on the earlier fixed head, plus Windows syntax checks with and without Tracy. Seven reported failures also occurred on that earlier base. The supplied current CI has checks still running and lists no head-only failures.
Overlaps and merge order: The PR reports one conflicting mutex-wait flag read in #1953 and in #1957, which carries that commit. Land #2019 first, then rebase both and keep the atomic read at its new location. If #1953 lands first, rebase #2019. The overlap and merge order come from the PR description.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -36,3 +41,3 @@void hos_chain_note(int kind, uint64_t key, uint64_t lr) {if (!hos_chain_on) return;if (!__atomic_load_n(&hos_chain_on, __ATOMIC_ACQUIRE)) return;const LONG64 i = InterlockedIncrement64(&nrecs) - 1;L3: Checking the flag and reserving a slot are separate operations. A producer can pass the acquire load, pause, then reserve after the recorder clears the flag. If it reserves just before the recorder snapshots the count and pauses again, the recorder can wait five seconds for a slot admitted after close. Put reservation and close under one atomic state or lock, then take the final count from that state.
@ -40,3 +44,4 @@if (i >= CHAIN_CAP) { __atomic_store_n(&hos_chain_on, 0, __ATOMIC_RELAXED); return; }if (my_index < 0) {const LONG k = InterlockedIncrement(&nthreads) - 1;my_index = k < CHAIN_THREADS ? (int)k : CHAIN_THREADS - 1;L6 (minor): The 1,025th distinct producer shares a thread ID with the 1,024th. A capture with enough short-lived threads can pair one thread's wait with another thread's wake and attribute both to the same name. Stop admitting records with unrepresentable thread IDs and report the limit, or size the table for the record format.
@ -67,2 +85,4 @@HosThread *t = __atomic_load_n(&threads[i], __ATOMIC_RELAXED);if (t) snprintf(name, sizeof name, "%s", hos_thread_name(t));else snprintf(name, sizeof name, "host-%d", i); /* (a host thread: the vsync timer, the renderer's) */for (char *q = name; *q; q++) if (*q == ' ' || *q == '\n') *q = '_';L7 (minor): An empty guest thread name produces a two-field header that the real reader rejects. A tab in a name can also cause the reader to drop part of it. Use a fallback for empty names, sanitize whitespace consistently with the reader and check the output with the real reader.
@ -60,0 +70,4 @@if (hos_now_ns() > give_up) {hos_log("chain: record %lld of %lld still not filled in after 5 s: the file ends before it",(long long)i, (long long)n);n = i;L5 (minor): The deadline throws away every later record, including whole records from unrelated threads. If slot zero is held while other threads record frames, the file can contain zero records. Choose whether a timeout should fail the capture explicitly or preserve whole records with a clear indication that events were lost.
L8: The writer ignores header, payload and close errors and logs success even if a full disk leaves a truncated file. Check the writes and fclose before reporting success. Write to a sibling temporary file and publish it only after a successful close so a failed capture cannot replace a valid one.
@ -64,3 +83,3 @@for (int i = 0; i < nt; i++) {char name[64];if (threads[i]) snprintf(name, sizeof name, "%s", hos_thread_name(threads[i]));HosThread *t = __atomic_load_n(&threads[i], __ATOMIC_RELAXED);L1: The atomic load protects the pointer slot but leaves the pointed-to thread unowned. A worker can record an event, exit and be freed by DestroyThread before the recorder formats its name. Keep a recorder-owned name snapshot or resolve a stable thread ID through a lifetime-safe API, with a fallback for exited threads.
L10 (minor): Any marker open failure is treated as removal. An existing marker that temporarily cannot be opened, for example because another Windows process opened it without sharing, can end a capture. Query marker existence without opening it for reading and distinguish absence from other errors.
@ -65,2 +84,3 @@char name[64];if (threads[i]) snprintf(name, sizeof name, "%s", hos_thread_name(threads[i]));HosThread *t = __atomic_load_n(&threads[i], __ATOMIC_RELAXED);if (t) snprintf(name, sizeof name, "%s", hos_thread_name(t));L2: A live thread name can still race with SetThreadName. The acquire of a record's kind orders publication of that record, while another guest thread can rewrite the plain name array during snprintf. Copy the name through an accessor synchronized with the setter, and give the file writer ownership of the resulting bytes.
@ -95,2 +112,4 @@(!g || !__atomic_load_n(&hos_chain_on, __ATOMIC_RELAXED))) {__atomic_store_n(&hos_chain_on, 0, __ATOMIC_RELAXED);chain_write(path);done = 1; /* (one recording a run) */L11 (minor): The one-shot recorder keeps its buffer and continues waking every 100 ms to open the marker for the rest of the process, even after a failed write. Exit the recorder after its final result and reclaim the buffer once admitted producers have drained. If the write times out, defer reclamation until those producers are done.
@ -93,2 +110,2 @@hos_chain_on = 0;Sleep(50); /* (notes under way finish) */__atomic_store_n(&hos_chain_on, 1, __ATOMIC_RELEASE);} else if (!done && __atomic_load_n(&nrecs, __ATOMIC_RELAXED) &&L4: Removing .go from an active capture with zero reservations leaves the flag set. If the first event happens seconds later, it enters a capture that should already have ended. Track an explicit active state and close it on marker removal even with zero records. Use the same state to stop when the last available slot is reserved, since a full buffer currently needs one more note to turn recording off.
@ -0,0 +38,4 @@clone's folder), else clang, gcc or cc on PATH"""for c in (given, os.environ.get("CC")):if c:return shutil.which(c) or cL16 (minor): The fixture treats CC as a single executable. A setting such as CC='ccache clang' fails with an executable lookup error even though the toolchain is installed. Preserve direct executable paths, and parse command-style CC values into an argument list for both compile steps.
@ -0,0 +73,4 @@#include <time.h>#endifstruct HosThread { char name[32]; };static _Thread_local struct HosThread me;L13: The fixture gives the recorder pointers into each worker's TLS, then joins the workers before the recorder reads their names. Those objects have ended their lifetime. Linux can reuse the TLS storage for the held thread, and Windows can release it. Use fixture-owned storage that stays live through the write, and cover exited-thread handling separately.
@ -0,0 +108,4 @@hold_next = 0;__atomic_store_n(&held, 1, __ATOMIC_SEQ_CST);const int64_t t0 = mono_ns();while (!__atomic_load_n(&written, __ATOMIC_SEQ_CST) && mono_ns() - t0 < 2000000000ll) Sleep(1);L14: The held note releases itself after two seconds even if the recorder has never observed the stop. On a loaded runner the old code or a build with the wait deleted can then pass. Gate release on an observed stop and an explicit fixture handshake. Make missing handshakes fail, use monotonic deadlines for the fixture waits, and verify the wait-deleted build fails.
@ -0,0 +142,4 @@static void join(Thread t) { WaitForSingleObject(t, INFINITE); }#elsetypedef pthread_t Thread;static Thread start(void *(*fn)(void *), void *arg) { pthread_t t; pthread_create(&t, NULL, fn, arg); return t; }L17 (minor): On Linux a failed pthread_create leaves join with an uninitialized thread handle. The Windows path also ignores creation and wait failures and leaves worker handles open, and marker removal is unchecked on both. Check these operations and report the specific setup failure before interpreting recording results.
@ -0,0 +220,4 @@str(ROOT / "host" / "posix" / "win32_posix.c"), "-o", str(shim)], check=True)cmd[1:1] = ["-I", str(ROOT / "host" / "posix")]cmd += [str(shim), "-pthread", "-ldl"]subprocess.run(cmd + ["-o", str(exe)], check=True)L15: The Windows fixture omits the UTF-8 manifest that Longshot uses for narrow argv, getenv and fopen. An emoji in the temporary-directory path can make the child fail before it starts recording on a system using a legacy code page. Link the existing manifest resource and run the fixture from a deliberately non-ASCII directory.
@ -0,0 +230,4 @@raise SystemExit("test_chain_recorder: FAILED: the fixture exited %d" % r.returncode)names, recs = records(out)want = WORKERS * NOTES + 1bad = [(i, rec) for i, rec in enumerate(recs) if not (1 <= rec[4] <= 6 and rec[0] and rec[1] and rec[3] in names)]L18 (minor): The fixture checks the held record's identity but accepts duplicate worker records, any valid kind for worker events and any caller address. Its pre-recording probe also runs before the recorder starts. Compare the expected worker keys, kinds, names and a nonzero caller address, and exercise the started recorder while the marker is absent.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.