WIP: fix(runtime): the log writer's flush no longer deadlocks with the writer; the writer runs on Linux too #1284
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1284
Loading…
Reference in a new issue
No description provided.
Delete branch "Molier/Longshot:linux/async-log"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: medium · Tested on: Linux (CachyOS, kernel 7.4.0-rc1), Radeon RX 9070 XT, RADV (Mesa 26.2.6): game-free tests, and a local build with the writer on · Needs: Windows, any GPU (
log_flush()is shared code)Why
On Linux every line the game logs is written to stderr by the thread that logs it, one write per line, unbuffered. Windows has had a writer thread that queues and batches those lines, but its code was gated to
_WIN32, so on Linux that writer never ran. The per-frame lines used for performance work (RX_VK_SLOW_LOG,HOS_FPS_LOG,HOS_THREAD_TIMES) are logged by the replay thread, so on Linux a slow disk holds that thread and the frame. Windows' own comment inhost/hos_thread.crecords the render thread held 140-310 ms in one line's write before its writer existed. This change enables the existing writer on Linux. No tracker issue fits.How to reproduce
Seen on: main, Linux, any disk
Settings and switches: defaults
Steps:
slowcase oftests/log_batch_test.cwith the writer off (HOS_LOG_SYNC=1, main's Linux behaviour): 100 lines against a 50 ms sink.inversioncase. Main'slog_flush()takes the write lock then the queue lock;log_main()takes the queue lock then the write lock.Expected: the calling thread returns in well under a millisecond per line; a flush during a write completes.
Actual:
slow: 100 lines in 5110.33 ms(writer off);DEADLOCKin 12 of 12 runs ofinversion(exit 3).What changed
Enable the existing log writer on Linux and fix a lock-order deadlock in its flush that Windows also has. Document the two existing log switches.
host/hos_thread.c: the writer gate is(defined(_WIN32) || defined(__linux__)) && !defined(__EMSCRIPTEN__). Every call it needs has a Linux shim inhost/posix: SRW locks, condition variables,CreateThread, the unhandled-exception filter (the crash signals),GetStdHandle/WriteFile,GetTickCount64. Emscripten keeps its stub; batching and the kill switch are unchanged.host/hos_thread.c,log_flush(): take the queue lock then the write lock (the writer's order), and copy the batch into a local array (a shared static one raced between two flushes). This lock-order fix is in shared code that Windows also runs.report_crash, installed byhos_commit_initafter the log's own filter, ends inhos_log_flush(); the crash signals installed bySetUnhandledExceptionFilter(SEGV, BUS, ILL, FPE, ABRT) reach it. The explicit flush sites (hos_fatal,svcExitProcess, window close,hos_relaunchbeforeexecve) are shared and unchanged.docs/development.md: rows forHOS_LOG_SYNC=1andHOS_LOG_BATCH_MS=<ms>, previously undocumented.tests/log_batch_test.c,tests/test_log_batch.py: 12 game-free cases, now runnable on Linux.Measurements
Verdict: the deadlock is fixed: the
inversioncase oftests/log_batch_test.cdeadlocks with main's lock order (12 of 12 runs, measured before the rebase) and passes here. Onmain,log_main()takeslog_lockand thenlog_write_lockwhile it still holdslog_lock;log_flush()takeslog_write_lockand thenlog_lock. In the game on Linux the writer brought no frame-time gain: the median is the same, and the tail figures are higher with the writer on (table below); whether that is beyond run-to-run variation is not established.Baseline: one build of nightly-338 (main @
e9e7cf0b) plus local patches (this change among them), withHOS_LOG_SYNC=1in one arm and unset in the other. It is not main against this PR; that run is listed under Must be tested.Hardware: Linux (CachyOS, kernel 7.4.0-rc1), Ryzen 7 5800X, 32 GB, Radeon RX 9070 XT, RADV (Mesa 26.2.6), gamescope
Scene: "heavy" = Castle Town market -> Hyrule Castle -> Hyrule Field, twice, capped 60; 3+3 interleaved runs (6 legs), 21,600 frames pooled per arm
Method: per-frame times from the heavy benchmark's frame-time logs; async writer on against off (
HOS_LOG_SYNC=1) on the same build, after a warm-up; 3+3 interleaved runs, median reported per leg and pooledHow to reproduce: the heavy route at capped 60, one arm with
HOS_LOG_SYNC=1, the other with it unset, on the same buildHOS_LOG_SYNC=1(old: per-line writes, no writer)HOS_LOG_SYNCunset (writer on)Per leg (median of 3 runs each): 1 of 6 legs differs beyond its runs' spread (Hyrule Castle, second pass: 3.7 % faster with the writer on); the other five are within ±3.4 % either way. No A/A pair of an unchanged configuration was run, so the spread of the tail figures is not known.
Flags and settings
No new switches. Two existing ones are documented; neither ships with a changed default.
HOS_LOG_SYNC=1HOS_LOG_BATCH_MS=<ms>Testing
Tested: Linux (CachyOS, kernel 7.4.0-rc1), Ryzen 7 5800X, Radeon RX 9070 XT, RADV (Mesa 26.2.6), gamescope : game-free tests, and in the game the A/B in Measurements (the local build ran both benchmark routes with the writer on; no run hung).
python -I -B tests/test_log_batch.py: 12 of 12 cases pass (batch,default,off,burst,flush,threads,slow,sync,exit,crash,crash_locked,inversion); the same test against main'shost/hos_thread.cfails ("log queue block not found"). Stress under load (8 runs at once, 6 CPU-bound helpers):crash100 of 100,crash_locked100 of 100,threads300 of 300,exit100 of 100,slow20 of 20,inversion24 of 24 ok. Host compile with clang from the installed toolchain, host build flags,-Wall -Wextra: only the pre-existingpick_cache_coresunused-function warning, exit 0.Not tested: Windows (see Must be tested); a live game crash exercising the deadlock fix (the game-free
crashandcrash_lockedcases cover the flush on the crash path); a real disk stall (the sink is simulated); Emscripten (stub unchanged, not compiled here);tests/run_all.pyin full on this rebase.Must be tested:
python -I -B tests/test_log_batch.py(12 cases; the Linux-only crash cases are skipped there) and a deliberate crash in a test build with lines still queued, expecting them to reach the log file (Test Farm: @Beanstalk windows,python -I -B tests/test_log_batch.py).hf48run(a save in Hyrule Field, the character running a fixed route, 1440p output, render scale 1, frame generation off, uncapped) and onheavy, expecting no change in log content or order and crash-path lines to reach the log before exit (by hand).ac778751against this PR @269ad429, 2 interleaved runs each (still to do): runhf48runandheavy, expecting the fps and frame-time numbers in Measurements to be confirmed or corrected; the current numbers are from a single-build A/B, not main against this PR (by hand).Risks
What could break: every log line on Linux now goes through a queue and a writer thread. A line logged just before a crash or exit that misses a flush site could be lost; the crash report and
atexitflush the queue, and the explicit flush sites are shared and unchanged.log_flush()now takes the queue lock before the write lock: a flush that finds a write in progress waits while holding the queue lock (a producer logging at the same moment waits too). Only the full-queue, exit and fatal paths flush, so this is rare.Known, not changed: the writer's own crash filter (
log_drain_raw, oneTryAcquireon the queue lock) can drop lines queued while the writer holds that lock at a crash: 63 of 80 crash runs under load kept their lines. Not reached on Linux (the crash report replaces that filter); on Windows it is reached whenHOS_LARGE_PAGES=1skipshos_commit_init. Left for its own PR.Turning it off:
HOS_LOG_SYNC=1(per run, no rebuild) restores the per-line path; the deadlock fix has no switch — revert the commit.Compatibility:
longshot_log.txton Linux: lines reach the file up toHOS_LOG_BATCH_MS(250 ms) after they are logged. Lines written straight to stderr by the platform shim or the crash paths' raw drain can now appear before earlier queued lines. Nothing in saves, options, caches or the ABI changes.Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks. Not ticked yet: Measurements show higher tail frame times with the writer on, and the risk of lost log lines is unmeasured.
main.python -I -B tests/run_all.pypasses (not run in full on this rebase; the PR's own tests pass (named in Testing)).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.Tested in game: this commit has been in the local build the Measurements come from since 2026-11-16 (nightly-338 base; Linux, CachyOS, Ryzen 7 5800X, RX 9070 XT, RADV Mesa 26.2.6, gamescope), with the writer on in every run except the
HOS_LOG_SYNC=1arm: both benchmark routes, several hours in total, and no run hung. Not run: this branch built from its own commit on current main, a real crash with lines still queued, and anything on Windows, where the changedlog_flush()also runs.Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.