WIP: fix(os): mutex_lock reads HOS_MUTEX_TRACE once and names the holder while it cannot end #2063
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2063
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:fix/os-mutex-wait-diagnostics"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
mutex_lock(host/nn_os_sync.c), when a guest mutex is held and its short spin did not win it, sleeps; on the wayit has three diagnostics, and each had a race:
HOS_MUTEX_TRACE=<ms>(log waits longer than that) was cached in a plain static that two first waits could write atonce, and parsed with
atoi:-1stayed "unread" and logged every wait (read again at each),1xread as 1. It isnow read once through an atomic (relaxed: every reader stores the same value) and kept in a local for the whole
wait; only a whole number of milliseconds from 1 to 2147483647 turns it on, anything else is off. The switch row says
so, and that
1logs nearly every wait.HosThreadafter the wait, when that thread couldhave unlocked, ended and been freed. The name is now copied while the waiter is counted in
mwaitand holdsm->lk:the holder's unlock (
mutex_give) then sees the waiter and waits form->lk, so it cannot end during the copy. Theowner load there is
__ATOMIC_SEQ_CST: it pairs withmutex_give's owner exchange andmwaitload (each sidestores, then loads what the other stored), which C11 orders only when all four are seq_cst (on x86-64 the same
mov).The 10 s "still waiting" log reads the name under the same conditions.
m->count, which the owner changes withoutm->lk: dropped (mwait, read atomically, stays).How to see it on
main:python -I -B tests/test_cv_handoff_threads.pyfails atHOS_MUTEX_TRACE=1: the log readsthe record of a holder that has unlocked and ended (see Testing). In the game it would show, rarely, with
HOS_MUTEX_TRACEset (off by default): a garbled or wrong holder name, a fault reading a freed thread's name, or-1logging every wait.Found in #1953's reviews, where these lines were first fixed inside the
perfchange that moved them: BeanBot's61949 (the trace switch's lazy read and
atoi), 61950 (the holder's lifetime)and the review 61957 (B1-B3,
m->countin the 10 s log); futondestroyer's 62896
("deserve their own fixes"); dyst's 65813
(take them out as a
fix(os)of their own), with his two notes on these lines applied here: the seq_cst owner load(65811) and the switch row's text
(65812). The two commits are #1953's
d3e02c70and18e90465, adapted tomain'smutex_lock(onmainthe wait is still inside it; #1953 moves it intomutex_lock_wait). #1953 no longer carries them.Read first:
mutex_lockandmutex_giveinhost/nn_os_sync.c; thetracegroup intests/test_cv_handoff_threads.py.Overlap: #2019 (
fix/os-chain-flag-atomic@75107f9e, stacked on #2020) changesconst int chain = hos_chain_on;inthe same block to an atomic load, and the two conflict on that one line (
git merge-tree); whichever lands secondkeeps both: this PR's lines with #2019's
__atomic_load_n(&hos_chain_on, __ATOMIC_RELAXED). #2020 (c29a9378), #1953(d99ce34c, now the bare split), #1966, #1967, #1968, #1969, #1970, #1971, #1972 and #1974 at their current heads merge
with this without a conflict. #1957 (
perf/mutex-ctx-thread@a912c5b8) still carries an earlier copy of these twocommits (
72ede7ee,6f64422b) and is rebased when #1953 settles.Testing
Tested: Windows 11, i7-9700, llvm-mingw clang 23.1.5; game-free checks only. This has NOT run in the game on this
PC, nor on Linux (Still to test). On
main@9dd41326. Each check ran in a hidden console of its own, in one job ofthe PC's job queue (fb3-mx-1-light):
python -I -B tests/test_cv_handoff_threads.py: passes at the head28cca9e5, with no--cc(LONGSHOT_MINGW'sclang) and again with
CCset. At the test commit7492bb61(main's code) it fails at its first value that logs,HOS_MUTEX_TRACE=1: "FAIL:105 !__atomic_load_n(&t->dead, __ATOMIC_SEQ_CST)" (the log read the ended holder's record;the run stops there, so
-1,1xand the oversized values after it are not reached onmain).host/nn_os_sync.ccompiled with the host build's own flags (-O2 -march=native -mcx16 ..., from a local build'scompile_commands.json), with and without Tracy (-DTRACY_ENABLE -DTRACY_ON_DEMAND): no error.python -I -B tests/run_all.py --strict --changed origin/mainat the head (the checks that read a changed file, 8 atonce, the Vulkan headers through
RX_CPM; build job fb3-mxn-changed): 32 passed, 7 failed, 3 skipped(
test_driver_env_choose.py,test_mesh_fallback_layout.py,test_tracy_funcs.py: Linux only). The 7(
test_crash_dialog,test_dev_ab,test_frame_overlap,test_linux_pressure,test_present_queue_default,test_target_reuse_memory,test_vk_occlusion_queries; picked becausegpu/vk/switches.defchanged) fail onmain@
9dd41326too, run by name from amainworktree in the same job, with the same messages; none fails here alone.mise run lint(ruff and pre-commit are not installed on this PC) and the whole suite (mise run test).Still to test:
HOS_MUTEX_TRACE=5, expecting "mutex ...: waited ... ms (lr ...); held by " lines (60 at most) withreal thread names and no crash, and none with the switch unset (by hand)
main, expecting the same picture and frametimes within the runs' spread (by hand)
python3 -I -B tests/test_cv_handoff_threads.py --cc clangand--cc gcc, expectingit to pass (by hand)
main, no switch set,expecting the same frame times (the Test Farm: play Castle Town's market for two minutes on Linux with an AMD CPU and
the Steam Deck, this PR against main, comparing frame times)
Measurements
No game measurement yet: the game runs are in Still to test. The change is on the path a lock takes only after its
spin lost (it is about to sleep): what it adds there is one atomic load where a plain one was, and, only with
HOS_MUTEX_TRACEset or in a Tracy build, a copy of the holder's name of at most 47 bytes underm->lk. Theuncontended lock and the spin are unchanged. The before-and-after proof is the game-free fixture, run on this PC
(i7-9700, Windows 11, llvm-mingw clang) in fb3-mx-1-light:
tests/test_cv_handoff_threads.pyHOS_MUTEX_TRACEcases (unset, 0, 1, -1, 1x, 2147483648, 99999999999)HOS_NO_HANDOFFways)Verdict: the fixture fails on
mainand passes here; no game measurement (no switch, and nothing on the paths that donot sleep changes).
Risks
With
HOS_MUTEX_TRACEset or Tracy built in, a waiter now copies the holder's name while it holdsm->lk, before itsleeps: the holder's unlock, if it comes in those few hundred nanoseconds, waits for
m->lkas it already does for anycounted waiter. A value of
HOS_MUTEX_TRACEthat is not a whole number from 1 to 2147483647 (-1,1x,5ms) nowturns it off where it used to log (every wait for
-1). The name in the long-wait log is the holder's when the waitstarted to sleep (the switch row says so), as before. No switch of its own: a revert gives back
main's code. Playersnotice nothing after updating.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.gpu/vk/switches.def.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Tested: the changed-file checks at
28cca9e5ended (fb3-mxn-changed): 32 passed, 7 failed, 3 skipped (Linux only); the 7 fail onmain@9dd41326too, with the same messages. The description says so in place of "queued".@Beanstalk This changes only the HOS_MUTEX_TRACE diagnostics of the guest mutex waits (host/nn_os_sync.c), off unless HOS_MUTEX_TRACE is set; with nothing set, nothing should change. Please start from the market-entrance save and play Castle Town Market for two minutes, the PR against main on the same PC with HOS_MUTEX_TRACE unset, and one more run of the PR with HOS_MUTEX_TRACE set, on a Windows PC and on a Linux PC. No hang, stall or crash either way; unset, frame time p99 equal to main's, within the run-to-run spread; set, the log has the mutex trace lines.
@siantm only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.