WIP: fix(os): mutex_lock reads HOS_MUTEX_TRACE once and names the holder while it cannot end #2063

Draft
siantm wants to merge 2 commits from siantm/Longshot:fix/os-mutex-wait-diagnostics into main
Contributor

Summary

mutex_lock (host/nn_os_sync.c), when a guest mutex is held and its short spin did not win it, sleeps; on the way
it has three diagnostics, and each had a race:

  • HOS_MUTEX_TRACE=<ms> (log waits longer than that) was cached in a plain static that two first waits could write at
    once, and parsed with atoi: -1 stayed "unread" and logged every wait (read again at each), 1x read as 1. It is
    now read once through an atomic (relaxed: every reader stores the same value) and kept in a local for the whole
    wait; only a whole number of milliseconds from 1 to 2147483647 turns it on, anything else is off. The switch row says
    so, and that 1 logs nearly every wait.
  • The long-wait log and the Tracy zone named the holder through its HosThread after the wait, when that thread could
    have unlocked, ended and been freed. The name is now copied while the waiter is counted in mwait and holds m->lk:
    the holder's unlock (mutex_give) then sees the waiter and waits for m->lk, so it cannot end during the copy. The
    owner load there is __ATOMIC_SEQ_CST: it pairs with mutex_give's owner exchange and mwait load (each side
    stores, then loads what the other stored), which C11 orders only when all four are seq_cst (on x86-64 the same mov).
    The 10 s "still waiting" log reads the name under the same conditions.
  • That 10 s log printed m->count, which the owner changes without m->lk: dropped (mwait, read atomically, stays).

How to see it on main: python -I -B tests/test_cv_handoff_threads.py fails at HOS_MUTEX_TRACE=1: the log reads
the record of a holder that has unlocked and ended (see Testing). In the game it would show, rarely, with
HOS_MUTEX_TRACE set (off by default): a garbled or wrong holder name, a fault reading a freed thread's name, or
-1 logging every wait.

Found in #1953's reviews, where these lines were first fixed inside the perf change that moved them: BeanBot's
61949 (the trace switch's lazy read and
atoi), 61950 (the holder's lifetime)
and the review 61957 (B1-B3, m->count
in the 10 s log); futondestroyer's 62896
("deserve their own fixes"); dyst's 65813
(take them out as a fix(os) of their own), with his two notes on these lines applied here: the seq_cst owner load
(65811) and the switch row's text
(65812). The two commits are #1953's
d3e02c70 and 18e90465, adapted to main's mutex_lock (on main the wait is still inside it; #1953 moves it into
mutex_lock_wait). #1953 no longer carries them.

Read first: mutex_lock and mutex_give in host/nn_os_sync.c; the trace group in tests/test_cv_handoff_threads.py.

Overlap: #2019 (fix/os-chain-flag-atomic @ 75107f9e, stacked on #2020) changes const int chain = hos_chain_on; in
the same block to an atomic load, and the two conflict on that one line (git merge-tree); whichever lands second
keeps both: this PR's lines with #2019's __atomic_load_n(&hos_chain_on, __ATOMIC_RELAXED). #2020 (c29a9378), #1953
(d99ce34c, now the bare split), #1966, #1967, #1968, #1969, #1970, #1971, #1972 and #1974 at their current heads merge
with this without a conflict. #1957 (perf/mutex-ctx-thread @ a912c5b8) still carries an earlier copy of these two
commits (72ede7ee, 6f64422b) and is rebased when #1953 settles.

Testing

Tested: Windows 11, i7-9700, llvm-mingw clang 23.1.5; game-free checks only. This has NOT run in the game on this
PC, nor on Linux (Still to test). On main @ 9dd41326. Each check ran in a hidden console of its own, in one job of
the PC's job queue (fb3-mx-1-light):

  • python -I -B tests/test_cv_handoff_threads.py: passes at the head 28cca9e5, with no --cc (LONGSHOT_MINGW's
    clang) and again with CC set. At the test commit 7492bb61 (main's code) it fails at its first value that logs,
    HOS_MUTEX_TRACE=1: "FAIL:105 !__atomic_load_n(&t->dead, __ATOMIC_SEQ_CST)" (the log read the ended holder's record;
    the run stops there, so -1, 1x and the oversized values after it are not reached on main).
  • host/nn_os_sync.c compiled with the host build's own flags (-O2 -march=native -mcx16 ..., from a local build's
    compile_commands.json), with and without Tracy (-DTRACY_ENABLE -DTRACY_ON_DEMAND): no error.
  • python -I -B tests/run_all.py --strict --changed origin/main at the head (the checks that read a changed file, 8 at
    once, the Vulkan headers through RX_CPM; build job fb3-mxn-changed): 32 passed, 7 failed, 3 skipped
    (test_driver_env_choose.py, test_mesh_fallback_layout.py, test_tracy_funcs.py: Linux only). The 7
    (test_crash_dialog, test_dev_ab, test_frame_overlap, test_linux_pressure, test_present_queue_default,
    test_target_reuse_memory, test_vk_occlusion_queries; picked because gpu/vk/switches.def changed) fail on main
    @ 9dd41326 too, run by name from a main worktree in the same job, with the same messages; none fails here alone.
  • Not run: mise run lint (ruff and pre-commit are not installed on this PC) and the whole suite (mise run test).

Still to test:

Measurements

No game measurement yet: the game runs are in Still to test. The change is on the path a lock takes only after its
spin lost (it is about to sleep): what it adds there is one atomic load where a plain one was, and, only with
HOS_MUTEX_TRACE set or in a Tracy build, a copy of the holder's name of at most 47 bytes under m->lk. The
uncontended lock and the spin are unchanged. The before-and-after proof is the game-free fixture, run on this PC
(i7-9700, Windows 11, llvm-mingw clang) in fb3-mx-1-light:

tests/test_cv_handoff_threads.py Before: main @ 9dd41326 (test commit 7492bb61) After: this PR @ 28cca9e5
HOS_MUTEX_TRACE cases (unset, 0, 1, -1, 1x, 2147483648, 99999999999) FAIL at 1 (the ended holder's record read) pass
the cases run before them (hand-off, ordering, lives, semaphores, both HOS_NO_HANDOFF ways) pass pass
the broken variants run after them not reached pass

Verdict: the fixture fails on main and passes here; no game measurement (no switch, and nothing on the paths that do
not sleep changes).

Risks

With HOS_MUTEX_TRACE set or Tracy built in, a waiter now copies the holder's name while it holds m->lk, before it
sleeps: the holder's unlock, if it comes in those few hundred nanoseconds, waits for m->lk as it already does for any
counted waiter. A value of HOS_MUTEX_TRACE that is not a whole number from 1 to 2147483647 (-1, 1x, 5ms) now
turns it off where it used to log (every wait for -1). The name in the long-wait log is the holder's when the wait
started to sleep (the switch row says so), as before. No switch of its own: a revert gives back main's code. Players
notice nothing after updating.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A fix: a game-free test that fails on main and passes here.
  • A switch added or changed: it ships at the default the measurements chose, with its row in gpu/vk/switches.def.
  • A check under tests/ that reads changed source is updated, not weakened.
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths.
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.
## Summary `mutex_lock` (`host/nn_os_sync.c`), when a guest mutex is held and its short spin did not win it, sleeps; on the way it has three diagnostics, and each had a race: - `HOS_MUTEX_TRACE=<ms>` (log waits longer than that) was cached in a plain static that two first waits could write at once, and parsed with `atoi`: `-1` stayed "unread" and logged every wait (read again at each), `1x` read as 1. It is now read once through an atomic (relaxed: every reader stores the same value) and kept in a local for the whole wait; only a whole number of milliseconds from 1 to 2147483647 turns it on, anything else is off. The switch row says so, and that `1` logs nearly every wait. - The long-wait log and the Tracy zone named the holder through its `HosThread` after the wait, when that thread could have unlocked, ended and been freed. The name is now copied while the waiter is counted in `mwait` and holds `m->lk`: the holder's unlock (`mutex_give`) then sees the waiter and waits for `m->lk`, so it cannot end during the copy. The owner load there is `__ATOMIC_SEQ_CST`: it pairs with `mutex_give`'s owner exchange and `mwait` load (each side stores, then loads what the other stored), which C11 orders only when all four are seq_cst (on x86-64 the same `mov`). The 10 s "still waiting" log reads the name under the same conditions. - That 10 s log printed `m->count`, which the owner changes without `m->lk`: dropped (`mwait`, read atomically, stays). How to see it on `main`: `python -I -B tests/test_cv_handoff_threads.py` fails at `HOS_MUTEX_TRACE=1`: the log reads the record of a holder that has unlocked and ended (see Testing). In the game it would show, rarely, with `HOS_MUTEX_TRACE` set (off by default): a garbled or wrong holder name, a fault reading a freed thread's name, or `-1` logging every wait. Found in #1953's reviews, where these lines were first fixed inside the `perf` change that moved them: BeanBot's [61949](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1953#issuecomment-61949) (the trace switch's lazy read and `atoi`), [61950](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1953#issuecomment-61950) (the holder's lifetime) and the review [61957](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1953#issuecomment-61957) (B1-B3, `m->count` in the 10 s log); futondestroyer's [62896](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1953#issuecomment-62896) ("deserve their own fixes"); dyst's [65813](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1953#issuecomment-65813) (take them out as a `fix(os)` of their own), with his two notes on these lines applied here: the seq_cst owner load ([65811](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1953#issuecomment-65811)) and the switch row's text ([65812](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1953#issuecomment-65812)). The two commits are #1953's d3e02c70 and 18e90465, adapted to `main`'s `mutex_lock` (on `main` the wait is still inside it; #1953 moves it into `mutex_lock_wait`). #1953 no longer carries them. Read first: `mutex_lock` and `mutex_give` in `host/nn_os_sync.c`; the `trace` group in `tests/test_cv_handoff_threads.py`. Overlap: #2019 (`fix/os-chain-flag-atomic` @ 75107f9e, stacked on #2020) changes `const int chain = hos_chain_on;` in the same block to an atomic load, and the two conflict on that one line (`git merge-tree`); whichever lands second keeps both: this PR's lines with #2019's `__atomic_load_n(&hos_chain_on, __ATOMIC_RELAXED)`. #2020 (c29a9378), #1953 (d99ce34c, now the bare split), #1966, #1967, #1968, #1969, #1970, #1971, #1972 and #1974 at their current heads merge with this without a conflict. #1957 (`perf/mutex-ctx-thread` @ a912c5b8) still carries an earlier copy of these two commits (72ede7ee, 6f64422b) and is rebased when #1953 settles. ## Testing **Tested:** Windows 11, i7-9700, llvm-mingw clang 23.1.5; game-free checks only. This has NOT run in the game on this PC, nor on Linux (Still to test). On `main` @ 9dd41326. Each check ran in a hidden console of its own, in one job of the PC's job queue (fb3-mx-1-light): - `python -I -B tests/test_cv_handoff_threads.py`: passes at the head 28cca9e5, with no `--cc` (`LONGSHOT_MINGW`'s clang) and again with `CC` set. At the test commit 7492bb61 (`main`'s code) it fails at its first value that logs, `HOS_MUTEX_TRACE=1`: "FAIL:105 !__atomic_load_n(&t->dead, __ATOMIC_SEQ_CST)" (the log read the ended holder's record; the run stops there, so `-1`, `1x` and the oversized values after it are not reached on `main`). - `host/nn_os_sync.c` compiled with the host build's own flags (`-O2 -march=native -mcx16 ...`, from a local build's `compile_commands.json`), with and without Tracy (`-DTRACY_ENABLE -DTRACY_ON_DEMAND`): no error. - `python -I -B tests/run_all.py --strict --changed origin/main` at the head (the checks that read a changed file, 8 at once, the Vulkan headers through `RX_CPM`; build job fb3-mxn-changed): 32 passed, 7 failed, 3 skipped (`test_driver_env_choose.py`, `test_mesh_fallback_layout.py`, `test_tracy_funcs.py`: Linux only). The 7 (`test_crash_dialog`, `test_dev_ab`, `test_frame_overlap`, `test_linux_pressure`, `test_present_queue_default`, `test_target_reuse_memory`, `test_vk_occlusion_queries`; picked because `gpu/vk/switches.def` changed) fail on `main` @ 9dd41326 too, run by name from a `main` worktree in the same job, with the same messages; none fails here alone. - Not run: `mise run lint` (ruff and pre-commit are not installed on this PC) and the whole suite (`mise run test`). **Still to test:** - Windows 11, this PC (i7-9700, RTX 4060 (NVIDIA 610.88)): Castle Town's market, two minutes with `HOS_MUTEX_TRACE=5`, expecting "mutex ...: waited ... ms (lr ...); held by <a thread name>" lines (60 at most) with real thread names and no crash, and none with the switch unset (by hand) - Windows 11, this PC: the same scene with no switch set, this PR against `main`, expecting the same picture and frame times within the runs' spread (by hand) - Linux: build the host and run `python3 -I -B tests/test_cv_handoff_threads.py --cc clang` and `--cc gcc`, expecting it to pass (by hand) - Linux with an AMD CPU and the Steam Deck: Castle Town's market, two minutes, this PR against `main`, no switch set, expecting the same frame times (the Test Farm: play Castle Town's market for two minutes on Linux with an AMD CPU and the Steam Deck, this PR against main, comparing frame times) ## Measurements No game measurement yet: the game runs are in Still to test. The change is on the path a lock takes only after its spin lost (it is about to sleep): what it adds there is one atomic load where a plain one was, and, only with `HOS_MUTEX_TRACE` set or in a Tracy build, a copy of the holder's name of at most 47 bytes under `m->lk`. The uncontended lock and the spin are unchanged. The before-and-after proof is the game-free fixture, run on this PC (i7-9700, Windows 11, llvm-mingw clang) in fb3-mx-1-light: | `tests/test_cv_handoff_threads.py` | Before: main @ 9dd41326 (test commit 7492bb61) | After: this PR @ 28cca9e5 | |---|---|---| | `HOS_MUTEX_TRACE` cases (unset, 0, 1, -1, 1x, 2147483648, 99999999999) | FAIL at 1 (the ended holder's record read) | pass | | the cases run before them (hand-off, ordering, lives, semaphores, both `HOS_NO_HANDOFF` ways) | pass | pass | | the broken variants run after them | not reached | pass | Verdict: the fixture fails on `main` and passes here; no game measurement (no switch, and nothing on the paths that do not sleep changes). ## Risks With `HOS_MUTEX_TRACE` set or Tracy built in, a waiter now copies the holder's name while it holds `m->lk`, before it sleeps: the holder's unlock, if it comes in those few hundred nanoseconds, waits for `m->lk` as it already does for any counted waiter. A value of `HOS_MUTEX_TRACE` that is not a whole number from 1 to 2147483647 (`-1`, `1x`, `5ms`) now turns it off where it used to log (every wait for `-1`). The name in the long-wait log is the holder's when the wait started to sleep (the switch row says so), as before. No switch of its own: a revert gives back `main`'s code. Players notice nothing after updating. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. - [ ] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A fix: a game-free test that fails on `main` and passes here. - [x] A switch added or changed: it ships at the default the measurements chose, with its row in `gpu/vk/switches.def`. - [x] A check under `tests/` that reads changed source is updated, not weakened. - [ ] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold.
tests/test_cv_handoff_threads.py runs mutex_lock's long-wait log in a fresh process per HOS_MUTEX_TRACE value (unset,
0, 1, -1, 1x, 2147483648, 99999999999), two waits of about 20 ms with the variable flipped between them: only a number
of milliseconds from 1 to 2147483647 logs, read once, and the log names the thread that held the mutex after that
thread has unlocked and ended (its record marked dead, which the fixture's hos_thread_name refuses). It fails on the
current code at its first value that logs, 1: the log reads the ended holder's record (the run stops there; the
values after it, -1 and 1x among them, are not reached).

The compiler is --cc, else $CC, else LONGSHOT_MINGW's clang on Windows, else clang, gcc or cc on PATH (no
RUN_ALL_ARGS forcing gcc).

From d3e02c70 on #1953 (perf/mutex-lock-wait-cold), there for the same wait moved into mutex_lock_wait.

Assisted-by: Claude Code
fix(os): mutex_lock reads HOS_MUTEX_TRACE once through an atomic and names the holder while it cannot end
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 5s
beanstalk/approval waiting for a BigBean's 👍 on the request
28cca9e5a1
mutex_lock's wait for a held mutex had three races in its diagnostics:
- HOS_MUTEX_TRACE was cached in a plain static two first waits could write at once, and parsed with atoi. It is read
  through an atomic (relaxed: both store the same value) and kept in a local for the whole wait; only a number of
  milliseconds from 1 to 2147483647 turns it on (strtoll, the whole string), anything else is off (-1 logged every
  wait and was read again at each; 1 logs nearly every wait, the switch row says).
- The long-wait log and the Tracy zone named the holder through its HosThread after the wait, when it could have
  unlocked, ended and been freed. The name is copied with the wait counted in mwait and m->lk held: the holder's unlock
  (mutex_give) then waits for m->lk, so it cannot end meanwhile. The owner load there is seq_cst, as mutex_give's
  exchange and mwait load are (a store-buffer pairing C11 orders only at seq_cst; the same mov on x86-64). The 10 s log
  reads it there too.
- The 10 s log printed m->count, which the owner changes without m->lk: dropped (mwait is read as an atomic).
The switch row says the name is the holder's when the wait started to sleep.

From 18e90465 on #1953 (perf/mutex-lock-wait-cold), with dyst's two notes on it (#1953 review 65813).

Assisted-by: Claude Code
Author
Contributor

Tested: the changed-file checks at 28cca9e5 ended (fb3-mxn-changed): 32 passed, 7 failed, 3 skipped (Linux only); the 7 fail on main @ 9dd41326 too, with the same messages. The description says so in place of "queued".

Tested: the changed-file checks at 28cca9e5 ended (fb3-mxn-changed): 32 passed, 7 failed, 3 skipped (Linux only); the 7 fail on `main` @ 9dd41326 too, with the same messages. The description says so in place of "queued".
Author
Contributor

@Beanstalk This changes only the HOS_MUTEX_TRACE diagnostics of the guest mutex waits (host/nn_os_sync.c), off unless HOS_MUTEX_TRACE is set; with nothing set, nothing should change. Please start from the market-entrance save and play Castle Town Market for two minutes, the PR against main on the same PC with HOS_MUTEX_TRACE unset, and one more run of the PR with HOS_MUTEX_TRACE set, on a Windows PC and on a Linux PC. No hang, stall or crash either way; unset, frame time p99 equal to main's, within the run-to-run spread; set, the log has the mutex trace lines.

@Beanstalk This changes only the HOS_MUTEX_TRACE diagnostics of the guest mutex waits (host/nn_os_sync.c), off unless HOS_MUTEX_TRACE is set; with nothing set, nothing should change. Please start from the market-entrance save and play Castle Town Market for two minutes, the PR against main on the same PC with HOS_MUTEX_TRACE unset, and one more run of the PR with HOS_MUTEX_TRACE set, on a Windows PC and on a Linux PC. No hang, stall or crash either way; unset, frame time p99 equal to main's, within the run-to-run spread; set, the log has the mutex trace lines.
Member

@siantm only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@siantm only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 5s
beanstalk/approval waiting for a BigBean's 👍 on the request
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/os-mutex-wait-diagnostics:siantm-fix/os-mutex-wait-diagnostics'
git switch 'siantm-fix/os-mutex-wait-diagnostics'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2063
No description provided.