WIP: perf(os): HOS_CV_PARK=1, events, semaphores and message queues sleep on a park that blocks at once after their own spin (experiment, off) #1970
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1970
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/event-park"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
A guest wait on an event, a semaphore or a message queue spins for its flag first (
spin_until_set) and then sleepswith
SleepConditionVariableSRW, which spins again before it blocks: 10240 cycles of pause in ntdll(
RtlSleepConditionVariableSRW: 10240 /KUSER_SHARED_DATA.CyclesPerYieldpauses), after the wait's own spin hasalready missed.
This adds an experiment, off by default: with
HOS_CV_PARK=1(Windows) those sleeps go to a park inhost/hos_thread.c(hos_park_sleep,hos_park_wake) that uses the condition variable's own block and wake(
NtWaitForAlertByThreadId,NtAlertThreadByThreadId) without its second spin when the caller has just spun (a wait'sfirst sleep after its own spin, or after the spin policy left that spin out). The waiters are an oldest-first list from
the cv's word under one of 64 striped SRW locks; a wake that finds a waiter still spinning makes no system call. A
woken waiter takes its virtual core back, then the object's lock once (
hos_cv_sleeptakes it, lets it go for the coreand takes it again). Mutexes, guest condition variables, read-write locks, timers and WaitAny stay on Windows'
condition variable. Unset,
=0, an ntdll without the two calls, or a wait that does not answer at setup: everything isas on
main(hos_park_sleepishos_cv_sleep,hos_park_wakethe Win32 wake). Linux keeps its condition variable(the same shims).
How the park keeps a wake its own (after the review): the switch and the two calls are set up once
(
InitOnceExecuteOnce, the mode published after the pointers). An alert is the thread's, so a sleep the park has wokenleaves only when its wake is done with it (
PARK_DONE, set after the alert is sent or has failed) and then takes anyalert still pending for its thread with a zero-time wait, so none reaches the thread's next wait: Windows'
WaitOnAddress returns at the first alert its thread gets, its wait block still listed, and the game's job barriers
(
host/oot_barrier.c) wait that way on the same threads (on the local branch this came from, a start with the firstversion of the park stopped with an access violation in
WakeByAddressAllunderhos_oot_barrier_wake, one start offive). A failed alert or a failing wait is logged once; a blocked sleep waits at most 100 ms at a time and looks at its
state, and a failing wait goes on in 1 ms steps until its wake or deadline, on the list all along.
spin_until_setreads the thread's spin budget once (spin_beginhands it on): with the park off it makes as manyhos_spin_ns()calls asmainwhen the spin is left out and one fewer when it spins.Six commits: the park; a test that no alert is left for a thread after a park sleep; its fix; tests for the review's
points (first callers, alert ownership, failing wait and alert, timed semaphore and queue peek, first-sleep policy, one
budget lookup); their fix; the budget.
From the local branch
exp/x86-asm-sync: c565d05a (the park), 78c83308 (its first part: one lock take after a parkwake), 939b5324 (the test), 76de03b5 (the fix). There the park was on by default; here it is off.
Read first: the park block after
hos_cv_sleepinhost/hos_thread.c(park_init,park_sleep,hos_park_wake,hos_park_sleep); inhost/nn_os_sync.cspin_begin,wait_until'sparkargument,spin_until_set's return,event_wait,sema_acquire,queue_get;tests/test_event_park.py(new).Testing
Tested: Windows 11, i7-9700, RTX 4060 (NVIDIA 610.88), llvm-mingw clang 23.1.5; game-free checks only. This change
has not run in the game on this PC yet (Still to test). Each check ran in a hidden console of its own, through the PC's
job queue (job fb3-1970-1), on this branch rebased onto
main@f1f2dc5b(headff7e2362):python -I -B tests/test_event_park.pyat the test commit1e2adc46(job fb3-1970-1): exit 1, each new part failing as itshould without its fix: "budget" ("spin_until_set read the spin budget 2 and 2 times, not once", with and without a
budget), "first callers" with
HOS_CV_PARK=1("8 first callers looked ntdll's two calls up 16 times (want 2)"),"own" ("a untimed park sleep left on another alert for its thread before its wake's alert was sent"), "failing wait"
("stalled": the 30 ms sleep never returned), "failing alert" ("an untimed sleep whose wake's alert failed ends (0 of
1)"), "a wait that does not answer" ("a park whose wait answers STATUS_NOT_IMPLEMENTED is used"). The races, "timed"
and "policy" pass there.
e4b273b1(the park fix, before the budget commit; job fb3-1970-2): exit 1, only "budget" failing (2 and 2).ff7e2362(job fb3-1970-2): exit 0, every part passing, unset,HOS_CV_PARK=1, both with no spinfirst and
HOS_CV_PARK=0("ok", 20,000 auto-clear signals, 2,000 manual-clear rounds of 4, 5,000 timed-wait signals,60,000 through a semaphore and a queue, each message once); "timed" 68 of 100 racing releases taken and 32 after the acquire timed out, 76 of 100 racing sends seen and 24 after the peek timed out (on the park); "policy": first sleeps told spun=1 and
0 pauses before blocking with a budget, spun=0 and 66 pauses with none; "budget" 1 and 1; "first callers" 2 lookups;
"own", "failing wait" (2 waits for a 30 ms sleep), "failing alert" pass; with the wait answering
STATUS_NOT_IMPLEMENTEDthe sleeps stay on Windows' condition variable; both controls fail. WithCCset tollvm-mingw's clang (job fb3-1970-1, before the own part's check was moved ahead of the held wake, where its control
crashed instead of failing): every part passed the same way.
test_cv_handoff_threads,test_cv_signal_race,test_cv_signal_early,test_event_handoff,test_multiwait_wake,test_os_lifted_siblings,test_oot_barrier,test_switch_table(job fb3-1970-1) andtest_spin_by_role(job fb3-1970-2, its wiring check now reading the budget wherespin_beginreads it) at the head:exit 0.
python -I -B tests/run_all.py --strict --changed origin/main(job fb3-1970-1, at the head before the two test-onlychanges of job fb3-1970-2): exit 1, "69 passed, 19 failed, 4 skipped". Two were this PR's and are fixed since
(
test_event_park's own-part control,test_spin_by_role's wiring check, both passing in job fb3-1970-2); 14 of theother 17 fail on
main@f1f2dc5btoo (run by name in the same job):test_crash_dialog,test_crash_record,test_dev_ab,test_dev_packs,test_frame_overlap,test_host_guest_writes,test_libc_hooks,test_linux_pressure,test_mods_list,test_profiling_mode,test_slow_frames,test_vcore_escape,test_vcore_mask0,test_walk_key. The other three (test_present_queue_default,test_target_reuse_memory,test_vk_occlusion_queries) compiled nothing in that run nor in the run onmain(no Vulkan headers on the job'spath: "not found: vulkan_headers (--strict)"). Rerun with the headers (job fb3-vk-1,
RX_CPMset) at the headff7e2362and onmain@f1f2dc5b, each fails at both with the same first error:test_present_queue_default"srccut:'{TAB_DEBUG, "Present queue", &opts.present_queue, ...' is not in gpu/vk/vk_menu.cpp";
test_target_reuse_memory2 ofits 8 tests ("'vmaGetAllocationInfo2' unexpectedly found" in the VMA source, and its fixture's "use of undeclared
identifier 'destroy_image_later'");
test_vk_occlusion_queries"the ring fixture did not compile" ("no member named'policy'"). None fails on this branch alone.
Still to test:
HOS_CV_PARK=1and unset (the earlier local branch failed one start in five), then Castle Town's market,main@f1f2dc5bagainst this PR withHOS_CV_PARKunset and with it set, at least two warmed, interleaved runs each, CPU perframe, average fps, 1% lows, median and p99 frame times, expecting the log line "os: HOS_CV_PARK: events, semaphores
and message queues sleep on a park that does not spin again after their own spin" with it set, no crash (none under
hos_oot_barrier_wakein particular), and the unset run withinmain's spread (by hand)python3 -I -B tests/run_all.py --strict --changed origin/main,expecting
tests/test_event_park.pyskipped (Windows only) and the cut checks passing (by hand)mainagainst this PRunset and with
HOS_CV_PARK=1, expecting theos: HOS_CV_PARKlog line, no crash and frame times as unset or betterMeasurements
Verdict: not measured: no micro-benchmark was run for this PR, and no play measurement exists yet.
HOS_CV_PARKships off (kind
attempt), an experiment for A/B runs;tests/test_event_park.pyhas no timing mode. The disabledhead against
mainand the in-game A/B (main, unset,HOS_CV_PARK=1) are in Still to test: a game-free benchmark ofthe wait's entry would time a stand-in for
hos_spin_ns()(it reads the host's thread table), not the call itself.The table holds the fixes' tests, before and after.
Baseline: this PR's test commits, on
main@f1f2dc5b:mainhas no park for the tests to run on.Setup: Windows 11, i7-9700, RTX 4060 (NVIDIA 610.88), llvm-mingw clang 23.1.5;
tests/test_event_park.py, one runeach side, each in a hidden console of its own through the PC's job queue.
1e2adc46(ande4b273b1for the budget)ff7e2362tests/test_event_park.py"first callers",HOS_CV_PARK=1hos_spin_ns()calls a wait)e4b273b1)Risks
Off (the default), the events', semaphores' and queues' sleeps and wakes go through
hos_park_sleepandhos_park_wake, which call Windows' condition variable as before after one test of a switch read once (its firstcaller runs the setup once, the others wait for it);
spin_until_setreturns whether it spun. On, the park relies ontwo undocumented ntdll calls and two words of
KUSER_SHARED_DATA(the unparked processor count andCyclesPerYield,read as ntdll's own condition variable reads them); a missed wake would stall a guest thread, and a stale alert would
wake another wait on the thread. Each sleep the park blocks and then wakes makes one more zero-time
NtWaitForAlertByThreadId(to take a pending alert), and a long sleep wakes every 100 ms to look at its state.Turn it off by leaving
HOS_CV_PARKunset; a revert is enough. Players notice nothing: the switch is not set for them.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named). (the checks reading the change ran, each failure named and failing onmaintoo; not the whole suite)main, or Measurements says why there is nothing to measure. (not measured yet: an experiment, off)third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def. (the row is there; off as an experiment, not measured yet)tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Possibly related, for context:
Recommendation: Request changes. The opt-in Windows park can race during initialization and can lose the correspondence between a wake and its thread alert. It adds a second synchronization implementation to the host, and there is no measured in-game gain against main.
Adds an off-by-default Windows experiment for guest events, semaphores and message queues. The park can skip a second spin and lets a woken thread reacquire its virtual core before retaking the object lock. The cost is a custom waiter list, striped locks and an undocumented alert protocol on paths that also serve audio, video and frame delivery.
@fixwill do (13)host/hos_thread.c:2272: The atomic flag publishes the pointers but does not protect their initialization. (more on the line)host/hos_thread.c:2323: STATUS_ALERTED and PARK_WOKEN do not establish that the alert came from this wake. (more on the line)host/hos_thread.c:2326: An unexpected status from the native wait retries immediately with the same inputs. (more on the line)host/hos_thread.c:2359: The wake detaches the waiter, marks it woken and ignores the alert's return status. (more on the line)host/hos_thread.c:2299: The spin divisor is not the advertised CyclesPerYield calculation. (more on the line)tests/test_event_park.py:35: Both the suite declaration and the direct default hard-code gcc. (more on the line)tests/test_event_park.py:226: The teardown writes signaled with a plain store while event_peek can still read or claim it without taking the lock. (more on the line)tests/test_event_park.py:224: release_all treats WAIT_FAILED like a successful join. (more on the line)tests/test_event_park.py:189: The check for blocking without a second spin waits for PARK_BLOCK before waking the thread. (more on the line)tests/test_event_park.py:92: The stale-alert regression check depends on stress scheduling. (more on the line)docs/dev/reference/switches.md:282: The switch reference and table describe the removed spin as about 2.5 µs without identifying the CPU or Windows build. (more on the line)host/hos_thread.c:2293: Record enqueue depth and lock time for the experiment. If contention shows up, keep a tail for constant-time insertion instead of traversing the list under the stripe lock.For the author (2) (a run, a measurement or a decision)
host/hos_thread.c:2293: The test result measures the stale-alert fix. (more on the line)Checks: Still running:
CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request),Website / Website on Cloudflare (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven. On Windows 11 with an i7-9700 and RTX 4060, the author reports 27 stale alerts in one game-free run of the park before its fix and zero after it on an earlier rebase. The earlier local branch reportedly saw no in-play difference. There are no matched runs against
6e1521a7for CPU time, wake latency, average FPS, 1% lows or median and p99 frame times.Other systems: Normal, light and system events share the change, including the audio renderer, vsync, video and GPU completion signals. The job barriers use WaitOnAddress on some of the same threads, so a pending alert can affect a later wait. Queue producers and consumers share one park. WaitAny still waits on its own condition variable and receives the same notifications. Mutexes, guest condition variables, reader-writer locks and timers still use their existing waits. Linux and Steam Deck use the original futex-backed condition variable through the new wrappers. There is no direct renderer, save, input, build or VRAM change. Windows 10, AMD hardware, lower-core machines, Linux and Steam Deck have no reported integration coverage for this revision.
Switches: HOS_CV_PARK is a read-once Windows developer switch, classified as attempt, and listed in switches.def and the developer reference. Unset and =0 select the existing condition variable, as does a missing ntdll export. Off is the safe interim value while correctness is open. The repository requires measurements to choose a shipping default. HOS_SPIN_US and HOS_SPIN_OTHER_US retain their defaults and determine whether the park can omit its spin.
Reviewed:
82d36ed100against6e1521a70a: Read AGENTS.md, CONTRIBUTING.md, docs/dev/standards.md, the Forgejo template and tests/test_event_park.py in full. Read all three commit messages and all 11 file diffs, including every changed region of host/hos.h, host/hos_thread.c, host/nn_os_sync.c and the five existing fixtures. Traced object creation and reuse, wait and wake callers, virtual-core handoff, Tracy lock wrappers, the Linux shim, the job barrier, switch parsing and test-runner compiler discovery. Unrelated portions of existing files were outside the review. The supplied run summaries were the available evidence. There was no docs/review-checklist.md or adjacent Tooling checkout, and there were no changed translations, generated files or prior reviews.; nothing run (a code review).Checked and fine:
Testing: Read-only review. git diff --check passed. The author reports that the Windows fixtures passed on an earlier rebase and that the 16 selected-suite failures also failed on its base. The reported range-diff was unchanged through
82d36ed1. The supplied CI checks are still running, with no head-only failures listed.Overlaps and merge order: None found in the checkout history or supplied PR material.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -279,6 +279,7 @@ The memory budget and its `vk: budget:` lines are in [architecture](architecture| `HOS_CORES=<1..8>` | The cores the game sees (6) || `HOS_RESERVE_CORE=0/1` | Keep a core for the replay (one is kept with 8 or more logical processors, but none on Windows CPUs whose L3 caches differ, unless `=1`) || `HOS_AB=<switch>=<value>[,<s>[,<warm-up s>]]` | Alternate a switch within one run for an A/B ([developing](../developing.md#an-ab-within-one-run)) || `HOS_CV_PARK=1` | Experiment, off, Windows: the game's events, semaphores and message queues sleep on a park that blocks at once after their own spin (`NtWaitForAlertByThreadId`, `NtAlertThreadByThreadId`: `hos_park_sleep` in `host/hos_thread.c`) instead of on Windows' condition variable, which spins 10240 cycles more (about 2.5 us) before it blocks. Other sleeps (a full queue's put, every wait with `HOS_SPIN_US=0`) keep that spin; mutexes, condition variables, read-write locks and WaitAny stay on Windows' condition variable. Read once; set, the log says which way they sleep (`os: HOS_CV_PARK: ...`) |L11 (minor): The switch reference and table describe the removed spin as about 2.5 µs without identifying the CPU or Windows build. The implementation also skips the spin when the adaptive policy skips the guest spin, and retains a park spin on timed semaphore waits and on sleeps after the first. Describe those cases and label the duration as an estimate for the reported setup.
@ -2244,0 +2269,4 @@v = 0;if (env_on("HOS_CV_PARK")) {const HMODULE nt = GetModuleHandleA("ntdll.dll");park_wait = (ParkWaitFn)(void *)GetProcAddress(nt, "NtWaitForAlertByThreadId");L1: The atomic flag publishes the pointers but does not protect their initialization. Two first callers on different objects can both read -1. One can publish and start using park_wait while the other is still assigning the same plain function pointer. Initialize the switch and both exports once before worker threads start, or use a real once primitive. Exercise simultaneous first calls in the fixture, which currently initializes the park before creating workers.
@ -2244,0 +2290,4 @@void **at = &((PCONDITION_VARIABLE)cv)->Ptr;SRWLOCK *s = park_stripe(cv);AcquireSRWLockExclusive(s);while (*at) at = &((Parked *)*at)->next;L12 (minor): The test result measures the stale-alert fix. It provides no evidence of a player gain or of the cost when the experiment is off. Compare
6e1521a7with the revised head both unset and enabled. Use interleaved runs in the Market and a heavy scene with matched settings, and report CPU per frame, wake latency, average FPS, 1% lows and median and p99 frame times on the affected Windows setups.@ -2244,0 +2296,4 @@ReleaseSRWLockExclusive(lk);/* (as RtlSleepConditionVariableSRW: no spin with one processor unparked; KUSER_SHARED_DATA's UnparkedProcessorCountand CyclesPerYield) */const int spins = spun || *(volatile USHORT *)0x7FFE036A <= 1 ? 0 : 10240 / (*(volatile USHORT *)0x7FFE02D6 | 1);L5 (minor): The spin divisor is not the advertised CyclesPerYield calculation. For example, an even value of 2 is treated as 3, and zero produces 10,240 pauses. Export lookup also does not establish that the two shared-data fields have the expected layout on the running Windows build. Avoid the private offsets or validate the supported layout and use the exact nonzero divisor with a conservative fallback.
@ -2244,0 +2320,4 @@}const LONG status = park_wait(cv, to);if (status == 0x101 /* STATUS_ALERTED */) {if (__atomic_load_n(&me.state, __ATOMIC_ACQUIRE) == PARK_WOKEN) break; /* (its wake's alert, taken) */L2: STATUS_ALERTED and PARK_WOKEN do not establish that the alert came from this wake. If a waiter receives an unrelated thread alert and the park waker publishes PARK_WOKEN before the waiter checks it, the waiter returns while its own alert is still to come. That later alert can reach the next job-barrier WaitOnAddress, the same failure mode reported in the PR. Use a waiter-specific notification whose delivery cannot be confused with another thread alert, and test an unrelated alert with the real wake held back.
@ -2244,0 +2323,4 @@if (__atomic_load_n(&me.state, __ATOMIC_ACQUIRE) == PARK_WOKEN) break; /* (its wake's alert, taken) */continue; /* (not this sleep's: none is left by the park, above; waited again as before) */}if (status != 0x102 /* STATUS_TIMEOUT */) continue;L3: An unexpected status from the native wait retries immediately with the same inputs. An implementation that exports the symbol but returns STATUS_NOT_IMPLEMENTED, or any persistent error, leaves the guest thread in a hot loop even after a timed wait's deadline. Establish which back end is usable before publishing it and define how an active park handles an unexpected wait result without dropping its registered waiter. Add a test that injects a persistent error.
@ -2244,0 +2356,4 @@while (w) { /* (off the list: only this wake reads them; each is the waiter's stack until its state is PARK_WOKEN) */Parked *next = all ? (Parked *)w->next : NULL;const DWORD tid = w->tid;if (__atomic_exchange_n(&w->state, PARK_WOKEN, __ATOMIC_SEQ_CST) == PARK_BLOCK) park_alert((HANDLE)(uintptr_t)tid);L4: The wake detaches the waiter, marks it woken and ignores the alert's return status. If the alert fails, an infinite waiter has no notification and future wakes cannot find it. A timed waiter also switches to an infinite wait once it sees that it was detached. Use a wake protocol that handles delivery failure without abandoning the detached waiter, and exercise a failed alert in the fixture.
@ -0,0 +32,4 @@import tempfilefrom pathlib import PathRUN_ALL_ARGS = ["--cc", "gcc"]L6: Both the suite declaration and the direct default hard-code gcc. A supported bundled llvm-mingw installation with clang and CC set can still skip the new Windows check or fail to launch its compiler. Honor CC and use the project's Windows toolchain discovery for the default so the check runs with the supported compiler.
@ -0,0 +89,4 @@now.QuadPart = 0;while (park_on() && park_wait((void *)&alerts_left, &now) != 0x102 /* STATUS_TIMEOUT */) InterlockedIncrement(&alerts_left);}static int probed_park_sleep(void *cv, void *lk, unsigned ms, int spun) {L10 (minor): The stale-alert regression check depends on stress scheduling. Its two controls remove an alert or change wake-all, so neither forces the window fixed by the last commit. Pause the waker after it changes the blocked waiter's state and before it sends the alert, then prove the waiter stays in the park until delivery. A control restoring the old early return should fail on that exact assertion.
@ -0,0 +186,4 @@WaitForMultipleObjects(4, h, TRUE, INFINITE);if (parked()) for (int i = 0; i < 4; i++) CHECK(order[i] == i);printf("wake-one order %ld %ld %ld %ld\n", (long)order[0], (long)order[1], (long)order[2], (long)order[3]);if (parked()) { /* (a sleep that spun blocks at once: its wake is one alert) */L9: The check for blocking without a second spin waits for PARK_BLOCK before waking the thread. It passes if park_sleep always spins, and no assertion sees whether event, semaphore or queue callers pass the intended spun value. Instrument the spin and sleep paths and drive each caller to a known blocked state. Assert the first-sleep policy, including the zero-budget case, and cover each object's actual back end. The printed switch state alone would also pass if a semaphore or queue were switched back to Win32 for both sleep and wake.
@ -0,0 +221,4 @@}static void release_all(HANDLE *h, int n) { /* (stop set: the waiters still asleep are let out) */__atomic_store_n(&stop, 1, __ATOMIC_SEQ_CST);while (WaitForMultipleObjects(n, h, TRUE, 1) == WAIT_TIMEOUT) {L8 (minor): release_all treats WAIT_FAILED like a successful join. If one CreateThread call fails during an auto-clear case, the remaining workers can still take every signal, then the invalid handle makes teardown return before those workers exit. Check allocation, thread creation and wait results, join the workers before reusing the globals, and close their handles.
@ -0,0 +223,4 @@__atomic_store_n(&stop, 1, __ATOMIC_SEQ_CST);while (WaitForMultipleObjects(n, h, TRUE, 1) == WAIT_TIMEOUT) {AcquireSRWLockExclusive(&ev->lk);ev->signaled = 1;L7: The teardown writes signaled with a plain store while event_peek can still read or claim it without taking the lock. The auto-clear stress cases can therefore race in the fixture itself. Use the production atomic publication for the teardown flag. Also use atomic reads for the sleeps snapshots taken while workers increment that counter.
Recommendation: Request changes. Fix how the Windows park initializes and delivers wakeups before testing it in the game. The existing exact-revision review documents those blockers. The disabled path also adds a spin-budget lookup that applies on Windows and x86-64 Linux. Existing review
The experiment lets events, semaphores and message queues block after their guest-side spin, and it takes the virtual core back before locking the object again. This could reduce CPU time spent waiting on a frame, but wakeup errors can stall the game or interfere with the next job-barrier wait on the same host thread.
Does it help? Unproven. The author reports 27 leftover alerts before the fix and none after it in one Windows fixture run per revision. That is evidence for the specific stale-alert fix. A matched game run against main is still needed to measure CPU time per frame, wake latency, average FPS, 1% lows, and median and p99 frame times.
Other systems. Normal, light and system events share the path. That includes notifications from host workers. The separate job barrier calls
WaitOnAddresson some of the same threads. WaitAny retains its own condition variable and is still notified throughpoke_multiwait.Switches.
HOS_CV_PARKis a Windows developer experiment, off by default and documented in the switch table and reference. Linux and a Windows installation missing either export use the existing condition-variable implementation. Measurements are needed to choose a shipping default.Reviewed. Head
82d36ed1000c93c10526953b3c316782ff2c2409against merge base6e1521a70a32603ed60a8263e4072625885669b7. Read all 11 changed files where they changed and traced the wait and wake paths, virtual-core handoff, Linux shim and job barrier with revision-matched local Graphify maps and direct source checks. Source review only. No changes made.Checked and fine. The waiter registers before releasing the object lock. The park releases its stripe before retaking the virtual core and the object lock. The wake-one path selects the oldest waiter, and the guest predicate is rechecked under the object lock. Finite wait steps stay below
INFINITE. The disabled path still calls the same platform condition-variable operations.Blocking
host/nn_os_sync.c:483repeatshos_spin_ns()when adaptive spinning skips a wait or the thread has a zero spin budget. Events, untimed semaphores and blocking queue reads pay for this extra lookup on x86-64 Linux and on Windows with the experiment off. Both paths discard the result. Carry the budget or skip reason out of the first lookup, then measure the disabled head against main.Testing. The reported Windows fixture passed at
06c08f8b, which the author says was rebased unchanged to this head. The strict changed-file run had 16 failures reported on both sides and skipped checks. CI was pending on this head, with the cached checks reporting blocked conditions. Full Windows and Linux host builds, in-game runs, and the full checks remain open. Add native Windows fixture coverage for timed semaphore acquisition and timed queue peek. Once the source issues are fixed, first validate repeated Windows starts through the title screen and Market play locally. Then compare exact main and the revised candidate with the switch on and off on supported Windows 10 and 11 CPUs, including AMD and a low-core system. The PR description proposes a farm run. The cached PR has no farm request comment.Overlaps and merge order. The auto-clear event handoff branch already contains this exact park head. Fix and measure the park on its own before rebasing that branch, and test the four combinations of the two switches on the combined head. The wake-latency diagnostic changes the same event loop and conflicts in a read-only merge check. Preserve its sleep count and the park's first-sleep behavior when rebasing. Event handoff · Wake-latency diagnostic · Related wake-path issue
@ -470,2 +481,3 @@int misses;if (__atomic_load_n(flag, __ATOMIC_RELAXED) > 0 || !spin_begin(o, &misses)) return;if (__atomic_load_n(flag, __ATOMIC_RELAXED) > 0) return 0;if (!spin_begin(o, &misses)) return hos_spin_ns() > 0;spin_begin() already reads hos_spin_ns(). This repeats the role/budget lookup on adaptive skips and for a thread with a zero spin budget, even with the park disabled or x86-64 Linux discarding the result. Carry the budget or skip reason from the first lookup, and include a disabled-head versus main comparison.
@ -0,0 +305,4 @@}return 0;}static void test_sema_queue(void) {Please cover timed semaphore acquisition and timed queue peek with the real Windows backend. This fixture only invokes the untimed semaphore and queue paths. Run each timed path through deadline expiry and a racing release or send, then check the waiter is removed and readiness is preserved.
82d36ed100ff7e23622bDescription updated for the push of
ff7e2362(onmain@f1f2dc5b): the Summary describes the review's fixes (set up once, a sleep ends only after its own wake is done, failed waits and alerts, one spin-budget lookup), Testing lists what ran in jobs fb3-1970-1 and fb3-1970-2, Still to test adds repeated starts and themain/ unset / set A/B, Measurements holds the new checks before and after, and Risks names the park's added costs.@futondestroyer thanks. Pushed
ff7e2362, rebased ontomain@f1f2dc5b(the first three commits unchanged,git range-diff"="). Each fix has a check intests/test_event_park.pythat fails at the test commit1e2adc46and passes at the head, run through the PC's job queue (jobs fb3-1970-1 and fb3-1970-2).e4b273b1):park_initruns once (InitOnceExecuteOnce) and publishes the mode after the two pointers (release, acquire). Check "first callers": 8 threads call the park at once with ntdll's lookups slowed; 16 lookups at1e2adc46, 2 at the head.PARK_DONEafter its alert (sent or failed); the sleep leaves only atPARK_DONE, then takes any alert still pending for its thread with a zero-time wait. Check "own": the wake's alert is held after it took the sleeper off the list, another alert is sent to the thread and, for a timed sleep, its deadline passes. At1e2adc46the sleep left on the other alert; at the head it stays asleep until the held alert. A control that leaves atPARK_WOKENfails it.park_inittries a zero-time wait and keeps the park off unless it answersSTATUS_TIMEOUTorSTATUS_ALERTED. A later unexpected status is logged once, and the sleep goes on inSleep(1)steps, still on the list, until its wake or its deadline. Checks "probe" and "fault_wait" (STATUS_NOT_IMPLEMENTED): at1e2adc46the park was used and a 30 ms sleep never returned. At the head the park stays off, the sleep times out after 2 waits, and a wake ends an untimed one.PARK_DONEis still set. A blocked sleep waits at most 100 ms per step and then checks its state. Check "fault_alert": with the alert failing, an untimed sleep and a 5 s timed sleep end at the wake. At1e2adc46the untimed one never ended.:483:spin_beginhands the budget it read tospin_until_setandspin_until_free(ff7e2362). Check "budget" countshos_spin_ns()calls: 2 before this commit, 1 at the head, whether the spin is left out, runs, or has no budget. With the park off, a wait now makes as many lookups asmainwhen the spin is left out, and one fewer when it spins. The disabled head againstmainis pending, asked of HostLiftPRs (CPU per frame and frame times in the Market:main, head unset, head withHOS_CV_PARK=1). I did not run a game-free benchmark: it would time a stand-in forhos_spin_ns(), which reads the host's thread table, not the call itself.tests/test_event_park.py:289: check "timed" runssema_acquire_forandqueue_get's timed peek on the real Windows backend, unset and on the park. It covers the deadline (it ends in the time asked, nothing taken, the list empty) and 100 rounds each against a release or send arriving 0-30 ms into a 5 ms wait. At the head on the park: 68 of 100 releases taken and 32 after the acquire timed out; 76 of 100 sends seen by the peek and 24 after it timed out. When it timed out, the count or message was still there; the peek left the message; the list was empty after each round. These also pass at1e2adc46: coverage, no fault found.git merge-treeof #1967's new head4d2ebc2awith this head conflicts inevent_waitat the spin's line (this PR'smissed, #1967'snapsand its comment): whichever lands second keeps #1967's sleep state (naps) and the park's first-sleep argument (missedand the first sleep only).BeanBot's review, at
ff7e2362(test commit1e2adc46, fixese4b273b1andff7e2362;tests/test_event_park.pythrough the PC's job queue):CyclesPerYielddivisor; 0 spins none.KUSER_SHARED_DATAhas a fixed layout that Windows only appends to; its offsets are the ones ntdll's own condition variable reads.--cc, then$CC, thenLONGSHOT_MINGW's clang, then clang, gcc or cc on PATH.RUN_ALL_ARGSis gone. It ran with no argument and withCCset.ev->signaledand atomic reads of the counters.PARK_WOKENfails it.@Beanstalk This adds a switch, HOS_CV_PARK, off by default: with HOS_CV_PARK=1, guest events, semaphores and message queues sleep on a park that does not spin again after their own spin (host/hos_thread.c, host/nn_os_sync.c). It has not run in the game yet. Please play Castle Town Market for two minutes, the PR against main on the same PC, on the PR's side with HOS_CV_PARK=1 and with it unset, on Windows 10, on a Windows 11 PC with an AMD processor and on a Windows PC with 6 or fewer logical processors. No crash or hang; set, the log must have the line starting "os: HOS_CV_PARK:" and frame time p99 equal to unset or better, within the run-to-run spread; unset, frame times as main's.
@siantm only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.The three renderer checks (
test_present_queue_default,test_target_reuse_memory,test_vk_occlusion_queries) did not compile in the earlier run (no Vulkan headers on that job's path); rerun with them in job fb3-vk-1, at the headff7e2362and onmain@f1f2dc5b: all three fail at both with the same first error, so they fail onmaintoo; Tested's sentence and counts corrected.Recommendation: Request changes. The new recovery path can return from a park with a native alert still pending. The next job-barrier wait on that Windows thread can receive it.
The experiment replaces the Windows condition variable used by guest events, semaphores and message queues. It skips the park's spin after the caller has already tried its own spin or the adaptive policy skipped it, and reacquires the virtual core before the object's lock. Normal, light and system events and notifications from host workers share this path. The OoT job barriers use
WaitOnAddresson some of those same threads. Guest condition variables, mutexes, read-write locks and WaitAny keep their existing backend. WaitAny is still notified throughpoke_multiwait.Does it help? Unproven. The author's Windows fixtures establish progress for the injected wait and alert errors, two native lookups for eight simultaneous first callers versus 16 before the fix, and one spin-budget lookup versus two. No matched game or microbenchmark comparison shows a gain. Measure CPU time per frame, wake latency, average FPS, 1% low FPS, and median and p99 frame times against main and the switched-off candidate. A blocked wake adds a zero-time system call and long sleeps check their state every 100 ms. The disabled Windows path still adds a mode check and the shared spin-budget change also affects x86-64 Linux.
Switches.
HOS_CV_PARKis documented, Windows-only and off by default as an experiment pending measurement. Windows falls back to its condition variable when the native exports or the initialization probe are unavailable. Linux uses the existing condition-variable shims.Reviewed. Head
ff7e23622b749cff8ac14e28323bb8818addd44dagainst merge basef1f2dc5bd2821cef061798ba40a043d281ac1707of the head and official target9dd41326e71094d8b774459dac5154eb5a3306d9. Read all 12 changed files where they changed and the three new commits relative to the first review. Verified both source-only Graphify maps and their extraction logs. Traced the wait, wake, virtual-core, compatibility and job-barrier callers, and checked the native dispatch and macros directly in source. Source review only. Longshot guidance9dd41326and Tooling0b0059ac.Checked and fine. The first three patches are unchanged after the rebase.
InitOnceExecuteOnceserializes setup and publishes the pointers with the mode. ThePARK_DONEhandshake now keeps a stack waiter alive until the waker finishes even when another alert arrives or the deadline passes. The 100 ms steps make a failed alert observable without waiting for the original deadline. A native wait error can reach a timed deadline, and the Windows fixture now covers the timed semaphore and queue-peek paths. The earlier duplicate spin-budget lookup is gone. Author's responseBlocking
host/hos_thread.c:2369discards the status of the final zero-time wait. With the runtime wait fault exercised bytest_fault_wait, the wait can fail while the waker'sNtAlertThreadByThreadIdsucceeds. The waiter seesPARK_DONEwhile polling, the drain also fails without taking the alert, andpark_sleepreturns. The nextWaitOnAddresscan consume the alert and leave the wait block behind forWakeByAddressAll, the same crash path described in the PR. The fixture disables its leftover-alert probe in this case, so it proves the waiter exits while the alert can still be pending. Make cleanup reliable after a wait error, or use a wake mechanism that cannot leave a thread alert for the next wait. Add a deterministic check that sends a real alert, makes the park's wait fail after successful initialization, then checks for a pending alert with the real native wait before the waiter thread exits.Testing. The author reports the focused Windows checks passed at this head, and 17 other strict changed-run failures also reproduce on the base, including the three rerun with Vulkan headers. The complete suite, full Windows and Linux host builds, repeated startup through the title screen, and gameplay measurements remain open. The cached description check passed while CI was pending and the farm awaited BigBean approval. Once the alert path is fixed and the local Windows start check passes, compare main and both switch arms in the Market and a heavier scene on Windows 10 and 11 across AMD CPU and low-core targets. For enabled runs, verify the positive park-mode diagnostic and check for native-call failures. The prefix in the current request also matches fallback. Strict fixture follow-up · Farm request
Overlaps and merge order. The event-handoff branch contains this exact park head. Establish the standalone park result first, then rebase the event-handoff branch and test all four combinations of its switch and the park switch on Windows. The wake-latency work changes the same
event_waitspin and sleep bookkeeping. Preserve the diagnostic's sleep count and the park's first-sleep argument in whichever lands second. Event handoff · Wake-latency diagnostic@ -2244,0 +2366,4 @@}LARGE_INTEGER now;now.QuadPart = 0;park_wait(cv, &now); /* (the alert its wake sent, if another alert or a step's end woke the sleep first) */If the native wait fails at runtime while the wake successfully sends an alert, the polling waiter can reach PARK_DONE and return even when this drain fails. The pending alert can then reach the same thread's WaitOnAddress in the job barrier. Please make the cleanup reliable or use a wake primitive without a pending thread alert. Extend the wait-failure fixture to send a real alert and probe for it with the real native wait before the worker exits.
View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.