perf(host): the game's job barriers wait for their mask words instead of sleeping 20 us (HOS_OOT_BARRIER_WAIT) #1662
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1662
Loading…
Reference in a new issue
No description provided.
Delete branch "GummiNaomi/Longshot:perf/oot-barrier-wait"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: medium · Tested on: Windows 11, RTX 4070 SUPER, Ryzen 9 9950X3D (SMT off) · Needs: a build of this head; a Steam Deck or another CPU of 4 cores; Linux
Stacked on #1661 (the in-run A/B tool): its commits come first; this PR is the last commit.
TL;DR
Three of the game's parallel jobs make their threads wait for each other by sleeping 20 microseconds and looking again. On Windows with few cores such a short sleep wakes about half a millisecond late, and the game's main thread spends a large part of each frame in it. This PR makes those threads wait until the job's progress word actually changes, woken by the thread that changes it. Standing in Castle Town's market with the game on 4 cores, frames were 3.2 % shorter in both of two in-run A/Bs, and a third fewer frames missed 60 fps; on 16 cores no difference.
Breakdown
Why
OoT runs three of its parallel-for jobs (f_800998a0, the drawing-command job; f_800738e0; f_8007e200) on several threads over one job block, each thread taking items by atomic ORs into bitmasks. A thread with nothing left to take sleeps 20 microseconds (
nn::os::SleepThread) and looks again, until the masks it polls are full. On Windows a sleep that short wakes about half a millisecond late when there are few logical processors, and the main thread is one of the waiters: on 4 cores in Castle Town's market it was asleep there in 11.4 % of its profile's samples, about 2 ms a frame. Waiting for the word itself to change, and being woken by the OR that changes it, ends the wait when the work is done instead of on the next timer tick.What changed
The lift's patch routes those three functions' barrier sleeps and their ORs through the host. A sleeping thread waits on the polled mask word until it moves (
WaitOnAddresson Windows, a futex throughhost/posixon Linux) with a 1 ms limit, and an OR into a mask wakes its waiters. After the wait the thread looks again exactly as after its sleep, so a missed wake costs at most the old sleep's time.host/oot_barrier.h,host/oot_barrier.c(new): the wait and the wake, the switch (on unlessHOS_OOT_BARRIER_WAIT=0),WaitOnAddresslooked up at the start (without it, the sleeps as before and a log line), and abarrier:log line every 16,384 looks (found moved already, waited, timed out, the average wait, the wakes). The switch is read throughhos_ab_value, soHOS_ABmay flip it between two looks.nx2/oot_patches.pypatch_barrier_wait: in each of the three functions, the OR into the mask gets the wake after it and the sleep gets the wait in front of it; a function is hooked whole or not at all (a warning names a function it could not hook, which keeps the game's sleeps). The places accept the page-mark lines a finished lift puts between lines and a loop's back edge with or without its poll; the three functions keep their registers in the context (their addresses are named in the file).HOS_OOT_BARRIER_WAIT=0row indocs/development.mdandgpu/vk/switches.def.tests/test_oot_barrier.py(new): the realoot_barrier.cand header over stand-in waits: threads that wait and are woken, no lost wake-up, the 1 ms limit, the switch's values (on for unset, 1 and yes; off for 0 and off); the patch on a small lifted text run through the lift's real poll and page-mark passes (hooked whole, the same text twice, before or after the polls, over a finished lift),touches()andmentioned_addresses()for the three functions; 9 broken variants of the code and 8 of the patch, each caught; on Windows (clang) and Linux (gcc 13).tests/test_hos_ab.py's list of wired switches gainsHOS_OOT_BARRIER_WAIT.Measurements
Verdict: faster on 4 cores: frame time -3.2 % in both runs (95 % intervals -5.1 to -1.9 % and -4.2 to -2.4 %), frames of 16.7 ms or more down 10 to 11 points (30 to 19 %, 26 to 16 %); on 16 cores no difference (intervals around 0).
Hardware: Windows 11, RTX 4070 SUPER (NVIDIA 617.42), Ryzen 9 9950X3D with SMT off, 32 GB
Scene: Castle Town's market (child), standing after a debug teleport from the
kokiri-startsave; output 2560x1440, uncapped (HOS_FPS_MAX=1000), no window (HOS_HEADLESS=1). "4 cores": the game kept to cores 8-11 (affinity 0xF00,HOS_CACHE_CORES=0); "16 cores": no affinity, the game on the 8 cores of the 96 MB L3 by defaultMethod: in-run A/Bs (
HOS_AB=HOS_OOT_BARRIER_WAIT=<value>,3,130: 3 s periods after 130 s), about 160 s standing,RX_VK_FRAME_TRACE=1;scripts/ab_summary.pyon each trace (transitions left out, a bootstrap of whole periods for the interval); two runs eachHow to reproduce: on this PR
HOS_AB=HOS_OOT_BARRIER_WAIT=0,3,130(side 1 is the old way); the runs also hadRX_VK_ASTC_BC_ALL=1 RX_VK_IMAGE_BLOCK_MB=32 RX_VK_LOAD=create RX_VK_BC_LAYERS=8 RX_VK_LATE_FIX=0 HOS_THREAD_TIMES=5Baseline: both sides of each run come from one build, main @
54a42715(93 commits before this PR's base) with the 18 Nov changes, the switch flipping every 3 s within the run. On 4 cores that build's default was the old sleeps and side 1 this PR's wait; on 16 cores (a later build of the same tree) the default was the wait and side 1 the old sleeps.54a4271554a42715(The 4-core runs' A/B value was
HOS_OOT_BARRIER_WAIT=1because that build's switch had the old sleeps as its default; the 16-core runs' was=0. Side 1 is the A/B's value in each.)Flags and settings
One switch added; the new behaviour is the default.
HOS_OOT_BARRIER_WAIT=0brings the sleeps backTesting
Tested: in the game, at 93 commits behind this base: the A/Bs above, on 4 and 16 cores, every run to its end with 0 errors, the market drawn in its screenshots, the
barrier:lines counting waits and no 1 ms time-outs. On this head:tests/test_oot_barrier.pypasses on Windows (clang) and on Linux (gcc 13, WSL Ubuntu 24.04);tests/test_hos_ab.pyandtests/test_patch_hooks_defined.pypass; of the 54 checks that read the changed files, the 22 that fail on this PC failed the same way on the base (run before the rebase onto main's last 11 commits; after it, the tests above pass again).Not tested: a build of this exact head: no lift of main exists on my PC, so the patch's places were read from the older tree's lift; the lift's
oot_patches: WARNING: the job barriers were found in N of 3 functionswould show a function main's lift writes differently (that function then keeps the game's sleeps). Linux in the game (the futex path is compiled and run by the test).Must be tested:
barrier:lines; thenHOS_AB=HOS_OOT_BARRIER_WAIT=0,3,60with the game on 4 cores in the market, expecting side 1 (the old sleeps) slower (by hand: me)Risks
What could break: a missed wake would make a barrier wait its 1 ms limit (the
barrier:line counts those: 0 in the runs above) and then look again as before, so the cost is bounded by the limit; a function whose text the patch doesn't find keeps the game's sleeps (with a warning). The job functions themselves are edited only at their sleeps and ORs.Turning it off:
HOS_OOT_BARRIER_WAIT=0.Compatibility: the lifted code changes (three functions keep their registers in the context), so the game is translated again; saves, options and caches otherwise as before.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). (The changed files' checks on this PC; failures are the base's own here.)third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Possibly related, for context:
5a8be09ec66328ed7577Rebased onto main (its base, #1661, landed with #1663; the one conflict, tests/test_hos_ab.py's WIRED list, now lists HOS_OOT_BARRIER_WAIT beside HOS_OS_YIELD). Approved to merge at the maintainer's request.
mooki referenced this pull request2026-11-18 23:27:04 +00:00