WIP: perf(os): each virtual core's owner slot on its own cache line #1973
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1973
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/vcore-owner-lines"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Depends on #1972 (
perf/vcore-fast, theHOS_VCORE_FASTpull request): this branch is that one plus onecommit. Its commit shows in this diff until it merges; only the last commit is this change.
With
HOS_VCORE_FAST=1a holder takes and frees its core's slot invc_ownerwithoutvc_lock. The eight slots areeight pointers on one 64-byte cache line, so each take and each release on one core takes the line from the holders of
all the others, who are taking and releasing their own slots at the same time.
This puts each slot on a cache line of its own:
vc_owner[8]becomes eight 64-byte-aligned slots (.t). The atomicsand their orders are unchanged. With
HOS_VCORE_FASToff every write is undervc_lock, and nothing changes but 512bytes instead of 64. No switch.
From cafb4100 (its
host/hos_thread.chalf) onexp/x86-asm-sync.Read first:
vc_ownerandvc_holderinhost/hos_thread.c.Testing
Tested: Windows 11, i7-9700, RTX 4060 (NVIDIA 610.88), llvm-mingw clang 23.1.5; game-free checks and a
micro-benchmark only. This change has not run in the game on this PC yet (Still to test). Each check
ran in a hidden console of its own, through the PC's job queue (batch 8), on this branch rebased onto
main@2e1f6c32(head 57257d0d, on
perf/vcore-fast3e65cad4):python -I -B tests/test_vcore_escape.py --cxx clang++: exit 0, "the defaults: ok", "the defaults,HOS_VCORE_STARVE_US=0: ok", "HOS_VCORE_FAST=1: ok", "HOS_VCORE_FAST=1, HOS_VCORE_STARVE_US=0: ok"; its layout check
(64 bytes a slot, the array on a line boundary) is part of it. On
main@2e1f6c32it does not build (its fixturedeclares
hos_fps_timingtwice).python -I -B tests/test_vcore_mask0.py --cxx clang++: exit 0, "the defaults: ok", "HOS_VCORE_FAST=1: ok". Onmain@2e1f6c32it does not build (noenv_on_orin its fixture).python -I -B tests/test_switch_table.py --cxx clang++: exit 0, "ok: 735 switches (diag 295, tune 109, kill 129,attempt 81, hw 67, player 54), 144 removed, 743 names read".
python -I -B tests/run_all.py --strict --changed origin/main(5 files changed withperf/vcore-fast, 54 checksread them): exit 1, "44 passed, 11 failed, 4 skipped".
test_vcore_escape,test_vcore_mask0,test_switch_table,test_invalidate_vcore,test_reserved_core,test_reserved_core_mask,test_short_sleep_timerandtest_spin_by_rolepass. All 11 failures fail onmain@2e1f6c32too (run by name in the same batch):test_crash_dialog,test_crash_record,test_dev_ab,test_dev_packs,test_frame_overlap,test_linux_pressure,test_present_queue_default,test_profiling_mode,test_slow_frames,test_target_reuse_memory,test_vk_occlusion_queries.The branch was then rebased onto
main@6e1521a7unchanged (git range-diff: "=" for both commits, head8f250090,on
perf/vcore-faste6acd10c);main's commits since2e1f6c32touch none of its files, so the checks were not runagain there.
Still to test:
HOS_VCORE_FAST=1against unset, at least two interleaved runs each, expecting the log line "vcores: takes andreleases without vc_lock while nobody waits (HOS_VCORE_FAST)" with it set, no "ran without a core" line, no crash,
and fps and frame times within the runs' spread or better (by hand)
python3 -I -B tests/test_vcore_escape.py, expecting it to pass withHOS_VCORE_FASToff and on (by hand)HOS_VCORE_FAST=1againstunset on this PR, expecting frame times within the runs' spread or better and no "ran without a core" line
(@Beanstalk play Castle Town's market for two minutes on PCs with 6 or fewer logical processors and on the Steam
Deck, this PR with HOS_VCORE_FAST=1 against unset, comparing frame times and looking for "ran without a core" in the
log)
Measurements
Verdict: a micro-benchmark only; no play measurement exists yet. In this micro-benchmark, with
HOS_VCORE_FAST=1a take and a release cost 74-89 % less than on
perf/vcore-fast, and 67-89 % less than the locked path(
HOS_VCORE_FASTunset: 170.1, 366.5, 587.0 ns at 2, 4, 6 threads).Baseline:
perf/vcore-fast(the pull request this one depends on), asmainhas noHOS_VCORE_FAST; both withHOS_VCORE_FAST=1.Setup: Windows 11, i7-9700 (8 cores, no SMT), llvm-mingw clang++ 23.1.5,
-O2; micro-benchmark:host/hos_thread.c'svcore block compiled with Windows' SRW locks, 2, 4 and 6 threads each taking and giving up its own core
(
hos_vcore_back,hos_vcore_away) with 8 virtual cores, so nobody waits; 15 interleaved rounds of 400 ms, the medianand the rounds' range; one timing job of the PC's queue. Measured at 15e55f96 and 026a1abc, these commits on
main@51f5f9da; their rebases since, now onto6e1521a7, changed none of their lines (git range-diff"=").(fast-H1:
perf/vcore-fast15e55f96 withHOS_VCORE_FAST=1; fast-H2: this PR, 026a1abc, with it; locked-H1:perf/vcore-fastwith it unset.)Risks
Nothing changes in what the scheduler does; 448 more bytes of static data. A mistake in the layout would show as a
build error (every access goes through
.t) or a failed layout check. A revert is enough. Players notice nothing.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Possibly related, for context:
Recommendation: Request changes. The stacked scheduler change can miss an affinity update, overwrite the guest’s TLS core value with stale data, and race with its own diagnostics. Padding the slots cuts isolated take-and-release time by 74–89%, but gameplay and the cost on the default path are still unproven.
The last commit gives each of the eight virtual-core owner slots its own 64-byte cache line, adding 448 bytes of static data. That is a small, focused improvement to the opt-in fast path. The prerequisite also changes core acquisition, handoff, affinity checks, and profiling, including some work done with the switch unset.
@fixwill do (13)host/hos_thread.c:2145: The mask saved before the search is separate from the mask used to choose a core. (more on the line)host/hos_thread.c:1895: The affinity store is atomic, but vc_mask, vc_allowed, vc_free_for and hos_thread_core still read core_mask directly. ideal_core is also shared without synchronization. For example, SetThreadCoreMask can update a running target while its unlocked take reads the old mask and new ideal. Use atomic access throughout and pass a consistent affinity snapshot to the selection helpers.host/hos_thread.c:168: hos_thread_core reads vc_held directly while a fast take or release can update it without vc_lock. The cross-thread GetThreadCoreMask and host-pinning paths can call it during that change. Take one atomic snapshot and use it for both the test and subtraction. Use an atomic snapshot for vc_last too when a remask caller reads it while the target’s fast take updates it.host/hos_thread.c:1934: vc_sync_tls_as samples the mask before taking tls_slot_lock. (more on the line)host/hos_thread.c:1881: The fast core preference reads the volatile slot index and its cached value without the lock used for slot allocation and freeing. If another thread frees and reuses the slot between the index snapshot and the cell read, the taker can treat an unrelated TLS value of 1 through 8 as a cached core. Validate the slot and read its cell under tls_slot_lock, and use consistent atomic access to cells that the scheduler and guest can access concurrently.host/hos_thread.c:2131: The fast CAS exposes the holder before it initializes vc_held_at and resets vc_kick_at. (more on the line)host/hos_thread.c:2041: The plain read of vc_st.takes and the struct-wide memset can race with a fast taker’s atomic batch addition. A batch arriving during the report can be cleared without appearing in either interval. Keep the counter separate and consume it with one atomic exchange. Flush each thread’s remaining batch on exit if the log is to count those takes too.host/hos_thread.c:2099: A waiter that finds a freshly released slot before the releaser gets vc_lock can grant it to another waiter. (more on the line)tests/test_slow_frames.py:127: The existing source check requiresvc_grant(c, t);invc_release. The assertion matches6e1521a7and cannot match this head, where the call takes three arguments and the locked path delegates to a helper. Update the check to verify that the releaser’s identity reaches a waiter on both paths. The earlier reported failure on both builds does not cover this new mismatch.tests/test_vcore_escape.py:310: The handoff tests only switch simulated threads once the waiter is asleep. (more on the line)tests/test_vcore_escape.py:373: Both documented standalone fixture invocations default to clang++ even when CXX selects another compiler. A Linux GCC-only checkout can run the suite through run_all.py but cannot run the proposed standalone check without supplying an extra option. Use the suite’s compiler selection order in both fixtures, with an explicit --cxx taking precedence.host/hos_thread.c:1823: Keep the original locked operations when the experiment is off to reduce the default-path cost. Make the take, wait and escape counters reliable for the gameplay comparisons.For the author (1) (a run, a measurement or a decision)
host/hos_thread.c:1823: The proposed head-on versus head-off run compares two scheduler modes and cannot isolate the padding or show what ships versus main. (more on the line)Checks: Still running:
CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request),Website / Website on Cloudflare (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven in gameplay. The PR reports 15 interleaved Windows microbenchmark rounds at 15e55f96 and 026a1abc with HOS_VCORE_FAST=1 and no queued waiters. Median take-and-release times fell from 218.5 to 56.0 ns at two threads, 436.3 to 59.0 ns at four, and 633.7 to 67.2 ns at six, with disjoint ranges. The locked comparison used the prerequisite build. There are no gameplay FPS, 1% low, median or p99 frame-time results, and no main-versus-head comparison with the switch unset.
Other systems: The scheduler is shared by Windows and Linux and can affect pacing on any GPU, particularly on PCs with few logical processors and on Steam Deck. Guest sync, file and memory operations, video, and Vulkan waits give up and retake cores through this path. Audio and input can be affected through guest scheduling. Frame Breakdown, Tracy, HOS_CHAIN and long-wait logs also observe it. The diff contains no renderer API, save format, installer, or build-flag changes.
Switches: The last commit adds no switch. #1972 adds HOS_VCORE_FAST as a developer A/B switch, off when unset, with matching entries in switches.def and the reference docs. Off is appropriate while the fast path has correctness issues. The final default still needs gameplay measurements. HOS_VCORES=0 continues to bypass virtual-core scheduling.
Reviewed:
8f250090a5against6e1521a70a: Read host/hos_thread.c, both changed test files, both commits and the full diff, plus AGENTS.md, CONTRIBUTING.md, docs/dev/standards.md and the Forgejo template. Read the relevant host startup, affinity, TLS, thread destruction, Linux wait, and source-check code. Skimmed the unchanged switch reference and registry entries, CI definitions, profiling consumers, and reported run output. There was no local docs/review-checklist.md or sibling Tooling checkout. Comments and reviews were empty.; nothing run (a code review).Checked and fine:
Testing: Read-only review. No project code, build, or test was run. The author reports passing Windows fixture and switch-table checks on the earlier equivalent rebase. Linux host build, gameplay, Tier 1 runs, and head CI results are still pending.
Overlaps and merge order: #1972 is the scheduler prerequisite and supplies the fast-path baseline. The repository groups steps in one performance track into one PR, so fold the padding into #1972. If maintainers keep the stack, land #1972 first and rebase #1973 onto it.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -1818,1 +1821,3 @@static HosThread *vc_owner[8];/* (each core's slot on its own cache line: a take or a release on one core does not take the line from the other cores'holders, whose own takes and releases read and write their slots without vc_lock: HOS_VCORE_FAST) */static struct { HosThread *t; } __attribute__((aligned(64))) vc_owner[8];L13 (minor): The proposed head-on versus head-off run compares two scheduler modes and cannot isolate the padding or show what ships versus main. Run warmed, interleaved comparisons of #1972 versus head with fast mode enabled, and main versus head at shipping defaults. Report average FPS, 1% lows, median and p99 frame times in the market and a heavy scene, with the author’s in-game run and coverage on Windows, Linux and Steam Deck.
@ -1890,1 +1931,4 @@did under vc_lock: with HOS_VCORE_FAST, t may have let c go and taken another core without the lock since; that takestores vc_held before it writes the cell, and a release clears vc_held before it frees the slot) */static void vc_sync_tls_as(HosThread *t, int c, int held_only) {const uint32_t mask = vc_mask(t);L4: vc_sync_tls_as samples the mask before taking tls_slot_lock. A fast taker can sample a multi-bit mask, pause, then overwrite the core number that a concurrent same-core remask to one bit just wrote. No move is requested, so the cell can stay zero until another take. Read the current mask inside the TLS critical section and cover the order where the remask writes first.
@ -2026,0 +2096,4 @@bound is for cores freed again meanwhile, whose releases give them on themselves */static void vc_settle(HosThread *t, int queued) {int c;for (int n = 0; n < 64 && !t->vc_held && (c = vc_free_for(t)) >= 0; n++) {L8 (minor): A waiter that finds a freshly released slot before the releaser gets vc_lock can grant it to another waiter. That emits a HOS_CHAIN signal from the settling waiter and loses the actual release attribution. Tracy can also print an empty “given by” label for grants with no named releaser. Distinguish claims of an already free slot from release handoffs in the trace and give unnamed grants a valid label.
@ -2026,0 +2128,4 @@__atomic_store_n(&t->vc_held, c + 1, __ATOMIC_SEQ_CST);t->vc_last = c + 1;t->vc_starved = 0;t->vc_held_at = hos_now_ns(), t->vc_kick_at = 0; /* (vc_long_wait: how long a holder has held its core) */L6: The fast CAS exposes the holder before it initializes vc_held_at and resets vc_kick_at. A waiter holding vc_lock can read vc_held_at or write vc_kick_at while the fast taker writes them. A kick recorded just after the CAS can also lose its timestamp. Publish initialized per-take metadata before exposing it to readers, synchronize subsequent updates, and read vc_kick atomically even in diagnostics.
@ -2026,0 +2142,4 @@ReleaseSRWLockExclusive(&vc_lock);if (wanted) return 0;}if (vc_mask_changed(mask, __atomic_load_n(&t->core_mask, __ATOMIC_RELAXED))) {L1: The mask saved before the search is separate from the mask used to choose a core. A taker can save mask 1, choose core 1 while the mask is 2, then claim it after the mask returns to 1. Both setters can see no held core, and the final comparison sees 1 on both sides, so the thread runs on a core its final mask excludes. Select with one affinity snapshot, then validate the claimed core against a fresh snapshot and retry if it is excluded.
@ -240,0 +307,4 @@check(!h0.vc_kick && h0.vc_held == 1, "a kick with nobody queued is cleared at the checkpoint, and the core kept");check(hos_vcore_away() == 1, "a holder gives its core up");h0.vc_kick = 2; /* (a kick set on a thread that has just let its core go: cleared at its next take) */hos_vcore_back();L10: The handoff tests only switch simulated threads once the waiter is asleep. They never queue a waiter between the fast taker’s two queue reads, race two claimants, or destroy a thread while a scanner still holds its record. The fixture could pass with the post-claim queue check or destruction barrier removed. Add controlled cases that exercise those boundaries and fail when the corresponding protection is removed.
Recommendation: Request changes. Fold the owner-slot padding into the virtual-core fast-path change after fixing that branch's affinity and synchronization issues. Address the owner scans on the locked path and measure the combined change in gameplay.
New finding. At
host/hos_thread.c:2208,2213and2218, the wait loop reads the owner before checking whether the waiter can use that core. With this layout, a waiter pinned to a single core can read all eight owner cache lines undervc_lock. It can do this again on each timed wake, even withHOS_VCORE_FASTunset. The equal-priority scan on main checks affinity before loading the owner. Check affinity before each of these loads and measure a route with queued core waits. The no-waiter benchmark never enters these scans.Inherited blockers. The padding leaves the prerequisite's ownership and affinity protocol in place. The earlier review found that a fast take can miss an affinity update and race with scheduler metadata readers. I also confirmed that when a fast take chooses a core under a nonzero mask and a concurrent remask restores mask zero before the claim, the take can overwrite the old cached preference before its requested checkpoint checks it. It can then keep the newly chosen core even if the cached core was free. Validate affinity and the preferred-core selection before publishing the cache, and cover this interleaving with a focused check. The old
test_slow_frames.pyassertion is already stale on main becausevc_releasedelegates tovc_release_locked. Update it for the new signature and verify attribution through both release paths. Existing exact-head review · PrerequisiteDoes it help? The author reports a 74–89% reduction in median uncontended take-and-release time with the fast path enabled on an i7-9700. Across 15 interleaved Windows rounds, medians went from 218.5 to 56.0 ns at two threads, 436.3 to 59.0 ns at four and 633.7 to 67.2 ns at six, with disjoint ranges. The locked reference numbers came from the unpadded prerequisite with its switch unset. The benchmark arms leave the gameplay gain and the cost of the shipping path against main unproven. Compare current main with the corrected combined head at its shipping default. Compare the corrected, unpadded and padded fast paths with the switch enabled if attributing the gain to padding. PR measurements
Other systems and switches. This changes a host scheduler used on Windows, Linux and Steam Deck. Guest synchronization, thread joins and NVN waits give up and reacquire cores through it. The padding adds 448 bytes of static storage and changes the layout in both modes. The last commit adds no switch. The prerequisite registers
HOS_VCORE_FASTand defaults it off. With the switch unset, the stack also introduces stronger atomics on the locked path.Checked and fine. All seven production uses of the owner array still access the same core index and preserve the prerequisite's atomic operations and memory order. The extracted fixture checks the 64-byte stride and base address alignment. The padding leaves the core-selection rules and thread teardown protocol untouched.
Testing. The author reports both virtual-core fixtures and the switch table passing on an earlier Windows Clang rebase. The changed-file sweep reported 44 passes, 11 failures and four skips. The captured Forgejo description check passed and source checks were pending. Author gameplay, Linux host build and Tier 1 gameplay coverage are open. Use warmed, interleaved Market and busy outdoor runs, including a route with measurable core waits. Report average FPS, 1% lows, median and p99 frame times, and wait and escape diagnostics.
Overlap. Keep this padding with the corrected virtual-core fast path as one scheduler PR. Keep the separate serial-core spin experiment out of the comparison so its changes do not confound the result. Fast path · Serial-core spinning
Reviewed. Head
8f250090a5fcdb586f84e2a82bc4b73b68d87a7eagainst main6e1521a70a32603ed60a8263e4072625885669b7and the unpadded parente6acd10c727c3fefbd3981129756c58bd3c005eb. Read all five changed files at the touched areas, both fixtures, and relevant wait, affinity, lifecycle and statistics callers. Used revision-matched local source maps and checked the findings in Git source. Source review only. Longshot guidance6e1521a7, Tooling0b0059ac.@ -2063,3 +2206,3 @@HosThread *worst = NULL;for (int k = 0; k < vc_cores(); k++) {HosThread *h = vc_owner[k];HosThread *h = vc_holder(k);Check the waiter's affinity before loading the owner here and in the equal-priority and starvation scans below. A waiter pinned to one core now reads up to eight separate cache lines under vc_lock and can repeat the sweep on each timed wake, even with HOS_VCORE_FAST unset. Include a route with actual queued core waits in the comparison. The reported no-waiter benchmark does not exercise these scans.
Folded into #1972 as asked in 62914: its commit is
1cde9675there.Pull request closed