WIP: perf(os): HOS_OBJ_CACHE=sets, the per-thread sync object cache as 2048 sets of 8 ways of object numbers (experiment) #1969
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1969
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/obj-cache-sets"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Every
nn::osmutex, event or queue call looks its host object up by guest address (obj_nockptinhost/nn_os_sync.c), first in a per-thread cache. #1781 made that cache 16,384 entries of four ways (HOS_OBJ_CACHE).exp/x86-asm-syncsolved the same problem in another shape, and the two were never measured against each other. ThisPR adds that shape beside #1781's, off unless
HOS_OBJ_CACHE=sets:thread, in place of the default sized cache's 256 KB, made at its first lookup and freed as the thread ends). The
cache is the host thread's: its pointer a thread-local, and its release the thread's own exit, whatever fiber it runs
then (on Windows the image's TLS callback,
DLL_THREAD_DETACH; on Linux a pthread key's destructor). Not a fiber'sslot (FLS): a guest fiber is a Win32 fiber (
hos_thread.c) that the game may delete while the thread that ran it runson, and its slot's callback would free the cache under that thread;
can fill while others are empty; a miss replaces a way of the address's tag whose object is no longer its, else an
empty way, else the next of the two sets' 16 ways in turn (a per-thread counter);
setsthe table numbers its objects as it makes them (obj_new_locked, chunks of1,024 that are never freed, as objects never are), so an entry is a 32-bit number (
obj_of_id). Withoutsets(unset,a size,
0, orHOS_NO_OBJ_CACHE), each object is made alone, as onmain.Every lookup that reads an object without
table_lock(every cache andfind_unlocked) checks it withobj_is, whichreads its key and kind as one identity, of one life, in one load: beside its key and kind an object keeps one 64-bit
word of both (
ident: the kind in the top 4 bits over a key under 2^60, a guest address; 0 for a tombstone or a largerkey), written after every change of its kind (
obj_kind_set, release) and compared once (acquire) with the word thelookup makes from its key and kind (
obj_ident). An object is given another identity (a spare taken by another address,a tombstone made again at its address) only as a tombstone, whose word is 0, and gets its new word last. Read separately, as before, a kind read
before a reassignment could pair with a key read after it: an object given to another address and then back to its
own as another kind passed for the old kind (a mutex lookup returning the address's new event), and a spare could
reach
holder_readyundermw_lockfor the wrong address.main's pointer caches (the sized default, the fixed one)and
find_unlockedread the same two fields separately and use the same check, so they are fixed here too. What alookup returns was that address's object of that kind when it read it, and stays so until the guest drops it: objects
are never freed, and a dropped one is reused only after 30 s in the graves and not while a condition variable waiter or a
multi-wait holds it (
bury_locked).HOS_NO_OBJ_CACHEandHOS_OBJ_CACHEare read once into one atomic word (obj_cache_mode_read; two first lookups atonce both store the same value), as is
HOS_SYNC_TABLE_LOCK;obj_cache_entriesreturnsOBJ_CACHE_SETSforsets, sothe lookup's path compares no string. A thread whose sets cannot be had (allocation or owner) goes without and does not
ask again. An object that cannot be had ends the process through
hos_fatal, not a write through NULL. The hits andmisses count in the existing
sync: object cachelog line. The switch's row ingpu/vk/switches.def(now a string)names the new value.
From 3cf76270 on
exp/x86-asm-sync, where it replaced the 1,024-entry cache (itsHOS_OBJ_CACHE_SMALLisHOS_OBJ_CACHE=0onmain).Read first:
obj_ident,obj_kind_set,obj_isand the two reassignments inobj_lookup;obj_lookup_sets,obj_set_find,obj_sets_thread_end,obj_new_lockedandobj_nockptinhost/nn_os_sync.c;tests/test_obj_cache_size.py(itsreuse,churnandfibermodes).Testing
Tested: Windows 11, i7-9700, RTX 4060 (NVIDIA 610.88), llvm-mingw clang 23.1.5; game-free checks and a
micro-benchmark only. This change has not run in the game on this PC yet, with the switch on or off (Still to test).
Each check ran in a hidden console of its own, at most 20 minutes, in two jobs of the PC's job queue (fb3-1969-r4l,
fb3-1969-r4b), on this branch on
main@9dd41326(head712ee7e7; the first job's tests ran at c02418f4, the same treebefore its commit message was reworded):
python -I -B tests/test_obj_cache_size.py(with--ccllvm-mingw clang, withCC, and with the compiler foundfrom
LONGSHOT_MINGW): passes. "ok: obj_cache_entries"; the 4,000-object walk's last pass reaches the table 47 timesunset, 4,000 times with
=0and=256, once with=sets(the sets case must stay at 10 or under); every lookup thetable's object over 200,000 random operations; the log line;
reusewithsets, unset,0andHOS_NO_OBJ_CACHE=1:the object reassigned before the lookup and after each of its loads (3 with
sets, 2 unset and0, 6 or 7unlocked), three ways, "ok";
churn: "64 threads made and ended: 2621511 bytes in allocations of 64 KB and up before,2621511 after";
fiber: "2752646 bytes ... with it, 2752646 once the fiber is deleted (the thread running on); thethread's next 4000 lookups reached the table 0 times; 0 bytes written in freed memory; 2621511 after the thread ended,
2621511 before it". With the same test file (
b7554f7b) on 479a4f90's source (the sequence check) it passes too; on9baebe0d's source (kind, then key) both cases fail:
reusein all four on the third way ("returned the object nowits own address's again, of another kind (another's in between) ... reassigned right after the lookup's load 2" unset
and
0, load 3 withsets, load 6 unlocked), andfiber("2719878 ... with it, 2588743 once the fiber is deleted...; reached the table 4000 times; 30053 bytes written in freed memory").
python -I -B tests/test_multiwait_wake.py: passes ("targeted: ok", "all: ok", the source check;__atomic_thread_fenceno longer allowed undermw_lock: none is made; the FLS calls no longer).python -I -B tests/test_cv_handoff_threads.py(it builds the real struct andobj_lookup's branch for an addressmade again): passes.
sizeof(Obj)(the struct as the tests cut it, llvm-mingw clang for x86-64 Windows): 344 bytes onmainand at479a4f90, 352 at this head (the identity word: the fields beforeowner40 bytes, now 48).host/nn_os_sync.ccompiled with the host build's flags (-fsyntax-only, with and without Tracy): no error. (At479a4f90, job fb3-1969-r3l, as an object with-ffunction-sections -fdata-sections, its.CRT$XLFsection (notCOMDAT) held the one relocation to
obj_sets_thread_end; this round does not touch it.)python -I -B tests/run_all.py --strict --timeout 1200 --changed origin/main(4 files changed, 37 checks read them; 8 atonce;
RX_CPMset, so the Vulkan headers were found): 32 passed, 7 failed, 3 skipped (Linux only). All seven fail onmain@9dd41326too, run by name in the same job with the same first error:test_crash_dialog(aUnicodeDecodeErrorreading a child's output),test_dev_ab("'"Skip title screen", &opts.skip_title' not found" inthe menu source),
test_frame_overlap("the frame ahead line is gone from the log or from the page"),test_linux_pressure("texture_frame() no longer has: pool_pressed = true;"),test_present_queue_default("srccut:'{TAB_DEBUG, "Present queue", ...' is not in gpu/vk/vk_menu.cpp"),
test_target_reuse_memory("'vmaGetAllocationInfo2'unexpectedly found"),
test_vk_occlusion_queries("no member named 'policy'" in its ring fixture). None fails on thisbranch alone.
Earlier, before the last two rounds' changes, the same change on
main@51f5f9da(fff612ab), withmain@e0ef1b57beside it:python -I -B tests/test_obj_cache_size.py: passes on both. Here also withsets:obj_cache_entries(withsetsand its near misses), the 4,000-object walk's last pass reaches the table 47 times unset, 4,000 times with
HOS_OBJ_CACHE=0and=256, once with=sets; every lookup the table's object over 200,000 random operations; thelog line.
python -I -B tests/test_multiwait_wake.py: passes on both. It now allows__builtin_ctzundermw_lock(thecompiler's, one instruction, used to walk a set's ways); its other checks and controls are unchanged.
python -I -B tests/run_all.py --strict --changed origin/main(35 checks read the 4 changed files): 31 passed, 6failed, 3 skipped (Linux only). The six (
test_crash_dialog,test_dev_ab,test_frame_overlap,test_linux_pressure,test_present_queue_default,test_vk_occlusion_queries) also fail in the whole suite onmain@e0ef1b57on this PC.Still to test:
HOS_OBJ_CACHE=setsand unset, interleaved, expecting fewer misses in the
sync: object cacheline withsets, fps and frame times withinthe runs' spread or better, and unset the same as
main(by hand)python3 -I -B tests/test_obj_cache_size.py, expecting it to pass (by hand)market for two minutes with
HOS_OBJ_CACHE=setsagainst unset, expecting frame times within the runs' spread orbetter (asked of the Test Farm)
main@9dd41326against this head unset and withHOS_OBJ_CACHE=sets, interleaved and warmed: the time tomake the game's objects at a cold start, resident memory after the threads' churn, and the Market's frame times
(by hand)
main:tests/test_obj_cache_size.py --benchtwice each, interleaved,main@9dd41326(its own test file), 479a4f90's check (sequence, kind, key, sequence) and this head's (one word),the last two with this head's test file (its
--benchbuilt without the per-load watch), expecting this head unsetwithin the runs' spread of
main's unset and below 479a4f90's (by hand: held for the PC's queue of run-alone jobs,fb3-1969-bench-r4)
Measurements
Verdict: micro-benchmark only, not measured in play yet: on a walk of 4,000 mutexes among 60,000 objects, the sets
took 9.8 and 10.6 ns a lookup against 14.2 and 13.6 ns for
main's default (16,384 entries of four ways) and 11.2 and11.0 ns for 65,536 entries.
HOS_OBJ_CACHE=setsships off (unset keepsmain's default) until a measurement in playchooses it.
The identity check (micro-benchmark). Round two's check (sequence, kind, key, sequence;
479a4f90) against roundone's (kind, then key;
9baebe0d), both with round two's test file, on i7-9700, Windows 11, llvm-mingw clang 23.1.5-O2:python -I -B tests/test_obj_cache_size.py --bench(ns a lookup for the walk below, best of 7), twice each,interleaved, in one
timing-cpujob of the PC's job queue (fb3-1969-bench-r3). Median of the two runs (range):9baebe0d479a4f90HOS_OBJ_CACHE=0(the fixed 1,024 entries)HOS_OBJ_CACHE=4096HOS_OBJ_CACHE=65536HOS_OBJ_CACHE=setsThe sequence check cost about 3 ns a lookup in every mode, the default included (+16 % unset), and left
setsonlyabout 5 % ahead of unset. That harness calls a watch function after every atomic load of the cut code, in
--benchtoo, so a check with more loads also paid for more calls. This head reads one word in one load (a lookup unset makes 2
atomic loads, the mode word and that word, where
479a4f90made 5) and its--benchis built without the watch; thishead against
main@9dd41326and479a4f90is held for the PC's run-alone queue (fb3-1969-bench-r4): Still to test.mainitself has not been measured with this harness yet.Earlier figures. Not measured again since the last three rounds' changes (the identity check above; without
setsobjects are made one at a time again, as on
main): the figures below are of fff612ab, both arms with that build'schunk allocator and its key-then-kind check. Cold creation and resident memory after thread churn (
main, this headunset, this head with
sets) are not measured yet: Still to test.Baseline:
main's cache sizes as they ran in that PR build (their code unchanged there), so both columns are one buildand one run of
tests/test_obj_cache_size.py --bench.Setup: Windows 11, i7-9700, llvm-mingw clang 23.1.5,
-O2;python -I -B tests/test_obj_cache_size.py --bench(thereal table, caches and lookup cut from the source, one thread): ns a lookup for a walk of 4,000 mutexes at a 0x80
stride among 60,000 other objects, best of 7, run twice in one
timing-cpujob of the PC's job queue, at fff612ab (thischange on
main@51f5f9da; the rebases since, now onto6e1521a7, changed none of its lines:git range-diff"=").sets(16,384 entries, 2,048 sets of 8)HOS_OBJ_CACHE=65536HOS_OBJ_CACHE=4096HOS_OBJ_CACHE=0(the fixed 1,024 entries)Risks
Off by default: unset,
main's 16,384-entry cache runs as before, objects made one at a time as before. What changesfor everyone: every cache and
find_unlockedcheck an object's identity with its identity word (the fix above), andan object that cannot be allocated ends the process with a message instead of a crash. The check's cost is on every
cached lookup, sets or not: a hit reads one word of the object (
mainreads its key and its kind, two loads) and makesthe word to compare from the key and kind (a shift, an or and a test of the key's top bits); every change of an
object's kind (under
table_lock) adds one store. Round two's check measured about 3 ns a lookup over round one's inevery mode (Measurements); this one against
mainis not measured yet (Still to test). An object is 8 bytes larger(344 to 352 bytes on Windows: the word). A guest address of 2^60 or more (none on the Switch, whose addresses are under
2^39) is never a cache hit: its lookup takes
table_lockevery time.With
HOS_OBJ_CACHE=setseach host thread that looks an object up holds 128 KB (the default's 256 KB not made), freed bythe thread's exit callback (the image's TLS callback on Windows: it runs for every thread that ends, the game's and
the host's, and does nothing for one without a cache), and objects are made in chunks of 1,024. A wrong entry would
hand a lookup another object; an entry is only taken while its object is still that address's of that kind
(
obj_is), andtests/test_obj_cache_size.pycompares every lookup with the table over 200,000 random operations andreassigns the object a lookup reads at every point of the lookup. Turn it off by leaving
HOS_OBJ_CACHEunset; arevert is enough. Players notice nothing.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Recommendation: Request changes. The new cache can return a recycled object for the wrong guest address and leaks its allocation when a thread exits. The synthetic walk is faster, but the benchmark leaves the allocator change shared by both sides and there are no in-game results.
Adds an opt-in, two-choice cache for guest synchronization objects. It cuts misses on the synthetic walk from 47 to 1 per 4,000 lookups and uses a 128 KiB heap cache per participating thread instead of the default's 256 KiB. It adds ID decoding and changes object allocation even for players who leave the switch unset.
@fixwill do (15)host/nn_os_sync.c:350: The cache validates key and kind from separate object lifetimes. (more on the line)host/nn_os_sync.c:360: Thread exit loses the only reference to each sets allocation, and the code discards the original pointer immediately after alignment. (more on the line)host/nn_os_sync.c:394: The new mode dispatch reads the inherited plain shared off and entries variables while another thread can initialize them. (more on the line)host/nn_os_sync.c:445: The new allocator can return NULL if both the chunk and individual allocation fail. (more on the line)host/nn_os_sync.c:352: A matching tag with a different live key is treated as a stale entry. (more on the line)host/nn_os_sync.c:361: After a cache allocation fails, every lookup on that thread retries the same 128 KiB allocation, including lookups under mw_lock. (more on the line)host/nn_os_sync.c:325: The inherited cache logger now also serves sets. It reads obj_cache_logged_ns and obj_cache_made through volatile accesses while other threads update them with Interlocked operations. Use atomic loads for those reads. The looks and misses totals also need a coherent snapshot so a thread adding a batch between the two exchanges cannot put its misses in a different reporting interval from its lookups.host/nn_os_sync.c:373: Hits return before checking whether the thread's counters are due. (more on the line)tests/test_obj_cache_size.py:140: The sets case also passes if the new dispatch line is removed. (more on the line)tests/test_obj_cache_size.py:143: The fixture claims to exercise a buried object's memory being reused at a different address. (more on the line)tests/test_obj_cache_size.py:221: The child process inherits HOS_KEEP_DEAD_OBJECTS and HOS_SYNC_TABLE_LOCK, both of which are read by the extracted code. (more on the line)tests/test_obj_cache_size.py:194: Direct runs default to clang, and RUN_ALL_ARGS forces clang for suite runs. A supported Linux setup with CC=gcc and no clang cannot run this check through either entry point without overriding the test command. Honor CC when --cc is absent, use an available supported compiler as the fallback, and remove the suite's hardcoded override.gpu/vk/switches.def:135: The switch row still declares int although sets is an accepted literal. (more on the line)host/nn_os_sync.c:445: Keep single-object allocation outside sets mode so the experimental switch also gates the allocator change. Measure the selected mode's gain in play.For the author (1) (a run, a measurement or a decision)
host/nn_os_sync.c:445: The same-build microbenchmark isolates cache shape but cannot establish the cost of chunk allocation for everyone. (more on the line)Checks: Still running:
CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven in play. The PR reports 9.8 and 10.6 ns per lookup with sets versus 14.2 and 13.6 ns unset, 22–31% lower, on an i7-9700 with llvm-mingw Clang at fff612ab. Each figure is the best of seven, and both modes use the PR's allocator. There are no matched runs against the merge base for frame times, 1% lows, startup time or RAM.
Other systems: The lookup serves mutexes, condition variables, events, timers, semaphores, queues, reader/writer locks and WaitAny, including system events used by video, vsync and applets. Cache misses can allocate while holding table_lock, including on a WaitAny scan under mw_lock. The host path is shared across GPU vendors on Windows and Linux, including Steam Deck. The source shows no new virtual-core scheduling point and no direct change to the renderer API, shaders, VRAM, saves, input mappings or build flags. Gameplay, wake timing and memory pressure remain unverified.
Switches: HOS_OBJ_CACHE adds the exact value sets. Unset, empty and invalid values still select the 16,384-entry four-way cache. Numeric sizes and 0 retain their meanings, and HOS_NO_OBJ_CACHE still overrides cache lookup. The row lists sets but labels the value type int. Keeping the existing cache as the comparison default makes sense for the experiment. The shared allocator and shipping default still need evidence on supported hardware.
Reviewed:
b8c6309640against6e1521a70a: Read host/nn_os_sync.c, tests/test_obj_cache_size.py and tests/test_multiwait_wake.py in full. Read the full diff and commit, AGENTS.md, CONTRIBUTING.md, the Forgejo template and developer standards. Read the changed switches.def row and related rows, the switch reader, test runner, Linux lock shim, thread exit and host event callers. Skimmed the rest of the switch catalog, related source checks and recorded object-cache runs. The older PR test heads were unavailable locally. There was no local review checklist or adjacent Tooling checkout. No translations or generated files changed.; nothing run (a code review).Checked and fine:
Testing: Read-only review. The author reports the two focused fixtures passing on Windows at f2c967b1 and seven broader-suite failures reproduced on its then-base. The author reports an unchanged rebase to
b8c63096. Linux, gameplay and Forgejo CI results are pending.Overlaps and merge order: No overlapping open PR was found in the supplied evidence or local history. #1781 is already on main. The description identifies exp/x86-asm-sync as the source of the experiment.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -133,3 +133,3 @@SW(HOS_NVN_WATCH_RT, diag, string, "off", "Logs who sets the render target at that guest address (hex), as guest call chains.")SW(HOS_NVN_WATCH_TEX, diag, string, "off", "Logs who binds that texture handle (hex), as guest call chains, each once.")SW(HOS_OBJ_CACHE, tune, int, "16384: each thread's cache of the host objects behind guest mutexes, events and queues has that many entries, four ways a set with a mixing hash, and a sync: object cache line is logged every 10 s", "<entries> (256 to 65536, a power of two): that many entries; =0: the fixed per-thread cache of 1024 direct-mapped entries, as before.")SW(HOS_OBJ_CACHE, tune, int, "16384: each thread's cache of the host objects behind guest mutexes, events and queues has that many entries, four ways a set with a mixing hash, and a sync: object cache line is logged every 10 s", "<entries> (256 to 65536, a power of two): that many entries; =0: the fixed per-thread cache of 1024 direct-mapped entries, as before; =sets (experiment): 2048 sets of 8 ways of object numbers, an address in either of two sets, 128 KB a thread, to compare with the default.")L13 (minor): The switch row still declares int although sets is an accepted literal. Use string metadata for the mixed input and say that numeric entries are rounded up to a power of two.
@ -308,6 +394,7 @@ static Obj *obj_nockpt(uint64_t key, int kind) {static int off = -1, entries = -1;L3: The new mode dispatch reads the inherited plain shared off and entries variables while another thread can initialize them. Two threads entering their first object lookup together can race even if both compute the same values. Publish the resolved mode atomically or initialize it before any lookup thread starts. Apply the same treatment to locked_reads on the first concurrent table miss.
@ -304,0 +347,4 @@for (unsigned m = obj_set_ways(s, tag); m; m &= m - 1) {const unsigned i = (unsigned)__builtin_ctz(m);Obj *o = obj_of_id(s->id[i]);if (o && __atomic_load_n(&o->key, __ATOMIC_RELAXED) == key && __atomic_load_n(&o->kind, __ATOMIC_ACQUIRE) == kind)L1: The cache validates key and kind from separate object lifetimes. A cache entry for A can still point at a buried object after the table installs A's replacement. If a reader sees the old object's key A just before it is reassigned to B, then sees B's published kind, it can return B as A. The pointer cache has the same pattern. Validate a single published identity or use a versioned snapshot shared by the cache paths, and cover concurrent spare reuse in the test.
@ -304,0 +349,4 @@Obj *o = obj_of_id(s->id[i]);if (o && __atomic_load_n(&o->key, __ATOMIC_RELAXED) == key && __atomic_load_n(&o->kind, __ATOMIC_ACQUIRE) == kind)return o;*stale = i;L5 (minor): A matching tag with a different live key is treated as a stale entry. Two live addresses with the same low 32 bits and a shared candidate set can keep evicting each other even when there are empty ways. Prefer an empty way over an ambiguous tag match and only give stale priority to entries whose stored object can be shown to have changed. Add a forced tag-collision case.
@ -304,0 +357,4 @@static _Thread_local ObjSet *sets;static _Thread_local unsigned victim;if (!sets) {void *raw = calloc(((size_t)1 << OBJ_SET_BITS) * sizeof(ObjSet) + 63, 1); /* (the thread's, for its life) */L2: Thread exit loses the only reference to each sets allocation, and the code discards the original pointer immediately after alignment. Each short-lived guest worker that makes a lookup leaves 128 KiB plus 63 bytes behind. Give the allocation a thread-exit owner on Windows and Linux and retain the allocation base for cleanup. Host-side event producers can enter this path too.
@ -304,0 +358,4 @@static _Thread_local unsigned victim;if (!sets) {void *raw = calloc(((size_t)1 << OBJ_SET_BITS) * sizeof(ObjSet) + 63, 1); /* (the thread's, for its life) */if (!raw) return obj_lookup(key, kind);L6 (minor): After a cache allocation fails, every lookup on that thread retries the same 128 KiB allocation, including lookups under mw_lock. During sustained memory pressure, a hot mutex loop becomes an allocator call plus a table lookup per access. Keep a per-thread fallback state or retry outside the hot lookup path.
@ -304,0 +370,4 @@obj_cache_tl_looks++;Obj *o = obj_set_find(s1, tag, key, kind, &st1);if (!o) o = obj_set_find(s2, tag, key, kind, &st2);if (o) return o; /* (a hit: no shared write) */L8 (minor): Hits return before checking whether the thread's counters are due. A thread that warms a small cache and then hits indefinitely can account for none of those lookups in the reported miss rate. The sized cache shares this behavior. Schedule counter publication at the threshold on hits too, keeping the actual flush outside mw_lock, and test an all-hit stretch.
@ -356,3 +443,3 @@__atomic_store_n(&o->kind, 0, __ATOMIC_RELAXED);} else {o = (Obj *)calloc(1, sizeof *o);o = obj_new_locked();L4: The new allocator can return NULL if both the chunk and individual allocation fail. The caller immediately initializes locks through that pointer, repeating the old unchecked allocation failure. Check the result and use the host's allocation failure reporting without introducing a retry or virtual-core wait while the lookup locks are held.
L15 (minor): The same-build microbenchmark isolates cache shape but cannot establish the cost of chunk allocation for everyone. For one new object, the first chunk requests hundreds of KiB under table_lock instead of a single object. Before merge, compare the actual base with head unset and head with sets, after warm-up in interleaved runs. Report cold and warm load times, RAM, cache counts, average FPS, 1% lows, median and p99 frame times in the Market and a heavy scene. The required author play run and Tier 1 Windows, Linux and Steam Deck results are still missing.
@ -136,3 +138,3 @@const int entries = obj_cache_entries(size);if (!entries) require(last_pass > SLOTS / 2, "=0: the fixed cache misses a 4,000-object walk pass after pass");else if (entries >= 16384) require(last_pass <= SLOTS / 50, "sized: the walk's later passes miss under 2 %");else if (entries >= 16384 || entries == OBJ_CACHE_SETS) require(last_pass <= SLOTS / 50, "sized or sets: the walk's later passes miss under 2 %");L9: The sets case also passes if the new dispatch line is removed. The sentinel is nonzero, so the lookup falls into the sized cache with entries equal to 1. That variant always indexes the first four ways of its five-entry allocation, so the long walk fails today, but routing sets to the ordinary 16,384-entry default passes the same 80-miss bound and all later assertions. Give sets a workload or threshold that distinguishes it from the default and verify it rejects a dispatch that falls back to that default.
@ -139,3 +140,4 @@else if (entries >= 16384 || entries == OBJ_CACHE_SETS) require(last_pass <= SLOTS / 50, "sized or sets: the walk's later passes miss under 2 %");else require(last_pass > SLOTS / 2, "a cache smaller than the walk misses again: the size is honoured");/* objects changing under the cache: compared with the table itself */enum { KEYS = 3000 };L10: The fixture claims to exercise a buried object's memory being reused at a different address. Its clock stays fixed through the random operations, and the later 20-second advance is below the 30-second grave delay. It exercises same-address reinitialization but never takes a spare. Force burial, advance past the delay, assert the old ID has been reassigned, then look up the old address and compare it with the table.
@ -218,2 +218,4 @@runs = [("entries", None)] + ([("bench", v) for v in ("0", "4096", None, "65536", "sets")] if a.bench else[("check", v) for v in (None, "0", "256", "sets")])for mode, value in runs:env = dict(os.environ)L11 (minor): The child process inherits HOS_KEEP_DEAD_OBJECTS and HOS_SYNC_TABLE_LOCK, both of which are read by the extracted code. For example, a developer's HOS_KEEP_DEAD_OBJECTS=1 would disable the grave path that the reuse check needs. Set or clear both explicitly for each fixture case.
Recommendation: Request changes. The new sets cache can return a reused object for the wrong address and leaves its allocation behind on thread exit. Both blockers are covered by the existing exact-head review.
obj_set_findcan read the original key before reassignment and the new object's kind after reassignment. The acquire on kind comes after the key read. An ID identifies the storage, which stays allocated while the spare can change owners. A false hit can reachholder_readyundermw_lock, where it can consult or consume the other object's event. The pointer caches already have the same key/kind pattern. Fix how the lookup obtains a consistent identity and protects the selected lifetime through use. Cover concurrent spare reuse and a same-kind replacement with controlled interleavings. The single-thread fixture's random operations establish neither condition.Does it help? The Windows i7-9700 fixture reports 9.8 and 10.6 ns per lookup with
setsversus 14.2 and 13.6 ns with the default cache, best of seven, for a single-thread walk of 4,000 objects among 60,000 records. Both arms used this PR's object allocator. Chunk allocation runs even withsetsunset orHOS_NO_OBJ_CACHEset. Compare exact main, candidate unset, and candidatesetsfor cold creation, resident memory after churn, and warmed gameplay frame times before choosing a default. Author gameplay is still pending.Reviewed.
b8c630964074ecbfeac68f5f95533828f933fe6aagainst6e1521a70a32603ed60a8263e4072625885669b7. Reviewed all four changed files and the object table, reuse, cache, and multi-wait callers. The author reports passing focused Windows fixtures onf2c967b1and seven broad-suite failures reproduced on its then-base. The merge bases differ in none of the four changed files. Exact-head CI and cross-platform gameplay evidence are pending. No PR code run in this review.b8c63096409baebe0d44Pushed
9baebe0d(rebased ontomain@f1f2dc5b): every object cache (sets, andmain's sized and fixed caches) andfind_unlockedread an object's kind before its key; a thread's sets are freed as it ends (Windows FLS, Linux pthread key); objects are made in chunks only withsets; the switches are read once through atomics; newreuseandchurnmodes intests/test_obj_cache_size.py. Summary, Testing, Measurements (the figures predate this round; cold creation and memory after churn not measured yet) and Risks follow; the Test Farm line no longer holds a mention.@futondestroyer, on your review, head
9baebe0d(rebased ontomain@f1f2dc5b; the PR's own commit changed only wheremain's newHOS_OCB_HOLD_MSrow sits beside its switch row, a context conflict):9baebe0d. Every cache now checks an entry's object withobj_is: kind first (acquire), then key. A spare's reuse stores kind 0, the new key, then the new kind (release), so a lookup that reads the new kind reads the new key and one that reads the old life's kind (0 since it was dropped) fails.main's pointer caches (the sized default, the fixed one) andfind_unlockedhad the same key-then-kind order; the fix there is the same two loads swapped, so it is in this commit rather than a separate change. Lifetime: an object is never freed (the numbered chunks neither), and a dropped one is reused only after 30 s in the graves and never while a condition variable waiter or a multi-wait holds it (bury_locked), so what a lookup returns stays that address's until the guest drops it, as onmain. Test (a72f1502):tests/test_obj_cache_size.pymodereuse, run withsets, unset and0, controlled interleaving: the table gives the spare to a new address right after the lookup's first atomic read of the stale entry's object (the harness hooks each load), with the same kind and with another, and before the lookup as the control; it checks the spare (same number) went to the new address and the lookup returned the old address's own object. Ata72f1502, without the fix, the same-kind case fails in all three caches ("a lookup of 0x5000001000 returned the spare now 0x70000de130's"); with it all pass.9baebe0d: the base is kept and handed to an FLS slot's callback on Windows (ExitThread too) or a pthread key's destructor on Linux (the POSIX layer's ExitThread ispthread_exit), slot or key made once; a thread whose cache or owner cannot be had goes without and is not retried. Test: modechurnmakes and ends 64 threads one after another, each looking 4,000 objects up, with calloc/free of 64 KB and up counted: live bytes after equal live bytes before. Ata72f1502it fails (2,588,743 bytes live before, 10,981,383 after: 64 caches of 128 KB left); at the head both are 2,588,743. (The sized default's per-thread cache still stays behind onmainas its comment says; left for a separate change.)setsunset /HOS_NO_OBJ_CACHE: avoided rather than stated: objects are numbered in chunks only withHOS_OBJ_CACHE=sets; otherwise each is made alone, as onmain(9baebe0d).main, candidate unset, candidatesets: cold creation, resident memory after churn, warmed gameplay): not run. The fixture's--benchmeasures lookups only, and its old figures predate this round's changes (the description says so); the gameplay comparison is asked of the game runs on this PC, pending.Checks:
tests/test_obj_cache_size.pyandtests/test_multiwait_wake.pypass;run_all.py --strict --changed origin/main: 31 passed, 7 failed, 3 skipped, all 7 failing onmain@f1f2dc5btoo (Testing in the description).BeanBot's review, head
9baebe0d:find_unlocked; sets freed at thread exit; chunks only withsets). The play measurement L15 asks for is pending.HOS_NO_OBJ_CACHE/HOS_OBJ_CACHEread once into one atomic word (obj_cache_mode_read),HOS_SYNC_TABLE_LOCKlikewise.hos_fatal, no retry, no core wait.reuseburies, ages past 30 s, asserts the spare's number went to the new address and compares the old address's lookup.HOS_KEEP_DEAD_OBJECTSandHOS_SYNC_TABLE_LOCKtoo.--cc, thenCC, thenLONGSHOT_MINGW's clang, then cc/clang/gcc;RUN_ALL_ARGSno longer forces clang.main's sized cache), L8 (an all-hit stretch flushes no counts, asmain's sized cache).@Beanstalk This adds a switch, HOS_OBJ_CACHE, off by default: with HOS_OBJ_CACHE=sets, the per-thread cache of guest sync objects is 2048 sets of 8 ways (host/nn_os_sync.c). It has not run in the game yet. Please play Castle Town Market for two minutes, the PR against main on the same PC, on the PR's side with HOS_OBJ_CACHE=sets and with it unset, on a Linux PC with an AMD processor, a Windows PC with an AMD processor, a PC with 6 or fewer logical processors, and the Steam Deck. No crash; set, fewer misses in the log's sync object cache line than unset and frame time p99 equal to unset or better, within the run-to-run spread; unset, frame times as main's. Please report the object cache line's miss counts.
@siantm only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was for9baebe0, and the PR has changed since (479a4f9), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.The three renderer checks (
test_present_queue_default,test_target_reuse_memory,test_vk_occlusion_queries) did not compile in the earlier run (no Vulkan headers on that job's path); rerun with them in job fb3-vk-1, at the head9baebe0dand onmain@f1f2dc5b: all three fail at both with the same first error, so they fail onmaintoo; Tested's sentence and counts corrected.Recommendation: Request changes. The revised cache still needs coherent object identity and a Windows cleanup lifetime that matches its owner.
obj_isstill reads kind and key independently. Reading kind first fixes the particular reuse ordering in the new fixture, but it can still combine a kind observed before a reassignment with a key observed afterward. The new predicate also runs in the default pointer cache and the unlocked table path. Validate a coherent lifetime before returning a cache hit, and extend the regression coverage to check reuse across the full lookup.The parameter cache now publishes atomically, allocation failure falls back without repeated cache allocation, and chunk allocation is limited to sets mode. The author reports Windows fixture coverage at the new head. Earlier single-thread cache timing favored sets on that workload. Production memory and frame-time evidence and Linux validation remain open. Defer the existing Market request until the source issues are fixed. Author follow-up · Existing Farm request
Reviewed.
9baebe0d44c1bf9f46b67569fed9524240731fceagainstf1f2dc5bd2821cef061798ba40a043d281ac1707. Reviewed the interdiff, the four changed-file paths, and the object table, reclamation, thread and fiber owners. Both source-only Graphify maps verified and queried. Source only.@ -197,0 +206,4 @@object is never freed, and a dropped one is reused only after 30 s in the graves, not while a condition variablewaiter or a multi-wait holds it (bury_locked) */static inline int obj_is(Obj *o, uint64_t key, int kind) {return __atomic_load_n(&o->kind, __ATOMIC_ACQUIRE) == kind && __atomic_load_n(&o->key, __ATOMIC_RELAXED) == key;The kind-first check still samples two separate properties across object reuse. The cache hit needs one coherent identity for the object's current lifetime, including the default cache and unlocked table readers.
@ -304,0 +380,4 @@}static int obj_sets_own(void *raw) { /* 1: raw is freed as this thread ends */return InitOnceExecuteOnce(&obj_sets_once, obj_sets_slot_make, NULL, NULL) && obj_sets_slot != FLS_OUT_OF_INDEXES &&FlsSetValue(obj_sets_slot, raw);This registers a thread-local allocation with a fiber-local destructor. Deleting a guest fiber can free the allocation while the host thread still holds its cache pointer. Keep the pointer and its cleanup at the same lifetime and cover Windows guest fiber teardown. The thread-churn fixture does not exercise that owner.
9baebe0d44479a4f906aPushed
479a4f90(rebased ontomain@9dd41326, the earlier commits unchanged): a sets cache is now its host thread's and freed by the thread's exit callback (the image's TLS callback on Windows), not a fiber's FLS slot (03ea7d4a); every lookup withouttable_lockreads an object's key and kind as one identity through a per-object sequence (a seqlock,479a4f90);tests/test_obj_cache_size.pygains thefibermode and reassigns the object at every point of a lookup inreuse, now also on the unlocked table path (28f27eed). Summary, Testing (the new base), Measurements and Risks (the check's cost) follow.@futondestroyer, on your review, head
479a4f90(rebased ontomain@9dd41326;git range-diff: the three earlier commits unchanged, no conflict: #2012'sspin_beginchange is elsewhere in the file):03ea7d4a. Both are now the host thread's: the pointer stays a thread-local, and the FLS slot is gone; on Windows the image's TLS callback (DLL_THREAD_DETACH, run by the loader on the ending thread with its thread-locals still there, whatever fiber it runs,ExitThreadtoo) frees it from its base; on Linux the pthread key's destructor, as before. Keeping the pointer in FLS instead would give each guest fiber its own 128 KB and aFlsGetValuecall per lookup, and a fiber can resume on another thread (hos_thread.c, fibers), so the thread is the owner that matches. Test (28f27eed):tests/test_obj_cache_size.pymodefiber(Windows): a thread makes its cache on a guest fiber, switches back,DeleteFibers it and looks the same 4,000 objects up again; freed allocations of 64 KB and up are kept filled with 0xA5. At28f27eed(and at9baebe0ditself with the new test file) it fails: "2719878 bytes ... with it, 2588743 once the fiber is deleted (the thread running on); the thread's next 4000 lookups reached the table 4000 times; 30053 bytes written in freed memory". At the head: 2719878 after the deletion, 0 lookups to the table, 0 bytes written, and 2588743 after the thread ended (= before).churn(64 threads) still passes.479a4f90, a seqlock per object.obj_lookupsteps the object's newseqto odd (then a release fence) before it gives it another identity (a spare taken, a tombstone made again at its address) and back to even (release) after;obj_isreads seq (acquire), kind, key, an acquire fence, seq again, and takes the pair only if seq was even and unchanged. The sets cache, the sized default, the fixed cache andfind_unlockedall go throughobj_is. Test (28f27eed): modereusenow reassigns the object the lookup will read through the table's own paths before the lookup and right after each of its atomic loads in turn (every load the cut code makes outsidetable_lock), three ways (another address, same kind; another address, another kind; another address and then back to its own as another kind), withsets, unset,0andHOS_NO_OBJ_CACHE=1(the unlocked table path); the lookup must return the table's object of that address and kind, or the old one only if it read its identity wholly before the change. At28f27eedand at9baebe0dall four fail on the third way, e.g. unset "a lookup of 0x70009ce770 (a mutex) returned the object now its own address's again, of another kind ... reassigned right after the lookup's load 2"; at the head every position passes (6 loads with sets, 5 unset and0, 9 to 10 unlocked).sets) are asked again at this head of the person who does them, pending. The deferred Market request to the Test Farm is handed back with this head now that the two source points are fixed; it is not resent here.Checks (Testing in the description):
tests/test_obj_cache_size.py(three ways of choosing the compiler) andtests/test_multiwait_wake.pypass;host/nn_os_sync.ccompiles with the host build's flags, with and without Tracy, and its object has the callback in.CRT$XLF(not COMDAT);run_all.py --strict --changed origin/main: 32 passed, 7 failed, 3 skipped, all 7 failing onmain@9dd41326too with the same first error (named in Testing).@Beanstalk This adds a switch, HOS_OBJ_CACHE, off by default: with HOS_OBJ_CACHE=sets, the per-thread cache of guest sync objects is 2048 sets of 8 ways (host/nn_os_sync.c). It has not run in the game yet. This head is on main
9dd41326: please compare with main at that merge base. Please start from the market-entrance save and play Castle Town Market for two minutes, the PR against main on the same PC, on the PR's side with HOS_OBJ_CACHE=sets and with it unset, on a Linux PC with an AMD processor, a Windows PC with an AMD processor, a PC with 6 or fewer logical processors, and the Steam Deck. No crash or hang; set, fewer misses in the log's "sync: object cache" line than unset and frame time p99 equal to unset or better, within the run-to-run spread; unset, frame times as main's. Please report the object cache line's miss counts.@siantm only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was for479a4f9, and the PR has changed since (712ee7e), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.Pushed
712ee7e7(onmain@9dd41326, the earlier commits unchanged):obj_isnow reads an object's key and kind as one 64-bit word in one load, in place of round two's sequence check (sequence, kind, key, sequence), which the micro-benchmark put at about 3 ns a lookup in every cache mode (712ee7e7); the object cache test makes its tombstones throughdropand builds--benchwithout its per-load watch (b7554f7b). Description: Summary (the identity word), Testing (jobs fb3-1969-r4l and fb3-1969-r4b), Measurements (the sequence check's figures; this head againstmainheld for the run-alone queue) and Risks (the check's cost, 8 bytes an object).@futondestroyer, on your review (65541, one coherent identity), head
712ee7e7: round two's sequence check (sequence, kind, key, sequence;479a4f90) cost about 3 ns a lookup in every cache mode, the default included, in the micro-benchmark (i7-9700, Windows 11, ns a lookup, best of 7, two runs each: unset 16.7/16.7 → 19.6/19.3,HOS_OBJ_CACHE=6553613.7/13.6 → 16.7/17.0,sets15.8/16.0 → 18.5/18.5,=028.0/28.3 → 31.7/31.1 against 9baebe0d's kind-then-key check), so712ee7e7replaces it with one 64-bit identity word per object (the kind in the top 4 bits over the key), written after every change of kind undertable_lockand read byobj_isin one load (acquire); an object only takes another identity as a tombstone, whose word is 0, and gets its new word last, so a kind never pairs with another identity's key.tests/test_obj_cache_size.pyreuse(sets, unset,0,HOS_NO_OBJ_CACHE=1, a reassignment before the lookup and after each of its loads) andfiberstill fail on 9baebe0d's source and pass at the head (b7554f7bonly makes the harness's tombstones throughdropand builds--benchwithout its per-load watch, which had charged a call to every load). The cost: 8 bytes an object (344 → 352 bytes on Windows); this head's timing againstmain@9dd41326and479a4f90is held for the PC's run-alone queue, pending (Measurements says so).@Beanstalk This adds a switch, HOS_OBJ_CACHE, off by default: with HOS_OBJ_CACHE=sets, the per-thread cache of guest sync objects is 2048 sets of 8 ways (host/nn_os_sync.c); in every cache mode, a lookup now checks the object's identity with one word in one load. It has not run in the game yet. Please start from the market-entrance save and play Castle Town Market for two minutes, the PR against main on the same PC, on the PR's side with HOS_OBJ_CACHE=sets and with it unset, on a Linux PC with an AMD processor, a Windows PC with an AMD processor, a PC with 6 or fewer logical processors, and the Steam Deck. No crash or hang; set, fewer misses in the log's "sync: object cache" line than unset and frame time p99 equal to unset or better, within the run-to-run spread; unset, frame times as main's. Please report the object cache line's miss counts.
@siantm only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.