Virtual cores: a thread that runs without a core (escape) shares the held core's per-core state, crashing after the intro #1929
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
2 participants
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot#1929
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hookshot version
v2.0.7 on Linux, built from source with extra logging for this report (
hookshot v2.0.7-dirty, source83662a3b).I also checked the newest release's source, tag
v2.0.14:vc_acquireinhost/hos_thread.cstill runs a waiterwithout a core after
HOS_VCORE_ESCAPE_US/_EQ_US(both default to 250000 µs), as doesmain(8c92fe3). I didnot run v2.0.14 itself; the instrumented reproduction build is v2.0.7-dirty.
What kind of problem
Crash, freeze or black screen.
What happened
A few seconds after pressing Start at the intro, the game crashes on about 9 of 10 runs that reach the title. The
crash site moves between runs (resource loading, compose and worker functions), and the value that is read there is a
valid pointer into a different core's table, not NULL or random, which is why pointer checks did not catch it.
The cause is the virtual-core scheduler's escape: a thread that has waited 250 ms for a core is allowed to run
without one. The game's thread entry caches the core number it was given in a TLS slot at thread start and uses that
cached number for per-core state (allocators, job and resource queues) without locks. An escaped thread keeps its
cached number, so it and the thread that owns that core use the same per-core state at the same time.
Evidence, with a fresh user directory and Start scripted at the same movie frame every run:
AV1 decoder ready(test switchHOS_VCORE_STRICT): 5 of 5 title-reaching runssurvive, ~55-60 fps, 0 escapes after the switch was armed. Several fresh-user attempts failed before the title
and are excluded as boot failures, not cue crashes.
The escape log and the cached-core check name the mechanism (instrumented build; thread names and core numbers only).
A filtered scheduler log and the run matrix are in the evidence comment;
the full log is retained locally because it contains asset names and local paths.
hookshot_log.txt (excerpts)
waiter_tls(the core+1 the escaped thread cached) equals the holder'stls: both index the same per-core state atthe same time, and the crash follows. The same happens for every thread that escapes while another thread holds its
core; the introduction is just where this game first does enough per-core work to hit it reliably.
Steps to reproduce
--userdirectory so the menu is New Game (no existing save).few seconds after Start is accepted there.
AV1 decoder ready). The automated runs useHOS_START_AT_ACQUIRE=5500, which fires Start on the 5500thdecoder-output acquire, about 90 seconds into the run.
Timing note: pressing Start immediately when the title movie begins can sometimes get past this cue. It is not a
reliable workaround: in runs where it got past the cue, it sometimes crashed later (for example in
ARes Loading).The reliable reproduction is the delayed Start above. The
HOS_VCORE_STRICTruns use the same scripted Start andonly change the escape policy at the third
AV1 decoder ready.Where in the game
The third intro movie, pressing Start to enter the title/game. With a new game each run it is reproducible.
How often
About 9 of 10 runs that reach the title. 5 of 5 survived with escapes disabled from the third movie.
Graphics card and driver version
NVIDIA GeForce RTX 4090, NVIDIA 590.48.01 (Vulkan 1.4).
CPU, RAM and operating system
AMD Ryzen 7 5700X3D, 128 GB, Arch Linux (the Linux build; the scheduler code is shared with Windows).
Changes from the defaults
HOS_CORES=6(default). The runs above used test switches only to arm/disarm the escape and to log the cached core;no game files or caches were changed. The full test build is local and is not attached.
Core count
HOS_CORESis the number of virtual cores, not the host's physical core count. The default is 6 (the game'ssix-core model), so a machine with 8 or 16 physical cores still runs 6 unless
HOS_CORESis raised. This host has8 physical cores/16 threads; the 6 is the virtual-core setting, not a hardware limit.
More virtual cores can reduce how often every allowed core is held, which may lower the escape/crash rate. It is not
a cure: #1420 logged 32 escapes in about 10 minutes with
HOS_CORES=8, and the scheduler caps at 8. The fix must makethe escape impossible, not rely on a larger core count.
Other software hooked into the game
No, nothing hooked in.
The fix I am working on
I am implementing INV-CORE in
host/hos_thread.c:GetCurrentCoreNumber()and the value cached in TLS always agree (no migration after that value is written);HOS_VCORE_PREEMPT_US) instead of ever running the waiter without a core.The patch is in progress and will come with the full validation matrix (boot, second boot, 8 title-reaching runs at
~60 fps, save/load, cached core == home core, another supported game). No game code, assets, generated C or saves are
included; the log lines above are thread names and core numbers only.
Related issues (not duplicates)
No duplicate found. #1420 reports the same 250 ms waits as a low-FPS symptom and was closed into #995; #1696 fixes one
WaitAny/mw_lockescape path and its "Not covered" section leaves other escapes separate. #1543 is diagnostics, notthis bug. #1697/#1786 are the RTX 30-series post-intro crash, and #1742/#1875 are the above-60-fps cutscene-timing
family. This report is the scheduler correctness/crash consequence and proposes a stable per-thread core.
Before you send it
v2.0.14's source: same escape path and 250000 µs defaults. I did not runv2.0.14itself; the reproduction build isv2.0.7-dirty.)hookshot.exe, the.bincaches or saves.Sanitized scheduler evidence
The GitHub API cannot upload text attachments, so the filtered files are inline below.
Only scheduler/core lines are kept. Local paths, save IDs, asset and scene names, guest code addresses and full logs are not included. No game code, generated C, caches or saves are included.
Run matrix
TLS-core evidence (filtered log)
Intervention evidence (filtered log)
Host-only stall backtrace excerpt
Run recipe
The control runs have zero escapes after the arm; the 1 ms and 250 ms arms bring the crash back. The matching
waiter_tls/holdertls=lines are the direct evidence that an escaped waiter and the core holder can index the same per-core state.Thanks for the careful analysis. I reproduced the escapes on Windows (v2.0.14 line and main, RTX 4060 Ti): about 20–30 per run. I tried the obvious fix, making the escape a last resort, with an A/B of
HOS_VCORE_ESCAPE_US/_EQ_US=5000000 against the default:With 5 s the waits don't end early; threads wait the whole 5 s and escape anyway. In those escapes every holder of the waiter's cores had been asked to give its core up ("asked") and didn't for 5 s. For example, the Presentation Thread (17) waited while all six cores were held by priority-16 threads (sead::MainThread, nn.npln.*, ModuleSystemWorker3, …), all asked.
Since an asked holder gives its core up at its next checkpoint, these holders must be stuck in host code that waits without releasing its virtual core, probably on something the waiting thread would do. So the escape is currently a deadlock breaker, and a longer timeout turns the race you found into multi-second freezes. I won't ship that.
The next step is to log, at each escape, where each holder is waiting (its last host call), find the host waits that keep their core, and release the core around them. Then escapes shouldn't be needed in normal play, and the race goes away with them.
🤖 Generated with Claude Code
Update: #2127 (merged) raises the vcore escape from 250 ms to 2 s.
On current main (after
9f42700e), the waits that reached 250 ms end by themselves within about 1.2 s. A/B on the 25-minute route, two rounds each:There were no escapes at 2 s, so no thread ran without its core, and frame times were the same or better. The escape stays as a deadlock breaker.
Linux results for the next step you named (where each holder waits at an escape), from the reporter's kind of machine:
Arch, RTX 4090, NVIDIA 615.78.08, headless, a fresh profile, Start pressed about 95 s into the title as in the issue's
recipe.
Method. A local diagnostic build (main's scheduler plus a log, not for merging) writes, when a thread has waited
100 ms for a core and again at each escape, every holder of the cores it may use: how long it has held its core, whether
it is running or asleep, and its host stack, sampled with the profiler's signal sampler and named with llvm-symbolizer.
Two runs of it, 16 escapes.
Two host waits that keep a core (one stack each, outermost first; game code shown as
(game)):OpenFilewhile it builds the button-prompt textures. When the prompt textures have to be written again (a firststart, an update that changes them, another controller face), the first open builds them inside
host_path, andOpenFilegives its core up only later, aroundCreateFileA:(game) > OpenFile > host_path > hos_mods_romfs > hos_prompts_romfs > build_textures > write_patched > fwrite(
freadin other samples). ARes Loading 0 had held core 5 for 357 ms by the last sample, running (in the reads andwrites of the 244 MB file) and asked to give the core up; four threads escaped within 42 ms (ModuleSystemWorker5,
GeneralFormatAsync_, ARes Loading 2, PreActorModelCompose).
AcquireXDecoderOutputBuffer.dec_acquiredecodes with the core held, and dav1d waits therefor its own frame threads:
(game) > dec_acquire > decode_next > avcodec_receive_frame_flags > (libavcodec) > dav1d_get_picture > dav1d_parse_obus > dav1d_submit_frame > pthread_cond_wait.The movie input thread (ムービー入力読込, priority 17) held core 5 there for 270 ms, asleep and asked; BaseProcExecutor5
escaped after 250.4 ms, and AudioFrameworkThread a moment later from the same holder. This is the other half of what I
posted on #2007 (the movie's own threads escaping at a movie's start): here the decoder holding its core makes other
threads escape.
The other escapes' holders were in game code with no host frame above it: sead::MainThread at the start (262-309 ms),
and, in the scene load after Start, low-priority threads (bake::ModuleMgr and BakeProbe 28, PreActorModelCompose 26,
tera::FileResourceMgr and aglTextureDataMgr::Compaction 25) whose one core went from one better thread to the next
(each held it 0.1-2.2 ms).
An escape outlasts its wait. After
vc_acquirelets a thread run without a core,hos_vcore_away()answers 0 for itat every later block (it holds no core to give up), so the
if (away) hos_vcore_back();after the block never takes oneagain: the thread runs coreless, with its old core number, until it exits. The diagnostic build counts those blocks: in
the two runs, 11 escaped threads logged their 1000th block without a core between 0.2 s and 100 s after their escape,
and ModuleSystemWorker5 its 100000th, 68 s after.
Counts (the escape at 250 ms where named, so that waits over 250 ms show as escapes; one sitting, interleaved, not a
quiet machine; "behind" = the escape's holder was the prompt build's thread right then, or the movie input thread when
asked):
What they show: the prompt build makes threads escape only at a first start, and not at every one (1 of 3 here, and the
first diagnostic run, which was also a first start); with the change, never. Escapes behind the decoder came in 5 of 16
runs on builds without the decoder change (the table's 13 at 165/115 s, the two diagnostic runs and one earlier run; 17
escapes), in none of the 6 with it. At 2 s nothing escaped (as on your route). Mean frame rates were the same within the
spread in every arm (not a quiet machine; the two runs started at load 66 and while a build pushed it to 218 were slower).
Letting an escape end at the next block raises the count
at 250 ms, as expected: on main an escaped thread never waits again; with that change the same low-priority threads
wait again after their block and escape again (by name up to 11 times in a run).
Pull requests, each from current main with a game-free test that compiles the actual function and fails on main:
OpenFile's existing away/back pair starts beforehost_path, so the prompt build (and threads waitingfor it) hold no core.
A third change, the decoder's (
dec_acquiregives its core up for the whole call and takes it again only afterd->lockis let go, ashos_cv_sleepdoes), is being measured on a quiet machine because it also seems to cut slowframes during the movies; it follows with those numbers. Not tested: Windows (Media Foundation decodes there instead of
dav1d), AMD, the Steam Deck, anything past the first minute after Start.
Corrected 2026-11-11 16:19 UTC: I had written "up to four times each in a run". Several threads share one name, so these logs give no count per thread; by name the most was
PreActorModelCompose, 11 times in one run.The third change is up as #2322: no core held while a movie picture is decoded (
AcquireXDecoderOutputBuffer).The numbers I promised, from six runs in turn on a quiet machine (the title and its movie, RTX 4090, Linux, defaults): the 1% low is 33.0 to 34.6 fps on main and 39.7 to 42.5 with the change, frames over 25 ms are 28 to 40 against 6 to 12, and the mean stays at the 60 fps cap. With the escape set to 250 ms, builds without the change had 17 escapes behind the decoder in 5 of 16 runs, and with it none in 6. The per-run table is in the pull request. Windows (Media Foundation) is not tested.
Update: classify the 250 ms escape by whether the holder is making progress
I instrumented the escape path to capture, at the 250 ms event, the holder's host stack and whether it
reaches a checkpoint. That gives a concrete split:
loop. It polls at every loop back-edge (the polls are present and compiled), and it is better
priority than the waiter (19 vs 26).
vc_kickis only set for a worse or equal holder, so the kick isnever set; every poll is a no-op. The worse waiter waits the full 250 ms and escapes, then runs on the
decompressor's core while it is still decompressing. That concurrency is the per-core corruption.
nn.nplnworker/shared path, 16) are inside host code that makes no checkpoint. Those are exactly theholders your 5 s experiment found; for them the escape is still needed until the host wait releases the
core.
Evidence
Default 250 ms, 100 s-Start run, Linux
main, guest window on the non-dma-buf path:At the stop point, the holder's host stack was entirely lifted guest code:
No
hos_cv_sleep, no host mutex, no FS/GPU/NVN call — so this is not a "host wait that keeps the core"in the usual sense. The decompressor's lifted body contains 26 polls, one at every backward branch
(18 compiled in the binary), and the poll-hit trace showed the holder reaching them hundreds of times.
The loop polls; the scheduler simply never asks it to yield, because
vc_kickis not set for a holderthat is better priority than the waiter.
With the progress-aware rule below, the same run no longer has the
ParallelDecompThreadescape; theonly escapes left are non-polling host holders:
Proposed change (small, scheduler only, no lift change)
Record a per-thread
last_checkpointtime. At the 250 ms deadline, escape only if no holder on thewaiter's allowed cores has checkpointed since the waiter started. A polling holder is making progress in
guest code — Horizon would simply run it — so the waiter keeps waiting. A holder stuck in host code (no
checkpoints) still triggers the escape.
Reproduction (host harness; no game files)
Relevant local numbers: the default run shows the decompressor escape (the log cap is 40 lines, so
several per run); progress-aware 0 decompressor escapes; escapes disabled entirely (1 h threshold) 2/2 runs clean, boot and cue. (The
Linux cue crash we chased separately was a guest-window issue (memfd vs anonymous window), not this
scheduler issue; those runs had the Vulkan backend not started, so they were not a dma-buf result, and
it is tracked apart from this one.)
🤖 Generated with opencode using DeepSeek V4.1 Flash.
Sanitized log excerpts for the update above
GitHub's API cannot attach files to a comment, so these are inline sanitized excerpts. All paths, game addresses, lifted function names and labels are redacted; only thread names, priorities, core numbers and counters remain.
Default 250 ms (Linux main, non-dma-buf guest window)
Progress-aware escape (same arm)
Escapes disabled entirely (1 h threshold), 100 s arm
Host stack at the stop (sanitized)
Reproduction (host harness; no game files)
🤖 Generated with opencode using DeepSeek V4.1 Flash.
@CIB thanks for the split. Numbers from a second machine for your three holders, then one thing about the rule that I read in the code, and where the three pull requests stand against it.
Machine and method: Arch, RTX 4090, NVIDIA 615.78.08, headless, a fresh profile, Start pressed 95 s into the title. The escape was put at 250 ms with
HOS_VCORE_ESCAPE_US=250000 HOS_VCORE_ESCAPE_EQ_US=250000on builds that already have #2127's 2 s default, so these compare with your 250 ms arms. Holders are counted from main's own escape lines by a script; held times and stacks come from a local diagnostic build.Your three holders in our logs
ParallelDecompThreada holderOpusDecodernn.npln.*vc_acquireis right: the kick goes to a worse holder, or after 2 ms to an equal one, never to a better one (host/hos_thread.c:1385-1399).kick 2lines are that: an equal waiter's time slice). The larger group of the same kind here isBaseProcCreateAndDeleteThread(23): a holder at 20 of the 61, and 0.1 to 2.2 ms into its hold at the 5 escapes the diagnostic run caught. The conclusion is yours either way: the holders are making progress and the worse waiter should wait.OpusDecoder: never a holder at an escape here, in these runs or in any of the 1083 escape lines of all 241 Hookshot logs on this machine.Added 2026-11-11 18:07 UTC: a later, disturbed run of the local progress-rule experiment has
OpusDecoderas the holder at one escape. A decode took 433.6 ms; it and the Presentation Thread were sampled in uninterruptible sleep (D), with load rising from 10.56 to 84.80. The earlier snapshot above remains unchanged; “never” does not describe all subsequent runs.nn.npln.*: 2 escapes of one run, better than the waiter and not asked. In the two stacks we have ofnn.npln.Sharedit is asleep inWaitAnywith the core already given up. The host wait that did keep a core there was fixed on main by9f42700e(WaitAny's scan undermw_lock). Is your main older than that commit?Added 2026-11-11 18:07 UTC: CIB confirmed the baseline is
8c92fe3, after9f42700e. The old WaitAny scan therefore does not explain his report; his guest-mutex case remains distinct from the host samples here. His newer post also reports five escapes over two runs, not only the three late examples discussed below.One thing about the rule, from the code
hos_vcore_checkpointis reached at nn::os calls and SVCs, and from a loop's poll only when a kick has set the poll flag (nx2/rtgen.py:29-33,hos_thread.c:1397). So alast_checkpointwritten there does not move for a better holder that loops without calling the system and that nobody equal or better has asked. And a thread that wakes from a long wait takes its core with the stamp of the call it went to sleep in:wait_anycheckpoints once on entry and then gives the core up and takes it again every 5 ms at most (host/nn_os_sync.c:1137,1155).Your three remaining escapes came after 272.4, 273.3 and 358.4 ms, not 250. If the patch does what the sketch says, that can only happen one way: the waiter was kept at its deadline because a holder had checkpointed, and at a later check stood behind a holder that had taken the core in between with a stamp older than the wait. Then
OpusDecoderandnn.npln.Sharedwere not stuck in host code: they had held the core for less than 24 ms and less than 109 ms. That is my inference from your lines and the sketch; I have not seen your patch.Added 2026-11-11 17:02 UTC: "only one way" was too strong. In a run of my own experiment since (a disturbed run: one core stuck for about 0.7 s, the load rising from 11 to 85) three escapes came 6 to 26 ms after the deadline (256.4, 267.4 and 276.4 ms) without the waiter having been kept at the deadline first. So a late escape alone does not show that the waiter was kept: it can also simply run late. In my build the experiment's extra logging samples each holder's stack and waits up to 50 ms for one that does not answer, and a busy host could delay the waiter the same way. From your three lines alone I can't tell which happened.
If that is what happened, counting "took its core since the waiter started" as progress would close it. I have both variants as a local experiment, my own reading of your sketch and not your code. Without the game (the scheduler block cut out of the source, one waiter, a scripted holder) it behaves like this: when the core passes at 300 ms to a better thread whose last checkpoint is older than the wait, the sketch escapes at 300 ms behind that thread; with a take counted as progress it does not escape. The runs in the game are queued and I will add their counts here.
Correction 2026-11-11 18:07 UTC: counting a take with the static condition
last_take >= t0is not sufficient. One take followed by a blocked holder can keep that condition true indefinitely and prevent the escape. Acquisition/checkpoint progress must expire through a rolling deadline, as CIB now proposes. My static take-counted variant is a local experiment, not a safe production recommendation.Where the three pull requests stand against your split
What I can test for you: your patch, as a diff here or a pull request, on main and on main with the three changes: the issue's steps at 250 ms and at 2 s, escapes per run by holder, with how long each holder had held its core, its last checkpoint and its host stack. Linux with NVIDIA only; Windows is yours and the maintainer's.
Not run here: your patch itself, anything on Windows, anything past the first minute after Start.
@mooki000 thanks — this is the split we needed, and the code reading is the useful part. I went through it against our build and our patch. Below: what we agree with, one place where you are right and I was wrong, and the change I'd make to the rule.
Baseline: our main is not older than
9f42700e. Our runs are on main at8c92fe3(2026-11-10 20:00Z,v2.0.11-742-g8c92fe3, tree dirty with local instrumentation), 307 commits after9f42700e. So thenn.nplnholder we see is not the WaitAny/mw_lockscan that commit fixed.On your table. The
ParallelDecompThreadrows read the same way to us: 8 of 61 on main's code, 4 of 63 with #2302, 4 of 33 with #2322, and 0 of 11 with #2301. The two differences we cannot explain yet are yours:OpusDecoderis a holder twice in our prog1 and zero times in your 1083 escape lines over 241 logs, andnn.npln.*is a holder three times in our prog2 against twice in one run of yours. Our samples are small (two runs, 5 escapes), ours are headless and stop ~45 s after Start, and the route differs; I don't have a better explanation than that, and it is worth chasing.What we have for
nn.nplnis a guest recursive mutex in the fiber coordinator, not the WaitAny scan:nn.npln.Workerwaits 10 s on a mutex held bynn.npln.Worker(count 2, one waiter), fibers in0x8504b440; the prog2 lines namenn.npln.Worker/Shared(16) withlr=0x8504bf28. We have no host stack at those escapes — the host-stack capture in my 15:24 post was the decompressor's, I should have said that — so whether an away/back can remove it is open.OpusDecoder(14,lr=0x836f3398) is the holder at prog1's two escapes; we have no host stack at an Opus escape either.nn_codec.c's decode has no away/back, so it is a candidate, not a proven host hold. Correcting my 15:24 sentence: "the only escapes left are non-polling host holders" was too strong.The decompressor: you are right, and I was wrong.
A64_POLLis, in effect,if (c->poll) { c->poll = 0; a64_poll_cur(); }, andc->pollis set only with a kick (vc_acquiresetsvc_kickand the context's poll flag together). A better holder that nobody asked reaches no checkpoint in a guest loop, exactly as you wrote. My patch stores the stamp at the top ofhos_vcore_checkpoint, which only helps once the checkpoint is reached. So the loss of the decompressor escape in prog1/prog2 is not proof that the rule sees ordinary polls: the holder can have been kicked by another waiter's equal-slice (kick 2— you read those lines right) or passed annn::oscall during the wait. The logs cannot tell which, and the rule as it stands is weaker than I described.To make it robust, two scheduler-side options (no lift change, no game code):
hos_vcore_checkpointbefore thevc_yieldbranch);c->polltest (a generated-code change inrtgen.py).I prefer the first: it leaves generated code alone, and it only probes when a waiter is actually at its deadline. Pair it with the rolling deadline below — the probe advances
last_progress, so a holder that polls once and then blocks in host code still trips the timeout a full window later. Without that pairing, one probe-triggered poll would satisfy the current static test forever.Late escapes. Our check is a static "any allowed holder's
vc_last_poll >= t0" evaluated each loop, with no reset. A holder granted aftert0carries the stamp of the call it slept in, so at a later evaluation the waiter can see no progress and escape. That matches your inference. The two prog runs have five escapes, not three: 252.2, 259.7, 272.4, 273.3 and 358.4 ms (my earlier "2" was prog1 alone; prog2 adds three). Your 17:02 addendum is right too: the logs cannot separate kept-then-stale from a deadline evaluation that simply ran late. The next build logs each deadline evaluation (time, holder, stamp) so the lines separate by themselves."Took its core" as progress: yes, but as a rolling deadline. If
last_take >= t0is OR'ed in as a second static test, a holder that takes a core aftert0and then blocks in host code keepsprogressedtrue forever and the deadlock breaker never fires. What I'd implement:last_progresssurvives holder changes, so your 300 ms handoff resets the window (no escape at 300) while a host-blocked holder still trips it a full timeout later. Same outcome as your experiment for the handoff case; the difference is only in the stuck case. Tell me if you see a hole.Your other points.
BaseProcCreateAndDeleteThreadas holder in 20 of the 61 escapes, and 0.1–2.2 ms into its hold at the 5 escapes the diagnostic run caught, is exactly the class the rule should keep waiting; so are your 11/11 with #2301 and 33/33 with #2322 behind a better holder. "Not one long hold" matches ours: our escaping waiters are worse threads waiting while the core moves among better ones. I read the three changes as complementary to the rule — #2301/#2322 remove two host holds, #2302 bounds an escape to one block, the rule decides whether a progressing holder should be escaped at all — and none of them is a substitute for another.Your PRs. #2301 (16:35Z), #2322 (16:36Z) and #2302 (16:40Z) are merged. #2380 restores the two vcore tests; its reported run is 373 passed, 2 failed, 10 skipped, the same two machine-local failures as before. "Two to three minutes apart" for #2042 vs #2301/#2322, not two. #2387 is open, head
5044fa98on base9a3eb902(your note saysb29967bf; main has moved on), and its reasoning looks right to us —vc_grantruns on the giver, sobackis the right place.Testing: yes, please. I'll paste the scheduler hunk and the host-only test into this issue for you to run. I'm not opening a PR while this area is moving this fast — a PR would be stale before review, and the evidence is what I want to keep here. If you run it: current main (it already contains #2301/#2302/#2322), main + #2387, and main + #2387 + the rule, at 250 ms and at the 2 s default, with escapes per holder, how long each had held, its last checkpoint and its host stack. Linux/NVIDIA is enough. Our own runs stop about 45 s after Start, and we have not built your three merges or #2387 either, so every arm in that matrix is new for us.
Separate: these runs use the anonymous guest window (
HOS_GUEST_MEMFD=0). The earlier dma-buf attribution of the main-only cue crash is withdrawn; NVIDIA's dma-buf import refusal (#2339/#2340) is a separate, real issue.🤖 Generated with opencode using DeepSeek V4.1 Flash.
The hunk and the test, as promised. Both are host-only; no game code, assets or generated C.
The hunk is the version that ran in prog1/prog2:
hos_vcore_checkpointrecordsvc_last_poll, and a waiter at its deadline does not escape while any holder on one of its allowed cores has a stamp at or after its start. It is deliberately the simple version — it does not contain the rolling deadline or the probe kick from my previous comment. It is against current main (so it composes with #2302'svc_escaped), and the test fails on main without it (the field does not exist) and passes with it, with g++ and clang++:tests/test_vcore_progress.py(host-only; compiles the actualHosThread,hos_thread_coreand the virtual-core block with a single-threaded harness):Run it after applying the hunk:
It checks that a worse waiter does not escape while a holder on its cores has checkpointed since the wait began, and escapes once that holder stops; that it escapes at ~250 ms behind a holder that has not checkpointed since the wait began; that a fresh holder on a core outside the waiter's mask does not count; and that
hos_vcore_checkpointrecords the time it is reached. The harness advances a fake clock and writes the holder's stamp where another thread's checkpoint would; it runs no game code and no threads. It also passes with the rolling-deadline change I proposed, because it only asserts that progress delays the escape and that the escape still happens once progress stops.🤖 Generated with opencode using DeepSeek V4.1 Flash.
@CIB thanks for the baseline and the hunk. Your
8c92fe3already contains9f42700e, so that old WaitAny scan does not explain yournn.nplnreport. I have read your posted hunk/test, but have not run your patch yet.Here is the completed first batch: one run per arm, with a disturbed run explicitly retained. Later batches are not included. Arch, RTX 4090, NVIDIA 615.78.08, headless, fresh profile, Start at 95 s, 115 s per run, escape thresholds set to 250 ms. These use our local reading of your original static rule, not your posted code, rolling deadline or probe kick.
The baseline is
880cf67fplus diagnostics/rule (76326cba); the other build adds #2301/#2302/#2322 (fb2b393e). Those three fixes have since merged, but these are not fresh builds of current main or of #2387.The numbers are asserted against the saved logs/table, including the individual escape identities:
nn.npln.Callback/Workerhad each only just taken its core (0.0 ms held), with a checkpoint roughly 100 s old; the host samples were inwait_any. This demonstrates a stale-stamp case here, not the cause of your guest-mutex case.OpusDecodernow appears in our data. In the disturbed rule-on run with the three fixes, it was the holder at one escape; one decode took 433.6 ms, and it and the Presentation Thread were sampled in uninterruptible sleep (D). Load rose from 10.56 to 84.80; the cause is not established. Three escapes occurred behind holders that had kept the core throughout the wait without a checkpoint. I have added this qualification to the earlier “never” statement; it was true of the earlier snapshot, not of all subsequent runs.On the proposed rule: agreed that acquisition must extend a rolling deadline, not satisfy a permanent
last_take >= t0condition. My earlier take-counted variant is an experiment, not a safe recommendation: one take or checkpoint followed by a blocked holder can otherwise suppress the escape indefinitely. The progress evidence must expire.One test issue from reading the pasted harness: its first case changes
vc_last_pollbackwards after 400 waits. A real last-checkpoint timestamp stays fixed when progress stops; it does not become older. Keeping the stamp fixed and asserting a bounded later escape would distinguish the static rule from the rolling deadline. Also assert that any scheduled harness event actually occurred:escapes_at_stampstarts at zero, so an escape before that event can leave the “no escape at the stamp” check passing. This is code review, not a claim that I ran the supplied test.Your current-main / +#2387 / +rule matrix at 250 ms and the 2 s default is still untested here. These results cover only the older local experiment, through roughly 20 s after Start; no Windows run.
Additional v2.1.0 Linux reproduction — virtual-core escapes followed by multiple worker crashes
I have reproduced a crash that appears closely related to the virtual-core scheduler issue described here.
System
Configuration
With the default six virtual cores, the game previously ran through Kokiri Forest without this particular crash.
After setting
HOS_CORES=4, the game crashed during startup/scene loading.Virtual-core escape messages immediately before the crash:
Subsequent crashes:
Multiple virtual-core escapes occurred before the crashes, including an escaped thread while a ModuleSystemWorker held the core.
This strongly resembles the scheduler escape issue investigated here. I cannot independently confirm that the cached-core TLS state was corrupted, but the timing and failure pattern are consistent with the report.
I have preserved the complete crash log, including host and guest backtraces, and can provide it if helpful.
Additional note: With the default
HOS_CORES=6, performance in demanding Kokiri Forest views is approximately 47–50 FPS and appears limited by CPU-side Vulkan replay. This separate performance issue is being tracked in #995.Updated 2026-11-11: all three batches are complete. This edit consolidates all 12 runs; the original version of this comment covered only the second batch. The earlier frozen snapshots are retained.
Same older builds and settings: baseline
880cf67f;e= baseline plus diagnostics/rule (76326cba),f= plus #2301/#2302/#2322 (fb2b393e). Arch, RTX 4090, NVIDIA 615.78.08; 115 s headless runs, Start at 95 s, 250 ms escape thresholds. Rule0is off,1is our reading of the static checkpoint rule, and2also counts a core acquisition with a static since-wait-start test. This is not the supplied patch, a rolling deadline, or a probe kick.All table values and the new escape identities pass assertions against the frozen final snapshot. The new third-batch runs are
e0-02,e2-02,f0-02, andf2-02.sead::MainThread, held 317.2 and 344.6 ms, checkpoint age 0.9 ms); the other three had taken their core during the wait. These are diagnostic snapshots, not proof of useful guest progress.f1-02named newly granted holders with stale checkpoint stamps: 12nn.npln.*, onetera::FileResourceMgr, each reporting 0.0 ms held after rounding. None of those holder samples was in state D. The disturbedf1-01remains in the table: it included the previously reported 433.6 ms Opus decode and D-state samples.No current-main/+2387 matrix, rolling/probe implementation, or Windows test is represented here. No additional game runs were made to assemble this table.
@max855k please attach the complete
hookshot_log.txt, including its first version/build line and the backtraces (personal paths can be redacted; no saves, game files or core dump). The escapes preceding the faults are useful evidence, but do not by themselves establish cached-core/TLS corruption. Keeping the working six-core configuration is sensible while this is investigated; the separate performance report can stay in #995.Rolling deadline + probe, corrected test, and the 250 ms / 2 s matrix on
940eb9ef@mooki000 — you were right on both test points and on the static stamp. Here is what changed, the corrected test, and the matrix. The matrix ran on
940eb9ef+ the diff below (a dirty build,source 287f6032); I am giving the exact head rather than claiming current main.The rule. I replaced the static
vc_last_poll >= t0with a rolling window and added the probe kick you preferred:vc_last_poll(last checkpoint) andvc_last_take(last core acquisition) are stamped inhos_vcore_checkpointandvc_take.vc_kick = 3to better holders on its allowed cores. The holder reaches a checkpoint, records it, and clears the probe without yielding. It is not a yield request.last_progress = max(t0, holders on the waiter's allowed cores: last_poll, last_take); the escape fires only whennow - last_progress > timeout. A take resets the window; a holder that stops making progress trips the deadlock breaker one window later.HOS_VCORE_PROGRESS=0keeps the old unconditional path for the control arm.The
held/stampfields in the escape lines are the diagnostics added with this diff (the log hunk is in the diff above).The test. Both review points are fixed:
probes_answered >= 3), checksescapes_when_probes_stop == 0, and the watchdog fails if the awaited escape never comes.It passes with g++ and clang++; a copy of
hos_thread.cwith the static rule fails at the 3 s watchdog. I can paste the fulltests/test_vcore_progress.py.Matrix on
940eb9ef+ the diff (it already contains #2301/#2302/#2322 and #2387). Four runs per arm, anonymous window, 0 crashes in all 12, fps 59.3–60.8. These are the harness's ~230 s runs (boot/title plus 45 s after Start); the escape counts are over the whole run, not only the 45 s.BootupBackgroundWorkThread(30), a worse holder in host codeBaseProcCreateAndDeleteThread30,sead::MainThread21,ParallelDecompThread19,EffectResLoadingThread11,BootupBackgroundWorkThread7,ARes Loading 13,ムービー入力読込3,ModuleSystemWorker42,tera::FileResourceMgr1,Resource Memory1BootupBackgroundWorkThread(30), a worse holder in host code, held 252.8–315.6 ms with a 250.4–312.4 ms old stamp. No better-holder escape in this arm.ParallelDecompThread(19) held 0.3–4.8 ms with a 0.3–77.6 ms stamp;BaseProcCreateAndDeleteThread(23) held 0.0–13.7 ms.sead::MainThread(16) is 21 escapes; 16 of them are the long holds (301.6–1381.3 ms, stamp 0.0–3.5 ms), the other five held 0.0–0.1 ms. The ones with a 0–3.5 ms stamp were still making progress when the rule-off build escaped them — the harmful case the rule is for. The static-vs-rolling proof is test case 1 (a fixed fresh stamp), not this sample.Not measured: Windows, the probe path in a game run, and the kernel log (the journal has no entries during the matrix window on this boot, so I am not claiming a clean kernel check).
Does this answer the take/static concern? The window length and probe rate are the knobs I would expect to tune.
🤖 Generated with opencode using DeepSeek V4.1 Flash.
@CIB thanks. I applied your 22:10 diff unchanged to
940eb9ef. It applies cleanly and builds.Your test. The corrected
tests/test_vcore_progress.pyisn't in the thread yet ("I can paste the full ..."), so I haven't run it, and I can't confirm the two fixes from the description alone (case 1 no longer rewinding its stamp, andescapes_at_stampno longer starting at zero). Please paste it, and I'll run it with g++ and clang++, on your diff and against main's source. In the rule itself I found nothing that moves a stamp backwards: both stamps are written only fromhos_now_ns().Your rule, with our own game-free check. The virtual-core block is cut out of your
hos_thread.cand run single-threaded with scripted holders, with the escape at 250 ms, on g++ and clang++. All times below are that harness's clock, not the game's.vc_kick. Take a holder H, a worse waiter A that probes it, and a waiter B that is better than H. While A's probe is pending, B can't set kick 1, and H's answer to the probe clearsvc_kickwithout a yield. If A's look always comes just after H's checkpoint, B gets the core only when H blocks. In the test below that is 1000 ms with your diff, against 1 ms on main and 1 ms with the change below. The test fails on your diff and passes with the change, with g++ and clang++. It also passes on main, which has no probe. With the change, every other case of our check comes out the same as on your diff.The first hunk is the fix the test checks. The second hunk keeps the holder's clear of the probe from wiping a yield request that replaced the probe in the meantime. That race needs two threads, so the single-threaded test can't show it; I found it by reading the code. I can paste the test if you want it beside yours.
vc_last_pollwith a plain load, where your 17:31 hunk used__atomic_load_n. And a probed holder now showsaskedin the escape line, so "asked" there no longer means only a yield request.The matrix. Our side of it (250 ms and 2 s, your rule on and off, two runs per arm) isn't run yet, and I'll add it when this machine is quiet. The build's shader cache started empty, and the preparation got through only 2543 of 38809 programs in 7 minutes while the machine's load was at 190 to 339. Escape counts taken at that load wouldn't compare with yours or with our earlier runs.
@mooki000 — the corrected
tests/test_vcore_progress.py, as asked. Both review points are in it:probes_answered >= 3,escapes_when_probes_stop == 0, and the watchdog fails if the awaited escape never comes.It passes with g++ and clang++ on the diff I posted; a copy of
hos_thread.cwith the static rule fails at the 3 s watchdog. The fixture defines no-opHOS_CHAINmacros because current main's virtual-core block uses them.🤖 Generated with opencode using DeepSeek V4.1 Flash.
@CIB thanks for pasting it. I ran
tests/test_vcore_progress.pyexactly as posted, with g++ and with clang++; both give the same result:host/hos_thread.c940eb9efokok940eb9ef+ your 17:31 static-rule hunkFAIL: no escape after 3 s of waiting (a static rule?)940eb9efno member named 'vc_last_poll'(the field comes with your diff)Both fixes are in. Case 1 sets a fresh stamp once and never moves it. Case 2 passes only if the holder answered a
vc_kick == 3probe at least three times (escapes_when_probes_stopstarts at -1 and must end at 0). The test doesn't cover the shared-slot case, though: it passes on your diff, where my probe-slot test fails (1000 ms against 1 ms, on the harness's clock). So the two tests would sit well side by side. Our side of the matrix still waits for a quiet machine.@CIB here is our side of the matrix, on
940eb9efwith your 22:10 diff unchanged, two runs per arm. Settings as in our earlier runs: Arch, RTX 4090, NVIDIA 615.78.08, headless, a fresh profile, Start at 95 s, 115 s per run. The build's shader cache was empty, so every run went without the preparation (RX_VK_PRECOMPILE=0), after one uncounted warm-up run of the same steps.¹ Another job's shader preparation ran beside that run, and the game ran at 34 to 47 fps; kept, not dropped. The waits are the longest from the game's 5-second
vcores:lines.stampfield).BaseProcCreateAndDeleteThreadwas the holder at 8 (held 0.1 to 9.7 ms),ParallelDecompThreadat 5 (held 0.1 to 1.4 ms) andsead::MainThreadat 2 (held 312 and 333 ms, last checkpoint 0.1 to 0.2 ms before). With the rule on, none of those escapes was left. The same three threads lead your rule-off list.BootupBackgroundWorkThread; we had none. The thread runs here too, but it never held a core at an escape in any of our runs. Our runs are shorter than yours (20 s after Start against your 45 s), use the default memfd guest window rather than your anonymous one (HOS_GUEST_MEMFD=0), and skip the preparation, so I can't say which of these explains it.kickscounter, which includes your probes, roughly triples with the rule on: 443,647 and 352,440 per run against 145,376 and 132,839 at 250 ms.Not tested: Windows, anything more than 20 s after Start, a prepared shader cache, and the probe-slot case in the game (it needs two waiters' looks to fall in the wrong order).
github-sync referenced this issue2026-11-12 06:22:54 +00:00
Status update: the rule is not the fix
I agree with the audit conclusion and mooki's matrix: the progress-rule + probe is a mitigation, not the right solution. It leaves the escape as a tokenless identity violation, it can mask a spin-waiting better holder, it has no measured benefit at the 2 s default, and at 250 ms it trades escapes for 1.2-1.4 s core waits. I am not proposing it as the fix.
The direction I am taking is the deeper one: stable home core, no escapes, and host waits releasing their cores. Before changing that invariant I am building debug-only vcore diagnostics - a low-overhead event ring, strict token/live/TLS-core identity checks, a watchdog, and host-block markers - all off by default. Its job is to name the real culprits (host lock, host syscall, fiber, guest spin, scheduler) instead of inferring them.
I will open a PR for the diagnostics when it is validated, and keep the scheduler refactor for after that. No PR is open yet.
🤖 Generated with opencode using DeepSeek V4.1 Flash.