fix(mem): a thread's stack page refused at the commit limit waits for commit, then the thread goes on #1885
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1885
Loading…
Reference in a new issue
No description provided.
Delete branch "d4nkn3ss/Longshot:fix/stack-commit-limit"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
When the PC runs out of memory (Windows' commit limit: RAM plus the page file), Longshot could crash with a "stack overflow" (
0xc00000fd) on whichever game thread happened to need one more page of stack. The thread wasn't deep in anything. A thread's stack is only reserved (the guest's 64 MB), and Windows commits each page as the stack grows. At the commit limit it can't, and it raises a stack overflow with nearly all of the stack unused. Guest memory already waits there for the renderer's textures to give commit back (commit_wait); a thread's stack didn't. That was the first crash in #1772: a 24 GB PC at its limit,0xc00000fdon PreActorModelCompose, inucrtbase.dll, after the log'smem: commit of 0x10000 bytes at the commit limit made after ...lines.Now a first vectored handler takes that one stack overflow. It acts only when more than 64 KB of the thread's reservation is left. It waits for commit as a guest page does (up to 5 s), sets the guard page again below the stack, and the thread goes on. If commit never comes back, the out-of-memory line and notice come first, then the crash report as before. A stack that really ran out goes straight to the crash report. Windows only.
On main: run the game with a per-process commit limit below what it needs, a Job object with
JOB_OBJECT_LIMIT_PROCESS_MEMORYat 14 GB here, and load a save: 2 of 3 starts crashed with0xc00000fd(onrmbl::VibrationThreadandGeneralFormatAsync_, both inucrtbase.dll). A real PC gets there with a small fixed page file.See #1772, and likely #1919 (the same signature on the Resource Control thread, v3.0.0). #1772's later crash, in
nvoglv64.dllunder VRAM and commit pressure, isn't this one: see Risks.Read first:
stack_on_touch()inhost/mkw_glue.cand its registration inhos_commit_init(); thentests/test_stack_commit.pyandtests/stack_commit_test.c.Testing
Tested:
89c3d21e(both built here): an adult save in Hyrule Field, the first teleport to the Market, then quit.CRASH,FATALorVK_ERRORline.0xc00000fdin 2 of 3. None of this PR's did. In one, a stack page was refused and recovered (mem: a thread's stack grew at the commit limit after waiting for commit). They ended instead in NVIDIA's driver (nvoglv64.dll,nvwgf2umx.dll: writes near 0), in the renderer's C++ allocation refused for 5 s, and once in a host write to 0. Those are other allocations failing at the limit; see Measurements and Risks.python -I -B tests/test_stack_commit.py: fails on main (a thread whose stack page was refused at the commit limit died with a stack overflow (0xc00000fd)), passes here.mkw_glue.c's real code fromcommit_waitto the crash report, with the handlershos_commit_initregisters from it. It puts itself in a Job with a commit limit, fills that, and grows a reserved 64 MB stack while another thread gives commit back after 300 ms.0xc00000fd), and a stack that really runs out (0xc00000fdat once, not retried).PAGE_GUARDstill passes; so does dropping the 64 KB "really ran out" margin, since the handler stops at the reservation's end either way. Both are kept as Windows expects them (_resetstkoflwsets the guard page again; the margin leaves the crash report its room).host/mkw_glue.cpass as on main: test_commit_limit, test_host_alloc_wait, test_crash_record, test_sampler_stack_walk, test_renderer_new_handler, test_input_background, test_nvn_host_tables. test_profiling_mode and test_slow_frames fail here and on main @89c3d21ealike.host/mkw_glue.cwith-fsyntax-only -Wall -Wextra: clang (llvm-mingw, Windows) and gcc 14.2 (Linux: Debian under WSL 2), here and on main. The same warnings on both (5 on Windows, 1 on Linux), none in the new code. The Windows game builds and links here; the Linux host wasn't linked.python -I -B tests/run_all.py(Windows, llvm-mingw from the Longshot Builder;miseisn't installed on this PC), this PR on main @89c3d21eagainst main @89c3d21e, both run the same way: 764 passed, 65 failed here; 765 passed, 63 failed on main. The 63 fail on both (test_vbytesby its 600 s timeout). The other two:test_stack_commitfailed in that first full run, at 24 checks at once: the fixture's thread that gives commit back had an ordinary stack and itself needed a stack page at the limit, so it never gave any back. The fixture now commits that thread's 256 KB of stack up front (in the test commit). After that, 12 copies run at once all passed, and the test commit alone still fails on main.test_recording_admissionis flaky on this PC on main too (its parallel case).test_job_queuewas run from Git Bash, because started from this PowerShell it gets a Ctrl+C that ends the run: 20 of its 21 pass,Environment.test_ram_file_systemfails, as on main.2210931c(no change to these files there): test_stack_commit, test_commit_limit, test_host_alloc_wait and test_crash_record pass.Still to test:
0xc00000fdand, when a stack page is refused,a thread's stack grew at the commit limit(by hand: #1772's reporter is on Windows 10 19045)@Beanstalk start the game, load a save and play a minute, Windows AMD and Intel)Measurements
Verdict: fixed: at the commit limit a thread's stack waits for commit instead of crashing with a stack overflow. On main @
89c3d21eit crashed so in 2 of 3 starts under a forced limit, this PR in none of 4. A PC that stays at its limit still crashes elsewhere: the driver's and the renderer's own allocations.Setup: Windows 11 Pro 26300, RTX 3080 10 GB (NVIDIA 621.14), i9-12900K, 64 GB. An adult save in Hyrule Field, the first teleport to the Market. The game runs in a Job object whose per-process commit limit is 14 GB (its peak with no limit is 17.0 GB), one start per run, interleaved. Read from
longshot_log.txtand the crash record. Not measured again on main @2210931c, which this PR is now on: the commits since89c3d21edon't touchhost/mkw_glue.c, the threads' stacks or the commit waits, so the numbers hold.89c3d21e89c3d21e)0xc00000fd(stack overflow)rmbl::VibrationThread,GeneralFormatAsync_)nvoglv64.dllwrite 0x10,nvwgf2umx.dllwrite 0x8, the renderer's allocation refused for 5 s, a host write to 0Risks
SetThreadStackGuarantee(none of Longshot's do) would have a real overflow carried into its guarantee, and crash a little later with less room.nvoglv64.dllcrash. The player's way out stays the out-of-memory notice's: a bigger page file.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Tested in game: Windows 11 Pro 26300, RTX 3080 10 GB (NVIDIA 621.14): an adult save in Hyrule Field, the first teleport to the Market. Without a limit, 2 starts arrived and closed normally. Under a 14 GB per-process commit limit (a Job object; the game peaks at 17.0 GB), interleaved with main @
89c3d21e: main crashed with 0xc00000fd in 2 of 3 starts, this PR in 0 of 4, with one stack page refused, waited for and grown (mem: a thread's stack grew at the commit limit after waiting for commit). Its runs ended elsewhere at that limit (NVIDIA's driver, the renderer's allocation), as Measurements says.WIP: fix(mem): a thread's stack page refused at the commit limit waits for commit, then the thread goes onto fix(mem): a thread's stack page refused at the commit limit waits for commit, then the thread goes onPossibly related, for context:
Recommendation: Wait. Approve once the forge's Windows build compiles the three commits on top (pushed to your branch) and a Windows run at a small page file passes. This PC has no Windows toolchain: the new handler code was only syntax-checked against stub declarations, and the new test cases weren't run here (CI runs them). The cause is right and the design is right. One real bug: the handler runs out of stack in its own log lines. The commits on top fix that. Without them I'd request changes, because the path where commit never comes back would die silently.
Does it fix the cause?
longshot_log.1.txt, run 6e14ecec) hasvk: commit limit near: 491 MB of commit left, then 8mem: commit of 0x10000 bytes at the commit limit made after ...lines. The record says0xc00000fdon PreActorModelCompose inucrtbase.dll, 28 s in. That is a reserved-only 64 MB stack (hos_thread.c:609) refused its next page, not deep recursion. The laternvoglv64.dllcrash isn't this one, and the PR says so.ucrtbase.dll+0x10bfbframe, Resource Control thread, but no log to confirm the limit.0xc00000fdon Resource Control. #433's other reports (Intel Arc, the 3070 laptop, logs that end with a normal close) aren't shown to be this. It shouldn't be closed by this PR.The bug: the handler has almost no stack
When Windows can't commit the next guard page, the thread's stack pointer is in the page it just committed. Below that page there is only reserved memory, with no guard page. The exception frames (CONTEXT and record, about 1.5 KB) go there, and then the handler runs on what is left: under 4 KB, often much less.
hos_logtakes 4.3 KB of buffers (hos_vlog: 1024 + 1100 + 2200, plushos_log_redact). So:commit_wait's "still refused after 5 s" line, then the handler's own line, run off the page into reserved memory. That is an access violation inside the vectored handler, on a stack with no room to dispatch it, and the process ends without the out-of-memory line, the notice or the crash report. On main, at least the crash report had a chance.commit_wait's success line runs into the one 4 KB page just committed. Whether it fits depends on where in the page the fault came. Your in-game recovery fit; it isn't guaranteed.The test didn't catch this, for two reasons. Its frames are 16 KB, so the fault comes from a
___chkstk_msprobe 16 KB below a stack pointer with plenty of committed stack above it. And itshos_logstand-in takes 512 bytes.What the commits on top do
smallcases with 512-byte frames, so the stack pointer is in the last committed page, as inucrtbase.dll. The stand-inhos_lognow takes the real one's 4.3 KB.never smallshould fail on your head (unverified: no Windows here).commit_wait's 100 x 50 ms, but saying nothing.StackLimitasPAGE_READWRITE | PAGE_GUARD. Windows hands those guard pages to the stack one at a time as it grows, asSetThreadStackGuarantee's region is handed out, so there is no re-entry andStackLimitstays right. Only then does it log.stack_sparecommitted inhos_commit_initis given back for that 64 KB, so the line, the notice and the crash report have a stack to run on.VirtualProtectis gone, since the commit is made as guard pages to begin with.docs/user/common-issues.md, "Windows: an out-of-memory notice, or a crash with0xc00000fd".Safety (with the fix)
InterlockedExchangePointer).report_fatalstays last.#ifdef _WIN32, and the preprocessed Linux code is unchanged.StackLimitsemantics: recovery relies onStackLimit - 1being reserved when the overflow is raised. Your Windows 11 runs show that, but Windows 10 19045 (the reporter's) is untested.Switches: none, and none needed: it only acts on an exception that would otherwise end the game.
Before merge
python -I -B tests/test_stack_commit.pyhere, and on your head to seenever smallfail.mem: a thread's stack grew at the commit limit after waiting N ms for commit (k so far); the line's text changed. Update the PR body's quoted line.@ -225,0 +239,4 @@static LONG CALLBACK stack_on_touch(EXCEPTION_POINTERS *info) {if (info->ExceptionRecord->ExceptionCode != EXCEPTION_STACK_OVERFLOW) return EXCEPTION_CONTINUE_SEARCH;char *const limit = (char *)((NT_TIB *)NtCurrentTeb())->StackLimit;MEMORY_BASIC_INFORMATION mbi;The handler runs on what is left of the page Windows did commit: under 4 KB, less the exception's own frames, with only reserved pages below.
commit_waitlogs on both of its ends (hos_log, 4.3 KB of buffers inhos_vlog). When commit never comes back, its "still refused after 5 s" line runs into reserved memory. That is an access violation inside the vectored handler, with no room to dispatch it, and the process ends with no out-of-memory line, notice or crash report. Fixed on top: a quiet wait loop (the same 100 x 50 ms), and the 64 KB belowStackLimitcommitted as guard pages before any line.@ -225,0 +234,4 @@own loop, as commit_wait's but saying nothing, and the 64 KB is committed as guard pages, which Windows then hands to thestack one at a time as it grows, as its own. Commit that never came back: stack_spare, 1 MB committed at the start, is givenback for that 64 KB, so the line, the notice and the crash report have a stack to run on */enum { STACK_ROOM = 0x10000, STACK_SPARE = 1 << 20 };With the 64 KB room committed below the stack, "really ran out" is now under 128 KB left (twice the room), so the room never reaches the reservation's last pages. A real overflow leaves a few pages, so it still goes straight to the crash report.
@ -225,0 +245,4 @@if (left < 2 * STACK_ROOM) return EXCEPTION_CONTINUE_SEARCH; /* (it really ran out) */char *const room = limit - STACK_ROOM;int ms = 0;while (!VirtualAlloc(room, STACK_ROOM, MEM_COMMIT, PAGE_READWRITE | PAGE_GUARD)) {This runs with nothing committed below the stack (the guard page was refused), so this line is where the process dies when commit never comes back. On top, a 1 MB
stack_sparecommitted inhos_commit_initis given back for the 64 KB room once the 5 s are up, so this line,hos_out_of_memory(1)and thenreport_crashhave a stack. For an error other than the commit limit there's no room to say anything, so it goes on to the crash report as on main.@ -0,0 +22,4 @@/* ---- stand-ins for what the cut code calls ---- */#define GUEST_SPACE (1ull << 30)static uint8_t *A64_MEM;#define HOS_GUEST_SPACE GUEST_SPACEThe real
hos_logtakes 4.3 KB of stack, and this stand-in 512 bytes. With it, a handler that logs before it has room passes here and dies in the game. On top, the stand-in takes the real one's size.@ -0,0 +54,4 @@#include "stack_commit_handlers.inc"}/* ---- the case ---- */With 16 KB frames the fault comes from a
___chkstk_msprobe up to 16 KB below the stack pointer, so the handler runs with plenty of committed stack above it. The game's faults are inucrtbase.dll's small frames, with the stack pointer in the last committed page. On top,smallcases (512-byte frames) forrecoversandnever.Pushed three commits on top for the review above (test with 512-byte frames, the handler commits room before it logs plus a 1 MB spare, a help-page section). Not compiled on Windows here and no tests run here: CI and a Windows run need to confirm.
@Beanstalk Windows 10 and Windows 11, any GPU: with a small fixed page file (e.g. 2 GB on a 16 GB PC), load a save and teleport between Hyrule Field and the Market for a few minutes. Expect no 0xc00000fd crash; a "mem: a thread's stack grew at the commit limit after waiting" line means it recovered; if the PC stays out of memory, the out-of-memory notice shows before the game closes. Also a normal start on Windows AMD and Intel: play a minute, expect it to play and close as on main.
Beanstalk test farm · ⛔ can't be tested as asked
Commit
3b1ad48· approved by HumanBean (untrusted) · dashboardNot run: the farm can't test this as asked, so nothing is built or run:
⚠️ Note: The recovery and persistent-out-of-memory branches cannot be reliably reached or judged without controlled memory pressure.
⚠️ Note: The farm cannot run the new standalone stack-commit fixture through a gameplay plan.
What it could test instead: The alternative plan loads a save and runs a minute of Hyrule Field gameplay on Windows AMD and Intel PCs, comparing main with the pull request. It takes about 12 minutes per tester and fails on a crash or a 15% increase in p99 frametime. No Intel tester is currently online, so that target will wait. No tester online now matches windows+intel: that target waits for one.
A member of
BigBeanscan comment@Beanstalk <new words>to ask for something else, or@Beanstalk test it anywayto run the plan regardless.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.