HOS_SAMPLE_ALL still crashes the game on Windows after #1343: the fault is inside RtlVirtualUnwind, past unwind_step's checks #1584

Open
opened 2026-11-18 13:55:01 +00:00 by enum · 3 comments
Contributor

The all-thread sampler (HOS_SAMPLE_ALL, which HOS_PROFILE starts) can still crash the game on Windows after #1343: the fault is now inside RtlVirtualUnwind, past the checks unwind_step() makes before calling it.

Seen on

Windows 11 (build 26300), RTX 4070 (NVIDIA 32.0.16.1656), Ryzen 9 7900X, 64 GB. main @ 310e7c9d (v3.0.0-rc.7-61-g310e7c9d), which has #1343 (92c5fca2).

Settings and switches

HOS_PROFILE=5, HOS_SAMPLE_ALL=<run>/all.txt, HOS_FPS_MAX=240, a 1280x720 window. Child Link in ZoraDomain_Child, standing at the stairs behind King Zora (about 56 fps, the game's job workers busy with Havok cloth: #1485).

Steps

  1. Start the game with the switches above and teleport to ZoraDomain_Child (child).
  2. Create <run>/all.txt.go to start sampling.

Expected: the run goes on, and the sampler file holds the window.
Actual: the game closed within the 30 s window (the sampler file had about 230,000 lines by then):

[hos host] CRASH: exception 0xc0000005 (read 0x73722082c0, guest 0xfffff073722082c0) at ntdll.dll+0x1ff2a
[hos host] CRASH: host backtrace: ntdll.dll+0x1ff2a ntdll.dll+0x1f549 ntdll.dll+0x213cb KERNEL32.DLL+0x281c8 longshot.exe+0x1c27d11b longshot.exe+0x1c27c523 KERNEL32.DLL+0x2dda7 ntdll.dll+0xc9c5c

From the build's longshot.map and a disassembly of longshot.exe:

  • longshot.exe+0x1c27c523 is sample_all_thread +0xb23.
  • longshot.exe+0x1c27d11b is unwind_step +0xcb, the instruction right after call *__imp_RtlVirtualUnwind (the RtlLookupFunctionEntry call is at +0x44). So the fault is in RtlVirtualUnwind, after unwind_step has checked that Rsp, and the frame register when the unwind data names one, are on the walked thread's committed stack.

How often: once in 2 runs. The same scene with two 10 s windows in a new session ran without a crash. #1343's own run crashed after 135 s on main before it.

What might cause it (not checked)

unwind_step checks the stack pointer and the frame register before the step, but RtlVirtualUnwind also reads at offsets from them: the saved registers (UWOP_SAVE_NONVOL and UWOP_SAVE_XMM128 at Rsp + offset), and the return address after a large UWOP_ALLOC_LARGE. A thread stopped where its unwind data doesn't match its real stack would pass the check at Rsp and still read far past it. The faulting address (0x73722082c0) is not near any stack the check accepted, though I didn't record the walked thread's [lo, hi). Checking each read's address against the stack before the step, or walking in a thread that can take the fault (a vectored handler for the sampler's own thread), would cover it.

Related: #383 (closed), #1343 (merged), #1044 (WIP, the same sampler's timestamps).

The all-thread sampler (`HOS_SAMPLE_ALL`, which `HOS_PROFILE` starts) can still crash the game on Windows after #1343: the fault is now inside `RtlVirtualUnwind`, past the checks `unwind_step()` makes before calling it. ### Seen on Windows 11 (build 26300), RTX 4070 (NVIDIA 32.0.16.1656), Ryzen 9 7900X, 64 GB. main @ 310e7c9d (`v3.0.0-rc.7-61-g310e7c9d`), which has #1343 (92c5fca2). ### Settings and switches `HOS_PROFILE=5`, `HOS_SAMPLE_ALL=<run>/all.txt`, `HOS_FPS_MAX=240`, a 1280x720 window. Child Link in ZoraDomain_Child, standing at the stairs behind King Zora (about 56 fps, the game's job workers busy with Havok cloth: #1485). ### Steps 1. Start the game with the switches above and teleport to ZoraDomain_Child (child). 2. Create `<run>/all.txt.go` to start sampling. **Expected:** the run goes on, and the sampler file holds the window. **Actual:** the game closed within the 30 s window (the sampler file had about 230,000 lines by then): ``` [hos host] CRASH: exception 0xc0000005 (read 0x73722082c0, guest 0xfffff073722082c0) at ntdll.dll+0x1ff2a [hos host] CRASH: host backtrace: ntdll.dll+0x1ff2a ntdll.dll+0x1f549 ntdll.dll+0x213cb KERNEL32.DLL+0x281c8 longshot.exe+0x1c27d11b longshot.exe+0x1c27c523 KERNEL32.DLL+0x2dda7 ntdll.dll+0xc9c5c ``` From the build's `longshot.map` and a disassembly of `longshot.exe`: - `longshot.exe+0x1c27c523` is `sample_all_thread` +0xb23. - `longshot.exe+0x1c27d11b` is `unwind_step` +0xcb, the instruction right after `call *__imp_RtlVirtualUnwind` (the `RtlLookupFunctionEntry` call is at +0x44). So the fault is in `RtlVirtualUnwind`, after `unwind_step` has checked that `Rsp`, and the frame register when the unwind data names one, are on the walked thread's committed stack. How often: once in 2 runs. The same scene with two 10 s windows in a new session ran without a crash. #1343's own run crashed after 135 s on main before it. ### What might cause it (not checked) `unwind_step` checks the stack pointer and the frame register before the step, but `RtlVirtualUnwind` also reads at offsets from them: the saved registers (`UWOP_SAVE_NONVOL` and `UWOP_SAVE_XMM128` at `Rsp + offset`), and the return address after a large `UWOP_ALLOC_LARGE`. A thread stopped where its unwind data doesn't match its real stack would pass the check at `Rsp` and still read far past it. The faulting address (0x73722082c0) is not near any stack the check accepted, though I didn't record the walked thread's `[lo, hi)`. Checking each read's address against the stack before the step, or walking in a thread that can take the fault (a vectored handler for the sampler's own thread), would cover it. Related: #383 (closed), #1343 (merged), #1044 (WIP, the same sampler's timestamps).
Member

Possibly related, for context:

  • issue #383 fix(host): HOS_SAMPLE_ALL can crash the game: the sampler's stack walk (RtlVirtualUnwind) is unguarded — Same HOS_SAMPLE_ALL RtlVirtualUnwind crash; this reports recurrence after guard checks
  • PR #1343 fix(host): the samplers' stack walks stay on the walked thread's stack, so HOS_PROFILE=1 no longer crashes the game on Windows — Merged stack-walk safety fix that issue says did not prevent this occurrence
Possibly related, for context: - issue #383 fix(host): HOS_SAMPLE_ALL can crash the game: the sampler's stack walk (RtlVirtualUnwind) is unguarded — Same HOS_SAMPLE_ALL RtlVirtualUnwind crash; this reports recurrence after guard checks - PR #1343 fix(host): the samplers' stack walks stay on the walked thread's stack, so HOS_PROFILE=1 no longer crashes the game on Windows — Merged stack-walk safety fix that issue says did not prevent this occurrence
Contributor

A possibly related observation from Windows, but not the same symptom as this issue: we saw startup nonprogress, not an observed RtlVirtualUnwind crash.

Setup:

  • main 430eef4f, built with llvm-mingw;
  • Windows 11 Pro 26300.9818, i7-13700K;
  • an RTX 5090 (NVIDIA 621.14) and an Intel UHD 770;
  • RTSS running, with its Vulkan implicit layer loaded.
  • These runs were headless with HOS_PROFILE=1, which starts the nominal 2 ms all-thread sampler during boot, and RX_VK_PASS_TIMING=0.

What happened: across the two retained profile attempts, two of five game launches stalled during Vulkan initialization. The other three launches completed.

  • One stayed at frame 0 for ~32 s. Its log ends after the output-resolution sizes, and no quit completed.
  • The other stopped logging after Intel device enumeration, for ~25 s. Its launcher also had a separate WinError 5 replacing manifest.json.
  • MainThread CPU was reported near zero.

The samples: the retained all.txt files from the stalls contained 20 stack records through 59 ms (4,130 bytes) and 45 through 120 ms (12,369 bytes).

  • They contain early-init evidence only, not a stack showing where the later stall occurred.
  • The three completed launches wrote about 55–99 MB.
  • Because output is buffered, the last retained record is not an exact time at which the sampler stopped making progress.

Without the sampler: ten earlier windowed launches on 430eef4f and the control 79d4a143 had no observed stall without HOS_PROFILE. A different presentation mode, build cohort and other profiling instrumentation make that an uncontrolled contrast, not an estimated sampler failure rate.

Unconfirmed source-level lead:

  • sample_all_thread performs module lookup/name resolution and unwind_step while the target remains suspended.
  • If a target holds a loader/function-table-related lock needed by those calls, that could create a suspend/lock deadlock.
  • ResumeThread precedes hos_sample_write, so the output writer is not inside that suspended-target interval.
  • We have no hang dump or lock-owner evidence, so the cause and any relationship to this issue remain unresolved. The early stacks do not implicate RTSS or the Intel driver.
A possibly related observation from Windows, but **not** the same symptom as this issue: we saw startup nonprogress, not an observed `RtlVirtualUnwind` crash. **Setup:** - main `430eef4f`, built with llvm-mingw; - Windows 11 Pro 26300.9818, i7-13700K; - an RTX 5090 (NVIDIA 621.14) and an Intel UHD 770; - RTSS running, with its Vulkan implicit layer loaded. - These runs were headless with `HOS_PROFILE=1`, which starts the nominal 2 ms all-thread sampler during boot, and `RX_VK_PASS_TIMING=0`. **What happened:** across the two retained profile attempts, two of five game launches stalled during Vulkan initialization. The other three launches completed. - One stayed at frame 0 for ~32 s. Its log ends after the output-resolution sizes, and no quit completed. - The other stopped logging after Intel device enumeration, for ~25 s. Its launcher also had a separate WinError 5 replacing `manifest.json`. - MainThread CPU was reported near zero. **The samples:** the retained `all.txt` files from the stalls contained 20 stack records through 59 ms (4,130 bytes) and 45 through 120 ms (12,369 bytes). - They contain early-init evidence only, not a stack showing where the later stall occurred. - The three completed launches wrote about 55–99 MB. - Because output is buffered, the last retained record is not an exact time at which the sampler stopped making progress. **Without the sampler:** ten earlier windowed launches on `430eef4f` and the control `79d4a143` had no observed stall without `HOS_PROFILE`. A different presentation mode, build cohort and other profiling instrumentation make that an uncontrolled contrast, not an estimated sampler failure rate. **Unconfirmed source-level lead:** - `sample_all_thread` performs module lookup/name resolution and `unwind_step` while the target remains suspended. - If a target holds a loader/function-table-related lock needed by those calls, that could create a suspend/lock deadlock. - `ResumeThread` precedes `hos_sample_write`, so the output writer is not inside that suspended-target interval. - We have no hang dump or lock-owner evidence, so the cause and any relationship to this issue remain unresolved. The early stacks do not implicate RTSS or the Intel driver.
Contributor

Claiming this: I have a fix from a local branch that hit the same crash (0xc0000005 inside RtlVirtualUnwind, reading just past the walked thread's StackBase, from unwind_step). The cause found there: clang puts a vzeroupper between an epilogue's last pop and its ret in AVX code, Windows' unwinder does not take that for an epilogue, so a thread stopped among the pops is unwound as if in the body and the whole prologue is undone again on a stack pointer the pops already moved: it reads up to the frame's size too high. The fix unwinds such an epilogue in the walk itself and works out every read RtlVirtualUnwind will make from the frame's unwind codes before the call, as this issue suggests. A pull request with a game-free single-step test follows once it has run in the game here (Windows 11, RTX 4060).

Claiming this: I have a fix from a local branch that hit the same crash (0xc0000005 inside `RtlVirtualUnwind`, reading just past the walked thread's StackBase, from `unwind_step`). The cause found there: clang puts a `vzeroupper` between an epilogue's last `pop` and its `ret` in AVX code, Windows' unwinder does not take that for an epilogue, so a thread stopped among the pops is unwound as if in the body and the whole prologue is undone again on a stack pointer the pops already moved: it reads up to the frame's size too high. The fix unwinds such an epilogue in the walk itself and works out every read `RtlVirtualUnwind` will make from the frame's unwind codes before the call, as this issue suggests. A pull request with a game-free single-step test follows once it has run in the game here (Windows 11, RTX 4060).
Sign in to join this conversation.
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1584
No description provided.