HOS_SAMPLE_ALL still crashes the game on Windows after #1343: the fault is inside RtlVirtualUnwind, past unwind_step's checks #1584
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#1584
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The all-thread sampler (
HOS_SAMPLE_ALL, whichHOS_PROFILEstarts) can still crash the game on Windows after #1343: the fault is now insideRtlVirtualUnwind, past the checksunwind_step()makes before calling it.Seen on
Windows 11 (build 26300), RTX 4070 (NVIDIA 32.0.16.1656), Ryzen 9 7900X, 64 GB. main @
310e7c9d(v3.0.0-rc.7-61-g310e7c9d), which has #1343 (92c5fca2).Settings and switches
HOS_PROFILE=5,HOS_SAMPLE_ALL=<run>/all.txt,HOS_FPS_MAX=240, a 1280x720 window. Child Link in ZoraDomain_Child, standing at the stairs behind King Zora (about 56 fps, the game's job workers busy with Havok cloth: #1485).Steps
<run>/all.txt.goto start sampling.Expected: the run goes on, and the sampler file holds the window.
Actual: the game closed within the 30 s window (the sampler file had about 230,000 lines by then):
From the build's
longshot.mapand a disassembly oflongshot.exe:longshot.exe+0x1c27c523issample_all_thread+0xb23.longshot.exe+0x1c27d11bisunwind_step+0xcb, the instruction right aftercall *__imp_RtlVirtualUnwind(theRtlLookupFunctionEntrycall is at +0x44). So the fault is inRtlVirtualUnwind, afterunwind_stephas checked thatRsp, and the frame register when the unwind data names one, are on the walked thread's committed stack.How often: once in 2 runs. The same scene with two 10 s windows in a new session ran without a crash. #1343's own run crashed after 135 s on main before it.
What might cause it (not checked)
unwind_stepchecks the stack pointer and the frame register before the step, butRtlVirtualUnwindalso reads at offsets from them: the saved registers (UWOP_SAVE_NONVOLandUWOP_SAVE_XMM128atRsp + offset), and the return address after a largeUWOP_ALLOC_LARGE. A thread stopped where its unwind data doesn't match its real stack would pass the check atRspand still read far past it. The faulting address (0x73722082c0) is not near any stack the check accepted, though I didn't record the walked thread's[lo, hi). Checking each read's address against the stack before the step, or walking in a thread that can take the fault (a vectored handler for the sampler's own thread), would cover it.Related: #383 (closed), #1343 (merged), #1044 (WIP, the same sampler's timestamps).
Possibly related, for context:
A possibly related observation from Windows, but not the same symptom as this issue: we saw startup nonprogress, not an observed
RtlVirtualUnwindcrash.Setup:
430eef4f, built with llvm-mingw;HOS_PROFILE=1, which starts the nominal 2 ms all-thread sampler during boot, andRX_VK_PASS_TIMING=0.What happened: across the two retained profile attempts, two of five game launches stalled during Vulkan initialization. The other three launches completed.
manifest.json.The samples: the retained
all.txtfiles from the stalls contained 20 stack records through 59 ms (4,130 bytes) and 45 through 120 ms (12,369 bytes).Without the sampler: ten earlier windowed launches on
430eef4fand the control79d4a143had no observed stall withoutHOS_PROFILE. A different presentation mode, build cohort and other profiling instrumentation make that an uncontrolled contrast, not an estimated sampler failure rate.Unconfirmed source-level lead:
sample_all_threadperforms module lookup/name resolution andunwind_stepwhile the target remains suspended.ResumeThreadprecedeshos_sample_write, so the output writer is not inside that suspended-target interval.Claiming this: I have a fix from a local branch that hit the same crash (0xc0000005 inside
RtlVirtualUnwind, reading just past the walked thread's StackBase, fromunwind_step). The cause found there: clang puts avzeroupperbetween an epilogue's lastpopand itsretin AVX code, Windows' unwinder does not take that for an epilogue, so a thread stopped among the pops is unwound as if in the body and the whole prologue is undone again on a stack pointer the pops already moved: it reads up to the frame's size too high. The fix unwinds such an epilogue in the walk itself and works out every readRtlVirtualUnwindwill make from the frame's unwind codes before the call, as this issue suggests. A pull request with a game-free single-step test follows once it has run in the game here (Windows 11, RTX 4060).