Game: shader preparation hard-freezes the whole PC (no TDR, no watchdog) — RX 9070 XT, Windows #462
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#462
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Longshot version
nightly-132-5e072fc8 (also reproduced on v3.0.0-rc.3)
Channel
Nightly
What kind of problem
Crash, freeze or black screen
What happened
Shader preparation does not hang the game — it hangs the whole PC. Six times, always in the
"Preparing shaders" stage, at different percentages, and every time the machine had to be
hard-reset: the desktop stops responding (mouse frozen, no Ctrl+Alt+Del), and the log stops
mid-line with no
stall, nodevice lost, nodevice fault:line, and no TDR event in theWindows event log afterwards. So
RX_VK_PREPARE_STALL_Snever gets to run: the whole OS isgone, not just the process.
System: AMD Radeon RX 9070 XT (RDNA4, 16 GB), driver 32.0.32015.12012 = Adrenalin 26.10.2
WHQL, Vulkan 1.4.359; Ryzen 7 9800X3D, 32 GB RAM, Windows 11 build 28000.
Game: program 0400486024DEC000, update v131072, build e0f3ccfa (the supported one).
Runs (all --prepare-shaders or the builder's prepare):
Run 6 matters: one background worker and no compute precompile still wedged the machine in
about 90 seconds, so this is not queue pressure or parallelism — it looks like one specific
shader on which AMD's compiler deadlocks the GPU driver, which matches the comment in
vk_programs.cpp: "Its compiler took 10 to 60 seconds on some shaders and froze the game while
it did. (and crashed on others, Lichii #200, #210)".
Nothing else is loaded: 12-17 GB of RAM free, 6 GB of VRAM free, 12+ logical CPUs idle, and the
translation is fine every time (100% of 294,522 functions).
Workaround that works: with cache\longshot_prepare_skipped present, the on-demand path is
stable — 2 h of play at 60-165 fps, 7-20 GB private memory, no freeze; "program census (play)"
shows 1,768 pipelines built in play with 288 waits totalling 24.7 s; saves load and save
normally. So the game is playable on this card, only the preparation is not.
Related: #316 (GPU hang on this same card, triggered by losing focus: capture write-back
timeout -> 251 ms slow submit -> GPU hang; my runs never switch focus, so the same driver-level
stall has a second trigger), #407 (preparation does not finish on Intel Arc), #365 (RX 6600
GPU timeout).
Request:
already quarantines programs the driver faulted on (issue 996) — could a compiler hang be
recorded the same way (write the program's key before asking the driver, so the next run can
skip it)? Even a RX_VK_SKIP_PROG-like list built from the last preparation would let me
finish a preparation by excluding the one or two bad programs.
preparation's pipelines as quick unoptimized ones only, so a wedge costs one batch instead of
the session.
I can run any instrumented build and report where it stops: the reproduction is 100% reliable
(about 1.5 minutes to a dead machine).
Where in the game
Before the game starts: the "Preparing shaders" stage (no area, no menu yet). It stops between 26% and 77% of the bar, a different percentage each run; run 6 stopped at 51.7% about 90 seconds after the start.
How often
Every time
Graphics card and driver version
AMD Radeon RX 9070 XT (RDNA4, 16 GB); Adrenalin 26.10.2 WHQL, driver 32.0.32015.12012, Vulkan 1.4.359, 224 device extensions
CPU, RAM and operating system
Ryzen 7 9800X3D (8C/16T), 32 GB, Windows 11 build 28000
Changes from the defaults
longshot_env.txt (this is what the working setup uses):
RX_VK_NOOPT=1
RX_VK_NO_PREDICT=1
RX_VK_WORKERS=1
RX_VK_PRECOMPILE_COMPUTES=none
RX_VK_VRAM_MB=6000
RX_PIPELINE_CAP=6000
RX_PIPELINE_TTL_S=180
RX_VK_CAPTURE_POOL_MB=192
RX_MEM_EVERY=600
Every preparation run used RX_VK_NOOPT=1 and RX_VK_NO_PREDICT=1; run 5 added
RX_VK_PREPARE_STALL_S=45; run 6 added RX_VK_WORKERS=1 and RX_VK_PRECOMPILE_COMPUTES=none.
No overlay, no ReShade, no shader-cache tool, no frame-generation injector: nothing is
hooked into the game. FSR 4 is the build's own (amd_fidelityfx_* from --fetch-fsr-runtime),
frame generation off, output 2560x1440, render scale 67%.
The only thing that makes the game run is cache\longshot_prepare_skipped (the preparation
skipped, pipelines built as play asks): 60-165 fps for 2 h with no freeze.
longshot_log.txt
Note: the attached longshot_prepare_frozen.log is the run that froze the machine
(run 6, --prepare-shaders, stopped at 0.517); longshot_log.txt is a normal session
that runs fine after it, for comparison.
Screenshots or video
No response
Before you send it
Update: updating the GPU driver helped, but only part of the way — and the watchdog now works
I updated the graphics driver and re-ran the exact configuration of run 6 (the one
that hard-froze the PC). The only thing that changed is the driver:
before 32.0.32015.12012, Adrenalin 26.10.2 WHQL (binary 2026-10-28)
after 32.0.32015.16001, Adrenalin 26.11.1 (binary 2026-11-11)
Everything else is unchanged: game nightly-132-5e072fc8 (not 135), same build of
longshot.exe, same settings:
RX_VK_WORKERS=1
RX_VK_PRECOMPILE_COMPUTES=none
RX_VK_NOOPT=1
RX_VK_NO_PREDICT=1
RX_VK_PREPARE_STALL_S=45
RX_VK_VRAM_MB=6000
longshot.exe --prepare-shaders
Result:
The watchdog did its job this time:
vk: preparation: waiting: nothing in the noted pipelines has finished for 30 s
(26716 of 26716), waiting on the GPU driver or the disk.
Stopped at 45 s if nothing moves
vk: compute pipelines: 686 of 4058 pipelines in 41 s, about 204 s left
vk: preparation: STOPPED: nothing moved for 45 s in the noted pipelines
(26716 of 26716): the GPU driver stopped responding. The rest is built in
play, when first drawn; updating the GPU driver may help
So the advice in that message is what fixed the hard freeze: with 26.11.1 the same
command gets through the shaders and through every noted pipeline, and only the
compute pipelines stall — and then the watchdog ends the run cleanly (exit 4, it wrote
cache\longshot_prepare_skipped) instead of taking Windows down with it.
What that leaves:
by default (vk_draw.cpp: serial only when vendorID == 0x8086).
shader stage reported 1.000, so the next start reuses it.
Two things this suggests for the code: (1) the same program-key quarantine that
longshot_driver_faults.txt already does for driver faults would make a stall like this
recoverable in one run rather than needing a rerun with the skip marker; (2) the
"updating the GPU driver may help" line is worth keeping — it is what turned a dead
machine into a caught stall here.
Second update: compute pipelines are where it stops, and serial does not save them
Same install (nightly-132-5e072fc8), same driver (32.0.32015.16001), same game files.
Only the pipeline switches changed:
RX_VK_COMPUTE_SERIAL=1 (compute pipelines one at a time, as on Intel)
RX_VK_WORKERS=1
RX_VK_PREPARE_STALL_S=30
RX_VK_NOOPT=1
RX_VK_NO_PREDICT=1
RX_VK_PRECOMPILE_COMPUTES=none
RX_VK_VRAM_MB=6000
vk: compute pipelines: 857 of 4058 pipelines in 9 s, about 35 s left
vk: compute pipelines: 1023 of 4058 pipelines in 19 s, about 58 s left
vk: compute pipelines: 1172 of 4058 pipelines in 29 s, about 73 s left
vk: preparation: STOPPED: nothing moved for 30 s in the noted pipelines
(26716 of 26716): the GPU driver stopped responding.
Serial is 1.7x further along, but it still stalls, and the machine stays alive both
times. Together with run 6 on the old driver that is four configurations — parallel,
serial, one worker, no predictions — all stopping inside the compute pipelines. It is
not concurrency: a few specific compute programs wedge AMD's compiler and it makes no
difference how many are in flight.
Two things I noticed:
Between the two runs, one start refused to precompile at all:
vk: precompile: the last two starts did not get through building the noted pipelines
into the cache: left as they are, ...
That guard is sensible, but it means two stalls use up the install's willingness to
try, so the third start cannot be used to test a new switch.
With RX_VK_PRECOMPILE_COMPUTES=none, the preparation still went on to build 4,058
compute pipelines. Is that intended — does "none" only cover the compute programs,
while the state list's compute entries are still built as listed ones?
Third update: nightly-135 completes the whole preparation on the same card
Same machine, same driver (32.0.32015.16001, Adrenalin 26.11.1), same game dump. The
only change is the build: nightly-132-5e072fc8 → nightly-135-26a2eab0, built from the
same romfs/exefs.
vk: preparation: 16 workers for the shader programs (16 CPUs usable; the segment
heap: as many as the free memory allows)
vk: one build of a program goes to the driver at a time (AMD; RX_VK_DRIVER_FAULTS=0:
as many as ask)
vk: precompile: 24453 noted pipelines of 15566 programs and 9687 compute pipelines
of 4058 compute programs built into the pipeline cache and let go
(0 and 0 not made, 0 whose program variant is not kept),
on 16 threads in 518.4 s (the compute ones 4.2 s)
So on nightly-135 everything that stopped on 132 now finishes: the shaders, all 24,453
noted pipelines, and all 9,687 compute pipelines — the compute part in 4.2 s, where 132
stalled at 686 or 1,172 of the same 4,058 compute programs. No skip marker was written,
longshot_prepared.bin is there, and the first start after it ran at 165 fps with
present intervals under 15.5 ms for every frame counted.
One more thing from the same build, for #316: the alt-tab stall is much smaller but not
gone. With the defaults this install had (frame_generation=2, frame_pacing=1, vsync
auto) losing focus gave
hid: game window not in front (foreground window: class "Chrome_WidgetWin_1")
vk: slow submit: 234.0 ms:
vk: slow submit: 215.0 ms:
Never a hang, never a device loss — the game kept running at 165 fps and the process
survived. With frame_generation=0, frame_pacing=0, vsync off, RX_VK_PRESENT_THREAD=0,
RX_VK_STAGING_IDLE_S=2, RX_VK_SPARSE_IDLE=1, RX_VK_BATCH_IDLE=1, two scripted focus
losses produced one slow submit of 15.1 ms and then none at all. So on this card the
interpolation's display thread looks like a large part of what #316 measures: the
frames it holds make the swapchain rebuild cost hundreds of milliseconds.
Thanks — 135 plus the driver update turned a machine that could not prepare at all into
one that prepares in under nine minutes.
Alt-tab stall on RX 9070 XT: interpolation + V-Sync + HDR is most of it
Related to #316 (same card). On this machine the game never hangs on focus loss any
more, but it used to stall for 200-700 ms every time the window lost the foreground,
and on the older driver the same event ended in the
capture wait for write-back ... timed out -> slow submit: 251 ms -> GPU hangchain #316 describes.What the log showed with the settings the install came with
(frame_generation=2, frame_pacing=1, vsync=auto, hdr=0):
hid: game window not in front (foreground window: class "Chrome_WidgetWin_1")
vk: slow submit: 234.0 ms:
vk: slow submit: 215.0 ms: memory sync ...
vk: slow submit: 131.2 ms: draw 177.2 commands 176.8
and the interpolation's own counters showed late frames at the same moments:
vk: frame interpolation: 463 frames shown on time, 8 late (over 2 ms);
shown apart: < 7 ms 460, 7-9 3, 9-12 1, 12-17 2, 17-25 1, over 25 4
What fixed it (all four together):
Frame generation off (frame_generation=0)
Frame pacing off (frame_pacing=0)
V-Sync off (vsync=2)
HDR off (hdr=0)
plus, in longshot_env.txt: RX_VK_PRESENT_THREAD=0, RX_VK_STAGING_IDLE_S=2,
RX_VK_SPARSE_IDLE=1, RX_VK_BATCH_IDLE=1
After that, two scripted focus losses (a Notepad window foregrounded for five seconds,
twice) gave:
first test: one slow submit of 15.1 ms (draw 14.8 descriptors 14.6 texture lookups 14.4)
second test: no slow submit at all
The game stayed at 165 fps, 17,698 frames presented with every present interval under
15.5 ms, and there was no GPU hang and no device loss at any point.
Reading: the interpolation's display thread holds two or three finished frames, so a
swapchain rebuild waits for them, and that is the 200-700 ms; V-Sync adds a queue wait
on top, and an HDR swapchain is one more thing to recreate. With all three off, losing
focus costs about one frame. If #316 is really the same mechanism taking the next step
into a hang, the interpolation display thread is worth a look first — it is the only
thing on this side that keeps frames in flight across the focus change.
Frame generation off costs nothing measurable here: the game's own 165 fps cap is what
the display gets either way, and 165 fps sustained beats 330 fps with a visible hitch
whenever the window loses focus.