fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107) #2123

2026-11-11T04:02:49Z

nullifyshot commented

2026-11-11 04:02:49 +00:00

(Migrated from github.com)

Since v2.1.0, builds can stop for good while preparing shaders (#2107). The reports come from players on a range of graphics cards, AMD and NVIDIA alike, so this isn't one vendor's or one series' bug. The Builder's details freeze on vertex format NVN 0xe -> vk 17 and nothing says why. This makes the preparation go on, or end with a reason, and keeps the log saying what it's doing.

What happens

One player's log (an RX 580) shows a likely cause:

vk: the driver faulted compiling a graphics pipeline (exception 0xc0000005): that one is left out
vk: graphics pipeline for 6ee656c480d6beea recovered with optimization disabled

guarded_graphics catches the driver's crash by jumping out of the compile, and that leaves the compile unfinished inside the driver. #1823's process dump of issue 996 shows what follows on AMD: later builds of that program wait inside the driver for ever.
The step it stops in is predict_run, which runs at build time since #2032 (before that, new builds predicted 0 pipelines). Unlike precompile_run, it had no stall guard.
It also started all its workers' first pipelines at once. precompile_run already avoids that, because of a known NVIDIA fault (621.14, RTX 4070 Ti Super).
Other reports in #2107 read as very slow rather than stuck. This PR changes nothing for those (no fault, still moving) except that the log now says what's going on.

Changes

predict_run (gpu/vk/vk_draw.cpp)

Builds its first pipeline alone, as precompile_run does.
If no pipeline finishes for 30 s (RX_VK_PREDICT_STALL_S, 0 waits for all), it goes on and leaves the stuck workers in the driver, with their state on the heap as in precompile_run. This happens:
- anywhere in the run, only after a driver fault was caught in it (driver_faults());
- otherwise only once every program was handed out, as precompile_run's guard does, so only the last stragglers are given up, never the bulk. A slow driver that still works keeps its pipelines, at the game's own start too. A real hang before then is the watchdog's to end.
An abandoned worker builds no more of its program's list.
The count now moves per pipeline, not per program, so a slow program doesn't look stuck.
predict_run and precompile_run record when they leave workers in the driver (prepare_left_workers()).

A watchdog for the whole --prepare-shaders run (gpu/vk/vk_main.cpp), covering every part on every GPU:

Live status in the log. After 30 s with no other line, it writes what it's working on with its count, or, if the count hasn't moved, what it's waiting on and for how long. That's "the GPU driver or the disk", "the GPU driver, which faulted (above)", or "the GPU driver, which still has pipelines it did not finish". A driver fault it caught gets a line of its own as soon as it happens.
Stops when nothing moves for 5 minutes (RX_VK_PREPARE_STALL_S, 0 never stops). It then:
- logs STOPPED: ... the GPU driver stopped responding
- leaves hookshot_prepare_skipped, so the game's own starts don't hang in the same place (what wasn't prepared is built when first drawn)
- ends with exit status 4 through TerminateProcess, not exit(), which would unload the stuck driver and could wait on it
Going on to the next part wouldn't help once the driver is stuck, because every part needs it. The 5-minute limit is long on purpose: reading the caches and deciding the predictions report no count for a while on a slow disk or CPU.
The pipeline cache's save, the last step:
- If workers were left in the driver, or a fault was caught: the save is stopped the same way, since vkGetPipelineCacheData can wait on them. A run that gets through ends at once with TerminateProcess (after @@done shaders), not exit().
- Otherwise: the save is only reported, and its lines stop after 300 s so that make_game's PREPARE_SILENCE (15 minutes with no line) still ends a save that never finishes.
The part, its count and the time are read under one lock.

Elsewhere

make_game.py maps exit status 4 to "Left to the game's first start: the GPU driver stopped responding". Before, players waited for PREPARE_SILENCE, 15 minutes, and the game's first start could then stop in the same place.
driver_faults(): the fault guards count what they catch. These are the same four guard blocks #1823 changes, so whichever lands second needs a mechanical rebase. After #1823, the count would follow its kDriverFault path.

Tested

tests/test_prepare_stall.py (new) compiles predict_run and the watchdog from the source with a stand-in driver. All 12 cases pass:
- predict_run:
  - every pipeline built, with the first one alone
  - the last pipelines never finish: the run goes on
  - a slow driver that still finishes pipelines, each program past the limit: no stop
  - a fault, then nothing finishes early in the run: the run goes on
  - the same with no fault: waited for
- The watchdog:
  - stuck, after a fault, and while the count moves
  - a long clean save: reported, then silent, never stopped
  - a save with workers left in the driver: stopped
  - a run ending through prepare_end
  - in each case, the lines it logs, the exit status and the skip file
- make_game.py: its message for exit status 4
vk_main.cpp and vk_draw.cpp compile with the Windows toolchain's flags (llvm-mingw, -fms-extensions), and the new functions link between them.
Updated fixtures for the new calls: test_prepare_cache_save, test_precompile_compute_guard, test_linux_driver_fault_guard.
python -I -B tests/run_all.py on Windows: 341 passed, 3 failed. test_recorder_sync, test_fpx_sticky and test_precompile_compute_guard fail the same way on an untouched copy of main on this PC (the last needs MSVC's libraries). The Linux-only checks, test_linux_driver_fault_guard among them, are left to CI.
Not tested: a real build on one of the affected cards. I couldn't reproduce the hang here (RTX 5080).

🤖 Generated with Claude Code

Since v2.1.0, builds can stop for good while preparing shaders (#2107). The reports come from players on a range of graphics cards, AMD and NVIDIA alike, so this isn't one vendor's or one series' bug. The Builder's details freeze on `vertex format NVN 0xe -> vk 17` and nothing says why. This makes the preparation go on, or end with a reason, and keeps the log saying what it's doing. ### What happens One player's log (an RX 580) shows a likely cause: ``` vk: the driver faulted compiling a graphics pipeline (exception 0xc0000005): that one is left out vk: graphics pipeline for 6ee656c480d6beea recovered with optimization disabled ``` - `guarded_graphics` catches the driver's crash by jumping out of the compile, and that leaves the compile unfinished inside the driver. #1823's process dump of issue 996 shows what follows on AMD: later builds of that program wait inside the driver for ever. - The step it stops in is `predict_run`, which runs at build time since #2032 (before that, new builds predicted 0 pipelines). Unlike `precompile_run`, it had no stall guard. - It also started all its workers' first pipelines at once. `precompile_run` already avoids that, because of a known NVIDIA fault (621.14, RTX 4070 Ti Super). - Other reports in #2107 read as very slow rather than stuck. This PR changes nothing for those (no fault, still moving) except that the log now says what's going on. ### Changes **`predict_run`** (`gpu/vk/vk_draw.cpp`) - Builds its first pipeline alone, as `precompile_run` does. - If no pipeline finishes for 30 s (`RX_VK_PREDICT_STALL_S`, 0 waits for all), it goes on and leaves the stuck workers in the driver, with their state on the heap as in `precompile_run`. This happens: - **anywhere in the run, only after a driver fault was caught in it** (`driver_faults()`); - **otherwise only once every program was handed out**, as `precompile_run`'s guard does, so only the last stragglers are given up, never the bulk. A slow driver that still works keeps its pipelines, at the game's own start too. A real hang before then is the watchdog's to end. - An abandoned worker builds no more of its program's list. - The count now moves per pipeline, not per program, so a slow program doesn't look stuck. - `predict_run` and `precompile_run` record when they leave workers in the driver (`prepare_left_workers()`). **A watchdog for the whole `--prepare-shaders` run** (`gpu/vk/vk_main.cpp`), covering every part on every GPU: - **Live status in the log.** After 30 s with no other line, it writes what it's working on with its count, or, if the count hasn't moved, what it's waiting on and for how long. That's "the GPU driver or the disk", "the GPU driver, which faulted (above)", or "the GPU driver, which still has pipelines it did not finish". A driver fault it caught gets a line of its own as soon as it happens. - **Stops when nothing moves for 5 minutes** (`RX_VK_PREPARE_STALL_S`, 0 never stops). It then: - logs `STOPPED: ... the GPU driver stopped responding` - leaves `hookshot_prepare_skipped`, so the game's own starts don't hang in the same place (what wasn't prepared is built when first drawn) - ends with exit status 4 through `TerminateProcess`, not `exit()`, which would unload the stuck driver and could wait on it Going on to the next part wouldn't help once the driver is stuck, because every part needs it. The 5-minute limit is long on purpose: reading the caches and deciding the predictions report no count for a while on a slow disk or CPU. - **The pipeline cache's save, the last step:** - **If workers were left in the driver, or a fault was caught:** the save is stopped the same way, since `vkGetPipelineCacheData` can wait on them. A run that gets through ends at once with `TerminateProcess` (after `@@done shaders`), not `exit()`. - **Otherwise:** the save is only reported, and its lines stop after 300 s so that make_game's `PREPARE_SILENCE` (15 minutes with no line) still ends a save that never finishes. - The part, its count and the time are read under one lock. **Elsewhere** - `make_game.py` maps exit status 4 to "Left to the game's first start: the GPU driver stopped responding". Before, players waited for `PREPARE_SILENCE`, 15 minutes, and the game's first start could then stop in the same place. - `driver_faults()`: the fault guards count what they catch. These are the same four guard blocks #1823 changes, so whichever lands second needs a mechanical rebase. After #1823, the count would follow its `kDriverFault` path. ### Tested - `tests/test_prepare_stall.py` (new) compiles `predict_run` and the watchdog from the source with a stand-in driver. All 12 cases pass: - **`predict_run`:** - every pipeline built, with the first one alone - the last pipelines never finish: the run goes on - a slow driver that still finishes pipelines, each program past the limit: no stop - a fault, then nothing finishes early in the run: the run goes on - the same with no fault: waited for - **The watchdog:** - stuck, after a fault, and while the count moves - a long clean save: reported, then silent, never stopped - a save with workers left in the driver: stopped - a run ending through `prepare_end` - in each case, the lines it logs, the exit status and the skip file - **`make_game.py`:** its message for exit status 4 - `vk_main.cpp` and `vk_draw.cpp` compile with the Windows toolchain's flags (llvm-mingw, `-fms-extensions`), and the new functions link between them. - Updated fixtures for the new calls: `test_prepare_cache_save`, `test_precompile_compute_guard`, `test_linux_driver_fault_guard`. - `python -I -B tests/run_all.py` on Windows: 341 passed, 3 failed. `test_recorder_sync`, `test_fpx_sticky` and `test_precompile_compute_guard` fail the same way on an untouched copy of main on this PC (the last needs MSVC's libraries). The Linux-only checks, `test_linux_driver_fault_guard` among them, are left to CI. - **Not tested:** a real build on one of the affected cards. I couldn't reproduce the hang here (RTX 5080). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hckrman101 (Migrated from github.com) reviewed

2026-11-11 04:13:14 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks for picking this up. The status lines alone are a real improvement for #2107: players can't tell "slow" from "stuck" right now. Moving the count per pipeline and building the first pipeline alone in predict_run both look right to me. Keeping the stuck workers' state on the heap and detaching them follows precompile_run's pattern and is safe as written: they only touch Pre, never the stack. Exit status 4 and its make_game message line up, and RUN_ALL_ARGS is declared.

Some things I'd like sorted before this goes in:

CI is red on Windows because of the new test. test_prepare_stall.py fails in part 2/2. clang++ targeting MSVC turns getenv (and fopen in the watchdog piece too) into -Wdeprecated-declarations errors under -Werror. Add -D_CRT_SECURE_NO_WARNINGS on Windows, as the other compiled tests do. This is the PR's own test, not one of main's known failures.
The save step can now hang forever (inline). Part 3 is never stopped, because "every pipeline is made". That's no longer true once predict_run (or precompile_run) has moved on and left a worker inside vkCreateGraphicsPipelines. If that worker holds the driver's lock or the cache's lock, vkGetPipelineCacheData in pipeline_cache_save(true) waits on it. Worse, the watchdog then logs "still writing ... this is the disk" every 30 s. That output keeps make_game's PREPARE_SILENCE from ever firing, so the 15-minute backstop this PR replaces is gone exactly in the #2107 case. The same goes for the plain return 0 → exit() after @@done, with a detached thread still in the driver (the unload hazard the description itself cites for exit()). Some ideas: let part 3 stop too when a part moved on (or a fault was counted), or after a longer cap. Stop the status lines after N minutes so the silence timeout still applies. And end the run with _Exit/TerminateProcess after the save when workers were left behind.
30 s for predict_run may catch slow drivers that still work (inline). precompile_run's guard only fires once everything has been handed out (next >= total), so it only ever drops the last stragglers. This one fires whenever nothing finishes for 30 s, wherever the run is, and gives up on everything left (tens of thousands of pipelines, which are then built in play). It also runs at the game's own start (the start screen and console paths), where there's no watchdog. In #2107, oniryuuko describes a run that was "working, just very very slow", with the progress sometimes stopping "for up to 30 seconds as it hangs on a particular pipeline". That's right at this limit, and right after the first pipeline, which the others now wait for. I'd only fire early once driver_faults() has counted something (the case the log shows). Otherwise use the tail-only rule, or a much longer limit, and leave the real hang to the watchdog in --prepare-shaders.
Overlap with #1823 (approved, open). Both change the __except / sigsetjmp blocks of guarded_graphics/guarded_compute, test_linux_driver_fault_guard.py and test_precompile_compute_guard.py, so whichever lands second needs a rebase. Two semantic points as well. #1823 makes the graphics guard return -77 like compute and adds *pl = VK_NULL_HANDLE, and its gate serializes a program's builds on AMD so that "a build behind a faulted one" doesn't hang (issue 996). That sounds a lot like the fault-then-stuck theory here, and the #2107 reporter is on an RX 580. Please say how the two fit together, e.g. whether the fault counter should also count #1823's kDriverFault path and whether its quarantine would already have kept this run from hanging.

Smaller:

Nit: a worker that's been abandoned keeps building the rest of its current program's list. abandoned is only checked at the top of the loop, so those pipelines can run alongside precompile_run and the save. A check inside the per-pipeline loop would stop it.
Nit: prepare_moved updates part/done/total as three separate atomics, so the watchdog can read a mixed snapshot. That's harmless for a log line, but worth knowing if it ever decides anything.
The description's root cause comes from one log, and nobody has reproduced it on an affected card yet ("Not tested"). Other reports in #2107 read like slowness rather than a hang. That's fine for a diagnostic change, but the 30 s default (point 3) makes it change behaviour too.

The watchdog/diagnostic part is close. Points 1–3 are what I'd want fixed first.

Thanks for picking this up. The status lines alone are a real improvement for #2107: players can't tell "slow" from "stuck" right now. Moving the count per pipeline and building the first pipeline alone in `predict_run` both look right to me. Keeping the stuck workers' state on the heap and detaching them follows `precompile_run`'s pattern and is safe as written: they only touch `Pre`, never the stack. Exit status 4 and its make_game message line up, and `RUN_ALL_ARGS` is declared. Some things I'd like sorted before this goes in: 1. **CI is red on Windows because of the new test.** `test_prepare_stall.py` fails in part 2/2. clang++ targeting MSVC turns `getenv` (and `fopen` in the watchdog piece too) into `-Wdeprecated-declarations` errors under `-Werror`. Add `-D_CRT_SECURE_NO_WARNINGS` on Windows, as the other compiled tests do. This is the PR's own test, not one of main's known failures. 2. **The save step can now hang forever (inline).** Part 3 is never stopped, because "every pipeline is made". That's no longer true once `predict_run` (or `precompile_run`) has moved on and left a worker inside `vkCreateGraphicsPipelines`. If that worker holds the driver's lock or the cache's lock, `vkGetPipelineCacheData` in `pipeline_cache_save(true)` waits on it. Worse, the watchdog then logs "still writing ... this is the disk" every 30 s. That output keeps make_game's `PREPARE_SILENCE` from ever firing, so the 15-minute backstop this PR replaces is gone exactly in the #2107 case. The same goes for the plain `return 0` → `exit()` after `@@done`, with a detached thread still in the driver (the unload hazard the description itself cites for `exit()`). Some ideas: let part 3 stop too when a part moved on (or a fault was counted), or after a longer cap. Stop the status lines after N minutes so the silence timeout still applies. And end the run with `_Exit`/`TerminateProcess` after the save when workers were left behind. 3. **30 s for `predict_run` may catch slow drivers that still work (inline).** `precompile_run`'s guard only fires once everything has been handed out (`next >= total`), so it only ever drops the last stragglers. This one fires whenever nothing finishes for 30 s, wherever the run is, and gives up on everything left (tens of thousands of pipelines, which are then built in play). It also runs at the game's own start (the start screen and console paths), where there's no watchdog. In #2107, oniryuuko describes a run that was "working, just very very slow", with the progress sometimes stopping "for up to 30 seconds as it hangs on a particular pipeline". That's right at this limit, and right after the first pipeline, which the others now wait for. I'd only fire early once `driver_faults()` has counted something (the case the log shows). Otherwise use the tail-only rule, or a much longer limit, and leave the real hang to the watchdog in `--prepare-shaders`. 4. **Overlap with #1823** (approved, open). Both change the `__except` / `sigsetjmp` blocks of `guarded_graphics`/`guarded_compute`, `test_linux_driver_fault_guard.py` and `test_precompile_compute_guard.py`, so whichever lands second needs a rebase. Two semantic points as well. #1823 makes the graphics guard return `-77` like compute and adds `*pl = VK_NULL_HANDLE`, and its gate serializes a program's builds on AMD so that "a build behind a faulted one" doesn't hang (issue 996). That sounds a lot like the fault-then-stuck theory here, and the #2107 reporter is on an RX 580. Please say how the two fit together, e.g. whether the fault counter should also count #1823's `kDriverFault` path and whether its quarantine would already have kept this run from hanging. Smaller: - Nit: a worker that's been abandoned keeps building the rest of its current program's list. `abandoned` is only checked at the top of the loop, so those pipelines can run alongside `precompile_run` and the save. A check inside the per-pipeline loop would stop it. - Nit: `prepare_moved` updates part/done/total as three separate atomics, so the watchdog can read a mixed snapshot. That's harmless for a log line, but worth knowing if it ever decides anything. - The description's root cause comes from one log, and nobody has reproduced it on an affected card yet ("Not tested"). Other reports in #2107 read like slowness rather than a hang. That's fine for a diagnostic change, but the 30 s default (point 3) makes it change behaviour too. The watchdog/diagnostic part is close. Points 1–3 are what I'd want fixed first.

gpu/vk/vk_draw.cpp

Unlike precompile_run's guard (next.load() >= total &&), this fires anywhere in the run and gives up on all that's left, here and at the game's own start (no watchdog there). With the first pipeline built alone, a single 30 s compile (which #2107 reports describe on slow-but-working runs) is enough. I'd fire early only once driver_faults() > 0, and otherwise keep the tail-only rule or a much longer limit.

Unlike `precompile_run`'s guard (`next.load() >= total &&`), this fires anywhere in the run and gives up on all that's left, here and at the game's own start (no watchdog there). With the first pipeline built alone, a single 30 s compile (which #2107 reports describe on slow-but-working runs) is enough. I'd fire early only once `driver_faults() > 0`, and otherwise keep the tail-only rule or a much longer limit.

gpu/vk/vk_main.cpp

					
				@ -727,6 +733,117 @@ std::string skipped_path() {

				    std::string path(exe, n);

These lines reset make_game's silence timer, so if the save (or the exit() after @@done) hangs, the build waits forever. Maybe stop emitting after a few minutes in part 3, or end the run here too when driver_faults() or a moved-on part says the driver may be stuck.

These lines reset make_game's silence timer, so if the save (or the `exit()` after `@@done`) hangs, the build waits forever. Maybe stop emitting after a few minutes in part 3, or end the run here too when `driver_faults()` or a moved-on part says the driver may be stuck.

gpu/vk/vk_main.cpp

"Every pipeline is made" isn't true when predict/precompile moved on and left a worker inside vkCreate*Pipelines. vkGetPipelineCacheData on the same cache can then wait on it forever. Because part 3 keeps logging "still writing", make_game's 15 min PREPARE_SILENCE never fires either. I think the save needs a cap, or a stop, when workers were left behind.

"Every pipeline is made" isn't true when predict/precompile moved on and left a worker inside `vkCreate*Pipelines`. `vkGetPipelineCacheData` on the same cache can then wait on it forever. Because part 3 keeps logging "still writing", make_game's 15 min `PREPARE_SILENCE` never fires either. I think the save needs a cap, or a stop, when workers were left behind.

tests/test_prepare_stall.py

					
				@ -0,0 +1,359 @@

				# SPDX-License-Identifier: GPL-3.0-or-later

This is what fails on windows-2022 (part 2/2): getenv is deprecated under MSVC's CRT, and with -Werror that's an error (fopen in the watchdog piece would hit the same). Add -D_CRT_SECURE_NO_WARNINGS when os.name == "nt".

This is what fails on windows-2022 (part 2/2): `getenv` is deprecated under MSVC's CRT, and with `-Werror` that's an error (`fopen` in the watchdog piece would hit the same). Add `-D_CRT_SECURE_NO_WARNINGS` when `os.name == "nt"`.

nullifyshot commented

2026-11-11 04:32:26 +00:00

(Migrated from github.com)

Thanks for the careful review, hckrman101. All of points 1–3 and both nits are in c69b5a6.

CI: the test compiles with -D_CRT_SECURE_NO_WARNINGS, as the other compiled checks do.
The save: predict_run and precompile_run now record when they go on and leave workers in the driver (prepare_left_workers()).
- If workers were left, or a driver fault was counted: the watchdog stops the save as it does the other parts (exit status 4, with the skip file). A run that gets through the save ends with TerminateProcess (prepare_end), not with return → exit(). It prints @@done shaders first, as the host would.
- A clean run: the save is only reported. Its status lines stop after 300 s, so PREPARE_SILENCE still ends a save that never finishes.
The 30 s in predict_run: it now goes on from anywhere in the run only if driver_faults() rose during that run. Otherwise it uses precompile_run's tail-only rule (next >= keys.size()), so a slow driver that still works keeps its pipelines, at the game's own start too. A real hang before the tail is left to the watchdog. The test has a case for each: a fault and then a hang early in the run goes on, the same hang with no fault is waited for, and a slow program (1.5 s, past a 1 s limit) isn't given up.

Nits:

An abandoned worker checks abandoned before each pipeline of its list.
The watchdog now reads part, count and time under one lock (PrepareSeen).

#1823: I read its dump, and it describes the same mechanism. After a caught fault, AMD's driver leaves that program's compile unfinished, and later builds of the program wait inside the driver for ever. The RX 580 log in #2107 fits that, so I've reworded the code comment to point at it instead of a general "lock". As I see it, the two are complementary:
- #1823 prevents the known cause on AMD: one build of a program at a time, and a faulted program is never asked of the driver again.
- This PR is the backstop and the diagnostics for what that doesn't cover: other drivers (where #1823's gate is off), other causes, and saying what the run is doing either way.
- Would #1823 alone have prevented this run's hang? I'm not sure it would. In the #2107 log the faulted program then logs recovered with optimization disabled. As I read #1823's build_graphics, that AMD retry runs before fault_quarantine().end(p.key, r == kDriverFault), so r is VK_SUCCESS by then and the program isn't quarantined. Later builds of it would go back to the driver. It may be worth quarantining on the first fault even when the retry recovers, but that's #1823's call.
- The fault counter: yes, it should follow #1823's kDriverFault path. My note_driver_fault() calls sit in the same four guard blocks, next to #1823's *pl = VK_NULL_HANDLE and return (VkResult)-77, and so do the two test stubs. The conflict is mechanical. Since #1823 is approved, I'd let it land first and rebase this onto it.

On the evidence: you're right that the root cause comes from one log, and that other reports read as slowness. With point 3 changed, the only behaviour change without a counted fault is the watchdog's 5-minute stop in --prepare-shaders. Slow-but-working runs now get status lines and otherwise run as before.

Locally: test_prepare_stall passes all 12 cases. Full suite on Windows: 341 passed, 3 failed. The 3 (test_recorder_sync, test_fpx_sticky, test_precompile_compute_guard) fail the same way on an untouched copy of main on this PC.

Thanks for the careful review, hckrman101. All of points 1–3 and both nits are in c69b5a6. 1. **CI:** the test compiles with `-D_CRT_SECURE_NO_WARNINGS`, as the other compiled checks do. 2. **The save:** `predict_run` and `precompile_run` now record when they go on and leave workers in the driver (`prepare_left_workers()`). - **If workers were left, or a driver fault was counted:** the watchdog stops the save as it does the other parts (exit status 4, with the skip file). A run that gets through the save ends with `TerminateProcess` (`prepare_end`), not with `return` → `exit()`. It prints `@@done shaders` first, as the host would. - **A clean run:** the save is only reported. Its status lines stop after 300 s, so `PREPARE_SILENCE` still ends a save that never finishes. 3. **The 30 s in `predict_run`:** it now goes on from anywhere in the run only if `driver_faults()` rose during that run. Otherwise it uses `precompile_run`'s tail-only rule (`next >= keys.size()`), so a slow driver that still works keeps its pipelines, at the game's own start too. A real hang before the tail is left to the watchdog. The test has a case for each: a fault and then a hang early in the run goes on, the same hang with no fault is waited for, and a slow program (1.5 s, past a 1 s limit) isn't given up. **Nits:** - An abandoned worker checks `abandoned` before each pipeline of its list. - The watchdog now reads part, count and time under one lock (`PrepareSeen`). 4. **#1823:** I read its dump, and it describes the same mechanism. After a caught fault, AMD's driver leaves that program's compile unfinished, and later builds of the program wait inside the driver for ever. The RX 580 log in #2107 fits that, so I've reworded the code comment to point at it instead of a general "lock". As I see it, the two are complementary: - **#1823 prevents the known cause on AMD:** one build of a program at a time, and a faulted program is never asked of the driver again. - **This PR is the backstop and the diagnostics** for what that doesn't cover: other drivers (where #1823's gate is off), other causes, and saying what the run is doing either way. - **Would #1823 alone have prevented this run's hang?** I'm not sure it would. In the #2107 log the faulted program then logs `recovered with optimization disabled`. As I read #1823's `build_graphics`, that AMD retry runs before `fault_quarantine().end(p.key, r == kDriverFault)`, so `r` is `VK_SUCCESS` by then and the program isn't quarantined. Later builds of it would go back to the driver. It may be worth quarantining on the first fault even when the retry recovers, but that's #1823's call. - **The fault counter:** yes, it should follow #1823's `kDriverFault` path. My `note_driver_fault()` calls sit in the same four guard blocks, next to #1823's `*pl = VK_NULL_HANDLE` and `return (VkResult)-77`, and so do the two test stubs. The conflict is mechanical. Since #1823 is approved, I'd let it land first and rebase this onto it. **On the evidence:** you're right that the root cause comes from one log, and that other reports read as slowness. With point 3 changed, the only behaviour change without a counted fault is the watchdog's 5-minute stop in `--prepare-shaders`. Slow-but-working runs now get status lines and otherwise run as before. Locally: `test_prepare_stall` passes all 12 cases. Full suite on Windows: 341 passed, 3 failed. The 3 (`test_recorder_sync`, `test_fpx_sticky`, `test_precompile_compute_guard`) fail the same way on an untouched copy of main on this PC.

nullifyshot (Migrated from github.com) reviewed

2026-11-11 04:32:37 +00:00

gpu/vk/vk_draw.cpp

Done in c69b5a6. It goes on from anywhere in the run only if driver_faults() rose during this run. Otherwise it uses the tail-only rule, as precompile_run does (next >= keys.size()). test_prepare_stall.py checks that a hang early in the run with no fault is waited for.

Done in c69b5a6. It goes on from anywhere in the run only if `driver_faults()` rose during this run. Otherwise it uses the tail-only rule, as `precompile_run` does (`next >= keys.size()`). `test_prepare_stall.py` checks that a hang early in the run with no fault is waited for.

nullifyshot (Migrated from github.com) reviewed

2026-11-11 04:32:39 +00:00

gpu/vk/vk_main.cpp

Agreed. predict_run and precompile_run now call prepare_left_workers_note() when they go on. When that's set, or a fault was counted, the watchdog stops the save the same way it stops the other parts.

Agreed. `predict_run` and `precompile_run` now call `prepare_left_workers_note()` when they go on. When that's set, or a fault was counted, the watchdog stops the save the same way it stops the other parts.

nullifyshot (Migrated from github.com) reviewed

2026-11-11 04:32:40 +00:00

gpu/vk/vk_main.cpp

					
				@ -727,6 +733,117 @@ std::string skipped_path() {

				    std::string path(exe, n);

Both are handled in c69b5a6. In a clean run, the save's status lines stop after 300 s, so PREPARE_SILENCE applies again. In a risky run (workers left, or a fault), the save is stopped at 300 s, and a run that gets through ends with TerminateProcess (prepare_end), not exit().

Both are handled in c69b5a6. In a clean run, the save's status lines stop after 300 s, so `PREPARE_SILENCE` applies again. In a risky run (workers left, or a fault), the save is stopped at 300 s, and a run that gets through ends with `TerminateProcess` (`prepare_end`), not `exit()`.

nullifyshot (Migrated from github.com) reviewed

2026-11-11 04:32:42 +00:00

tests/test_prepare_stall.py

					
				@ -0,0 +1,359 @@

				# SPDX-License-Identifier: GPL-3.0-or-later

Fixed: -D_CRT_SECURE_NO_WARNINGS is added to the compile command, as in the other compiled checks.

Fixed: `-D_CRT_SECURE_NO_WARNINGS` is added to the compile command, as in the other compiled checks.

nullifyshot commented

2026-11-11 07:02:29 +00:00

(Migrated from github.com)

Closing in favour of #2195.

#2186 landed on main in the meantime. It has the preparation's pipeline steps log their progress every 10 s, which covers the "is it slow or stuck?" half of this PR and conflicted with it. #2195 is this work rebased onto #2186, with two changes on top:

Dropped: this PR's "still working on …" lines for the pipeline steps. #2186's lines report that better, with time left.
Changed: the "waiting on the GPU driver …" line now comes every 30 s that nothing has moved. It used to wait for 30 s of a quiet log, and #2186's lines, which repeat the same count during a hang, would never have let it appear.

Everything else carries over unchanged, including the changes from hckrman101's review here:

predict_run's first pipeline built alone
the stall guard: anywhere in the run only after a driver fault, otherwise the tail-only rule
the 5-minute stop, with its reason, skip file and exit status 4
the cache-save handling, and ending through TerminateProcess when workers were left
the driver-fault lines

A new PR keeps this one's review and history readable as they were, instead of force-pushing over them.

Closing in favour of #2195. #2186 landed on main in the meantime. It has the preparation's pipeline steps log their progress every 10 s, which covers the "is it slow or stuck?" half of this PR and conflicted with it. #2195 is this work rebased onto #2186, with two changes on top: - **Dropped:** this PR's "still working on …" lines for the pipeline steps. #2186's lines report that better, with time left. - **Changed:** the "waiting on the GPU driver …" line now comes every 30 s that nothing has moved. It used to wait for 30 s of a quiet log, and #2186's lines, which repeat the same count during a hang, would never have let it appear. Everything else carries over unchanged, including the changes from hckrman101's review here: - `predict_run`'s first pipeline built alone - the stall guard: anywhere in the run only after a driver fault, otherwise the tail-only rule - the 5-minute stop, with its reason, skip file and exit status 4 - the cache-save handling, and ending through `TerminateProcess` when workers were left - the driver-fault lines A new PR keeps this one's review and history readable as they were, instead of force-pushing over them.

Commenting is not possible because the repository is archived.

No reviewers