Stop the replay and exit when the GPU device is lost #1291

Merged

berelels merged 2 commits from fix/exit-on-device-lost into main

2026-11-09 23:32:14 +00:00

berelels commented

2026-11-09 20:58:44 +00:00

(Migrated from github.com)

What

After VK_ERROR_DEVICE_LOST the replay wrote the checkpoint report and kept
running. Nothing it submitted completed again, so captures and staging were
never retired: in #1275 captures kept went from 25 to 6527 and staging from
256 MB to 1408 MB, until system RAM ran out with a black window.

The four VK_ERROR_DEVICE_LOST checks in gpu/vk/vk_replay.cpp now call a new
device_lost(), which writes the report once through std::call_once and then
ends the process through the existing replay_fatal (exit code 3). With
call_once, a second thread that sees the loss at the same time (the #1275 log
shows several) waits for the report instead of ending the process before it is
written.

This does not fix the GPU hang itself in #1275, only what happens after it.

Tests

tests/test_device_lost_exit.py (new, game-free): every
VK_ERROR_DEVICE_LOST check must call device_lost(); device_lost() is cut
into a fixture and called from five threads with a slow report, which must be
written once before the exit. It fails on the old code path and on a
check-once flag without call_once.
tests/test_parallel_init_order.py: its fixture gained a device_lost stub
(the segment-submit block it cuts now calls it); passes.
Not tested: a real device lost on hardware, and a full build (only the
changed function was compiled, in the fixture). Linux, g++.

Question

Do you want the process to stay alive after a device lost in some debugging
setup (e.g. with RX_VK_FAULT_INFO)? If so, I can put the exit behind an
opt-out variable.

Refs #1275

AI assistance

Assisted-by: Claude Opus 5.5
Assisted-by: gpt-oss-120b (LLM council)
Assisted-by: Qwen3.8-27B (LLM council)
Assisted-by: Gemini 3.5 Flash-Lite (LLM council)
Assisted-by: dots-3-note-preview (LLM council)
Assisted-by: Nemotron 3.5 Lightning (LLM council)
Assisted-by: Nemotron 3 Ultra 550B (LLM council)
Assisted-by: Nemotron 3.5 Super VL (LLM council)
Co-Authored-By: Claude Opus 5.5 [email protected]

The LLM council (several models answer the same question independently, then
rank each other's answers anonymously) was used to define the approach and the
solution: which problem to take on, how to fix it and what to avoid. It saw a
summary of the issues, not the code. The change and its test were written with
Claude Code, then reviewed and tested by me.

## What After `VK_ERROR_DEVICE_LOST` the replay wrote the checkpoint report and kept running. Nothing it submitted completed again, so captures and staging were never retired: in #1275 `captures kept` went from 25 to 6527 and staging from 256 MB to 1408 MB, until system RAM ran out with a black window. The four `VK_ERROR_DEVICE_LOST` checks in `gpu/vk/vk_replay.cpp` now call a new `device_lost()`, which writes the report once through `std::call_once` and then ends the process through the existing `replay_fatal` (exit code 3). With `call_once`, a second thread that sees the loss at the same time (the #1275 log shows several) waits for the report instead of ending the process before it is written. This does not fix the GPU hang itself in #1275, only what happens after it. ## Tests - `tests/test_device_lost_exit.py` (new, game-free): every `VK_ERROR_DEVICE_LOST` check must call `device_lost()`; `device_lost()` is cut into a fixture and called from five threads with a slow report, which must be written once before the exit. It fails on the old code path and on a check-once flag without `call_once`. - `tests/test_parallel_init_order.py`: its fixture gained a `device_lost` stub (the segment-submit block it cuts now calls it); passes. - Not tested: a real device lost on hardware, and a full build (only the changed function was compiled, in the fixture). Linux, g++. ## Question Do you want the process to stay alive after a device lost in some debugging setup (e.g. with `RX_VK_FAULT_INFO`)? If so, I can put the exit behind an opt-out variable. Refs #1275 ## AI assistance Assisted-by: Claude Opus 5.5 Assisted-by: gpt-oss-120b (LLM council) Assisted-by: Qwen3.8-27B (LLM council) Assisted-by: Gemini 3.5 Flash-Lite (LLM council) Assisted-by: dots-3-note-preview (LLM council) Assisted-by: Nemotron 3.5 Lightning (LLM council) Assisted-by: Nemotron 3 Ultra 550B (LLM council) Assisted-by: Nemotron 3.5 Super VL (LLM council) Co-Authored-By: Claude Opus 5.5 <[email protected]> The LLM council (several models answer the same question independently, then rank each other's answers anonymously) was used to define the approach and the solution: which problem to take on, how to fix it and what to avoid. It saw a summary of the issues, not the code. The change and its test were written with Claude Code, then reviewed and tested by me.

hckrman101 (Migrated from github.com) reviewed

2026-11-09 22:47:50 +00:00

hckrman101 (Migrated from github.com) left a comment

Nice find, and the call_once ordering is right. I read through the paths around it:

Locks: replay_fatal ends in std::_Exit(3), so no atexit/static destructors run and nothing takes a lock. checkpoints_report (and device_fault_report) take no present_mu and never wait on a fence/semaphore, so calling it under present_mu from queue_flush_locked can't deadlock and can't re-enter device_lost(). Worker threads (segment submit) are fine for the same reason.
Save on exit: nothing is skipped that the normal exit does. Window close is ExitProcess(0) with no settings or pipeline-cache save either, and the cache is saved every 300 presents in the present loop, so you lose at most that.
Other DEVICE_LOST returns: wait_value, the slot wait, vk_timeline_wait and the *ok wait still ignore it, and present/acquire only log. They all return immediately on a lost device, and the next submit or wait() hits one of your four checks, so I think this is covered in practice. Not blocking.

One thing I'd fix: on Windows with a console, hos_log is queued and written by a background thread. hos_fatal calls hos_log_flush() before exiting, but replay_fatal doesn't (its own comment says the queued log goes with the process). That was fine for its one-line use, but now the whole point of device_lost() is the checkpoint report, which is all logf lines, and a _Exit right after can drop them. The fprintf(stderr) in replay_fatal would still show, but the report might not. Suggest giving the Vulkan side a way to flush the host log (an extra callback in MkwVkHost, or the host's hos_log_flush) and calling it in replay_fatal before _Exit. Linux isn't affected (log_queue is a no-op there).

Smaller: with call_once, if checkpoints_report itself hangs (the comment in vk_draw.cpp says vkGetDeviceFaultInfoEXT didn't come back on NVIDIA, only with RX_VK_FAULT_INFO), every thread that sees the loss now blocks behind it, so the process stays up with a black window. Previously other threads would at least keep running. Acceptable for a debug-only env var, just be aware.

On your question: I'd keep exit-by-default; the RX_VK_FAULT_INFO case is the only one I can see wanting to stay alive, and it already reports first.

Nice find, and the call_once ordering is right. I read through the paths around it: - Locks: `replay_fatal` ends in `std::_Exit(3)`, so no atexit/static destructors run and nothing takes a lock. `checkpoints_report` (and `device_fault_report`) take no `present_mu` and never wait on a fence/semaphore, so calling it under `present_mu` from `queue_flush_locked` can't deadlock and can't re-enter `device_lost()`. Worker threads (segment submit) are fine for the same reason. - Save on exit: nothing is skipped that the normal exit does. Window close is `ExitProcess(0)` with no settings or pipeline-cache save either, and the cache is saved every 300 presents in the present loop, so you lose at most that. - Other DEVICE_LOST returns: `wait_value`, the slot wait, `vk_timeline_wait` and the `*ok` wait still ignore it, and present/acquire only log. They all return immediately on a lost device, and the next submit or `wait()` hits one of your four checks, so I think this is covered in practice. Not blocking. One thing I'd fix: on Windows with a console, `hos_log` is queued and written by a background thread. `hos_fatal` calls `hos_log_flush()` before exiting, but `replay_fatal` doesn't (its own comment says the queued log goes with the process). That was fine for its one-line use, but now the whole point of `device_lost()` is the checkpoint report, which is all `logf` lines, and a `_Exit` right after can drop them. The `fprintf(stderr)` in `replay_fatal` would still show, but the report might not. Suggest giving the Vulkan side a way to flush the host log (an extra callback in `MkwVkHost`, or the host's `hos_log_flush`) and calling it in `replay_fatal` before `_Exit`. Linux isn't affected (log_queue is a no-op there). Smaller: with `call_once`, if `checkpoints_report` itself hangs (the comment in vk_draw.cpp says `vkGetDeviceFaultInfoEXT` didn't come back on NVIDIA, only with RX_VK_FAULT_INFO), every thread that sees the loss now blocks behind it, so the process stays up with a black window. Previously other threads would at least keep running. Acceptable for a debug-only env var, just be aware. On your question: I'd keep exit-by-default; the RX_VK_FAULT_INFO case is the only one I can see wanting to stay alive, and it already reports first.

hckrman101 (Migrated from github.com) approved these changes

2026-11-09 23:23:43 +00:00

hckrman101 (Migrated from github.com) left a comment

Still open, and only a nit: std::call_once makes every other thread wait if checkpoints_report hangs (the NVIDIA RX_VK_FAULT_INFO case), so the process would never reach _Exit. A timeout or try-lock around the report would avoid that, but it is fine to leave as is.

No CI checks are reported on the branch.

Thanks, the flush fix looks right. replay_fatal now calls host->log_flush() (hos_log_flush, which takes the write lock and is safe from any thread) before the _Exit, and device_lost() goes through replay_fatal on every path, worker threads included. The API version bump, the vk_host table entry and the test fixture all match, and the new check in test_device_lost_exit.py covers the flush-before-_Exit ordering. Still open, and only a nit: std::call_once makes every other thread wait if checkpoints_report hangs (the NVIDIA RX_VK_FAULT_INFO case), so the process would never reach _Exit. A timeout or try-lock around the report would avoid that, but it is fine to leave as is. No CI checks are reported on the branch.

berelels commented

2026-11-10 00:15:04 +00:00

(Migrated from github.com)

Thanks for the careful read. Fixed in fb879db: MkwVkHost gets log_flush (hos_log_flush, API version 10), and replay_fatal calls it before _Exit; test_device_lost_exit.py checks both. On the report hanging with RX_VK_FAULT_INFO: agreed, I left it, since it only happens with that debug variable. Keeping exit-by-default as you suggest.

Thanks for the careful read. Fixed in fb879db: `MkwVkHost` gets `log_flush` (`hos_log_flush`, API version 10), and `replay_fatal` calls it before `_Exit`; `test_device_lost_exit.py` checks both. On the report hanging with `RX_VK_FAULT_INFO`: agreed, I left it, since it only happens with that debug variable. Keeping exit-by-default as you suggest.

Commenting is not possible because the repository is archived.

No reviewers

hckrman101