fix(vk): automatically query VK_EXT_device_fault on device lost when NV checkpoints and buffer markers are absent #2117

Open

Cloudiny wants to merge 1 commit from Cloudiny/vk/auto-device-fault-report into main

Cloudiny commented

2026-11-11 03:36:48 +00:00

(Migrated from github.com)

Summary

In gpu/vk/vk_draw.cpp (checkpoints_report()), line 3613 logs "vk: device lost: reading the fault info (RX_VK_CHECKPOINTS=1 for markers)" when !dev.checkpoints && !marker_words, yet FaultLast at line 3614 still gated device_fault_report() solely on std::getenv("RX_VK_FAULT_INFO"). That guard exists because on NVIDIA (dev.checkpoints == true), vkGetDeviceFaultInfoEXT can hang before checkpoint data finishes reporting.

When both dev.checkpoints and marker_words are false (typical on AMD RADV / Intel when RX_VK_CHECKPOINTS=1 is not set), invoking device_fault_report() automatically captures VK_EXT_device_fault addresses and vendor fault info on the first crash without needing a manual reproduction run with RX_VK_FAULT_INFO=1.

NVIDIA & Cross-Platform Compatibility

Zero impact on NVIDIA (dev.checkpoints == true) or when buffer markers are active (marker_words != nullptr): device_fault_report() remains gated on RX_VK_FAULT_INFO, preserving the NVIDIA hang guard.

Files Changed

gpu/vk/vk_draw.cpp

Test Plan / Verification

Verified clean git apply --check against main (ca082bf).
Syntax-checked gpu/vk/vk_draw.cpp (clang++ -fsyntax-only -std=gnu++20 -Wall -Wextra -Wno-missing-field-initializers -Wno-c++11-narrowing -DVK_NO_PROTOTYPES): 0 errors, 0 warnings in modified lines.

## Summary In `gpu/vk/vk_draw.cpp` (`checkpoints_report()`), line 3613 logs `"vk: device lost: reading the fault info (RX_VK_CHECKPOINTS=1 for markers)"` when `!dev.checkpoints && !marker_words`, yet `FaultLast` at line 3614 still gated `device_fault_report()` solely on `std::getenv("RX_VK_FAULT_INFO")`. That guard exists because on NVIDIA (`dev.checkpoints == true`), `vkGetDeviceFaultInfoEXT` can hang before checkpoint data finishes reporting. When both `dev.checkpoints` and `marker_words` are false (typical on AMD RADV / Intel when `RX_VK_CHECKPOINTS=1` is not set), invoking `device_fault_report()` automatically captures `VK_EXT_device_fault` addresses and vendor fault info on the first crash without needing a manual reproduction run with `RX_VK_FAULT_INFO=1`. ## NVIDIA & Cross-Platform Compatibility - **Zero impact on NVIDIA (`dev.checkpoints == true`) or when buffer markers are active (`marker_words != nullptr`):** `device_fault_report()` remains gated on `RX_VK_FAULT_INFO`, preserving the NVIDIA hang guard. ## Files Changed - `gpu/vk/vk_draw.cpp` ## Test Plan / Verification - [x] Verified clean `git apply --check` against `main` (`ca082bf`). - [x] Syntax-checked `gpu/vk/vk_draw.cpp` (`clang++ -fsyntax-only -std=gnu++20 -Wall -Wextra -Wno-missing-field-initializers -Wno-c++11-narrowing -DVK_NO_PROTOTYPES`): 0 errors, 0 warnings in modified lines.

Joey-Magnus (Migrated from github.com) reviewed

2026-11-11 20:06:35 +00:00

Joey-Magnus (Migrated from github.com) left a comment

should fix gpu/vk/vk_draw.cpp:3628: !dev.checkpoints && !marker_words is not the same as "not NVIDIA". dev.checkpoints is on by default, but RX_VK_CHECKPOINTS=0 turns it off (gpu/vk/vk_main.cpp:272, documented in docs/development.md as the way to run without checkpoints). On NVIDIA with that switch set, this change calls vkGetDeviceFaultInfoEXT automatically, which is exactly the call the comment four lines above says has not come back after a fault on NVIDIA. That turns a crash report into a hang for someone who is already debugging a lost device. Gating on the vendor, the way the rest of the renderer does, keeps the NVIDIA guard whatever the switches are:

struct FaultLast { ~FaultLast() {
    if ((dev.props.vendorID != 0x10DE && !dev.checkpoints && !marker_words) || env_on("RX_VK_FAULT_INFO")) device_fault_report();
} } fault_last;

should fix (merge conflict): the PR conflicts with main in gpu/vk/vk_draw.cpp. #2042 replaced std::getenv("RX_VK_FAULT_INFO") with env_on(...), and main now also calls device_fault_report() early, at the top of checkpoints_report(), when RX_VK_FAULT_INFO is set. After rebasing, please use env_on as above, so RX_VK_FAULT_INFO=0 means off like the other switches.

minor: the PR hasn't been tested on a real lost device. One is hard to provoke, so I won't insist, but one line from an AMD or Intel log with the new vk: device fault: ... output (or VK_EXT_device_fault not available where the driver lacks the extension) would show the path works end to end.

Reading the fault info by default when there is nothing else to read is a good idea: on AMD and Intel a first crash would then log the faulting address without a second run. Two things before it can go in. **should fix** `gpu/vk/vk_draw.cpp:3628`: `!dev.checkpoints && !marker_words` is not the same as "not NVIDIA". `dev.checkpoints` is on by default, but `RX_VK_CHECKPOINTS=0` turns it off (`gpu/vk/vk_main.cpp:272`, documented in `docs/development.md` as the way to run without checkpoints). On NVIDIA with that switch set, this change calls `vkGetDeviceFaultInfoEXT` automatically, which is exactly the call the comment four lines above says has not come back after a fault on NVIDIA. That turns a crash report into a hang for someone who is already debugging a lost device. Gating on the vendor, the way the rest of the renderer does, keeps the NVIDIA guard whatever the switches are: ```cpp struct FaultLast { ~FaultLast() { if ((dev.props.vendorID != 0x10DE && !dev.checkpoints && !marker_words) || env_on("RX_VK_FAULT_INFO")) device_fault_report(); } } fault_last; ``` **should fix** (merge conflict): the PR conflicts with main in `gpu/vk/vk_draw.cpp`. #2042 replaced `std::getenv("RX_VK_FAULT_INFO")` with `env_on(...)`, and main now also calls `device_fault_report()` early, at the top of `checkpoints_report()`, when `RX_VK_FAULT_INFO` is set. After rebasing, please use `env_on` as above, so `RX_VK_FAULT_INFO=0` means off like the other switches. **minor**: the PR hasn't been tested on a real lost device. One is hard to provoke, so I won't insist, but one line from an AMD or Intel log with the new `vk: device fault: ...` output (or `VK_EXT_device_fault not available` where the driver lacks the extension) would show the path works end to end.

Joey-Magnus (Migrated from github.com) requested changes

2026-11-11 21:23:59 +00:00

Joey-Magnus (Migrated from github.com) left a comment

The safety claim doesn't hold. With RX_VK_CHECKPOINTS=0 (vk_main.cpp:282-283), NVIDIA has neither dev.checkpoints nor marker_words, so this would call vkGetDeviceFaultInfoEXT by default on the driver the comment says it hangs on. Keying it on vendorID != 0x10DE would be safer.

Also:

dev.device_fault isn't combined with fault.deviceFault after the feature query (vk_main.cpp:369-370). That gap predates this PR, but it matters once the call is on by default.
It conflicts with main: env_on("RX_VK_FAULT_INFO") at vk_draw.cpp:3776, and checkpoints_report() now already calls device_fault_report() once at :3767.

The safety claim doesn't hold. With `RX_VK_CHECKPOINTS=0` (`vk_main.cpp:282-283`), NVIDIA has neither `dev.checkpoints` nor `marker_words`, so this would call `vkGetDeviceFaultInfoEXT` by default on the driver the comment says it hangs on. Keying it on vendorID != 0x10DE would be safer. Also: - `dev.device_fault` isn't combined with `fault.deviceFault` after the feature query (`vk_main.cpp:369-370`). That gap predates this PR, but it matters once the call is on by default. - It conflicts with main: `env_on("RX_VK_FAULT_INFO")` at `vk_draw.cpp:3776`, and `checkpoints_report()` now already calls `device_fault_report()` once at `:3767`.

hckrman101 commented

2026-11-11 22:55:45 +00:00

(Migrated from github.com)

This now conflicts with main in gpu/vk/vk_draw.cpp. Could you merge main into the branch (or rebase) when you get a chance? I'll take another look once it's up to date.

This now conflicts with main in `gpu/vk/vk_draw.cpp`. Could you merge main into the branch (or rebase) when you get a chance? I'll take another look once it's up to date.