vk: RX_VK_CHECKPOINTS on AMD and Intel (VK_AMD_buffer_marker), so GPU hang reports name the program #734

Merged

SkippySteve merged 1 commit from feat/amd-buffer-markers into main

2026-11-08 20:17:01 +00:00

SkippySteve commented

2026-11-08 19:08:06 +00:00

(Migrated from github.com)

Refs #602, #604, #620, #581 (AMD GPU hangs after the intro; on Linux, a failed ring reset leads to a full GPU reset, and sometimes a reboot).

The gap

When the device is lost, the log says device lost: reading the fault info (RX_VK_CHECKPOINTS=1 for markers). But those markers are VK_NV_device_diagnostic_checkpoints only. On AMD, with both RADV and the Windows driver, the switch does nothing, and VK_EXT_device_fault isn't available either (#604, #602). So none of the AMD hang reports can say which program the GPU was running. The most we have is "it goes away with RX_VK_NO_MESH=1" (#602, RX 9070 on v2.0.2, which already has the 1024-attempt compare-exchange ceiling).

What this does

When RX_VK_CHECKPOINTS is set (not 0 or ext), the NV extension is missing, and VK_AMD_buffer_marker is present (RADV, the AMD Windows driver, and Intel ANV all have it), each existing checkpoint (checkpoint() and checkpoint_mark(), so every draw, dispatch, upload, mirror sync and write-back) writes its sequence number into a 512 KB host-memory buffer twice:

at TOP_OF_PIPE, when the GPU reaches it (word 2i);
at BOTTOM_OF_PIPE, once all the work before it has finished (word 2i + 1).

The buffer is in system memory if the driver offers it, because a MODE1 reset loses VRAM. After a device loss, checkpoints_report() logs:

vk: device lost: reading the buffer markers
vk: device lost: N markers set, R reached; the last whose earlier work all finished: S
vk:   finished before: marker S-6..S-1 (...)
vk:   RUNNING after: marker S (draw of program XXXXXXXXXXXXXXXX, frame F)   <- the work the GPU never finished
vk:   running: marker S+1 ...
vk:   reached, earlier work unfinished: marker ...

followed by the same 40-line watch ring the NV report prints.

It only turns on when asked for: markers twice per draw are not free, and on NVIDIA the checkpoints are known to change timing (vk_main.cpp's comment). The NV path is unchanged apart from sharing the kind names.

Testing

hookshot_vk.so builds with the AppImage toolchain (RX_VK_ONLY=1 nx2/build.py --standalone) with no warnings, and tests/run_all.py passes (88 passed, 4 skipped).
Not run on AMD hardware: I only have an NVIDIA card, which doesn't expose VK_AMD_buffer_marker. The reporting logic was checked by hand: with markers 1-100 recorded, the GPU done through 57 and reached through 60, it names marker 57's draw as the one running and lists 58-60 as reached with earlier work unfinished.
Markers are numbered when commands are recorded, so with RX_VK_PARALLEL the order can interleave across command buffers. The "reached, earlier work unfinished" list is printed for that case.

If someone with an affected AMD card could run once with RX_VK_CHECKPOINTS=1 in hookshot_env.txt and attach the log, the "RUNNING after" line should name the hanging program.

Refs #602, #604, #620, #581 (AMD GPU hangs after the intro; on Linux, a failed ring reset leads to a full GPU reset, and sometimes a reboot). ## The gap When the device is lost, the log says `device lost: reading the fault info (RX_VK_CHECKPOINTS=1 for markers)`. But those markers are `VK_NV_device_diagnostic_checkpoints` only. On AMD, with both RADV and the Windows driver, the switch does nothing, and `VK_EXT_device_fault` isn't available either (#604, #602). So none of the AMD hang reports can say which program the GPU was running. The most we have is "it goes away with `RX_VK_NO_MESH=1`" (#602, RX 9070 on v2.0.2, which already has the 1024-attempt compare-exchange ceiling). ## What this does When `RX_VK_CHECKPOINTS` is set (not `0` or `ext`), the NV extension is missing, and `VK_AMD_buffer_marker` is present (RADV, the AMD Windows driver, and Intel ANV all have it), each existing checkpoint (`checkpoint()` and `checkpoint_mark()`, so every draw, dispatch, upload, mirror sync and write-back) writes its sequence number into a 512 KB host-memory buffer twice: - at `TOP_OF_PIPE`, when the GPU reaches it (word `2i`); - at `BOTTOM_OF_PIPE`, once all the work before it has finished (word `2i + 1`). The buffer is in system memory if the driver offers it, because a MODE1 reset loses VRAM. After a device loss, `checkpoints_report()` logs: ``` vk: device lost: reading the buffer markers vk: device lost: N markers set, R reached; the last whose earlier work all finished: S vk: finished before: marker S-6..S-1 (...) vk: RUNNING after: marker S (draw of program XXXXXXXXXXXXXXXX, frame F) <- the work the GPU never finished vk: running: marker S+1 ... vk: reached, earlier work unfinished: marker ... ``` followed by the same 40-line watch ring the NV report prints. It only turns on when asked for: markers twice per draw are not free, and on NVIDIA the checkpoints are known to change timing (vk_main.cpp's comment). The NV path is unchanged apart from sharing the kind names. ## Testing - `hookshot_vk.so` builds with the AppImage toolchain (`RX_VK_ONLY=1 nx2/build.py --standalone`) with no warnings, and `tests/run_all.py` passes (88 passed, 4 skipped). - **Not run on AMD hardware**: I only have an NVIDIA card, which doesn't expose `VK_AMD_buffer_marker`. The reporting logic was checked by hand: with markers 1-100 recorded, the GPU done through 57 and reached through 60, it names marker 57's draw as the one running and lists 58-60 as reached with earlier work unfinished. - Markers are numbered when commands are recorded, so with `RX_VK_PARALLEL` the order can interleave across command buffers. The "reached, earlier work unfinished" list is printed for that case. If someone with an affected AMD card could run once with `RX_VK_CHECKPOINTS=1` in `hookshot_env.txt` and attach the log, the "RUNNING after" line should name the hanging program.

shayderrr commented

2026-11-08 20:16:56 +00:00

(Migrated from github.com)