MagicBeanDev/Longshot

Watch

Fork 98

WIP: fix(vk): a marker before every GPU command on AMD's integrated GPUs, so Kokiri Forest no longer loses the device #466

Closed

Yeetbot90 wants to merge 1 commit from Yeetbot90/Longshot:fix/amd-igpu-markers into main

Yeetbot90 commented

2026-11-14 15:40:19 +00:00

Contributor

Why

On an AMD integrated GPU (a Ryzen 9 7900X's Radeon Graphics, Windows) the game loses the GPU device (VK_ERROR_DEVICE_LOST, then FATAL) within seconds of Kokiri Forest loading, every time. With a marker written before every GPU command, as NVIDIA's checkpoints already are by default for the same kind of loss, it kept the device for 110 minutes.

What changed

What the GPU was doing when it was lost: with markers it stood still for 4-6 s (nvn: GPU waits in 5 s: 1 sync waits (4176-6305 ms)) before Windows reset it. Every marker it had reached had its earlier work finished, so it was not inside any draw, dispatch, upload, mirror sync, write-back or present copy: it stood between commands. Turning off each suspect in turn did not keep the device (Measurements). Markers before every command did. Each one waits for the work before it, so like the NV checkpoints (vk_main.cpp: "The race they hide is not found yet") they hide a race that is still to be found; this PR does not claim to have found it.

markers: gpu/vk/vk_work.h. The work macros that are not draws or dispatches (barriers, render pass begin/end, copies, blits, resolves, clears, fills, updates, queries) call fine_mark before the command, through a lambda that takes the macro's arguments once and passes them on, so no argument is evaluated twice. fine_mark does nothing unless dev.fine_markers.
default: create_device() in gpu/vk/vk_main.cpp. On AMD integrated GPUs on Windows (vendorID 0x1002, VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU) VK_AMD_buffer_marker is enabled and dev.fine_markers set with no switch. RX_VK_CHECKPOINTS=all asks for the same anywhere the NV checkpoints are missing, RX_VK_CHECKPOINTS=1 keeps them to draws and dispatches, RX_VK_CHECKPOINTS=0 turns them off.
report: buffer_markers_report() in gpu/vk/vk_draw.cpp. Each marker now keeps its command buffer, and after a lost device the log lists, for each command buffer, its last markers done and its first not reached. Before, an upload command buffer's markers interleaved in number with its recording's draws, and the single "RUNNING after" line could name the wrong command. New marker kinds name the commands, a recording's start (begin_recording in gpu/vk/vk_replay.cpp) and the present's copy (present_image).
tests: tests/test_recorder_sync.py accepts the lambda form (the packer still called, the driver never directly); tests/test_inline_write_order.py stubs fine_mark and removes the work count by the macro's start, so its negative control still fails; tests/fsr_uniform_ownership_test.cpp and tests/recorder_sync_test.cpp stub checkpoint_mark and checkpoint_fine.
docs: docs/development.md, RX_VK_CHECKPOINTS=1 / all.
Deliberately unchanged: every other GPU (dedicated AMD cards, Intel, NVIDIA, Linux and the Steam Deck). Nothing runs there unless RX_VK_CHECKPOINTS asks for it.

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
`RX_VK_CHECKPOINTS=all` (new value)	`VK_AMD_buffer_marker` before every command, not only draws and dispatches; the lost-device report by command buffer	On for AMD integrated GPUs on Windows; off elsewhere	On there: without it Kokiri Forest loses the device every run	Ryzen 9 7900X Radeon Graphics (RDNA 2, 2 CUs), Windows	Not needed on dedicated cards: off there	Other AMD APUs (680M, 780M, 890M), Linux APUs
`RX_VK_CHECKPOINTS=0`	Turns those markers off (and the NV checkpoints, as before)	Unset	For A/B tests only	-	-	-

Measurements

main: 5e072fc8 (2026-11-14)
this PR: fix/amd-igpu-markers @ 0cb75e95
Baseline: main @ 5e072fc8. In the game its side is e072164e's renderer without these markers: the game folder on the iGPU was built from e072164e (#404's branch: main @ 0caf0618 plus two Intel commits), and each run below swapped only its longshot_vk.dll, built from that source with the switch or change named. A renderer from 5e072fc8 changes the shader translator and would restart the shader preparation each time. The renderer files this PR touches are the same on both bases. Without markers, or with RX_VK_CHECKPOINTS=1, is main's behaviour on this GPU.
Hardware: Ryzen 9 7900X's AMD Radeon(TM) Graphics (RDNA 2, 2 CUs), AMD proprietary driver 26.10.1 (Vulkan driver 0x800161), no mesh shaders (the compute fallback), Windows 11; the screen on an RX 9070 XT, so the iGPU's frames are presented across adapters
Scene: New Game, then Kokiri Forest (map: KokiriForest_Child); output resolution 2048x1152
Method: one run each, played by hand until the device was lost or the run was stopped; the lines from each run's longshot_log.txt. The checks: tests/run_all.py on main and on this PR
How to reproduce: on an AMD APU on Windows, walk into Kokiri Forest on main (or with RX_VK_CHECKPOINTS=1), then with this PR and no switch

The runs on the iGPU, each a renderer built from e072164e (frame and submit where the device was lost; where the GPU stood, the last marker whose earlier work was done):

No markers (main's behaviour): lost seconds after Kokiri Forest loaded, about frame 1109, submit 7773.
RX_VK_CHECKPOINTS=1: lost seconds after, frame 794, submit 5574; after a dispatch of 53fcaee4dc45c6ee.
=1 with RX_VK_SKIP_PROG=53fcaee4dc45c6ee,53509044401d78d2: lost, frame 3678, submit 25762; after a dispatch of 0ba2f076e4a94acf.
=1 with RX_VK_MIRROR=1: lost, frame 2463, submit 17257; after a pool's write-back.
=1 with the swapchain image waited for on the CPU (not in this PR): lost, frame 853, submit 5980; after a texture upload.
=1 with RX_VK_SKIP_CODE=2c56dc1006942f74 (issue 817's task shader): lost, frame 840, submit 5889; after the present's copy.
RX_VK_CHECKPOINTS=all (this PR's default): not lost, 14:28 to past 16:15 (110 min), Kokiri Forest in them.
This PR's change with no switch: run under way (Testing).

	main @ `5e072fc8`	this PR @ `0cb75e95`
Kokiri Forest on the iGPU (e072164e's renderer without markers / with this PR's markers)	device lost in 6 runs of 6	device kept in 1 run of 1 (110 min)
Kokiri Forest frame rate on the iGPU	not measurable: the device is lost	median 11 fps (8 to 25.7) over the 85 fps lines logged there
`tests/run_all.py`	518 passed, 8 failed, 35 skipped	520 passed, 7 failed, 34 skipped

RX_VK_CHECKPOINTS=1:
[hos sead::MainThread] vk: a capture's wait for write-back 5566 timed out (done 5562)
[hos sead::MainThread] nvn: GPU waits in 5 s: 1 sync waits (4176 ms), 0 polls timed out (0 ms), 0 queue finishes (0 ms)
[hos Presentation Thread] vk: submit 5574 failed (-4)
[hos Presentation Thread] vk: device lost: 23670 markers set, 21030 reached; the last whose earlier work all finished: 21030
[hos Presentation Thread] vk:   RUNNING after: marker 21030 (dispatch of program 53fcaee4dc45c6ee, frame 794)
[hos Presentation Thread] vk: FATAL: the GPU device was lost (VK_ERROR_DEVICE_LOST): the driver stopped or reset the GPU, ...
(no marker "reached, earlier work unfinished" in any of the five runs with markers)

RX_VK_CHECKPOINTS=all, in Kokiri Forest:
[hos Presentation Thread] vk: 378480 presents, 13.8 fps; a frame: 1150 draws (0 skipped), 126 dispatches, 162 passes, 177 barriers, 6 submits, ...
[hos Presentation Thread] vk: GPU time 60.8 ms a frame (the longest submission 24.6 ms); 4.9 textures made a frame

this PR @ 0cb75e95's change, no switch:
[hos host] vk: RX_VK_CHECKPOINTS: VK_AMD_buffer_marker, a marker before every command (RX_VK_CHECKPOINTS=all, or AMD's integrated GPUs on Windows; =0: off) (memory type 1)

run_all.py, failing on both sides (this PC's setup; none reads the files changed here): test_release, test_oot_state_tap,
test_tested_configurations, test_dev_packs, test_framegen_resources, test_pad_layout, test_module_ids. Main's eighth,
test_mesh_args_cleanup (26 s), passed in this PR's runs.

Testing

Tested: On the iGPU above, the seven runs in Measurements (Kokiri Forest, played by hand). tests/run_all.py on main and on this PR, Windows 11, llvm-mingw from nightly-132-5e072fc8: the same seven fail on both. The three checks updated here (test_recorder_sync, test_inline_write_order, test_fsr_uniform_ownership) failed on this PR's source before their updates and pass with them. The renderer (RX_VK_ONLY) builds on 5e072fc8 and on e072164e. With no switch, the log on the iGPU says the markers are on.

Not tested: This PR's own build with no switch through Kokiri Forest: running now, result to follow in a comment. The frame-rate cost of the markers on the iGPU: main loses the device there, so there is nothing to compare with. Other AMD APUs, newer APU drivers. A Linux build was not run: the change is standard C++ and the default sits under #ifdef _WIN32, so Linux keeps main's behaviour. Dedicated cards are untouched.

Needs other testers:

AMD Radeon 780M or 680M laptop on Windows: Kokiri Forest for 10 minutes with this PR, expecting no lost device and the log line "a marker before every command"; then RX_VK_CHECKPOINTS=0 to see whether that APU loses the device without them (request to open)

Screenshots

Nothing on screen: the change is in how commands are recorded.

Risk and rollback

GPU time on AMD integrated GPUs: each marker waits for the work before it, so the GPU overlaps less work. Not measured against main, which loses the device there. RX_VK_CHECKPOINTS=0 turns them off.
CPU time: a vkCmdWriteBufferMarkerAMD pair before each command on those GPUs; elsewhere one predictable branch per command (checkpoint_fine()).
The race is hidden, not fixed: it may still show elsewhere, or come back when timing changes.
Reverting the commit restores main's behaviour.

Reviewer notes

Start with the macros in gpu/vk/vk_work.h and the default in create_device() (gpu/vk/vk_main.cpp).

A lead for the race: in five of the six lost runs (all but the issue 817 one), the device was lost in the frame of, or the frame after, these lines: the same seven programs rebinding the same 149 MB storage buffer, logged once as Kokiri Forest starts. The run with markers before every command passed the same moment (frame 377250) and went on.

vk: table entry bound again at frame 793: prog 507e946d64f2d55e ssbo 0 of 3 (c0+0x880) addr 0x108c704000 size 0x9596000 (before 0)
vk: table entry bound again at frame 794: prog 10dc355939390bdc ssbo 0 of 3 (c0+0x880) addr 0x108c704000 size 0x9596000 (before 0)

The per-command-buffer report is the tool for the follow-up: with RX_VK_CHECKPOINTS=all the device is no longer lost, so the next step is fewer markers, by kind, to see which one keeps it. Open question: should this cover every AMD integrated GPU, or only RDNA 2 APUs, until other APUs are tried?

Issues

See the Kokiri Forest device loss on AMD integrated graphics (no issue on the tracker yet).

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (Pass/fail: six runs lost the device, the one with these markers did not; the checks on both sides on this PC. Main has no frame rate to compare: Not tested.)
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags noted, and the switch reference updated (docs/development.md, the README).
A check under tests/ that reads changed source updated in this PR, not weakened.
Docs updated.

## Why On an AMD integrated GPU (a Ryzen 9 7900X's Radeon Graphics, Windows) the game loses the GPU device (`VK_ERROR_DEVICE_LOST`, then FATAL) within seconds of Kokiri Forest loading, every time. With a marker written before every GPU command, as NVIDIA's checkpoints already are by default for the same kind of loss, it kept the device for 110 minutes. ## What changed What the GPU was doing when it was lost: with markers it stood still for 4-6 s (`nvn: GPU waits in 5 s: 1 sync waits (4176-6305 ms)`) before Windows reset it. Every marker it had reached had its earlier work finished, so it was not inside any draw, dispatch, upload, mirror sync, write-back or present copy: it stood between commands. Turning off each suspect in turn did not keep the device (Measurements). Markers before every command did. Each one waits for the work before it, so like the NV checkpoints (`vk_main.cpp`: "The race they hide is not found yet") they hide a race that is still to be found; this PR does not claim to have found it. - markers: `gpu/vk/vk_work.h`. The work macros that are not draws or dispatches (barriers, render pass begin/end, copies, blits, resolves, clears, fills, updates, queries) call `fine_mark` before the command, through a lambda that takes the macro's arguments once and passes them on, so no argument is evaluated twice. `fine_mark` does nothing unless `dev.fine_markers`. - default: `create_device()` in `gpu/vk/vk_main.cpp`. On AMD integrated GPUs on Windows (`vendorID 0x1002`, `VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU`) `VK_AMD_buffer_marker` is enabled and `dev.fine_markers` set with no switch. `RX_VK_CHECKPOINTS=all` asks for the same anywhere the NV checkpoints are missing, `RX_VK_CHECKPOINTS=1` keeps them to draws and dispatches, `RX_VK_CHECKPOINTS=0` turns them off. - report: `buffer_markers_report()` in `gpu/vk/vk_draw.cpp`. Each marker now keeps its command buffer, and after a lost device the log lists, for each command buffer, its last markers done and its first not reached. Before, an upload command buffer's markers interleaved in number with its recording's draws, and the single "RUNNING after" line could name the wrong command. New marker kinds name the commands, a recording's start (`begin_recording` in `gpu/vk/vk_replay.cpp`) and the present's copy (`present_image`). - tests: `tests/test_recorder_sync.py` accepts the lambda form (the packer still called, the driver never directly); `tests/test_inline_write_order.py` stubs `fine_mark` and removes the work count by the macro's start, so its negative control still fails; `tests/fsr_uniform_ownership_test.cpp` and `tests/recorder_sync_test.cpp` stub `checkpoint_mark` and `checkpoint_fine`. - docs: `docs/development.md`, `RX_VK_CHECKPOINTS=1` / `all`. - Deliberately unchanged: every other GPU (dedicated AMD cards, Intel, NVIDIA, Linux and the Steam Deck). Nothing runs there unless `RX_VK_CHECKPOINTS` asks for it. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `RX_VK_CHECKPOINTS=all` (new value) | `VK_AMD_buffer_marker` before every command, not only draws and dispatches; the lost-device report by command buffer | On for AMD integrated GPUs on Windows; off elsewhere | On there: without it Kokiri Forest loses the device every run | Ryzen 9 7900X Radeon Graphics (RDNA 2, 2 CUs), Windows | Not needed on dedicated cards: off there | Other AMD APUs (680M, 780M, 890M), Linux APUs | | `RX_VK_CHECKPOINTS=0` | Turns those markers off (and the NV checkpoints, as before) | Unset | For A/B tests only | - | - | - | ## Measurements main: 5e072fc8 (2026-11-14) this PR: fix/amd-igpu-markers @ 0cb75e95 Baseline: main @ 5e072fc8. In the game its side is e072164e's renderer without these markers: the game folder on the iGPU was built from e072164e (#404's branch: main @ 0caf0618 plus two Intel commits), and each run below swapped only its `longshot_vk.dll`, built from that source with the switch or change named. A renderer from 5e072fc8 changes the shader translator and would restart the shader preparation each time. The renderer files this PR touches are the same on both bases. Without markers, or with `RX_VK_CHECKPOINTS=1`, is main's behaviour on this GPU. Hardware: Ryzen 9 7900X's AMD Radeon(TM) Graphics (RDNA 2, 2 CUs), AMD proprietary driver 26.10.1 (Vulkan driver 0x800161), no mesh shaders (the compute fallback), Windows 11; the screen on an RX 9070 XT, so the iGPU's frames are presented across adapters Scene: New Game, then Kokiri Forest (`map: KokiriForest_Child`); output resolution 2048x1152 Method: one run each, played by hand until the device was lost or the run was stopped; the lines from each run's `longshot_log.txt`. The checks: `tests/run_all.py` on main and on this PR How to reproduce: on an AMD APU on Windows, walk into Kokiri Forest on main (or with `RX_VK_CHECKPOINTS=1`), then with this PR and no switch The runs on the iGPU, each a renderer built from e072164e (frame and submit where the device was lost; where the GPU stood, the last marker whose earlier work was done): - No markers (main's behaviour): lost seconds after Kokiri Forest loaded, about frame 1109, submit 7773. - `RX_VK_CHECKPOINTS=1`: lost seconds after, frame 794, submit 5574; after a dispatch of 53fcaee4dc45c6ee. - `=1` with `RX_VK_SKIP_PROG=53fcaee4dc45c6ee,53509044401d78d2`: lost, frame 3678, submit 25762; after a dispatch of 0ba2f076e4a94acf. - `=1` with `RX_VK_MIRROR=1`: lost, frame 2463, submit 17257; after a pool's write-back. - `=1` with the swapchain image waited for on the CPU (not in this PR): lost, frame 853, submit 5980; after a texture upload. - `=1` with `RX_VK_SKIP_CODE=2c56dc1006942f74` (issue 817's task shader): lost, frame 840, submit 5889; after the present's copy. - `RX_VK_CHECKPOINTS=all` (this PR's default): **not lost**, 14:28 to past 16:15 (110 min), Kokiri Forest in them. - This PR's change with no switch: run under way (Testing). | | main @ 5e072fc8 | this PR @ 0cb75e95 | |---|---|---| | Kokiri Forest on the iGPU (e072164e's renderer without markers / with this PR's markers) | device lost in 6 runs of 6 | device kept in 1 run of 1 (110 min) | | Kokiri Forest frame rate on the iGPU | not measurable: the device is lost | median 11 fps (8 to 25.7) over the 85 fps lines logged there | | `tests/run_all.py` | 518 passed, 8 failed, 35 skipped | 520 passed, 7 failed, 34 skipped | ``` RX_VK_CHECKPOINTS=1: [hos sead::MainThread] vk: a capture's wait for write-back 5566 timed out (done 5562) [hos sead::MainThread] nvn: GPU waits in 5 s: 1 sync waits (4176 ms), 0 polls timed out (0 ms), 0 queue finishes (0 ms) [hos Presentation Thread] vk: submit 5574 failed (-4) [hos Presentation Thread] vk: device lost: 23670 markers set, 21030 reached; the last whose earlier work all finished: 21030 [hos Presentation Thread] vk: RUNNING after: marker 21030 (dispatch of program 53fcaee4dc45c6ee, frame 794) [hos Presentation Thread] vk: FATAL: the GPU device was lost (VK_ERROR_DEVICE_LOST): the driver stopped or reset the GPU, ... (no marker "reached, earlier work unfinished" in any of the five runs with markers) RX_VK_CHECKPOINTS=all, in Kokiri Forest: [hos Presentation Thread] vk: 378480 presents, 13.8 fps; a frame: 1150 draws (0 skipped), 126 dispatches, 162 passes, 177 barriers, 6 submits, ... [hos Presentation Thread] vk: GPU time 60.8 ms a frame (the longest submission 24.6 ms); 4.9 textures made a frame this PR @ 0cb75e95's change, no switch: [hos host] vk: RX_VK_CHECKPOINTS: VK_AMD_buffer_marker, a marker before every command (RX_VK_CHECKPOINTS=all, or AMD's integrated GPUs on Windows; =0: off) (memory type 1) run_all.py, failing on both sides (this PC's setup; none reads the files changed here): test_release, test_oot_state_tap, test_tested_configurations, test_dev_packs, test_framegen_resources, test_pad_layout, test_module_ids. Main's eighth, test_mesh_args_cleanup (26 s), passed in this PR's runs. ``` ## Testing **Tested:** On the iGPU above, the seven runs in Measurements (Kokiri Forest, played by hand). `tests/run_all.py` on main and on this PR, Windows 11, llvm-mingw from `nightly-132-5e072fc8`: the same seven fail on both. The three checks updated here (`test_recorder_sync`, `test_inline_write_order`, `test_fsr_uniform_ownership`) failed on this PR's source before their updates and pass with them. The renderer (`RX_VK_ONLY`) builds on 5e072fc8 and on e072164e. With no switch, the log on the iGPU says the markers are on. **Not tested:** This PR's own build with no switch through Kokiri Forest: running now, result to follow in a comment. The frame-rate cost of the markers on the iGPU: main loses the device there, so there is nothing to compare with. Other AMD APUs, newer APU drivers. A Linux build was not run: the change is standard C++ and the default sits under `#ifdef _WIN32`, so Linux keeps main's behaviour. Dedicated cards are untouched. **Needs other testers:** - AMD Radeon 780M or 680M laptop on Windows: Kokiri Forest for 10 minutes with this PR, expecting no lost device and the log line "a marker before every command"; then `RX_VK_CHECKPOINTS=0` to see whether that APU loses the device without them (request to open) ## Screenshots Nothing on screen: the change is in how commands are recorded. ## Risk and rollback - GPU time on AMD integrated GPUs: each marker waits for the work before it, so the GPU overlaps less work. Not measured against main, which loses the device there. `RX_VK_CHECKPOINTS=0` turns them off. - CPU time: a `vkCmdWriteBufferMarkerAMD` pair before each command on those GPUs; elsewhere one predictable branch per command (`checkpoint_fine()`). - The race is hidden, not fixed: it may still show elsewhere, or come back when timing changes. - Reverting the commit restores main's behaviour. ## Reviewer notes Start with the macros in `gpu/vk/vk_work.h` and the default in `create_device()` (`gpu/vk/vk_main.cpp`). A lead for the race: in five of the six lost runs (all but the issue 817 one), the device was lost in the frame of, or the frame after, these lines: the same seven programs rebinding the same 149 MB storage buffer, logged once as Kokiri Forest starts. The run with markers before every command passed the same moment (frame 377250) and went on. ``` vk: table entry bound again at frame 793: prog 507e946d64f2d55e ssbo 0 of 3 (c0+0x880) addr 0x108c704000 size 0x9596000 (before 0) vk: table entry bound again at frame 794: prog 10dc355939390bdc ssbo 0 of 3 (c0+0x880) addr 0x108c704000 size 0x9596000 (before 0) ``` The per-command-buffer report is the tool for the follow-up: with `RX_VK_CHECKPOINTS=all` the device is no longer lost, so the next step is fewer markers, by kind, to see which one keeps it. Open question: should this cover every AMD integrated GPU, or only RDNA 2 APUs, until other APUs are tried? ## Issues See the Kokiri Forest device loss on AMD integrated graphics (no issue on the tracker yet). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (Pass/fail: six runs lost the device, the one with these markers did not; the checks on both sides on this PC. Main has no frame rate to compare: Not tested.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.

Yeetbot90 added 1 commit

2026-11-14 15:40:22 +00:00

fix(vk): a marker before every GPU command on AMD's integrated GPUs, so Kokiri Forest no longer loses the device

PR description / Description follows the template (pull_request_target) Successful in 27s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

CI / Conventional Commit title (pull_request) Has been cancelled

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled

Details

CI / Workflow lint (actionlint) (pull_request) Has been cancelled

Details

Third-party parts / As their notes say (pull_request) Has been cancelled

Details

Third-party parts / The same as upstream (pull_request) Has been cancelled

Details

0cb75e95a1

A Ryzen 9 7900X's Radeon Graphics (2 CUs, driver 26.10.1, no mesh shaders) lost the device
(VK_ERROR_DEVICE_LOST) in six runs of six within seconds of Kokiri Forest loading: with no markers and with
VK_AMD_buffer_marker before each draw and dispatch (RX_VK_CHECKPOINTS=1). With markers the GPU stood still for 4-6 s
before Windows reset it; every marker it had reached had its earlier work done, so it was on no draw or dispatch. Skipping the
dispatch it stopped after, the VRAM mirrors (RX_VK_MIRROR=1), waiting for the swapchain image on the CPU and leaving out
issue 817's culling task shader (RX_VK_SKIP_CODE=2c56dc1006942f74) all crashed the same way. With a marker before every
command (barriers, render passes, copies, clears, queries too) it kept the device for 110 minutes, Kokiri Forest in
them. As with the NVIDIA
checkpoints that are on by default for the same reason, the markers hide a race that is not found yet.

- vk_work.h: the work macros that are not draws or dispatches put a marker before the command (fine_mark), through a
  lambda that takes the arguments once, where dev.fine_markers.
- vk_main.cpp: VK_AMD_buffer_marker and dev.fine_markers on by default on AMD integrated GPUs on Windows;
  RX_VK_CHECKPOINTS=all asks for them anywhere, RX_VK_CHECKPOINTS=0 turns them off.
- vk_draw.cpp: each marker keeps its command buffer, and the lost-device report says, for each command buffer, its last
  marker done and its first not reached (an upload's markers interleave in number with its recording's draws).
- vk_replay.cpp: markers at a recording's start and at the present's copy.
- tests: the checks that read vk_work.h and begin_recording() take the lambda form and stub the two functions.

Co-Authored-By: Cursor <[email protected]>

mooki added the

labels

2026-11-14 16:09:17 +00:00

Yeetbot90 referenced this pull request

2026-11-14 16:30:26 +00:00

fix(vk): the graphics in AMD's desktop CPUs decode ASTC on the CPU, so Kokiri Forest no longer resets the GPU #489

Yeetbot90 commented

2026-11-14 16:30:37 +00:00

Author

Contributor

Closing: the claim here does not hold. This PR's own build, markers before every command by default, lost the device in Kokiri Forest all the same (submit 470388 failed (-4), the GPU waited for 5468 ms first), so the 110-minute run was not the fix.

The cause is the GPU's decode and BC3 re-encode of the terrain's two 1365 x 1365 x 90 ASTC arrays in one upload submission; decoded on the CPU, Kokiri Forest loaded 3 times of 3. That is #489, which replaces this one.

Closing: the claim here does not hold. This PR's own build, markers before every command by default, lost the device in Kokiri Forest all the same (`submit 470388 failed (-4)`, the GPU waited for 5468 ms first), so the 110-minute run was not the fix. The cause is the GPU's decode and BC3 re-encode of the terrain's two 1365 x 1365 x 90 ASTC arrays in one upload submission; decoded on the CPU, Kokiri Forest loaded 3 times of 3. That is #489, which replaces this one.

Yeetbot90 closed this pull request

2026-11-14 16:30:38 +00:00