WIP: fix(vk): the write-back diff pass has a checkpoint of its own in device-lost reports #1220

Draft
TrashPanda wants to merge 1 commit from TrashPanda/Longshot:fix/wb-diff-checkpoint into main
Contributor

Type: fix · Risk: low · Tested on: Linux, RTX 4070 Ti · Needs: none

Why

When the game loses the GPU (VK_ERROR_DEVICE_LOST, the 30-second hangs), the NVIDIA checkpoints are meant to name
what the GPU was doing. In the reports from an RTX 5060 (#899) and a GTX 1660 Ti (#1008) both pipeline stages stop at
"write-back of pool", and that is ambiguous: the write-back diff compute pass runs right after the last pool's copy
and set no marker of its own, so a fault in that pass reads as the copy. This gives the pass its own marker, so the
next report says which of the two it was. It does not fix the hangs; it is what the next step on them needs.

How to reproduce

Seen on: Windows, RTX 5060 (#899) and GTX 1660 Ti (#1008), from their reports' logs; not reproduced here (this PC has lost no device)
Settings and switches: defaults (checkpoints are on by default on NVIDIA)
Steps: in Kakariko (#899: its graveyard, every time; #1008: walking around the village, once), play until the device is lost, then read the vk: device lost lines in longshot_log.txt
Expected: the checkpoint at the stop names the work the GPU was doing
Actual: both stages stop at write-back of pool <address>, which is either that pool's copy or the write-back diff pass dispatched right after it

What changed

The write-back diff pass sets a checkpoint before its dispatch, and the device-lost report names it.

  • gpu/vk/vk_memory.cpp, memory_end_submission(): checkpoint_mark(cmd, n, 'w') between the pass's push constants
    and its vkCmdDispatch, with the chunk count as its key.
  • gpu/vk/vk_draw.cpp, checkpoint_kind(): 'w' reads "write-back diff pass, chunk count".
  • tests/test_wb_diff_checkpoint.py: the marker sits between the push and the dispatch, and the report names it.

Measurements

Nothing to measure: one vkCmdSetCheckpointNV more in a submission with a write-back diff, recorded only with checkpoints on, read only after a lost device.

Flags and settings

None.

Testing

Tested: python -I -B tests/run_all.py on Linux, before the rebase and on the rebased code: every failure also
fails on main on this PC, whose toolchain lacks what those checks need (listed in a comment).
tests/test_wb_diff_checkpoint.py fails on main and passes here. In the game: Ubuntu 26.04.1, RTX 4070 Ti, NVIDIA
595.99.02, checkpoints on (the default on NVIDIA): ten 6-minute tours of Kakariko and the Market
(scripts/memtour.py) and two screenshot runs with this marker recording, with no device lost.

Not tested: a device-lost report showing the new marker. It needs a GPU that loses the device, which this PC
has not done.

Must be tested: None before merge. After it, the next device-lost log from an affected card (#899, #1008, #839)
says "write-back diff pass" or "write-back of pool" at the stop, which tells which one to look at.

Risks

What could break: nothing that renders. A checkpoint is a marker in the command buffer.

Turning it off: RX_VK_CHECKPOINTS=0 (all checkpoints), or revert the commit.

Compatibility: Nothing changes.

Issues

See #899
See #1008
See #839
See #811

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.
**Type:** fix · **Risk:** low · **Tested on:** Linux, RTX 4070 Ti · **Needs:** none ## Why When the game loses the GPU (`VK_ERROR_DEVICE_LOST`, the 30-second hangs), the NVIDIA checkpoints are meant to name what the GPU was doing. In the reports from an RTX 5060 (#899) and a GTX 1660 Ti (#1008) both pipeline stages stop at "write-back of pool", and that is ambiguous: the write-back diff compute pass runs right after the last pool's copy and set no marker of its own, so a fault in that pass reads as the copy. This gives the pass its own marker, so the next report says which of the two it was. It does not fix the hangs; it is what the next step on them needs. ## How to reproduce **Seen on:** Windows, RTX 5060 (#899) and GTX 1660 Ti (#1008), from their reports' logs; not reproduced here (this PC has lost no device) **Settings and switches:** defaults (checkpoints are on by default on NVIDIA) **Steps:** in Kakariko (#899: its graveyard, every time; #1008: walking around the village, once), play until the device is lost, then read the `vk: device lost` lines in `longshot_log.txt` **Expected:** the checkpoint at the stop names the work the GPU was doing **Actual:** both stages stop at `write-back of pool <address>`, which is either that pool's copy or the write-back diff pass dispatched right after it ## What changed The write-back diff pass sets a checkpoint before its dispatch, and the device-lost report names it. - `gpu/vk/vk_memory.cpp`, `memory_end_submission()`: `checkpoint_mark(cmd, n, 'w')` between the pass's push constants and its `vkCmdDispatch`, with the chunk count as its key. - `gpu/vk/vk_draw.cpp`, `checkpoint_kind()`: `'w'` reads "write-back diff pass, chunk count". - `tests/test_wb_diff_checkpoint.py`: the marker sits between the push and the dispatch, and the report names it. ## Measurements Nothing to measure: one `vkCmdSetCheckpointNV` more in a submission with a write-back diff, recorded only with checkpoints on, read only after a lost device. ### Flags and settings None. ## Testing **Tested:** `python -I -B tests/run_all.py` on Linux, before the rebase and on the rebased code: every failure also fails on `main` on this PC, whose toolchain lacks what those checks need (listed in a comment). `tests/test_wb_diff_checkpoint.py` fails on `main` and passes here. In the game: Ubuntu 26.04.1, RTX 4070 Ti, NVIDIA 595.99.02, checkpoints on (the default on NVIDIA): ten 6-minute tours of Kakariko and the Market (`scripts/memtour.py`) and two screenshot runs with this marker recording, with no device lost. **Not tested:** a device-lost report showing the new marker. It needs a GPU that loses the device, which this PC has not done. **Must be tested:** None before merge. After it, the next device-lost log from an affected card (#899, #1008, #839) says "write-back diff pass" or "write-back of pool" at the stop, which tells which one to look at. ## Risks **What could break:** nothing that renders. A checkpoint is a marker in the command buffer. **Turning it off:** `RX_VK_CHECKPOINTS=0` (all checkpoints), or revert the commit. **Compatibility:** Nothing changes. ## Issues See #899 See #1008 See #839 See #811 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here.
fix(vk): the write-back diff pass has a checkpoint of its own in device-lost reports
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
6f5b1f9929
The device-lost logs of an RTX 5060 (#899) and a GTX 1660 Ti (#1008) stop with both pipeline stages at "write-back of
pool": the last marker before the write-back diff's dispatch, which had none, so a fault in the pass read as the last
pool's copy. The pass now sets one ('w', with its chunk count), so the next report says which of the two it was.

Refs #899, #1008, #839, #811

Assisted-by: Claude
Author
Contributor

Tested in game: Ubuntu 26.04.1, RTX 4070 Ti, NVIDIA 595.99.02, checkpoints on (the NVIDIA default): ten 6-minute tours of Kakariko and the Market and two screenshot runs with the marker recording; no device lost, play as on main.

Tested in game: Ubuntu 26.04.1, RTX 4070 Ti, NVIDIA 595.99.02, checkpoints on (the NVIDIA default): ten 6-minute tours of Kakariko and the Market and two screenshot runs with the marker recording; no device lost, play as on main.
Author
Contributor

Checks that fail on main @ 0cab3548 on this PC (Ubuntu 26.04.1; its toolchain lacks what they need), python -I -B tests/run_all.py:

  • tests/test_audio_output_devices.py
  • tests/test_bug_report.py
  • tests/test_bug_report_defaults.py
  • tests/test_crash_record.py
  • tests/test_crt_options.py
  • tests/test_dlss_feature_size.py
  • tests/test_failed_compile_keeps_objects.py
  • tests/test_fsr_uniform_ownership.py
  • tests/test_guest_imported.py
  • tests/test_hardware_report.py
  • tests/test_import_reach.py
  • tests/test_lift_lookups.py
  • tests/test_linux_preflight.py
  • tests/test_linux_update.py
  • tests/test_make_game_cache.py
  • tests/test_missing_nup.py
  • tests/test_mods_size_table.py
  • tests/test_nonuniform_image_arrays.py
  • tests/test_parallel_init_order.py
  • tests/test_pending_guest_import.py
  • tests/test_replay_write_order.py
  • tests/test_run_all_changed.py
  • tests/test_vk_occlusion_queries.py
  • tests/test_walk.py
  • tests/test_walk_key.py

This PR's run fails the same ones and nothing else (tests/test_control.py timed out once in the full run on #1222's branch and passes alone).

**Checks that fail on `main` @ 0cab3548 on this PC** (Ubuntu 26.04.1; its toolchain lacks what they need), `python -I -B tests/run_all.py`: - `tests/test_audio_output_devices.py` - `tests/test_bug_report.py` - `tests/test_bug_report_defaults.py` - `tests/test_crash_record.py` - `tests/test_crt_options.py` - `tests/test_dlss_feature_size.py` - `tests/test_failed_compile_keeps_objects.py` - `tests/test_fsr_uniform_ownership.py` - `tests/test_guest_imported.py` - `tests/test_hardware_report.py` - `tests/test_import_reach.py` - `tests/test_lift_lookups.py` - `tests/test_linux_preflight.py` - `tests/test_linux_update.py` - `tests/test_make_game_cache.py` - `tests/test_missing_nup.py` - `tests/test_mods_size_table.py` - `tests/test_nonuniform_image_arrays.py` - `tests/test_parallel_init_order.py` - `tests/test_pending_guest_import.py` - `tests/test_replay_write_order.py` - `tests/test_run_all_changed.py` - `tests/test_vk_occlusion_queries.py` - `tests/test_walk.py` - `tests/test_walk_key.py` This PR's run fails the same ones and nothing else (`tests/test_control.py` timed out once in the full run on #1222's branch and passes alone).
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/wb-diff-checkpoint:TrashPanda-fix/wb-diff-checkpoint'
git switch 'TrashPanda-fix/wb-diff-checkpoint'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1220
No description provided.