fix(vk): a timeline wait that comes back device-lost is said as a lost device, not as a timeout #670

Open
mooki wants to merge 1 commit from mooki/Longshot:fix/lost-wait-said into main
Member

Type: fix · Risk: low · Tested on: macOS 26.7.1, Apple M4 Max (MoltenVK 1.4.2), arm64 and x86_64 under Rosetta · Needs: Windows and Linux, any GPU (build and start)

Why

When the GPU device is lost while the game waits on one of its timelines, the log says the wait timed out, which points at the
wrong thing. In the macOS port's start 21 the device was lost during a scene load, and the last line before the end was
vk: a capture's wait for write-back 9646 timed out (done 0): the wait had come back at once with VK_ERROR_DEVICE_LOST, and
done 0 was a counter the driver never wrote once the device was lost. Whoever reads such a log looks for a slow GPU or a
deadlock instead of a lost device. This PR says a lost wait as a lost device. Nothing else changes. It came out of the macOS port
(See #420), but every platform runs this code.

How to reproduce

Seen on: macOS 26.7.1, Apple M4 Max, MoltenVK 1.4.2 (the macOS port's integration build 09f9a171; main does not run on
macOS yet); the same code path runs on every driver
Settings and switches: defaults
Steps: play until the GPU device is lost while a capture waits for its write-back (start 21: the field-dawn save's load
under memory pressure).
Expected: a line that says the device is lost.
Actual: vk: a capture's wait for write-back 9646 timed out (done 0) (about 17 ms after the wait began), then the game ends
on the next submit (reports/2026-11-14-mac-startup-21.md; analysis in reports/2026-11-14-device-lost-21.md §1: MoltenVK
returns VK_ERROR_DEVICE_LOST from the wait at once and leaves the semaphore's counter unwritten once the device is marked lost,
so completed() read 0).

What changed

A timeline wait that comes back VK_ERROR_DEVICE_LOST is said as vk: a wait for recording N came back VK_ERROR_DEVICE_LOST: the GPU device is lost (or a capture's wait for write-back N …; 8 lines at most), and no longer as a wait that timed out. The game still ends where it did: at the next
submit or by the GPU hang watch (device_lost, issue 1275); a wait that times out is said as before.

  • gpu/vk/vk_replay.cpp wait_rec_seq: keeps its wait's result (wait_rec_result, thread-local) and says a lost one through
    wait_lost ("a wait for recording N").
  • vk_timeline_wait: after a recordings' wait, a lost one is not said again as a capture's wait that timed out; its own
    timeline wait says a lost one through wait_lost ("a capture's wait for write-back N").
  • tests: tests/test_lost_wait_said.py (new): both waits cut from vk_replay.cpp and run against a made-up driver: a lost
    recordings' wait and a lost capture wait each said as a lost device and not as a timeout, a timeout said as before, a finished
    wait silent, the line capped, no wait ending the process; and every VK_ERROR_DEVICE_LOST check kept in
    test_device_lost_exit's form.

Measurements

Nothing to measure: the change is one log line where a wait comes back device-lost; the game-free test shows the lines.

Flags and settings

None.

Testing

Tested: on macOS 26.7.1, Apple M4 Max, at 3a4dfd5a, this PR's head before the rebases onto e090cc48 and aa95d35f
(main 6c9edffe plus this PR):

  • tests/test_lost_wait_said.py --arch arm64 and --arch x86_64 (the fixture runs under Rosetta): pass on both. On main
    6c9edffe with the test file copied in (sha256 in lw-test-files.sha256): fails on both: the recordings' lost wait said as
    a capture's wait for write-back 9646 timed out (done 0), the capture's lost wait said not at all
    (reports/PubWave-files/PubLate1/lw-proof-01/).
  • tests/test_device_lost_exit.py (with /usr/bin/clang++, and through clang++ -arch x86_64 under Rosetta): passes.
  • The 88 checks that read vk_replay.cpp, and the new one, through tools/ls-suite.py: 84 pass, 5 fail; 4 fail on main 6c9edffe
    on macOS too (reports/Baseline-files/main-6c9edffe/), and test_present_aspect_vulkan is skipped on macOS in a full run and
    fails only because it was named (no Vulkan library to link on this Mac) (lost-wait-said-related-01/, -compare.txt).
  • gpu/vk/vk_replay.cpp compiled whole for arm64 and x86_64 with main's renderer flags (with #476's, #490's and #492's macOS
    compile pieces merged in a scratch tree only): both objects made, exit 0 (reports/PubWave-files/PubLate1/compile/lost-wait-said/).
  • Off Apple (unifdef -U__APPLE__): vk_replay.cpp differs, as intended: this is every platform's change
    (reports/PubWave-files/PubLate1/offapple-lost-wait-said.txt).
  • After the rebase onto main aa95d35f, at 0c916d9f (3a4dfd5a before the rebases onto e090cc48 and aa95d35f):
    Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they
    apply) gave the same results as before; python -I -B tests/run_all.py on wave S1's merge 11a5b4e7, which carries
    this head: 568/22/17 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure
    (reports/2026-11-15-rebase-sweep-3.md).

Not tested: a real lost device on any platform with this head (the line is shown by the test's made-up driver); Windows and
Linux: not built on Windows or Linux; its test request covers it.

Must be tested:

  • Windows and Linux, any GPU: build this PR (head 0c916d9f, on main alone) and main aa95d35f the same way and start
    each to the title, expecting no came back VK_ERROR_DEVICE_LOST line in a run that keeps its device and
    tests/test_lost_wait_said.py passing, and every other line to match main's (by hand: test request T-5A9G).

Risks

What could break: only log text where a timeline wait comes back VK_ERROR_DEVICE_LOST: one line (at most 8 in a run) in place
of the "timed out" line, or where none was said. The early return after a lost recordings' wait skips only that "timed out" line;
the function returned there anyway. Every VK_ERROR_DEVICE_LOST check still goes to device_lost as before.

Turning it off: revert the PR.

Compatibility: Nothing changes for players: no switch, shader stamp, cache format or MKWVK_API_VERSION change.

Reviewer notes

  • Review 3a4dfd5a (one commit on main 6c9edffe).
  • Split from the macOS giant compiles' memory gate (#669) at the owner's rule that every-platform changes go in
    their own PRs: there the Mac also ends the game at such a wait (device_lost), Apple-gated, and its other platforms keep main's
    text. git merge-tree with that branch's head conflicts in gpu/vk/vk_replay.cpp (both change the two waits): whichever lands
    second keeps this PR's wait_rec_result and line on every platform and adds the Mac's device_lost() after the line, as the
    topic first had it (macos/giant-mem-gate 230dde4c).
  • No conflict (git merge-tree) with the other open macOS PRs (#423, #427, #428, #431, #438, #476, #479, #480, #490, #491, #492,
    #498), #642, #650, #651, #652, #653 or #666 (reports/PubWave-files/PubLate1/merge-check-lost-wait-said.txt).
  • Conflicts with main: none.

Issues

See #420

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how. (Test request T-5A9G.)
  • Rebased on current main: on aa95d35f (#761, 2026-11-15), head 0c916d9f, without a conflict; this PR's own
    commits, old → new: 3a4dfd5a → 0c916d9f; the commit names elsewhere in this description are from before the
    rebase.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). (At 3a4dfd5a: 532 passed, 32 failed, 17 skipped against main 6c9edffe's 531/32/17 on this Mac; the 32 failures are main's own, nothing that passes on main fails, and test_lost_wait_said is new and passes; reports/PubWave-files/lost-wait-full-3a4dfd5a/. After the rebase onto main aa95d35f (head 0c916d9f): wave S1's merge 11a5b4e7, 568/22/17 against main aa95d35f's 543/33/17, no new failure.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.
**Type:** fix · **Risk:** low · **Tested on:** macOS 26.7.1, Apple M4 Max (MoltenVK 1.4.2), arm64 and x86_64 under Rosetta · **Needs:** Windows and Linux, any GPU (build and start) ## Why When the GPU device is lost while the game waits on one of its timelines, the log says the wait timed out, which points at the wrong thing. In the macOS port's start 21 the device was lost during a scene load, and the last line before the end was `vk: a capture's wait for write-back 9646 timed out (done 0)`: the wait had come back at once with `VK_ERROR_DEVICE_LOST`, and `done 0` was a counter the driver never wrote once the device was lost. Whoever reads such a log looks for a slow GPU or a deadlock instead of a lost device. This PR says a lost wait as a lost device. Nothing else changes. It came out of the macOS port (See #420), but every platform runs this code. ## How to reproduce **Seen on:** macOS 26.7.1, Apple M4 Max, MoltenVK 1.4.2 (the macOS port's integration build 09f9a171; main does not run on macOS yet); the same code path runs on every driver **Settings and switches:** defaults **Steps:** play until the GPU device is lost while a capture waits for its write-back (start 21: the field-dawn save's load under memory pressure). **Expected:** a line that says the device is lost. **Actual:** `vk: a capture's wait for write-back 9646 timed out (done 0)` (about 17 ms after the wait began), then the game ends on the next submit (`reports/2026-11-14-mac-startup-21.md`; analysis in `reports/2026-11-14-device-lost-21.md` §1: MoltenVK returns `VK_ERROR_DEVICE_LOST` from the wait at once and leaves the semaphore's counter unwritten once the device is marked lost, so `completed()` read 0). ## What changed A timeline wait that comes back `VK_ERROR_DEVICE_LOST` is said as `vk: a wait for recording N came back VK_ERROR_DEVICE_LOST: the GPU device is lost` (or `a capture's wait for write-back N …`; 8 lines at most), and no longer as a wait that timed out. The game still ends where it did: at the next submit or by the GPU hang watch (`device_lost`, issue 1275); a wait that times out is said as before. - `gpu/vk/vk_replay.cpp` `wait_rec_seq`: keeps its wait's result (`wait_rec_result`, thread-local) and says a lost one through `wait_lost` ("a wait for recording N"). - `vk_timeline_wait`: after a recordings' wait, a lost one is not said again as a capture's wait that timed out; its own timeline wait says a lost one through `wait_lost` ("a capture's wait for write-back N"). - tests: `tests/test_lost_wait_said.py` (new): both waits cut from `vk_replay.cpp` and run against a made-up driver: a lost recordings' wait and a lost capture wait each said as a lost device and not as a timeout, a timeout said as before, a finished wait silent, the line capped, no wait ending the process; and every `VK_ERROR_DEVICE_LOST` check kept in `test_device_lost_exit`'s form. ## Measurements Nothing to measure: the change is one log line where a wait comes back device-lost; the game-free test shows the lines. ### Flags and settings None. ## Testing **Tested:** on macOS 26.7.1, Apple M4 Max, at `3a4dfd5a`, this PR's head before the rebases onto e090cc48 and aa95d35f (main 6c9edffe plus this PR): - `tests/test_lost_wait_said.py --arch arm64` and `--arch x86_64` (the fixture runs under Rosetta): pass on both. On main 6c9edffe with the test file copied in (sha256 in `lw-test-files.sha256`): fails on both: the recordings' lost wait said as `a capture's wait for write-back 9646 timed out (done 0)`, the capture's lost wait said not at all (`reports/PubWave-files/PubLate1/lw-proof-01/`). - `tests/test_device_lost_exit.py` (with `/usr/bin/clang++`, and through `clang++ -arch x86_64` under Rosetta): passes. - The 88 checks that read `vk_replay.cpp`, and the new one, through `tools/ls-suite.py`: 84 pass, 5 fail; 4 fail on main 6c9edffe on macOS too (`reports/Baseline-files/main-6c9edffe/`), and `test_present_aspect_vulkan` is skipped on macOS in a full run and fails only because it was named (no Vulkan library to link on this Mac) (`lost-wait-said-related-01/`, `-compare.txt`). - `gpu/vk/vk_replay.cpp` compiled whole for arm64 and x86_64 with main's renderer flags (with #476's, #490's and #492's macOS compile pieces merged in a scratch tree only): both objects made, exit 0 (`reports/PubWave-files/PubLate1/compile/lost-wait-said/`). - Off Apple (`unifdef -U__APPLE__`): `vk_replay.cpp` differs, as intended: this is every platform's change (`reports/PubWave-files/PubLate1/offapple-lost-wait-said.txt`). - **After the rebase onto main aa95d35f**, at `0c916d9f` (`3a4dfd5a` before the rebases onto e090cc48 and aa95d35f): Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they apply) gave the same results as before; `python -I -B tests/run_all.py` on wave S1's merge `11a5b4e7`, which carries this head: 568/22/17 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure (`reports/2026-11-15-rebase-sweep-3.md`). **Not tested:** a real lost device on any platform with this head (the line is shown by the test's made-up driver); Windows and Linux: not built on Windows or Linux; its test request covers it. **Must be tested:** - Windows and Linux, any GPU: build this PR (head `0c916d9f`, on main alone) and main aa95d35f the same way and start each to the title, expecting no `came back VK_ERROR_DEVICE_LOST` line in a run that keeps its device and `tests/test_lost_wait_said.py` passing, and every other line to match main's (by hand: test request T-5A9G). ## Risks **What could break:** only log text where a timeline wait comes back `VK_ERROR_DEVICE_LOST`: one line (at most 8 in a run) in place of the "timed out" line, or where none was said. The early return after a lost recordings' wait skips only that "timed out" line; the function returned there anyway. Every `VK_ERROR_DEVICE_LOST` check still goes to `device_lost` as before. **Turning it off:** revert the PR. **Compatibility:** Nothing changes for players: no switch, shader stamp, cache format or `MKWVK_API_VERSION` change. ## Reviewer notes - Review `3a4dfd5a` (one commit on main 6c9edffe). - Split from the macOS giant compiles' memory gate (#669) at the owner's rule that every-platform changes go in their own PRs: there the Mac also ends the game at such a wait (`device_lost`), Apple-gated, and its other platforms keep main's text. `git merge-tree` with that branch's head conflicts in `gpu/vk/vk_replay.cpp` (both change the two waits): whichever lands second keeps this PR's `wait_rec_result` and line on every platform and adds the Mac's `device_lost()` after the line, as the topic first had it (`macos/giant-mem-gate` `230dde4c`). - No conflict (`git merge-tree`) with the other open macOS PRs (#423, #427, #428, #431, #438, #476, #479, #480, #490, #491, #492, #498), #642, #650, #651, #652, #653 or #666 (`reports/PubWave-files/PubLate1/merge-check-lost-wait-said.txt`). - Conflicts with main: none. ## Issues See #420 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. (Test request T-5A9G.) - [x] Rebased on current `main`: on aa95d35f (#761, 2026-11-15), head `0c916d9f`, without a conflict; this PR's own commits, old → new: `3a4dfd5a` → `0c916d9f`; the commit names elsewhere in this description are from before the rebase. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (At `3a4dfd5a`: 532 passed, 32 failed, 17 skipped against main 6c9edffe's 531/32/17 on this Mac; the 32 failures are main's own, nothing that passes on main fails, and `test_lost_wait_said` is new and passes; `reports/PubWave-files/lost-wait-full-3a4dfd5a/`. After the rebase onto main aa95d35f (head `0c916d9f`): wave S1's merge `11a5b4e7`, 568/22/17 against main aa95d35f's 543/33/17, no new failure.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here.
fix(vk): a timeline wait that comes back device-lost is said as a lost device, not as a timeout
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Third-party parts / As their notes say (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 15s
3a4dfd5aaf
Run 21 (macOS, MoltenVK 1.4.2) lost the GPU device while a capture waited for
its write-back, and the log said "a capture's wait for write-back 9646 timed
out (done 0)": the driver marks the device lost, signals every timeline waited
on, returns VK_ERROR_DEVICE_LOST from the wait and leaves the counter's value
unwritten, so completed() read 0. The log showed a timeout where the GPU was
gone.

wait_rec_seq keeps its wait's result (wait_rec_result) and says a lost one as
"vk: a wait for recording N came back VK_ERROR_DEVICE_LOST: the GPU device is
lost"; vk_timeline_wait then does not say it again as a wait that timed out,
and its own timeline wait says "a capture's wait for write-back N came back
VK_ERROR_DEVICE_LOST" (wait_lost, 8 lines at most). Nothing else changes: the
next submit or the GPU hang watch ends the game, as before.

tests/test_lost_wait_said.py runs the two waits cut from vk_replay.cpp against
a made-up driver.

Assisted-by: Claude Opus 5.5

Co-Authored-By: Claude Opus 5.5 <[email protected]>
mooki force-pushed fix/lost-wait-said from 3a4dfd5aaf
All checks were successful
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Third-party parts / As their notes say (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 15s
to 8dfd549b55
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
CI / Workflow lint (actionlint) (pull_request) Successful in 21s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 20s
Third-party parts / As their notes say (pull_request) Successful in 34s
PR description / Description follows the template (pull_request_target) Successful in 4s
2026-11-15 07:14:30 +00:00
Compare
mooki changed title from WIP: fix(vk): a timeline wait that comes back device-lost is said as a lost device, not as a timeout to fix(vk): a timeline wait that comes back device-lost is said as a lost device, not as a timeout 2026-11-15 11:46:07 +00:00
mooki force-pushed fix/lost-wait-said from 8dfd549b55
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
CI / Workflow lint (actionlint) (pull_request) Successful in 21s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 20s
Third-party parts / As their notes say (pull_request) Successful in 34s
PR description / Description follows the template (pull_request_target) Successful in 4s
to 0c916d9f86
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 7s
Third-party parts / As their notes say (pull_request) Successful in 10s
PR description / Description follows the template (pull_request_target) Successful in 5s
2026-11-15 13:55:06 +00:00
Compare
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 5s
CI / Workflow lint (actionlint) (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 7s
Third-party parts / As their notes say (pull_request) Successful in 10s
PR description / Description follows the template (pull_request_target) Successful in 5s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/lost-wait-said:mooki-fix/lost-wait-said'
git switch 'mooki-fix/lost-wait-said'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!670
No description provided.