WIP: fix(vk): the cooperative matrix test waits for its dispatch before destroying what it uses #912

Draft

mooki wants to merge 1 commit from mooki/Longshot:fix/coopmat-safe-free into main

pull from: mooki/Longshot:fix/coopmat-safe-free

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 0 Commits 1 Files changed 2 +207 -2

mooki commented 2026-11-15 20:18:15 +00:00

Member

• Copy link
• Copy Markdown

Type: fix · Risk: low · Tested on: macOS 26.7, Apple M4 Max (game-free checks only, arm64 and x86_64) · Needs: Windows or Linux, an NVIDIA GPU with cooperative matrix

Why

At start, on a GPU with cooperative matrices, the renderer runs a small test on the GPU to decide whether the game's matrix multiplies (the movie upscaler's and TAAU's networks) can run natively. It waits 5 s for that test. If the GPU has not finished by then, the renderer destroys everything the test uses anyway, while the GPU may still be running it. That can end in a GPU fault or a lost device at start, on any GPU that offers cooperative matrices.

It has not been seen in a game; it was found by reading the source (it is the same pattern #788 fixes for the texture read-backs), and the new test shows it on main's code. No title in the issue tracker (snapshot of 2026-11-14 20:01) names the cooperative matrix test or a start-up timeout: the device losses and GPU hangs there (#130, #316, #365, #583, #606) come in play, not at start.

How to reproduce

Seen on: not seen in a game; shown on main's own run_shader by tests/test_coopmat_safe_free.py, on any OS
Settings and switches: defaults, on a GPU whose driver offers VK_KHR_cooperative_matrix with 16x8x16 f16 (vk: native matrix multiplies on in longshot_log.txt), and a GPU stall of more than 5 s during the test at start (vk: cooperative matrix test: the dispatch did not complete)
Steps: run python -I -B tests/test_coopmat_safe_free.py --headers <Vulkan-Headers/include> on main; in a game run, RX_VK_VALIDATION=1 and a GPU busy with other work at start
Expected: nothing the test's dispatch uses is destroyed while it may still be running
Actual: the test reports the fence, the command pool, the descriptor pool, the pipeline, the buffer and its memory destroyed under a dispatch that has not finished. The validation layer would name it (for example VUID-vkDestroyFence-fence-01120, VUID-vkDestroyCommandPool-commandPool-00041) when the GPU is that late

What changed

A dispatch not seen done in 5 s is now waited for, without a limit, before what it uses is destroyed, as #788 does for the read-backs (readback_settle). The test's result is as before: a dispatch not done in 5 s still fails it, and the multiplies run in software. Nothing is freed earlier than before.

• the cooperative matrix test, gpu/vk/vk_coopmat.cpp:
  • run_shader (main :114), called by measure_order and test_multiply from coopmat_init (:411, once the device is made, gpu/vk/vk_main.cpp:691), waited 5 s for its dispatch (:211) and then destroyed the fence, the command pool, the descriptor pool and the pipeline (:214-:217), and the buffer and its memory (:224-:225), whether the dispatch had finished or not. Vulkan forbids destroying a fence of a pending submit, a pool with a pending command buffer, and a pipeline or buffer a pending command uses.
  • Now the submit's result is kept (submitted), and when the 5 s wait gave up the fence is waited for without a limit before the destroys. A submit that failed is not waited for: nothing will signal its fence.
  • The GPU hang watch (RX_VK_GPU_HANG_SEC) starts at the first present, after this test, so it does not cover this wait; a lost device ends it (VK_ERROR_DEVICE_LOST).

Measurements

Verdict: nothing to measure. A normal start does the same work at the same time; only a test that runs past its 5 s wait changes when its objects are destroyed.

Flags and settings

None

Testing

Tested: game-free, on macOS 26.7, Apple M4 Max, arm64 and x86_64 (Rosetta):

• tests/test_coopmat_safe_free.py (new) fails on main 7bacf846 and passes on this PR, on both architectures. It runs the real run_shader against a GPU that finishes in time (as before), one that does not finish in 5 s (main destroys six objects under it; this PR waits first), and a submit that fails (no wait without a limit).
• Whole-file compiles of gpu/vk/vk_coopmat.cpp with the macOS port's own nx2/build.py commands (main has no macOS build), on both architectures: each exits 0, as on main. (Those commands turn warnings off, so only the exit codes count.)
• python -I -B tests/run_all.py on this Mac, where main's runner takes macOS for Linux: 559 passed, 34 failed, 17 skipped. Main 7bacf846 gives 558 passed, 34 failed, 17 skipped, and by name every result is the same apart from the new test (pass). The 34 fail on main too, on this Mac.

Not tested: in the game, on any platform: no GPU on this Mac offers cooperative matrices (MoltenVK has none), so the test does not run here at all, and a dispatch that runs past its 5 s wait cannot be made on demand. Not built on Windows (llvm-mingw) or Linux. The code has no platform branch, so it runs the same on both.

Must be tested:

• Windows or Linux, an NVIDIA RTX GPU whose driver offers VK_KHR_cooperative_matrix with 16x8x16 f16: with the Khronos validation layer installed, build this PR (head c45cad61) and main 7bacf846 the same way and run python -I -B tests/run_all.py. Then start each with RX_VK_VALIDATION=1 and play to the title, expecting every check to pass, vk: native matrix multiplies on (cooperative matrices, the console's fragment layout checked) in longshot_log.txt on both, and no validation error at start on this PR that main does not also show. A dispatch that runs past its 5 s wait cannot be made on demand (by hand: test request T-SUWS)

Risks

What could break: a cooperative matrix test that runs past its 5 s wait now holds the start until the GPU finishes it, instead of going on with its objects destroyed under it. A lost device still ends the wait. A GPU that hangs at that point without reporting the device lost would hold the start there, after the log line vk: cooperative matrix test: the dispatch did not complete; the hang watch does not run yet at that point.

Turning it off: revert the commit (RX_VK_NATIVE_MMA=0 or RX_VK_MMA_SOFTWARE=1 skip the test altogether, as on main)

Compatibility: Nothing changes

Checklist

Everyone

• One topic.
• The title is a Conventional Commit (type(scope): description) of the status line's type.
• Why gives the reason plainly and links the issues that fit (the tracker searched).
• Measurements has a verdict, or one line saying why there is nothing to measure.
• Testing names what was tested, what wasn't, and what must still be tested, where and how. (Test request T-SUWS.)
• Rebased on current main (main 7bacf846, head c45cad61).
• python -I -B tests/run_all.py passes (any failure that also fails on main is named).
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• A game-free test added for the fix, failing on main and passing here.
• A check under tests/ that reads changed source updated in this PR, not weakened. (None needed a change: test_recorder_sync already names vk_coopmat.cpp's submit and fence wait; test_intel_compiler_gate, test_vendor_fixes and test_global_table_loop read other parts of it.)
• Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Not built on either here, named under Not tested; ticked once test request T-SUWS has built it.)

**Type:** fix · **Risk:** low · **Tested on:** macOS 26.7, Apple M4 Max (game-free checks only, arm64 and x86_64) · **Needs:** Windows or Linux, an NVIDIA GPU with cooperative matrix ## Why At start, on a GPU with cooperative matrices, the renderer runs a small test on the GPU to decide whether the game's matrix multiplies (the movie upscaler's and TAAU's networks) can run natively. It waits 5 s for that test. If the GPU has not finished by then, the renderer destroys everything the test uses anyway, while the GPU may still be running it. That can end in a GPU fault or a lost device at start, on any GPU that offers cooperative matrices. It has not been seen in a game; it was found by reading the source (it is the same pattern #788 fixes for the texture read-backs), and the new test shows it on main's code. No title in the issue tracker (snapshot of 2026-11-14 20:01) names the cooperative matrix test or a start-up timeout: the device losses and GPU hangs there (#130, #316, #365, #583, #606) come in play, not at start. ## How to reproduce **Seen on:** not seen in a game; shown on main's own `run_shader` by `tests/test_coopmat_safe_free.py`, on any OS **Settings and switches:** defaults, on a GPU whose driver offers `VK_KHR_cooperative_matrix` with 16x8x16 f16 (`vk: native matrix multiplies on` in `longshot_log.txt`), and a GPU stall of more than 5 s during the test at start (`vk: cooperative matrix test: the dispatch did not complete`) **Steps:** run `python -I -B tests/test_coopmat_safe_free.py --headers <Vulkan-Headers/include>` on main; in a game run, `RX_VK_VALIDATION=1` and a GPU busy with other work at start **Expected:** nothing the test's dispatch uses is destroyed while it may still be running **Actual:** the test reports the fence, the command pool, the descriptor pool, the pipeline, the buffer and its memory destroyed under a dispatch that has not finished. The validation layer would name it (for example `VUID-vkDestroyFence-fence-01120`, `VUID-vkDestroyCommandPool-commandPool-00041`) when the GPU is that late ## What changed A dispatch not seen done in 5 s is now waited for, without a limit, before what it uses is destroyed, as #788 does for the read-backs (`readback_settle`). The test's result is as before: a dispatch not done in 5 s still fails it, and the multiplies run in software. Nothing is freed earlier than before. - the cooperative matrix test, `gpu/vk/vk_coopmat.cpp`: - `run_shader` (main `:114`), called by `measure_order` and `test_multiply` from `coopmat_init` (`:411`, once the device is made, `gpu/vk/vk_main.cpp:691`), waited 5 s for its dispatch (`:211`) and then destroyed the fence, the command pool, the descriptor pool and the pipeline (`:214`-`:217`), and the buffer and its memory (`:224`-`:225`), whether the dispatch had finished or not. Vulkan forbids destroying a fence of a pending submit, a pool with a pending command buffer, and a pipeline or buffer a pending command uses. - Now the submit's result is kept (`submitted`), and when the 5 s wait gave up the fence is waited for without a limit before the destroys. A submit that failed is not waited for: nothing will signal its fence. - The GPU hang watch (`RX_VK_GPU_HANG_SEC`) starts at the first present, after this test, so it does not cover this wait; a lost device ends it (`VK_ERROR_DEVICE_LOST`). ## Measurements **Verdict:** nothing to measure. A normal start does the same work at the same time; only a test that runs past its 5 s wait changes when its objects are destroyed. ### Flags and settings None ## Testing **Tested:** game-free, on macOS 26.7, Apple M4 Max, arm64 and x86_64 (Rosetta): - `tests/test_coopmat_safe_free.py` (new) fails on main `7bacf846` and passes on this PR, on both architectures. It runs the real `run_shader` against a GPU that finishes in time (as before), one that does not finish in 5 s (main destroys six objects under it; this PR waits first), and a submit that fails (no wait without a limit). - Whole-file compiles of `gpu/vk/vk_coopmat.cpp` with the macOS port's own `nx2/build.py` commands (main has no macOS build), on both architectures: each exits 0, as on main. (Those commands turn warnings off, so only the exit codes count.) - `python -I -B tests/run_all.py` on this Mac, where main's runner takes macOS for Linux: 559 passed, 34 failed, 17 skipped. Main `7bacf846` gives 558 passed, 34 failed, 17 skipped, and by name every result is the same apart from the new test (pass). The 34 fail on main too, on this Mac. **Not tested:** in the game, on any platform: no GPU on this Mac offers cooperative matrices (MoltenVK has none), so the test does not run here at all, and a dispatch that runs past its 5 s wait cannot be made on demand. Not built on Windows (llvm-mingw) or Linux. The code has no platform branch, so it runs the same on both. **Must be tested:** - Windows or Linux, an NVIDIA RTX GPU whose driver offers `VK_KHR_cooperative_matrix` with 16x8x16 f16: with the Khronos validation layer installed, build this PR (head `c45cad61`) and main `7bacf846` the same way and run `python -I -B tests/run_all.py`. Then start each with `RX_VK_VALIDATION=1` and play to the title, expecting every check to pass, `vk: native matrix multiplies on (cooperative matrices, the console's fragment layout checked)` in `longshot_log.txt` on both, and no validation error at start on this PR that main does not also show. A dispatch that runs past its 5 s wait cannot be made on demand (by hand: test request T-SUWS) ## Risks **What could break:** a cooperative matrix test that runs past its 5 s wait now holds the start until the GPU finishes it, instead of going on with its objects destroyed under it. A lost device still ends the wait. A GPU that hangs at that point without reporting the device lost would hold the start there, after the log line `vk: cooperative matrix test: the dispatch did not complete`; the hang watch does not run yet at that point. **Turning it off:** revert the commit (`RX_VK_NATIVE_MMA=0` or `RX_VK_MMA_SOFTWARE=1` skip the test altogether, as on main) **Compatibility:** Nothing changes ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. (Test request T-SUWS.) - [x] Rebased on current `main` (main `7bacf846`, head `c45cad61`). - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. (None needed a change: `test_recorder_sync` already names `vk_coopmat.cpp`'s submit and fence wait; `test_intel_compiler_gate`, `test_vendor_fixes` and `test_global_table_loop` read other parts of it.) - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Not built on either here, named under Not tested; ticked once test request T-SUWS has built it.)

mooki added 1 commit 2026-11-15 20:18:15 +00:00

fix(vk): the cooperative matrix test waits for its dispatch before destroying what it uses ... c45cad61bf

The test at start (vk_coopmat.cpp run_shader, from coopmat_init) waited 5 s for its one dispatch, then destroyed the
fence, the command pool, the descriptor pool, the pipeline, the buffer and its memory whether the dispatch had finished
or not: a hazard on every GPU, reached when the GPU runs late. A fence of a pending submit, a pool with a pending
command buffer, and a pipeline or buffer a pending dispatch uses must not be destroyed. Now a dispatch not seen done in
5 s is waited for, without a limit, before they are destroyed, as the read-backs do after their 5 s wait
(readback_settle). The test's result is as before: a dispatch not done in 5 s still fails it and the multiplies run in
software. Nothing is freed earlier than before. A submit that failed is not waited for (nothing will signal its fence).

tests/test_coopmat_safe_free.py runs the real run_shader against a GPU that finishes in time, one that does not, and a
submit that fails.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

BeanIssues added the

kind

bug

area/renderer

risk

low

labels 2026-11-15 20:18:18 +00:00

Some checks failed

Hide checks

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 5s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 4s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'fix/coopmat-safe-free:mooki-fix/coopmat-safe-free'

git switch 'mooki-fix/coopmat-safe-free'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/tooling

Developer tools: scripts, CI, build and release tooling

  

area/upscaling

DLSS, FSR, frame generation

  

ci: approved

A maintainer approved CircleCI for this fork PR (sticky; removed after 14 quiet days)

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Affects Linux (a change for every system carries both Windows and Linux)

  

platform/macos

macOS (Apple silicon or Intel Macs)

  

platform/steam-deck

Steam Deck

  

platform/windows

Affects Windows (a change for every system carries both Windows and Linux)

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

regression

Worked in an earlier build, broken in a newer one: highest priority, above other bugs

  

risk

high

Broad or core change: the renderer, defaults on many systems, saves, installs, security

  

risk

low

Small, contained change: little can break

  

risk

medium

Changes behaviour in one area, or a default on some systems

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

  

ci: approved

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/tooling

area/upscaling

ci: approved

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/macos

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

regression

risk

high

risk

low

risk

medium

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

ci: approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!912

No description provided.