fix(vk): cooperative matrix probe checks map and fence, keeps in-flight resources #1247
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1247
Loading…
Reference in a new issue
No description provided.
Delete branch "jcabarca/Longshot:fix/vk-coopmat-probe"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Windows, AMD Radeon RX 9070 XT · Needs: none
Why
At start-up the cooperative matrix probe writes through an unchecked mapping, waits on a fence it may not have, and frees the pipeline, pool, fence and buffers while its dispatch may still run after a timed-out wait (VUID-vkDestroyPipeline-pipeline-00765 and others): a crash or a lost device at start-up on a slow or failing driver.
The same fix was first sent as part of #287 together with other changes; this PR sends it alone, one fix per PR.
How to reproduce
Seen on: not seen in the game: found by reading
run_shader()ingpu/vk/vk_coopmat.cpp; it happens whenvkMapMemoryorvkCreateFencefails, or when the probe's dispatch does not finish within its 5 s waitSettings and switches: defaults
Steps: game-free:
python -I -B tests/test_coopmat_probe_failures.py --cxx clang++ --headers <Vulkan-Headers include>with this PR's test on a main checkoutExpected: a failed map or fence returns cleanly; a dispatch that did not finish keeps what it uses
Actual: on main the
map-failcase fails (exit 2): the probe writes through a null mappingWhat changed
run_shader()checksvkMapMemoryandvkCreateFence, submits only with a fence, waits only after a successful submit, and when a submitted dispatch did not finish it returns without destroying what the dispatch uses (only the shader module, which may be destroyed once the pipeline exists).gpu/vk/vk_coopmat.cpp:run_shader()as above.tests/test_coopmat_probe_failures.py(new): four cases (map failure, fence failure, submit failure, wait timeout) against a cut-out ofrun_shader(); the fixture stands in for the Intel compiler gate so every case reaches the probe.The bug, step by step:
vkMapMemory's result was ignored andmemcpywrote throughwords.vkCreateFence's result was ignored;vkQueueSubmit2andvkWaitForFencesthen used a null fence.vkWaitForFencestimed out or failed, the code destroyed the fence, command pool, pipeline and buffers while the dispatch could still be pending.The rules:
When the dispatch did not finish the branch leaks the fence, pool, pipeline and buffers on purpose: Vulkan has no way to free them safely until the work completes, and the probe runs once at start-up.
Measurements
Verdict: it fixes the bug:
tests/test_coopmat_probe_failures.pyfails on main and passes with this PR, in both interleaved runs. No speed change is expected: the change is on a failure or check path.Hardware: AMD Radeon RX 9070 XT, Windows; the test itself is game-free (it runs on the CPU, compiled with clang++ from llvm-mingw where it builds a fixture); Python 3.12
Scene: none, game-free
Method: this PR's test files on each side's checkout, 2 runs each, interleaved (main, PR, main, PR); the result is pass/fail, so the median is the result both runs gave
How to reproduce:
python -I -B tests/test_coopmat_probe_failures.py --cxx clang++ --headers <Vulkan-Headers include>on each sideac778751b59cbcdftests/test_coopmat_probe_failures.py, run 1map-fail)tests/test_coopmat_probe_failures.py, run 2map-fail)Flags and settings
None.
Testing
Tested: on this branch, Windows, Python 3.12, clang++ from llvm-mingw:
tests/test_coopmat_probe_failures.pypasses (fails on main).Built and played: this commit builds on Windows (llvm-mingw back end) and on Linux (
longshot_vk.so, without and with AMD FSR 4). Played on Windows, AMD Radeon RX 9070 XT: the game runs normally. (The build that was played had this change before a whitespace-only fix: indentation, no code change.)Not tested: the failure path itself in the game (it needs a driver failure or a format pair the RX 9070 XT does not produce on demand). Linux was built, not run.
Must be tested: None: the change is the same on every GPU and OS, and the game-free test covers it.
Risks
What could break: a probe whose dispatch did not finish leaks one small buffer, a pipeline, a pool and a fence, once, at start-up.
Turning it off: revert the commit.
Compatibility: Nothing changes.
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). (the check that reads the changed code, on this PR, Windows:tests/test_coopmat_probe_failures.py)third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here. (tests/test_coopmat_probe_failures.py)tests/that reads changed source updated in this PR, not weakened.Tested in game: Played on Windows, AMD Radeon RX 9070 XT: the game runs normally.
Possibly related, for context:
I am juliotec, the original author of this fix. It was first sent as part of #287 together with other changes; this PR sends it alone, one fix per PR, rebased on current main.
WIP: fix(vk): cooperative matrix probe checks map and fence, keeps in-flight resourcesto fix(vk): cooperative matrix probe checks map and fence, keeps in-flight resourcesThe cooperative-matrix probe no longer writes through a failed mapping, submits without a fence, or destroys a dispatch that has not finished. I read head
b59cbcdf. Approved.run_shaderchecksvkMapMemoryand frees the buffer and its memory when the map fails. A fence that does not create is left null, and the submit is skipped, sovkWaitForFencesis not called on a null fence. When a submit did happen and the 5 s wait did not succeed, the function destroys the shader module and returns. The fence, command pool, descriptor pool, pipeline, pipeline layout, descriptor-set layout, buffer and memory stay. A pending dispatch still refers to all of those except the shader module, which Vulkan allows to be destroyed once the pipeline exists. A failed submit takes the old path and frees them, because nothing is pending.The fixture compiles that function and runs it. Map failure, fence failure and a timed-out wait are real calls. A failed
vkQueueSubmit2is not one of the cases, though the description names it; that path frees everything becausesubmittedis false. The Windows play on an RX 9070 XT did not hit a failed map, fence or wait. #912 is the other fix for the same wait: it waits without a limit and then destroys. It is still work in progress. Take this pull request. Do not land both.Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.