feat(vk): matrix multiplies through WMMA 16x16x16 workgroup tiles on AMD RDNA 3/4 #1038
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1038
Loading…
Reference in a new issue
No description provided.
Delete branch "jcabarca/Longshot:feat/vk-wmma16"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Windows, AMD Radeon RX 9070 XT · Needs: none
Why
On AMD RDNA 3 and 4 the driver's cooperative matrices offer only the 16x16x16 f16 shape, not the 16x8x16 one the console's
HMMA_nx2maps to, socoopmat_init()sent every matrix multiply (the game's neural-network shaders) to the slow software path on subgroup shuffles.How to reproduce
Seen on: AMD Radeon RX 9070 XT, Windows (RDNA 4):
vkGetPhysicalDeviceCooperativeMatrixPropertiesKHRlists 16x16x16 f16 but no 16x8x16, and the log says the matrix multiplies run in softwareSettings and switches: defaults
Steps: game-free:
python -I -B tests/test_wmma16.py --source <main checkout>andpython -I -B tests/test_wmma16_spirv_val.py --source <main checkout> --headers <SPIR-V headers> --cxx clang++ --spirv-val <spirv-val>(the tests from this PR)Expected: on a GPU with only 16x16x16 cooperative matrices,
HMMA_nx2runs through them, gives A*B+C at the console's positions, and its SPIR-V is valid for Vulkan 1.3Actual: on main Sirit has no
OpCooperativeMatrixStoreKHRand there is no 16x16x16 path; both tests failWhat changed
Where the GPU has only 16x16x16 f16 cooperative matrices and full 32-lane compute subgroups, each
HMMA_nx2goes through a per-subgroup tile of shared memory: the lanes write their console fragments at their PTX m16n8k16 positions, one or two 16x16x16 multiplies run, and the lanes read their results back. Elsewhere nothing changes.coopmat_init()ingpu/vk/vk_coopmat.cppalso looks for the 16x16x16 f16 subgroup shape; with it and without 16x8x16, newwmma16_init()needsshaderFloat16, 32-lane compute subgroups andcomputeFullSubgroups, then runstest_multiply_wmma16()(a 16x16 multiply on the GPU checked against the CPU, 128 elements). Any miss takes the software multiply and logs why.DefineWmma16MatrixFunction()inemit_spirv_warp.cpp;EmitMatrixMulAddF16()calls it when the program has it. It returns false (software multiply) unless full subgroups are on and the workgroup X width is a multiple of 32, and unless the tiles fit in the device's shared memory limit (mma_wmma16_shared_limit). Withexplicit_layoutthe tiles are a Block at an offset after the program's own shared memory,Aliased.mma_layout.hgainsmma_wmma16,mma_wmma16_fast, the tile size, the shared limit andmma_wmma16_full_subgroups;spirv_emit_context.hgainswmma16_function;emit_spirv.cppdefines it.OpCooperativeMatrixLoadKHRwith a memory operand andOpCooperativeMatrixStoreKHR.vk_shader.cpp: the translator stamp includesmma_wmma16andmma_wmma16_fast(programs translated for another path are not reused); the network gate lets AMD through whenmma_wmma16is on.gpu/vk/VULKAN.md, a section on the path and its log lines.tests/test_wmma16.pyandtests/test_wmma16_spirv_val.py(new).How it works, step by step:
OpControlBarrier, Subgroup scope, AcquireRelease, WorkgroupMemory, MakeAvailable, MakeVisible) follows.OpCooperativeMatrixLoadKHRloads A, B's K half and C as 16x16x16 matrices;OpCooperativeMatrixMulAddKHRruns once or twice (one K=16 multiply withmma_wmma16_fast, else two K=8 halves);OpCooperativeMatrixStoreKHRwrites D back to the tile. The loads and the store carryNonPrivatePointer.pin_subgroup()(vk_draw.cpp, as on main) creates the stage withREQUIRE_FULL_SUBGROUPSandrequiredSubgroupSize = 32, i.e. a compute stage whose X width is a multiple of 32 on a device withcomputeFullSubgroups.The Vulkan rules this relies on, from
validusage.json:pin_subgroup()sets it only withcomputeFullSubgroups, andwmma16_init()requires it.DefineWmma16MatrixFunction()refuses any other width.pin_subgroup()gets a workgroup width only for compute stages.spirv-val --target-env vulkan1.3on five modules built from the realDefineWmma16MatrixFunction()(Measurements).Before (
coopmat_init()invk_coopmat.cpp):After:
DefineWmma16MatrixFunction(), its gate:Measurements
Verdict: it adds the path:
tests/test_wmma16.pyandtests/test_wmma16_spirv_val.pyfail on main and pass with this PR, in both interleaved runs; every WMMA16 module passesspirv-val. No frame-time numbers: the speed in the game was not measured side by side with main.Hardware: AMD Radeon RX 9070 XT, Windows; the tests themselves are game-free (they read source, build cut-out fixtures and validate SPIR-V, they do not use the GPU); Python 3.12, clang++ (llvm-mingw), spirv-val from SPIRV-Tools v2025.1
Scene: none, game-free
Method: both tests from this PR with
--sourceset to each side's checkout, 2 runs each, interleaved (main, PR, main, PR); the result is pass/fail, so the median is the result both runs gaveHow to reproduce:
python -I -B tests/test_wmma16.py --source <checkout>andpython -I -B tests/test_wmma16_spirv_val.py --source <checkout> --headers <SPIR-V headers> --cxx clang++ --spirv-val <spirv-val>for each sidecefce0d14aa76728tests/test_wmma16.py, run 1tests/test_wmma16_spirv_val.py, run 1tests/test_wmma16.py, run 2tests/test_wmma16_spirv_val.py, run 2What the tests check:
test_wmma16.py: Sirit has the store and the load with a memory operand;coopmat_init()looks for 16x16x16 and callswmma16_init(), which needscomputeFullSubgroups; the translator stamp and the network gate knowmma_wmma16; the tile positions, run on the CPU, give A*B+C at the console's positions.test_wmma16_spirv_val.py: the realDefineWmma16MatrixFunction()built into a Sirit fixture for five programs (one and two subgroups; with the program's own shared memory and an explicit layout; one and two multiplies); each module passesspirv-val --target-env vulkan1.3; a 48-lane-wide workgroup and a device without full subgroups take the software multiply.Flags and settings
No new switch: the two existing ones below turn this path off.
RX_VK_NATIVE_MMA=0RX_VK_MMA_SOFTWARE=1Testing
Tested: on this branch, Windows, Python 3.12, clang++ (llvm-mingw), spirv-val (SPIRV-Tools v2025.1):
test_wmma16andtest_wmma16_spirv_valpass (both fail on main).Built and played: this commit builds on Windows (llvm-mingw back end) and on Linux (
longshot_vk.so, without and with AMD FSR 4). Played on Windows, AMD Radeon RX 9070 XT: the game runs normally.Not tested: reading the game log for the startup test's result on this GPU; RDNA 3 (only RDNA 4 was run); the frame time against the software path. Linux was built, not run.
Must be tested: None: RDNA 3 has the same 16x16x16 shape, and any device where the startup test fails takes the software multiply.
Risks
What could break: on RDNA 3/4 the neural-network shaders now run on cooperative matrices instead of subgroup shuffles; a driver bug in them would show as wrong output there, which the startup test (a full 16x16 multiply checked against the CPU) is meant to catch. Programs whose workgroup X width is not a multiple of 32, or whose tiles do not fit in shared memory, keep the software multiply.
Turning it off:
RX_VK_NATIVE_MMA=0orRX_VK_MMA_SOFTWARE=1, or revert the commit.Compatibility: On RDNA 3/4 the translator stamp changes, so the translated shaders are made again once (the shader cache rebuilds); nothing changes on other GPUs.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). (the checks that read the changed code, on this PR, Windows:test_wmma16,test_wmma16_spirv_valpass)third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here. (tests/test_wmma16.py,tests/test_wmma16_spirv_val.py)tests/that reads changed source updated in this PR, not weakened. (both tests added; no check removed)longshot_vk.sowithout and with AMD FSR 4; no OS-specific code or paths)gpu/vk/VULKAN.md)Tested in game: Windows 11 IoT Enterprise LTSC 24H2 (build 26100.9711), AMD Radeon RX 9070 XT, AMD Software: Adrenalin Edition 26.10.1: played with this change; the game runs normally, no rendering errors or crashes seen.
WIP: feat(vk): matrix multiplies through WMMA 16x16x16 workgroup tiles on AMD RDNA 3/4to feat(vk): matrix multiplies through WMMA 16x16x16 workgroup tiles on AMD RDNA 3/4Rebased onto main
6c8d6e6ato clear the conflict with #1086 (software MMA unroll).After the rebase: test_wmma16 fails on main
6c8d6e6aand passes here; test_wmma16_spirv_val and test_mma_software also pass here.4aa76728c67b54ddcc4fView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.