feat(vk): matrix multiplies through WMMA 16x16x16 workgroup tiles on AMD RDNA 3/4 #1038

Open
jcabarca wants to merge 3 commits from jcabarca/Longshot:feat/vk-wmma16 into main
Contributor

Type: feat · Risk: medium · Tested on: Windows, AMD Radeon RX 9070 XT · Needs: none

Why

On AMD RDNA 3 and 4 the driver's cooperative matrices offer only the 16x16x16 f16 shape, not the 16x8x16 one the console's HMMA_nx2 maps to, so coopmat_init() sent every matrix multiply (the game's neural-network shaders) to the slow software path on subgroup shuffles.

How to reproduce

Seen on: AMD Radeon RX 9070 XT, Windows (RDNA 4): vkGetPhysicalDeviceCooperativeMatrixPropertiesKHR lists 16x16x16 f16 but no 16x8x16, and the log says the matrix multiplies run in software
Settings and switches: defaults
Steps: game-free: python -I -B tests/test_wmma16.py --source <main checkout> and python -I -B tests/test_wmma16_spirv_val.py --source <main checkout> --headers <SPIR-V headers> --cxx clang++ --spirv-val <spirv-val> (the tests from this PR)
Expected: on a GPU with only 16x16x16 cooperative matrices, HMMA_nx2 runs through them, gives A*B+C at the console's positions, and its SPIR-V is valid for Vulkan 1.3
Actual: on main Sirit has no OpCooperativeMatrixStoreKHR and there is no 16x16x16 path; both tests fail

What changed

Where the GPU has only 16x16x16 f16 cooperative matrices and full 32-lane compute subgroups, each HMMA_nx2 goes through a per-subgroup tile of shared memory: the lanes write their console fragments at their PTX m16n8k16 positions, one or two 16x16x16 multiplies run, and the lanes read their results back. Elsewhere nothing changes.

  • vk: coopmat_init() in gpu/vk/vk_coopmat.cpp also looks for the 16x16x16 f16 subgroup shape; with it and without 16x8x16, new wmma16_init() needs shaderFloat16, 32-lane compute subgroups and computeFullSubgroups, then runs test_multiply_wmma16() (a 16x16 multiply on the GPU checked against the CPU, 128 elements). Any miss takes the software multiply and logs why.
  • shader recompiler: new DefineWmma16MatrixFunction() in emit_spirv_warp.cpp; EmitMatrixMulAddF16() calls it when the program has it. It returns false (software multiply) unless full subgroups are on and the workgroup X width is a multiple of 32, and unless the tiles fit in the device's shared memory limit (mma_wmma16_shared_limit). With explicit_layout the tiles are a Block at an offset after the program's own shared memory, Aliased.
  • shader recompiler: mma_layout.h gains mma_wmma16, mma_wmma16_fast, the tile size, the shared limit and mma_wmma16_full_subgroups; spirv_emit_context.h gains wmma16_function; emit_spirv.cpp defines it.
  • Sirit: OpCooperativeMatrixLoadKHR with a memory operand and OpCooperativeMatrixStoreKHR.
  • vk: vk_shader.cpp: the translator stamp includes mma_wmma16 and mma_wmma16_fast (programs translated for another path are not reused); the network gate lets AMD through when mma_wmma16 is on.
  • docs: gpu/vk/VULKAN.md, a section on the path and its log lines.
  • tests: tests/test_wmma16.py and tests/test_wmma16_spirv_val.py (new).

How it works, step by step:

  1. Each lane of a 32-lane subgroup holds A (16x16), B (16x8) and C (16x8) fragments in the console's register order.
  2. The lanes store them into their subgroup's tile at the PTX m16n8k16 positions; B and C are padded to 16 columns with zeros. A subgroup barrier (OpControlBarrier, Subgroup scope, AcquireRelease, WorkgroupMemory, MakeAvailable, MakeVisible) follows.
  3. OpCooperativeMatrixLoadKHR loads A, B's K half and C as 16x16x16 matrices; OpCooperativeMatrixMulAddKHR runs once or twice (one K=16 multiply with mma_wmma16_fast, else two K=8 halves); OpCooperativeMatrixStoreKHR writes D back to the tile. The loads and the store carry NonPrivatePointer.
  4. After another barrier each lane reads its two result words from its PTX positions, so the device's own lane layout of the matrices never matters.
  5. Cooperative matrix operations need every lane of the subgroup: the path is used only where pin_subgroup() (vk_draw.cpp, as on main) creates the stage with REQUIRE_FULL_SUBGROUPS and requiredSubgroupSize = 32, i.e. a compute stage whose X width is a multiple of 32 on a device with computeFullSubgroups.

The Vulkan rules this relies on, from validusage.json:

  • VUID-VkPipelineShaderStageCreateInfo-flags-02785: "If flags has the VK_PIPELINE_SHADER_STAGE_CREATE_REQUIRE_FULL_SUBGROUPS_BIT flag set, the computeFullSubgroups feature must be enabled": met, pin_subgroup() sets it only with computeFullSubgroups, and wmma16_init() requires it.
  • VUID-VkPipelineShaderStageCreateInfo-pNext-02757: "If a VkPipelineShaderStageRequiredSubgroupSizeCreateInfo structure is included in the pNext chain, and flags has the VK_PIPELINE_SHADER_STAGE_CREATE_REQUIRE_FULL_SUBGROUPS_BIT flag set, the local workgroup size in the X dimension of the pipeline must be a multiple of VkPipelineShaderStageRequiredSubgroupSizeCreateInfo::requiredSubgroupSize": met, the flag is set only for X % 32 == 0, and DefineWmma16MatrixFunction() refuses any other width.
  • VUID-VkPipelineShaderStageCreateInfo-flags-08988: "If flags includes VK_PIPELINE_SHADER_STAGE_CREATE_REQUIRE_FULL_SUBGROUPS_BIT, stage must be one of VK_SHADER_STAGE_MESH_BIT_EXT, VK_SHADER_STAGE_TASK_BIT_EXT, or VK_SHADER_STAGE_COMPUTE_BIT": met, pin_subgroup() gets a workgroup width only for compute stages.
  • The SPIR-V module (capabilities, the explicit workgroup layout, the cooperative matrix instructions and their memory operands): checked by spirv-val --target-env vulkan1.3 on five modules built from the real DefineWmma16MatrixFunction() (Measurements).

Before (coopmat_init() in vk_coopmat.cpp):

    bool shape = false;
    for (const auto& p : props)
        shape |= p.MSize == 16 && p.NSize == 8 && p.KSize == 16 && p.AType == VK_COMPONENT_TYPE_FLOAT16_KHR &&
                 p.BType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.CType == VK_COMPONENT_TYPE_FLOAT16_KHR &&
                 p.ResultType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.scope == VK_SCOPE_SUBGROUP_KHR;

After:

    bool shape = false, shape16 = false;
    for (const auto& p : props) {
        const bool f16 = p.AType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.BType == VK_COMPONENT_TYPE_FLOAT16_KHR &&
                         p.CType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.ResultType == VK_COMPONENT_TYPE_FLOAT16_KHR &&
                         p.scope == VK_SCOPE_SUBGROUP_KHR;
        shape |= f16 && p.MSize == 16 && p.NSize == 8 && p.KSize == 16;
        shape16 |= f16 && p.MSize == 16 && p.NSize == 16 && p.KSize == 16;
    }
    if (!shape && shape16) {   // (AMD RDNA 3/4: WMMA 16x16x16 only)
        wmma16_init();
        return;
    }

DefineWmma16MatrixFunction(), its gate:

    if (!mma_wmma16_full_subgroups || program.workgroup_size[0] == 0 || program.workgroup_size[0] % 32 != 0) return false;
    ...
    if (mma_wmma16_shared_limit != 0 && own_bytes + words * 4 + 4 > mma_wmma16_shared_limit) return false;

Measurements

Verdict: it adds the path: tests/test_wmma16.py and tests/test_wmma16_spirv_val.py fail on main and pass with this PR, in both interleaved runs; every WMMA16 module passes spirv-val. No frame-time numbers: the speed in the game was not measured side by side with main.

Hardware: AMD Radeon RX 9070 XT, Windows; the tests themselves are game-free (they read source, build cut-out fixtures and validate SPIR-V, they do not use the GPU); Python 3.12, clang++ (llvm-mingw), spirv-val from SPIRV-Tools v2025.1
Scene: none, game-free
Method: both tests from this PR with --source set to each side's checkout, 2 runs each, interleaved (main, PR, main, PR); the result is pass/fail, so the median is the result both runs gave
How to reproduce: python -I -B tests/test_wmma16.py --source <checkout> and python -I -B tests/test_wmma16_spirv_val.py --source <checkout> --headers <SPIR-V headers> --cxx clang++ --spirv-val <spirv-val> for each side

Before: main @ cefce0d1 After: this PR @ 4aa76728
tests/test_wmma16.py, run 1 fails passes
tests/test_wmma16_spirv_val.py, run 1 fails passes
tests/test_wmma16.py, run 2 fails passes
tests/test_wmma16_spirv_val.py, run 2 fails passes
Median fails passes
run 1, main @ cefce0d1:
FAIL: Sirit has no OpCooperativeMatrixStoreKHR
exit 1
FAIL: emit_spirv_warp.cpp has no DefineWmma16MatrixFunction
exit 1
run 1, this PR @ 4aa76728:
HMMA_nx2 through 16x16x16 cooperative matrices: wired, and the tiles give A*B+C at the console's positions: ok
exit 0
(five modules) spirv-val ok
48 lanes wide: software multiply
no full subgroups: software multiply
PASS: every WMMA16 module is valid SPIR-V for Vulkan 1.3, and partial subgroups take the software multiply
exit 0
run 2, main @ cefce0d1:
(the same two failures)
exit 1
run 2, this PR @ 4aa76728:
(the same lines as run 1)
exit 0

What the tests check:

  • test_wmma16.py: Sirit has the store and the load with a memory operand; coopmat_init() looks for 16x16x16 and calls wmma16_init(), which needs computeFullSubgroups; the translator stamp and the network gate know mma_wmma16; the tile positions, run on the CPU, give A*B+C at the console's positions.
  • test_wmma16_spirv_val.py: the real DefineWmma16MatrixFunction() built into a Sirit fixture for five programs (one and two subgroups; with the program's own shared memory and an explicit layout; one and two multiplies); each module passes spirv-val --target-env vulkan1.3; a 48-lane-wide workgroup and a device without full subgroups take the software multiply.

Flags and settings

No new switch: the two existing ones below turn this path off.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
RX_VK_NATIVE_MMA=0 existing: no native matrix multiplies software on RDNA 3/4 software (turns this path off) the old path not measured leave unset
RX_VK_MMA_SOFTWARE=1 existing: matrix multiplies in software software software the old path not measured leave unset

Testing

Tested: on this branch, Windows, Python 3.12, clang++ (llvm-mingw), spirv-val (SPIRV-Tools v2025.1): test_wmma16 and test_wmma16_spirv_val pass (both fail on main).

Built and played: this commit builds on Windows (llvm-mingw back end) and on Linux (longshot_vk.so, without and with AMD FSR 4). Played on Windows, AMD Radeon RX 9070 XT: the game runs normally.

Not tested: reading the game log for the startup test's result on this GPU; RDNA 3 (only RDNA 4 was run); the frame time against the software path. Linux was built, not run.

Must be tested: None: RDNA 3 has the same 16x16x16 shape, and any device where the startup test fails takes the software multiply.

Risks

What could break: on RDNA 3/4 the neural-network shaders now run on cooperative matrices instead of subgroup shuffles; a driver bug in them would show as wrong output there, which the startup test (a full 16x16 multiply checked against the CPU) is meant to catch. Programs whose workgroup X width is not a multiple of 32, or whose tiles do not fit in shared memory, keep the software multiply.

Turning it off: RX_VK_NATIVE_MMA=0 or RX_VK_MMA_SOFTWARE=1, or revert the commit.

Compatibility: On RDNA 3/4 the translator stamp changes, so the translated shaders are made again once (the shader cache rebuilds); nothing changes on other GPUs.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched). (no issue fits)
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). (the checks that read the changed code, on this PR, Windows: test_wmma16, test_wmma16_spirv_val pass)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here. (tests/test_wmma16.py, tests/test_wmma16_spirv_val.py)
  • A check under tests/ that reads changed source updated in this PR, not weakened. (both tests added; no check removed)
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (built on Windows with llvm-mingw and on Linux, longshot_vk.so without and with AMD FSR 4; no OS-specific code or paths)
  • Docs updated. (gpu/vk/VULKAN.md)
**Type:** feat · **Risk:** medium · **Tested on:** Windows, AMD Radeon RX 9070 XT · **Needs:** none ## Why On AMD RDNA 3 and 4 the driver's cooperative matrices offer only the 16x16x16 f16 shape, not the 16x8x16 one the console's `HMMA_nx2` maps to, so `coopmat_init()` sent every matrix multiply (the game's neural-network shaders) to the slow software path on subgroup shuffles. ## How to reproduce **Seen on:** AMD Radeon RX 9070 XT, Windows (RDNA 4): `vkGetPhysicalDeviceCooperativeMatrixPropertiesKHR` lists 16x16x16 f16 but no 16x8x16, and the log says the matrix multiplies run in software **Settings and switches:** defaults **Steps:** game-free: `python -I -B tests/test_wmma16.py --source <main checkout>` and `python -I -B tests/test_wmma16_spirv_val.py --source <main checkout> --headers <SPIR-V headers> --cxx clang++ --spirv-val <spirv-val>` (the tests from this PR) **Expected:** on a GPU with only 16x16x16 cooperative matrices, `HMMA_nx2` runs through them, gives A*B+C at the console's positions, and its SPIR-V is valid for Vulkan 1.3 **Actual:** on main Sirit has no `OpCooperativeMatrixStoreKHR` and there is no 16x16x16 path; both tests fail ## What changed Where the GPU has only 16x16x16 f16 cooperative matrices and full 32-lane compute subgroups, each `HMMA_nx2` goes through a per-subgroup tile of shared memory: the lanes write their console fragments at their PTX m16n8k16 positions, one or two 16x16x16 multiplies run, and the lanes read their results back. Elsewhere nothing changes. - vk: `coopmat_init()` in `gpu/vk/vk_coopmat.cpp` also looks for the 16x16x16 f16 subgroup shape; with it and without 16x8x16, new `wmma16_init()` needs `shaderFloat16`, 32-lane compute subgroups and `computeFullSubgroups`, then runs `test_multiply_wmma16()` (a 16x16 multiply on the GPU checked against the CPU, 128 elements). Any miss takes the software multiply and logs why. - shader recompiler: new `DefineWmma16MatrixFunction()` in `emit_spirv_warp.cpp`; `EmitMatrixMulAddF16()` calls it when the program has it. It returns false (software multiply) unless full subgroups are on and the workgroup X width is a multiple of 32, and unless the tiles fit in the device's shared memory limit (`mma_wmma16_shared_limit`). With `explicit_layout` the tiles are a Block at an offset after the program's own shared memory, `Aliased`. - shader recompiler: `mma_layout.h` gains `mma_wmma16`, `mma_wmma16_fast`, the tile size, the shared limit and `mma_wmma16_full_subgroups`; `spirv_emit_context.h` gains `wmma16_function`; `emit_spirv.cpp` defines it. - Sirit: `OpCooperativeMatrixLoadKHR` with a memory operand and `OpCooperativeMatrixStoreKHR`. - vk: `vk_shader.cpp`: the translator stamp includes `mma_wmma16` and `mma_wmma16_fast` (programs translated for another path are not reused); the network gate lets AMD through when `mma_wmma16` is on. - docs: `gpu/vk/VULKAN.md`, a section on the path and its log lines. - tests: `tests/test_wmma16.py` and `tests/test_wmma16_spirv_val.py` (new). How it works, step by step: 1. Each lane of a 32-lane subgroup holds A (16x16), B (16x8) and C (16x8) fragments in the console's register order. 2. The lanes store them into their subgroup's tile at the PTX m16n8k16 positions; B and C are padded to 16 columns with zeros. A subgroup barrier (`OpControlBarrier`, Subgroup scope, AcquireRelease, WorkgroupMemory, MakeAvailable, MakeVisible) follows. 3. `OpCooperativeMatrixLoadKHR` loads A, B's K half and C as 16x16x16 matrices; `OpCooperativeMatrixMulAddKHR` runs once or twice (one K=16 multiply with `mma_wmma16_fast`, else two K=8 halves); `OpCooperativeMatrixStoreKHR` writes D back to the tile. The loads and the store carry `NonPrivatePointer`. 4. After another barrier each lane reads its two result words from its PTX positions, so the device's own lane layout of the matrices never matters. 5. Cooperative matrix operations need every lane of the subgroup: the path is used only where `pin_subgroup()` (`vk_draw.cpp`, as on main) creates the stage with `REQUIRE_FULL_SUBGROUPS` and `requiredSubgroupSize = 32`, i.e. a compute stage whose X width is a multiple of 32 on a device with `computeFullSubgroups`. The Vulkan rules this relies on, from `validusage.json`: - VUID-VkPipelineShaderStageCreateInfo-flags-02785: "If flags has the VK_PIPELINE_SHADER_STAGE_CREATE_REQUIRE_FULL_SUBGROUPS_BIT flag set, the computeFullSubgroups feature must be enabled": met, `pin_subgroup()` sets it only with `computeFullSubgroups`, and `wmma16_init()` requires it. - VUID-VkPipelineShaderStageCreateInfo-pNext-02757: "If a VkPipelineShaderStageRequiredSubgroupSizeCreateInfo structure is included in the pNext chain, and flags has the VK_PIPELINE_SHADER_STAGE_CREATE_REQUIRE_FULL_SUBGROUPS_BIT flag set, the local workgroup size in the X dimension of the pipeline must be a multiple of VkPipelineShaderStageRequiredSubgroupSizeCreateInfo::requiredSubgroupSize": met, the flag is set only for X % 32 == 0, and `DefineWmma16MatrixFunction()` refuses any other width. - VUID-VkPipelineShaderStageCreateInfo-flags-08988: "If flags includes VK_PIPELINE_SHADER_STAGE_CREATE_REQUIRE_FULL_SUBGROUPS_BIT, stage must be one of VK_SHADER_STAGE_MESH_BIT_EXT, VK_SHADER_STAGE_TASK_BIT_EXT, or VK_SHADER_STAGE_COMPUTE_BIT": met, `pin_subgroup()` gets a workgroup width only for compute stages. - The SPIR-V module (capabilities, the explicit workgroup layout, the cooperative matrix instructions and their memory operands): checked by `spirv-val --target-env vulkan1.3` on five modules built from the real `DefineWmma16MatrixFunction()` (Measurements). Before (`coopmat_init()` in `vk_coopmat.cpp`): ```cpp bool shape = false; for (const auto& p : props) shape |= p.MSize == 16 && p.NSize == 8 && p.KSize == 16 && p.AType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.BType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.CType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.ResultType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.scope == VK_SCOPE_SUBGROUP_KHR; ``` After: ```cpp bool shape = false, shape16 = false; for (const auto& p : props) { const bool f16 = p.AType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.BType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.CType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.ResultType == VK_COMPONENT_TYPE_FLOAT16_KHR && p.scope == VK_SCOPE_SUBGROUP_KHR; shape |= f16 && p.MSize == 16 && p.NSize == 8 && p.KSize == 16; shape16 |= f16 && p.MSize == 16 && p.NSize == 16 && p.KSize == 16; } if (!shape && shape16) { // (AMD RDNA 3/4: WMMA 16x16x16 only) wmma16_init(); return; } ``` `DefineWmma16MatrixFunction()`, its gate: ```cpp if (!mma_wmma16_full_subgroups || program.workgroup_size[0] == 0 || program.workgroup_size[0] % 32 != 0) return false; ... if (mma_wmma16_shared_limit != 0 && own_bytes + words * 4 + 4 > mma_wmma16_shared_limit) return false; ``` ## Measurements **Verdict:** it adds the path: `tests/test_wmma16.py` and `tests/test_wmma16_spirv_val.py` fail on main and pass with this PR, in both interleaved runs; every WMMA16 module passes `spirv-val`. No frame-time numbers: the speed in the game was not measured side by side with main. Hardware: AMD Radeon RX 9070 XT, Windows; the tests themselves are game-free (they read source, build cut-out fixtures and validate SPIR-V, they do not use the GPU); Python 3.12, clang++ (llvm-mingw), spirv-val from SPIRV-Tools v2025.1 Scene: none, game-free Method: both tests from this PR with `--source` set to each side's checkout, 2 runs each, interleaved (main, PR, main, PR); the result is pass/fail, so the median is the result both runs gave How to reproduce: `python -I -B tests/test_wmma16.py --source <checkout>` and `python -I -B tests/test_wmma16_spirv_val.py --source <checkout> --headers <SPIR-V headers> --cxx clang++ --spirv-val <spirv-val>` for each side | | Before: main @ cefce0d1 | After: this PR @ 4aa76728 | |---|---|---| | `tests/test_wmma16.py`, run 1 | fails | passes | | `tests/test_wmma16_spirv_val.py`, run 1 | fails | passes | | `tests/test_wmma16.py`, run 2 | fails | passes | | `tests/test_wmma16_spirv_val.py`, run 2 | fails | passes | | Median | fails | passes | ``` run 1, main @ cefce0d1: FAIL: Sirit has no OpCooperativeMatrixStoreKHR exit 1 FAIL: emit_spirv_warp.cpp has no DefineWmma16MatrixFunction exit 1 run 1, this PR @ 4aa76728: HMMA_nx2 through 16x16x16 cooperative matrices: wired, and the tiles give A*B+C at the console's positions: ok exit 0 (five modules) spirv-val ok 48 lanes wide: software multiply no full subgroups: software multiply PASS: every WMMA16 module is valid SPIR-V for Vulkan 1.3, and partial subgroups take the software multiply exit 0 run 2, main @ cefce0d1: (the same two failures) exit 1 run 2, this PR @ 4aa76728: (the same lines as run 1) exit 0 ``` What the tests check: - `test_wmma16.py`: Sirit has the store and the load with a memory operand; `coopmat_init()` looks for 16x16x16 and calls `wmma16_init()`, which needs `computeFullSubgroups`; the translator stamp and the network gate know `mma_wmma16`; the tile positions, run on the CPU, give A*B+C at the console's positions. - `test_wmma16_spirv_val.py`: the real `DefineWmma16MatrixFunction()` built into a Sirit fixture for five programs (one and two subgroups; with the program's own shared memory and an explicit layout; one and two multiplies); each module passes `spirv-val --target-env vulkan1.3`; a 48-lane-wide workgroup and a device without full subgroups take the software multiply. ### Flags and settings No new switch: the two existing ones below turn this path off. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `RX_VK_NATIVE_MMA=0` | existing: no native matrix multiplies | software on RDNA 3/4 | software (turns this path off) | the old path | not measured | leave unset | | `RX_VK_MMA_SOFTWARE=1` | existing: matrix multiplies in software | software | software | the old path | not measured | leave unset | ## Testing **Tested:** on this branch, Windows, Python 3.12, clang++ (llvm-mingw), spirv-val (SPIRV-Tools v2025.1): `test_wmma16` and `test_wmma16_spirv_val` pass (both fail on main). **Built and played:** this commit builds on Windows (llvm-mingw back end) and on Linux (`longshot_vk.so`, without and with AMD FSR 4). Played on Windows, AMD Radeon RX 9070 XT: the game runs normally. **Not tested:** reading the game log for the startup test's result on this GPU; RDNA 3 (only RDNA 4 was run); the frame time against the software path. Linux was built, not run. **Must be tested:** None: RDNA 3 has the same 16x16x16 shape, and any device where the startup test fails takes the software multiply. ## Risks **What could break:** on RDNA 3/4 the neural-network shaders now run on cooperative matrices instead of subgroup shuffles; a driver bug in them would show as wrong output there, which the startup test (a full 16x16 multiply checked against the CPU) is meant to catch. Programs whose workgroup X width is not a multiple of 32, or whose tiles do not fit in shared memory, keep the software multiply. **Turning it off:** `RX_VK_NATIVE_MMA=0` or `RX_VK_MMA_SOFTWARE=1`, or revert the commit. **Compatibility:** On RDNA 3/4 the translator stamp changes, so the translated shaders are made again once (the shader cache rebuilds); nothing changes on other GPUs. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). (no issue fits) - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). (the checks that read the changed code, on this PR, Windows: `test_wmma16`, `test_wmma16_spirv_val` pass) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. (`tests/test_wmma16.py`, `tests/test_wmma16_spirv_val.py`) - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. (both tests added; no check removed) - [x] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (built on Windows with llvm-mingw and on Linux, `longshot_vk.so` without and with AMD FSR 4; no OS-specific code or paths) - [x] Docs updated. (`gpu/vk/VULKAN.md`)
Cards whose cooperative matrices offer only 16x16x16 f16 ran HMMA_nx2 in software. Sirit gains OpCooperativeMatrixStoreKHR and a Load with a memory operand; vk_coopmat.cpp probes the tile multiply; emit_spirv_warp.cpp defines DefineWmma16MatrixFunction; the program cache stamp and the network gate know mma_wmma16. tests/test_wmma16.py checks the wiring and the tile layout.

Assisted-by: GitHub Copilot
Programs whose workgroup X width is not a multiple of 32, or devices without computeFullSubgroups, take the software multiply.

Assisted-by: GitHub Copilot
test(vk): validate the WMMA16 multiply's SPIR-V with spirv-val
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
4aa76728c6
Assisted-by: GitHub Copilot
Author
Contributor

Tested in game: Windows 11 IoT Enterprise LTSC 24H2 (build 26100.9711), AMD Radeon RX 9070 XT, AMD Software: Adrenalin Edition 26.10.1: played with this change; the game runs normally, no rendering errors or crashes seen.

Tested in game: Windows 11 IoT Enterprise LTSC 24H2 (build 26100.9711), AMD Radeon RX 9070 XT, AMD Software: Adrenalin Edition 26.10.1: played with this change; the game runs normally, no rendering errors or crashes seen.
jcabarca changed title from WIP: feat(vk): matrix multiplies through WMMA 16x16x16 workgroup tiles on AMD RDNA 3/4 to feat(vk): matrix multiplies through WMMA 16x16x16 workgroup tiles on AMD RDNA 3/4 2026-11-16 04:14:16 +00:00
Author
Contributor

Rebased onto main 6c8d6e6a to clear the conflict with #1086 (software MMA unroll).

  • mma_layout.h: kept #1086's software forms and this PR's HMMA_nx2 path together.
  • translator_stamp (vk_shader.cpp): #1086 now uses bits 256, 512 and 1024 for the software form and quad broadcasts, so this PR's two bits moved to 2048 (mma_wmma16) and 4096 (mma_wmma16_fast); tests/test_wmma16.py checks the new values.
  • When the tiles do not fit, the software fallback keeps form 0 (the loop), as before.

After the rebase: test_wmma16 fails on main 6c8d6e6a and passes here; test_wmma16_spirv_val and test_mma_software also pass here.

Rebased onto main 6c8d6e6a to clear the conflict with #1086 (software MMA unroll). - mma_layout.h: kept #1086's software forms and this PR's HMMA_nx2 path together. - translator_stamp (vk_shader.cpp): #1086 now uses bits 256, 512 and 1024 for the software form and quad broadcasts, so this PR's two bits moved to 2048 (mma_wmma16) and 4096 (mma_wmma16_fast); tests/test_wmma16.py checks the new values. - When the tiles do not fit, the software fallback keeps form 0 (the loop), as before. After the rebase: test_wmma16 fails on main 6c8d6e6a and passes here; test_wmma16_spirv_val and test_mma_software also pass here.
jcabarca force-pushed feat/vk-wmma16 from 4aa76728c6
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to 7b54ddcc4f
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
2026-11-16 12:34:26 +00:00
Compare
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'feat/vk-wmma16:jcabarca-feat/vk-wmma16'
git switch 'jcabarca-feat/vk-wmma16'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1038
No description provided.