fix(vk): pin compute, task and mesh stages to 32 lanes only within maxComputeWorkgroupSubgroups #1556
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1556
Loading…
Reference in a new issue
No description provided.
Delete branch "jcabarca/Longshot:fix/vk-subgroup-pin-limit"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Windows 11 IoT Enterprise LTSC 24H2 (build 26100.9711), AMD Radeon RX 9070 XT, AMD Software: Adrenalin Edition 26.10.1 · Needs: none
TL;DR
On GPUs that can fix a shader to 32-lane groups, the emulator fixed every compute, task and mesh shader that way, even when the shader's workgroup is larger than 32 lanes times the number of groups the GPU allows in one workgroup. Vulkan forbids that, and such a pipeline can fail to build or misbehave. This PR pins a stage only when its workgroup fits, and translates the larger compute programs for a non-pinned subgroup. A new test fails on main and passes here.
Breakdown
Why
pin_subgroupchainsrequiredSubgroupSize32 on every compute, task and mesh stage the GPU lists, without readingmaxComputeWorkgroupSubgroups, which breaks VUID-VkPipelineShaderStageCreateInfo-pNext-02756 for any workgroup larger than 32 times that limit. See #1448, which changes the sameprofile_pinned().How to reproduce
Seen on: not seen in the game: found by reading
pin_subgroup()ingpu/vk/vk_draw.cpp. It applies to any GPU that lists compute, task or mesh inrequiredSubgroupSizeStageswithmaxComputeWorkgroupSubgroupsbelow the workgroup's invocations divided by 32 (the limit has no required minimum)Settings and switches: defaults
Steps: game-free:
python -I -B tests/test_subgroup_pin_limit.py --cxx clang++on a main checkout with this PR'stests/test_subgroup_pin_limit.py; it compiles the pin code against stand-in Vulkan calls and a device with 16 subgroups per workgroupExpected: a 1024 x 1 x 1 compute workgroup on that device gets no
VkPipelineShaderStageRequiredSubgroupSizeCreateInfoActual: on main it is pinned:
CHECK failed (1024 x 1 x 1, 16 subgroups): (ci.stage.pNext != nullptr) == pinnedWhat changed
The 32-lane pin now also checks the stage's workgroup against
maxComputeWorkgroupSubgroups, and the translator emits an unpinned compute program for a non-pinned subgroup.gpu/vk/vk_internal.h: newdev.max_workgroup_subgroupsand an inlinepins_32(stage, invocations = 64): false unless the stage is indev.sg32_stages; for compute, task and mesh, true only wheninvocations <= 32 * max_workgroup_subgroups.gpu/vk/vk_main.cpp:dev.max_workgroup_subgroupsis read fromVkPhysicalDeviceSubgroupSizeControlProperties::maxComputeWorkgroupSubgroups(core in 1.3, already in main's properties chain).gpu/vk/vk_draw.cpp:pin_subgroup()takes the workgroup's invocations and usespins_32; the compute pipelines pass x * y * z.gpu/vk/vk_shader.cpp:profile_pinned(invocations)returns the pinned profile only whenpins_32holds, elseprofile();profile_graphics()usespins_32(stage);translate_into()passes the program's workgroup;translator_stamp()adds the limit only when 32 times it is below 1024, so other GPUs keep their caches.tests/test_subgroup_pin_limit.py;test_amd_fragment_warp,test_amd_subgroup_profiles,test_compute_pin,test_pipeline_state_computesandtest_precompile_compute_guardfollow the new signatures.The bug, step by step:
subgroupSizeControl,dev.sg32_stagesholdsrequiredSubgroupSizeStages.pin_subgroup()chainsVkPipelineShaderStageRequiredSubgroupSizeCreateInfowithrequiredSubgroupSize32 on every stage in that mask.maxComputeWorkgroupSubgroupshas no required minimum (not in the required-limits table), so on a GPU whose limit is below 32 such a workgroup exceeds 32 times the limit, and the pipeline breaks 02756.The rules (validusage.json):
REQUIRE_FULL_SUBGROUPSis set only after the pin is kept and x is a multiple of 32, as on main.Before (main):
After:
What the test checks: on devices with 32, 16 and 1 subgroups per workgroup, compute, task and mesh stages of several sizes get the pin struct only when the workgroup fits, full subgroups only when x is a multiple of 32; it also checks that the compute translation no longer takes the pinned profile whatever its workgroup.
Start reading at
pins_32ingpu/vk/vk_internal.h.Measurements
Verdict: it fixes the bug:
test_subgroup_pin_limitfails on main and passes here. Both interleaved runs gave the same result. No speed change is expected on a GPU whose limit covers 1024 invocations: it pins as before; elsewhere only the oversized workgroups lose the pin.Hardware: Windows 11 IoT Enterprise LTSC 24H2 (build 26100.9711), AMD Radeon RX 9070 XT, AMD Software: Adrenalin Edition 26.10.1; the test is game-free (it compiles a cut-out of the pin code with llvm-mingw, it does not use the GPU); Python 3.12
Scene: none, game-free
Method: this PR's test run against each side's checkout, 2 runs each, interleaved (main, PR, main, PR); the result is pass/fail, so the median is the result both runs gave
How to reproduce:
python -I -B tests/test_subgroup_pin_limit.py --cxx clang++from each side's checkout6fbeca963c96b782tests/test_subgroup_pin_limit.py, run 1tests/test_subgroup_pin_limit.py, run 2tests/test_subgroup_pin_limit.py, medianFlags and settings
None.
Testing
Tested: on this branch, Windows, Python 3.12, llvm-mingw clang++:
test_subgroup_pin_limitpasses (fails on main), and the checks that read the changed code pass:test_amd_fragment_warp,test_amd_subgroup_profiles,test_compute_pin,test_pipeline_state_computes,test_precompile_compute_guard,test_fallback_pipeline_state.Built and played: this commit builds on Windows (llvm-mingw back end) and on Linux (
longshot_vk.so, without and with AMD FSR 4). Played on Windows 11 IoT Enterprise LTSC 24H2 (build 26100.9711), AMD Radeon RX 9070 XT, AMD Software: Adrenalin Edition 26.10.1: the game runs normally.Not tested: a GPU with
maxComputeWorkgroupSubgroupsbelow 32: the RX 9070 XT's limit covers 1024 invocations, so the unpinned path did not run on a GPU; the test covers it. Linux was built, not run.Must be tested: None.
Risks
What could break: on a GPU whose limit is below 32, the larger compute programs are no longer pinned and take the long warp path, the one main already uses on a GPU without the compute pin; their shaders are translated again once (the cache key adds the limit only there). Other GPUs keep their pins and caches.
Turning it off: revert the commit.
Compatibility: Nothing changes on a GPU whose limit covers 1024 invocations. Elsewhere the shader cache is rebuilt once.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). (the checks that read the changed code, on this PR, Windows: all listed in Tested pass)third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened. (five checks follow the newpin_subgroupandprofile_pinnedsignatures)Tested in game: Windows 11 IoT Enterprise LTSC 24H2 (build 26100.9711), AMD Radeon RX 9070 XT, AMD Software: Adrenalin Edition 26.10.1: played with this change; the game runs normally, no rendering errors or crashes seen.
WIP: fix(vk): pin compute, task and mesh stages to 32 lanes only within maxComputeWorkgroupSubgroupsto fix(vk): pin compute, task and mesh stages to 32 lanes only within maxComputeWorkgroupSubgroupsRebased onto main @
7daafa7a; the related tests pass on it.3c96b78262a9762fa447View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.