fix(vk): dark lighting and glowing bloom on AMD: 64-lane graphics warps and Lichii #273's fragment shuffle fallback #1507
No reviewers
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!1507
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/amd-dark-lighting-64-lane-warps-shuffles"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Fixes dark lighting and white-glowing objects (a basket in Link's house, bloom) on AMD's Windows driver. Two causes: 64-lane graphics waves that the translator treated as 32-lane, and AMD's broken fragment-stage subgroup shuffles (Lichii #273).
Part of splitting up #1023. Needs testers with RDNA2 or newer: I only have RDNA1.
Problem
requiredSubgroupSizeStages= 0x20 on the RX 5500M). Fragment and vertex stages run at 64. The translator assumed 32 lanes everywhere (warp_size_potentially_larger_than_guestfalse), so the ballots and shuffles in the game's fragment shaders ran across both halves of a 64-lane wave, giving dark lighting and glowing bloom.Change
make_profilesetswarp_size_potentially_larger_than_guest, andmax_subgroup_sizeto the device's largest, only when a graphics stage can't be pinned and the default is wider than 32. Eden's long path then keeps each 32-lane guest warp to its half. NVIDIA (32 lanes) is unchanged.profile_graphics): a graphics stage the GPU does pin to 32 lanes (pin_subgroup) keeps main's short path, so only stages that really run at 64 lanes change:0x20, compute only): vertex and fragment take the long path.0xf0, reported from an RX 7800 XT in this thread): only the vertex stage. Fragment, task and mesh keep main's translation.profile_pinned), including the mesh fallback's compute stages, because they are pinned to 32. Giving every compute program the long path changed its SPIR-V, missed the driver's cache, and AMD recompiled shaders for hours at start.has_broken_spirv_fragment_shuffle), and so doesSHFL.BFLY. The quad swap stays native: its programs have neither the lane-id input nor the Ballot capability the fallback needs, and rerouting it crashed shader preparation.RX_VK_WARP64=0|1andRX_VK_FRAG_SHUFFLE_FALLBACK=0|1override both, whatever the GPU, to help bisect reports.The depth-compare part of #273 is already on main (#834); only the test checks it here.
Risk
On RDNA1, this changes the vertex and fragment programs' translation, so those players re-translate on their first start. On RDNA2/RDNA3, only vertex programs change (plus the shuffle fallback below), because their other graphics stages are pinned to 32 lanes. An RX 6000/7000 result would still be welcome before merging. NVIDIA and Intel are unaffected by the warp part. The shuffle part is limited to
VK_DRIVER_ID_AMD_PROPRIETARY.Tests
tests/test_amd_fragment_warp.py(new) checks the profile, the per-stage choice (pinned stages keep the short path;RX_VK_WARP64still sets them all), the pinned profile's uses, the stamp, the shuffle fallback and the depth-compare rule.Tested on
Data point for the testers asked for here, from an RX 7800 XT (RDNA3), Windows 11, driver 0x8001a0, v2.0.12:
The log line is
vk: subgroups: default 64 lanes, 32-64, pinned to 32 on stages 0xf0. By the Vulkan stage bits that is fragment (0x10), compute (0x20), task (0x40) and mesh (0x80): on this card the fragment stage can be pinned to 32 lanes, and only the vertex stage cannot. So the "64-lane graphics warps" part of this PR may not apply to RDNA3 at all; the broken fragment-stage shuffles of AMD's proprietary driver (Lichii #273) might still be relevant. On the RX 5500M the value is 0x20 (compute only), which is why it matters there.I have not tested this PR on this card. If it helps I can build it on top of v2.0.12 and report the first-start preparation time (the PR says graphics programs are translated again on AMD), whether the picture changes anywhere, and the log lines, using
RX_VK_WARP64=0|1andRX_VK_FRAG_SHUFFLE_FALLBACK=0|1to separate the two parts.Thanks, that's exactly the data this needs.
On your card this PR does turn the 64-lane path on. The condition is
subgroup_size > 32and not every graphics stage pinnable to 32 lanes ((sg32_stages & VK_SHADER_STAGE_ALL_GRAPHICS) != VK_SHADER_STAGE_ALL_GRAPHICS), and with0xf0the vertex stage isn't. So on RDNA3 the vertex and fragment stages both get the long warp path, the fragment stage even though it runs at 32 lanes there. The long path is still correct at 32 lanes, so this should cost translation time, not correctness. Narrowing it to the stages that can't be pinned is a reasonable follow-up, and your results would show whether it's worth it.The two parts are independent. The most useful runs, set in
hookshot_env.txt:RX_VK_WARP64RX_VK_FRAG_SHUFFLE_FALLBACKFor each run: the first-start preparation time, and whether lighting, bloom or anything else looks different. On the RX 5500M it showed most in Kokiri Forest and in Link's house (a basket drew white). Both switches are in the program cache's stamp, so each run re-translates once.
First result for run 3 (both parts, the PR's default) on an RX 7800 XT (RDNA3), Windows 11, driver 0x8001a0,
RX_VK_WARP64andRX_VK_FRAG_SHUFFLE_FALLBACKunset,RX_VK_PRESENT_QUEUE=1. Thanks for the correction above: you are right that the long warp path is on here (the vertex stage is not in0xf0); my earlier note that it might not apply was wrong.Build: v2.0.12 plus this PR (merged with no conflicts), plus PR #1130 at
6f76660and two local memory edits for AMD (image pool preallocation, larger capture buffer classes), so it is not a clean main + PR build.First start:
prepared 38809 shader programs (46556 new, 111 failed, ...) in 65.4 s(111 failed is the same count as without the PR); the log's total translation time was 475 s across threads; from launch to a steady 60 fps at the title (headless, including the driver's pipelines) it took about 5 minutes.What I saw (normal play, about 12 minutes): lighting and bloom looked normal in Kokiri Forest, in Link's house and in front of Ganon's Castle as adult Link. No white-glowing objects, no too-dark areas, no errors, no device lost. I did not see those problems on this card before the PR either, so "normal" shows no regression, not that the PR fixes anything here.
Frame times (not conclusive): mean 57.7 game fps (median 60, 5% worst 46), GPU 4.9 ms a frame on average, p95 9.5 ms. 28 submits of 40 ms or more (15 of 100 ms or more, longest 1.4 s) against 12 in a 9.6-minute session on the same GPU without this PR; the dominant part of the long ones is
drawin both, and the routes differ (this one had more area loads), so I am not reading it as a cost of the PR yet.Not done yet: runs 1 (0/0, main's behaviour) and 2 (0/1), on the same route, to separate the two parts and the frame times. I will post them here.
Following your point, I've pushed a narrower version (
6943ac0). The 64-lane path now applies per stage, only to the graphics stages the GPU can't pin to 32 lanes. On your RX 7800 XT that means only the vertex stage changes; fragment, task and mesh keep main's translation, while the AMD fragment shuffle fallback still applies.The test plan stays the same. In run 3 (both parts on), only your vertex programs are re-translated with the long path, so the first-start time should come out close to run 2's.
Run 1 (
RX_VK_WARP64=0,RX_VK_FRAG_SHUFFLE_FALLBACK=0, main's behaviour) on the RX 7800 XT, same build and settings as my run 3 above (RX_VK_PRESENT_QUEUE=1), same route (Kokiri Forest, Link's house, in front of Ganon's Castle as adult). This is the head before your narrower6943ac0: my run 3 was ondae962a, so run 3 has to be redone; I will rebuild on6943ac0and post runs 2 and 3 with it.Picture: normal in all three places in both runs. Nothing white, nothing too dark, no errors, no device lost.
First start:
prepared 38809 shader programs ... in 58.4 swith 10091 stage libraries; run 3 ondae962awas 65.4 s with 0 stage libraries listed in that line.Frame times over the same first 26,400 presents (about 7.3 minutes), one run each, run 3 done first:
dae962a)SLOW driver compilelines in playI would not read this as a cost of the PR: one run each, the order was fixed, and the PR changes the translated shaders, so run 3 also compiled many pipelines for the first time in play (21 against 5 slow driver compiles), which can explain part of the hitches. I cannot separate that from a real cost with these two runs. The longest hitch is the other way round (3.2 s in run 1).
Thanks for narrowing it per stage. For a fair comparison I will repeat run 3 on
6943ac0after a warm-up pass, and add run 2 (0 / 1).Read through the diff against the description and it matches. The per-stage split looks right:
profile_pinned()for compute and the mesh fallback,profile_graphics(stage)for graphics,make_profile()only going wide when some graphics stage can't be pinned. The cache stamp covers the warp choice,max_subgroup_size, the shuffle flag andsg32_stages, so a stale cache won't be reused across GPUs. Leaving the quad swap native (no lane-id input or Ballot capability in those programs) is explained in the code. CI is green, and the new test is picked up by run_all's glob.Nit:
RX_VK_WARP64is read inmake_profile,profile_graphicsand the stamp viaprofile(), which is fine, but onlyprofile_graphicstreats "set to anything" as forced whilemake_profilecheckse[0] == '1'.RX_VK_WARP64=0still works as intended (long path off everywhere), just a bit asymmetric.As you say in the description, RDNA2/3 results for the shuffle fallback are still welcome, but nothing I can see in the code blocks merging.
Thanks for runs 1 and 3. #1507 has merged since, including the per-stage change (
6943ac0, on main asc433687), so it's main's default now.Your run 3 had more long submits than run 1 (15 against 7 of 100 ms or more) and more slow driver compiles (21 against 5). But it was on the wider version, which put the long warp path on every fragment shader with warp operations. On your card the merged version gives it only to the vertex stage, and only 59 of the 17,233 distinct vertex shaders in my program cache use warp operations.
The other part that could still cost RDNA3 something is the fragment shuffle fallback. On AMD's Windows driver it turns each fragment-shader shuffle into 32 broadcasts, and pinning the fragment stage to 32 lanes doesn't turn it off. Your run 1 had it off and the picture was normal, and the white basket it fixes has only been seen on an RDNA1 card so far. So it may only be needed where the fragment stage runs at 64 lanes.
Could you test that on current main? No PR build is needed. Use the same route both times, each after a warm-up pass. The switch is in the program cache's stamp, so changing it re-translates once.
hookshot_env.txtRX_VK_FRAG_SHUFFLE_FALLBACK=0For each run, the most useful numbers are the frame times (game fps mean and 5% worst, submits of 100 ms or more) and the count of
SLOW driver compilelines. Please also note anything that looks different, especially Link's house (the basket) and Kokiri Forest's plants. If B is faster and draws the same, I'll limit the fallback to fragment stages that run at 64 lanes. That keeps it for RDNA1 and leaves RDNA2/3 as they were before #1507.