WIP: perf(vk): the fog's lighting pass in 256-invocation workgroups on the Steam Deck (RX_VK_WG_SPLIT=1, off by default) #1377
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1377
Loading…
Reference in a new issue
No description provided.
Delete branch "dukenheimer/Longshot:brett-d/fog-wg-split"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: low · Tested on: Steam Deck OLED, handheld on AC (SteamOS, RADV VANGOGH, Mesa 26.2.0-devel git-5c65d4fc31); Windows, RTX 5090 (pictures only, debug switch) · Needs: a second Deck (LCD), and Deck runs with the switch on by more people
Why
On the Steam Deck the ground fog costs about 5 ms of GPU time a frame, 13–16% of the frame. One of its four programs is the lighting pass (27ee9883), and that pass runs well below the GPU's occupancy.
The cause is in Mesa 26.x's shader compiler (ACO), which the Deck's current SteamOS ships. It aims for about 10 waves per SIMD for this shader. It then rounds down to whole workgroups: one 512-invocation workgroup is 4 waves per SIMD, so 10 becomes 8, and the compiler hands out 128 registers to each thread. The same shader on Mesa 25.2 got 80 registers and 12 waves.
Splitting the workgroup in half (8x8x4, 256 invocations) lets the Deck's own compiler choose 80 registers and 12 waves again, with no spills.
See #373, which asks for an option to reduce the fog. This PR keeps the fog as it is and only makes it cheaper.
What changed
An opt-in renderer switch,
RX_VK_WG_SPLIT=1. The gate is a RADV driver whose device name says VANGOGH, with compute pinned to 32 lanes; it is not tied to a Mesa version. There, the renderer rewrites the fog lighting pass's SPIR-V to half-height workgroups. It builds that as one extra shader module and pipeline of its own, and binds it for the pass's direct dispatches, with the grid's z doubled.The translator, the translated modules, and the program cache's format and stamp are unchanged: nothing is translated or prepared again. The driver's own pipeline cache can gain the split pipeline as an entry of its own. With the switch off, nothing changes.
gpu/vk/wg_split.{h,cpp}, inwgsplit::rewrite().gpu/vk/vk_draw.cpp, inwg_split_on,wg_split_build,wg_split_forand the bind and dispatch indispatch().wgsplit::Slot.gpu/vk/vk_progcache.{h,cpp}.program_cache_code()reads a program's SPIR-V back from the cache file.wg_split_code_seenhook keeps the code a listed program was read with.gpu/vk/switches.defgains two rows,RX_VK_WG_SPLIT(attempt) andRX_VK_WG_SPLIT_ANY_GPU(diag).Measurements
Verdict: an opt-in, measured GPU saving, on this driver, this program and this shape.
Hardware: Steam Deck OLED. AMD Custom APU 0932 (4 cores, 8 threads), 16 GB; AMD Custom GPU 0932, RADV VANGOGH, Mesa 26.2.0-devel git-5c65d4fc31, compute pinned to 32 lanes. SteamOS 3.9.5 (BUILD_ID 20261023.100), kernel 7.2.7-valve2-2-neptune-72-g31504582a073.
Display and power, checked in every run counted:
profile_peak), frame generation off.Scene: the repo save kokiri-start. Kokiri Forest standing still, 45 s; then a teleport to Hyrule Field (child), turning, 30 s.
Settings: each run used its own copy of the setup's options file. Within a setup, all copies came back identical after the game's exit-time rewrite: D 7757e809…, H 4f7df9fa…. They were made from the templates hisfg0.ini (552eb6cc…) and hh.ini (872edeea…). No switch overrides a row.
RX_VK_COLD_CHECK=0, a Mesa shader cache of 5 GB per game copy, and Fossilize disabled. The arms differ only inRX_VK_WG_SPLIT.Method: one build (bda836f4) and two arms,
RX_VK_WG_SPLITunset (O) and=1(N). A warm-up run of each arm, then 3 interleaved runs each; medians and observed ranges.RX_VK_PASS_TIMING=2in a separate run per arm, for the pass's span (instrumented runs, not the fps windows).How to reproduce: the repo save kokiri-start, the switch unset and set, in turn.
RX_VK_PASS_TIMING=2from a separate run per arm.screen 1280x800 at 90 Hzandvk: swapchain 1280x800, and on N for thebound for its direct dispatchesline before the window.Baseline: this PR @ bda836f4 with the switch unset, which runs main's pipeline and dispatch for every program.
6d72ca93.544229a2. The only difference is main's #1332 (which Vulkan driver the loader picks on Windows); the split's sources are byte-identical. So the numbers are labelled bda836f4 and were not re-measured.Setup D (Brett's options, 100% render), handheld, on AC, 1280x800 at 90 Hz
† D-TN's Field timing window also had 4 pipelines built in it. It is shown for completeness, not as a clean measurement. H's clean Field pair is the evidence for the Field.
Setup H (the Deck's Auto choice: Minimum, 50% render), handheld, on AC, 1280x800 at 90 Hz
Reading:
Side note: docked, on AC, 1280x720 at 60 Hz, setup H's options. These are earlier runs, not the measurement above. They were taken before the Deck was undocked at 21:38:30; H-3N, taken after the undock, is left out.
Offline, the Deck's Mesa: RADV 26.2.0-devel git-5c65d4fc31 (steamos-26.05), VANGOGH through amdgpu drm-shim, with RADV_DEBUG=shaderstats. The split module was made at bda836f4; the split's sources are byte-identical at this head.
Flags and settings
Two new switches, both off by default, plus a debug value of the first.
RX_VK_WG_SPLIT1: the fog lighting pass in 8x8x4 workgroups for its direct dispatches, on a RADV driver whose device is VANGOGH, with compute at 32 lanes (no Mesa version gate)RX_VK_WG_SPLIT=<hex,…>RX_VK_WG_SPLIT_ANY_GPU1allows the split on any GPU whose compute runs 32 lanes, to look at the picture there; logged as "a debug run, not a measurement". This widens the risk beyond RADV Van GoghTesting
Tested:
tests/test_wg_split.py, withtests/wg_split/wg_split_test.cppandfixture.spvasm, a module shaped like the fog's pass.gpu/vk/wg_split.cpp. The correctness evidence is the emitted-op mutations above.544229a2; tested at 4d619558, which differs from it only in the docs/tested rows):tests/test_wg_split.pypasses on Windows and on Linux (with spirv-val), and the renderer builds (dev_build.py vk, 0 failed, exports checked).python -I -B tests/run_all.pyon Windows; the receipts are in the evidence comment below.544229a2: 577 passed, 47 failed.test_longshot_ctl, which is flaky: run alone, it passes on the branch and fails on main.test_gpu_hang_watch.py, a 2 s timing test that failed under a 24-way parallel run. Run alone, it passes on the branch and on main.test_wg_splitis the branch's extra test, and it passes.6d72ca93matched this: 46 common failures, plustest_longshot_ctlon the branch only.lane_probe.candprobe.comp, and its Deck output are in the evidence comment below.RX_VK_WG_SPLIT=1+RX_VK_WG_SPLIT_ANY_GPU=1(debug).RX_VK_WG_SPLITunset vs=1): no obvious fog change by eye. The side-by-side of H-2O (unset) and H-2N (=1), Kokiri and the Field, is in the evidence comment below, with the PC pair.Not tested: the Deck LCD; other Mesa versions (the gate is the device, not the version); other RADV GPUs (the switch declines there); cold translations (they decline until the next start, by design).
Must be tested:
RX_VK_WG_SPLITunset and=1in turn, 3 runs each, plus oneRX_VK_PASS_TIMING=2run per arm. Expecting the log's "workgroup split of 27ee9883… bound for its direct dispatches from now on" line before each=1window, no "declined" line, the fog looking the same, and a lower 27ee9883 time underRX_VK_PASS_TIMING=2(Test Farm: @Beanstalk steamdeck, the fog split off vs on in Kokiri Forest and Hyrule Field on an LCD Deck; or by hand: the same, by anyone with an LCD Deck)Risks
What could break: only with the switch on, on a RADV Van Gogh driver: the fog's lighting pass. A wrong remap would show as blocky or shifted fog. The rewrite refuses the program, and today's pipeline stays, when it can't show that no invocation reaches outside its new group. The lane assumption was shown on one Mesa (26.2.0-devel); another Mesa version is untested. The debug values (hex prefixes,
RX_VK_WG_SPLIT_ANY_GPU) widen the risk to other programs and GPUs.Turning it off: unset
RX_VK_WG_SPLIT, which is the default.Compatibility: nothing changes for a player who doesn't set the switch.
Issues
See #373 (an option to reduce the fog; this PR makes it cheaper instead).
Checklist
Everyone
type(scope): description) of the status line's type.main. (544229a2; re-check before opening)python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). (The 47 common failures listed; the two timing-flaky tests explained.)tests/wg_split/fixture_spv.his generated from the hand-writtenfixture.spvasm, with its SHA-256 checked; it is not game-derived.)third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.gpu/vk/switches.def)docs/tested/for each new measured run (format indocs/tested-configurations.md). (12 files: D and H, off and on, runs 1–3; commite4a1471d.)mainand passing here. (It fails to build on main.)tests/that reads changed source updated in this PR, not weakened. (test_intel_quick_compute.py, plus stubs in the two program-cache tests)Tested in game: Steam Deck OLED, SteamOS 3.9.5, RADV VANGOGH (Mesa 26.2.0-devel git-5c65d4fc31), handheld on AC at 1280x800 and 90 Hz, at the user's settings and at Auto (Minimum): Kokiri Forest and Hyrule Field, RX_VK_WG_SPLIT unset vs =1, 3 interleaved runs each at bda836f4. The split was bound before every window and never declined, the fog looked the same, and the lighting pass 27ee9883 took 27-28% less GPU time in Kokiri. Also Windows, RTX 5090, driver 621.14, with the debug RX_VK_WG_SPLIT_ANY_GPU=1: the same spots, no visible change.
Evidence for the description: screenshots, the lane probe, the receipts at bda836f4, and the suite's failure list.
Pictures, Steam Deck OLED, handheld on AC, setup H. Left: Before, RX_VK_WG_SPLIT unset: this PR @ bda836f4 (H-2O). Right: After, RX_VK_WG_SPLIT=1: this PR @ bda836f4 (H-2N). Top: Kokiri Forest; bottom: Hyrule Field.
Pictures, Windows, RTX 5090, driver 621.14. This is a debug run with
RX_VK_WG_SPLIT_ANY_GPU=1, not a measurement. Taken headless at arrival +900 frames, at bda836f4.The lane probe's output. The Deck run used the first binary. The PC rerun used the revised binary, 9068d6fa…, whose PASS also requires lane = index%32 and size 32.
probe.comp
lane_probe.c
Receipts at bda836f4: source hashes, the test with spirv-val, ASan/UBSan, spirv-val of the real module, Mesa 26.2 stats
The 47 failures common to this branch and main
544229a2(Windows run_all)Description: one cell corrected. The setup-D Kokiri CPU row was the process's CPU divided by the control bridge's game-frame counter, under a vaguer name and from an earlier reading. It is now labelled 'process CPU ms per bridge game frame', with the corrected medians 152.76 -> 152.33 ms: a sub-percent difference, not a change. The GPU, fps and 27ee9883 numbers are unchanged.
On this Steam Deck OLED, at the player's settings, the split binds and the frame's GPU time does not move. Not a no: it does not show here.
RX_VK_WG_SPLIT=1workgroup split of ...line is in both=1runs, and nodeclinedline.RX_VK_PASS_TIMING=2run, so this is the whole frame's GPU time, not the pass's own.Hardware and method: Steam Deck OLED (AMD Custom APU 0932, 4 cores / 8 threads, RADV VANGOGH), SteamOS 3.9.5, 16 GB, on the charger, the GPU at a manual 1600 MHz. One game build on the Deck by the Linux builder (
--jobs 4), no hot list: main544229a2with this PR and three others merged locally (#120900ad3e5b, #11584e889b09, #1377e4a1471d, #875d51401bc), each measured by its own switch in that one binary. Headless runs from a save, 720 presents measured once the scene has settled, the player's settings (a low internal resolution, 768x432 passes; a texture budget of 2,800 MB) with the frame cap lifted (HOS_FPS_MAX=1000), numbers from the game's own log lines. No picture check (no display). 0 fatal lines in the 18 starts and 2 tours of the session.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.