perf(vk): submits are not batched on a discrete RDNA 4 card on RADV #1893

Open
Molier wants to merge 1 commit from Molier/Longshot:perf/radv-rdna4-no-batch into main
Contributor

Summary

On a discrete RDNA 4 card under RADV, batching the replay's queue submits makes the frame about a quarter longer. While submits are held the GPU has short idle gaps, and the card's clock management reads them as low load: the shader clock sits near 1.6-1.7 GHz instead of 2.3-2.6 GHz. With each submit sent at once the same work finishes much sooner. This PR leaves batch_submits off for rdna4() && discrete() on RADV. Every other RADV card keeps today's rule: the Radeon 8060S of issue 413, where batching took the submit from 70 to 6 ms a frame, and the RDNA 2/3 cards nobody has measured either way.

To see it on main: an RX 9070 XT on Linux, hf48run uncapped, with and without RX_VK_BATCH=0.

Read first: gpu/vk/device_policy.h (batch_submits, its comment and the one-line rule), tests/test_device_policy.py (the new RX 9070 XT, RADV fixture).

Testing

Tested: CachyOS (kernel 7.4), Ryzen 7 5800X, RX 9070 XT, Mesa 26.2.6 RADV; renderer built from main @ 89c3d21e and from this PR in the same way and swapped into one game folder. The game boots and plays through hf48run and the heavy route (Hyrule Field, Market, Castle) with the log saying batched submits off on this PR and on on main. Picture: I saw nothing wrong in play, but have no pixel comparison. Shots from the two arms are not of the same frame, and two runs of the same arm already differ in 38 % of pixels, so a diff says nothing. tests/test_device_policy.py and tests/test_switch_table.py pass; the new fixture fails on main (RX 9070 XT, RADV: batch_submits is 1, expected off).

Still to test:

  • RX 6000/7000 on RADV: RX_VK_BATCH=0 against the default on hf48run, to see whether the rule should cover RDNA 2/3 too (by hand, or @Beanstalk on an RDNA 2/3 runner).
  • Radeon 8060S / other AMD APUs: unchanged by this PR (the rule needs discrete()); nothing to do.

Measurements

Verdict: throughput +33 % (median frame 17.0 -> 12.8 ms on hf48run); hitch time on the capped heavy route unchanged.

Setup: CachyOS, Ryzen 7 5800X, RX 9070 XT, Mesa 26.2.6 RADV; 2560x1440, render scale 1, no upscaler, fullscreen. hf48run uncapped, 3 interleaved runs a side after one warm-up each; heavy route capped at 60, 2 interleaved runs a side. GPU busy and the current pp_dpm_sclk level sampled at 10 Hz during the hf48run runs.

Before: main @ 89c3d21e After: this PR @ 8f6712b1
hf48run fps, median per run (run_a) 61.2 / 58.5 / 56.9 80.4 / 76.4 / 77.7
hf48run median frame, run_a (ms) 16.3 / 17.1 / 17.6 12.4 / 13.1 / 12.9
shader clock, median of samples (MHz) 1587 2311
GPU busy, median (%) 84 85
heavy route: time lost in frames >= 40 ms (2 runs pooled) 10.9 s 11.0 s
heavy route: frames > 25 ms 470 376

The same A/B as a switch on the official nightly-612 (3 runs a side, RX_VK_BATCH=0 against the default) gave 62 -> 81 fps, with clock medians of 1715 against 2589 MHz. For comparison, holding amdgpu's peak power state (RX_VK_GPU_CLOCK=peak, the Steam Deck's rule) pins this card at about 2.48 GHz and gives 71 fps; with batching off as well it is 79, slightly below batching off alone. So the clock hold is not the fix on a desktop card.

The work per frame is not identical: in an earlier pair of runs, batching off had the same frames, presents and submits, but 5.8 % fewer draws and 8 % fewer dispatches, and the game waited on GPU results about half as long (9 % against 22 % of the frame). The likely reason is that occlusion results arrive sooner, which culls more. The clock difference accounts for the rest: GPU time per frame was 8.8 against 11.8 ms.

Risks

  • Driver/kernel dependency: if a later Mesa or kernel keeps the clock up during the gaps, batching may win again on RDNA 4. RX_VK_BATCH=1 turns it back on.
  • Picture: occlusion culling sees fresher results, so the set of draws differs slightly (see Measurements). If anything pops in or out that did not before, RX_VK_BATCH=1 is the A/B.
  • Other cards: nothing changes off RDNA 4 discrete cards on RADV.
  • For players: nothing to do after updating; saves, options and caches are unchanged.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game, as Testing says.
  • tests/test_device_policy.py and tests/test_switch_table.py pass (the full mise run test not run here).
  • Measured against main.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A switch's default changes: the RX_VK_BATCH row in gpu/vk/switches.def says the new default, and Measurements has the numbers that chose it.
  • AI assistance recorded with an Assisted-by: trailer.

Ready to merge

  • Approved in review, every comment answered.
  • CI passes on the head.
## Summary On a discrete RDNA 4 card under RADV, batching the replay's queue submits makes the frame about a quarter longer. While submits are held the GPU has short idle gaps, and the card's clock management reads them as low load: the shader clock sits near 1.6-1.7 GHz instead of 2.3-2.6 GHz. With each submit sent at once the same work finishes much sooner. This PR leaves `batch_submits` off for `rdna4() && discrete()` on RADV. Every other RADV card keeps today's rule: the Radeon 8060S of issue 413, where batching took the submit from 70 to 6 ms a frame, and the RDNA 2/3 cards nobody has measured either way. To see it on `main`: an RX 9070 XT on Linux, `hf48run` uncapped, with and without `RX_VK_BATCH=0`. Read first: `gpu/vk/device_policy.h` (`batch_submits`, its comment and the one-line rule), `tests/test_device_policy.py` (the new `RX 9070 XT, RADV` fixture). ## Testing **Tested:** CachyOS (kernel 7.4), Ryzen 7 5800X, RX 9070 XT, Mesa 26.2.6 RADV; renderer built from `main` @ `89c3d21e` and from this PR in the same way and swapped into one game folder. The game boots and plays through `hf48run` and the heavy route (Hyrule Field, Market, Castle) with the log saying `batched submits off` on this PR and `on` on `main`. Picture: I saw nothing wrong in play, but have no pixel comparison. Shots from the two arms are not of the same frame, and two runs of the same arm already differ in 38 % of pixels, so a diff says nothing. `tests/test_device_policy.py` and `tests/test_switch_table.py` pass; the new fixture fails on `main` (`RX 9070 XT, RADV: batch_submits is 1, expected off`). **Still to test:** - RX 6000/7000 on RADV: `RX_VK_BATCH=0` against the default on `hf48run`, to see whether the rule should cover RDNA 2/3 too (by hand, or @Beanstalk on an RDNA 2/3 runner). - Radeon 8060S / other AMD APUs: unchanged by this PR (the rule needs `discrete()`); nothing to do. ## Measurements **Verdict:** throughput +33 % (median frame 17.0 -> 12.8 ms on hf48run); hitch time on the capped heavy route unchanged. Setup: CachyOS, Ryzen 7 5800X, RX 9070 XT, Mesa 26.2.6 RADV; 2560x1440, render scale 1, no upscaler, fullscreen. `hf48run` uncapped, 3 interleaved runs a side after one warm-up each; heavy route capped at 60, 2 interleaved runs a side. GPU busy and the current `pp_dpm_sclk` level sampled at 10 Hz during the hf48run runs. | | Before: main @ 89c3d21e | After: this PR @ 8f6712b1 | |---|---|---| | hf48run fps, median per run (run_a) | 61.2 / 58.5 / 56.9 | 80.4 / 76.4 / 77.7 | | hf48run median frame, run_a (ms) | 16.3 / 17.1 / 17.6 | 12.4 / 13.1 / 12.9 | | shader clock, median of samples (MHz) | 1587 | 2311 | | GPU busy, median (%) | 84 | 85 | | heavy route: time lost in frames >= 40 ms (2 runs pooled) | 10.9 s | 11.0 s | | heavy route: frames > 25 ms | 470 | 376 | The same A/B as a switch on the official nightly-612 (3 runs a side, `RX_VK_BATCH=0` against the default) gave 62 -> 81 fps, with clock medians of 1715 against 2589 MHz. For comparison, holding amdgpu's peak power state (`RX_VK_GPU_CLOCK=peak`, the Steam Deck's rule) pins this card at about 2.48 GHz and gives 71 fps; with batching off as well it is 79, slightly below batching off alone. So the clock hold is not the fix on a desktop card. The work per frame is not identical: in an earlier pair of runs, batching off had the same frames, presents and submits, but 5.8 % fewer draws and 8 % fewer dispatches, and the game waited on GPU results about half as long (9 % against 22 % of the frame). The likely reason is that occlusion results arrive sooner, which culls more. The clock difference accounts for the rest: GPU time per frame was 8.8 against 11.8 ms. ## Risks - **Driver/kernel dependency:** if a later Mesa or kernel keeps the clock up during the gaps, batching may win again on RDNA 4. `RX_VK_BATCH=1` turns it back on. - **Picture:** occlusion culling sees fresher results, so the set of draws differs slightly (see Measurements). If anything pops in or out that did not before, `RX_VK_BATCH=1` is the A/B. - **Other cards:** nothing changes off RDNA 4 discrete cards on RADV. - **For players:** nothing to do after updating; saves, options and caches are unchanged. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game, as Testing says. - [x] `tests/test_device_policy.py` and `tests/test_switch_table.py` pass (the full `mise run test` not run here). - [x] Measured against `main`. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] A switch's default changes: the `RX_VK_BATCH` row in `gpu/vk/switches.def` says the new default, and Measurements has the numbers that chose it. - [x] AI assistance recorded with an `Assisted-by:` trailer. ### Ready to merge - [ ] Approved in review, every comment answered. - [ ] CI passes on the head.
perf(vk): submits are not batched on a discrete RDNA 4 card on RADV
Some checks failed
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Failing after 9s
Website / Website on Cloudflare (pull_request_target) Has been skipped
8f6712b15c
On an RX 9070 XT (RADV, Mesa 26.2.6) the held submits leave gaps in which
the card's clock drops: 1715 MHz median while held against 2589 MHz with
each submit sent at once (gpu_busy and pp_dpm_sclk sampled at 10 Hz over
three hf48run runs each). The frame is about a quarter longer with
batching: hf48run median 16.1 -> 12.2 ms with RX_VK_BATCH=0 on nightly-612.

batch_submits stays on for the other RADV cards it was measured on (the
Radeon 8060S of issue 413) and for the ones never measured; RX_VK_BATCH=1
turns it back on on an RDNA 4 card. tests/test_device_policy.py gains an
RX 9070 XT RADV fixture, which fails on main (batch_submits is 1).

Assisted-by: Claude (Anthropic)
Member

Possibly related, for context:

  • PR #1839 fix(vk): recognize RDNA 4 GPUs by PCI device ID — Open RDNA 4 device detection change; this rule relies on rdna4() classification.
  • issue #402 Deck: the frame is CPU-bound in the submission path, not GPU-bound (16 fps at 50% internal, GPU-bound wait 0.4 ms) — Open RADV submit-path discussion on Steam Deck; adjacent concern, but different GPU and direction.
Possibly related, for context: - PR #1839 fix(vk): recognize RDNA 4 GPUs by PCI device ID — Open RDNA 4 device detection change; this rule relies on `rdna4()` classification. - issue #402 Deck: the frame is CPU-bound in the submission path, not GPU-bound (16 fps at 50% internal, GPU-bound wait 0.4 ms) — Open RADV submit-path discussion on Steam Deck; adjacent concern, but different GPU and direction.
BeanBot left a comment

Recommendation: Request changes. The RX 9070 XT shows a substantial throughput gain, but the new default also applies to other discrete RDNA 4 cards without evidence that the extra unbatched submit cost is safe there. The reported runs also omit the low-frame-rate and tail-frame-time measurements needed to rule out worse pacing.

Disabling batching improves the measured RX 9070 XT route and keeps the existing override, with a small, focused policy change. Before applying that default across the whole discrete RDNA 4 family, either narrow it to the measured card or measure representative affected cards and CPU-bound settings; also report average FPS, 1% lows and p99 frame time on the target and heavy routes.

Does it help: Yes on the tested RX 9070 XT: three interleaved runs show median frame times of 16.3–17.6 ms before versus 12.4–13.1 ms after, and median-per-run FPS of 56.9–61.2 versus 76.4–80.4. Heavy-route time in frames >=40 ms is 10.9 versus 11.0 s pooled, but 1% lows and p99 are not reported; benefit and cost on other affected RDNA 4 cards are unproven.
Other systems: RADV discrete RDNA 4 alone changes; integrated RADV, RDNA 2/3, Windows AMD, NVIDIA, Intel and Steam Deck retain their policy. Earlier submits change occlusion-result timing and increase queue-submit calls; the author saw no visual issue in play, but has no matched-frame comparison and CPU/pacing costs are not established across affected setups.
Switches: RX_VK_BATCH defaults off instead of on for discrete RDNA 4 on RADV; on for other non-GCN RADV, off elsewhere. The table is updated and RX_VK_BATCH=1 restores batching. The broader new default is not yet supported by measurements beyond one card.

Reviewed: 8f6712b15c against 89c3d21ea1: Read the full diff, changed policy rule and surrounding facts, fixture and switch row, PR description and comments, base AGENTS.md, CONTRIBUTING.md, standards, measurement skill and PR template. Tool output truncated the full device_policy.h, switches.def and test_device_policy.py; their changed sections were read but their remaining contents were only skimmed. Searched batch policy consumers and open overlapping PRs; did not run code or tests. docs/review-checklist.md does not exist at base.; nothing run (a code review).
Checked and fine:

  • The new fixture exercises the policy result and reportedly fails on main; existing fixtures cover RADV RDNA 2, GCN, Steam Deck and AMD's Windows driver.
  • No generated files, third-party code, paths, shader interface or new switch are changed.

Blocking (to change before merge)

  1. gpu/vk/device_policy.h:554: The default disables batching for every discrete RDNA 4 RADV GPU, although the evidence is one RX 9070 XT on one CPU and driver. On a different RDNA 4 card or CPU-limited setup the additional queue submits can cost more than the saved GPU time. Limit the rule to measured hardware or supply representative A/B measurements that justify the family-wide default.
  2. gpu/vk/device_policy.h:554: The change affects every frame and can trade throughput for tail latency, but neither the target nor heavy-route results include 1% lows or p99 frame times; pooled hitch counts alone cannot establish pacing per run. Report average FPS, 1% lows and median/p99 frame times for each arm and investigate any regression.

Testing: Author reports in-game RX 9070 XT runs and passing device-policy and switch-table tests; full mise run test and CI are pending. No independent execution in this review.
Overlaps and merge order: Open #1839 changes DeviceFacts::rdna4() to include PCI IDs, expanding this new batching exception to additional devices if both merge. Coordinate and re-evaluate the combined default and tests whichever merges second; no direct duplicate found.


Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The RX 9070 XT shows a substantial throughput gain, but the new default also applies to other discrete RDNA 4 cards without evidence that the extra unbatched submit cost is safe there. The reported runs also omit the low-frame-rate and tail-frame-time measurements needed to rule out worse pacing. Disabling batching improves the measured RX 9070 XT route and keeps the existing override, with a small, focused policy change. Before applying that default across the whole discrete RDNA 4 family, either narrow it to the measured card or measure representative affected cards and CPU-bound settings; also report average FPS, 1% lows and p99 frame time on the target and heavy routes. **Does it help:** Yes on the tested RX 9070 XT: three interleaved runs show median frame times of 16.3–17.6 ms before versus 12.4–13.1 ms after, and median-per-run FPS of 56.9–61.2 versus 76.4–80.4. Heavy-route time in frames >=40 ms is 10.9 versus 11.0 s pooled, but 1% lows and p99 are not reported; benefit and cost on other affected RDNA 4 cards are unproven. **Other systems:** RADV discrete RDNA 4 alone changes; integrated RADV, RDNA 2/3, Windows AMD, NVIDIA, Intel and Steam Deck retain their policy. Earlier submits change occlusion-result timing and increase queue-submit calls; the author saw no visual issue in play, but has no matched-frame comparison and CPU/pacing costs are not established across affected setups. **Switches:** RX_VK_BATCH defaults off instead of on for discrete RDNA 4 on RADV; on for other non-GCN RADV, off elsewhere. The table is updated and RX_VK_BATCH=1 restores batching. The broader new default is not yet supported by measurements beyond one card. **Reviewed:** 8f6712b15c82 against 89c3d21ea169: Read the full diff, changed policy rule and surrounding facts, fixture and switch row, PR description and comments, base AGENTS.md, CONTRIBUTING.md, standards, measurement skill and PR template. Tool output truncated the full device_policy.h, switches.def and test_device_policy.py; their changed sections were read but their remaining contents were only skimmed. Searched batch policy consumers and open overlapping PRs; did not run code or tests. docs/review-checklist.md does not exist at base.; nothing run (a code review). **Checked and fine:** - The new fixture exercises the policy result and reportedly fails on main; existing fixtures cover RADV RDNA 2, GCN, Steam Deck and AMD's Windows driver. - No generated files, third-party code, paths, shader interface or new switch are changed. **Blocking** (to change before merge) 1. gpu/vk/device_policy.h:554: The default disables batching for every discrete RDNA 4 RADV GPU, although the evidence is one RX 9070 XT on one CPU and driver. On a different RDNA 4 card or CPU-limited setup the additional queue submits can cost more than the saved GPU time. Limit the rule to measured hardware or supply representative A/B measurements that justify the family-wide default. 2. gpu/vk/device_policy.h:554: The change affects every frame and can trade throughput for tail latency, but neither the target nor heavy-route results include 1% lows or p99 frame times; pooled hitch counts alone cannot establish pacing per run. Report average FPS, 1% lows and median/p99 frame times for each arm and investigate any regression. **Testing:** Author reports in-game RX 9070 XT runs and passing device-policy and switch-table tests; full mise run test and CI are pending. No independent execution in this review. **Overlaps and merge order:** Open #1839 changes DeviceFacts::rdna4() to include PCI IDs, expanding this new batching exception to additional devices if both merge. Coordinate and re-evaluate the combined default and tests whichever merges second; no direct duplicate found. --- <sub>Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -550,3 +552,3 @@
// measured on one card
// (corrected: GCN by the one definition, radv_gcn. To go back: f.radv() && !(f.sg32_stages == 0 && f.subgroup_size > 32))
p.batch_submits = f.radv() && !f.radv_gcn();
p.batch_submits = f.radv() && !f.radv_gcn() && !(f.rdna4() && f.discrete());
Member

This changes the default for every discrete RDNA 4 RADV card, but the A/B covers one 9070 XT/CPU/driver. On a different card or CPU-bound system, the extra queue submits could outweigh the gain. Please narrow the rule to measured hardware or measure representative affected configurations before applying it family-wide.

This changes the default for every discrete RDNA 4 RADV card, but the A/B covers one 9070 XT/CPU/driver. On a different card or CPU-bound system, the extra queue submits could outweigh the gain. Please narrow the rule to measured hardware or measure representative affected configurations before applying it family-wide.
Member

Could the A/B also report average FPS, 1% lows and median/p99 frame times for both the target and heavy routes? The pooled >=40 ms totals do not show whether an individual run or the tail got worse when submits were unbatched.

Could the A/B also report average FPS, 1% lows and median/p99 frame times for both the target and heavy routes? The pooled >=40 ms totals do not show whether an individual run or the tail got worse when submits were unbatched.
Some checks failed
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Failing after 9s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/radv-rdna4-no-batch:Molier-perf/radv-rdna4-no-batch'
git switch 'Molier-perf/radv-rdna4-no-batch'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1893
No description provided.