WIP: perf(vk): one map reservation per task workgroup, not per meshlet (#2245) #1460
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1460
Loading…
Reference in a new issue
No description provided.
Delete branch "omgitsgela/Longshot:perf/2245-task-atomic-batch"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: medium · Tested on: Steam Deck LCD (VanGogh, RADV), desktop mode · Needs: RDNA2 desktop (RX 6600-class) and Windows AMD for the policy's other half
Why
Issue #2245 (upstream Hookshot:
https://git.notrexed.dev/Hookshot/Hookshot/issues/2245, closed by the compute-by-default workaround #2362) left one half open: is the Steam Deck's automatic compute fallback the right call, or a bandaid that costs the Deck frame rate it does not need to lose? The Deck's driver reports task and mesh shaders, so "the Deck should just use them" is the natural assumption, and the open half asked whether it should.Measured here on a Deck LCD: it should not. Native task/mesh runs clean on this GPU — no device loss, no amdgpu reset — but it is slower than the fallback on both sides of the pipeline. That answer, with the numbers, is the point of this PR. The code change it carries is a real inefficiency found while getting there.
What changed
The measurement work found that
TaskEpiloguereserved its place in the mesh dispatch map with one device-scope atomic per started meshlet, so OoT's vegetation tasks — a thread a grass blade, fern and tree crown — hit that single device-wide word dozens of times per task workgroup. It now batches the reservation the way the mesh stage already does.In the game this changed nothing measurable (see Measurements: a wash), and it is documented as such. The valuable result is the A/B it made possible: native mesh on the Deck is slower than the fallback, so #2245's compute-by-default policy is the faster path on VanGogh and should not be relaxed.
gpu/vk/eden/shader_recompiler/backend/spirv/emit_spirv_mesh.cpp—TaskEpilogue: thread 0 adds the workgroup's count to the dispatch word and hands the other threads their base through the workgroup's shared word (the patternMeshFallbackEpiloguealready uses for records); thread i takes base + i. Same entries, same slot per index, same dispatch total — only the order indices arrive in changes, and nothing downstream reads an order (each mesh workgroup takes its slot by index). The past-capacity guard keeps a whole workgroup out of the reservation: a block added with no slots written would dispatch mesh workgroups over unwritten map entries.gpu/vk/eden/shader_recompiler/backend/spirv/spirv_emit_context.cpp— the hidden half: the task module returned fromDefineMeshbefore the workgroup word existed, so the batch's shared base had no definition and the task module would not emit. The word is now defined on the task path too (each module holds its own copy).tests/mesh_fallback_layout_ir.cpp— the probe now emits a task program through the real SPIR-V back end (a task that keeps its meshlets and runs the actualTaskEpilogue), so this path is validated byspirv-valrather than only by a source grep.tests/test_mesh_fallback.py— source pins: no per-thread device atomic returns, exactly one device atomic in the epilogue, the base travels through the shared word, and the past-capacity guard still holds.docs/tested/— seven one-row files for the Deck runs: the fallback's stages measured, this change, native task/mesh, the 50% host-bound baseline,RX_VK_PARALLEL=4as a regression, and the second interleaved run of each fallback arm.Measurements
Verdict: an honest negative on frame rate, and a decision on the issue. The batching removes real atomic traffic but did not measurably change fps: two runs of each arm, per-frame medians, turns 19.4/19.3 fps before against 19.5/18.4 after. It stays because it is correct,
spirv-val-validated and strictly less device traffic. The result that matters is the A/B: native mesh on the Deck costs 8 ms more GPU time and 3-5 fps, so #2245's compute-by-default policy is the right call on VanGogh and should stay. No 30 fps claim — see Risks.Hardware: Steam Deck LCD, AMD Custom GPU 0405 (RADV VANGOGH), kernel 6.18.50-valve2, Mesa of that stack, Zen 2 4C/8T, 16 GB shared VRAM/CPU.
Scene: Hyrule Field, child save (slot 5 copied to a separate
--savefolder; the installed save untouched), output 1280x800, render resolution 1 (100%) except the 50% row below, the game's own upscaler, Quality preset, frame generation off, vsync off, no fps cap (HOS_FPS_MAX=1000).Method: one arm per
systemd-run --userunit, the same scripted route every time (warm 1200 presents, 4 forward walks, left/right pans, 4 full-stick turns, walk+circle). Two runs per arm, arms adjacent within a round, in two separate rounds. Numbers are medians of per-frame times fromRX_VK_FRAME_LOG'sframe-times.csv, windowed on each unit's journal phase marks; the log's rolling 5 sfps over the lastwindows are shown to be unusable for A/B (below).RX_VK_PASS_TIMING=2for the GPU spans. Each run'slongshot_options.iniis seeded before launch so settings cannot race shader prep. Overlays are hard-linked copies of the installed runtime with the writable caches unlinked, so no arm can write into the installed build.Baseline: the before column is this PR's tree with the batching left out (
46973fff: the fallback-stage spans, a measurement aid carried in both arms), built on 0697c0e4. main against this PR was not the comparison because the mesh mode is a switch and the two fallback arms must share one build for the comparison to mean anything.How to reproduce: the Deck LCD, the copied Hyrule Field child save,
RX_VK_MESH_SHADERS=compute(the RDNA2 rule's own choice) versus=native, at render resolution 1 and 4, withRX_VK_FRAME_LOGandRX_VK_PASS_TIMING=2, driven by the scripted route in~/longshot-investigation/mesh2245-arm2.sh(one arm persystemd-run --user --collectunit; run folder must be new).46973fff72c8f21372c8f213submita frameWhy native loses, which is not a Longshot defect: on RDNA2 one mesh invocation writes about one vertex and one primitive, so the driver launches more invocations than the API asked for and stores and reloads the outputs; RDNA3's attribute ring and row export removed that. See #2245's desktop RDNA2 fault reports (fixed-VA TCP reads on Navi 21/23); the Mesa task-ring fix "emit task rings after cache flush and wait" is 26.3, newer than this stack — so the RDNA2 policy stays for safety and is now known to be the faster path here.
Flags and settings
None added. The aids used already exist and are unchanged:
RX_VK_MESH_SHADERS(default Automatic, which picks compute on AMD),RX_VK_PASS_TIMING,RX_VK_GPU_HANG_SEC,RX_VK_FRAME_LOG.Testing
Tested:
tests/test_mesh_fallback.py,tests/test_mesh_fallback_failed_once.pyandtests/test_mesh_fallback_layout.pyon the Deck's own toolchain (g++,/usr/bin/spirv-val): 62 layout/emit cases, 0 failed, 42 modulesspirv-valclean, task-stage modules included. In the game, Deck LCD desktop mode, Hyrule Field child save: the full scripted walk/turn route (~11 minutes) run twice per arm (fallback and native) — no device loss, no hang, 0amdgpukernel lines, no missing grass, no flicker, no wrong geometry counts.Not tested: Windows AMD and desktop RDNA2 (no such machine here); RDNA3/RDNA4; other titles (Kokiri Forest measured only by the same scene class). The Windows cross-compile was not run locally (no llvm-mingw on this host).
Must be tested:
steamdeck, linux, radv: render resolution 100%, Hyrule Field child save,RX_VK_PASS_TIMING=2, walk and turn, expectingfb task prog/fb mesh progrows in the report, fps within the fallback's band, and no device loss.linux, amd, rdna2andwindows, amd, rdna2(desktop Navi 21/23, the hardware #2245's faults are from): Kokiri Forest or Hyrule Field with this change, expecting identical visuals and no fault — this change rewrites every task program's SPIR-V, so the fault picture must not shift.build-windows.ymljob on this PR (by hand: read the check on the PR page), expecting the Windows build green for this branch'semit_spirv_mesh.cppandspirv_emit_context.cpp, since both compile on Windows too.Risks
What could break: the task programs' map reservation, so any title whose task programs cull per thread. A mistake shows as wrong meshlet counts or missing geometry, not a crash: the past-capacity guard keeps an over-capacity workgroup out of the reservation entirely. Hyrule Field grass, the heaviest case I have, is covered twice.
Turning it off: revert the commit; there is no switch.
Compatibility: every task program translates again on the first launch after this lands (shader preparation runs; the pipeline cache rebuilds). Saves and
longshot_options.iniuntouched;MKWVK_API_VERSIONuntouched.Not claimed: this PR does not reach a stable 30 fps. At render resolution 4 with the installed build's caches warm (run08) the honest steady medians are ~27.5 fps warming, ~26 walking, ~20 turning, ~22 circling: wall ~36 ms with GPU 25-27 ms and host submit 19-23 ms, so neither thread is alone the wall and the frame loses ~10 ms to the gap between them.
RX_VK_PARALLEL=4is a regression on this GPU (18.1 against 21.7 fps). An earlier arm here read host submit over budget at 36 ms; run08 showed that was the first run's cold page cache, not the steady state. Closing the gap is per-draw host work (the run10 itemization), a separate task from the mesh question this PR answers. All corrected numbers are on issue 402.Issues
See Hookshot/Hookshot#2245 —
https://git.notrexed.dev/Hookshot/Hookshot/issues/2245(closed upstream by #2362; this covers the open "is the Deck's fallback costing performance?" half and answers it: no, the fallback wins here).Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main. Base is544229a2withmainatc14641a6:git merge --no-commitof origin/main into this branch reports 0 conflicts, so no rebase is needed (docs/agent-pull-requests.md: rebase if it conflicts; main moving past the base is not, by itself, a reason).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).main@c14641a6: 38 failed; this branch: 38 failed, 576 passed, 107 skipped, no branch-only failure (diffed by test name). The one that reads changed source istests/test_tested_configurations.py, which fails onmainand here because fourdocs/testedfiles carry several runs each; #1457 fixes that, and with it this branch is 37 failed / 577 passed.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
docs/tested/for each new measured run (format indocs/tested-configurations.md).tests/that reads changed source updated in this PR, not weakened.Tested in game: Steam Deck LCD, AMD Custom GPU 0405 (RADV VANGOGH), kernel 6.18.50-valve2, desktop mode: Hyrule Field child save at render resolution 100%, warmed then 4 forward walks, left/right pans, 4 full-stick turns, walk+circle (~11 min). Vegetation draws normally (no missing grass, no flicker, no wrong geometry counts), 21.7 fps warming and 22.2/14.2/17.8 fps through the maneuvers, no device loss, no hang, 0 amdgpu job timeouts or resets in journalctl -k -b. Same route on the same build with RX_VK_MESH_SHADERS=native also clean (0 faults) but slower: 18.8 and 18.7/9.9/12.7 fps, GPU 41.8 vs 33.7 ms.
Two more runs, one per arm (docs/tested runs 06 and 07): fb-compute-r2 and fb-atomics-r2, same save, route and settings as runs 01 and 02, so each arm now has two runs and the comparison is medians, not a single pair.
They also invalidate the log's rolling 5 s fps windows for A/B work: between two runs of the identical build the maneuver windows read 22.3 fps and 15.9 fps while the median frame time was the same to 0.1 ms (43.2 ms warming both times). The table now reports per-frame medians from RX_VK_FRAME_LOG's frame-times.csv, windowed on the run's own phase marks.
Verdict unchanged and now two-runs-deep: the batching is a wash (turns 19.4/19.3 fps before against 19.5/18.4 after; walks 22.2/22.7 against 22.2/23.4). Native mesh stays slower (18.8 fps, GPU 41.8 ms) and still does not fault on this GPU.
@Beanstalk please test this change on hardware I do not have: (1) linux, amd, radv on a desktop RDNA2 card (RX 6600/6650 XT/6950 XT class, the cards issue 2245's GPU read faults came from) - open a Kokiri Forest or Hyrule Field save, walk and turn for a few minutes, expecting the vegetation to look exactly as on main with no GPU fault, no device loss and no amdgpu job timeout in the kernel log; (2) windows, amd, rdna2 with the same scene and expectations; (3) linux, steamdeck: at render resolution 100% with RX_VK_PASS_TIMING=2, walk and turn in grass, expecting fb task prog / fb mesh prog rows in the report, frame times in the same band as main's fallback, and no device loss. This change rewrites every task program's SPIR-V, so the first launch retranslates (shader preparation runs once). Report whether anything looks different in the vegetation.
@omgitsgela only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was for9338fa9, and the PR has changed since (4c3a4f2), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).
Delegated code review of this branch (five own commits through
4c3a4f20, merge imported changes excluded) finished: no blocking issues in the batched reservation or rendering correctness. The reviewer independently brute-forced the batching arithmetic — old per-started-lane count equals the batched reservation across 42,980 boundary cases (threads 1..1024; zero, partial, full and oversized counts; final partial workgroup); source-pin test and git diff --check pass; the worktree stayed clean. Its one open caveat (native IR emission and spirv-val not rerun on the review machine) is already covered here: the Deck's own toolchain ran tests/test_mesh_fallback_layout.py on this exact emission code — 62 cases, 0 failed, 42 modules spirv-val clean, task stages included — and the code under validation is unchanged since.It found one real non-blocking defect, now fixed in
685f98e3: the measurement spans formatted their per-program label with snprintf on every fallback dispatch even with RX_VK_PASS_TIMING off — gprof_on() gates the span's timestamps and report but not the caller's formatting. The label buffer is now written only when the profiler is on (GprofSpan reads the label only when it opened), so the shipped path pays exactly the cached bool test it paid before this PR. tests/test_mesh_fallback.py still ok.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.