perf(vk): three small costs of the replay thread removed: unread bookkeeping without mirrors, 16 vertex bindings a change, a lock a sampled texture #590
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8015c9a
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!590
Loading…
Reference in a new issue
No description provided.
Delete branch "p0lux/Longshot:perf/three-cpu-costs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: low · Tested on: Linux (SteamOS), Steam Deck OLED (RADV VANGOGH) · Needs: Windows, any GPU (in the game)
Why
In heavy places the frame is limited by the thread that replays the game's draws. On a Steam Deck that thread spent about 3% of its time on three things that buy nothing: notes kept for write-backs that a GPU without VRAM mirrors never makes, 16 vertex bindings handed to the driver when one changed, and a lock taken for every sampled texture to ask a question whose answer is nearly always no.
What changed
Three commits, each removing one of those costs and keeping the result the same. A fourth updates three test fixtures that cut the changed code out of the source, and a fifth moves an include.
gpu/vk/vk_memory.cpp).memory_cpu_owned(the report words the replay wrote on the CPU),shadow_written_toandguest_cpu_wrote(the guest pages the CPU wrote) note things that only a mirror's write-back reads (cut_cpu_words,shadows_forget_written, both underif (!writebacks.empty())). Without mirrors (mirror_on()false: an integrated GPU, a GCN card,RX_VK_MIRROR=0) no write-back is ever noted, so the words piled up to 65,536 to be sorted, merged and dropped, and the page lists to 8 million entries (32 MB) before being dropped. They now return at once there.gpu/vk/vk_draw.cpp).vertex_bind_rangegives the bindings that differ from what the command buffer has, anddraw()binds that range instead of all 16; all 16 are still remembered.gpu/vk/wrote_regions.h, new;gpu/vk/vk_texture.cpp).texture_gpu_wrotenotes the 2 MB regions a GPU buffer write reaches (64 KB of bits, never cleared), before the map of written ranges has the write.gpu_wrote_intoanswers no from the table when the texture's storage is in none of them; otherwise the map answers under its lock, as before.RX_VK_VB_RANGEindocs/development.md; one run file indocs/tested/.Measurements
Verdict: it helps the replay thread: 4.8% fewer instructions and 3.8% fewer cycles a frame, 1.7 ms less replay, 28.5 to 29.9 fps on a Steam Deck in a heavy view. In a build with the committed play profile the functions it changes are compiled without their profile, and it is level on fps (31.5 against 31.5) with 2.8% fewer instructions.
Hardware: SteamOS 3.9.5 (kernel 7.2.7-valve2), Steam Deck OLED: AMD Custom APU 0932 (4 cores, 8 threads), RADV VANGOGH, Mesa 26.2.0-devel, 16 GB shared with the GPU; the GPU's clock fixed at 1600 MHz
Scene: a heavy view from a save, standing still, about 2,150 draws a frame; no screen (headless), handheld profile, Minimum preset, the game's own upscaler, no frame generation, no cap; mesh programs on the compute path, no VRAM mirrors
Method: the game and host built from main
9c9d69ed; the back end built alone (clang, same flags) from main414f56e2(nothing undergpu/differs from9c9d69ed) and from this PR's three commits on it, and swapped in that game folder. Each run starts the game, loads the save by script, waits 3 log windows, then measures 720 presents: fps by the clock, the log's replay time, the render thread's user-space cycles and instructions a frame (perf stat -t). Two runs each after a warm-up start, main then this PR, the second round in the reverse order; the median of two is their mean. No per-frame log in these runs, so no p99 or 1% low: the target is the replay thread's work, which the instruction count gives independently of the CPU's frequency.How to reproduce: a save in a place with over 2,000 draws a frame,
RX_VK_VB_RANGE=0against unset for the vertex bindings; the other two have no switch and need the two buildsthis PR: perf/three-cpu-costs @ 243c3a0d (its three code commits on main
414f56e2; on the fork asmeasured/three-cpu-costs-243c3a0d). The branch is now on main25726217:git range-diffshows the same three patches (cf67e86b,1c2bdc6b,dd342a08). Main's commits in between touchvk_draw.cppandvk_texture.cppelsewhere (the pipeline cache,dispatch,texture_frame), nottexture_gpu_wrote,gpu_wrote_intoor the vertex bindings ofdraw(), and notvk_memory.cppat all, so the numbers hold.Built without a play profile (medians; each run in brackets, in the order run):
414f56e2Built with the committed play profile (
RX_VK_PGO=<assets/profiles/longshot_vk.profdata.xz, unpacked>; the build log saysbuilt with the profile):414f56e2, play profileRead with care. Two runs each on one machine and one scene; the fps of two runs of one renderer differ by up to 2 (the play profile's arms went from 30.3-30.7 in the first round to 32.3-32.6 in the second), while the instruction counts repeat within 0.6%. The three commits were not measured apart. With the play profile,
draw(),gpu_wrote_into,texture_gpu_wrote,memory_cpu_owned,shadow_written_toandguest_cpu_wroteno longer match the recording and are compiled as if there were none, which is what takes back most of the gain there until the profile is recorded again (assets/profiles/README.md: "a function changed since the recording is compiled as if there were no profile").Flags and settings
One new A/B switch; the other two changes have none.
RX_VK_VB_RANGE=0Testing
Tested:
python -I -B tests/run_all.pyon macOS with the Vulkan headers given: 499 passed, 41 failed, 39 skipped; the 41 fail the same way on main25726217on that machine (496 passed, 41 failed: no/proc, noposix_fallocate, Python 3.9, the Linux launch scripts and the like).tests/test_mirror_bookkeeping.py(the three functions cut out of the source, without mirrors and with, and that only a write-back reads what they keep),tests/test_vertex_bind_range.py,tests/test_wrote_regions.py. Each fails on main. Updated, no check changed:tests/test_report_words.py,tests/test_wb_diff_stale_shadow.py(their pools have mirrors: amirror_on()stand-in) andtests/test_fallback_pipeline_state.py(takesvertex_bind_rangefrom the source).414f56e2(clang, on the Deck): 0 failed.longshot_vk.dllmade. Not run.Not tested:
vkCmdBindVertexBuffers2.Must be tested:
Risks
What could break: a draw's vertex streams, if a driver mishandles a partial bind (models missing or stretched); a texture the GPU wrote not uploaded again, if a write's regions were not noted (the table is filled before the map and never cleared, and addresses past 40 bits share bits, so it can only say "ask the map" too often); the mirrors' write-backs, if a mirror existed while
mirror_on()said no (a pool gets one only inmirror_make, whose two callers,gpu_buffer_bindandmemory_begin_submission, are both behindmirror_on()).Turning it off:
RX_VK_VB_RANGE=0for the vertex bindings; reverting the commit for each of the other two.Compatibility: Nothing changes: no shader or pipeline cache is rebuilt (no stamped file is touched),
MKWVK_API_VERSIONis as it was.Issues
See #402
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each new measured run (format indocs/tested-configurations.md).mainand passing here.tests/that reads changed source updated in this PR, not weakened.WIP: perf(vk): three small costs of the replay thread removed: unread bookkeeping without mirrors, 16 vertex bindings a change, a lock a sampled textureto perf(vk): three small costs of the replay thread removed: unread bookkeeping without mirrors, 16 vertex bindings a change, a lock a sampled textureRead all three code changes and the tests.
vertex_bind_rangelooks right. It returns the span of bindings that differ, or all 16 when nothing is valid yet. All 16 are still remembered inlast_vb, andlast_vb_validis still reset wherever the command buffer state is reset. That draw is the onlyvkCmdBindVertexBuffers2call, and the partial bind is legal with dynamic strides.mirror_on()is a once-computed static, so it can't flip mid-run and leave a write-back with notes it never took. The early returns cover exactly what only a write-back reads.wrote_regions: the bits are a superset, never cleared, and set before the map insert. A false positive just falls back to the map, and every path that fillsgpu_wrotegoes throughtexture_gpu_wrote. Relaxed ordering is fine for a lock-free "maybe" filter.RX_VK_VB_RANGEis documented and defaults to on. NoMKWVK_API_VERSIONbump is needed becausevk_api.his untouched.I did not run anything. The remaining risk is a driver mishandling a partial vertex bind, which the PR already lists, and its "Must be tested" Windows run should still happen. Nothing in the diff blocks the merge. Approving.