WIP: perf(vk): a draw of the program before it keeps its texture and storage descriptors while their bindings stand (RX_VK_DESC_SECTIONS, off) #2210
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2210
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/vk-desc-sections"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
A draw keeps the descriptors of the draw before it only while nothing at all was bound since (
same_descindraw(),gpu/vk/vk_draw.cpp), and the game binds a uniform buffer for nearly every draw: most draws resolved every texture's handle, view and sampler and every storage buffer's pool again, though only a uniform buffer's range had changed.build_descriptorsnow treats a program's descriptors as three sections by what they are read from: the uniform buffers (built at every build, as before), the texture section (textures with their samplers and read-only storage images, whose handles are words of c2 of their own stage) and the storage section (read-only storage buffers that are one entry of their stage's c0 table). A draw of the program whose build its command buffer bound last, with no other build since, keeps a section's infos from that build while everything it was resolved from stands:tex_gen, the texture destroy generation, the samplers' registrations, the texture epoch, the slots' versions (or the registrations' generation, assame_descchecks them) and each texture's GPU-written state stand (desc_tex_kept);ssbo_genandgpu_bind_stampstand and none of its buffers came out unbound or lies in a virtual pool (desc_ssbo_kept);desc_sections_of): a handle program, another bank, a table, a texel buffer, anything the program writes, a simulated-turf private buffer, or a compute, task or mesh stage leave a section out;RX_VK_DESC_REUSE=never, or while a diagnostic that looks at every binding is set;RX_VK_DESC_REFRESH: when every descriptor but the uniform buffers is kept (or there is none of the rest), the writer stays as it is and only the uniform buffers are resolved again, in the build's order (refresh_uniform_descriptors); otherwise the kept infos are copied back and every write is made again. The descriptors written are meant to be the same either way; the result expected is less replay time for draws that change only their uniform buffers. The frame log gets a line every 120 presents counting the draws that kept a section.Switches (rows in
gpu/vk/switches.def):RX_VK_DESC_SECTIONSis an attempt and ships off: unset or0is main's build,1(any value but0,false,off,no) turns the sections on. The default follows dazemc's review: it stays off until the in-game comparison exists (Still to test).RX_VK_DESC_REFRESH(kind kill, on) acts only under it: withRX_VK_DESC_SECTIONSunset or0no build keeps a section, so the refresh does nothing and is never read; withRX_VK_DESC_SECTIONS=1,RX_VK_DESC_REFRESH=0keeps the sections but copies them back and makes every write again. Each is read once, at its first use (soHOS_ABcannot alternate them); a set value is logged once, and an unsetRX_VK_DESC_SECTIONSlogs nothing.With the sections off,
build_descriptorsanddraw()do none of their work: no count of builds (a shared atomic), no build number, no lookup of the kept infos or of the texture record, no note per storage buffer write, dummy texture or unbound storage buffer. What is left of the change on that path is the switch's test (a function-local static, read once) in the build and indraw(), a call reading it at each bind, and branches on it; not timed.Prior work: dyst's #794 (closed) kept a stage's descriptors across draws in a recording part while what they were resolved from stood. This takes the same idea up again in the branch's narrower form: only the last build of the program the command buffer bound, as the writer holds it, and never the uniform buffers (#794 kept every program's stages in a table of their own, uniform buffers too). #794 measured a loss: its own A/B had the descriptor time slower than main (3.4 to 3.9 ms a frame against 2.5 to 2.9 ms), and dyst closed it. From its code, each build checked and re-applied kept keys for textures, storage buffers and uniform buffers, and the uniform buffers move with nearly every draw; this keeps only the bound program's last build and never the uniform buffers, which should check less a build, but this pull request's own descriptor time is not measured yet. Open pull requests in the same function: #1232 (jcabarca) changes which sampler a texture binding asks
sampler_forfor, in the loop this change skips for a kept section; a kept section holds the samplers its build chose, so the two compose (the same textures give the same views), and the second to merge may need a rebase. #2045, #2055, #2057 and #2058 (mine) touchbuild_descriptorstoo: neighbouring lines only.Since the branch, main lost the parallel replay (its check is gone here) and gained
gp_table_cover, which looks at a virtual pool's pages at every binding (a storage section with a buffer in a virtual pool is bound again at the next build, andRX_VK_GP_TABLEturns the sections off), and the simulated turf's private storage buffers (#2093), whichsim_turf_private_bindingsizes and binds by their demand at every build (a storage section with one is never kept).From d1e33769 (as at 6a4a8ec0) and 498b07fa on
exp/x86-asm-sync.Read first: the comment above
kSectTex, thendesc_sections_on,desc_tex_kept,desc_ssbo_keptandrefresh_uniform_descriptors, the head ofbuild_descriptorsand the sections' condition indraw()(gpu/vk/vk_draw.cpp).Testing
Tested: Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head
f1249421on main @a2dc5385. Every check ran through the PC's job queue in build jobs, in a hidden console, with the Vulkan headers found (jobsdsc2210,dsc2210-m2):tests/test_desc_sections.py: pass. It cuts from the source, compiles and runs the rules for which descriptors a section can keep, the checks against what a section was made from (each part of a section's record moved alone fails its check; a texture that went, on a page made inaccessible after the destroy generation moved, fails the check without being read; an eviction whose memory is made into another texture at once fails it on the generation alone) and the refresh; runs both switches' readers in fresh processes (unset,0,off,false,no,1,onand another value, twice each, set the other way after the first read:RX_VK_DESC_SECTIONSunset is off and logs nothing,RX_VK_DESC_REFRESHunset is on;RX_VK_DESC_SECTIONS=1is off with each per-binding diagnostic set); and readsvk_texture.cpp(a texture leaves the texture table only afterdestroy_latermoved the generation),draw()andbuild_descriptors(where a section is kept, that a kept one looks nothing up, and that with the switch off no statement naming the sections' own state runs). The first new commit alone (6a8299fa: the two texture cases on the earlier code) passes too.tests/test_desc_counts.pyandtests/test_hidden_mesh_depth.py(they read the build's signature): pass.gpu_writtenloop moved to the top ofdesc_tex_kept(the texture-that-went case: it reads the page and stops with its FAIL line); thedestroy_laterbefore one erase from the texture table removed; the default turned on; and each of the eight off-path statements put back outside the switch.--changed a2dc5385; its own are among them): 181 passed, 11 failed, 6 skipped (Linux only). Failed:tests/test_crash_dialog.py,tests/test_dev_ab.py,tests/test_dev_packs.py,tests/test_draw_stream_visibility.py,tests/test_fsr_uniform_ownership.py,tests/test_input_movie.py,tests/test_recording_admission.py,tests/test_sampler_admission.py,tests/test_sim_turf_pools.py,tests/test_switch_table.py,tests/test_tls_objects.py: each fails on main @a2dc5385too (the same 11 in the earlier batch's run of main,t-DescCut3-base);tests/test_switch_table.pyhere only for main's own unsortedRX_VK_GRASS_DISTANCErow.gpu/vk/vk_draw.cppcompiled with the game build's own command (syntax only): pass.It has NOT run in the game on this PC yet.
Still to test:
RX_VK_DESC_SECTIONSunset (main's build) againstRX_VK_DESC_SECTIONS=1in this PR's build, a warm-up a side then 8 a side interleaved,RX_VK_PROF_FINE=1on every run (the descriptor time and draw time a frame), andRX_VK_ASYNC_PARTSandRX_VK_ASYNC_FENCES(#2211, #2212) set the same on both sides; the counter is the frame log'svk: a frame: ... draws kept their texture section, ... their storage section (of ... that built descriptors)line on the=1side (the unset side has none); then the Castle Town Market a run a side. Expected: the same picture both ways, and with=1the descriptor time a frame the same as or lower than with=0(by hand)RX_VK_DESC_SECTIONS=1, warmed, in ABBA order, expecting the same picture both ways and no crash or validation error (Test Farm: linux+nvidia, linux+amd, windows+amd, windows+intel, steamdeck: Kokiri Forest a minute, main against this PR with RX_VK_DESC_SECTIONS=1, ABBA, the same picture)tests/test_desc_sections.pywith--cxx g++and--cxx clang++, expecting PASS (by hand)Measurements
Verdict: not measured: no micro-benchmark of this port and no run in the game on this PC yet, so it is not shown that this avoids the loss #794 measured (its descriptor time slower than main); the in-game A/B (descriptor and draw time a frame,
RX_VK_DESC_SECTIONS=0against=1) is in Still to test. SoRX_VK_DESC_SECTIONSships off (an attempt), as dazemc's review asks, until that comparison exists;RX_VK_DESC_REFRESHacts only under it. Off, the build does none of the sections' work (Summary); what is left on that path, the switch's test and branches on it, is not timed. The other names the diff adds are existing diagnostics, each already ingpu/vk/switches.defwith its own default (off, unset), which this change does not change:RX_VK_DRAW_STATS,RX_VK_DRAW_STATS_PROGS,RX_VK_SLOW_DESC,RX_VK_WATCH_PROG,RX_VK_PEEK_PROG,RX_VK_TRACE_TEX,RX_VK_LOG_IMG,RX_VK_LOG_HITS,RX_VK_SSBO_STRETCH,RX_VK_SSBO_WIDE,RX_VK_ASYNC_IMPORTandRX_VK_GP_TABLE: set (any value), each turns the sections off, since a kept section would hide the bindings it looks at or changes.Risks
A kept section that building it again would not give is a wrong texture, a stale storage buffer, or a view or sampler already freed (a crash or a lost device). Every part of what a section was made from is checked before it is kept (the test moves each alone), and anything not known to stand builds the section again. The samplers are kept on the same terms as main's own whole-set reuse (
same_desc): while the samplers' registration generation stands. A texture's pointer in a kept section's record is read only after the destroy generation's check, and a texture leaves the texture table only afterdestroy_latermoved that generation (the test checks both, with a texture that went and one made again in its memory). The new frame-log line is printed every 120 presents while the sections are on. On only withRX_VK_DESC_SECTIONS=1; unset or0is main's build;RX_VK_DESC_REFRESH=0turns off only the in-place refresh. The renderer is compiled again after updating;Program(gpu/vk/vk_shader.h) gains a field and no file innx2/translator_files.pychanges, so players' programs are not translated again.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named). Not yet: only the checks in Testing ran.main, or Measurements says why there is nothing to measure. Not yet (Measurements).third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def. Its rows are in;RX_VK_DESC_SECTIONSships off (an attempt) until it is measured.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.@Beanstalk This keeps a stage's texture and storage descriptor sections across draws while what they were resolved from stands (gpu/vk), in every scene, draw-heavy ones most; RX_VK_DESC_SECTIONS=0 turns it off (RX_VK_DESC_REFRESH acts only under it). Please start from the kokiri-start save and take the picture 5 s (300 frames at 60 fps) after the first frame whose picture is not a single uniform colour, with no input in between (a picture during the load's fade-in from black is not comparable), with a main-against-itself picture on the same PC from the same point; then, for frame times only, play Kokiri Forest and Hyrule Field, two minutes each, and Castle Town Market for two minutes from the market-entrance save, with a picture there taken the same way; at least two starts a side, interleaved, the PR against main on the same PC, on the PR's side unset (the default, on) and with RX_VK_DESC_SECTIONS=0 (off), on Windows with an RTX 4070, Windows with an Arc B580, Linux with an RX 6600 (RADV) and the Steam Deck. The same picture as main within main's own difference from itself; with it unset, frame time p99 equal to main's or better, within the run-to-run spread, and equal to RX_VK_DESC_SECTIONS=0 or better. The log must have no FATAL, device lost or validation error line. No stale or wrong texture in any picture.
@siantm only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was for65da04c, and the PR has changed since (f124942), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.Description: Prior work now says #794 measured a loss (its descriptor time slower than main) and what this narrower form leaves out; this PR's own descriptor time is not measured yet (Verdict says the same).
The switch ships on with no measurement, after #794 measured a loss — make it
attemptand off until the A/B.The body's Measurements says "Verdict: not measured: no micro-benchmark of this port and no run in the game on this PC yet"; the two rows are
kill/on (gpu/vk/switches.def), and the checklist's "ships at the default the measurements chose" is unticked. The Prior work names #794, the same idea, whose own A/B measured descriptor time slower than main (3.4–3.9 ms against 2.5–2.9 ms) and was closed. A perf switch whose only prior evidence is a regression should ship off /attemptuntil the in-game A/B; otherwise every player gets the unmeasured path by default. This is the same pattern as #2212. The code may well win, but the default is a product decision and the measurement is the thing that should make it.Lifetime, read from the source (not run).
bufs/imgs/viewsfor the immediately preceding build of the same program, only in the same recording and only over the build that was bound (cb.desc.build == desc_build_no,out.prog == &p, equal counts). Released by another build, a dispatch, a draw left out between build and bind, a new recording/clear/upscaler, the mesh fallback, a frame dump,RX_VK_DESC_REUSE=never, or any failed check.tex_genper stage,texture_destroy_generation()(stored and current), sampler registration generation, texture epoch, slot versions (or the registrations' generation when the row is off), and each resolved texture'sgpu_writtenstate.destroy_genis bumped bydestroy_later, compaction and deferred uploads, so destroyed, evicted and reused textures fail the check.ssbo_genper stage,gpu_bind_stamp()(pools_genandmirror_epoch, the stampgpu_buffer_bind's own memo uses), anddesc_ssbo_againfor an unbound buffer or one in a virtual pool. There is no separate buffer destroy generation, butgpu_bind_stampis the renderer's own binding identity; no path found where a section outlives the buffer it names within a recording.desc_tex_seen'sconst Texture*; it is safe only because the destroy-generation checks come first. Notexture_alivecall, but no reordering issue found in the code.Test.
tests/test_desc_sections.pyfails on main because the cut functions/fields do not exist (the extraction raises), so it is more than a switch parse; it runs the rules, bothdesc_tex_kept/desc_ssbo_keptand the refresh. It coversg_destroy++anddgen+1, but never a destroyedTexture:texsstay alive, and the only lifetime state it moves is the generation andgpu_written. So a real eviction/reuse/free, and the ordering "destroy generation before theTexture*deref", are untested. Add one case with a deadTexture*/texture_alive(or a comment pinning the order) and one eviction/reuse case.Off path. Behaviour is main's (same
out.reset(), same writes, no refresh), but the cost is not only a branch:st_desc_buildsis an atomic add per build,desc_build_no++a thread-local add,out.has_storage_buffersa write per storage write,desc_tex_dummiesan add per dummy,desc_ssbo_againon an unbound buffer. No timer is read. The atomic per build is the one to measure or drop when off.Pins. Our #1988 texture drain uses the same
destroy_gen; sparse shadows usegpu_bind_stamp; #2211 (RX_VK_ASYNC_PARTS) and #2212 (RX_VK_ASYNC_FENCES) touchvk_replay.cpp. No hunk overlap with #2210'svk_draw.cpp/vk_shader.h(rebase only), but an A/B must pinRX_VK_DESC_SECTIONS=0against unset and those switches on both arms; the frame log's kept-section line is the counter.No fix claim: the title says
perf, and the body does not claim one. The default is the concern, not the wording.WIP: perf(vk): a draw of the program before it keeps its texture and storage descriptors while their bindings stand (RX_VK_DESC_SECTIONS, on)to WIP: perf(vk): a draw of the program before it keeps its texture and storage descriptors while their bindings stand (RX_VK_DESC_SECTIONS, off)@dazemc thank you for the review (72495). Pushed
6a8299faandf1249421on top of65da04cc(main is stilla2dc5385, so no rebase). The description and the title are updated.Default.
f1249421:RX_VK_DESC_SECTIONSis now anattemptrow and ships off. Unset or0is main's build, and unset logs nothing;1turns the sections on.RX_VK_DESC_REFRESHacts only under it: with the sections off no build keeps a section, so the refresh does nothing and is never read. The reader's test covers unset (off),0/off/false/no(off) and1/on/any other value (on). The description says the default follows your review and stays off until the in-game comparison exists.Test gaps.
6a8299faadds two cases, cut from the source as before:Textureis on a page of its own, which is made inaccessible after the destroy generation moves (asdestroy_latermoves it before the texture table frees theTexture).desc_tex_keptmust fail, with the draw's generation read either before or after, and any read of that page stops the run with a FAIL line.The test also reads
vk_texture.cpp:destroy_latermoves the generation first, and every erase from the texture table has adestroy_laterbefore it in its function. Mutants on copies, each case run alone: without the destroy-generation check, both cases fail; with thegpu_writtenloop moved to the top ofdesc_tex_kept, the first one fails; with thedestroy_laterbefore one erase removed, the source check fails.6a8299faalso passes on its own, on the earlier code.Off path. I dropped it rather than keep and measure it (
f1249421). With the switch off, none of these run now: the build countst_desc_builds, the build number (its add, and its read at the bind), the two thread-local lookups per build (the kept infos and the texture record, which your list did not name), thehas_storage_bufferswrite, the dummy add anddesc_ssbo_again. Each is under the switch or a kept section. What the off path still adds to main's work is the switch's test, a function-local static, in the build and indraw(), a call to its reader at each bind, and branches on it. I have not timed that remainder. The test now readsbuild_descriptorsanddraw()and fails on any statement that names the sections' own state outside the switch or a kept section. Each of the eight statements above, put back outside the switch one at a time, fails it.Pins. No commit. The held A/B on this PC is now
RX_VK_DESC_SECTIONS=0against=1in this PR's build, withRX_VK_ASYNC_PARTSandRX_VK_ASYNC_FENCESset the same on both sides and the frame log's kept-section line as the counter (on the=1side). Still to test says the same. It has not run yet.Testing (in the description):
tests/test_desc_sections.pypasses. The 198 checks that read a changed file give 181 passed, 11 failed and 6 skipped (Linux only); the 11 failures are the same 11 that fail on maina2dc5385.vk_draw.cppcompiles with the game build's command.@Beanstalk This replaces my request 71737, made for the earlier head
65da04cc, where the switch was on by default. This adds RX_VK_DESC_SECTIONS, an attempt off by default: with RX_VK_DESC_SECTIONS=1 a stage's texture and storage descriptor sections are kept across draws while what they were resolved from stands (gpu/vk), in every scene, draw-heavy ones most; unset or =0 is main's way (RX_VK_DESC_REFRESH acts only under it). Please start from the kokiri-start save and take the picture 5 s (300 frames at 60 fps) after the first frame whose picture is not a single uniform colour, with no input in between (a picture during the load's fade-in from black is not comparable), with a main-against-itself picture on the same PC from the same point; then, for frame times only, play Kokiri Forest and Hyrule Field, two minutes each, and Castle Town Market for two minutes from the market-entrance save, with a picture there taken the same way; at least two starts a side, interleaved, the PR against main on the same PC, on the PR's side unset (the default, off) and with RX_VK_DESC_SECTIONS=1 (on), on Windows with an RTX 4070, Windows with an Arc B580, Linux with an RX 6600 (RADV) and the Steam Deck. The same picture as main within main's own difference from itself; unset, frame time p99 equal to main's; with RX_VK_DESC_SECTIONS=1, equal to unset or better, within the run-to-run spread. The log must have no FATAL, device lost or validation error line. No stale or wrong texture in any picture.@siantm only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.