perf(vk): textures prepared on worker threads as the game makes them (RX_VK_TEX_PREP, off by default) #1847
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1847
Loading…
Reference in a new issue
No description provided.
Delete branch "GummiNaomi/Longshot:perf/tex-prep-workers"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Arriving in an area, the replay thread copies, hashes and lays out the storage of every new texture at its first draw, inside the frames that draw the new area. This moves that work to worker threads as the game makes each texture (
RX_VK_TEX_PREP): the upload then takes the worker's buffer when the storage still hashes the same, and does as before otherwise. It is an attempt, off by default on every PC:=1turns it on,=2also checks each upload.Measured at this head on main, it does not help: in two teleport tours a side on 4 cores (where it is on by default), against main @
65488813, an arrival's time over 16.7 ms a frame went 350 -> 396 ms (median), its longest frame 104 -> 133 ms (median) and 244 -> 211 ms at most, and the game's most private memory went up by 1.3-1.4 GB (22.9-23.0 -> 24.2-24.4 GB). The earlier gain (an arrival's excess 496 -> 370 ms) was measured on a build with #1810's moves and a carry-over of the prepared buffers across them, which this PR leaves out; main without those arrives smoother than that build did, and this head adds nothing to it. So it now ships off on every PC (ebd6c08a); before, it was on with 8 usable CPUs or fewer. It stays as an opt-in until a PC where it helps is found (Still to test).Why: a texture's first draw makes its image and fills it from the game's memory (
upload_inner,gpu/vk/vk_texture.cpp): the storage is read, hashed for the ASTC decoder's reuse, and staged (ASTC) or unswizzled (native BC and uncompressed) on the replay thread. An arrival makes hundreds of textures in its first frames, so this lands in them. The game announces each texture well before it draws it (nvnTextureInitialize), and its storage is mostly written by then.What changes:
texture_prep_made(called fromtexture_created_of,gpu/vk/vk_replay.cpp) queues each texturetex_prep.htakes: 2D, 2D array, cube and cube array textures, ASTC up to 256 MB (whole layers, laid out as the GPU decoder reads them) and native BC or uncompressed ones under 8 MB (unswizzled level by level as the CPU route does), not render targets, depth, multisampled or non-contiguous ones. One worker (two from 8 usable CPUs,RX_VK_TEX_PREP_THREADS), at below-normal priority and only once the renderer has presented, copies the storage once, hashes the copy (the same chained XXH64 in 64 MB pieces asreuse_upload) and fills a cached host buffer. A whole upload (upload_inner, no dirty ranges) takes the buffer of its texture's newest serial: for ASTC it compares the hashreuse_uploadmakes anyway and decodes from the worker's buffer; for the CPU route it hashes the storage as the upload reads it (guest_bytes), checks every level's geometry, and leaves out a level that runs past the storage while a kept capture page lies there; then one copy from the worker's buffer. Any mismatch, or a buffer not ready in time, uploads as before. Buffers the GPU used are freed once their recording completes; at mostRX_VK_TEX_PREP_MB(1024) are held, the oldest let go first, and a texture finalized drops its own.RX_VK_TEX_PREP=2also does each such upload the old way and compares. The log has a "textures prepared" line every 600 presents and the two routes in "a frame's uploads took". Unchanged: partial uploads,reuse_uploadand its ASTC cache, what is drawn.This was written and first measured on top of #1810 (open), where a texture the game's compaction moves keeps its image; there a move also carried its prepared buffer over. Main has no such moves (a moved texture is a new one, prepared on its own), so this PR leaves that part out; it goes in with whichever of the two lands second.
Read first:
gpu/vk/tex_prep.h, thenprep_worker,prep_takeand their use inupload_inneringpu/vk/vk_texture.cpp, andtests/test_tex_prep.py.Testing
Tested: Windows 11 Pro 26200, RTX 4070 SUPER (NVIDIA 617.42), Ryzen 9 9950X3D with SMT off, 32 GB.
At
a35e5edd(on main @65488813), the code of this head before it was rebased and its default turned off (with the old default, unset meant on with 4 usable CPUs, as=1does now): the builder's Windows (llvm-mingw) builds of this head and of main @65488813, and four teleport tours on 4 cores, main, this PR, this PR, main (Measurements). No crash, FATAL or device loss in any; every teleport arrived (in one tour a side one was refused at the first try, "an event is playing"). With it unset on 4 cores the log says 1 worker; in a tour 34,000-43,600 textures announced and prepared (16.7-18.4 GB, 13-15 s of the worker), of which 4,278-5,286 CPU-route and 3,118-3,821 ASTC first uploads took the worker's buffer; 4-10 were not ready in time and none written since; 1,017-1,019 MB of prepared buffers held at the tour's end (the 1024 MB limit). The pictures were not compared at this head.Earlier, on builds of main @
f505340bwith #1810-#1813's changes and this one (with the move carry-over above): teleport tours on 4 and 16 cores, two a side; no crash in the six tours with it on of the last two builds; the arrivals' pictures alike both ways (mean pixel difference 2.8-6.6 of 255: Navi and particles); a check run withRX_VK_TEX_PREP=2on 16 cores, 15 teleports: 10,298 unswizzled and 10,736 ASTC uploads from the workers' buffers, every one the same as the old path's ("check: all the same", no DIFFER line).Checks:
tests/test_tex_prep.py(tex_prep.h's rules compiled and run: which textures, their layers and levels, the hash, the ASTC layout, the ready list's room and age, the workers' count and the CPU rule; the wiring in the source, and no call that both moves a prepared texture and reads it, the start-up crash an earlier version had) passes on this head on Windows (clang) and Linux (g++ 13.3, WSL Ubuntu 24.04) and fails on main;tests/test_switch_table.py,tests/test_astc_reuse.py,tests/test_bc_upload_retry.py,tests/test_astc_upload_chunking.py(stand-ins for the prep's names),tests/test_texture_staging_publication.pyand the BC scratch checks pass on Windows and in WSL.Suite at this head,
tests/run_all.py --changed 65488813in WSL Ubuntu 24.04: 156 pass, 30 fail, 5 skipped. 29 of the 30 fail on main @65488813there too:test_capture_readback_visibility,test_crash_dialog,test_dev_ab,test_dev_packs,test_draw_stream_visibility,test_early_split_before_draw,test_frame_overlap,test_fsr_image_retirement,test_fsr_uniform_ownership,test_hold_gpu,test_hud_layout_row,test_image_view_retry,test_linux_pressure,test_pass_barrier,test_present_queue_default,test_present_source_crop,test_profiling_mode,test_readback_early,test_readback_late,test_readback_timeout_source,test_replay_write_order,test_slow_frames,test_split_before_draw,test_staging_burst,test_texel_buffer_ownership,test_ub_update_batch,test_upscaler_rows,test_vk_occlusion_queries,test_walk_key. The 30th,test_low_vram_texture_mips, fails in WSL on the text ofvk_menu.cpp, a file this PR leaves alone; run on Windows it fails on main @65488813with the same error as on this head (its marker is invk_texture.cpptwice). (The full suite with every check on main @068fb8ebin WSL, and this head's--changedchecks on Windows as it was on068fb8eb, are in this PR's earlier description: every failure there also failed on main.)No Linux build was done: on Linux only the checks above ran, in WSL.
Still to test:
RX_VK_TEX_PREP=1, ABBA, to see whether the arrivals are any smoother there, expecting the private memory about 1 GB higher with=1(@Beanstalk steamdeck, teleport tour 4 minutes x 4, RX_VK_TEX_PREP=1 against unset, ABBA, with the frame trace and the private memory)RX_VK_TEX_PREP=2, expecting "check: all the same" (@Beanstalk amd and intel GPUs, a teleport tour with RX_VK_TEX_PREP=2, the log's check line)Measurements
Rebased and turned off: this head,
ebd6c08a, isa35e5eddrebased onto main89c3d21ewith no conflicts (1fb19086), then one commit that makesRX_VK_TEX_PREPoff by default on every PC and an attempt in its row. The numbers below are froma35e5eddon65488813, where unset was on with 4 usable CPUs, the same as=1now. Atebd6c08a,tests/test_tex_prep.pyandtests/test_switch_table.pypass on Windows (clang), andvk_texture.cppandvk_replay.cppcompile syntax-only with the builder's flags;tests/test_astc_upload_chunking.pyfails to compile the same way on main89c3d21e.Verdict: no gain at this head, and a memory cost. On 4 cores, with it on (the old default there), against main @
65488813: an arrival's excess 350 -> 396 ms (median) and 422 -> 433 ms (mean), its longest frame 104 -> 133 ms (median) and 244 -> 211 ms at most, its uploads and lookups about the same; the tours' medians of the excess (361 / 333 against 396 / 396) and of the longest frame (108 / 101 against 130 / 136) do not overlap, so on those it is slightly worse, better only on the single longest frame. The most private memory in a tour is 1.3-1.4 GB higher.Setup: Windows 11 Pro 26200, RTX 4070 SUPER 12 GB (NVIDIA 617.42), Ryzen 9 9950X3D with SMT off, 32 GB; the game held to 4 cores (process affinity 0xF00). Teleport tours from
kokiri-start, 4 minutes each (14-15 arrivals: Castle Town's market, Hyrule Field, Kakariko, Kokiri Forest, the Lost Woods, Zora's River, round again); the control tool'stestprofile (window 1920x1080, headless), uncapped (HOS_FPS_MAX=1000); both sidesRX_VK_LATE_FIX=1 RX_VK_TEX_AHEAD=2 RX_VK_ASTC_BC_ALL=1 RX_VK_IMAGE_BLOCK_MB=32 RX_VK_LOAD=create RX_VK_BC_LAYERS=8 RX_VK_FRAME_TRACE=1, this PR ata35e5eddwithRX_VK_TEX_PREPunset, which with that head's default was on (4 usable CPUs), as=1is now. Order main, PR, PR, main, 20 Nov 13:12-13:32, one build a side. Each arrival's first 3 s from the frame trace: its excess (the time over 16.7 ms a frame), its longest frame, and the uploads' and lookups' columns (means over the arrivals); the save's load-in left out (27 and 28 teleports). The private bytes sampled every 2 s, the most in each tour.65488813a35e5eddA tour's average fps and median frame time mix the title, the loads and play, which take different shares in each tour, so they are not compared. Not measured: 16 cores, anything between 4 and 16 cores, the pictures.
Baseline (the earlier table): builds of main @
f505340bwith #1810-#1813's changes and this one, with the move carry-over this PR leaves out:RX_VK_TEX_PREP=0against on, two tours a side on each core count, in the order off, on, off, on; the same tour and method, andRX_VK_TEX_MOVE=1 HOS_NVN_FRAME_OVERLAP=1, switches main does not have. The off tours ran on an earlier and the final build of that change, the on tours on the last two.RX_VK_TEX_PREP=0@f505340b+ #1810-#1813f505340b+ #1810-#1813(Those tours ran in the same 1920x1080 window as above; the earlier description's 2560x1440 was a mistake.)
RX_VK_TEX_PREP(new, an attempt) ships off on every PC;=1turns it on and=2also checks. The earlier default, on with 8 usable CPUs or fewer, was chosen by the earlier table, on a build with #1810's moves; ata35e5eddon main the 4-core tours above showed no gain and 1.3-1.4 GB more memory, so it is off by default now (ebd6c08a).Risks
What could break: the prepared buffers hold up to 1 GB of host memory (
RX_VK_TEX_PREP_MB; 1.3-1.4 GB more private memory in the tours at this head), for a player who turns it on (RX_VK_TEX_PREP_MBlowers it). Off by default, nothing a frame runs changes apart from one check of the switch, read once. On main it did not make arrivals smoother on 4 cores (Measurements). An upload uses a worker's buffer only when the storage hashes the same as it reads it, and the check run found every one the same, but a texture written between the hash and the GPU's read would be as stale as on today's path, no more. The workers take CPU time at below-normal priority (13-15 s of one worker in a 4-core tour of about 4 minutes at this head). After updating, nothing changes unlessRX_VK_TEX_PREPis set; nothing changes either way for saves,longshot_options.inior the caches. A revert is enough.Checklist
Ready for review
main; the title is a Conventional Commit. (Rebased onto main89c3d21e, no conflicts; thenebd6c08aturns the default off.)mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure. (Two tours a side on 4 cores at this head against main @65488813, ABBA.)third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def. (Off: on main the 4-core tours showed no gain and 1.3-1.4 GB more memory. Its row is an attempt.)tests/that reads changed source is updated, not weakened.a35e5edd, run in the tours above. On Linux only checks:tests/test_tex_prep.py(tex_prep.hwith g++ 13.3) and the--changedsuite in WSL Ubuntu 24.04. No Linux build was done.)Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
429766c8d0abb0cb0e43abb0cb0e43a35e5eddd91GB more memory is big
@Beanstalk
Beanstalk test farm · an earlier run: commit
a35e5ed(the newer results are in a comment below)Commit
a35e5ed· tier reviewed · approved by HumanBean · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)A short teleport tour compares main and the PR at default settings, then repeats with texture-preparation byte checking forced on. It covers Kokiri Forest, Hyrule Field and Kakariko Village, including a return visit, in about 20 minutes per tester. The plan measures arrival hitches and peak process memory, and compares arrival pictures. Crashes, texture-check mismatches, materially worse hitches or memory, and large visual differences fail the plan; the memory limit allows for the reported extra allocation.
2(PR only).kokiri-start, then teleport to Hyrule Field, take the picture field to compare with main, teleport to Kakariko Village, take the picture village to compare with main, teleport to Kokiri Forest, measure forest for 20 s (after 2 s warm-up), take the picture forest to compare with main; 1 run on each side.kokiri-start, then teleport to Kokiri Forest, teleport to Hyrule Field, take the picture field-check to compare with main, teleport to Kakariko Village, measure checked-forest for 20 s (after 2 s warm-up), take the picture forest-check to compare with main; 1 run on each side.Done (started 08:00 UTC, last report 08:38 UTC). Comment
@Beanstalkto run it again.NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0; 2: windows+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 4080, 621.1.0Measurements · any
Verdict: could not be judged on any (2/1 report).
Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu
NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM; 2: windows+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 4080, 621.1.0AMD Ryzen 7 9800X3D 8-Core Processor, 32 GB RAMScene: default-tour, from the repository's save kokiri-start; checked-tour, from the repository's save kokiri-start; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); default-tour: 1 game start a side, forest 20 s after 2 s warm-up; checked-tour: 1 game start a side, checked-forest 20 s after 2 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)Nothing was measured: no metric came back.
Flags and settings
RX_VK_TEX_PREP2Markdown for the PR description
WIP: perf(vk): textures prepared on worker threads as the game makes them, on PCs of 8 usable CPUs or fewer (RX_VK_TEX_PREP)to perf(vk): textures prepared on worker threads as the game makes them, on PCs of 8 usable CPUs or fewer (RX_VK_TEX_PREP)a35e5eddd9ebd6c08accBeanstalk test farm · 🔄 testing (2 of 2 reports so far)
Commit
ebd6c08· approved by GummiNaomi (reviewed) · dashboardDone (started 14:53 UTC, last report 15:10 UTC). Comment
@Beanstalkto run it again.steamdeckwindowsCouldn't be judged: the tester's PC could not finish the test. The farm tries another PC, or the same one after a fix; it says nothing about the PR on its own.
windows, report 1:the main base failed (exit 1): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^windows, report 2:no scenario could be judged: a step ran out of time: timeout: not met in timeWhat was tested
What will be tested · AI plan (
gpt-6.1-terra)On Steam Deck and Windows, two runs per side will tour six destinations from Kokiri Forest, spending 3.5 minutes in measured segments per tour. The PR with texture preparation enabled is compared with main unset; arrival-segment longest frames, whole-run hitches, peak resident memory and screenshots are recorded. Crashes or material screenshot differences fail the plan, while smoothness and memory are reported without requiring a gain. The farm cannot select Windows PCs by core count, enable the requested frame trace, or measure peak private memory; resident memory is a proxy and arrival-segment maxima exclude the preceding load.
1(PR only).kokiri-start, then teleport to Market, take the picture market to compare with main, measure market for 35 s (after 0 s warm-up), teleport to Hyrule Field, take the picture field to compare with main, measure field for 35 s (after 0 s warm-up), teleport to Kakariko Village, take the picture kakariko to compare with main, measure kakariko for 35 s (after 0 s warm-up), teleport to Kokiri Forest, take the picture kokiri to compare with main, measure kokiri for 35 s (after 0 s warm-up), teleport to Lost Woods, take the picture woods to compare with main, measure woods for 35 s (after 0 s warm-up), teleport to Zora's River, take the picture river to compare with main, measure river for 35 s (after 0 s warm-up); 2 runs on each side.Measurements on `windows`, for the PR description
Verdict: could not be judged on windows (2 reports).
Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu
NVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAM; 2: windows+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 3090, 621.1.0AMD Ryzen 9 5900X 12-Core Processor, 128 GB RAMScene: teleport-tour, from the repository's save kokiri-start; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); teleport-tour: 2 game starts a side, market 35 s after 0 s warm-up, field 35 s after 0 s warm-up, kakariko 35 s after 0 s warm-up, kokiri 35 s after 0 s warm-up, woods 35 s after 0 s warm-up, river 35 s after 0 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)89c3d21ebd6c08RX_VK_TEX_PREP=1 (PR only)· forbidden log linesFlags and settings
RX_VK_TEX_PREP1As Markdown for the PR description:
The testers' hardware
windowsNVIDIA GeForce RTX 3050 Ti Laptop GPU, 621.24.0AMD Ryzen 7 5800H with Radeon Graphics, 16 GB RAMwindowsNVIDIA GeForce RTX 3090, 621.1.0AMD Ryzen 9 5900X 12-Core Processor, 128 GB RAMperf(vk): textures prepared on worker threads as the game makes them, on PCs of 8 usable CPUs or fewer (RX_VK_TEX_PREP)to perf(vk): textures prepared on worker threads as the game makes them (RX_VK_TEX_PREP, off by default)@HumanBean you were right about the memory. Measured at this PR's code on main (
65488813, before the rebase), 4 cores, two teleport tours a side, ABBA: no smoother arrivals (median excess 350 → 396 ms, longest frame 104 → 133 ms median) and 1.3–1.4 GB more peak private memory. The earlier gain came from a build that also had #1810's moves. SoRX_VK_TEX_PREPis now off by default on every PC (ebd6c08a, rebased onto89c3d21e), an opt-inattempt; the description has the numbers.@Beanstalk steamdeck, and windows with 6 to 8 cores: a 4-minute teleport tour from kokiri-start (Castle Town's market, Hyrule Field, Kakariko, Kokiri Forest, the Lost Woods, Zora's River),
RX_VK_TEX_PREP=1against unset, ABBA,RX_VK_FRAME_TRACE=1, with each arrival's longest frame and the peak private memory. The question is whether=1makes arrivals smoother anywhere; expecting about 1 GB more memory with it.(This replaces today's run, which ended in
errorbefore building: the farm's checkout of main failedgit rev-parse --verify HEAD^{commit}in packs.py's cmd_base.)Closing: no gain at this head, and 1.3-1.4 GB more peak private memory, as @HumanBean said. On 4 cores arrivals were slightly worse with it on (median excess 350 -> 396 ms, longest frame 104 -> 133 ms), and the farm couldn't judge it. The earlier gain came from a build that also had #1810's moves.
Pull request closed