perf(vk): 4K is CPU-bound by the memory sync (35.6 fps at 4K vs 60 at 1080p, GPU starved) — finish the remaining legs #2099
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot#2099
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
At 4K the game runs at 35.6 fps where the same scene holds 60 at 1080p, and the GPU sits far under its power limit: the frame is bounded by CPU-side memory-sync work whose size follows the framebuffer's bytes, not the CPU's own traffic. #1629 removed the largest leg (the blind size skip); the legs that are left — the mirror-sync page churn, the capture scans, the write-back compare and the game-thread readback waits — are what this issue proposes to finish. The groundwork is done and reviewed; what is missing is upstream implementation of the three designs below.
The evidence (RX 9070 XT, RADV, Linux, 4K, v2.1.0+)
From
hookshot_log.txt(the log holds no game data):gpu/vk/vk_memory.cpp:3420-3424.vk_memory.cpp:3427-3430; the passgpu/vk/shaders/wb_diff.comp).vk_memory.cpp:2824).invalidatewaits for the whole submission's write-back value (gpu/vk/vk_replay.cpp:2264-2272), not for the small result the CPU is about to read; the average is logged atvk_replay.cpp:2285-2289.Every leg scales with the bytes the GPU writes back, which is why resolution is the knob that moves them.
Proven already (completed local analysis, reviewed — not yet upstream)
Dirty-origin split (Phase 1). A parallel GPU-write generation stamp alongside the CPU's page generations, its own counter bumped per submission and its own per-pool sync baseline. A pure classifier,
dirty_origin_of, keys each 4 KB page clean / CPU-dirty / GPU-dirty / both, and the three consumers re-key on it:mirror_syncskips GPU-dirty-only pages (the mirror holds the truth; #1629's write-back brings it to guest memory), the capture scans prune ranges with no CPU-dirty page on even submissions (odd ones scan everything, so no range starves), and the write-back end narrows #1629's large both-dirty exceptions to the pages the pass must compare (pending pages never culled). Kill switchRX_VK_DIRTY_ORIGIN=0restores every pre-split path. Branchwb-tiles: stamps and classifiergpu/vk/vk_memory.cpp:631-689, the mirror-sync skip:2422-2463, the capture prune:1578-1614, the write-back cull:3704-3719and:3740-3760, the gate narrowing:2896-2915, counters:3623-3639. Two independent reviewers found it sound. It has not been built or run; that is the first thing upstream would do.The prefilter proof (Phase 2 attempts, both negative — and that is the useful result). Two independent Phase-2 implementations (branches
phase2-glm,phase2-mimo) tried to cull write-back compare bytes from CPU-observable signals and both hit the same wall: a tile mask marked at bind time covers every queued run by construction — the same binds that queue a write-back mark the tiles — so intersecting it culled nothing, while its overflow arm dropped unmarked tiles (data loss). Both attempts deleted their prefilters honestly and left the proof at the cull site:phase2-glmgpu/vk/vk_memory.cpp:3729-3735,phase2-mimogpu/vk/vk_memory.cpp:4024-4031. Both reached the same conclusion: the only exact subset signal is the word-diff pass's own differ output — the GPU's observation of what it actually changed — and using it needs new GPU-side machinery, not a better CPU-side prefilter. The three proposals below are what survives that proof.Proposed work
1. GPU-observed dirty tiles: feed wb_diff's per-tile differ output back as the next submission's cull input
Today the pass computes
differsper vector and folds the shadow (shaders/wb_diff.comp:29-44) but records nothing per tile: after it runs, the only trace of what the GPU changed is its own writes. The proposal:wbd_split's chunk size). The pass itself (or a tiny fold dispatch after it) sets a tile's bit with an atomics when any of its words differed — the branchphase2-glmnote atgpu/vk/vk_memory.cpp:3733-3735specifies this shape: "a GPU-resident mask with an indirect-dispatch count, not a CPU intersect".wbd_keep_page's GPU-dirty leg for the next submission's chunk-table build. At the log's 3.3 GB compare window that is ~53k tiles ≈ 6.6 KB of bits — a small, latency-tolerant readback, unlike the frame-sized ones proposal 2 fixes.RX_VK_DIRTY_ORIGIN=0opt-out covers it (or its ownRX_VK_WB_TILES=0), with the fallback being the Phase-1 page-level cull unchanged.2. Urgent/bulk write-back split: game-thread readbacks wait on a small-result signal, not whole-frame retire
The bookkeeping design exists (branch
phase2-glm); the signal machinery is what is missing. The design:memory_rb_refreshrefreshes (readback results the CPU is about to read), runs of 16 KB or less, and odd (non-word-aligned) leftovers the diff pass cannot take; BULK = the rest (framebuffer tiles). Pure classifiers:gpu/vk/vk_memory.cpp:1430-1449.:1451-1474; the copies at:3919-3943).invalidatewaits on that value only, falling back to the full value once for a region never read back before (gpu/vk/vk_internal.h:228-233,gpu/vk/vk_replay.cpp:2277-2301).mem_urgent_value == mem_rec_valueand the urgent wait is today's wait, only narrower —phase2-glmgpu/vk/vk_memory.cpp:3921-3923says exactly this: "a split submit signalling urgent right after these copies is staged, not this change". The work is recording a second timeline value (or semaphore) signalled by the submission's urgent-only tail, the catch-up path's handling of it, and the ordering argument (urgent copies after the first barrier, both tiers before the next submission's work,:1425-1429).3. Measured demotion for CPU/GPU-colliding pools
The churn in the log (289 dirty pages a submit, 99.6% from earlier captures) concentrates in pools where the CPU and the GPU write the same pages: the sync copies the CPU's version over the GPU's, the GPU writes it again, the next capture lags, repeat. Blind policies (idle timeouts, budget refusals) move whole pools on guesses. A related approach exists outside upstream — the Triforce fork's measured mirror demotion: per-pool guest-memory trials with GPU-time A/B and a persistent keep list. A pool suspected of colliding goes to guest memory for a trial window; if GPU time improved, the demotion is kept in a persistent list; if not, the mirror comes back. Upstream equivalent:
st_sync_*per window,direct_by_pool,live_by_pool—vk_memory.cpp:3420-3467);RX_opt-out like every switch here, off-or-neutral unless set;kIdleEpochs,vk_memory.cpp:2757) with measured placement.Related issues
RX_VK_WB_MAX_KB, the blind size skip (gpu/vk/vk_memory.cpp:2733-2745,readme.md:681-685). Phase 1 narrows its large-both-dirty exception; proposal 1 shrinks what remains compared.Validation
st_wbd_*,vk_memory.cpp:3427-3430), live pages read by pool (:3434-3452), the mirror-sync split (from-captures vs from-earlier,:3420-3424), imports (:2824), and Phase 1's additions — GPU-dirty-only pages skipped, capture scan ranges pruned, #1629's gate kept/skipped (wb-tiles:3623-3639). Targets: compared MB down with no flash, the from-earlier share down sharply, readback-wait averages down from ~17 ms to the urgent tail's length.RX_VK_LOG_READBACK_WAITS=1for per-wait lines and the 3000-invalidate averages (vk_replay.cpp:2277-2289).RX_switch that is off or neutral unless asked for (CONTRIBUTING.md, Code and data conventions), documented beside the surrounding switches. One run on, one run off, same spot, same settings, per card — theRX_VK_DIRTY_ORIGIN=0pattern the Phase-1 branch already documents in its readme section.RX_VK_FLASH=1,docs/development.md:114) — a culled tile that actually differed shows up as a flash, so a clean detector run is the correctness gate for proposals 1 and 3;RX_VK_LOG_LIVE=1(vk_memory.cpp:2339-2340) for the live-page decisions andRX_VK_LOG_READBACK(vk_replay.cpp:2226-2233) for wait values.tests/run_all.pyon its own. The analysis branches already carry the shape:tests/test_wb_diff_split.py,tests/test_wb_origin_split.py,tests/test_wb_diff_cpu_writes.py(Phase 1),tests/test_wb_tier_split.py,tests/test_wb_bulk_render.py,tests/test_wb_invalidate_tier.py(phase2-glm). Each proposal's pure parts (the tile-index and keep predicates, the tier classifiers, the trial decision) get the same treatment; nothing needs a GPU or a dump in CI.None of the above changes what the replay reads from guest memory, the per-queue wait order, or submission-order registration — the design notes' section 4 rules hold by construction in every piece.
I checked the public Hookshot-optimi fork at
15ec28db; its published branches aremainandbake-offline, and I couldn't find the dirty-origin or tier-split code described here. Could you publish links towb-tiles,phase2-glmandphase2-mimo, and say which parts you're still working on? I'd like to review the source and coordinate a non-overlapping piece before implementing anything. For the tile feedback, the main point to check is how an observation from submission N is invalidated by fresh GPU writes in N+1.wb-tiles follow-up: branches, status, and your GPU-write invalidation question
Branches (just published to blessedplummer/Hookshot-optimi; nothing merged anywhere):
wb-tiles@13b601a— Phase 1 dirty-origin split + review fixes + build fix. Unit suite green; full run shows only pre-existing env failures. Remaining: game smoke test (not yet run).phase2-glm@61fcb3c— urgent/bulk write-back bookkeeping, bulk-render capture exclusion, invalidate fallback; the bind tile prefilter is deleted as a proven no-op. Remaining: game measurement.phase2-mimo@22f5d61— sparse mirrors with uninitialized-fill fallback, GPU-aware shadow inherit, eviction bias; the bind-mark tile cull is deleted as a proven no-op. Remaining: game measurement.The builds + 4K A/B smokes assessment was interrupted and resumes later. Coordination: the one piece nobody has built is the GPU-fed tile observation kernel — consuming the wb_diff differ output (which chunks actually differed) and folding it into per-tile observations. All three branches deliberately stopped at the CPU side, so if you want to take that kernel (it needs the wb_diff feedback path), it's yours; we'll stay off it and off the branches' own remaining work.
Your invalidation question — how an observation from submission N isn't invalidated by fresh GPU writes in N+1. Short answer: it never has to survive as a live fact. N's observation is consumed within N's own submission (the write-back), and N+1 starts from re-stamped state. The mechanism, sites verified at gpu/vk/vk_memory.cpp @
13b601aunless noted:memory_end_submissionbuildswb_pending_pagesfrom this submission's write-back ranges wherewb_pages.get(pg) > mem_completed(lines 3707–3718; pending predicatewb_pages.pending, lines 1317–1321). The cull keeps them:wbd_keep_page(known, gpu_dirty, pending) = pending || !known || gpu_dirty(line 3562), applied withstd::binary_search(wb_pending_pages...)at lines 3750–3752.memory_end_submission(lines 3804/3840) and stamp each page's timeline value (wb_pages_note, line 3808). The CPU's readback (invalidate, gpu/vk/vk_replay.cpp lines 2270–2295) re-reads that value and waits whenv > completed()(wait_value(v)), so a CPU observation of the page happens only after N's write-back completed — no cross-submission stale-read window.for (Pool* p : mirror_synced) p->synced_gen = gen_now, p->gpu_synced = gpu_now;(line 3864), withgpu_now++at each submission's begin (line 3130). CapEntry GPU-origin marks are set during N's mirror sync (cap_entry_set(page, {gpu_page_gen(page), 0, true}), line 2460) and cleared by any later CPU copy (line 1831); capture-side readers skipe.gpuentries (lines 1198, 2471, 3274), so a mark never feeds a copy that could resurrect stale bytes.memory_stamp_gpu→stamp_gpu_range(lines 767–775; callers: vk_draw's storage buffers, vk_replay's copies/clears, vk_xfer's texture-to-buffer copies). The next cull readsgpu_page_gen(gpg) > p->gpu_synced(line 3751);gpu_syncedwas frozen at N'sgpu_now, so an N+1 stamp is strictly newer → gpu_dirty → kept.The rule, enforced at the two cull sites above: no page with a pending write-back value or a post-baseline stamp is ever culled. The remaining gap is exactly the kernel: turning the differ output into the tile observations N+1 reads — hence the offer.
Thanks, I'll take the GPU-fed observation path. Claimed in dev-work. I'll start with opt-in per-chunk actual-difference feedback tied to completed submissions, keeping the current copy decisions. That gives us a measured basis for a later cull and its freshness proof. Your three branches remain yours. I now have a repeated Windows/NVIDIA 1080p/4K baseline; readback waits stay around 0.01 ms here, so I won't treat the urgent-signal proposal as a demonstrated local win.
Another Windows/NVIDIA data point: v2.1.0, RTX 3090, 4K output, DLSS Quality (K), memory sync 13-15 ms,
RX_VK_WB_MAX_KB=1024changes nothingSame symptom as above (CPU-bound by memory sync, GPU not the limit), and this log shows that skipping large write-backs was not the dominant cost here.
Setup: Windows 11, RTX 3090 (24 GB), Windows driver version 32.0.16.2101, output 3840x2160, DLSS Quality (render 2560x1440), DLSS model K, frame cap 60, frame generation off, HDR off, Reflex on.
Scene: a busy area with many actors (about 850 draws, 140 dispatches, 140 passes a frame).
Results (same area each time, 600-frame windows; not a scripted run, so treat small differences as noise):
memory syncRX_VK_WB_MAX_KB=1024(log confirms: "write-backs only of written bindings up to 1024 KB")So the written-back volume dropped about 8x, but memory sync stayed at ~13-14 ms and fps did not move.
What the log shows for a frame in that scene:
The cost looks tied to the number of copies (about 1,790 small copies a frame for 13 MB), not the bytes, which matches the "mirror-sync page churn" leg above.
GPU-bound waitis 0, so lowering the DLSS render resolution is not expected to help in this scene.If it is useful for the baseline, I can run the same scene with the dirty-origin or GPU-fed observation builds when they have a branch to test. Happy to provide a fuller log.
The GPU-fed observation path is now #2131, with CI running. It is opt-in (
RX_VK_WB_OBSERVE=1) and reports actual changed chunks after their submission completes; no copy or culling decisions change. The existing analysis branches stay with blessedplummer.The six local 4K controls found differences in about 15% of chunks, covering about 13% of compared volume, with no unavailable results. The enabled mean was 1.7% below disabled, so this is measurement infrastructure rather than a speed-up. Details and limits are in the PR.
@builuanhoang-bot your copy-count-heavy scene would be useful external coverage once the checks finish. If you try the branch, the observation lines alongside the existing sync/copy counters would help compare the two workloads; a clean observation still is not a proof that a future write can be skipped.
@blessedplummer I'm investigating automatic compute storage write-footprints from IR, separately from #2131 observations and your dirty-origin/sparse-mirror branches. Scope posted in dev-work.
Authored IR-to-planner-to-GPU probes now cover dense ranges, strides, mixed plans and conservative fallbacks on Windows/NVIDIA. No renderer policy has changed, and there is no in-game speed-up claim. Experimental priority: S3; reach is not established; confidence 0.4 for an in-game performance opportunity.
Before integration, are store extents already part of any of your current branches? The proposed boundary keeps binding/hazard spans and your copy/culling policy unchanged; it supplies conservative per-dispatch write metadata with the existing fallback when a proof is unavailable. I'd like to agree on that boundary before touching overlapping renderer code.
There is a PR in the works taking a different approach. So far it is looking promising, so once it looks good, that will be made a PR on upstream (here).
@blessedplummer Thanks. I'll keep the write-footprint work isolated until we can check the overlap. Does the upcoming approach also derive per-dispatch store extents from shader IR, or does it work at the memory/copy-tracking layer? A branch or draft link when available would help me avoid duplicating it. No renderer or copy-policy changes have been made in this experiment, and there is no in-game speed-up claim.
Follow-up: this issue's remaining legs are implemented and measured in PR #2338 (
RX_VK_DGPU=1, default off) — the "render targets device-local, sync only what the CPU reads, mirrors only where a measured trial proves them" policy for dGPUs.Measured on RX 9070 XT / RADV, 4K, Uncapped cap, warp-normalized adjacent pairs (Hyrule Field, 3D gameplay, draw-count parity checked per pair):
One safety finding from developing this, directly relevant to the small-readback-ranges direction (#1150): page-granular write-back culling is NOT safe. With write-backs fully skipped, the same scene issued 3941 draws vs 1364 — the game reads GPU results outside the readback API, so page-sparse copies starve it and the apparent fps wins were scene mismatches. Gating at POOL level (skip a pool's sync only where the CPU never read that pool) restores draw parity within ~2% (705 vs 709 draws/frame in the final logs). Anything building on #1150 should use the draw-count-parity OFF-vs-ON check as a regression gate.
What the PR deliberately does not touch: the remaining "memory sync" ms is the submit-time capture of CPU-written pages (texture recreation correctness, ~30 MB/submit) — policy-independent, and page-sparse tricks on it are exactly the failure mode above. The next leg there is conflict-driven per-pool demotion (the Triforce fork's measured-trial approach, referenced in the original issue text) so pools that collide CPU/GPU-side leave the mirrors; that is drafted as a companion issue pending signal on the PR.
Branch:
blessedplummer/Hookshot-optimi@dgpu-pr(one commit off main @364d199; game-free checks green, suite 352/0 vs pristine 351/0; default-off verified — the added counters and logging are gated on the switch).@blessedplummer Thanks, I read #2338. I'll keep this separate from its pool policy, readback gating and trials. The remaining scope would be optional per-dispatch compute write-extent metadata, with unknown image/global effects falling back and no change to pending-writer, alias or lifetime rules. Authored tests now exercise eight disjoint ranges through IR, cache, fresh resolution and GPU consumption, including causal checks; there is still no game-performance claim. Would that metadata-only scope be useful for residual mirrored pools after #2338, or would you prefer it deferred?
Thanks for reading it — useful now, not deferred, and the scope you've drawn is exactly the one that composes.
Under #2338's policy the residual syncs split three ways: pools bound direct (
MIRROR_REFUSED) run no write-back at all — nothing for the metadata to feed there; pools that earn a mirror through the trial keep the stock full-pool machinery; and CPU-read pools keep stock write-backs by the pool-level gate (any readback page -> stock behavior for that pool). Those last two are precisely the "residual mirrored pools" your per-dispatch write-extents would sharpen — and they are the only place it can apply, so it composes instead of overlapping:No rush on the performance claim — once it's testable I can measure it on the residual mirrored pools directly (RX 9070 XT / RADV, 4K, warp-normalized adjacent pairs, the 600-submit counters — the same harness #2338's numbers came from), so the game-performance side your tests deliberately don't claim can come from the same rig the pool policy was measured on.
@blessedplummer Thanks, that separate metadata-only scope works. One clarification: the planned producer derives conservative possible-write extents from shader IR, resolved against replay-ordered dispatch constants/bindings; it is not a GPU-written dirty-page signal. Unknown image/global effects, incomplete bounds or unresolved aliases must retain full sync, and every pending writer still contributes. I'll keep the pool policy and existing lifetime/alias machinery unchanged, with draw-count parity and matched-scene checks before any performance claim. Please flag if you meant GPU-written extents rather than IR-derived metadata.
Thanks for the report. Reports of the same problem are now collected in #995, which lists this one with your details and links back here, so one fix closes them together and the open list stays readable. Follow #995 for the fix, and add new details there (or here). If this turns out to be a separate problem it will be reopened.