WIP: perf(vk): the submit's collection leaves the capture's ranges to the captures (RX_VK_COLLECT_ONCE=0 to disable) #2027
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2027
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/vk-collect-once"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
A submit's capture reads and clears the write marks of its ranges (the mirrored pools, the texture storage, the
pools up to 64 MB), and the collection of written pages that follows read every GPU-visible range again, those
included, only to put back the marks it found there for the next capture.
With this,
memory_collect_writtenasks the host for the marks outside those ranges (collect_written_inwith a countbelow 0,
host_collect_outsideinhost/rx_backend.c; the smallest int asks whether the host knows that form), and themarks inside stay as they are, which is what putting them back came to. The pools with a write-back shadow are still
scanned (
collect_skip_ranges, rebuilt when a shadow is made or freed), so the shadows are told of the same pages asbefore. The full scan stays before the first capture, with
RX_VK_CAPTURE=0and with a host without the outside form.A log line every 600 collections says what was found and how much was left to the captures. The result expected: one
scan of the capture's ranges a submit instead of two.
The switch:
RX_VK_COLLECT_ONCE(new, a kill switch, bool, its row ingpu/vk/switches.def), on by default as thebranch shipped it, read once;
=0gives back main's code: every range scanned and its marks put back.From d1e33769 on
exp/x86-asm-sync(there it was off with the render thread; here it is not: the ranges left out arethe ones the full scan's put-back reads, under the same
collect_mu, whichever thread collects).Read first:
memory_collect_writtenandcollect_skip_rangesingpu/vk/vk_memory.cpp,host_collect_outsideinhost/rx_backend.c, thentests/test_collect_outside.py.Testing
Tested: Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head
48e5a97eon main75317e1f. Build jobs of this PC's job queue (batchCapCut3-b3), the checks in a hidden console:tests/run_all.py --strict --exactontests/test_collect_outside.pyand every check--changedpicks for the files this changes: 231 checks, 198 passed, 33 failed, 0 skipped; its own check passes, the switch's read included (fresh processes); the failing ones fail on their base too, run from a base worktree in the same batch:tests/test_capture_read_once.py(fails on main @75317e1ftoo),tests/test_capture_readback_visibility.py(fails on main @75317e1ftoo),tests/test_crash_dialog.py(fails on main @75317e1ftoo),tests/test_crash_record.py(fails on main @75317e1ftoo),tests/test_dev_ab.py(fails on main @75317e1ftoo),tests/test_dev_build.py(fails on main @75317e1ftoo),tests/test_dev_packs.py(fails on main @75317e1ftoo),tests/test_device_lost_exit.py(fails on main @75317e1ftoo),tests/test_draw_stream_visibility.py(fails on main @75317e1ftoo),tests/test_flush_nothing_held.py(fails on main @75317e1ftoo),tests/test_frame_overlap.py(fails on main @75317e1ftoo),tests/test_fsr_image_retirement.py(fails on main @75317e1ftoo),tests/test_fsr_uniform_ownership.py(fails on main @75317e1ftoo),tests/test_gpu_wait_stats.py(fails on main @75317e1ftoo),tests/test_guest_imported.py(fails on main @75317e1ftoo),tests/test_host_guest_writes.py(fails on main @75317e1ftoo),tests/test_linux_pressure.py(fails on main @75317e1ftoo),tests/test_mesh_conv.py(fails on main @75317e1ftoo),tests/test_pending_guest_import.py(fails on main @75317e1ftoo),tests/test_perf_report.py(fails on main @75317e1ftoo),tests/test_present_queue_default.py(fails on main @75317e1ftoo),tests/test_present_source_crop.py(fails on main @75317e1ftoo),tests/test_profiling_mode.py(fails on main @75317e1ftoo),tests/test_recorder_own_idle.py(fails on main @75317e1ftoo),tests/test_renderer_new_handler.py(fails on main @75317e1ftoo),tests/test_replay_write_order.py(fails on main @75317e1ftoo),tests/test_slow_frames.py(fails on main @75317e1ftoo),tests/test_sparse_mirror.py(fails on main @75317e1ftoo),tests/test_target_reuse_memory.py(fails on main @75317e1ftoo),tests/test_texel_buffer_ownership.py(fails on main @75317e1ftoo),tests/test_ub_update_batch.py(fails on main @75317e1ftoo),tests/test_upscaler_rows.py(fails on main @75317e1ftoo),tests/test_vk_occlusion_queries.py(fails on main @75317e1ftoo).It has NOT run in the game on this PC yet.
Still to test:
75317e1f, this branch withRX_VK_COLLECT_ONCE=0and this branch with it unset (on), each warmed up a minute in the scene first, interleaved (ABBA), at least two runs each, expecting the same picture (no stale or flickering texture, no missing draw), no FATAL, and a median frame time unset lower than withRX_VK_COLLECT_ONCE=0and than main, andRX_VK_COLLECT_ONCE=0the same as main (by hand)75317e1f, this branch withRX_VK_COLLECT_ONCE=0and this branch with it unset (on), warmed up, ABBA, expecting the same picture, no stale texture and no FATAL, and a median frame time unset lower than withRX_VK_COLLECT_ONCE=0and than main, andRX_VK_COLLECT_ONCE=0the same as main (@Beanstalk linux+nvidia, linux+amd, windows+amd, windows+intel, steamdeck: Kokiri Forest three minutes: main @75317e1f, this branch with RX_VK_COLLECT_ONCE=0 and this branch with it unset (on), warmed up, ABBA, the log, frame times and a screenshot)python3 -I -B tests/test_collect_outside.py, expecting PASS (by hand)Measurements
Verdict: not measured: no micro-benchmark of this port and no run in the game on this PC yet; the in-game A/B is
in Still to test.
RX_VK_COLLECT_ONCEships on, as the branch shipped it; no measurement chose that default yet, the in-game A/B in Still to test does.Risks
What could break: a page written by the CPU and left out of both the capture and the collection would reach the GPU a
submit late: a stale or flickering texture, a wrong value read back. The test cuts
host_collect_inandhost_collect_outsidefromhost/rx_backend.cand checks against a plain walk that only the whole pages of the givenranges are left out; the write-back shadows' checks run
collect_skip_rangesand check the shadowed pools stayscanned. Turn it off with
RX_VK_COLLECT_ONCE=0. Host and renderer code: the runtime is rebuilt; nothing changes forplayers' saves, options or caches.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Possibly related, for context:
Recommendation: Request changes. The shortened scan can miss a dirty page when a pool leaves the capture list during collection. It also makes the early-readback diagnostic count partial scans as full coverage, and the performance gain on this head is unproven.
The change aims to reduce CPU work per Vulkan submit by leaving capture-owned write marks for the next capture while still scanning pools with write-back shadows. It adds a host callback protocol, cached range subtraction, a pool lock acquisition, and scratch storage. Players have no demonstrated frame-time gain yet.
@fixwill do (13)gpu/vk/vk_memory.cpp:938: The skip list is built before the host scan, while the put-back decision uses a fresh range snapshot afterward. (more on the line)gpu/vk/vk_memory.cpp:969: RX_VK_LOG_EARLY_CHECK still advances its full-scan generation for an outside scan. Its classifier closes an interval after the second full collection begun after a snapshot. With two submits already queued before a CPU store to an excluded unshadowed page, replay can finish both partial scans and close the interval before any later capture reports the store. The diagnostic can then attribute the changed word to the GPU. Keep the full scan while that diagnostic is enabled, or make interval closure depend on actual coverage.host/rx_backend.c:325: The low-memory fallback calls host_collect, which allocates another 64 KiB range buffer and passes it to hos_gpu_guest_ranges unchecked. (more on the line)host/rx_backend.c:313: This adds a new contract to MkwVkHost while vk_api.h still documents only an inside scan. (more on the line)host/rx_backend.c:327: The advertised negative-count protocol accepts counts above the renderer's local guard. (more on the line)gpu/vk/vk_memory.cpp:946: The C callback indexes an array of uint64_t[2], but the caller passes std::pair objects through a cast. (more on the line)gpu/vk/vk_memory.cpp:985: The 600-submit log combines averaged page counts with the mode, range count and bytes from only the last submit. (more on the line)tests/test_collect_outside.py:223: The new test ignores CC and uses one driver for both languages. (more on the line)tests/test_wb_diff_stale_shadow.py:23: The newly added shadow-exclusion case sits behind a literal g++ runner argument. A Windows llvm-mingw setup with clang++ and no g++ cannot run that coverage even with CXX configured. Use the runner's {cxx} tool and honor CXX for direct invocation. The direct defaults in the other changed C++ fixtures should use the same selection.tests/test_vk_collect_window.py:203: The new enabled run checks final marks and stamps, which also pass if the renderer silently uses the full scan and puts page 9 back. (more on the line)gpu/vk/switches.def:299: The switch row omits the full-scan fallback before capture coverage exists and when RX_VK_CAPTURE=0. (more on the line)e6196f0f, eight commits beyond75317e1f. Untick the box until the branch and its measurements are refreshed against current main.For the author (1) (a run, a measurement or a decision)
Checks: Still running:
CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),PR description / Description follows the template (pull_request_target),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven. The commit cites a profile from experimental commit d1e33769 on an i7-9700 in Kokiri Forest: collection used 5.1% of one core, the capture covered about 2.9 GB across 1,465 ranges, and the estimated upper bound was 0.11 ms saved per submit. There are no game timings for
48e5a97eagainst75317e1f, no same-head switch A/B, and no frame-time or 1% low results.Other systems: The Vulkan collection path is shared by Windows and Linux across NVIDIA, AMD, Intel and Steam Deck, including mesh fallback. Threaded replay can encounter the mapping change described below. Shadowed pools stay in the scan for a stable range snapshot, and capture-disabled runs retain the full scan. Early-readback diagnostics depend on full coverage. The shader stamp already excludes vk_api.h. The diff contains no save, audio, input, shader, or build-script changes.
Switches: RX_VK_COLLECT_ONCE is a developer kill switch. It is read once and defaults on with a capable host and eligible capture ranges. Empty, 0, off, false and no disable it. The row is present and sorted. Merge defaults still need measurements on supported hardware, with the correctness fixes in place.
Reviewed:
48e5a97e97against75317e1f31: Read AGENTS.md, CONTRIBUTING.md, docs/dev/standards.md, the Forgejo PR template, the commit and the full eight-file diff. Read all five changed test files and the changed production sections with their surrounding code. Traced collection and capture callers, pool mapping and shadow lifetimes, early-readback accounting, host API declarations, allocation helpers and shader stamp inputs. Skimmed the prior vector-walk run and related existing tests. No local review-checklist file, PR comments, or prior reviews were present. Unrelated portions of the large production files, other recorded runs, translations and generated files were outside the review.; nothing run (a code review).Checked and fine:
Testing: Read-only review. The PR reports 198 passes and 33 failures also seen on base out of 231 selected Windows checks, including a pass for the new fixture. Linux builds and game runs are pending. The supplied CI snapshot has no completed head-only failures and still lists checks in progress.
Overlaps and merge order: No overlapping open PR was identified from the supplied review data and local refs. The local main has a separate switch-table edit from merged #1977 with no overlapping hunk. Forgejo reports mergeable false and supplies no conflict details.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -296,6 +296,7 @@ SW(RX_VK_CASCADE_BIAS, tune, string, "0,0: nothing added", "Adds a depth bias <uSW(RX_VK_CHECKPOINTS, kill, enum, "on: NVIDIA diagnostic checkpoints before each draw where the driver has the extension", "0 turns off draw checkpoints; ext enables the extension only; 1 also uses AMD buffer markers where NV's are missing.")SW(RX_VK_CHECK_BANKS, diag, bool, "off: no check", "Compares each constant-bank fill against the per-word path and logs up to 20 mismatches.")SW(RX_VK_CLEAR_CPU_MAX, tune, int, "256 bytes", "Largest buffer clear also applied on the CPU in stream order; 0 leaves every clear to the GPU.")SW(RX_VK_COLLECT_ONCE, kill, bool, "on (unset): the submit's collection leaves the capture's ranges to the captures and scans the rest and the pools with a write-back shadow (not with a host without the collection outside ranges)", "0, off, false or no: the submit's collection scans every GPU-visible range, the capture's ranges included, as before. Read once, at the first collection (the log says which).")L11 (minor): The switch row omits the full-scan fallback before capture coverage exists and when RX_VK_CAPTURE=0. It also omits the empty value from the off values. Document both so an operator can tell when an unset switch will actually shorten a scan.
@ -861,0 +935,4 @@once = collect_once_on();if (once) { // (only what the captures do not own: see above)uint64_t owned_ver = 0;const auto& ranges = capture_ranges_merged(&owned_ver);L1: The skip list is built before the host scan, while the put-back decision uses a fresh range snapshot afterward. The game can replace a small capture-eligible pool with a larger GPU-visible pool at the same address between those snapshots. A dirty page in that span is skipped, and the next capture also excludes it. Main's full scan can stamp that page for the current submit, so a texture or buffer can stay stale until a later collection. Revalidate capture ownership before completing collection and scan any ranges that lost ownership, or synchronize the host scan and classification against a stable mapping snapshot. Cover removal during the scan in the window fixture.
@ -861,0 +943,4 @@if (once) {left_ranges = skip.size();for (const auto& r : skip) left_bytes += r.second - r.first;n = host->collect_written_in(reinterpret_cast<const uint64_t(*)[2]>(skip.data()), -(int)skip.size() - 1, &pages);L6 (minor): The C callback indexes an array of uint64_t[2], but the caller passes std::pair objects through a cast. C++ gives no array layout or aliasing guarantee for those pairs, so a build with a different pair layout could feed the host incorrect endpoints. Store the exclusion list in the actual wire type and share that type through the API.
@ -887,0 +982,4 @@calls++, found += n > 0 ? n : 0, back += put_back;if (calls % 600 == 0) {logf("vk: writes collected (600 submits): %.1f pages a submit, %.1f of them in the captures' ranges (marks put back for ""the next capture), %s", found / 600.0, back / 600.0, once ? "the captures' ranges not scanned again:" : "every range scanned");L7 (minor): The 600-submit log combines averaged page counts with the mode, range count and bytes from only the last submit. It also claims ranges were skipped when the host fell back to a full scan after realloc failed. Report actual completed full and outside scans for the window, with actual excluded bytes or an explicit fallback count. The current byte tally also walks every exclusion on every submit and can be computed when the list is rebuilt.
@ -307,0 +310,4 @@rest; the renderer names the ones it leaves to its captures, whose marks stay as they are. skip is sorted and itsranges are apart, as the renderer's list is; only the whole pages inside one are left out, and a list that is not inorder leaves less out, never more: a page is left out only where it lies in one of the ranges given.It is reached through collect_written_in's count, with no callback of its own (MkwVkHost is in gpu/vk/vk_api.h, whichL4: This adds a new contract to MkwVkHost while vk_api.h still documents only an inside scan. The probe assumes any accepted old callback safely handles an out-of-contract negative count and null ranges. The stated reason for avoiding a versioned callback is stale because nx2/translator_files.py explicitly excludes vk_api.h from the shader stamp. Declare the outside operation in the shared API, bump MKWVK_API_VERSION per the architecture rules, and gate the callback on host version and availability.
@ -307,0 +322,4 @@static uint64_t (*ranges)[2], (*parts)[2];static int parts_cap;if (!ranges) ranges = malloc(sizeof *ranges * 4096);if (!ranges) return host_collect(pages); /* (no memory for the list: the whole collection, as before) */L3: The low-memory fallback calls host_collect, which allocates another 64 KiB range buffer and passes it to hos_gpu_guest_ranges unchecked. If the first collection starts with captures enabled and both allocations fail at the commit limit, the host copies the mapping list through a null pointer. Use one shared mapping buffer for both paths or the checked host allocator. Exercise both mapping-buffer failure and parts-growth failure with a fixture whose full-scan fallback actually collects the marks.
@ -307,0 +324,4 @@if (!ranges) ranges = malloc(sizeof *ranges * 4096);if (!ranges) return host_collect(pages); /* (no memory for the list: the whole collection, as before) */const int n = hos_gpu_guest_ranges(ranges, 4096);if (parts_cap < n + nskip) { /* (a range's rest, and at most one part before each range left out) */L5 (minor): The advertised negative-count protocol accepts counts above the renderer's local guard. For example, INT_MIN + 1 decodes to INT_MAX - 1, so adding even two visible ranges overflows int before the allocation's size_t casts. Validate counts at the callback boundary and check the capacity, slack, and multiplication in size_t before allocating, or put a shared enforced limit in the new API.
@ -0,0 +220,4 @@p = argparse.ArgumentParser(description=__doc__)p.add_argument("--cc", default=None)a = p.parse_args()a.cc = a.cc or default_cxx()L8: The new test ignores CC and uses one driver for both languages. The documented --cc gcc invocation uses gcc to link the C++ switch fixture, which needs the C++ runtime for its function-local static initialization. Give the C and C++ fixtures separate compiler selection, honor CC and CXX, and pass the corresponding tools from the runner.
@ -178,6 +201,8 @@ void run(Enter e, const char* when) {int main() {run(kAfterTake, "just after the collection took the marks");run(kAtRemark, "as the collection marked the captured page again");L10 (minor): The new enabled run checks final marks and stamps, which also pass if the renderer silently uses the full scan and puts page 9 back. The switch fixture always supplies a capable host and checks compatibility selection through source text. Count calls to each callback and assert that the enabled path leaves the captured mark untouched throughout. Run the real selection with an unsupported or absent callback and with empty capture ranges to verify the fallback.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.