WIP: perf(vk): the submission walks only the retired pools still holding an import (RX_VK_RETIRED_HELD=0 to disable) #2029
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2029
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/vk-retired-held"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
At every part of a submission the give-back of retired imports (
retired_imports_give_back) walked the retiredpools, a deque of map nodes that only grows, and every free of the game's asked
memory_guest_imported, which walkedevery retired pool too. Main already passes over the oldest retired pools given back; the walk of the rest stayed, and
memory_guest_importedstill walks them all.With this,
retire_poollists a retired pool that still has an import, in the order of retirement(
retired_held, linked through the pools themselves:Pool::held_nextand a tail pointer, so listing one allocatesnothing), reaching the pool through its node once the table's node is extracted;
retired_imports_give_backunlinks itas it gives the import back (the one place a retired pool loses its import), writing nothing for the pools it keeps. The
give-back and
memory_guest_importedwalk that list instead. The same imports are given back, with the same budgetfigures and answers. Either way, a line after the
vk: imports:line counts the retired pools each walk looked at(parts and calls), and Tracy builds time both walks in zones of their own. The result expected: less CPU time a
submission when many pools have been retired; not measured yet.
The switch:
RX_VK_RETIRED_HELD(new, a kill switch, bool, its row ingpu/vk/switches.def), on by default asdocs/dev/standards.md (Switches) asks of a switch's first push; read once;
=0keeps main's walk, in the give-back andthe free's check both.
From 12168bb8 on
exp/x86-asm-sync(the half of that commit that rebuilt the GPU-visible ranges in asm is not here).Read first:
retire_pool,retired_imports_give_backandmemory_guest_importedingpu/vk/vk_memory.cpp, thentests/test_retired_held.py.Testing
Tested: Windows 11, i7-9700, RTX 4060, the builder's llvm-mingw clang 23; head
e26cc20con main9dd41326. The checks ran at91960bc2on mainb62932a9, the same commits before a rebase onto9dd41326, which added only lift files (#1905) that none of these checks reads. Light queue jobsfb-2029-3andfb-2029-4of this PC's job queue (the first found compile errors in the new fixtures, fixed before the second), each check in a hidden console throughtests/run_all.py --strict --exact --timeout 1200 --jobs 2:tests/test_retired_held.py,tests/test_import_reach.pyandtests/test_switch_table.pypass;tests/test_guest_imported.pyandtests/test_pending_guest_import.pyfail, with the same compile error (import_reachundeclared in their fixtures) on main @9dd41326in the same job. At the test commitcc8c828dtests/test_retired_held.pyfails, as expected (retire_poolallocates more with the vector list). The Windows game build's own compile ofgpu/vk/vk_memory.cpp(syntax only) passes. Not run this round: every check--changedpicks (a heavy batch), the Linux-target compile (the local tool could not read its build profile),tests/test_vbytes.py(hangs on this PC).It has NOT run in the game on this PC yet.
Still to test:
more, pools retire as scenes load) and a heavy view (Hyrule Field at dusk, the camera turned), on main @
9dd41326, this branch withRX_VK_RETIRED_HELD=0and this branch unset (on), each warmed a minute, interleaved (ABC ABC ABC), expectingthe same picture, no FATAL or device lost, the
vk: imports:lines alike, fewer pools looked at in thevk: retired walksline unset than with=0, and average fps, 1% lows, median and p99 frame times and peak RAM and VRAM of this branch unset at least those of the other two; a difference beyond the runs' spread is a gain or a loss, one within it is nodifference measured (by hand)
long session, main, this branch with
RX_VK_RETIRED_HELD=0and unset, warmed and interleaved, expecting the samepicture and no FATAL or device lost (the test farm: linux+nvidia, linux+amd, windows+amd, windows+intel, steamdeck:
Kokiri Forest to Hyrule Field and back five minutes, main @
9dd41326, this branch with RX_VK_RETIRED_HELD=0 and unset, warmed up, interleaved, the log, frame times and a screenshot)python3 -I -B tests/test_retired_held.py, expecting PASS (by hand)mise run testandmise run lintat this head, expecting every failure tofail on main too and lint to pass (by hand; neither tool is installed on this PC)
Measurements
Verdict: not measured in play yet: no micro-benchmark of this port and no run in the game on this PC yet.
RX_VK_RETIRED_HELDships on, as docs/dev/standards.md (Switches) asks of a switch's first push (
=0is main's way); the in-game A/B(unset against
=0and main) in Still to test decides whether it stays. The figures in the source commit 12168bb8 were taken on the experimentbranch before main's own pass over the oldest given-back pools; they say nothing about this head.
Risks
What could break (on, the default): a retired pool missing from the list would keep its import (memory held longer, its
pages read as imported); one listed after it gave its import back would be given back twice. The test runs the real
retire_pool,memory_guest_importedandretired_imports_give_backwith the switch off and on over the same 60,000random steps in the three give-back modes (the queued imports freed as the GPU passes them) and compares every import
given back, figure, log line and answer, the list against the retired pools holding one (by address, in order), and
that
retire_poolallocates no more with the list; directed cases check the wait mode's early give-back under heappressure and a pool bound again after it was retired. With
=0, what changes from main: the extracted poolis reached through its node, two counts are kept under
pools_muand logged with the imports line, andPoolis 8bytes larger. Turn it off with
RX_VK_RETIRED_HELD=0. Renderer code: the runtime is rebuilt; nothing changesfor players' saves, options or caches.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Possibly related, for context:
Recommendation: Request changes. The default path dereferences a pointer taken before its map node was extracted, while the node handle still owns the pool. Fix that lifetime and cover pressure-driven give-back in the fixture before using in-game results to choose the default.
The index can save CPU time during submissions and guest frees as retired records accumulate. It preserves retirement order and shares the give-back logic between both paths, at the cost of a second collection, heap growth during retirement, and pointer writes during submission.
@fixwill do (6)gpu/vk/vk_memory.cpp:4579: old comes from the live map before imported_take extracts its node. (more on the line)tests/test_retired_held.py:187: The fixture advances GPU completion without reaping dead_imports. (more on the line)tests/test_retired_held.py:140: Count, timestamp order and non-null handles do not establish exact membership. (more on the line)tests/test_retired_held.py:294: Both fixture namespaces use the same refactored give-back body, so agreement misses regressions in the shared conditions. (more on the line)gpu/vk/vk_memory.cpp:4579: Vector growth adds a fallible allocation under pools_mu after retirement has already changed the page table and moved the node. (more on the line)gpu/vk/vk_memory.cpp:5587: Add temporary timing and visit counts for both walks and record the vector's peak capacity. Skip pointer stores when a survivor is already in its destination slot.For the author (1) (a run, a measurement or a decision)
gpu/vk/vk_memory.cpp:5587: The planned median-only comparison misses tail latency and the cost when most retired imports are still held. (more on the line)Checks: Still running:
PR description / Description follows the template (pull_request_target),CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven. The commit reports 77.6 to 1.9 µs cold and 12.1 to 0.06 µs warm for 4,000 retired pools on branch commit 12168bb8. It also reports 6.7–19.6 µs down to about 1.1 µs per submission in Kokiri. The branch comparison predates optimizations already in the base. There are no timing or gameplay results for
3f6c6a2aagainst75317e1f.Other systems: Both walks hold pools_mu. Their timing can affect the replay thread and guest threads freeing pages. The give-back conditions are shared across immediate mode on Linux and Windows NVIDIA, grace mode on Windows AMD and Intel, and optional keep mode. Linux dma-buf and Windows host-pointer imports both rely on the guest-free check to keep backing pages intact. Mirror inheritance still uses the owning deque, and pending, evicted and background imports retain their existing checks. Static review found no changes to saves, audio, input, the renderer ABI or cache formats. Driver behavior, frame pacing and RAM/VRAM impact still need in-game validation, including on Steam Deck.
Switches: RX_VK_RETIRED_HELD is a developer kill switch, read once and on when unset. Set it to 0 to use the previous traversal in both paths. The switch table has a sorted row. On is the branch default for A/B testing. The shipping default remains unproven across affected hardware.
Reviewed:
3f6c6a2a4dagainst75317e1f31: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template and docs/dev/standards.md. No docs/review-checklist.md or sibling Tooling checkout was present. Read the complete six-file diff and commit, all four changed test files in full, and the changed regions of vk_memory.cpp and switches.def with their surrounding code. Traced retirement callers, pool ownership and locking, completion and accounting, host frees, switch parsing and test compiler selection. Skimmed the earlier #391 results. The supplied PR description was the only source for this head’s recorded runs. There were no tester comments or prior reviews.; nothing run (a code review).Checked and fine:
Testing: Read-only review. git diff --check passed. The author reports 82 of 96 Windows checks passed, 13 failures also seen on the base, and one timeout that passed on an isolated retry. The new fixture reportedly passed. The two older guest-import fixtures named in the base failures provide no passing evidence. No Linux or in-game result was supplied, and the listed CI checks were still running.
Overlaps and merge order: Merged #391 supplies the retired-prefix skip already in the base. Current main changes a separate switch-table row in #1870. No overlapping change requiring a merge order was found in the checkout.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -4565,6 +4576,7 @@ void retire_pool(std::map<uint64_t, Pool>::iterator it) {if (old->mstate == Pool::MIRROR_ON) old->inherit_pending = true, old->retired_epoch = mirror_epoch;pools_gen++;retired_pools.push_back(imported_take(it));if (old->memory && retired_held_on()) retired_held.push_back(old);L1: old comes from the live map before imported_take extracts its node. C++ permits that pointer to survive extraction, but accessing the element through it while a node handle owns it is undefined. The new default path reads old->memory at retirement and saves the pointer for subsequent frees and submissions. Reacquire the pointer from retired_pools.back().mapped() after extraction and use that pointer for the list and the existing post-extraction log.
L5 (minor): Vector growth adds a fallible allocation under pools_mu after retirement has already changed the page table and moved the node. At the host commit limit, an optional index growth can make the renderer wait up to five seconds and then stop even when the owning deque had room. Make index allocation failure fall back to the deque scan under the same lock, or use bookkeeping that needs no separate allocation.
@ -5571,1 +5584,4 @@retired_alive++, retired_bytes += p.hi - p.lo;return true;};if (retired_held_on()) { // (only the retired pools still holding an import, retired_held: those given back leave it)L6 (minor): The planned median-only comparison misses tail latency and the cost when most retired imports are still held. Run current main and the PR with the switch on and off in matched, warmed, interleaved in-game runs, including a long session, a heavy view and the affected Tier 1 hardware. Report both scan times, average FPS, 1% lows, median and p99 frame times, and peak RAM/VRAM. Record the author's own game run, drivers and results in Testing and Measurements.
@ -0,0 +137,4 @@}// (a pool is listed only with an import, and the give-back that takes one drops it in the same pass: so the list is// exactly the retired pools with one, in the order they were retired)CHECK(ref::retired_held.empty() && now::retired_held.size() == held);L3 (minor): Count, timestamp order and non-null handles do not establish exact membership. If two pools retire at the same timestamp, a duplicate of one can stand in for the other and pass these checks. Compare each index entry with the address of the next memory-owning node in retirement order. That invariant also lets the test drop the exact source-line checks for append and compaction.
@ -0,0 +184,4 @@const uint64_t rec = below(3), lag = below(4), epochs = below(300) + 1, ms = below(4000);const uint64_t used = (g_budget / 100) * (60 + below(40));g_now += (int64_t)ms, g_used = used, g_heap_ok = below(8) != 0;BOTH(mem_rec_value += rec, mem_completed = mem_rec_value - std::min(mem_rec_value, lag), mirror_epoch += epochs,L2: The fixture advances GPU completion without reaping dead_imports. Once about 1.12 GiB is queued in AT_ONCE, subtracting those bytes suppresses every later GRACE pressure probe at the fixture's 8 GiB budget and 99% maximum use. Reap completed entries and add a directed case that requires an early give-back under pressure, stops at the low watermark, and checks that the guest range is still reported imported until completion.
@ -0,0 +291,4 @@state = STATE.replace("@REACH@", reach)fixture = (FIXTURE.replace("@STATE_REF@", state.replace("@HELD@", "false")).replace("@STATE_NOW@", state.replace("@HELD@", "true")).replace("@CODE@", "\n".join([retire, imported, give])))L4 (minor): Both fixture namespaces use the same refactored give-back body, so agreement misses regressions in the shared conditions. For example, dropping the last_value gate would queue an import too early on both sides and still compare equal. Add a directed expectation that an import with a completed retirement value and an outstanding latest binding stays held until the binding completes.
3f6c6a2a4de26cc20c83Thank you for the review (#2029 (comment)). The branch is rebased on main @
9dd41326(the PR's own commit unchanged by the rebase, now2efd763d), with a test commitcc8c828dand a fix commite26cc20c; heade26cc20c. Point by point:retire_poolnow reaches the pool throughretired_pools.back().mapped()onceimported_takehas extracted its node, for the list and for the log line after it (e26cc20c). No test can tell the two pointers apart (same address); the fixture below runs the new code.reap_mirrorsdoes once the GPU is past them, at every part, so the wait mode's probe sees the settled use, and a directed case (fresh process, both sides) has 4 retired pools of 512 MB with the heap at 90 % of 8 GB: nothing goes before the 16th part's probe, then 3 go, oldest first, stopping once the 75 % watermark's worth is queued, the 4th is kept through the next probe, and the 3 stay reported imported bymemory_guest_importeduntil the GPU passes them (cc8c828d).cc8c828d).cc8c828d).Pool::held_nextand a tail pointer): listing a pool allocates nothing, and the give-back unlinks the pools it gives back without writing the ones it keeps (e26cc20c). The fixture checks thatretire_poolallocates no more with the list than without it (cc8c828d: fails on the vector, passes now).vk: imports:line and reset, either way the switch is set; Tracy builds time both walks in zones of their own (e26cc20c). There is no vector any more, so no peak capacity and no stores for pools kept. The in-game A/B you describe (main,=0, unset; a long session and a heavy view; Tier 1; average fps, 1% lows, median and p99 frame times, peak RAM and VRAM, and the walks line) is pending: it has not run in the game on this PC, and it is in Still to test.RX_VK_RETIRED_HELDstays on by default (a kill switch,=0is main's way), as docs/dev/standards.md (Switches) asks of a switch's first push; it is not measured in play yet, and the in-game A/B (unset against=0and main) in Still to test decides whether it stays.Tested here (Windows 11, i7-9700, RTX 4060, llvm-mingw clang 23; queue jobs fb-2029-3, fb-2029-4, light):
tests/test_retired_held.py,tests/test_import_reach.pyandtests/test_switch_table.pypass;tests/test_guest_imported.pyandtests/test_pending_guest_import.pyfail, with the same compile error (import_reachundeclared in their fixtures) on main @9dd41326in the same job. At the test commitcc8c828dtests/test_retired_held.pyfails, as expected (retire_poolallocates more with the vector list). The Windows game build's own compile ofgpu/vk/vk_memory.cpp(syntax only) passes. (Run at91960bc2onb62932a9; the push is the same commits rebased onto9dd41326, which added only lift files.) Not run this round: every check--changedpicks (a heavy batch), the Linux-target compile (the local tool could not read its build profile),tests/test_vbytes.py(hangs on this PC). The change has NOT run in the game on this PC yet. The description is updated.@Beanstalk This makes the renderer give back retired imports with walks of the retired pools at each submission part and each free (gpu/vk/vk_memory.cpp), on by default; RX_VK_RETIRED_HELD=0 is main's way. This head is on main
9dd41326: please compare with main at that merge base. Please play Kokiri Forest to Hyrule Field and back for five minutes, then one longer session of fifteen minutes, the PR against main on the same PC, on the PR's side with the switch unset and with RX_VK_RETIRED_HELD=0, with a picture in each scene compared with main's, on Linux with NVIDIA, Linux with AMD (RADV), Windows with Intel and the Steam Deck. The same picture, no crash, device lost or FATAL line; the log's "vk: imports:" lines alike on every side; unset, the "vk: retired walks" line shows fewer pools looked at than with it off; frame time p99 and peak VRAM no worse than main's. Please report those two lines.@siantm only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.