WIP: perf(vk): free retired pool records that hold nothing (RX_VK_REAP_RETIRED), so they stop piling up over a session #1037
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1037
Loading…
Reference in a new issue
No description provided.
Delete branch "TrashPanda/Longshot:perf/stage1-retired-reap"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: low · Tested on: Linux, RTX 4070 Ti · Needs: Windows AMD or Intel, and the Steam Deck, before the switch's default flips (a later PR)
Depends on #1034 (Stage 0: its memory line is what this is measured with). This branch is stacked on it. Review the last two commits,
perf(vk): reap retired pool records...anddocs(tested): ....Why
The renderer keeps a record for every game pool it had to remake larger, for the rest of the session, and walks all of them at every submission (five loops in
vk_memory.cpp). Their imports and mirrors are already given back, so the records hold nothing, but they pile up as you play: Stage 0's memory tour counted 159 after 25 minutes, about 10 more per area change. Over a long session that's thousands of empty records walked at every submit. The plan's Stage 1 asks for the renderer's memory and its per-submit walks to stay flat after warm-up. Of everything Stage 0's tour counted, this is the only thing that kept growing.What changed
A retired pool's record is freed once nothing is left in it and nothing still points at it. That was already the case for pools evicted to save memory (
evicted_pools), but not for pools remade larger.gpu/vk/retired_give_back.h:record_done(Leftover, completed). A record can go once its import, mirror and shadow are gone (each queued for freeing by value indead_imports/dead_mirrors), no pool waits to inherit its mirror (inherit_pending) or for a trim to settle (trimmed), and the GPU is past both the submission that retired it and the last one that bound it.gpu/vk/vk_memory.cpp:retired_pools_reap()inmemory_begin_submission, after the per-submission lists that holdPool*(mirror_synced,writebacks) are cleared. Other pointers to a retired pool are already dropped when it's retired (mirror_wanted,sparse_wanted,pool_cache,last_pool) or when its mirror is freed (dead_blocks), and the parallel replay's cache checkspools_genbefore it dereferences. The 600-submit memory line now also counts the records reaped.RX_VK_REAP_RETIRED=1, off by default per the plan's rule that a new path ships behind a switch; documented indocs/development.md.tests/retired_give_back_test.cpp: each condition that keeps a record, and a simulated session where reaping holds the list to one area change's worth;tests/test_retired_give_back.pychecks thatvk_memory.cppgoes through the rule, in that place.Measurements
Verdict: On one build and one route, the reap changes the retired-record count and nothing else measurable. With
RX_VK_REAP_RETIRED=1no retired pool record is left at the end (169 and 119 reaped), against 179 and 115 kept without it. Live imports, mirrors, captures, VRAM and the game's own memory overlap between the two settings, run to run. There were no errors. Frame time isn't measured here: skipping a walk over a few hundred records saves microseconds a submit.Hardware: Ubuntu 26.04.1 LTS (kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D (8 cores), 32 GB
Scene:
scripts/memtour.pythrough Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each, 2 laps (all 8 stops, about 7 minutes a run); the game folder's options. Kakariko was left out: an event there refused every later teleport in earlier tours.Method: one build of this PR's head (
a22b3411; the reap commit's changed lines are the same as at 13aa1d32 and at0ef6d036, this branch rebased onto mainac778751with headc328a0c4; only its base moved), four tours in turn (off, on, off, on), each a timed run throughscripts/job_queue.pywith nothing else running. The values are each run's lastvk: renderer memoryline (600 submits) andmemtour.py's summary.Baseline: the Before column is this PR's build with the switch unset, where the reap never runs and the records are kept as on main. Main itself wasn't the comparison: this PR is stacked on #1034, and main has no memory line with the record count to compare (it comes from #1034).
How to reproduce:
python -I scripts/memtour.py <game> --save <folder> --dests 13,3,7,9 --dwell 40 --laps 2 --settle 20, then the same with--env RX_VK_REAP_RETIRED=1RX_VK_REAP_RETIREDunset: this PR @a22b3411(runs 1 and 3)RX_VK_REAP_RETIRED=1: this PR @a22b3411(runs 2 and 4)Each run is in
docs/tested/2026-11-16-retired-reap-ab-trashpanda-run01.mdtorun04.md. The earlier single runs (Stage 0's 25-minute tour against this PR's 14-minute one, on different commits and routes) are replaced by this A/B.Flags and settings
One new switch, off by default.
RX_VK_MARK_AUDITandRX_VK_MARK_AUDIT_MBin the diff come from #1034, which this PR is stacked on, and are listed there.RX_VK_REAP_RETIRED=1Testing
Tested:
python -I -B tests/run_all.py --jobs 6on Linux atc328a0c4: 649 passed, 24 failed, 21 skipped;test_retired_give_backpasses, and every failure also fails onmainac778751on this PC (listed in #1034). The renderer cross-builds for Windows with llvm-mingw. In the game: the tour above on Linux with an RTX 4070 Ti, with no device loss or Vulkan error in the log.Not tested: Windows, where retired imports wait 30 s on AMD and Intel (
retired_modeGRACE) and are kept withRX_VK_KEEP_RETIRED=1. A record isn't reaped while it holds its import, so in those modes it waits for the import to go first. The two runs below are needed before the switch's default flips (a follow-up PR), not for this one, which ships it off.Must be tested:
RX_VK_REAP_RETIRED=1, the same memory tour or 20 minutes through several areas, expectingretired pools 0or a small number in the memory lines and no device loss (Test Farm:@Beanstalk windows amd, a memory tour with RX_VK_REAP_RETIRED=1).RX_VK_REAP_RETIRED=1, the same tour, expectingretired pools 0or a small number in the memory lines and no device loss (Test Farm:@Beanstalk steamdeck, a memory tour with RX_VK_REAP_RETIRED=1).Risks
What could break: a pointer to a retired pool kept somewhere this PR missed would now point into freed memory, and could crash or lose the device. The lists that point at pools are named above, and the records only go once the GPU is past them.
Turning it off: leave
RX_VK_REAP_RETIREDunset, the default.Compatibility: Nothing changes.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
docs/tested/for each new measured run (format indocs/tested-configurations.md).mainand passing here.tests/that reads changed source updated in this PR, not weakened.Tested in game: Linux (Ubuntu 26.04.1), RTX 4070 Ti, NVIDIA 595.99.02, Ryzen 7 9800X3D: a 14-minute memtour.py session (Zora's Domain, Kakariko, the Market, Hyrule Field, Death Mountain Trail, 2 laps and 2 areas) with RX_VK_REAP_RETIRED=1: retired pool records 0 at all 1,431 memory lines, 191 reaped, no device loss or Vulkan error; against 159 kept by the end of the same kind of tour without it.
98d3aaf7a7e1401f05daNice and small.
record_doneas a pure rule with its own test, and reaping only aftermirror_synced/writebacksare cleared, is the right shape. I went through everyPool*holder invk_memory.cpp(mirror_wanted,sparse_wanted,pool_cache,last_pool,dead_blocks,tl_pools, the inherit loop inmirror_make) and couldn't find one that outlives the conditions you check. Two things:retired_pools(~3707: "Retired pools are kept (a few, never reused)") and the one abovereap_mirrors(~4435: "The node stays (pointers to it are kept elsewhere)") now hold only with the switch off. Safety now rests on everyPool*list dropping a pool when it retires, and nothing tells the next person who adds such a list. I'd update those two comments, list the holders in one place (atretire_pool), and note attl_pools(~2968) thatpcan now dangle and is only safe because thepools_gencheck runs first.memtour.pyroute with and withoutRX_VK_REAP_RETIRED=1. That would show the reap changes nothing but the record count. It would also help to say whether the reap commit changed in the rebase.This is from reading the code only. I didn't build or run anything.
🤖 Generated with Claude Code
@ -4729,0 +4891,4 @@int64_t st_retired_reaped = 0;/// (pools_mu held, after the submission's per-submission lists are cleared) The records of retired pools that hold nothing/// any more (retired_give_back::record_done). RX_VK_REAP_RETIRED=1; off, they stay for the session, as beforevoid retired_pools_reap() {This is the first place a retired
Poolgets freed, so it now depends on everyPool*list dropping a pool atretire_pool. I checked thatmirror_wanted,sparse_wanted,pool_cacheandlast_pool(retire_pool),dead_blocks(mirror_free) andmirror_synced/writebacks(cleared just above) all do.tl_poolskeeps a danglingpbut is saved bypools_genbeing checked before the dereference. Could you write that list down here or atretire_pool, so a new list that holds aPool*across submissions doesn't become a use-after-free that only happens with the switch on?e1401f05daa22b3411e1@dyst thanks for going through every
Pool*holder. Both points are done.1. The comments.
a22b3411rewrites the two comments that still said records are never freed: aboveretired_poolsand abovereap_mirrors. It also lists every list that holds aPool*in one place atretire_pool:mirror_wanted,sparse_wanted,pool_cacheandlast_pool(dropped there),dead_blocks(mirror_free), andmirror_synced/writebacks(cleared before the reap). Attl_poolsit notes thatpcan dangle with the switch on and is safe only becausepools_genis checked first. A new list that holds aPool*across submissions has to be added to that list.2. A clean A/B. One build of this head (
a22b3411), the samememtour.pyroute four times in turn (off, on, off, on), each a timed run through the job queue with nothing else running. The route is Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each, 2 laps. Kakariko is left out because an event there refused every later teleport in the first set of runs. All four runs reached all 8 stops, with no errors.RX_VK_REAP_RETIRED=1(runs 2 / 4)The reap changes the record count. The other rows overlap between the two settings, run to run.
The rebase: the reap commit's added and removed lines are identical at 13aa1d32 and at today's
f938d258; only its base moved.The Measurements section now has this A/B in place of the two unequal runs. The four runs are in
docs/tested/2026-11-16-retired-reap-ab-trashpanda-run01.mdtorun04.md(ab9bda10).ab9bda100fc328a0c4c4Rebased onto the new #1034 head
7046f6da, which is on main @ac778751(headc328a0c4; wasab9bda10).Conflict: none in this PR's four commits.
git range-diffshows them identical (f938d258→0ef6d036, 83535f62→10a90583, a22b3411→186ef506, ab9bda10→c328a0c4). The reap commit's lines are unchanged by the rebase; only their line offsets moved. The measured build staysa22b3411. #1034's one conflict (itsChainSpanaroundtake_capture's late wait, which main's #977 changed) is described there.Tests:
test_retired_give_backpasses.python -I -B tests/run_all.py --jobs 6atc328a0c4: 649 passed, 24 failed, 21 skipped. All 24 also fail on mainac778751on this PC. The Windows renderer cross-builds and links with llvm-mingw 20261104.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.