WIP: perf(vk): free retired pool records that hold nothing (RX_VK_REAP_RETIRED), so they stop piling up over a session #1037

Draft
TrashPanda wants to merge 8 commits from TrashPanda/Longshot:perf/stage1-retired-reap into main
Contributor

Type: perf · Risk: low · Tested on: Linux, RTX 4070 Ti · Needs: Windows AMD or Intel, and the Steam Deck, before the switch's default flips (a later PR)

Depends on #1034 (Stage 0: its memory line is what this is measured with). This branch is stacked on it. Review the last two commits, perf(vk): reap retired pool records... and docs(tested): ....

Why

The renderer keeps a record for every game pool it had to remake larger, for the rest of the session, and walks all of them at every submission (five loops in vk_memory.cpp). Their imports and mirrors are already given back, so the records hold nothing, but they pile up as you play: Stage 0's memory tour counted 159 after 25 minutes, about 10 more per area change. Over a long session that's thousands of empty records walked at every submit. The plan's Stage 1 asks for the renderer's memory and its per-submit walks to stay flat after warm-up. Of everything Stage 0's tour counted, this is the only thing that kept growing.

What changed

A retired pool's record is freed once nothing is left in it and nothing still points at it. That was already the case for pools evicted to save memory (evicted_pools), but not for pools remade larger.

  • gpu/vk/retired_give_back.h: record_done(Leftover, completed). A record can go once its import, mirror and shadow are gone (each queued for freeing by value in dead_imports/dead_mirrors), no pool waits to inherit its mirror (inherit_pending) or for a trim to settle (trimmed), and the GPU is past both the submission that retired it and the last one that bound it.
  • gpu/vk/vk_memory.cpp: retired_pools_reap() in memory_begin_submission, after the per-submission lists that hold Pool* (mirror_synced, writebacks) are cleared. Other pointers to a retired pool are already dropped when it's retired (mirror_wanted, sparse_wanted, pool_cache, last_pool) or when its mirror is freed (dead_blocks), and the parallel replay's cache checks pools_gen before it dereferences. The 600-submit memory line now also counts the records reaped.
  • Behind RX_VK_REAP_RETIRED=1, off by default per the plan's rule that a new path ships behind a switch; documented in docs/development.md.
  • tests/retired_give_back_test.cpp: each condition that keeps a record, and a simulated session where reaping holds the list to one area change's worth; tests/test_retired_give_back.py checks that vk_memory.cpp goes through the rule, in that place.

Measurements

Verdict: On one build and one route, the reap changes the retired-record count and nothing else measurable. With RX_VK_REAP_RETIRED=1 no retired pool record is left at the end (169 and 119 reaped), against 179 and 115 kept without it. Live imports, mirrors, captures, VRAM and the game's own memory overlap between the two settings, run to run. There were no errors. Frame time isn't measured here: skipping a walk over a few hundred records saves microseconds a submit.

Hardware: Ubuntu 26.04.1 LTS (kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D (8 cores), 32 GB
Scene: scripts/memtour.py through Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each, 2 laps (all 8 stops, about 7 minutes a run); the game folder's options. Kakariko was left out: an event there refused every later teleport in earlier tours.
Method: one build of this PR's head (a22b3411; the reap commit's changed lines are the same as at 13aa1d32 and at 0ef6d036, this branch rebased onto main ac778751 with head c328a0c4; only its base moved), four tours in turn (off, on, off, on), each a timed run through scripts/job_queue.py with nothing else running. The values are each run's last vk: renderer memory line (600 submits) and memtour.py's summary.
Baseline: the Before column is this PR's build with the switch unset, where the reap never runs and the records are kept as on main. Main itself wasn't the comparison: this PR is stacked on #1034, and main has no memory line with the record count to compare (it comes from #1034).
How to reproduce: python -I scripts/memtour.py <game> --save <folder> --dests 13,3,7,9 --dwell 40 --laps 2 --settle 20, then the same with --env RX_VK_REAP_RETIRED=1

Before, RX_VK_REAP_RETIRED unset: this PR @ a22b3411 (runs 1 and 3) After, RX_VK_REAP_RETIRED=1: this PR @ a22b3411 (runs 2 and 4)
retired pool records at the end 179 / 115 0 / 0
records reaped 0 / 0 169 / 119
live imports at the end 109 (1,916 MB) / 155 (2,316 MB) 152 (2,148 MB) / 118 (1,784 MB)
mirrors at the end (of 2,048 MB) 825 / 825 MB 880 / 846 MB
captures at the end 370 / 496 MB 268 / 216 MB
VRAM in use at the end 7,344 / 8,210 MB 8,404 / 6,508 MB
game total (resident + swapped), peak 15,458 / 14,181 MB 14,291 / 15,653 MB
game total at the end 14,221 / 13,521 MB 13,881 / 14,146 MB
private memory, peak 5,790 / 5,564 MB 5,846 / 5,584 MB
shared memory, peak 9,842 / 9,146 MB 9,003 / 10,082 MB

Each run is in docs/tested/2026-11-16-retired-reap-ab-trashpanda-run01.md to run04.md. The earlier single runs (Stage 0's 25-minute tour against this PR's 14-minute one, on different commits and routes) are replaced by this A/B.

Flags and settings

One new switch, off by default. RX_VK_MARK_AUDIT and RX_VK_MARK_AUDIT_MB in the diff come from #1034, which this PR is stacked on, and are listed there.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
RX_VK_REAP_RETIRED=1 frees retired pool records that hold nothing - off the record list stays empty instead of growing about 10 per area change helps on Linux NVIDIA (above); untested on Windows, AMD, Intel and the Steam Deck off until it has run on Windows AMD and Intel and the Steam Deck, then on for everyone (a follow-up PR)

Testing

Tested: python -I -B tests/run_all.py --jobs 6 on Linux at c328a0c4: 649 passed, 24 failed, 21 skipped; test_retired_give_back passes, and every failure also fails on main ac778751 on this PC (listed in #1034). The renderer cross-builds for Windows with llvm-mingw. In the game: the tour above on Linux with an RTX 4070 Ti, with no device loss or Vulkan error in the log.

Not tested: Windows, where retired imports wait 30 s on AMD and Intel (retired_mode GRACE) and are kept with RX_VK_KEEP_RETIRED=1. A record isn't reaped while it holds its import, so in those modes it waits for the import to go first. The two runs below are needed before the switch's default flips (a follow-up PR), not for this one, which ships it off.

Must be tested:

  • Windows, AMD or Intel GPU: RX_VK_REAP_RETIRED=1, the same memory tour or 20 minutes through several areas, expecting retired pools 0 or a small number in the memory lines and no device loss (Test Farm: @Beanstalk windows amd, a memory tour with RX_VK_REAP_RETIRED=1).
  • Steam Deck: RX_VK_REAP_RETIRED=1, the same tour, expecting retired pools 0 or a small number in the memory lines and no device loss (Test Farm: @Beanstalk steamdeck, a memory tour with RX_VK_REAP_RETIRED=1).

Risks

What could break: a pointer to a retired pool kept somewhere this PR missed would now point into freed memory, and could crash or lose the device. The lists that point at pools are named above, and the records only go once the GPU is past them.

Turning it off: leave RX_VK_REAP_RETIRED unset, the default.

Compatibility: Nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md).
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.
**Type:** perf · **Risk:** low · **Tested on:** Linux, RTX 4070 Ti · **Needs:** Windows AMD or Intel, and the Steam Deck, before the switch's default flips (a later PR) **Depends on #1034** (Stage 0: its memory line is what this is measured with). This branch is stacked on it. Review the last two commits, `perf(vk): reap retired pool records...` and `docs(tested): ...`. ## Why The renderer keeps a record for every game pool it had to remake larger, for the rest of the session, and walks all of them at every submission (five loops in `vk_memory.cpp`). Their imports and mirrors are already given back, so the records hold nothing, but they pile up as you play: Stage 0's memory tour counted 159 after 25 minutes, about 10 more per area change. Over a long session that's thousands of empty records walked at every submit. The plan's Stage 1 asks for the renderer's memory and its per-submit walks to stay flat after warm-up. Of everything Stage 0's tour counted, this is the only thing that kept growing. ## What changed A retired pool's record is freed once nothing is left in it and nothing still points at it. That was already the case for pools evicted to save memory (`evicted_pools`), but not for pools remade larger. - `gpu/vk/retired_give_back.h`: `record_done(Leftover, completed)`. A record can go once its import, mirror and shadow are gone (each queued for freeing by value in `dead_imports`/`dead_mirrors`), no pool waits to inherit its mirror (`inherit_pending`) or for a trim to settle (`trimmed`), and the GPU is past both the submission that retired it and the last one that bound it. - `gpu/vk/vk_memory.cpp`: `retired_pools_reap()` in `memory_begin_submission`, after the per-submission lists that hold `Pool*` (`mirror_synced`, `writebacks`) are cleared. Other pointers to a retired pool are already dropped when it's retired (`mirror_wanted`, `sparse_wanted`, `pool_cache`, `last_pool`) or when its mirror is freed (`dead_blocks`), and the parallel replay's cache checks `pools_gen` before it dereferences. The 600-submit memory line now also counts the records reaped. - Behind `RX_VK_REAP_RETIRED=1`, off by default per the plan's rule that a new path ships behind a switch; documented in `docs/development.md`. - `tests/retired_give_back_test.cpp`: each condition that keeps a record, and a simulated session where reaping holds the list to one area change's worth; `tests/test_retired_give_back.py` checks that `vk_memory.cpp` goes through the rule, in that place. ## Measurements **Verdict:** On one build and one route, the reap changes the retired-record count and nothing else measurable. With `RX_VK_REAP_RETIRED=1` no retired pool record is left at the end (169 and 119 reaped), against 179 and 115 kept without it. Live imports, mirrors, captures, VRAM and the game's own memory overlap between the two settings, run to run. There were no errors. Frame time isn't measured here: skipping a walk over a few hundred records saves microseconds a submit. Hardware: Ubuntu 26.04.1 LTS (kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D (8 cores), 32 GB Scene: `scripts/memtour.py` through Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each, 2 laps (all 8 stops, about 7 minutes a run); the game folder's options. Kakariko was left out: an event there refused every later teleport in earlier tours. Method: one build of this PR's head (a22b3411; the reap commit's changed lines are the same as at 13aa1d32 and at 0ef6d036, this branch rebased onto main ac778751 with head c328a0c4; only its base moved), four tours in turn (off, on, off, on), each a timed run through `scripts/job_queue.py` with nothing else running. The values are each run's last `vk: renderer memory` line (600 submits) and `memtour.py`'s summary. Baseline: the Before column is this PR's build with the switch unset, where the reap never runs and the records are kept as on main. Main itself wasn't the comparison: this PR is stacked on #1034, and main has no memory line with the record count to compare (it comes from #1034). How to reproduce: `python -I scripts/memtour.py <game> --save <folder> --dests 13,3,7,9 --dwell 40 --laps 2 --settle 20`, then the same with `--env RX_VK_REAP_RETIRED=1` | | Before, `RX_VK_REAP_RETIRED` unset: this PR @ a22b3411 (runs 1 and 3) | After, `RX_VK_REAP_RETIRED=1`: this PR @ a22b3411 (runs 2 and 4) | |---|---|---| | retired pool records at the end | 179 / 115 | **0 / 0** | | records reaped | 0 / 0 | 169 / 119 | | live imports at the end | 109 (1,916 MB) / 155 (2,316 MB) | 152 (2,148 MB) / 118 (1,784 MB) | | mirrors at the end (of 2,048 MB) | 825 / 825 MB | 880 / 846 MB | | captures at the end | 370 / 496 MB | 268 / 216 MB | | VRAM in use at the end | 7,344 / 8,210 MB | 8,404 / 6,508 MB | | game total (resident + swapped), peak | 15,458 / 14,181 MB | 14,291 / 15,653 MB | | game total at the end | 14,221 / 13,521 MB | 13,881 / 14,146 MB | | private memory, peak | 5,790 / 5,564 MB | 5,846 / 5,584 MB | | shared memory, peak | 9,842 / 9,146 MB | 9,003 / 10,082 MB | Each run is in `docs/tested/2026-11-16-retired-reap-ab-trashpanda-run01.md` to `run04.md`. The earlier single runs (Stage 0's 25-minute tour against this PR's 14-minute one, on different commits and routes) are replaced by this A/B. ### Flags and settings One new switch, off by default. `RX_VK_MARK_AUDIT` and `RX_VK_MARK_AUDIT_MB` in the diff come from #1034, which this PR is stacked on, and are listed there. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `RX_VK_REAP_RETIRED=1` | frees retired pool records that hold nothing | - | off | the record list stays empty instead of growing about 10 per area change | helps on Linux NVIDIA (above); untested on Windows, AMD, Intel and the Steam Deck | off until it has run on Windows AMD and Intel and the Steam Deck, then on for everyone (a follow-up PR) | ## Testing **Tested:** `python -I -B tests/run_all.py --jobs 6` on Linux at c328a0c4: 649 passed, 24 failed, 21 skipped; `test_retired_give_back` passes, and every failure also fails on `main` ac778751 on this PC (listed in #1034). The renderer cross-builds for Windows with llvm-mingw. In the game: the tour above on Linux with an RTX 4070 Ti, with no device loss or Vulkan error in the log. **Not tested:** Windows, where retired imports wait 30 s on AMD and Intel (`retired_mode` GRACE) and are kept with `RX_VK_KEEP_RETIRED=1`. A record isn't reaped while it holds its import, so in those modes it waits for the import to go first. The two runs below are needed before the switch's default flips (a follow-up PR), not for this one, which ships it off. **Must be tested:** - Windows, AMD or Intel GPU: `RX_VK_REAP_RETIRED=1`, the same memory tour or 20 minutes through several areas, expecting `retired pools 0` or a small number in the memory lines and no device loss (Test Farm: `@Beanstalk windows amd, a memory tour with RX_VK_REAP_RETIRED=1`). - Steam Deck: `RX_VK_REAP_RETIRED=1`, the same tour, expecting `retired pools 0` or a small number in the memory lines and no device loss (Test Farm: `@Beanstalk steamdeck, a memory tour with RX_VK_REAP_RETIRED=1`). ## Risks **What could break:** a pointer to a retired pool kept somewhere this PR missed would now point into freed memory, and could crash or lose the device. The lists that point at pools are named above, and the records only go once the GPU is past them. **Turning it off:** leave `RX_VK_REAP_RETIRED` unset, the default. **Compatibility:** Nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`). - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.
HOS_CHAIN records ENTER/LEAVE spans around the renderer's calls (submit, present, GPU waits,
texel reads, memory work, slot and late waits); scripts/chain.py splits the critical path into
the game's own work, the renderer's, and GPU waits, and joins it with the GPU timestamps
(--log) into a ceiling report. MKWVK_API_VERSION 24 (chain_span, chain_on).

RX_VK_MARK_AUDIT=N compares a copy of the synced pages with guest memory every Nth submit and
logs differences no write mark covered.

tests/test_collect_concurrent.py: collect_in loses no mark set while it runs.
RX_VK_MARK_AUDIT found three host writes of guest memory that set no write mark, so the
captures kept the pages' old bytes:

- nvnCommandBufferEndRecording's copy mark in the command buffer's control memory
  (9 pages a window in Kokiri Forest, at offset 0x1c8), found with a hardware watchpoint;
- nvnQueueSubmitCommands' submit count in the game's queue object (offset 0x28);
- FreeMemoryPages, discard_pages and UnmapPhysicalMemory zeroing freed pages (12 MB of
  texture storage at one load).

Kokiri Forest and Hyrule Field now audit with 0 missed pages.
Co-authored-by: Cursor <[email protected]>
docs(tested): the Stage 0 baseline runs with the ceiling report (Kokiri Forest, Hyrule Field, RTX 4070 Ti, Linux)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
5b156bfdda
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Cursor <[email protected]>
docs(tested): a memory tour with the retired pool records reaped
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
98d3aaf7a7
Co-authored-by: Cursor <[email protected]>
Author
Contributor

Tested in game: Linux (Ubuntu 26.04.1), RTX 4070 Ti, NVIDIA 595.99.02, Ryzen 7 9800X3D: a 14-minute memtour.py session (Zora's Domain, Kakariko, the Market, Hyrule Field, Death Mountain Trail, 2 laps and 2 areas) with RX_VK_REAP_RETIRED=1: retired pool records 0 at all 1,431 memory lines, 191 reaped, no device loss or Vulkan error; against 159 kept by the end of the same kind of tour without it.

Tested in game: Linux (Ubuntu 26.04.1), RTX 4070 Ti, NVIDIA 595.99.02, Ryzen 7 9800X3D: a 14-minute memtour.py session (Zora's Domain, Kakariko, the Market, Hyrule Field, Death Mountain Trail, 2 laps and 2 areas) with RX_VK_REAP_RETIRED=1: retired pool records 0 at all 1,431 memory lines, 191 reaped, no device loss or Vulkan error; against 159 kept by the end of the same kind of tour without it.
TrashPanda force-pushed perf/stage1-retired-reap from 98d3aaf7a7
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to e1401f05da
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-16 04:21:45 +00:00
Compare
dyst left a comment

Nice and small. record_done as a pure rule with its own test, and reaping only after mirror_synced/writebacks are cleared, is the right shape. I went through every Pool* holder in vk_memory.cpp (mirror_wanted, sparse_wanted, pool_cache, last_pool, dead_blocks, tl_pools, the inherit loop in mirror_make) and couldn't find one that outlives the conditions you check. Two things:

  1. The "records are never freed" invariant is still written down in the code. The comment above retired_pools (~3707: "Retired pools are kept (a few, never reused)") and the one above reap_mirrors (~4435: "The node stays (pointers to it are kept elsewhere)") now hold only with the switch off. Safety now rests on every Pool* list dropping a pool when it retires, and nothing tells the next person who adds such a list. I'd update those two comments, list the holders in one place (at retire_pool), and note at tl_pools (~2968) that p can now dangle and is only safe because the pools_gen check runs first.
  2. Measurements: before and after differ in commit (d66b4ac6 vs 13aa1d32, neither of them the current head), route and length (25 vs 14 min). So the other rows (live imports, mirrors) can't be compared, even though the 0-records result clearly holds. The switch makes a clean A/B cheap: one build, the same memtour.py route with and without RX_VK_REAP_RETIRED=1. That would show the reap changes nothing but the record count. It would also help to say whether the reap commit changed in the rebase.

This is from reading the code only. I didn't build or run anything.

🤖 Generated with Claude Code

Nice and small. `record_done` as a pure rule with its own test, and reaping only after `mirror_synced`/`writebacks` are cleared, is the right shape. I went through every `Pool*` holder in `vk_memory.cpp` (`mirror_wanted`, `sparse_wanted`, `pool_cache`, `last_pool`, `dead_blocks`, `tl_pools`, the inherit loop in `mirror_make`) and couldn't find one that outlives the conditions you check. Two things: 1. **The "records are never freed" invariant is still written down in the code.** The comment above `retired_pools` (~3707: "Retired pools are kept (a few, never reused)") and the one above `reap_mirrors` (~4435: "The node stays (pointers to it are kept elsewhere)") now hold only with the switch off. Safety now rests on every `Pool*` list dropping a pool when it retires, and nothing tells the next person who adds such a list. I'd update those two comments, list the holders in one place (at `retire_pool`), and note at `tl_pools` (~2968) that `p` can now dangle and is only safe because the `pools_gen` check runs first. 2. **Measurements:** before and after differ in commit (d66b4ac6 vs 13aa1d32, neither of them the current head), route and length (25 vs 14 min). So the other rows (live imports, mirrors) can't be compared, even though the 0-records result clearly holds. The switch makes a clean A/B cheap: one build, the same `memtour.py` route with and without `RX_VK_REAP_RETIRED=1`. That would show the reap changes nothing but the record count. It would also help to say whether the reap commit changed in the rebase. This is from reading the code only. I didn't build or run anything. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
@ -4729,0 +4891,4 @@
int64_t st_retired_reaped = 0;
/// (pools_mu held, after the submission's per-submission lists are cleared) The records of retired pools that hold nothing
/// any more (retired_give_back::record_done). RX_VK_REAP_RETIRED=1; off, they stay for the session, as before
void retired_pools_reap() {
Member

This is the first place a retired Pool gets freed, so it now depends on every Pool* list dropping a pool at retire_pool. I checked that mirror_wanted, sparse_wanted, pool_cache and last_pool (retire_pool), dead_blocks (mirror_free) and mirror_synced/writebacks (cleared just above) all do. tl_pools keeps a dangling p but is saved by pools_gen being checked before the dereference. Could you write that list down here or at retire_pool, so a new list that holds a Pool* across submissions doesn't become a use-after-free that only happens with the switch on?

This is the first place a retired `Pool` gets freed, so it now depends on every `Pool*` list dropping a pool at `retire_pool`. I checked that `mirror_wanted`, `sparse_wanted`, `pool_cache` and `last_pool` (retire_pool), `dead_blocks` (mirror_free) and `mirror_synced`/`writebacks` (cleared just above) all do. `tl_pools` keeps a dangling `p` but is saved by `pools_gen` being checked before the dereference. Could you write that list down here or at `retire_pool`, so a new list that holds a `Pool*` across submissions doesn't become a use-after-free that only happens with the switch on?
TrashPanda force-pushed perf/stage1-retired-reap from e1401f05da
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to a22b3411e1
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 8s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-16 13:55:20 +00:00
Compare
docs(tested): the retired-record reap A/B, one build, the same tour with and without RX_VK_REAP_RETIRED
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
ab9bda100f
Author
Contributor

@dyst thanks for going through every Pool* holder. Both points are done.

1. The comments. a22b3411 rewrites the two comments that still said records are never freed: above retired_pools and above reap_mirrors. It also lists every list that holds a Pool* in one place at retire_pool: mirror_wanted, sparse_wanted, pool_cache and last_pool (dropped there), dead_blocks (mirror_free), and mirror_synced/writebacks (cleared before the reap). At tl_pools it notes that p can dangle with the switch on and is safe only because pools_gen is checked first. A new list that holds a Pool* across submissions has to be added to that list.

2. A clean A/B. One build of this head (a22b3411), the same memtour.py route four times in turn (off, on, off, on), each a timed run through the job queue with nothing else running. The route is Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each, 2 laps. Kakariko is left out because an event there refused every later teleport in the first set of runs. All four runs reached all 8 stops, with no errors.

off (runs 1 / 3) RX_VK_REAP_RETIRED=1 (runs 2 / 4)
retired pool records at the end 179 / 115 0 / 0
records reaped 0 / 0 169 / 119
live imports at the end 109 / 155 152 / 118
mirrors at the end 825 / 825 MB 880 / 846 MB
captures at the end 370 / 496 MB 268 / 216 MB
VRAM at the end 7,344 / 8,210 MB 8,404 / 6,508 MB
game total, peak 15,458 / 14,181 MB 14,291 / 15,653 MB
game total at the end 14,221 / 13,521 MB 13,881 / 14,146 MB

The reap changes the record count. The other rows overlap between the two settings, run to run.

The rebase: the reap commit's added and removed lines are identical at 13aa1d32 and at today's f938d258; only its base moved.

The Measurements section now has this A/B in place of the two unequal runs. The four runs are in docs/tested/2026-11-16-retired-reap-ab-trashpanda-run01.md to run04.md (ab9bda10).

@dyst thanks for going through every `Pool*` holder. Both points are done. **1. The comments.** `a22b3411` rewrites the two comments that still said records are never freed: above `retired_pools` and above `reap_mirrors`. It also lists every list that holds a `Pool*` in one place at `retire_pool`: `mirror_wanted`, `sparse_wanted`, `pool_cache` and `last_pool` (dropped there), `dead_blocks` (`mirror_free`), and `mirror_synced`/`writebacks` (cleared before the reap). At `tl_pools` it notes that `p` can dangle with the switch on and is safe only because `pools_gen` is checked first. A new list that holds a `Pool*` across submissions has to be added to that list. **2. A clean A/B.** One build of this head (a22b3411), the same `memtour.py` route four times in turn (off, on, off, on), each a timed run through the job queue with nothing else running. The route is Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each, 2 laps. Kakariko is left out because an event there refused every later teleport in the first set of runs. All four runs reached all 8 stops, with no errors. | | off (runs 1 / 3) | `RX_VK_REAP_RETIRED=1` (runs 2 / 4) | |---|---|---| | retired pool records at the end | 179 / 115 | **0 / 0** | | records reaped | 0 / 0 | 169 / 119 | | live imports at the end | 109 / 155 | 152 / 118 | | mirrors at the end | 825 / 825 MB | 880 / 846 MB | | captures at the end | 370 / 496 MB | 268 / 216 MB | | VRAM at the end | 7,344 / 8,210 MB | 8,404 / 6,508 MB | | game total, peak | 15,458 / 14,181 MB | 14,291 / 15,653 MB | | game total at the end | 14,221 / 13,521 MB | 13,881 / 14,146 MB | The reap changes the record count. The other rows overlap between the two settings, run to run. **The rebase:** the reap commit's added and removed lines are identical at 13aa1d32 and at today's f938d258; only its base moved. The Measurements section now has this A/B in place of the two unequal runs. The four runs are in `docs/tested/2026-11-16-retired-reap-ab-trashpanda-run01.md` to `run04.md` (ab9bda10).
TrashPanda force-pushed perf/stage1-retired-reap from ab9bda100f
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 6s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to c328a0c4c4
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
2026-11-16 19:45:00 +00:00
Compare
Author
Contributor

Rebased onto the new #1034 head 7046f6da, which is on main @ ac778751 (head c328a0c4; was ab9bda10).

Conflict: none in this PR's four commits. git range-diff shows them identical (f938d258→0ef6d036, 83535f62→10a90583, a22b3411→186ef506, ab9bda10→c328a0c4). The reap commit's lines are unchanged by the rebase; only their line offsets moved. The measured build stays a22b3411. #1034's one conflict (its ChainSpan around take_capture's late wait, which main's #977 changed) is described there.

Tests: test_retired_give_back passes. python -I -B tests/run_all.py --jobs 6 at c328a0c4: 649 passed, 24 failed, 21 skipped. All 24 also fail on main ac778751 on this PC. The Windows renderer cross-builds and links with llvm-mingw 20261104.

Rebased onto the new #1034 head 7046f6da, which is on main @ ac778751 (head c328a0c4; was ab9bda10). **Conflict:** none in this PR's four commits. `git range-diff` shows them identical (f938d258→0ef6d036, 83535f62→10a90583, a22b3411→186ef506, ab9bda10→c328a0c4). The reap commit's lines are unchanged by the rebase; only their line offsets moved. The measured build stays a22b3411. #1034's one conflict (its `ChainSpan` around `take_capture`'s late wait, which main's #977 changed) is described there. **Tests:** `test_retired_give_back` passes. `python -I -B tests/run_all.py --jobs 6` at c328a0c4: 649 passed, 24 failed, 21 skipped. All 24 also fail on main ac778751 on this PC. The Windows renderer cross-builds and links with llvm-mingw 20261104.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/stage1-retired-reap:TrashPanda-perf/stage1-retired-reap'
git switch 'TrashPanda-perf/stage1-retired-reap'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1037
No description provided.