perf(vk): RX_VK_WB_MAX_KB writes large GPU-written bindings back only where the CPU reads or writes them (opt-in)

github-sync commented

2026-11-12 10:34:12 +00:00

Owner

@klikso opened this pull request on GitHub · 2026-11-12 09:45 UTC · original
Merged on GitHub in 8e6c15af42

Why. Every frame OoT writes ~68 MB through the GPU into bindings over 1 MB (culling lists and per-frame buffers), and all of it is copied back to guest memory at the end of its submission (#384: ~4.4 ms a submit on an RTX 3080). The CPU touches only small parts of those bindings. RX_VK_WB_MAX_KB=1024 (opt-in, #384's experiment) skips them all and is fast, but it is not safe:

The CPU reads some large ones. With the plain cap and a diagnostic on invalidate, 236 of every 3,000 CPU reads in Hyrule Field landed in a skipped range: four 4.3 MB buffers (+0x44b040) read in full after every frame the GPU writes them, and an 8.4 MB range now and then. Those reads got what guest memory held before the GPU wrote them. The same in Kokiri Forest (168 of 3,000), the Market (76) and Kakariko (128).
The CPU writes pages inside them. ~12 pages a frame inside the 160 MB culling pool (0x1178000000), always the same ones. A mirror sync copies such a page whole from guest memory, so with the plain cap the rest of the page went back to stale bytes over the GPU's: ~28,000 pages in a 40 s run.

This keeps most of the saving and drops both hazards. It stays opt-in (RX_VK_WB_MAX_KB=1024): after review, on the two machines that ran it (this RTX 3060 Ti, and hckrman101's RX 580 with its mirror forced on) it writes back ~50 MB a frame less at the same frame rate, so the default waits for a machine where that shows as fps.

What

gpu/vk/wb_learn.h (plain rules, no Vulkan), held by vk_memory.cpp under its own mutex:

A written binding over RX_VK_WB_MAX_KB (unset or 0: off, the default; 1024 is the tested value) is written back whole in the first RX_VK_WB_PROBATION submissions that write it (8; several binds in one submission count once).
Meanwhile the CPU's reads (invalidate, before its write-back wait) and the CPU writes a mirror sync copies (each dirty page in a binding the table follows) are noted by range.
After the probation only those ranges are written back, plus the binding's first and last page when it starts or ends inside a page (a page shared with what is next to it); the rest of the binding is left out.
A range the CPU first touches after that finds the stale copy once, is written back from then on, and is logged (vk: the CPU read/wrote ..., in a binding whose write-back was left out).
Past 8,192 bindings or 8,192 merged ranges the table forgets everything and every binding starts its probation again (resets in the log line). CPU reads and writes are noted only inside a binding the table follows, and a mirror sync asks once per pool.
Without RX_VK_WB_MAX_KB every binding is written back, as before. RX_VK_LOG_WB_SKIPS=1 lists the ranges left out every 600 submits. A log line every 600 submits gives the count and MB left out.

Measured

Windows 11, RTX 3060 Ti 8 GB (driver 32.0.16.2114), Ryzen 5 5600X, 2560x1440 DLSS, main 5b96a55a; 40 s with the camera turning after a settle, RX_VK_FRAMEGEN=0, HOS_FPS_MAX=240, RX_VK_IMAGE_PREALLOC=0 in both arms (#2690), arms alternated.

	written back a frame	fps median	fps mean	GPU a frame
Hyrule Field, `RX_VK_WB_MAX_KB=0` (as before)	67.4 MB	58.8 / 58.2 / 58.6	57.2 / 56.4 / 56.8	12.4 ms
Hyrule Field, this PR	17.9 MB	60.0 / 59.4 / 60.5	58.6 / 57.3 / 58.0	12.15 ms
Kokiri Forest, as before	66.2 MB	59.1 / 63.4	59.6 / 62.7	11.8 ms
Kokiri Forest, this PR	16.1 MB	59.0 / 62.6	60.1 / 62.6	11.6 ms
Castle Town Market, as before	60.8 MB	27.5 / 40.6	29.6 / 38.9	19.5 / 14.0 ms
Castle Town Market, this PR	17.1 MB	38.3 / 43.5	40.2 / 42.1	11.9 / 11.8 ms

The Market's GPU time changes from launch to launch on 8 GB cards (blushOFF's note in #995), so two runs each there are suggestive, not proof. Late CPU touches of a range left out, which the log names: one page in one Market run; and, in the three runs started from a later save, one sweep of ~1,690 consecutive pages (0x117ffbd000 on, one 6.3 MB buffer) during the teleport's scene load, right after the new scene's pools are imported. That is the CPU setting up memory the title screen's buffers had used, whole pages, so nothing stale is mixed in; those pages are written back from then on. Fps of those runs: 58.2 / 57.6 / 55.7 mean.

The frame here is mostly the GPU and the once-a-frame exposure wait, so 50 MB less PCIe traffic a frame is worth ~2% on this card; it should be worth more where the copies are slower (laptops, x8 cards). The plain cap was faster (~61 fps, 9 MB a frame) because it also left out the pages the CPU writes and reads, which is the stale data above.

Screenshots at the same spot with and without it match.

Updates after review, the long-session counts and the flash check: see the comments below.

Tests

New tests/test_wb_learn.py compiles and runs tests/test_wb_learn.cpp (probation, a binding read every write, pages written during probation, touches clipped to the binding, a late touch reported once, merging, the bounds) and checks that vk_memory.cpp and vk_replay.cpp still call it where they should.
tests/direct_buffer_writes_test.cpp and tests/parallel_rb_refresh_test.cpp get stubs for the new calls (every write-back queued, as before).
python -I -B tests/run_all.py on Windows: 455 passed, 3 failed, 34 skipped. The three fail on unchanged main too here (nx2/test_nup_paths.py and tests/test_input_background.py: this machine's file links and input; tests/test_finish_output.py only when run beside the others, and passes alone).

🤖 Generated with Claude Code

> [@klikso](https://github.com/klikso) opened this pull request on GitHub · 2026-11-12 09:45 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2731) > **Merged on GitHub** in https://git.magicbean.dev/MagicBeanDev/Hookshot/commit/8e6c15af42e6e1af14aed7a6d75f4c34fa9f74d9 **Why.** Every frame OoT writes ~68 MB through the GPU into bindings over 1 MB (culling lists and per-frame buffers), and all of it is copied back to guest memory at the end of its submission (#384: ~4.4 ms a submit on an RTX 3080). The CPU touches only small parts of those bindings. `RX_VK_WB_MAX_KB=1024` (opt-in, #384's experiment) skips them all and is fast, but it is not safe: - **The CPU reads some large ones.** With the plain cap and a diagnostic on `invalidate`, 236 of every 3,000 CPU reads in Hyrule Field landed in a skipped range: four 4.3 MB buffers (`+0x44b040`) read in full after every frame the GPU writes them, and an 8.4 MB range now and then. Those reads got what guest memory held before the GPU wrote them. The same in Kokiri Forest (168 of 3,000), the Market (76) and Kakariko (128). - **The CPU writes pages inside them.** ~12 pages a frame inside the 160 MB culling pool (`0x1178000000`), always the same ones. A mirror sync copies such a page whole from guest memory, so with the plain cap the rest of the page went back to stale bytes over the GPU's: ~28,000 pages in a 40 s run. This keeps most of the saving and drops both hazards. **It stays opt-in** (`RX_VK_WB_MAX_KB=1024`): after review, on the two machines that ran it (this RTX 3060 Ti, and hckrman101's RX 580 with its mirror forced on) it writes back ~50 MB a frame less at the same frame rate, so the default waits for a machine where that shows as fps. ## What `gpu/vk/wb_learn.h` (plain rules, no Vulkan), held by `vk_memory.cpp` under its own mutex: - A written binding over `RX_VK_WB_MAX_KB` (unset or 0: off, the default; 1024 is the tested value) is written back whole in the first `RX_VK_WB_PROBATION` submissions that write it (8; several binds in one submission count once). - Meanwhile the CPU's reads (`invalidate`, before its write-back wait) and the CPU writes a mirror sync copies (each dirty page in a binding the table follows) are noted by range. - After the probation only those ranges are written back, plus the binding's first and last page when it starts or ends inside a page (a page shared with what is next to it); the rest of the binding is left out. - A range the CPU first touches after that finds the stale copy once, is written back from then on, and is logged (`vk: the CPU read/wrote ..., in a binding whose write-back was left out`). - Past 8,192 bindings or 8,192 merged ranges the table forgets everything and every binding starts its probation again (`resets` in the log line). CPU reads and writes are noted only inside a binding the table follows, and a mirror sync asks once per pool. - Without `RX_VK_WB_MAX_KB` every binding is written back, as before. `RX_VK_LOG_WB_SKIPS=1` lists the ranges left out every 600 submits. A log line every 600 submits gives the count and MB left out. ## Measured Windows 11, RTX 3060 Ti 8 GB (driver 32.0.16.2114), Ryzen 5 5600X, 2560x1440 DLSS, `main` 5b96a55a; 40 s with the camera turning after a settle, `RX_VK_FRAMEGEN=0`, `HOS_FPS_MAX=240`, `RX_VK_IMAGE_PREALLOC=0` in both arms (#2690), arms alternated. | | written back a frame | fps median | fps mean | GPU a frame | |---|---|---|---|---| | Hyrule Field, `RX_VK_WB_MAX_KB=0` (as before) | 67.4 MB | 58.8 / 58.2 / 58.6 | 57.2 / 56.4 / 56.8 | 12.4 ms | | Hyrule Field, this PR | 17.9 MB | 60.0 / 59.4 / 60.5 | 58.6 / 57.3 / 58.0 | 12.15 ms | | Kokiri Forest, as before | 66.2 MB | 59.1 / 63.4 | 59.6 / 62.7 | 11.8 ms | | Kokiri Forest, this PR | 16.1 MB | 59.0 / 62.6 | 60.1 / 62.6 | 11.6 ms | | Castle Town Market, as before | 60.8 MB | 27.5 / 40.6 | 29.6 / 38.9 | 19.5 / 14.0 ms | | Castle Town Market, this PR | 17.1 MB | 38.3 / 43.5 | 40.2 / 42.1 | 11.9 / 11.8 ms | The Market's GPU time changes from launch to launch on 8 GB cards (blushOFF's note in #995), so two runs each there are suggestive, not proof. Late CPU touches of a range left out, which the log names: one page in one Market run; and, in the three runs started from a later save, one sweep of ~1,690 consecutive pages (`0x117ffbd000` on, one 6.3 MB buffer) during the teleport's scene load, right after the new scene's pools are imported. That is the CPU setting up memory the title screen's buffers had used, whole pages, so nothing stale is mixed in; those pages are written back from then on. Fps of those runs: 58.2 / 57.6 / 55.7 mean. The frame here is mostly the GPU and the once-a-frame exposure wait, so 50 MB less PCIe traffic a frame is worth ~2% on this card; it should be worth more where the copies are slower (laptops, x8 cards). The plain cap was faster (~61 fps, 9 MB a frame) because it also left out the pages the CPU writes and reads, which is the stale data above. Screenshots at the same spot with and without it match. Updates after review, the long-session counts and the flash check: see the comments below. ## Tests - New `tests/test_wb_learn.py` compiles and runs `tests/test_wb_learn.cpp` (probation, a binding read every write, pages written during probation, touches clipped to the binding, a late touch reported once, merging, the bounds) and checks that `vk_memory.cpp` and `vk_replay.cpp` still call it where they should. - `tests/direct_buffer_writes_test.cpp` and `tests/parallel_rb_refresh_test.cpp` get stubs for the new calls (every write-back queued, as before). - `python -I -B tests/run_all.py` on Windows: 455 passed, 3 failed, 34 skipped. The three fail on unchanged `main` too here (`nx2/test_nup_paths.py` and `tests/test_input_background.py`: this machine's file links and input; `tests/test_finish_output.py` only when run beside the others, and passes alone). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

github-sync added 1 commit

2026-11-12 10:34:12 +00:00

perf(vk): large GPU-written bindings are written back only where the CPU reads or writes them be7b6d29ab

OoT writes ~68 MB a frame through the GPU into bindings over 1 MB (culling lists and per-frame
buffers), and every byte was copied back to guest memory at the end of its submission. The CPU
touches only small parts of them: four 4.3 MB buffers it reads in full, and ~12 pages a frame it
writes inside the culling pool. A binding over RX_VK_WB_MAX_KB (now 1024 by default) is written
back whole for its first RX_VK_WB_PROBATION writes (8) while the CPU's reads (invalidate) and the
writes a mirror sync copies are noted by range; after that only those ranges are written back.
A range the CPU first touches later is written back from then on, and logged.

Hyrule Field, RTX 3060 Ti: 67.4 -> 17.9 MB written back a frame, 56.8 -> 58.0 fps (mean of three
alternated runs each), no CPU touch of a range left out. RX_VK_WB_MAX_KB=0 writes everything
back as before.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

github-sync referenced this pull request from a commit

2026-11-12 11:35:41 +00:00

fix(vk): write-back probation counts submissions; the table resets instead of latching full

github-sync added 2 commits

2026-11-12 11:35:41 +00:00

fix(vk): a large binding's edge pages are always written back 0dd0459283

A binding that starts or ends inside a page shares that page with what is next to it. The 15 MB
culling lists end 0x80 bytes into a page the CPU writes, and a mirror sync copies that page whole,
so the binding's last bytes were found stale once (Goron City, Zora's Domain) before the page was
learned. Those pages are now written back from the binding's first write: at most 8 KB a binding.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

fix(vk): write-back probation counts submissions; the table resets instead of latching full bce3d17ec6

Review of #2731 (hckrman101):
- A binding bound by several dispatches in one submission counted each bind toward its probation,
  so it could finish in a frame. Probation now counts the submissions that write it (mirror_epoch),
  and a binding is left out only from the submission after its last one.
- Past its bounds the table latched "full" and wrote everything back for the rest of the session.
  It now forgets everything and every binding starts its probation again; the log counts resets.
- CPU reads are noted only in a large binding the table follows, as writes already were, so small
  readbacks elsewhere do not fill it.
- A mirror sync asks once per pool whether it overlaps a followed binding before noting its pages.
- The readme states the trade-off: a range first touched after the probation is stale once.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

github-sync added the

testing wanted

label

2026-11-12 11:36:24 +00:00

github-sync commented

2026-11-12 11:36:25 +00:00

Author

Owner

@hckrman101 commented on gpu/vk/wb_learn.h line 58 on GitHub · 2026-11-12 10:52 UTC · original

writes counts every written bind, and written binds skip the bind memo. A buffer that N dispatches write per frame therefore finishes probation after about 8/N frames, possibly before the CPU's first invalidate of its results is seen (the replay can be a frame ahead of that read). Counting at most once per submission would fix it: pass the recording's submission value into plan() and only increment when it changes.

> [@hckrman101](https://github.com/hckrman101) commented on `gpu/vk/wb_learn.h` line 58 on GitHub · 2026-11-12 10:52 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2731#discussion_r4549432909) `writes` counts every written bind, and written binds skip the bind memo. A buffer that N dispatches write per frame therefore finishes probation after about 8/N frames, possibly before the CPU's first `invalidate` of its results is seen (the replay can be a frame ahead of that read). Counting at most once per submission would fix it: pass the recording's submission value into `plan()` and only increment when it changes.

github-sync commented

2026-11-12 11:36:25 +00:00

Author

Owner

@hckrman101 commented on gpu/vk/wb_learn.h line 130 on GitHub · 2026-11-12 10:52 UTC · original

Once full_ is set it never clears, and neither seen_ nor read_ is ever trimmed. Every invalidated range ends up here, including ones outside any large binding, so a long session may drift into writing everything back with only one log line saying so. Two suggestions: filter reads through followed() in memory_note_cpu_access, as writes already are, and please report the counts from a long run.

> [@hckrman101](https://github.com/hckrman101) commented on `gpu/vk/wb_learn.h` line 130 on GitHub · 2026-11-12 10:52 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2731#discussion_r4549432920) Once `full_` is set it never clears, and neither `seen_` nor `read_` is ever trimmed. Every invalidated range ends up here, including ones outside any large binding, so a long session may drift into writing everything back with only one log line saying so. Two suggestions: filter reads through `followed()` in `memory_note_cpu_access`, as writes already are, and please report the counts from a long run.

github-sync commented

2026-11-12 11:36:25 +00:00

Author

Owner

@hckrman101 commented on gpu/vk/vk_memory.cpp line 2634 on GitHub · 2026-11-12 10:52 UTC · original

This takes wb_learn_mu and runs followed()'s backward walk over seen_ (up to 1 GB below) for every dirty page of every synced pool. Meanwhile the replay threads' binds wait on the same mutex in wb_leave_out. It might be worth checking once per pool, before the page loop, whether [p.lo, p.hi) overlaps anything followed, and skipping the per-page call otherwise.

> [@hckrman101](https://github.com/hckrman101) commented on `gpu/vk/vk_memory.cpp` line 2634 on GitHub · 2026-11-12 10:52 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2731#discussion_r4549432927) This takes `wb_learn_mu` and runs `followed()`'s backward walk over `seen_` (up to 1 GB below) for every dirty page of every synced pool. Meanwhile the replay threads' binds wait on the same mutex in `wb_leave_out`. It might be worth checking once per pool, before the page loop, whether `[p.lo, p.hi)` overlaps anything followed, and skipping the per-page call otherwise.

github-sync commented

2026-11-12 11:36:25 +00:00

Author

Owner

@hckrman101 commented on gpu/vk/vk_memory.cpp line 2734 on GitHub · 2026-11-12 10:52 UTC · original

When stale is true, the CPU is about to read (or the sync is about to copy) a range whose last write-back was left out. The mirror still has the GPU's data, since gpu_written keeps it. Could the read path queue that range's write-back and wait for it here, instead of only logging? That would turn the first-touch case from wrong data once into a slow read once.

> [@hckrman101](https://github.com/hckrman101) commented on `gpu/vk/vk_memory.cpp` line 2734 on GitHub · 2026-11-12 10:52 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2731#discussion_r4549432935) When `stale` is true, the CPU is about to read (or the sync is about to copy) a range whose last write-back was left out. The mirror still has the GPU's data, since `gpu_written` keeps it. Could the read path queue that range's write-back and wait for it here, instead of only logging? That would turn the first-touch case from wrong data once into a slow read once.

github-sync commented

2026-11-12 11:36:25 +00:00

Author

Owner

@hckrman101 commented on gpu/vk/vk_memory.cpp line 3158 on GitHub · 2026-11-12 10:52 UTC · original

Nit: 57 -> 61 fps is the plain cap's result in the description. This PR measured about 58.8 -> 60.0 median and 57.2 -> 58.6 mean in Hyrule Field, so I'd quote that, or just the 68 -> 18 MB a frame.

> [@hckrman101](https://github.com/hckrman101) commented on `gpu/vk/vk_memory.cpp` line 3158 on GitHub · 2026-11-12 10:52 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2731#discussion_r4549432948) Nit: 57 -> 61 fps is the plain cap's result in the description. This PR measured about 58.8 -> 60.0 median and 57.2 -> 58.6 mean in Hyrule Field, so I'd quote that, or just the 68 -> 18 MB a frame.

github-sync commented

2026-11-12 11:36:25 +00:00

Author

Owner

@hckrman101 reviewed on GitHub · 2026-11-12 10:52 UTC · original

Thanks, this is a careful follow-up to #384's experiment: the plain cap's two hazards are clearly explained, and learning what the CPU touches, rather than guessing from size, is the right shape. wb_learn.h is small, has no Vulkan in it, and is unit-tested, which makes it easy to review. CI is green.

What I checked and found sound:

Ordering in invalidate. The read is noted before the write-back wait. So work that is recorded during the catch-up / sync_settle wait already keeps the range, and test_wb_learn.py pins that order. Ranges the CPU reads keep their write-back, so memory_writeback_value still finds them and the readback wait (exposure meter, PR 430's refresh) is unchanged for them.
CPU writes in mirror_sync. They are noted before the page copy, and at page granularity, so a page the CPU writes during probation keeps the GPU's rest of the page from then on. That is the flicker case from the whole-page sync.
Learned state. It is keyed by guest range, not by pool, so retiring or re-importing a pool doesn't lose it. A binding at the same start with a different size starts a new probation. gpu_written already keeps these pools' mirrors from being given back, so the left-out data isn't dropped by mirrors_give_back.
Locking. wb_learn_mu is always taken last: after pools_mu from the bind path and mirror_sync, and on its own from invalidate. I didn't find a lock-order problem with the parallel replay or the readback refresh.
Off switch. RX_VK_WB_MAX_KB=0 restores the old behaviour exactly.

Points I'd like addressed before this becomes the default (details inline):

Probation counts bind calls, not submissions. Written binds aren't memoized, so a buffer that several dispatches write in one frame uses up its 8 writes in a frame or two. The CPU's first read of that buffer can come after the replay has already recorded the next frame. In that case the read finds stale data, which is exactly the case probation is meant to cover. Counting at most one write per submission would make RX_VK_WB_PROBATION=8 mean what the README says.
First touch is stale by design. A range the CPU first reads or writes after probation gets the old copy once. The logs in the description show this is rare in these scenes, but it is silent wrong data when it happens, and it's on by default. Could the late touch be repaired instead of only logged? For example, invalidate could queue that range's write-back from the mirror, which still holds the GPU's data, and wait for it as it does for kept ranges. If that's not practical, please at least say in the README that an occasional one-off stale read is the trade-off.
The table only grows. seen_ and read_ are never pruned, and every invalidated range goes into read_, including small readbacks that are in no followed binding. Once either table reaches 8192 entries, full_ latches and the session silently falls back to writing everything back. Please report the bindings followed / ranges the CPU read counts from the log after a long session (30+ min, several scene changes, a save reload). Filtering reads through followed(), as writes already are, would also help.
Per-page cost in mirror_sync. Every dirty page now takes wb_learn_mu and walks seen_ back up to 1 GB, and that work runs while the replay threads plan their binds. The walk is cheap while there are few followed bindings, but it grows with them (see 3). Checking once per pool whether it overlaps any followed binding would keep this off the common path.

Measurements (our rule from #2712). Thanks for the before/after table: it covers the same scenes, alternates the arms, names the GPU and driver, and includes the late-touch counts. That's what we ask for. For a new default I'd still like:

A second machine, ideally Linux and/or AMD, and the slower-PCIe case the description expects to gain most (laptop or x8). As measured, the gain is about 2% on the 3060 Ti in Hyrule Field and Kokiri Forest, and the Market is noise-bound, as you note. The default needs to stand on more than one NVIDIA/Windows box.
The screenshot pairs attached. The description says they match. Please also say explicitly that no white flashes, exposure steps or fog from the previous frame showed up during the runs, since those are the past symptoms of stale write-backs.

Nit: the comment in gpu_buffer_bind quotes "57 -> 61 fps and no frame over 50 ms". 61 fps is the plain cap's figure in the description, and this PR measured about 58.8 -> 60.0 median, so I'd quote this PR's own numbers.

No overlap with #2338 or #1377 that would block this, but both edit the same writebacks.push_back block in gpu_buffer_bind. #1377 marks a binding over the cap gpu_only. Whichever lands second needs a small rebase there, and with this PR "not written back" becomes partial rather than whole.

> [@hckrman101](https://github.com/hckrman101) reviewed on GitHub · 2026-11-12 10:52 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2731#pullrequestreview-5873123101) Thanks, this is a careful follow-up to #384's experiment: the plain cap's two hazards are clearly explained, and learning what the CPU touches, rather than guessing from size, is the right shape. `wb_learn.h` is small, has no Vulkan in it, and is unit-tested, which makes it easy to review. CI is green. What I checked and found sound: - **Ordering in `invalidate`.** The read is noted before the write-back wait. So work that is recorded during the catch-up / `sync_settle` wait already keeps the range, and `test_wb_learn.py` pins that order. Ranges the CPU reads keep their write-back, so `memory_writeback_value` still finds them and the readback wait (exposure meter, PR 430's refresh) is unchanged for them. - **CPU writes in `mirror_sync`.** They are noted before the page copy, and at page granularity, so a page the CPU writes during probation keeps the GPU's rest of the page from then on. That is the flicker case from the whole-page sync. - **Learned state.** It is keyed by guest range, not by pool, so retiring or re-importing a pool doesn't lose it. A binding at the same start with a different size starts a new probation. `gpu_written` already keeps these pools' mirrors from being given back, so the left-out data isn't dropped by `mirrors_give_back`. - **Locking.** `wb_learn_mu` is always taken last: after `pools_mu` from the bind path and `mirror_sync`, and on its own from `invalidate`. I didn't find a lock-order problem with the parallel replay or the readback refresh. - **Off switch.** `RX_VK_WB_MAX_KB=0` restores the old behaviour exactly. Points I'd like addressed before this becomes the default (details inline): 1. **Probation counts bind calls, not submissions.** Written binds aren't memoized, so a buffer that several dispatches write in one frame uses up its 8 writes in a frame or two. The CPU's first read of that buffer can come after the replay has already recorded the next frame. In that case the read finds stale data, which is exactly the case probation is meant to cover. Counting at most one write per submission would make `RX_VK_WB_PROBATION=8` mean what the README says. 2. **First touch is stale by design.** A range the CPU first reads or writes after probation gets the old copy once. The logs in the description show this is rare in these scenes, but it is silent wrong data when it happens, and it's on by default. Could the late touch be repaired instead of only logged? For example, `invalidate` could queue that range's write-back from the mirror, which still holds the GPU's data, and wait for it as it does for kept ranges. If that's not practical, please at least say in the README that an occasional one-off stale read is the trade-off. 3. **The table only grows.** `seen_` and `read_` are never pruned, and every invalidated range goes into `read_`, including small readbacks that are in no followed binding. Once either table reaches 8192 entries, `full_` latches and the session silently falls back to writing everything back. Please report the `bindings followed` / `ranges the CPU read` counts from the log after a long session (30+ min, several scene changes, a save reload). Filtering reads through `followed()`, as writes already are, would also help. 4. **Per-page cost in `mirror_sync`.** Every dirty page now takes `wb_learn_mu` and walks `seen_` back up to 1 GB, and that work runs while the replay threads plan their binds. The walk is cheap while there are few followed bindings, but it grows with them (see 3). Checking once per pool whether it overlaps any followed binding would keep this off the common path. **Measurements (our rule from #2712).** Thanks for the before/after table: it covers the same scenes, alternates the arms, names the GPU and driver, and includes the late-touch counts. That's what we ask for. For a new default I'd still like: - **A second machine**, ideally Linux and/or AMD, and the slower-PCIe case the description expects to gain most (laptop or x8). As measured, the gain is about 2% on the 3060 Ti in Hyrule Field and Kokiri Forest, and the Market is noise-bound, as you note. The default needs to stand on more than one NVIDIA/Windows box. - **The screenshot pairs attached.** The description says they match. Please also say explicitly that no white flashes, exposure steps or fog from the previous frame showed up during the runs, since those are the past symptoms of stale write-backs. Nit: the comment in `gpu_buffer_bind` quotes "57 -> 61 fps and no frame over 50 ms". 61 fps is the plain cap's figure in the description, and this PR measured about 58.8 -> 60.0 median, so I'd quote this PR's own numbers. No overlap with #2338 or #1377 that would block this, but both edit the same `writebacks.push_back` block in `gpu_buffer_bind`. #1377 marks a binding over the cap `gpu_only`. Whichever lands second needs a small rebase there, and with this PR "not written back" becomes partial rather than whole.

	written back a frame	fps median	fps mean	GPU a frame	late touches
`RX_VK_WB_MAX_KB=0`	68.2 / 68.0 MB	58.4 / 59.2	57.5 / 57.5	12.4 ms	-
this PR	18.0 / 18.0 MB	58.2 / 59.9	56.6 / 58.4	12.2 ms	0 / 0

scene	main (3)	PR (3)	PR `RX_VK_WB_MAX_KB=0` (1)
Kokiri	21.8 / 21.7 / 21.9 → 21.8; 8.7	21.7 / 22.0 / 21.8 → 21.8; 8.7	21.8; 8.1
Hyrule Field	30.6 / 30.4 / 32.4 → 30.6; 3.8	30.6 / 30.9 / 31.2 → 30.9; 4.0	31.6; 3.9
Market	39.9 / 39.5 / 40.9 → 39.9; 3.4	34.6 / 39.2 / 39.4 → 39.2; 3.7	36.2; 3.6

scene	main (3)	PR (3)	PR `RX_VK_WB_MAX_KB=0` (3)	written back a frame, main → PR
Kokiri	28.5 / 25.0 / 27.5 → 27.5; 9.2	28.1 / 28.4 / 27.7 → 28.1; 9.0	28.1 / 27.6 / 28.5 → 28.1; 9.0	67.1 → 17.1 MB
Hyrule Field	45.6 / 46.0 / 45.7 → 45.7; 6.5	38.0 / 43.8 / 43.3 → 43.3; 5.5	45.2 / 46.3 / 44.8 → 45.2; 5.1	68.2 → 18.0 MB
Market	40.4 / 41.0 / 39.4 → 40.4; 4.2	24.0 / 40.7 / 44.8 → 40.7; 4.0	38.5 / 33.7 / 30.4 → 33.7; 3.8	61.1 → 17.5 MB

`RX_VK_WB_MAX_KB`	Hyrule Field median fps (min-max), each run
0 (as before)	34.9 (26.1-37.9), 33.0 (28.1-36.0)
1024	33.0 (19.5-40.0), 33.3 (27.4-37.0)

Run	Arm	MB written back/frame	FPS	CPU ms/frame	1% low	Frames >20 / >25 ms	Worst frame ms	Load start→end
1	main `ba005ca4`	68.10	62.23	45.88	3.44	53 / 47	511.18	15.75→11.98
2	`db1359af`, default	67.91	69.07	46.89	3.86	38 / 32	624.40	11.98→8.75
3	`db1359af`, 1024	18.13	92.99	44.74	54.02	1 / 1	25.09	8.75→7.55
4	main `ba005ca4`	67.89	91.60	45.20	54.25	2 / 1	25.73	7.55→8.24
5	`db1359af`, default	67.97	91.64	44.99	51.00	4 / 1	27.46	8.24→8.75
6	`db1359af`, 1024	18.15	93.50	44.12	51.87	5 / 0	24.61	8.75→7.54

Rows
Columns

perf(vk): RX_VK_WB_MAX_KB writes large GPU-written bindings back only where the CPU reads or writes them (opt-in) #2731

What

Measured

Tests

Three arms, alternated

Correctness and late touches

Build and game-free checks