MagicBeanDev/Hookshot

Archived

Watch

Fork 0

perf(vk): prevent VRAM paging surges across fast scene transitions #2467

Closed

Erfboom wants to merge 1 commit from perf/vk-vram-surge-relief into main

Erfboom commented

2026-11-11 20:24:38 +00:00

(Migrated from github.com)

When moving quickly between areas (e.g. Kokiri Forest <-> Hyrule Field <-> Market within < 60 frames), min_unused = 60 prevented eviction of newly idle textures from the previous scene while streaming in hundreds of megabytes of new textures. This caused texture allocations to surge past 170% of budget, pushing total GPU allocation past 6-8 GB VRAM capacity and causing severe 60-70 ms PCIe driver paging hitches.

Changes

Eviction Gate Bypass: If total_bytes surges over own + (own >> 3) (> 112.5% of texture budget), bypass the 10-frame gate in texture_frame to immediately initiate cache rebalancing.
Surge Relief Pass: When cache exceeds 112.5% of budget or when render target pool memory is pressed, immediately evict CPU-streamed non-GPU-written textures unused for >= 8 frames back down to budget, without waiting for the full 60-frame aging cycle.
Budget Clamps on <= 6.5 GB Devices:
- pool_cap(): Default to 128 MB (was 256 MB) on devices with <= 6.5 GB allowed VRAM.
- keep_finalized_max: Default to 192 MB (was 512 MB) on devices with <= 6.5 GB allowed VRAM.
Environment Controls: Guard partial texture upload logging behind env_on("RX_VK_LOG_PARTIAL") to avoid spam during bursts of texture uploads.

When moving quickly between areas (e.g. Kokiri Forest <-> Hyrule Field <-> Market within < 60 frames), `min_unused = 60` prevented eviction of newly idle textures from the previous scene while streaming in hundreds of megabytes of new textures. This caused texture allocations to surge past 170% of budget, pushing total GPU allocation past 6-8 GB VRAM capacity and causing severe 60-70 ms PCIe driver paging hitches. ### Changes 1. **Eviction Gate Bypass**: If `total_bytes` surges over `own + (own >> 3)` (> 112.5% of texture budget), bypass the 10-frame gate in `texture_frame` to immediately initiate cache rebalancing. 2. **Surge Relief Pass**: When cache exceeds 112.5% of budget or when render target pool memory is pressed, immediately evict CPU-streamed non-GPU-written textures unused for >= 8 frames back down to budget, without waiting for the full 60-frame aging cycle. 3. **Budget Clamps on <= 6.5 GB Devices**: - `pool_cap()`: Default to 128 MB (was 256 MB) on devices with <= 6.5 GB allowed VRAM. - `keep_finalized_max`: Default to 192 MB (was 512 MB) on devices with <= 6.5 GB allowed VRAM. 4. **Environment Controls**: Guard partial texture upload logging behind `env_on("RX_VK_LOG_PARTIAL")` to avoid spam during bursts of texture uploads.

👍 1

Joey-Magnus (Migrated from github.com) reviewed

2026-11-11 21:26:49 +00:00

Joey-Magnus (Migrated from github.com) left a comment

The every-10-frames gate is bypassed whenever textures exceed 112.5% of own (vk_texture.cpp:3875). If the excess can't be evicted (mostly GPU-written textures), the full budget pass runs every frame and never stops.
The relief pass (:4262) evicts CPU-uploaded textures unused for only 8 frames and skips the compaction hold, so textures drawn every few frames can churn.
On 6.5 GB or smaller cards, pool_cap drops from 256 to 128 MB (:290) and keep_finalized_max from 512 to 192 MB (:4131). The finalized keep exists for the rainbow/garbage-texture issues #107 and #386, so this risks regressing exactly those cards.
test_target_image_pool.py was loosened rather than updated.
The RX_VK_LOG_PARTIAL change is unrelated.
This rewrites the same frame_no % 10 gate as #2484.

- The every-10-frames gate is bypassed whenever textures exceed 112.5% of `own` (`vk_texture.cpp:3875`). If the excess can't be evicted (mostly GPU-written textures), the full budget pass runs every frame and never stops. - The relief pass (`:4262`) evicts CPU-uploaded textures unused for only 8 frames and skips the compaction hold, so textures drawn every few frames can churn. - On 6.5 GB or smaller cards, `pool_cap` drops from 256 to 128 MB (`:290`) and `keep_finalized_max` from 512 to 192 MB (`:4131`). The finalized keep exists for the rainbow/garbage-texture issues #107 and #386, so this risks regressing exactly those cards. - `test_target_image_pool.py` was loosened rather than updated. - The `RX_VK_LOG_PARTIAL` change is unrelated. - This rewrites the same `frame_no % 10` gate as #2484.

hckrman101 (Migrated from github.com) reviewed

2026-11-11 22:20:48 +00:00

hckrman101 (Migrated from github.com) left a comment

No numbers yet. The description talks about allocation at 170% of budget and 60-70 ms paging hitches, but gives no log, frame times or GPU. A before/after on a 6-8 GB card (the vk: ... textures retired lines plus frame times for the Kokiri -> Field -> Market run) would show whether the gate, the relief pass or the two caps actually fixes it. Then the risky parts could be dropped. This changes behaviour on cards people actually play on, so it needs an in-game report.
The relief pass works against the existing 60-frame rule. The comment above min_unused explains why it is 60: with a shorter age, about 300 textures in a few minutes of Kokiri were evicted and made again, 200 of them within 10 frames. Evicting CPU-uploaded textures after 8 frames brings back that pattern of re-uploads and re-decodes, just limited to the surge. See inline.
An every-frame pass costs more than it looks. Each pass sorts every texture, calls vram_usage, and does the write-back batches, each of which is a submission plus a wait. Running it every present during a surge can put those waits on consecutive frames, in the very window this PR is trying to smooth out.
Nit: the comment above keep_finalized_max still says "512 by default".
CI: no checks have run on this head.

About your other open PRs: #2469 also edits vk_texture.cpp's staging and texture_frame code, but in a different hunk, so I expect no textual conflict. #2484 rewrites the same frame_no % 10 gate and the same line in test_target_image_pool.py, so one of the two will need a rebase. That one is easier to reason about, since it only stays open while dead targets pile up.

My suggestion: split this up. The first PR would be the gate and/or a relief pass that respects min_unused, with measurements. Keep the default caps out unless a log shows they matter. For now this is a comment rather than an approval.

Thanks for going after the scene-change hitches. I agree with all of Joey-Magnus's points above (the gate that can stay open every frame, the 8-frame relief pass, the smaller finalized keep on cards up to 6.5 GB and the risk to #107/#386, the loosened test, the unrelated logging change, and the clash with #2484). I won't go over them again. A few things to add: - **No numbers yet.** The description talks about allocation at 170% of budget and 60-70 ms paging hitches, but gives no log, frame times or GPU. A before/after on a 6-8 GB card (the `vk: ... textures retired` lines plus frame times for the Kokiri -> Field -> Market run) would show whether the gate, the relief pass or the two caps actually fixes it. Then the risky parts could be dropped. This changes behaviour on cards people actually play on, so it needs an in-game report. - **The relief pass works against the existing 60-frame rule.** The comment above `min_unused` explains why it is 60: with a shorter age, about 300 textures in a few minutes of Kokiri were evicted and made again, 200 of them within 10 frames. Evicting CPU-uploaded textures after 8 frames brings back that pattern of re-uploads and re-decodes, just limited to the surge. See inline. - **An every-frame pass costs more than it looks.** Each pass sorts every texture, calls `vram_usage`, and does the write-back batches, each of which is a submission plus a wait. Running it every present during a surge can put those waits on consecutive frames, in the very window this PR is trying to smooth out. - Nit: the comment above `keep_finalized_max` still says "512 by default". - CI: no checks have run on this head. About your other open PRs: #2469 also edits `vk_texture.cpp`'s staging and `texture_frame` code, but in a different hunk, so I expect no textual conflict. #2484 rewrites the same `frame_no % 10` gate and the same line in `test_target_image_pool.py`, so one of the two will need a rebase. That one is easier to reason about, since it only stays open while dead targets pile up. My suggestion: split this up. The first PR would be the gate and/or a relief pass that respects `min_unused`, with measurements. Keep the default caps out unless a log shows they matter. For now this is a comment rather than an approval.

gpu/vk/vk_texture.cpp

This is measured against own, while the relief pass below is measured against budget (which a preset, the 90% rule or the compaction hold can make smaller). If the surge is over budget but not own, the relief only runs on every tenth present. And as Joey said, if the excess is textures this pass won't evict, it runs every present indefinitely. If you keep the bypass, a hysteresis or a cap (e.g. at most N extra passes after a gate trigger) would help.

This is measured against `own`, while the relief pass below is measured against `budget` (which a preset, the 90% rule or the compaction hold can make smaller). If the surge is over `budget` but not `own`, the relief only runs on every tenth present. And as Joey said, if the excess is textures this pass won't evict, it runs every present indefinitely. If you keep the bypass, a hysteresis or a cap (e.g. at most N extra passes after a gate trigger) would help.

gpu/vk/vk_texture.cpp

Nit: stale now. The default is 192 MB on cards of 6.5 GB or less.

gpu/vk/vk_texture.cpp

8 frames is far below min_unused (60), and the comment on min_unused describes exactly the churn this causes: 200 textures made again within 10 frames. A texture the game samples every few frames can be evicted and uploaded/decoded again on every surge pass. Could this at least respect RX_VK_EVICT_MIN_UNUSED, or use a much larger floor than 8?

8 frames is far below `min_unused` (60), and the comment on `min_unused` describes exactly the churn this causes: 200 textures made again within 10 frames. A texture the game samples every few frames can be evicted and uploaded/decoded again on every surge pass. Could this at least respect `RX_VK_EVICT_MIN_UNUSED`, or use a much larger floor than 8?

DFN-master commented

2026-11-11 23:29:12 +00:00

(Migrated from github.com)

I tested this on Windows 11, RTX 3080 10 GB, i7-11700, 32 GB RAM, 1920x1080. The base is my fork build (4a53c897 + zstd/BC7), texture budget 4 GB, RX_VK_COMPACT_PCT=90, with this PR cherry-picked onto it. Same exe, caches and save for both sides; the DLL is swapped between runs. Route: scripted fast teleports Kakariko -> Hyrule Field -> Lake Hylia every 10 s from 100 s to 200 s, 240 s per run; the same fixed save is restored before each run.

	base	this PR
driver's 9 GB allowed
fps, whole run (avg)	45.39	45.06
fps, transitions (presents 6000-12000, avg)	32.56	29.04
slow-submit `memory sync`, max	263.4 ms	257.5 ms
VRAM peak	9547 MB	8895 MB
textures retired, total	12,596	15,419
simulated 6.5 GB (`RX_VK_TEST_ALLOWED_MB=6500`)
fps, whole run (avg)	50.24	48.62
fps, transitions (avg)	41.09	15.08
slow-submit `memory sync`, max	253.1 ms	253.4 ms
VRAM peak	6850 MB	6782 MB
textures retired, total	8,605	8,225

The PR does lower the VRAM peak (and slightly lowers the retired count at 6.5 GB), which matches its goal. In my runs it did not improve the frame rate; at 6.5 GB the transition window had a long low stretch with this PR (the sample counts differ because the two runs took different times to reach the same present range).

Caveats: the PR's own base is 6842f21d, while this test is on 4a53c897 + zstd/BC7; RX_VK_TEST_ALLOWED_MB simulates the allowed-VRAM value and the eviction policy, not actual PCIe paging; one pair per configuration, and the transition route is noisy on this machine. If useful, I can repeat it on a clean latest-main build, or with a 6-8 GB card's actual paging, and with more interleaved runs. I have not changed the PR.

I tested this on Windows 11, RTX 3080 10 GB, i7-11700, 32 GB RAM, 1920x1080. The base is my fork build (`4a53c897` + zstd/BC7), texture budget 4 GB, `RX_VK_COMPACT_PCT=90`, with this PR cherry-picked onto it. Same exe, caches and save for both sides; the DLL is swapped between runs. Route: scripted fast teleports Kakariko -> Hyrule Field -> Lake Hylia every 10 s from 100 s to 200 s, 240 s per run; the same fixed save is restored before each run. | | base | this PR | |---|---:|---:| | **driver's 9 GB allowed** | | | | fps, whole run (avg) | 45.39 | 45.06 | | fps, transitions (presents 6000-12000, avg) | 32.56 | 29.04 | | slow-submit `memory sync`, max | 263.4 ms | 257.5 ms | | VRAM peak | 9547 MB | **8895 MB** | | textures retired, total | 12,596 | 15,419 | | **simulated 6.5 GB (`RX_VK_TEST_ALLOWED_MB=6500`)** | | | | fps, whole run (avg) | 50.24 | 48.62 | | fps, transitions (avg) | 41.09 | 15.08 | | slow-submit `memory sync`, max | 253.1 ms | 253.4 ms | | VRAM peak | 6850 MB | **6782 MB** | | textures retired, total | 8,605 | 8,225 | The PR does lower the VRAM peak (and slightly lowers the retired count at 6.5 GB), which matches its goal. In my runs it did not improve the frame rate; at 6.5 GB the transition window had a long low stretch with this PR (the sample counts differ because the two runs took different times to reach the same present range). Caveats: the PR's own base is `6842f21d`, while this test is on `4a53c897` + zstd/BC7; `RX_VK_TEST_ALLOWED_MB` simulates the allowed-VRAM value and the eviction policy, not actual PCIe paging; one pair per configuration, and the transition route is noisy on this machine. If useful, I can repeat it on a clean latest-main build, or with a 6-8 GB card's actual paging, and with more interleaved runs. I have not changed the PR.

DFN-master commented

2026-11-11 23:43:35 +00:00

(Migrated from github.com)

For reference, the exact machine and test setup behind the table above:

Windows 11, 64-bit; Intel Core i7-11700, 8C/16T; 32 GB RAM; NVIDIA GeForce RTX 3080 10 GB; 1920x1080 @ 60 Hz.
Test build: 4a53c897 + zstd/BC7 (v2.0.11-1024-g9bc83387), texture budget 4 GB, RX_VK_COMPACT_PCT=90, RX_VK_READBACK_NO_CATCHUP=1. The PR was cherry-picked onto that base for the test; the same exe, caches and fixed save were used for both sides.
Route: headless, 240 s, fast teleports Kakariko -> Hyrule Field -> Lake Hylia every 10 s from 100 s to 200 s; the same save is restored before each run.

The result agrees with the review concerns above: this PR lowered the VRAM peak (9.55 -> 8.90 GB with the driver's 9 GB allowed; 6.85 -> 6.78 GB with RX_VK_TEST_ALLOWED_MB=6500), but it did not improve the frame rate in these runs, and at 6.5 GB the transition window was worse (the relief pass/8-frame churn and the every-frame gate are the likely reasons). I did not change the PR; these are only measurements, with the caveats already noted (different base, simulated allowed VRAM, one pair per configuration).

For reference, the exact machine and test setup behind the table above: - Windows 11, 64-bit; Intel Core i7-11700, 8C/16T; 32 GB RAM; NVIDIA GeForce RTX 3080 10 GB; 1920x1080 @ 60 Hz. - Test build: `4a53c897` + zstd/BC7 (`v2.0.11-1024-g9bc83387`), texture budget 4 GB, `RX_VK_COMPACT_PCT=90`, `RX_VK_READBACK_NO_CATCHUP=1`. The PR was cherry-picked onto that base for the test; the same exe, caches and fixed save were used for both sides. - Route: headless, 240 s, fast teleports Kakariko -> Hyrule Field -> Lake Hylia every 10 s from 100 s to 200 s; the same save is restored before each run. The result agrees with the review concerns above: this PR lowered the VRAM peak (9.55 -> 8.90 GB with the driver's 9 GB allowed; 6.85 -> 6.78 GB with `RX_VK_TEST_ALLOWED_MB=6500`), but it did not improve the frame rate in these runs, and at 6.5 GB the transition window was worse (the relief pass/8-frame churn and the every-frame gate are the likely reasons). I did not change the PR; these are only measurements, with the caveats already noted (different base, simulated allowed VRAM, one pair per configuration).

hckrman101 (Migrated from github.com) reviewed

2026-11-11 23:58:42 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks @DFN-master for the runs. That is the before/after I asked for, though on a 10 GB card with a simulated 6.5 GB limit instead of a real 6-8 GB card.

What they show: the VRAM peak goes down (9.55 -> 8.90 GB; 6.85 -> 6.78 GB with RX_VK_TEST_ALLOWED_MB=6500), but the frame rate doesn't get better. With 6.5 GB simulated, the transition window drops from 41 to 15 fps. The memory sync slow-submit maximum stays the same on both sides (~253-263 ms), so the stall this PR is aimed at doesn't shrink either.

The simulated limit only changes what vram_usage reports. It doesn't page anything. So the 41 -> 15 fps drop at 6.5 GB can't come from PCIe paging: it comes from the new policy itself, on the host side. That means the budget pass now runs on every present while textures are over 112.5% of own (each pass sorts every texture, queries the budget and does write-back submissions with a CPU wait each), plus whatever the 8-frame relief pass and the smaller pool/finalized caps make again. The textures retired count is a little lower at 6.5 GB, so the extra cost seems to be the passes running more often rather than more evictions. A vk: slow submit phase breakdown from the two 6.5 GB runs would confirm which part it is.

The code is unchanged since my last review (still 956ec66), so all the earlier points are still open: the gate compares against own and the relief against budget, the gate can stay open every frame, the 8-frame age goes under min_unused, the caps are lower on cards of 6.5 GB or less (the risk to #107/#386), the test was loosened, the logging change is unrelated, there's the clash with #2484, and the stale "512 by default" comment. CI still hasn't run on this head.

Given these numbers, I'd still split it: a gate/relief change that respects min_unused and the 10-frame cadence, measured on a real 6-8 GB card, and no default cap changes unless a log shows they help. Still a comment, not an approval.

Thanks @DFN-master for the runs. That is the before/after I asked for, though on a 10 GB card with a simulated 6.5 GB limit instead of a real 6-8 GB card. What they show: the VRAM peak goes down (9.55 -> 8.90 GB; 6.85 -> 6.78 GB with `RX_VK_TEST_ALLOWED_MB=6500`), but the frame rate doesn't get better. With 6.5 GB simulated, the transition window drops from 41 to 15 fps. The `memory sync` slow-submit maximum stays the same on both sides (~253-263 ms), so the stall this PR is aimed at doesn't shrink either. The simulated limit only changes what `vram_usage` reports. It doesn't page anything. So the 41 -> 15 fps drop at 6.5 GB can't come from PCIe paging: it comes from the new policy itself, on the host side. That means the budget pass now runs on every present while textures are over 112.5% of `own` (each pass sorts every texture, queries the budget and does write-back submissions with a CPU wait each), plus whatever the 8-frame relief pass and the smaller pool/finalized caps make again. The textures retired count is a little lower at 6.5 GB, so the extra cost seems to be the passes running more often rather than more evictions. A `vk: slow submit` phase breakdown from the two 6.5 GB runs would confirm which part it is. The code is unchanged since my last review (still 956ec66), so all the earlier points are still open: the gate compares against `own` and the relief against `budget`, the gate can stay open every frame, the 8-frame age goes under `min_unused`, the caps are lower on cards of 6.5 GB or less (the risk to #107/#386), the test was loosened, the logging change is unrelated, there's the clash with #2484, and the stale "512 by default" comment. CI still hasn't run on this head. Given these numbers, I'd still split it: a gate/relief change that respects `min_unused` and the 10-frame cadence, measured on a real 6-8 GB card, and no default cap changes unless a log shows they help. Still a comment, not an approval.

Erfboom commented

2026-11-12 03:47:02 +00:00

(Migrated from github.com)

Thank you @Joey-Magnus, @hckrman101, and especially @DFN-master for taking the time to test and provide concrete measurements.

The benchmark data across fast area transitions (Kakariko → Hyrule Field → Lake Hylia) with a 6.5 GB limit clearly demonstrates the issue: while peak VRAM was slightly reduced, running the budget pass on every frame past 112.5% caused heavy host-side submission stalls and the 8-frame relief pass induced re-upload churn, dropping transition performance from 41 FPS down to 15 FPS. On lower-spec hardware (6 GB VRAM laptops, Steam Deck), this is a noticeable regression rather than a performance win.

Given these findings, along with the overlap with #2484 and the potential risk to #107/#386 from lowering the finalized keep caps, I agree with closing this PR. Thanks again for the thorough review and testing!

Thank you @Joey-Magnus, @hckrman101, and especially @DFN-master for taking the time to test and provide concrete measurements. The benchmark data across fast area transitions (Kakariko → Hyrule Field → Lake Hylia) with a 6.5 GB limit clearly demonstrates the issue: while peak VRAM was slightly reduced, running the budget pass on every frame past 112.5% caused heavy host-side submission stalls and the 8-frame relief pass induced re-upload churn, dropping transition performance from 41 FPS down to 15 FPS. On lower-spec hardware (6 GB VRAM laptops, Steam Deck), this is a noticeable regression rather than a performance win. Given these findings, along with the overlap with #2484 and the potential risk to #107/#386 from lowering the finalized keep caps, I agree with closing this PR. Thanks again for the thorough review and testing!

Commenting is not possible because the repository is archived.

No reviewers