fix(vk): rainbow block-noise textures on low-VRAM cards: a failed BC3 upload is tried again, not drawn as noise #1505

Merged

rakefire-impala merged 1 commit from fix/rainbow-textures-retry-failed-bc3-upload into main

2026-11-10 14:47:37 +00:00

rakefire-impala commented

2026-11-10 05:43:13 +00:00

(Migrated from github.com)

On a low-VRAM card, a BC3 texture whose upload failed was drawn as coloured block noise and stayed that way until the area was left. This makes a failed upload get tried again instead.

Related: #993 (garbled / rainbow textures, VRAM over budget, 4-8 GB cards). Part of splitting up #1023, as asked there.

Problem

ASTC textures are decoded on the GPU and re-encoded to BC3. That needs staging memory for the ASTC blocks and a BC3 scratch (an image plus an output buffer, 160-288 MB), which is given back after 20 s unused. On a 4 GB RX 5500M, after an alt-tab or a scene change:

With the game in the background the driver gave the process less VRAM. The texture budget shrank, and about a thousand textures were evicted in seconds.
Made again when drawn, their BC3 uploads failed. The scratch's output buffer could only be made in VRAM, within the budget (create_device_buffer), and VRAM was full.
A failed upload was marked uploaded anyway, so it wouldn't be tried every draw. The BC3 image, whose encode never ran, was then drawn with whatever its memory held, and was never made again unless the game rewrote its memory, which static textures never are.

The result was paths, trees, the Kokiri's clothes and the minimap turning to rainbow block noise.

Change

A BC3 texture whose upload failed stays not uploaded and is tried again 30 frames later (Texture::bc_retry_frame).
The cause is logged once per texture: vk: BC3 texture #... its upload failed (no staging memory for its ASTC blocks): tried again in 30 frames. Other causes are "no memory for the BC3 scratch", "its layout" and so on.
The BC3 scratch's output buffer falls back to system memory when VRAM is full, as the scratch image already did.

Between a failed upload and its retry, a texture can still show noise briefly; before this change it was permanent.

How to check

On a low-VRAM card, alt-tab out for a few seconds and come back, or change areas. Instead of noise that stays, any failure appears in the log as above and the texture recovers within a moment.

Risk

Only the failure path changes. A texture whose upload succeeds is untouched, so cards with enough VRAM see no difference.

Overlaps #1130 (draft), which spreads large BC3 uploads over several submissions and retries a failed continuation. This PR covers a failed first upload being marked uploaded. Both touch upload_astc_gpu's callers, so whichever lands second needs a small rebase, which I'm happy to do.

Tests

tests/test_bc_upload_retry.py (new) checks the retry, its spacing, the logged cause and the scratch fallback.
One check in tests/test_mem_trim.py now matches the call without its old one-line form; the check itself is unchanged.

Tested on


GPU	AMD Radeon RX 5500M (RDNA1, 4 GB VRAM), laptop
Driver	AMD Windows driver, Windows 11
Result	The noise after alt-tab went away (tested on an earlier base together with the texture-squeeze PR; rebased on current main, game-free tests pass)

More low-VRAM cards would help, especially AMD ones.

On a low-VRAM card, a BC3 texture whose upload failed was drawn as coloured block noise and stayed that way until the area was left. This makes a failed upload get tried again instead. Related: #993 (garbled / rainbow textures, VRAM over budget, 4-8 GB cards). Part of splitting up #1023, as asked there. ## Problem ASTC textures are decoded on the GPU and re-encoded to BC3. That needs staging memory for the ASTC blocks and a BC3 scratch (an image plus an output buffer, 160-288 MB), which is given back after 20 s unused. On a 4 GB RX 5500M, after an alt-tab or a scene change: - With the game in the background the driver gave the process less VRAM. The texture budget shrank, and about a thousand textures were evicted in seconds. - Made again when drawn, their BC3 uploads failed. The scratch's output buffer could only be made in VRAM, within the budget (`create_device_buffer`), and VRAM was full. - **A failed upload was marked uploaded anyway**, so it wouldn't be tried every draw. The BC3 image, whose encode never ran, was then drawn with whatever its memory held, and was never made again unless the game rewrote its memory, which static textures never are. The result was paths, trees, the Kokiri's clothes and the minimap turning to rainbow block noise. ## Change - A BC3 texture whose upload failed stays **not uploaded** and is tried again 30 frames later (`Texture::bc_retry_frame`). - The cause is logged once per texture: `vk: BC3 texture #... its upload failed (no staging memory for its ASTC blocks): tried again in 30 frames`. Other causes are "no memory for the BC3 scratch", "its layout" and so on. - The BC3 scratch's output buffer falls back to system memory when VRAM is full, as the scratch image already did. Between a failed upload and its retry, a texture can still show noise briefly; before this change it was permanent. ## How to check On a low-VRAM card, alt-tab out for a few seconds and come back, or change areas. Instead of noise that stays, any failure appears in the log as above and the texture recovers within a moment. ## Risk Only the failure path changes. A texture whose upload succeeds is untouched, so cards with enough VRAM see no difference. Overlaps #1130 (draft), which spreads large BC3 uploads over several submissions and retries a failed *continuation*. This PR covers a failed *first* upload being marked uploaded. Both touch `upload_astc_gpu`'s callers, so whichever lands second needs a small rebase, which I'm happy to do. ## Tests - `tests/test_bc_upload_retry.py` (new) checks the retry, its spacing, the logged cause and the scratch fallback. - One check in `tests/test_mem_trim.py` now matches the call without its old one-line form; the check itself is unchanged. ## Tested on | | | |---|---| | GPU | AMD Radeon RX 5500M (RDNA1, 4 GB VRAM), laptop | | Driver | AMD Windows driver, Windows 11 | | Result | The noise after alt-tab went away (tested on an earlier base together with the texture-squeeze PR; rebased on current main, game-free tests pass) | More low-VRAM cards would help, especially AMD ones.