vk: tree-shadow flicker evened out; push-descriptor cache stale after the stand-ins' pushes #477

Closed

imbahzor wants to merge 1 commit from fix/tree-shadow-flicker into main

imbahzor commented

2026-11-08 05:04:41 +00:00

(Migrated from github.com)

Tree-shadow flicker evened out; push-descriptor cache marked stale after the stand-ins' own pushes

1. Tree shadows flickering in fast thin streaks

Symptom: over sunlit ground and along shadow edges near trees, fast thin streaks flicker in every direction while the camera stands still (constant, not single bad frames).

Found by bisection (live skip lists, captures):

leaving out 3427a1dbefd5e02c (a full-screen draw) removes the flicker, and the canopy shadows with it
leaving out both programs that draw the 2400x2400 D32F shadow map it reads removes it too; either one alone does not
not the depth bias (even extreme values), not the filter (the game already asks for linear on its depth-compare samplers), no per-frame seed in its constants (only camera and sun drift)

Mechanism (from its IR): the pass reads the scene depth, reconstructs a normal from an ImageGather of the neighbouring depths and offsets the shadow lookup by it, then takes 9 fixed-offset PCF taps. On grass blades and leaf edges the gathered neighbours flip between the blade and the ground behind it with the sub-pixel camera shift, so those pixels flip between lit and shadowed every frame. On the console the TAAU network averages that away.

Change: right after that draw, its mask (G and B of an RGBA8 target at the scene's size, MIN blended) is blended with its own history in a small compute pass (shaders/shadow_stab.comp): two history images in turn, then G and B copied back.

RX_VK_MASK_BLEND=<w>: default 0.85, 0 turns it off
RX_VK_MASK_RADIUS=<r>: bound the history to a (2r+1)^2 neighbourhood of this frame's mask; default 0 (unbounded). A 3x3 bound kept nothing, because the streaks move as blocks wider than that.

With 0.85 and no bound: no flicker while standing still, and no visible smearing of the tree shadows when turning the camera or walking (tested in the Kokiri / Lost Woods area at several times of day).

2. Push-descriptor cache after the stand-ins' own pushes

taau_stand_in and the movie upscale stub push descriptors of their own layout with vkCmdPushDescriptorSetKHR, but bind_descriptors keeps a cache of what it pushed (pushed[1]). The next dispatch with the cached layout and unchanged writes skips its push and reads the stand-in's bindings. This showed up as a reproducible device lost right after a save load once another compute pass was inserted after the network; both stand-ins now mark the cache stale (pushed[1].cmd = VK_NULL_HANDLE), as the new pass does.

Testing

RTX 3060 12 GB, Windows 11, 16 GB RAM
Played and measured on the no-visual-studio release with these changes applied (plus unrelated local patches)
This branch is rebased onto current main and builds cleanly (RX_VK_ONLY=1); not yet played on main itself

On this RTX 3060, the gameplay TAAU network on the native cooperative-matrix path crushes dark tones: shadowed areas go black (issue #59). With RX_VK_MMA_SOFTWARE=1 and RX_VK_TAAU_KEYS=- (the network kept, its multiplies done in software) shadows are right and the picture keeps the network's sharpness, at about 32 fps here. Separate issue to follow.

🤖 Generated with Claude Code

## Tree-shadow flicker evened out; push-descriptor cache marked stale after the stand-ins' own pushes ### 1. Tree shadows flickering in fast thin streaks **Symptom:** over sunlit ground and along shadow edges near trees, fast thin streaks flicker in every direction while the camera stands still (constant, not single bad frames). **Found by bisection (live skip lists, captures):** - leaving out `3427a1dbefd5e02c` (a full-screen draw) removes the flicker, and the canopy shadows with it - leaving out both programs that draw the 2400x2400 D32F shadow map it reads removes it too; either one alone does not - not the depth bias (even extreme values), not the filter (the game already asks for linear on its depth-compare samplers), no per-frame seed in its constants (only camera and sun drift) **Mechanism (from its IR):** the pass reads the scene depth, reconstructs a normal from an `ImageGather` of the neighbouring depths and offsets the shadow lookup by it, then takes 9 fixed-offset PCF taps. On grass blades and leaf edges the gathered neighbours flip between the blade and the ground behind it with the sub-pixel camera shift, so those pixels flip between lit and shadowed every frame. On the console the TAAU network averages that away. **Change:** right after that draw, its mask (G and B of an RGBA8 target at the scene's size, MIN blended) is blended with its own history in a small compute pass (`shaders/shadow_stab.comp`): two history images in turn, then G and B copied back. - `RX_VK_MASK_BLEND=<w>`: default 0.85, `0` turns it off - `RX_VK_MASK_RADIUS=<r>`: bound the history to a (2r+1)^2 neighbourhood of this frame's mask; default 0 (unbounded). A 3x3 bound kept nothing, because the streaks move as blocks wider than that. With 0.85 and no bound: no flicker while standing still, and no visible smearing of the tree shadows when turning the camera or walking (tested in the Kokiri / Lost Woods area at several times of day). ### 2. Push-descriptor cache after the stand-ins' own pushes `taau_stand_in` and the movie upscale stub push descriptors of their own layout with `vkCmdPushDescriptorSetKHR`, but `bind_descriptors` keeps a cache of what it pushed (`pushed[1]`). The next dispatch with the cached layout and unchanged writes skips its push and reads the stand-in's bindings. This showed up as a reproducible device lost right after a save load once another compute pass was inserted after the network; both stand-ins now mark the cache stale (`pushed[1].cmd = VK_NULL_HANDLE`), as the new pass does. ### Testing - RTX 3060 12 GB, Windows 11, 16 GB RAM - Played and measured on the `no-visual-studio` release with these changes applied (plus unrelated local patches) - This branch is rebased onto current `main` and builds cleanly (`RX_VK_ONLY=1`); not yet played on `main` itself ### Related, not in this PR On this RTX 3060, the gameplay TAAU network on the native cooperative-matrix path crushes dark tones: shadowed areas go black (issue #59). With `RX_VK_MMA_SOFTWARE=1` and `RX_VK_TAAU_KEYS=-` (the network kept, its multiplies done in software) shadows are right and the picture keeps the network's sharpness, at about 32 fps here. Separate issue to follow. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

NotRexed commented

2026-11-09 05:29:34 +00:00

(Migrated from github.com)