feat(vk): FXAA in the gameplay TAAU stand-in (edges on cards without the network) #1115

github-sync · 2026-11-09T12:25:30Z

t0masz407 commented

2026-11-09 12:25:30 +00:00

(Migrated from github.com)

What

The gameplay TAAU stand-in (taau_stand_in → typed_upscale, shaders/taau_upscale.comp) now runs FXAA on its input instead of the plain linear sample. It uses FXAA 3.11's quality variant, in the network's perceptual (square-root) encoding, at the input's texel size.

RX_VK_TAAU_FXAA=0: the linear upscale as before.
RX_VK_TAAU_FXAA=2: diagnostic colours for the pixels it moves, red along an edge and blue for the sub-pixel blend.

The other typed_upscale caller (the generic RGB9E5 fallback) is unchanged.

Why

On cards where the TAAU network does not run (software matrix multiplies: AMD, Intel; #52, tracked in #1006), the picture has no anti-aliasing. Every edge stays stair-stepped, and hair, Link's cap, leaves and thin branches shimmer as they move. The network reads the scene colour alone: its dispatch binds one texture (network fd029a317a9a4b94: … 1 textures 1 images), with no motion vectors or depth. Temporal accumulation would therefore smear everything that moves (the history bound's comment records that). A one-frame filter is what can stand in.

Tuning

At FXAA's default thresholds (0.125 / 0.0312, sub-pixel 0.75), RX_VK_TAAU_FXAA=2 showed most of Kokiri Forest in red: moss, cloth and leaf textures were taken for edges and softened. At the high-quality thresholds (0.166 / 0.0833) with half the sub-pixel blend (0.5), it acts on contours: Link against the background, the hair's outline, the cap, leaf edges. Textures keep their plain sample.

The push constant's third word is now read as bits: 1 the history bound (RX_VK_TAAU_CLAMP), 2 FXAA, 4 its colours. The bound tests its own bit only.

Tested

RX 7900 XTX, RADV (Mesa 26.2.6), Linux (Hyprland, native Wayland), v2.0.7 plus this change, Kokiri Forest at 3840x2160 output with dynamic resolution (the 3D at 2880x1620):

edges of hair, cap and foliage visibly smoother, and textures not blurred, compared side by side with RX_VK_TAAU_FXAA=0;
RX_VK_TAAU_FXAA=2 before and after the threshold change, as described above;
no frame-rate change noticed in play (84-89 fps shown, 41-44 the game's own). This is not a controlled measurement; the filter reads 5 texels per output pixel off edges and up to about 30 on one.

tests/test_taau_fxaa.py checks:

the history bound stays on its own bit;
only the gameplay stand-in sets FXAA's bits;
RX_VK_TAAU_FXAA switches it;
with glslangValidator of the header's version (16.6.0) on PATH, taau_upscale_comp_spv.h is compiled again from taau_upscale.comp and compared.

🤖 Generated with Claude Code

## What The gameplay TAAU stand-in (`taau_stand_in` → `typed_upscale`, `shaders/taau_upscale.comp`) now runs FXAA on its input instead of the plain linear sample. It uses FXAA 3.11's quality variant, in the network's perceptual (square-root) encoding, at the input's texel size. - `RX_VK_TAAU_FXAA=0`: the linear upscale as before. - `RX_VK_TAAU_FXAA=2`: diagnostic colours for the pixels it moves, red along an edge and blue for the sub-pixel blend. The other `typed_upscale` caller (the generic RGB9E5 fallback) is unchanged. ## Why On cards where the TAAU network does not run (software matrix multiplies: AMD, Intel; #52, tracked in #1006), the picture has no anti-aliasing. Every edge stays stair-stepped, and hair, Link's cap, leaves and thin branches shimmer as they move. The network reads the scene colour alone: its dispatch binds one texture (`network fd029a317a9a4b94: … 1 textures 1 images`), with no motion vectors or depth. Temporal accumulation would therefore smear everything that moves (the history bound's comment records that). A one-frame filter is what can stand in. ## Tuning At FXAA's default thresholds (0.125 / 0.0312, sub-pixel 0.75), `RX_VK_TAAU_FXAA=2` showed most of Kokiri Forest in red: moss, cloth and leaf textures were taken for edges and softened. At the high-quality thresholds (0.166 / 0.0833) with half the sub-pixel blend (0.5), it acts on contours: Link against the background, the hair's outline, the cap, leaf edges. Textures keep their plain sample. The push constant's third word is now read as bits: 1 the history bound (`RX_VK_TAAU_CLAMP`), 2 FXAA, 4 its colours. The bound tests its own bit only. ## Tested RX 7900 XTX, RADV (Mesa 26.2.6), Linux (Hyprland, native Wayland), v2.0.7 plus this change, Kokiri Forest at 3840x2160 output with dynamic resolution (the 3D at 2880x1620): - edges of hair, cap and foliage visibly smoother, and textures not blurred, compared side by side with `RX_VK_TAAU_FXAA=0`; - `RX_VK_TAAU_FXAA=2` before and after the threshold change, as described above; - no frame-rate change noticed in play (84-89 fps shown, 41-44 the game's own). This is not a controlled measurement; the filter reads 5 texels per output pixel off edges and up to about 30 on one. `tests/test_taau_fxaa.py` checks: - the history bound stays on its own bit; - only the gameplay stand-in sets FXAA's bits; - `RX_VK_TAAU_FXAA` switches it; - with `glslangValidator` of the header's version (16.6.0) on PATH, `taau_upscale_comp_spv.h` is compiled again from `taau_upscale.comp` and compared. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hckrman101 (Migrated from github.com) approved these changes

2026-11-09 23:26:54 +00:00

hckrman101 (Migrated from github.com) left a comment

Nice, small and well contained. FXAA sits behind its own push-constant bit inside the existing taau_upscale.comp dispatch, so there are no new images, descriptors, barriers or layout changes. Bindings, the push-constant range (still 12 bytes) and the history and resize handling all stay as they were. Only taau_stand_in's temporal=true call sets bits 2 and 4. The generic RGB9E5 fallback (temporal=false) is untouched, and the history bound now tests & 1, so FXAA can't turn the clamp on by accident.

What I checked:

The port follows FXAA 3.11 quality step by step. That covers the N/S to W/E swap after ns/we are taken, pair_n and the sign flip, the half-texel posB, end-of-span search with grad/4, good span, and the sub-pixel term over 12 with max(span, sub). The search schedule is a custom 12-step one, which is fine.
Luma is taken in the sqrt-encoded input, which is the perceptual space FXAA expects. The square is applied afterwards, as before.
taau_upscale_comp_spv.h matches the source. I rebuilt it with glslangValidator 16.6.0 (-V --target-env vulkan1.3 --vn TAAU_UPSCALE_COMP_SPV) and got a byte-identical file, and spirv-val passes. Of the -107 lines, 104 are the regenerated header and 3 are shader lines replaced by the bit tests.
DLSS/FSR: on #1008 (and the #841 chain it sits on), temporal_stand_in/dlss_stand_in runs before taau_stand_in, and #1008's temporal_rejected_network goes through upscale_stub, not typed_upscale. On #1335, FSR 3/4 are chosen inside taau_stand_in before it falls back to typed_upscale. So FXAA only runs when no temporal backend took the dispatch. The size[2] context lines are the same on both branches, so I don't expect a textual conflict in vk_draw.cpp, and neither branch touches taau_upscale.comp.
No settings option is added (env var only), so no option-id collisions. tests/test_taau_fxaa.py is picked up automatically by run_all.py's tests/test_*.py glob.

Nits only, inline. One design note, no change needed: the 3x3 neighbourhood is read at the output pixel's centre offset by input texels, so at a 1.33x scale every tap is already a bilinear blend. That's a reasonable choice, since snapping to input texel centres would go blocky, but it explains why the edge detection is a bit softer than native-res FXAA. FXAA is also on by default for everyone on the stand-in path with no menu toggle. That's fine as a default, and it could later go in #1335's Upscaling row if people want it off without an env var.

No CI has run on this branch yet (fork workflows not approved), so the test hasn't run in CI.

Nice, small and well contained. FXAA sits behind its own push-constant bit inside the existing `taau_upscale.comp` dispatch, so there are no new images, descriptors, barriers or layout changes. Bindings, the push-constant range (still 12 bytes) and the history and resize handling all stay as they were. Only `taau_stand_in`'s `temporal=true` call sets bits 2 and 4. The generic RGB9E5 fallback (`temporal=false`) is untouched, and the history bound now tests `& 1`, so FXAA can't turn the clamp on by accident. What I checked: - The port follows FXAA 3.11 quality step by step. That covers the N/S to W/E swap after `ns`/`we` are taken, `pair_n` and the sign flip, the half-texel `posB`, end-of-span search with `grad/4`, `good` span, and the sub-pixel term over 12 with `max(span, sub)`. The search schedule is a custom 12-step one, which is fine. - Luma is taken in the sqrt-encoded input, which is the perceptual space FXAA expects. The square is applied afterwards, as before. - `taau_upscale_comp_spv.h` matches the source. I rebuilt it with glslangValidator 16.6.0 (`-V --target-env vulkan1.3 --vn TAAU_UPSCALE_COMP_SPV`) and got a byte-identical file, and `spirv-val` passes. Of the -107 lines, 104 are the regenerated header and 3 are shader lines replaced by the bit tests. - DLSS/FSR: on #1008 (and the #841 chain it sits on), `temporal_stand_in`/`dlss_stand_in` runs before `taau_stand_in`, and #1008's `temporal_rejected_network` goes through `upscale_stub`, not `typed_upscale`. On #1335, FSR 3/4 are chosen inside `taau_stand_in` before it falls back to `typed_upscale`. So FXAA only runs when no temporal backend took the dispatch. The `size[2]` context lines are the same on both branches, so I don't expect a textual conflict in `vk_draw.cpp`, and neither branch touches `taau_upscale.comp`. - No settings option is added (env var only), so no option-id collisions. `tests/test_taau_fxaa.py` is picked up automatically by `run_all.py`'s `tests/test_*.py` glob. Nits only, inline. One design note, no change needed: the 3x3 neighbourhood is read at the output pixel's centre offset by input texels, so at a 1.33x scale every tap is already a bilinear blend. That's a reasonable choice, since snapping to input texel centres would go blocky, but it explains why the edge detection is a bit softer than native-res FXAA. FXAA is also on by default for everyone on the stand-in path with no menu toggle. That's fine as a default, and it could later go in #1335's Upscaling row if people want it off without an env var. No CI has run on this branch yet (fork workflows not approved), so the test hasn't run in CI.

gpu/vk/shaders/taau_upscale.comp

					
				@ -17,0 +35,4 @@

				    // (FXAA's high-quality thresholds: with 0.125 / 0.0312 it also took the moss, cloth and leaf textures for edges and

				    // softened them)

				    if (range < max(0.0833, hi * 0.166)) return rgbM;   // (no edge: flat areas and fine textures keep their plain sample)

				    const float nw = L(uv - px), se = L(uv + px), ne = L(uv + vec2(px.x, -px.y)), sw = L(uv + vec2(-px.x, px.y));

Nit: an inf texel (the header mentions frames that blow up) gives range = inf and hi * 0.166 = inf, so this test is false and FXAA carries on. Further down, sub becomes inf/inf = NaN, and clamp/max with NaN are undefined in SPIR-V, so the final uv can come out NaN for the bad pixel and its neighbours. That's harmless on RADV/NVIDIA in practice, since their fmax drops the NaN. Still, something like if (!(range < 65504.0)) return rgbM; (or an isinf/isnan check on hi) would keep this path from making a bad texel worse when the clamp is off, which is the default.

Nit: an `inf` texel (the header mentions frames that blow up) gives `range = inf` and `hi * 0.166 = inf`, so this test is false and FXAA carries on. Further down, `sub` becomes `inf/inf = NaN`, and `clamp`/`max` with NaN are undefined in SPIR-V, so the final `uv` can come out NaN for the bad pixel and its neighbours. That's harmless on RADV/NVIDIA in practice, since their fmax drops the NaN. Still, something like `if (!(range < 65504.0)) return rgbM;` (or an `isinf`/`isnan` check on `hi`) would keep this path from making a bad texel worse when the clamp is off, which is the default.

gpu/vk/vk_draw.cpp

					
				@ -5849,2 +5849,4 @@

				    const uint64_t now_frame = texture_frame_no();

				    size[2] = temporal && !no_clamp && hist_frame && now_frame - hist_frame <= 3 ? 1u : 0u;

				    // (the gameplay stand-in's edges: FXAA in the shader. RX_VK_TAAU_FXAA=0: the plain linear upscale as before; 2: the

				    // pixels it changed drawn red)

Nit: mode 2 draws the sub-pixel blend in blue as well as edges in red (shader line 83), so this comment could say "red/blue" to match the shader and the PR description.