Make the game's upscale dispatch addressable by an upscaler backend #821

Closed

notquiteog wants to merge 5 commits from feat/upscale-descriptor-order into main

notquiteog commented

2026-11-08 22:35:29 +00:00

(Migrated from github.com)

The finding this is built on

OoT evaluates its own neural TAAU network (program fd029a317a9a4b94), and taau_stand_in in gpu/vk/vk_draw.cpp already recognizes that dispatch and substitutes a compute resolve where the device has no hardware f16 matrix multiply. So the game already renders the 3D at a reduced internal resolution and resolves it temporally — a vendor upscaler takes that dispatch's place.

The composited window image, where the upscale filter and the frame interpolator work, is the wrong place for one: no depth, no motion, and the HUD already in the history. That is why the existing interpolator block-matches quarter-size luma.

What is in here

gpu/vk/upscale_contract.h — which sampled binding is colour, motion and depth, and the sets it must refuse rather than guess. No Vulkan include, nothing from guest memory.
gpu/vk/upscale_format.h — the one place that decides what a VkFormat means to that contract. Separate so it is testable: which formats count as motion is the part most likely to be wrong, and a wrong answer there is handed to an upscaler as motion.
gpu/vk/upscale_seam.h — a recognized dispatch resolved once into an upscale::Request, choose() to pick a backend, and temporal_possible() as the gate one must pass.
The existing per-network log now names each binding's role and says whether a temporal backend could stand in, so the contract can be read off a run instead of assumed.
docs/upscaling.md — the integration point, the input contract, what each vendor runtime permits, and the order the rest can land in.

No behaviour change. The stand-in is still the only backend; it is a spatial resolve that needs none of the contract, and choose() returns it under exactly the condition the old first guard tested. tests/upscale_seam_test.cpp pins that: making the stand-in require the contract fails the test.

The bug this caught on the way

The contract first read draw_sampled, which is the obvious list and the wrong one. It records a sampled texture only where the GPU has written it and records every storage image too, so its index i is not dw.imgs[i]. A backend wired to that would upscale whichever texture happened to be bound first and ghost, and the ghosting would be filed as a renderer bug. upscale_bindings now resolves the texture descriptors the way build_descriptors does — same view type, same ViewUse::Sample, same handle_is_storage_pointer guard for the case d603d98bb fixed — so an index is a descriptor slot.

Tested

Builds: the Vulkan renderer builds clean with RX_VK_ONLY=1 (nx2/build.py --standalone), 0 warnings, and a full game build from a v131072 dump (build e0f3ccfa) succeeded in 8m20s.
Runs: Linux, RTX 5060 Ti, driver 615.71.09, Xvfb, 240 s of the opening — 491 presents, swapchain 1280x720. The per-network log prints its roles and verdict on real networks: the movie upscalers report no upscaler inputs, correctly, since they sample one 1920x1080 texture and nothing else. The gameplay TAAU dispatch was not reached in that run, so the seam has not fired on it yet and the PR does not claim it has.
No fps claim. That run had RX_VK_PRECOMPILE=0 so every pipeline compiled on demand; it is not a performance measurement.
python -I -B tests/run_all.py: 89 passed. The 14 failures and 7 skips are identical on pristine main in this environment (87 passed there; the +2 are the two tests added here), and are a missing clang++ plus pre-existing SPIR-V and BC-level failures.

Not in here

No DLSS, no FSR, no frame generation, no change to the internal resolution, no new runtime dependency and no vendored runtime. Every new switch is off or neutral unless asked for.

AI assistance: recorded in the commit trailers.

Several open requests ask for DLSS, FSR or "a proper upscaling technique" (#744, #691, #687, #672). This does not add a backend. It does the groundwork a backend needs, and writes down where one belongs, so that the interesting part can be reviewed on its own afterwards. ### The finding this is built on OoT evaluates its own neural TAAU network (program `fd029a317a9a4b94`), and `taau_stand_in` in `gpu/vk/vk_draw.cpp` already recognizes that dispatch and substitutes a compute resolve where the device has no hardware f16 matrix multiply. So the game already renders the 3D at a reduced internal resolution and resolves it temporally — a vendor upscaler takes **that dispatch's** place. The composited window image, where the `upscale` filter and the frame interpolator work, is the wrong place for one: no depth, no motion, and the HUD already in the history. That is why the existing interpolator block-matches quarter-size luma. ### What is in here - `gpu/vk/upscale_contract.h` — which sampled binding is colour, motion and depth, and the sets it must refuse rather than guess. No Vulkan include, nothing from guest memory. - `gpu/vk/upscale_format.h` — the one place that decides what a `VkFormat` means to that contract. Separate so it is testable: which formats count as motion is the part most likely to be wrong, and a wrong answer there is handed to an upscaler as motion. - `gpu/vk/upscale_seam.h` — a recognized dispatch resolved once into an `upscale::Request`, `choose()` to pick a backend, and `temporal_possible()` as the gate one must pass. - The existing per-network log now names each binding's role and says whether a temporal backend could stand in, so the contract can be read off a run instead of assumed. - `docs/upscaling.md` — the integration point, the input contract, what each vendor runtime permits, and the order the rest can land in. **No behaviour change.** The stand-in is still the only backend; it is a spatial resolve that needs none of the contract, and `choose()` returns it under exactly the condition the old first guard tested. `tests/upscale_seam_test.cpp` pins that: making the stand-in require the contract fails the test. ### The bug this caught on the way The contract first read `draw_sampled`, which is the obvious list and the wrong one. It records a sampled texture only where the GPU has written it and records every storage image too, so its index `i` is not `dw.imgs[i]`. A backend wired to that would upscale whichever texture happened to be bound first and ghost, and the ghosting would be filed as a renderer bug. `upscale_bindings` now resolves the texture descriptors the way `build_descriptors` does — same view type, same `ViewUse::Sample`, same `handle_is_storage_pointer` guard for the case `d603d98bb` fixed — so an index is a descriptor slot. ### Tested - **Builds**: the Vulkan renderer builds clean with `RX_VK_ONLY=1` (`nx2/build.py --standalone`), 0 warnings, and a full game build from a v131072 dump (build `e0f3ccfa`) succeeded in 8m20s. - **Runs**: Linux, RTX 5060 Ti, driver 615.71.09, Xvfb, 240 s of the opening — 491 presents, swapchain 1280x720. The per-network log prints its roles and verdict on real networks: the movie upscalers report `no upscaler inputs`, correctly, since they sample one 1920x1080 texture and nothing else. The gameplay TAAU dispatch was not reached in that run, so the seam has not fired on it yet and the PR does not claim it has. - **No fps claim.** That run had `RX_VK_PRECOMPILE=0` so every pipeline compiled on demand; it is not a performance measurement. - `python -I -B tests/run_all.py`: 89 passed. The 14 failures and 7 skips are identical on pristine `main` in this environment (87 passed there; the +2 are the two tests added here), and are a missing `clang++` plus pre-existing SPIR-V and BC-level failures. ### Not in here No DLSS, no FSR, no frame generation, no change to the internal resolution, no new runtime dependency and no vendored runtime. Every new switch is off or neutral unless asked for. AI assistance: recorded in the commit trailers.

notquiteog commented

2026-11-08 22:53:29 +00:00

(Migrated from github.com)

Correction, from a warmed run that reached gameplay. The premise in the description above is wrong, and the log that this PR adds is what shows it:

vk: network fd029a317a9a4b94: dispatch 32x36x1, 1 textures 1 images: 0:1536x864; no upscaler inputs

One sampled texture, one written image, nothing else — no motion, no depth. All eleven cooperative-matrix networks in the game are that shape: at most two sampled textures, never a motion or depth target among them. So OoT's upscaler is a single-image neural upscaler, not a motion-vector temporal one, and a temporal upscaler cannot take that dispatch's place: it is never handed what one needs. temporal_possible() refuses there and will keep refusing however a backend is wired.

Two things follow, and I would rather say them here than have a reviewer find them.

The integration point in docs/upscaling.md is superseded, and #577's — at the final resolve, with depth carried from the scene passes and motion derived from the camera — is the right one for this game. The doc in the branch now says that above the section it contradicts. The inputs have to be assembled, not taken.

What stands: the contract and the gate did their job, refusing rather than handing an upscaler whichever texture happened to be bound first, which is the failure the descriptor-order commit is about. The per-network log is what produced this answer. The census is unaffected.

The run also confirms the opportunity is real: the network's input is 1536x864 against a 2048x1152 scene, so the game already renders the 3D at 75% and upscales.

Happy for this to be judged on the contract, the log and the descriptor-order fix alone — the "a vendor upscaler takes that dispatch's place" framing should not be read as a claim any more. Tested on Linux, RTX 5060 Ti, driver 615.71.09, a full build from a v131072 dump.

**Correction, from a warmed run that reached gameplay.** The premise in the description above is wrong, and the log that this PR adds is what shows it: ``` vk: network fd029a317a9a4b94: dispatch 32x36x1, 1 textures 1 images: 0:1536x864; no upscaler inputs ``` One sampled texture, one written image, nothing else — no motion, no depth. All eleven cooperative-matrix networks in the game are that shape: at most two sampled textures, never a motion or depth target among them. So OoT's upscaler is a **single-image** neural upscaler, not a motion-vector temporal one, and a temporal upscaler cannot take that dispatch's place: it is never handed what one needs. `temporal_possible()` refuses there and will keep refusing however a backend is wired. Two things follow, and I would rather say them here than have a reviewer find them. **The integration point in `docs/upscaling.md` is superseded**, and #577's — at the final resolve, with depth carried from the scene passes and motion derived from the camera — is the right one for this game. The doc in the branch now says that above the section it contradicts. The inputs have to be *assembled*, not taken. **What stands:** the contract and the gate did their job, refusing rather than handing an upscaler whichever texture happened to be bound first, which is the failure the descriptor-order commit is about. The per-network log is what produced this answer. The census is unaffected. The run also confirms the opportunity is real: the network's input is **1536x864** against a 2048x1152 scene, so the game already renders the 3D at 75% and upscales. Happy for this to be judged on the contract, the log and the descriptor-order fix alone — the "a vendor upscaler takes that dispatch's place" framing should not be read as a claim any more. Tested on Linux, RTX 5060 Ti, driver 615.71.09, a full build from a v131072 dump.

NotRexed commented

2026-11-09 05:30:23 +00:00

(Migrated from github.com)

Thank you, and thanks for the honest follow-up! As your own comment shows, OoT's upscale network takes a single image, so a temporal upscaler can't take that dispatch's place as described; #850 takes another route (motion from depth and the camera). Closing to keep the queue clear; a smaller PR with the parts that still apply would be welcome.

Commenting is not possible because the repository is archived.

No reviewers