Complete the temporal upscalers' motion vectors for content that moves without bones (water, foliage, projectiles, particles) #1449

Open
opened 2026-11-17 18:40:09 +00:00 by shitpost · 2 comments

Problem. The shared temporal chain (vk_temporal.cpp — DLSS, FSR 4, XeSS, FSR 3.1 on-chain, and FSR 3.1 frame-generation inputs) writes a per-pixel velocity into the motion image the upscaler's temporal filter consumes. Today it covers exactly two motion sources:

  • camera motion — depth reprojection in dlss_prep;
  • skinned actors — actor_motion (poses from the skinned draws, the dlss_actor redraw pipeline, including one-bone rigid models and, in WIP, matrix-moved rigid objects — #617, behind RX_VK_DLSS_RIGID).

Everything else that moves in the world — water, foliage/grass, projectiles (arrows, bombs), particles and effects (magic, the swoosh), mesh-shader-drawn content — has no velocity at all. The upscaler blends every one of its displaced pixels with stale history: the trails and ghosting players report:

  • #484 "Ghosting with DLSS enabled"
  • #1742 (the above-60-fps row): "water sharp and strange animated in 240fps"
  • #2510 (the frame-gen row): "ghosting while using framegen"
  • related: #1370 (DLSS/DLAA ghosting when the camera clips inside Link), #1236 (FSR 3.1 FG shows swinging limbs see-through in made frames), #1310 (FSR 3.1 FG UI glitching in camera motion)

Why not a game patch. The displacement of water, foliage and particles is computed inside the game's GPU shaders (its NVN programs); nx2/oot_patches.py patches lifted C (CPU-visible state, like the jitter in oot_jitter.py) and cannot rewrite the game's NVN programs. And the velocity is per-pixel GPU work regardless. So this is injected GPU work in gpu/vk/ with the per-game analysis in nx2/ — exactly the shape of the existing actor motion. Disocclusions need no new work: the upscalers already manage history from the depth we hand them.

Plan (staged; each phase a self-contained PR).

  • Phase 0 — Coverage inventory. A still-camera census (camera held perfectly still, so RX_VK_DLSS_DEBUG=motion shows only the objects' own motion — the coverage gap, in red/green): water (river, Lake Hylia shore), a grassy field, combat (arrows, magic, swoosh particles), plus whatever mesh-shader content the first pass surfaces. Per-class signatures from the existing draw debug log and nx2/census.py: program key, vertex/index layout, which UBO/SSBO offsets drive the animation (the time/wind values), whether that state is CPU-uploaded or GPU-computed, and the instance layout where instanced. A written analysis note in the nx2/ style, so the per-game knowledge is reviewable. Deliverable: a class table — class, program, state source, state size, pairing strategy, impact — that orders Phase 2.
  • Phase 1 — Framework: a scene-motion pass. Generalize actor motion. Per-class predicates in vk_draw.cpp (gated to the real scene target and temporal_wanted()/temporal_fg_inputs()) calling temporal_note_<class>_draw; state capture in the two actor flavors — a host-visible staging copy of small UBO state "as its draw read it" (the pose_bytes path), and a GPU-to-GPU copy of large/GPU-resident state into a device-address double buffer (the pose_buffers path); pairing reuses dlss_actor::pair_by_place with its reach, generalized; one redraw pipeline per class in the dlss_actor.vert/frag convention — current = this frame's camera plus this frame's state, previous = the frame before's camera without its jitter plus the previous state, written into s.motion in the existing convention, gated by the scene depth within a tolerance (the RX_VK_DLSS_ACTOR_TOL pattern); all classes draw into the same s.motion image over the dlss_prep camera base with the existing vkCmdBeginRendering LOAD pattern, the upscaler dispatch untouched; the same notes/state feed take_motion (vk_fsr3_inputs) so FSR 3.1 frame-generation's inputs get the new classes too; one kill switch per class plus a master, all behind runtime switches, default on, the game's own path reachable. Ship state: framework with zero classes enabled changes nothing; all checks green; zero measurable cost.
  • Phase 2 — Per-class PRs, ordered by impact. (1) Water — large, always visible, time-driven displacement, the simplest re-implementation, removes the most visible ghosting. (2) Foliage/grass — wind sway; check whether per-blade phase is a vertex attribute (formula then stateless) or uniform state; densest geometry, first cost risk. (3) Projectiles — likely instanced with per-instance state in a buffer: snapshot-and-redraw; fast small objects, very visible ghost today. (4) Particles/effects (magic, swoosh) — the state source (CPU upload vs GPU compute) decides the capture flavor; hardest class. (5) Mesh-shader content — whatever the inventory found; snapshot the generated geometry. Each PR: (a) displacement/state analysis from the game's shader, (b) predicate + note, (c) state capture, (d) redraw shader re-implementing the displacement parameterized by state (the dlss_actor precedent — it re-implements OoT's skinning rather than re-running the game's shader), (e) game-free test of the math via srccut, (f) in-game verification, (g) measurement. Each PR independently verifiable and revertible; if a class's state turns out unobservable, the framework degrades gracefully to today's camera-only motion for that class.
  • Phase 3 — Verification and quality gates. A/B matrix on the reported scenes (FSR 4 75%, FSR 4 Native AA, FSR 3.1 — before/after, same framing: water + grass + combat); the still-camera motion view clean — magnitudes and directions match the visible motion, wrong vectors are visible immediately there; cost budget — actor motion today is ~0.2 ms CPU/frame, added GPU time measured per class (the gprof spans exist; RX_VK_PASS_TIMING names the passes) — measure first, don't pre-optimize (if grass redraw is over budget: gate on state change, or re-draw at reduced detail); edge cases — dynamic-resolution changes, the headless probe (HOS_HEADLESS=1), the Off path (no motion built, zero cost), HDR/PsychoV35 (motion is encoding-agnostic), frame-gen inputs; python -I -B tests/run_all.py green (many checks cut functions out of sources by text, so reshaping vk_temporal.cpp/vk_draw.cpp can break distant checks — update them in the same PR).
  • Phase 4 — Docs and release. vk_temporal.cpp's header note (the "what moves by itself would ghost" line becomes history), the README's upscaler sections, switches.def rows, doc: anchors in docs/design.md if a rule is added; final team-updates post; the worklist row closed with links.

Relation to existing work.

  • #242 ("shared pose motion vectors for Link and attached equipment") is where actor motion came from; this generalizes that mechanism to non-boned content.
  • #617 (WIP, ballotstomper) covers matrix-moved rigid objects via RX_VK_DLSS_RIGID (~0.2 ms CPU/frame, windmill sails verified in the motion view on a RX 6750 XT). The Phase 1 framework subsumes that class: its matrix-pose logic is ported into the generalized machinery, credited and linked here. Coordinate with ballotstomper before closing it.
  • Merged foundation this builds on: actor motion (8b925b67, dfc8fa5e), gossip-stone place+reach pairing (5e072fc8), FSR 3.1 with the characters' motion (579c6d48), FSR 3.1 frame-generation with actor motion (983d607b), XeSS (b35943d2).

Expectation note. Completing the vectors removes the motion ghosting (trails behind moving things). It does not remove the temporal filter accumulating the game's own motion blur (the swoosh) — the realistic target is "Cyberpunk-like," not "zero ghosting."

Rules this follows. Every new injection behind a runtime switch, default on, the game's own path reachable; new RX_ names get rows in gpu/vk/switches.def; a MkwVkHost/MkwVkRenderer change bumps MKWVK_API_VERSION; game-free checks in tests/ via srccut; anything the game runs is run in the game before its PR opens (--tested-in-game).

Tested on: Linux, AMD Radeon RX 7900 XTX (RADV, Mesa 26.2.5), Fedora 44 — headless and windowed.

**Problem.** The shared temporal chain (`vk_temporal.cpp` — DLSS, FSR 4, XeSS, FSR 3.1 on-chain, and FSR 3.1 frame-generation inputs) writes a per-pixel velocity into the motion image the upscaler's temporal filter consumes. Today it covers exactly two motion sources: - camera motion — depth reprojection in `dlss_prep`; - skinned actors — `actor_motion` (poses from the skinned draws, the `dlss_actor` redraw pipeline, including one-bone rigid models and, in WIP, matrix-moved rigid objects — #617, behind `RX_VK_DLSS_RIGID`). Everything else that moves in the world — water, foliage/grass, projectiles (arrows, bombs), particles and effects (magic, the swoosh), mesh-shader-drawn content — has no velocity at all. The upscaler blends every one of its displaced pixels with stale history: the trails and ghosting players report: - #484 "Ghosting with DLSS enabled" - #1742 (the above-60-fps row): "water sharp and strange animated in 240fps" - #2510 (the frame-gen row): "ghosting while using framegen" - related: #1370 (DLSS/DLAA ghosting when the camera clips inside Link), #1236 (FSR 3.1 FG shows swinging limbs see-through in made frames), #1310 (FSR 3.1 FG UI glitching in camera motion) **Why not a game patch.** The displacement of water, foliage and particles is computed inside the game's GPU shaders (its NVN programs); `nx2/oot_patches.py` patches lifted C (CPU-visible state, like the jitter in `oot_jitter.py`) and cannot rewrite the game's NVN programs. And the velocity is per-pixel GPU work regardless. So this is injected GPU work in `gpu/vk/` with the per-game analysis in `nx2/` — exactly the shape of the existing actor motion. Disocclusions need no new work: the upscalers already manage history from the depth we hand them. **Plan (staged; each phase a self-contained PR).** - **Phase 0 — Coverage inventory.** A still-camera census (camera held perfectly still, so `RX_VK_DLSS_DEBUG=motion` shows only the objects' own motion — the coverage gap, in red/green): water (river, Lake Hylia shore), a grassy field, combat (arrows, magic, swoosh particles), plus whatever mesh-shader content the first pass surfaces. Per-class signatures from the existing draw debug log and `nx2/census.py`: program key, vertex/index layout, which UBO/SSBO offsets drive the animation (the time/wind values), whether that state is CPU-uploaded or GPU-computed, and the instance layout where instanced. A written analysis note in the `nx2/` style, so the per-game knowledge is reviewable. Deliverable: a class table — class, program, state source, state size, pairing strategy, impact — that orders Phase 2. - **Phase 1 — Framework: a scene-motion pass.** Generalize actor motion. Per-class predicates in `vk_draw.cpp` (gated to the real scene target and `temporal_wanted()`/`temporal_fg_inputs()`) calling `temporal_note_<class>_draw`; state capture in the two actor flavors — a host-visible staging copy of small UBO state "as its draw read it" (the `pose_bytes` path), and a GPU-to-GPU copy of large/GPU-resident state into a device-address double buffer (the `pose_buffers` path); pairing reuses `dlss_actor::pair_by_place` with its reach, generalized; one redraw pipeline per class in the `dlss_actor.vert/frag` convention — current = this frame's camera plus this frame's state, previous = the frame before's camera without its jitter plus the previous state, written into `s.motion` in the existing convention, gated by the scene depth within a tolerance (the `RX_VK_DLSS_ACTOR_TOL` pattern); all classes draw into the same `s.motion` image over the `dlss_prep` camera base with the existing `vkCmdBeginRendering` LOAD pattern, the upscaler dispatch untouched; the same notes/state feed `take_motion` (`vk_fsr3_inputs`) so FSR 3.1 frame-generation's inputs get the new classes too; one kill switch per class plus a master, all behind runtime switches, default on, the game's own path reachable. Ship state: framework with zero classes enabled changes nothing; all checks green; zero measurable cost. - **Phase 2 — Per-class PRs, ordered by impact.** (1) **Water** — large, always visible, time-driven displacement, the simplest re-implementation, removes the most visible ghosting. (2) **Foliage/grass** — wind sway; check whether per-blade phase is a vertex attribute (formula then stateless) or uniform state; densest geometry, first cost risk. (3) **Projectiles** — likely instanced with per-instance state in a buffer: snapshot-and-redraw; fast small objects, very visible ghost today. (4) **Particles/effects** (magic, swoosh) — the state source (CPU upload vs GPU compute) decides the capture flavor; hardest class. (5) **Mesh-shader content** — whatever the inventory found; snapshot the generated geometry. Each PR: (a) displacement/state analysis from the game's shader, (b) predicate + note, (c) state capture, (d) redraw shader re-implementing the displacement parameterized by state (the `dlss_actor` precedent — it re-implements OoT's skinning rather than re-running the game's shader), (e) game-free test of the math via `srccut`, (f) in-game verification, (g) measurement. Each PR independently verifiable and revertible; if a class's state turns out unobservable, the framework degrades gracefully to today's camera-only motion for that class. - **Phase 3 — Verification and quality gates.** A/B matrix on the reported scenes (FSR 4 75%, FSR 4 Native AA, FSR 3.1 — before/after, same framing: water + grass + combat); the still-camera motion view clean — magnitudes and directions match the visible motion, wrong vectors are visible immediately there; cost budget — actor motion today is ~0.2 ms CPU/frame, added GPU time measured per class (the gprof spans exist; `RX_VK_PASS_TIMING` names the passes) — measure first, don't pre-optimize (if grass redraw is over budget: gate on state change, or re-draw at reduced detail); edge cases — dynamic-resolution changes, the headless probe (`HOS_HEADLESS=1`), the Off path (no motion built, zero cost), HDR/PsychoV35 (motion is encoding-agnostic), frame-gen inputs; `python -I -B tests/run_all.py` green (many checks cut functions out of sources by text, so reshaping `vk_temporal.cpp`/`vk_draw.cpp` can break distant checks — update them in the same PR). - **Phase 4 — Docs and release.** `vk_temporal.cpp`'s header note (the "what moves by itself would ghost" line becomes history), the README's upscaler sections, `switches.def` rows, `doc:` anchors in `docs/design.md` if a rule is added; final team-updates post; the worklist row closed with links. **Relation to existing work.** - #242 ("shared pose motion vectors for Link and attached equipment") is where actor motion came from; this generalizes that mechanism to non-boned content. - #617 (WIP, ballotstomper) covers matrix-moved rigid objects via `RX_VK_DLSS_RIGID` (~0.2 ms CPU/frame, windmill sails verified in the motion view on a RX 6750 XT). The Phase 1 framework subsumes that class: its matrix-pose logic is ported into the generalized machinery, credited and linked here. Coordinate with ballotstomper before closing it. - Merged foundation this builds on: actor motion (8b925b67, dfc8fa5e), gossip-stone place+reach pairing (5e072fc8), FSR 3.1 with the characters' motion (579c6d48), FSR 3.1 frame-generation with actor motion (983d607b), XeSS (b35943d2). **Expectation note.** Completing the vectors removes the motion ghosting (trails behind moving things). It does not remove the temporal filter accumulating the game's own motion blur (the swoosh) — the realistic target is "Cyberpunk-like," not "zero ghosting." **Rules this follows.** Every new injection behind a runtime switch, default on, the game's own path reachable; new `RX_` names get rows in `gpu/vk/switches.def`; a `MkwVkHost`/`MkwVkRenderer` change bumps `MKWVK_API_VERSION`; game-free checks in `tests/` via `srccut`; anything the game runs is run in the game before its PR opens (`--tested-in-game`). **Tested on:** Linux, AMD Radeon RX 7900 XTX (RADV, Mesa 26.2.5), Fedora 44 — headless and windowed.
Collaborator

Possibly related, for context:

  • PR #617 WIP: feat(vk): motion vectors for rigid objects moved by their own matrix — Open WIP for motion vectors on rigid objects; this proposal generalizes that machinery.
  • issue #242 Idea: Add shared pose motion vectors for Link and attached equipment (discord Idea) — Existing actor-motion foundation that this proposal extends.
  • issue #1236 Game: FSR 3.1 frame generation shows swinging limbs as see-through copies in made frames — Related frame-generation ghosting report, but for moving skinned limbs rather than non-boned scene content.
  • issue #1370 Game: DLSS/DLAA ghosting when camera clips inside Link — Related DLSS ghosting report with a distinct camera-clipping cause.
Possibly related, for context: - PR #617 WIP: feat(vk): motion vectors for rigid objects moved by their own matrix — Open WIP for motion vectors on rigid objects; this proposal generalizes that machinery. - issue #242 Idea: Add shared pose motion vectors for Link and attached equipment (discord Idea) — Existing actor-motion foundation that this proposal extends. - issue #1236 Game: FSR 3.1 frame generation shows swinging limbs as see-through copies in made frames — Related frame-generation ghosting report, but for moving skinned limbs rather than non-boned scene content. - issue #1370 Game: DLSS/DLAA ghosting when camera clips inside Link — Related DLSS ghosting report with a distinct camera-clipping cause.
Author

Claimed.

  • What: the full plan above. Phase 0 (coverage inventory: the still-camera RX_VK_DLSS_DEBUG=motion census, plus the per-class signatures from the draw debug log and nx2/census.py) first, then the Phase 1 scene-motion framework PR, then one PR per class in impact order — water, foliage/grass, projectiles, particles/effects, mesh-shader content.
  • Why / score: S2 (3 — wrong but playable: visual corruption) × 1.0 upper bound (users of the temporal upscalers: DLSS, FSR 4, XeSS, FSR 3.1) × 0.7 (several reports: #484, the #1742 row, #2510; to be reproduced with a log in the Phase 0 census) = 2.1. The worklist row goes up with the PR that starts the work (the Phase 1 framework).
  • Hardware: Linux, AMD Radeon RX 7900 XTX (RADV, Mesa 26.2.5), Fedora 44 — headless and windowed.
  • Coordination: #617 (WIP — matrix-moved rigid objects, RX_VK_DLSS_RIGID) is subsumed by the Phase 1 framework: its matrix-pose logic is ported into the generalized machinery, credited and linked. Its author was flagged in the team chat.
Claimed. - **What:** the full plan above. Phase 0 (coverage inventory: the still-camera `RX_VK_DLSS_DEBUG=motion` census, plus the per-class signatures from the draw debug log and `nx2/census.py`) first, then the Phase 1 scene-motion framework PR, then one PR per class in impact order — water, foliage/grass, projectiles, particles/effects, mesh-shader content. - **Why / score:** S2 (3 — wrong but playable: visual corruption) × 1.0 upper bound (users of the temporal upscalers: DLSS, FSR 4, XeSS, FSR 3.1) × 0.7 (several reports: #484, the #1742 row, #2510; to be reproduced with a log in the Phase 0 census) = **2.1**. The worklist row goes up with the PR that starts the work (the Phase 1 framework). - **Hardware:** Linux, AMD Radeon RX 7900 XTX (RADV, Mesa 26.2.5), Fedora 44 — headless and windowed. - **Coordination:** #617 (WIP — matrix-moved rigid objects, `RX_VK_DLSS_RIGID`) is subsumed by the Phase 1 framework: its matrix-pose logic is ported into the generalized machinery, credited and linked. Its author was flagged in the team chat.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot#1449
No description provided.