Complete the temporal upscalers' motion vectors for content that moves without bones (water, foliage, projectiles, particles) #1449
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#1449
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem. The shared temporal chain (
vk_temporal.cpp— DLSS, FSR 4, XeSS, FSR 3.1 on-chain, and FSR 3.1 frame-generation inputs) writes a per-pixel velocity into the motion image the upscaler's temporal filter consumes. Today it covers exactly two motion sources:dlss_prep;actor_motion(poses from the skinned draws, thedlss_actorredraw pipeline, including one-bone rigid models and, in WIP, matrix-moved rigid objects — #617, behindRX_VK_DLSS_RIGID).Everything else that moves in the world — water, foliage/grass, projectiles (arrows, bombs), particles and effects (magic, the swoosh), mesh-shader-drawn content — has no velocity at all. The upscaler blends every one of its displaced pixels with stale history: the trails and ghosting players report:
Why not a game patch. The displacement of water, foliage and particles is computed inside the game's GPU shaders (its NVN programs);
nx2/oot_patches.pypatches lifted C (CPU-visible state, like the jitter inoot_jitter.py) and cannot rewrite the game's NVN programs. And the velocity is per-pixel GPU work regardless. So this is injected GPU work ingpu/vk/with the per-game analysis innx2/— exactly the shape of the existing actor motion. Disocclusions need no new work: the upscalers already manage history from the depth we hand them.Plan (staged; each phase a self-contained PR).
RX_VK_DLSS_DEBUG=motionshows only the objects' own motion — the coverage gap, in red/green): water (river, Lake Hylia shore), a grassy field, combat (arrows, magic, swoosh particles), plus whatever mesh-shader content the first pass surfaces. Per-class signatures from the existing draw debug log andnx2/census.py: program key, vertex/index layout, which UBO/SSBO offsets drive the animation (the time/wind values), whether that state is CPU-uploaded or GPU-computed, and the instance layout where instanced. A written analysis note in thenx2/style, so the per-game knowledge is reviewable. Deliverable: a class table — class, program, state source, state size, pairing strategy, impact — that orders Phase 2.vk_draw.cpp(gated to the real scene target andtemporal_wanted()/temporal_fg_inputs()) callingtemporal_note_<class>_draw; state capture in the two actor flavors — a host-visible staging copy of small UBO state "as its draw read it" (thepose_bytespath), and a GPU-to-GPU copy of large/GPU-resident state into a device-address double buffer (thepose_bufferspath); pairing reusesdlss_actor::pair_by_placewith its reach, generalized; one redraw pipeline per class in thedlss_actor.vert/fragconvention — current = this frame's camera plus this frame's state, previous = the frame before's camera without its jitter plus the previous state, written intos.motionin the existing convention, gated by the scene depth within a tolerance (theRX_VK_DLSS_ACTOR_TOLpattern); all classes draw into the sames.motionimage over thedlss_prepcamera base with the existingvkCmdBeginRenderingLOAD pattern, the upscaler dispatch untouched; the same notes/state feedtake_motion(vk_fsr3_inputs) so FSR 3.1 frame-generation's inputs get the new classes too; one kill switch per class plus a master, all behind runtime switches, default on, the game's own path reachable. Ship state: framework with zero classes enabled changes nothing; all checks green; zero measurable cost.dlss_actorprecedent — it re-implements OoT's skinning rather than re-running the game's shader), (e) game-free test of the math viasrccut, (f) in-game verification, (g) measurement. Each PR independently verifiable and revertible; if a class's state turns out unobservable, the framework degrades gracefully to today's camera-only motion for that class.RX_VK_PASS_TIMINGnames the passes) — measure first, don't pre-optimize (if grass redraw is over budget: gate on state change, or re-draw at reduced detail); edge cases — dynamic-resolution changes, the headless probe (HOS_HEADLESS=1), the Off path (no motion built, zero cost), HDR/PsychoV35 (motion is encoding-agnostic), frame-gen inputs;python -I -B tests/run_all.pygreen (many checks cut functions out of sources by text, so reshapingvk_temporal.cpp/vk_draw.cppcan break distant checks — update them in the same PR).vk_temporal.cpp's header note (the "what moves by itself would ghost" line becomes history), the README's upscaler sections,switches.defrows,doc:anchors indocs/design.mdif a rule is added; final team-updates post; the worklist row closed with links.Relation to existing work.
RX_VK_DLSS_RIGID(~0.2 ms CPU/frame, windmill sails verified in the motion view on a RX 6750 XT). The Phase 1 framework subsumes that class: its matrix-pose logic is ported into the generalized machinery, credited and linked here. Coordinate with ballotstomper before closing it.8b925b67,dfc8fa5e), gossip-stone place+reach pairing (5e072fc8), FSR 3.1 with the characters' motion (579c6d48), FSR 3.1 frame-generation with actor motion (983d607b), XeSS (b35943d2).Expectation note. Completing the vectors removes the motion ghosting (trails behind moving things). It does not remove the temporal filter accumulating the game's own motion blur (the swoosh) — the realistic target is "Cyberpunk-like," not "zero ghosting."
Rules this follows. Every new injection behind a runtime switch, default on, the game's own path reachable; new
RX_names get rows ingpu/vk/switches.def; aMkwVkHost/MkwVkRendererchange bumpsMKWVK_API_VERSION; game-free checks intests/viasrccut; anything the game runs is run in the game before its PR opens (--tested-in-game).Tested on: Linux, AMD Radeon RX 7900 XTX (RADV, Mesa 26.2.5), Fedora 44 — headless and windowed.
Possibly related, for context:
Claimed.
RX_VK_DLSS_DEBUG=motioncensus, plus the per-class signatures from the draw debug log andnx2/census.py) first, then the Phase 1 scene-motion framework PR, then one PR per class in impact order — water, foliage/grass, projectiles, particles/effects, mesh-shader content.RX_VK_DLSS_RIGID) is subsumed by the Phase 1 framework: its matrix-pose logic is ported into the generalized machinery, credited and linked. Its author was flagged in the team chat.