feat(vk): DLSS frame generation (NGX), more actor motion for DLSS, FidelityFX FSR 4 fallback (builds with #1833, #1834) #1831
Closed
FIocker wants to merge 1 commit from
super-build/upscaling-framegen into main
pull from: super-build/upscaling-framegen
merge into: MagicBeanDev:main
MagicBeanDev:main
MagicBeanDev:github/pr/2893
MagicBeanDev:github/pr/2890
MagicBeanDev:github/pr/2887
MagicBeanDev:github/pr/2878
MagicBeanDev:github/pr/2875
MagicBeanDev:github/pr/2874
MagicBeanDev:github/pr/2873
MagicBeanDev:github/pr/2871
MagicBeanDev:github/pr/2756
MagicBeanDev:github/pr/2897
MagicBeanDev:github/pr/2896
MagicBeanDev:github/pr/2718
MagicBeanDev:github/pr/2842
MagicBeanDev:github/pr/2691
MagicBeanDev:github/pr/2895
MagicBeanDev:github/pr/2743
MagicBeanDev:github/pr/2765
MagicBeanDev:github/pr/2708
MagicBeanDev:github/pr/2732
MagicBeanDev:github/pr/2755
MagicBeanDev:github/pr/2817
MagicBeanDev:github/pr/2876
MagicBeanDev:release-please--branches--main
MagicBeanDev:release-please--branches--main--release-notes
MagicBeanDev:github/pr/2778
MagicBeanDev:github/pr/2888
MagicBeanDev:github/pr/2866
MagicBeanDev:github/pr/2877
MagicBeanDev:github/pr/2800
MagicBeanDev:github/pr/2829
MagicBeanDev:github/pr/2850
MagicBeanDev:github/pr/2889
MagicBeanDev:github/pr/2882
MagicBeanDev:github/pr/2860
MagicBeanDev:github/pr/2849
MagicBeanDev:github/pr/2886
MagicBeanDev:github/pr/2794
MagicBeanDev:github/pr/2840
MagicBeanDev:github/pr/2830
MagicBeanDev:github/pr/2834
MagicBeanDev:github/pr/2719
MagicBeanDev:github/pr/2742
MagicBeanDev:github/pr/2741
MagicBeanDev:github/pr/2820
MagicBeanDev:github/pr/2771
MagicBeanDev:github/pr/2859
MagicBeanDev:github/pr/2858
MagicBeanDev:github/pr/2823
MagicBeanDev:github/pr/2757
MagicBeanDev:github/pr/2769
MagicBeanDev:github/pr/2844
MagicBeanDev:github/pr/2843
MagicBeanDev:github/pr/2673
MagicBeanDev:github/pr/2855
MagicBeanDev:github/pr/2739
MagicBeanDev:github/pr/2807
MagicBeanDev:github/pr/2851
MagicBeanDev:github/pr/2789
MagicBeanDev:github/pr/2814
MagicBeanDev:github/pr/2818
MagicBeanDev:github/pr/2774
MagicBeanDev:github/pr/2772
MagicBeanDev:github/pr/2750
MagicBeanDev:github/pr/2786
MagicBeanDev:github/pr/2785
MagicBeanDev:github/pr/2782
MagicBeanDev:github/pr/2779
MagicBeanDev:github/pr/2775
MagicBeanDev:github/pr/2706
MagicBeanDev:github/pr/2766
MagicBeanDev:github/pr/2747
MagicBeanDev:github/pr/2759
MagicBeanDev:github/pr/2745
MagicBeanDev:github/pr/2740
MagicBeanDev:github/pr/2733
MagicBeanDev:github/pr/2725
MagicBeanDev:github/pr/2727
MagicBeanDev:github/pr/2728
MagicBeanDev:github/pr/2729
MagicBeanDev:github/pr/2721
MagicBeanDev:github/pr/2722
MagicBeanDev:github/pr/2715
MagicBeanDev:github/pr/2714
MagicBeanDev:github/pr/2707
MagicBeanDev:github/pr/2703
MagicBeanDev:github/pr/2705
MagicBeanDev:github/pr/2697
MagicBeanDev:github/pr/2679
MagicBeanDev:github/pr/2676
MagicBeanDev:github/pr/2669
MagicBeanDev:docs/what-hookshot-connects-to
MagicBeanDev:fix/hdr-encode-test
MagicBeanDev:fix/main-tests
MagicBeanDev:fix/play-cache-files
MagicBeanDev:perf/game-barrier-order
MagicBeanDev:feat/heap-lazy
MagicBeanDev:feat/texture-detail
MagicBeanDev:perf/astc-opaque-bc1
MagicBeanDev:fix/fs-dir-lock
MagicBeanDev:fix/framegen-native-kept
MagicBeanDev:fix/update-no-downgrade
MagicBeanDev:port/k10-gcn-f16-mirrors
MagicBeanDev:fix/framegen-mult-cap
MagicBeanDev:chore/release-as-2.1.1
MagicBeanDev:perf/pcache-load-switch
MagicBeanDev:fix/prepare-progress-log
MagicBeanDev:fix/update-closes-stale-game
MagicBeanDev:fix/hdr-scaled-sampled
MagicBeanDev:perf/first-start-preset
MagicBeanDev:perf/zero-upload
MagicBeanDev:fix/vcore-escape-2s
MagicBeanDev:test/fake-gh-atomic
MagicBeanDev:perf/batch-idle-flush
MagicBeanDev:test/finish-stub-a
MagicBeanDev:chore/record-2.1.0
MagicBeanDev:fix/main-red-tests
MagicBeanDev:fix/test-display-startup-driver-cache
MagicBeanDev:feat/mesh-shader-mode
MagicBeanDev:fix/menu-probes-at-startup
MagicBeanDev:fix/fps-counter-classic
MagicBeanDev:fix/test-log-batch-flake
MagicBeanDev:fix/narrow-screen-hud
MagicBeanDev:ci/prereleases
MagicBeanDev:release/v2.0.15
MagicBeanDev:hotfix/v2.0.15
MagicBeanDev:fix/present-ahead-replay
MagicBeanDev:release/v2.0.13
MagicBeanDev:fix/bracket-paths-2013
MagicBeanDev:fix/hdr-overlays-2013
MagicBeanDev:fix/fps-counter-scale-2013
MagicBeanDev:fix/stick-seen-at-rest
MagicBeanDev:hotfix/menu-fits-screen
MagicBeanDev:fix/f1-always-opens-menu
MagicBeanDev:fix/ini-keep-lines
MagicBeanDev:claude/cranky-pike-7cf247
MagicBeanDev:perf/texture-answer-flat
MagicBeanDev:fix/stamp-test-modules
MagicBeanDev:fix/appimage-fonts
MagicBeanDev:fix/display-awake
MagicBeanDev:release/v2.0.12
MagicBeanDev:fix/discord-windows
MagicBeanDev:rc/v2.0.12-tests
MagicBeanDev:fix/dlss-keep-feature
MagicBeanDev:rc/v2.0.12
MagicBeanDev:button-icons
No reviewers
Labels
Clear labels
accessibility
Barrier affecting people with disabilities
autorelease: pending
autorelease: tagged
bug
Something isn't working
dependencies
Pull requests that update a dependency file
documentation
Improvements or additions to documentation
duplicate
This issue or pull request already exists
enhancement
New feature or request
github_actions
Pull requests that update GitHub Actions code
good first issue
Good for newcomers
help wanted
Extra attention is needed
invalid
This doesn't seem right
question
Further information is requested
testing wanted
wontfix
This will not be worked on
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!1831
Loading…
Reference in a new issue
No description provided.
Delete branch "super-build/upscaling-framegen"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This adds NVIDIA DLSS frame generation through NGX and gives more of OoT's moving models their own motion, so DLSS ghosts less behind them. It also offers AMD's FidelityFX FSR 4 where main's FSR 4 engine isn't built.
#1833, #1834 and this PR build only together, and should be reviewed and merged as one unit. The menu rows, the
vk_mainandvk_replayhooks and the build lists are in #1834; the neural rendering this chain feeds is in #1833.FIocker:super-build/allis the three together, rebuilt on main atd7a73062, and it's what was built and tested. #1830 is separate and builds alone.super-build/shaders(#1830)super-build/upscaling-framegen(#1831)super-build/neural(#1833)super-build/core(#1834)What it does
Temporal upscaler chain (
vk_temporal.cpp, on main's DLSS/FSR 4 chain):kSingle), two-influence sets (kPair), and rigid models bound to one bone without weights (kNoWeight), such as Navi's body, props and held items.dlss_actor.vertanddlss_actor.hread the new formats.temporal_upscale_recent()says whether a temporal upscaler drew in the last frames.NrFrame) instead of a flat depth and no motion. A temporal model then keeps the jitter the engine needs and doesn't ghost on camera motion.Frame generation:
vk_dlssg.cpp,vk_ngx.cpp), sharing DLSS's one NGX session (dlss_ngx_session).framegen_methods.h, with tests): Auto, NVIDIA DLSS and Built-in, of what runs here.RX_VK_FSR_FG=1, after #1635 and its duplicates: the HUD moved with the scene). This PR keeps that, and doesn't offer FidelityFX FSR 4's frame generation either, since it misbehaved the same way in play. The method enum keeps both values, so a saved choice reads as Auto and a future fix can bring them back.vk_slfg.cpp, env-only). It misbehaved in play, and its hooks in Reflex and the present were what main's tests tripped on.FSR 4 through AMD's FidelityFX runtime (
vk_ffx.cpp, MIT headers inthird_party/ffx-api): this is the Upscaler row's FSR 4 only in builds without main's FSR 4 engine (noHOOKSHOT_FSR). Wherevk_fsr4.cppis built and available, that engine takes the row.Window sharpening over a temporal upscaler is a fifth of the Sharpness row (
RX_VK_TEMPORAL_SHARPEN). At full strength over DLSS's motion-time softness, dense grass turned into an oily smear whenever the camera moved. Main has since fixed the black HDR picture under the scaling pass the same way this PR did (theSAMPLEDusage), so that part is gone from here.Camera motion pass (
vk_camera_motion.cpp) for the open NR backend, anddocs/upscaling.md.Main's recorder (#1473):
vk_camera_motion,vk_dlssgandvk_ffxincludevk_work.hlast, so theirvkCmd*go through it. NGX's and FidelityFX's own recording gets aB_DIRECTboundary first, asvk_dlss.cpphas.test_recorder_sync.pypasses.Testing
tests/run_all.pyon the three together, locally (clang targeting MSVC, the CPM Vulkan headers, glslang 11.12): 328 passed. The only failures are the same 18 that main atd7a73062has on this machine, all down to this PC's toolchain.frame generation method: NVIDIA DLSS.HOOKSHOT_FSRhere) and AMD GPUs.🤖 Generated with Claude Code
Heads-up: this conflicts with main now (a lot landed today). Could you merge main into the branch (or rebase) and resolve the conflicts? Thanks!
Rebased: the series is rebuilt on main at
d7a73062, the conflicts are resolved, and the descriptions are rewritten. The full replies to your reviews are on #1830, #1833 and #1834. For this PR:SAMPLEDfix: main fixed the black HDR picture under the scaling pass the same way, so ours is gone.test_recorder_sync.pypasses.#1831, #1833 and #1834 build only together, as the descriptions now say.
Thanks for the rebase and the rewritten description.
Overlap: #2118 adds the same DLSS Frame Generation through NGX, also on DLSS's session. Its description says it started from this PR's path, including its MvecScale of 1. Only one of the two should land, so they need to converge.
Scope. This PR bundles DLSS-G with FidelityFX FSR 4 through AMD's runtime (
vk_ffx.cppplus vendored ffx-api headers), more actor-motion layouts, neural-rendering hooks, a camera-motion pass and an 848-line doc. It also only builds together with #1833 and #1834, and #1834 currently conflicts with main. That makes it much harder to land than a focused PR.opts.framegen_methodis only in #1834'svk_menu.h.test_framegen_methods.pylooks for a menu row that is also only in #1834.I'd suggest splitting it: DLSS-G on its own, and the single/pair/no-weight actor-motion layouts on their own. #2118's author reports those layouts fix Link's shield and hair and suggests the same split. FSR 4 through FidelityFX would then be a third PR.
What #2118 does better on frame generation, and could be adopted here:
vk_dlssg.hsays so).MultiFrameCountMaxand leaves sets it can't make to the built-in method (see the inline note at 5x+).What this one does better:
Smaller points:
shutdown(), which isn't called in this PR.vk_slfg.cpp. That refers to the fork's earlier version, so it might be worth rewording.The decision on which of the two lands stays with @NotRexed.
@ -0,0 +168,4 @@const NVSDK_NGX_Result res =ngx::create_feature(cmd, NVSDK_NGX_Feature_FrameGeneration, params, &feature);if (NVSDK_NGX_FAILED(res) || !feature) {logf("vk: dlssg: the feature was not created for %ux%u (0x%08x)", g.width, g.height,When creation fails (for example with the device made without frame generation's extensions, which
asked()notes can happen when the row is turned on mid-game),ngx_featureretries and logs this line on every present with no cap. Consider giving up after a few failures until the extents change.@ -0,0 +209,4 @@// uses; the camera-motion pass uses the same pair and its picture agreed with the game's, which// is the only check available without NVIDIA's own tooling.if (in.have_matrices) {ngx::set(params, NVSDK_NGX_DLSSG_Parameter_ClipToPrevClip, (void*)in.clip_to_prev_clip);vk_dlssg.hsays the matrices are row major and "transposed on the way in", but here they are handed to NGX as they are. #2118 transposes the samedlss_math::reprojectionoutput (itsfg_put, "NGX's matrices are rows of a row-vector convention"). One of the two is wrong. Could you check which convention NGX expects and make the code and the header comment agree?@ -0,0 +331,4 @@g.render_height == render_height && g.present_format == present_format)return true;// Extents or format changed: the feature was made for the old ones.ngx_drop_feature();On a resize or multiplier change, this releases the NGX feature and then
drop_images the made frames immediately. The previous present's command buffer (its DLSS-G evaluation and the copy out ofg.made[]) can still be in flight, so both the feature and the images can be freed while the GPU uses them. They need a queue idle first (as main'svk_dlss.cpprelease and #2118'sfg_releasedo), ortemporal_later-style deferral for the images.At 5x to 8x,
wantis capped at 3 (kMaxMade).dlssg_framesthen returns 3,set->madebecomes 3, and the set is shown as 4 frames while the row says 5x+, so the built-in method never fills the rest. It also doesn't read NGX'sMultiFrameCountMax(1 on RTX 40), so 3x/4x there depend on NGX refusing each set. Leaving the set to the next method whenm - 1is more than DLSS-G can make would avoid both.@ -0,0 +41,4 @@static wchar_t wide[MAX_PATH + 1];static const bool once = []() {char path[MAX_PATH + 1]{};const DWORD n = GetModuleFileNameA(nullptr, path, MAX_PATH);GetModuleFileNameAplus widening byte by byte breaks for a game folder with non-ASCII characters (ANSI code page, against the UTF-8 path rule in AGENTS.md /test_utf8_paths.py). It also differs from the data pathngx_initgives NGX (game_dir() / "ngx"). Reusinggame_dir()and the same data folder would fix both.Thanks for all the work on this, @FIocker. We're going with #2118 for DLSS frame generation for now, so I'm closing this one.
#2118 builds on its own, its CI is green and testers have run it on RTX 50 cards. It also already has the 3x+ pacing and Reflex. This branch needs #1834 to build, and it bundles several separate changes. The parts this PR did better are listed on #2118 as suggestions, credited to you, so the ideas aren't lost.
The other pieces here (more actor motion for DLSS, the FidelityFX FSR 4 fallback) would be very welcome as small, separate PRs on top of main.
Thanks, @hckrman101, and for crediting the ideas on #2118. As suggested, the single, pair and no-weight actor-motion layouts are now their own PR on main: #2787. The neural rendering in #1833 is rebuilt on #2118's NGX session.
github-sync referenced this pull request2026-11-12 13:38:29 +00:00
github-sync referenced this pull request2026-11-12 13:39:06 +00:00