feat(dlss): DLSS Frame Generation through NGX, evenly paced at 3x and up #2118
No reviewers
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!2118
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/dlss-frame-gen-ngx"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
On an RTX 5090 at 5120x2160 the built-in frame generation couldn't follow motion. Its block search covers ±32 px per game frame, so a camera pan's vectors fell to zero and every made frame was a cross-fade of two game frames: smoother counters, same stepping motion. This PR makes those frames with NVIDIA's DLSS Frame Generation through NGX, from the temporal upscaler's own depth and motion. It also fixes two pacing problems that made 3x feel uneven even once the frames were right.
What changed (five commits, each on its own)
fix(vk): present 3x and 4x sets ahead too. #1702's present-ahead (a set's copies go to the GPU together, and only the presents wait for their times) ran for 2x only. At 3x and 4x each made frame's copy still queued behind the game's next frame and reached the screen unevenly. It now runs for any paced set the swapchain can hold, and at 3x/4x the swapchain asks for one spare image per frame (5 and 6 images on NVIDIA).feat(dlss): DLSS-G through NGX (vk_dlss.cpp, beside DLSS on the same NGX session, no Streamline):nvngx_dlssg.dllis looked for where DLSS's runtime is.RX_VK_DLSS_FG=0/1overrides it for a run.fix(vk): V-Sync Auto presents FIFO with DLSS Frame Generation at 2x and up. Auto was mailbox, which on NVIDIA is Fast Sync: below the refresh rate (~135 shown on 165 Hz) frames land on the refresh grid, so evenly timed presents still judder. With DLSS-G making the sets, Auto now asks for FIFO, which a VRR display shows as presented. The built-in frame generation and other GPUs keep Auto as before (from review, item 5). On and Off are unchanged. The swapchain is rebuilt when this changes, as it already is for V-Sync.feat(vk): NVIDIA Reflex with frame generation (added after review). Reflex was off whenever frame generation was on, so its held-back frame added its whole delay (nullifyshot measured +12 to +18 ms at 4x on an RTX 5080). The game's frame calls stay on: the sleep, the simulation markers and render submission. Each set keeps the game frame it was made at, and only that frame is presented with its present id and present markers; made frames get neither. Present ids only go up. Without frame generation nothing changes. On the 5090 at 3x, Reflex's own timings now appear under DLSS-G: simulation start to present end ~41 ms, against ~16 ms with frame generation off (details in the comments).fix(dlss): DLSS-G only where it runs (from review). NGX isn't asked for frame generation's device extensions, or started at a present, on a device without NVIDIA's extensions. V-Sync Auto's FIFO is for DLSS-G only. Frame generation at Off or Auto lets DLSS-G's feature go. A barrier follows a refused frame before the built-in passes write the same images.How it was tested
tests/run_all.py(Windows, at90c46837): 340 passed, 28 skipped, 3 failed. The same 3 also fail on untouched main at this branch's base here:test_recorder_sync's exit case;test_precompile_compute_guard;test_finish_output(aNameErrorinmake_game.py; it passes on current main).test_present_ahead.py: 3x/4x with and without enough spare images.test_vsync_row.py: the present-mode choice with DLSS-G and with the built-in frame generation.test_upscaler_rows.py: the new row order.present_source_crop_test.cppandtest_present_semaphores.py: stubs for the new calls.test_recorder_sync.py: knows NGX's DLSS-G create/evaluate calls as direct recordings with their B_DIRECT boundary.Not tested / known
VkLatencySubmissionPresentIdNVshould fix it; I'll do that as its own PR.🤖 Generated with Claude Code
Tested exact head
ec27f1fdon an RTX 5070 Ti / Windows 11, 3440x1440 output, DLSS Quality 2294x960, game cap 60, FG 2x, pacing on. Fresh guest generation and matching API 14 host/backend build. NGX loaded the pinned official SR runtime and the DLSS-G runtime supplied with Streamline 2.14.16 from an isolated ngx directory; no Streamline interposer was loaded for this path.NGX created its feature and logged repeated 600/600 sets made by DLSS-G, with zero refused and no fallback in those windows. Three controls used the same host/backend, copied checkpoint and active window. The controller reported 3440x1440 at 159 Hz; all three swapchains logged FIFO (V-Sync Auto). PresentMon 2.6 recorded actual display timestamps, separately from the native-present trace; no pixel capture during timing.
Native rates stayed around 60 in all modes. Displayed P95 in the first return second: 18.78 ms off, 12.51 ms built-in, 12.52 ms NGX. There was no missing display feedback in those measurement windows. Normal exit from inventory returned 0 with NGX active, and both controls also exited normally. Presented hearts/instructions and inventory text were legible in the frames inspected after timing.
This confirms the NGX path runs on this RTX 50 setup. It is one run per mode in the initial interior scene, not a latency, motion-quality, 3x/4x, fixed-refresh, or campaign test. With the game capped at 60, both FG methods reach about 120 displayed/s here; these measurements do not establish an overall winner. PresentMon marked frames Application, so individual generated-frame identity comes from neither that label nor the UI screenshots; the generator attribution above comes from the NGX set log.
I tested this on an RTX 5080 (1920x1080 at 360 Hz, G-SYNC, driver V-Sync on, in-game V-Sync Off) at 4x, measuring input latency with DLSS-G against frame generation off. DLSS-G worked well: it made every set in gameplay ("600 made by DLSS-G" in each window). It adds a fair amount of latency, though, and part of that is because NVIDIA Reflex is off while it runs.
How I measured. The game pressed its own buttons through
HOS_INPUT_LIVE. A local log line (not part of this PR) recorded when each press was due, on the QPC clock. Desktop Duplication timed the first composed frame that showed the reaction, usingLastPresentTimeon the same clock. That covers the game's input read through to the frame on the desktop. It leaves out the controller and the monitor. I used 20 presses per action, with Link idle in the same spot each time.These are medians. The 10–90% range was about ±5 ms around each, and the two modes don't overlap. The game's own frame rate was about 77 fps with DLSS-G and about 99 fps with frame generation off.
Reflex is off whenever frame generation is on, DLSS-G included.
on()ingpu/vk/vk_reflex.cpprequires!framegen_on(). That check gates the sleep, every marker and the timing log. Anti-Lag 2 has the same check. This PR doesn't touch either file, andvk_dlss.cppnever calls into Reflex.RX_VK_REFLEX_LOG=1, a DLSS-G session wrote no Reflex timing report in over 10,000 frames. The log writes one every 300 frames while Reflex runs.NVIDIA Reflex: on + boostis still logged once after each swapchain rebuild. It looks like the sleep mode is re-applied when the latency state is reset. So the driver may keep Boost even though nothing is paced.The frame-generation-off run had Reflex On + Boost and the DLSS-G run had none, so the gap above includes both frame generation's held-back frame and Reflex's pacing. NVIDIA's own integrations (Streamline) require Reflex while DLSS-G is on, to win back exactly this latency, so it seems worth having here too.
A possible fix. As far as I can tell, Reflex is off under frame generation because its per-frame calls assume one present per game frame. Its calls fall into two groups, and only the second actually conflicts:
Game-frame calls, all on the presentation thread in
do_present:SIMULATION_ENDandRENDERSUBMIT_START(reflex_present_begins, vk_replay.cpp:2940)SIMULATION_STARTandINPUT_SAMPLE(ReflexSleep, vk_replay.cpp:2941)RENDERSUBMIT_END(reflex_submit_ends, vk_replay.cpp:3678)These still describe the game's real frames with frame generation on, and the sleep that paces them is where the latency gets won back.
Present calls: the present id and
PRESENT_START/PRESENT_END(reflex_present_id, vk_replay.cpp:3704, andreflex_presented, :3724). These run in the present path the display thread also uses for made frames, so they would mark made frames as if they were game frames. This is the part that clashes with frame generation.So
on()could be split into two checks:framegen_on(), and it gates the sleep and the simulation and render-submit markers.reflex_present_idandreflex_presented.RX_VK_REFLEX_LOG=1would show it working: timing reports with frame generation on. Rerunning the measurement above would show what it wins back. Anti-Lag 2 could probably get the same split. I haven't tried any of this, so take it as a direction, not a tested patch.Thanks @Arkex-ai for the 5070 Ti run, and @breakthetargets for the latency numbers and the Reflex write-up. I pushed your split as a new commit here (
feat(vk): NVIDIA Reflex with frame generation):do_present, and render-submit end once the set's recording is submitted (reflex_game_submitted).reflex_frame()). On the display thread only the set's real frame is presented with that present id and PRESENT_START/END. Made frames get neither. A deferred (present-ahead) present carries it in itsPendingPresent.test_reflex_swapchain_lifetime.pycovers the new cases.On the 5090 (5120x2160, DLAA, Reflex On + Boost, same spot), Reflex's own timings (
RX_VK_REFLEX_LOG=1) now show up under DLSS-G. Before, it wrote none:That's Reflex's view, so it can't show what turning Reflex on under frame generation wins back: with Reflex off there's no timing to compare against. @breakthetargets, could you rerun your input-to-screen measurement on the new head? That's the comparison that counts.
Most of the remaining gap is the interpolation itself: the real frame is shown after its set's made frames, about ⅔ of a game frame later at 3x.
Thanks for the quick follow-up. I reran the same test on
3a39e3f, with the same PC, settings, spot and method as my comment above. This timeRX_VK_REFLEX_LOG=1was set in both modes.Reflex now runs with DLSS-G. The DLSS-G run wrote 59 Reflex timing reports while DLSS-G made every set. Before this commit it wrote none.
3a39e3f)3a39e3fThese are medians of 20 presses each. The walk gap shrank by about 5 ms and the sword gap by about 2 ms, but part of that may be run-to-run noise. Overall, the latency barely changed. Reflex's own timings show why:
So the change looks right. It should pay off in GPU-bound scenes, where frames do queue up: higher resolution, heavier settings or a slower card.
One thing looks off: the GPU-end timing with frame generation. With frame generation on, Reflex's "to the GPU's end" reads 243–442 ms, next to about 21 ms from simulation start to present end. With frame generation off it reads a sensible 12.1 ms. It looks like the driver is pinning the GPU's end of a frame on later work.
A possible fix (untested, just a direction). Hookshot never chains
VkLatencySubmissionPresentIdNVto its queue submits. That's VK_NV_low_latency2's way of telling the driver which frame a submission's GPU work belongs to. Without frame generation the driver seems to infer it correctly. With frame generation, the display thread's copy and DLSS-G submits sit between the game's frames, and it probably gets them wrong. What I'd try is tagging every submit through one helper, much likereflex_present_id, for examplereflex_submit_id(const void* next):submit_recording, including the batched path throughqueue_flush_locked, and the parallel recording jobs' submit (vk_replay.cpp around line 1555). ChainVkLatencySubmissionPresentIdNVwith the frame being made (r.idwhileframe_marked, between its simulation start and its render-submit end).present_image's submit (around line 3682): use the same rule as the present id. That'sr.idwithout frame generation, the set's game frame (reflex_present_frame) for the present that carries it, and no tag for made frames. That way the made frames' copy and DLSS-G work isn't counted as the next game frame's rendering.If this is the cause,
RX_VK_REFLEX_LOG=1should show "to the GPU's end" back below "simulation start to present end" with frame generation on, as it is with frame generation off. The driver would then also be pacing on correct GPU timings. I haven't tried this, so it's a lead to check, not a confirmed diagnosis.Thanks for this. It's focused, it builds alone, CI is green, and it has been tested on three RTX 50 cards (Arkex-ai's 5070 Ti run, nullifyshot's latency numbers, and the Reflex follow-up you added after them).
Overlap: #1831 adds the same feature (DLSS-G through NGX, on DLSS's session). Only one of them should land, so the two need to converge. Your description already says you're open to folding this into either.
Compared against #1831's diff, this PR:
MultiFrameCountMaxand leaves sets it can't make to the built-in method. #1831 hard-codes 3 made frames and quietly makes a 4x set at 5x to 8x.What #1831 does better, and could be adopted here:
framegen_methods.h) with a unit test. The newFrame generation byrow here has no test of its own beyond the row order.Inline notes below. Still open from the thread: nullifyshot saw Reflex's "to the GPU's end" read 243 to 442 ms with frame generation on, and suggested tagging submits with
VkLatencySubmissionPresentIdNV. That's fine as a follow-up, but please say so in the description so it isn't lost.The decision on which of the two lands stays with @NotRexed.
@ -447,0 +761,4 @@static const float id[16] = {1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1};const bool history = g.have_last && motion && !f.reset;float to_prev[16], from_prev[16];if (!history || !dlss_math::reprojection(vp, g.last_vp, to_prev)) std::memcpy(to_prev, id, sizeof id);This calls
NVSDK_NGX_VULKAN_GetFeatureDeviceExtensionRequirementsat every device creation in a DLSS build, on AMD and Intel GPUs too and with frame generation off. Main'sdlss_device_extensionsanddlss_probeavoid NGX unless the device has the NVX extensions (DLSS_NEEDS_RTX). Could this get the same gate, i.e. skip whenVK_NVX_binary_import/VK_NVX_image_view_handlearen't inall?fg_startcould also return early whendlss_why()says the device isn't NVIDIA's, so the first 2x present on an AMD card doesn't runngx_init.Nit: when frame k>1 is refused, frames 1..k-1 have already been written by NGX into
out[], and the built-in passes then write the same images in the same command buffer. No barrier is recorded on this path, so that's a write-after-write without a dependency. Afg_barrier(cmd)before returning would cover it.Nit: the feature is only let go when the
Frame generation byrow changes. With Frame generation set to Off or Auto,dlss_fg_framesisn't called anymore (m > 1), so DLSS-G's feature (large at 5K) stays allocated for the rest of the run. Releasing it when the multiplier drops below 2 would match the row's "changes at once".This changes V-Sync Auto for every GPU and for the built-in frame generation too, not only DLSS-G. On Windows it also skips the immediate fallback that the comment below keeps for AMD ("FIFO held anything under 60 fps at 30"). The description lists fixed-refresh displays as untested. Could this be limited to where it was measured (DLSS-G running, or at least NVIDIA), or else tested at 2x on a fixed-refresh/AMD setup?
@ -447,0 +761,4 @@static const float id[16] = {1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1};const bool history = g.have_last && motion && !f.reset;float to_prev[16], from_prev[16];if (!history || !dlss_math::reprojection(vp, g.last_vp, to_prev)) std::memcpy(to_prev, id, sizeof id);Done in
90c46837(fix(dlss): DLSS-G only where it runs).dlss_fg_device_extensionsnow returns before NGX when the device lacksVK_NVX_binary_importorVK_NVX_image_view_handle, the same check asdlss_device_extensions.fg_startreturns beforengx_initwhendlss_why()isn'tDLSS_OK, so the first 2x present on an AMD card doesn't start NGX. On an NVIDIA device NGX is still asked with frame generation off, so the Frame generation by row works without a restart.Limited to DLSS-G, where it was measured (
90c46837).swapchain_dlssg()is true only at a fixed multiplier with DLSS-G asked for, on a device it runs on, and NGX not having said no. The built-in frame generation and other GPUs keep Auto as on main, including the Windows immediate fallback for AMD. The swapchain is made again when that changes.test_vsync_rownow covers both cases. A fixed-refresh display is still untested, as the description says.Fixed in
90c46837:fg_barrier(cmd)before returning when frame k > 1 is refused.Fixed in
90c46837:dlss_fg_idle()runs at each present without a fixed multiplier (Off or Auto) and lets the feature go if one is held. When there's nothing to release it's one atomic load.Thanks for the review. The four inline points are fixed in a new commit,
90c46837fix(dlss): DLSS-G only where it runs(replies in each thread). The Reflex submit tagging (VkLatencySubmissionPresentIdNV) is now in the description as a follow-up.From #1831's side:
Agreed that the choice between this and #1831 is @NotRexed's.
Thanks, re-reviewed at
fb09afc8(90c46837plus a merge of main).All four earlier points are fixed:
VK_NVX_binary_import/VK_NVX_image_view_handle, andfg_startstops beforengx_initwhendlss_why()isn'tDLSS_OK.swapchain_dlssg()), so the built-in frame generation and AMD keep main's behaviour, including the Windows immediate fallback.test_vsync_rowcovers both cases, and the swapchain is rebuilt when it flips.fg_barrier(cmd)before the built-in passes write the same images.dlss_fg_idle()lets the feature go at Off/Auto. It runs on the presentation thread, wheredlss_fg_framesalso runs, and takesg.muthenpresent_muin the same order, so I don't see a lock-order problem.The Reflex submit tagging (
VkLatencySubmissionPresentIdNV, nullifyshot's GPU-end reading) is now in the description as a follow-up, which is what I asked for.One nit, inline: after three failed feature creations,
dlss_fg_runs()still says yes, so V-Sync Auto stays FIFO while the built-in method makes the sets. That's an edge case and doesn't block.From #1831, worth adopting. #1831 is being closed in favour of this PR. These are FIocker's ideas, and none of them blocks this one. They would be good follow-ups, with credit:
vk_camera_motion), so its DLSS-G runs with the Upscaler off. Here DLSS-G needs DLSS, DLAA or FSR. You've said this could be a follow-up, and that's fine.framegen_methods.h). In #1831, an unknown saved value reads as Auto, and the row only offers methods that run on this PC. Here, a saved out-of-range value clamps to Built-in, and an AMD card sees Auto and Built-in, which do the same thing. You offered to do this, and I'd take it, here or in the follow-up.dlss_actor.vert/bones_used(the single, pair and no-weight layouts: Link's shield and hair, rigid props). That helps DLSS and DLSS-G alike, and you plan it as its own PR.90c46837).CI on
fb09afc8was still queued when I reviewed, so please merge once it's green. Approving.@ -447,0 +759,4 @@fg_mul(ip, vp, view);if (!fg_inverse(view, cam)) return;static const float id[16] = {1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1};const bool history = g.have_last && motion && !f.reset;Nit: after
create_failures >= 3,dlss_fg_framesrefuses every set, but this still returns true, so V-Sync Auto stays FIFO while the built-in method makes the sets. Settingstart_failed(or a separate flag) once creation gives up would hand Auto back to main's mode.@ -447,0 +761,4 @@static const float id[16] = {1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1};const bool history = g.have_last && motion && !f.reset;float to_prev[16], from_prev[16];if (!history || !dlss_math::reprojection(vp, g.last_vp, to_prev)) std::memcpy(to_prev, id, sizeof id);Confirmed in
90c46837, thanks: both the extension query andfg_startare now gated.Confirmed: FIFO is now for DLSS-G only, and the built-in method and AMD keep main's Auto. Thanks.
@ -447,0 +759,4 @@fg_mul(ip, vp, view);if (!fg_inverse(view, cam)) return;static const float id[16] = {1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1};const bool history = g.have_last && motion && !f.reset;Fixed in
7c13821d: once creation gives up (create_failures >= 3),start_failedis set, sodlss_fg_runs()says no and the swapchain goes back to V-Sync Auto's mode without DLSS-G. Main is merged in again too (b3713596), for the conflict with #2042's menu rows.Thanks for the approval. Main is merged in again (#2042 touched the Reflex row), and the nit is fixed in
7c13821d. I'll take the method choice as a small header with a unit test in the follow-up, together with DLSS-G without a temporal upscaler, and the single/pair/no-weight actor-motion layouts as their own PR, with credit to FIocker.Re-checked at
fd6aaa5. The only code change since my approval is7c13821d(the rest are main merges). Settingstart_failedonce creation gives up at three failures makesdlss_fg_runs()say no, so V-Sync Auto goes back to its normal mode. That's what my nit asked for, and it matches the existing "off" log line (it was already permanent after three failures). Thanks for taking the #1831 ideas on as follow-ups.CI is still running on this head after the main merges (#2042 touched the Reflex row), so it's worth waiting for green before merging.
github-sync referenced this pull request2026-11-12 06:22:54 +00:00