fix(vk): resolve DLSS permanent failure latching and improve dynamic resolution handling #2574

Merged

dani10gl merged 1 commit from fix/dlss-dynamic-res-and-recovery into main

2026-11-12 03:05:28 +00:00

dani10gl commented

2026-11-11 23:54:09 +00:00

(Migrated from github.com)

Summary

This PR addresses several issues in the Vulkan temporal upscaler pipeline (vk_temporal.cpp, vk_dlss.cpp, vk_menu.cpp, dlss_checks.h) where NVIDIA DLSS would either fail to instantiate during dynamic resolution scaling (DRS) or permanently latch into a disabled state (engine.failed = true), silently falling back to bilinear upscaling without any user-facing recovery path.

Problem Description & Root Causes

Permanent Failure Latching in vk_temporal.cpp:
- When a transient evaluation error occurs (e.g. during screen transition, window resize, or dynamic resolution shift), vk_temporal.cpp unconditionally sets engine.failed = true.
- Once latched, temporal_upscale() perpetually rejects all future frames with "it failed earlier".
- Changing options in the Insert menu (switching between presets, render resolutions, or upscaler modes) never cleared this state, leaving DLSS dead for the remainder of the application lifecycle.
Off-by-One Rejection in dlss_checks.h:
- In scene_size_fits(), (uint64_t)w * 3 >= ow fails for fractional roundings at 33% Ultra Performance scaling (e.g., 853 \times 3 = 2559 < 2560).
- A tolerance margin of 2 pixels prevents false rejections of valid render targets.
retry flag left false for DLSS in vk_dlss.cpp:
- TemporalEngine contains a retry boolean intended for transient evaluation drops. Setting this to true allows DLSS to retry on subsequent frames if a single frame evaluation fails, rather than permanently falling back.

Changes

gpu/vk/dlss_checks.h:
- Adjusted scene_size_fits() to (uint64_t)(w + 2) * 3 >= ow && (uint64_t)(h + 2) * 3 >= oh to safely handle rounding in fractional 3x downscaling.
gpu/vk/vk_dlss.cpp:
- Set retry = true in TemporalEngine kEngine for DLSS to ensure transient evaluation failures reset history on the next frame instead of shutting down the engine permanently.
gpu/vk/vk_temporal.h & gpu/vk/vk_temporal.cpp:
- Implemented void temporal_reset_failure() to clear engine.failed across active engines, reset s.reason, and allow clean reinitialization.
gpu/vk/vk_menu.cpp:
- Hooked temporal_reset_failure() into set_row() when modifying opts.dlss, opts.resolution, or opts.dlss_model. Modifying scaling settings in the Insert menu now cleanly clears previous errors and hot-reloads the upscaler.

Verification

Tested with Hookshot v2.1.0 on Vulkan with an NVIDIA RTX GPU using DLSS runtime nvngx_dlss.dll (v310.9.16).
Verified seamless switching between:
- Dynamic Resolution ("Auto") and fixed targets (33%, 50%, 58%, 67%, 75%, 83%, 100%).
- DLSS models (Presets K, M, L, Default).
- Switching between FSR 3.1 and DLSS without image degradation or permanent fallback to bilinear scaling.
Compiled clean on LLVM-MinGW x86_64 with zero warnings or errors.

### Summary This PR addresses several issues in the Vulkan temporal upscaler pipeline (`vk_temporal.cpp`, `vk_dlss.cpp`, `vk_menu.cpp`, `dlss_checks.h`) where NVIDIA DLSS would either fail to instantiate during dynamic resolution scaling (DRS) or permanently latch into a disabled state (`engine.failed = true`), silently falling back to bilinear upscaling without any user-facing recovery path. --- ### Problem Description & Root Causes 1. **Permanent Failure Latching in `vk_temporal.cpp`:** - When a transient evaluation error occurs (e.g. during screen transition, window resize, or dynamic resolution shift), `vk_temporal.cpp` unconditionally sets `engine.failed = true`. - Once latched, `temporal_upscale()` perpetually rejects all future frames with `"it failed earlier"`. - Changing options in the Insert menu (switching between presets, render resolutions, or upscaler modes) never cleared this state, leaving DLSS dead for the remainder of the application lifecycle. 2. **Off-by-One Rejection in `dlss_checks.h`:** - In `scene_size_fits()`, `(uint64_t)w * 3 >= ow` fails for fractional roundings at 33% Ultra Performance scaling (e.g., $853 \times 3 = 2559 < 2560$). - A tolerance margin of 2 pixels prevents false rejections of valid render targets. 3. **`retry` flag left `false` for DLSS in `vk_dlss.cpp`:** - `TemporalEngine` contains a `retry` boolean intended for transient evaluation drops. Setting this to `true` allows DLSS to retry on subsequent frames if a single frame evaluation fails, rather than permanently falling back. --- ### Changes - **`gpu/vk/dlss_checks.h`**: - Adjusted `scene_size_fits()` to `(uint64_t)(w + 2) * 3 >= ow && (uint64_t)(h + 2) * 3 >= oh` to safely handle rounding in fractional 3x downscaling. - **`gpu/vk/vk_dlss.cpp`**: - Set `retry = true` in `TemporalEngine kEngine` for DLSS to ensure transient evaluation failures reset history on the next frame instead of shutting down the engine permanently. - **`gpu/vk/vk_temporal.h` & `gpu/vk/vk_temporal.cpp`**: - Implemented `void temporal_reset_failure()` to clear `engine.failed` across active engines, reset `s.reason`, and allow clean reinitialization. - **`gpu/vk/vk_menu.cpp`**: - Hooked `temporal_reset_failure()` into `set_row()` when modifying `opts.dlss`, `opts.resolution`, or `opts.dlss_model`. Modifying scaling settings in the Insert menu now cleanly clears previous errors and hot-reloads the upscaler. --- ### Verification - Tested with **Hookshot v2.1.0** on Vulkan with an NVIDIA RTX GPU using DLSS runtime `nvngx_dlss.dll` (v310.9.16). - Verified seamless switching between: - Dynamic Resolution ("Auto") and fixed targets (33%, 50%, 58%, 67%, 75%, 83%, 100%). - DLSS models (Presets K, M, L, Default). - Switching between FSR 3.1 and DLSS without image degradation or permanent fallback to bilinear scaling. - Compiled clean on LLVM-MinGW x86_64 with zero warnings or errors.