fix(vk): DLSS at handheld Ultra Performance, and NGX paths on Linux #1364

Merged

hckrman101 merged 1 commit from dlss-handheld-paths into main

2026-11-09 23:17:11 +00:00

hckrman101 commented

2026-11-09 23:11:27 +00:00

(Migrated from github.com)

Two fixes for NVIDIA DLSS (#841), both found by the validation report on #841. Thanks for the careful testing.

Handheld + Ultra Performance ran the game's own upscale

The handheld profile's output is 1536x864, so Ultra Performance draws the scene at a third of that: 512x288. dlss_note_scene_draw only took scenes of at least 640x360, so that scene never gave DLSS its camera. The game's own upscale ran while the menu said DLSS. dlss_skin_check (actor motion) and the RX_VK_DLSS_TRACE diagnostic had the same fixed floor.

The floor now follows the gameplay upscale's output, as last seen. A scene is taken when the output is 1 to 3 times its size on each axis, from DLAA to Ultra Performance. This is the same rule dlss_upscale already checks before it draws (ow < rw || ow > rw * 3). It is in dlss_checks.h now and used in all three places. Before the first upscale the output is not known, so nothing is taken. That first frame runs the game's upscale, as it already did.

The 640x360 floor was a cheap filter against other passes that look like the scene: B10G11R11 colour plus depth plus a camera, such as the pause screen's model views or reflections. These checks still turn them away:

the new floor: passes below a third of the output, such as 256x144 or a 384x384 target at 1536x864;
the colour format and the camera's own checks (its size field matches the target, a perspective projection);
the largest such target in a frame wins;
at the upscale, the input must be this frame's scene size, the depth must match, and the actor draws must use the scene's depth.

For docked 2048x1152 the new floor is 683x384, a little stricter than before.

Linux: NGX paths with non-ASCII characters

The Linux wide() widened each byte of the UTF-8 path on its own, so Café reached NGX as CafÃ©. It now decodes UTF-8 itself, without the process locale. Bytes that are not UTF-8 become U+FFFD, one per bad sequence, as Unicode recommends. I chose U+FFFD over keeping the raw byte: a raw byte such as 0xE9 would become U+00E9, which NGX writes back as the bytes for é, a different and possibly real path. A path with U+FFFD names nothing, so it can only fail.

Decoding alone is not enough. NGX's Linux library (libnvsdk_ngx.a, WStrToUTF8Str) turns every wide path back into bytes with newlocale(LC_ALL, "C") and wcsrtombs. In the C locale that fails on any character past ASCII, and the path comes out empty. I checked this by calling that function directly. So:

A folder whose path is not all ASCII now goes to NGX as /proc/self/fd/<n>: the folder is opened, kept open, and NGX gets an ASCII name for it. Nothing is written to disk. NGX's data folder (the game folder's ngx) works this way, and NGX finds the runtime through it.
The driver's part (libnvidia-ngx.so) then loads the runtime by its real path (realpath), and that fails the same way. So on Linux the DLSS runtime itself must be in a folder whose real path is ASCII. When it is only in other folders, the log and the menu now say so and name the fix. Before, they said "DLSS needs an NVIDIA RTX GPU". The fix is to put libnvidia-ngx-dlss.so.* in $XDG_DATA_HOME/hookshot/ngx or in another ASCII folder named by RX_VK_DLSS_DIR. The readme says this too.

Windows is unchanged. Its paths are wide end to end (GetModuleFileNameW, _wgetenv, fs::path::native()).

How it was tested

tests/test_dlss_checks.py (new) compiles and runs tests/test_dlss_checks.cpp with -Wall -Wextra -Werror (g++ and clang++):
- Accepted: 512x288 for 1536x864 (the report's case), 683x384 for 2048x1152, and every DLSS mode and DLAA at every output the menu has (using the menu's twelfths formula).
- Rejected: an unknown output, under a third on either axis (511x288, 512x287), 384x384, 256x144, 640x360 at 2048x1152, and larger than the output.
- UTF-8: ASCII, accented, CJK and 4-byte paths; a lone Latin-1 byte, stray continuation bytes, cut sequences, overlong forms, surrogates, values past U+10FFFF and F5-FF. All run in the C locale.
- It also checks that the sources no longer have a fixed 640/360 floor and that wide() widens byte by byte only for ASCII paths.
tests/run_all.py: 176 passed, 5 skipped. One check, nx2/test_nup_verify.py, timed out once under load (a 5 s subprocess limit) and passed when run again. It is unrelated.
hookshot_vk.dll cross-built with llvm-mingw and the NGX SDK (as the vulkan-dlss job does): compiles and links.
Headless runs on Linux, RTX 3090, NGX runtime 310.9.16:

Build	Folder	Setup	Result
main	ASCII	handheld, DLSS, Ultra Performance	no DLSS at all: the game's upscale throughout, no DLSS part in the status lines
this PR	ASCII	handheld, DLSS, Ultra Performance	`NVIDIA DLSS draws OoT's upscale: 512x288 -> 1536x864`, `feature 1 made ... 512x288 ... -> 1536x864, mode 3`, status `DLSS: 600 frames drawn ... actor motion 48924 draws in 600 frames`
this PR	ASCII	docked, DLSS Quality	`1366x768 -> 2048x1152`, `600 frames drawn`
main	`Hookshot-dlss-utf8-Café-ゼルダ`	docked, DLSS Quality	`not available on this GPU (0xbad00004): DLSS needs an NVIDIA RTX GPU` on an RTX 3090. 0xbad00004 is NGX's FeatureNotFound
this PR	same	same	`... is only in folders whose path is not all ASCII (...) ... put it in $XDG_DATA_HOME/hookshot/ngx, or RX_VK_DLSS_DIR=<an ASCII folder>`, and the menu says the same
this PR	same, runtime in an ASCII folder via `RX_VK_DLSS_DIR`	docked, DLSS Quality	`NGX started`, `NVIDIA DLSS draws OoT's upscale: 1366x768 -> 2048x1152`, `528 frames drawn` in the first status line

With RX_VK_DLSS_LOG=1, NGX's own lines show both steps. Through the alias it finds the runtime (Loading most recent 'libnvidia-ngx-dlss.so' snippet from '/proc/self/fd/135'... Found 1 snippets). Then the driver cannot load it from the real path (failed to load signed snippet - Unable to open fd).

Not tested: Windows end to end (no change there), and the U+FFFD fallback in a real run. That fallback only applies when a non-ASCII folder cannot be opened, and NGX would not find anything there anyway.

Two fixes for NVIDIA DLSS (#841), both found by the [validation report on #841](https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/841). Thanks for the careful testing. ## Handheld + Ultra Performance ran the game's own upscale The handheld profile's output is 1536x864, so Ultra Performance draws the scene at a third of that: 512x288. `dlss_note_scene_draw` only took scenes of at least 640x360, so that scene never gave DLSS its camera. The game's own upscale ran while the menu said DLSS. `dlss_skin_check` (actor motion) and the `RX_VK_DLSS_TRACE` diagnostic had the same fixed floor. The floor now follows the gameplay upscale's output, as last seen. A scene is taken when the output is 1 to 3 times its size on each axis, from DLAA to Ultra Performance. This is the same rule `dlss_upscale` already checks before it draws (`ow < rw || ow > rw * 3`). It is in `dlss_checks.h` now and used in all three places. Before the first upscale the output is not known, so nothing is taken. That first frame runs the game's upscale, as it already did. The 640x360 floor was a cheap filter against other passes that look like the scene: B10G11R11 colour plus depth plus a camera, such as the pause screen's model views or reflections. These checks still turn them away: - the new floor: passes below a third of the output, such as 256x144 or a 384x384 target at 1536x864; - the colour format and the camera's own checks (its size field matches the target, a perspective projection); - the largest such target in a frame wins; - at the upscale, the input must be this frame's scene size, the depth must match, and the actor draws must use the scene's depth. For docked 2048x1152 the new floor is 683x384, a little stricter than before. ## Linux: NGX paths with non-ASCII characters The Linux `wide()` widened each byte of the UTF-8 path on its own, so `Café` reached NGX as `CafÃ©`. It now decodes UTF-8 itself, without the process locale. Bytes that are not UTF-8 become U+FFFD, one per bad sequence, as Unicode recommends. I chose U+FFFD over keeping the raw byte: a raw byte such as 0xE9 would become U+00E9, which NGX writes back as the bytes for `é`, a different and possibly real path. A path with U+FFFD names nothing, so it can only fail. Decoding alone is not enough. NGX's Linux library (`libnvsdk_ngx.a`, `WStrToUTF8Str`) turns every wide path back into bytes with `newlocale(LC_ALL, "C")` and `wcsrtombs`. In the C locale that fails on any character past ASCII, and the path comes out empty. I checked this by calling that function directly. So: - A folder whose path is not all ASCII now goes to NGX as `/proc/self/fd/<n>`: the folder is opened, kept open, and NGX gets an ASCII name for it. Nothing is written to disk. NGX's data folder (the game folder's `ngx`) works this way, and NGX finds the runtime through it. - The driver's part (`libnvidia-ngx.so`) then loads the runtime by its real path (`realpath`), and that fails the same way. So on Linux the DLSS runtime itself must be in a folder whose real path is ASCII. When it is only in other folders, the log and the menu now say so and name the fix. Before, they said "DLSS needs an NVIDIA RTX GPU". The fix is to put `libnvidia-ngx-dlss.so.*` in `$XDG_DATA_HOME/hookshot/ngx` or in another ASCII folder named by `RX_VK_DLSS_DIR`. The readme says this too. Windows is unchanged. Its paths are wide end to end (`GetModuleFileNameW`, `_wgetenv`, `fs::path::native()`). ## How it was tested - `tests/test_dlss_checks.py` (new) compiles and runs `tests/test_dlss_checks.cpp` with `-Wall -Wextra -Werror` (g++ and clang++): - Accepted: 512x288 for 1536x864 (the report's case), 683x384 for 2048x1152, and every DLSS mode and DLAA at every output the menu has (using the menu's twelfths formula). - Rejected: an unknown output, under a third on either axis (511x288, 512x287), 384x384, 256x144, 640x360 at 2048x1152, and larger than the output. - UTF-8: ASCII, accented, CJK and 4-byte paths; a lone Latin-1 byte, stray continuation bytes, cut sequences, overlong forms, surrogates, values past U+10FFFF and F5-FF. All run in the C locale. - It also checks that the sources no longer have a fixed 640/360 floor and that `wide()` widens byte by byte only for ASCII paths. - `tests/run_all.py`: 176 passed, 5 skipped. One check, `nx2/test_nup_verify.py`, timed out once under load (a 5 s subprocess limit) and passed when run again. It is unrelated. - `hookshot_vk.dll` cross-built with llvm-mingw and the NGX SDK (as the `vulkan-dlss` job does): compiles and links. - Headless runs on Linux, RTX 3090, NGX runtime 310.9.16: | Build | Folder | Setup | Result | |---|---|---|---| | main | ASCII | handheld, DLSS, Ultra Performance | no DLSS at all: the game's upscale throughout, no DLSS part in the status lines | | this PR | ASCII | handheld, DLSS, Ultra Performance | `NVIDIA DLSS draws OoT's upscale: 512x288 -> 1536x864`, `feature 1 made ... 512x288 ... -> 1536x864, mode 3`, status `DLSS: 600 frames drawn ... actor motion 48924 draws in 600 frames` | | this PR | ASCII | docked, DLSS Quality | `1366x768 -> 2048x1152`, `600 frames drawn` | | main | `Hookshot-dlss-utf8-Café-ゼルダ` | docked, DLSS Quality | `not available on this GPU (0xbad00004): DLSS needs an NVIDIA RTX GPU` on an RTX 3090. 0xbad00004 is NGX's FeatureNotFound | | this PR | same | same | `... is only in folders whose path is not all ASCII (...) ... put it in $XDG_DATA_HOME/hookshot/ngx, or RX_VK_DLSS_DIR=<an ASCII folder>`, and the menu says the same | | this PR | same, runtime in an ASCII folder via `RX_VK_DLSS_DIR` | docked, DLSS Quality | `NGX started`, `NVIDIA DLSS draws OoT's upscale: 1366x768 -> 2048x1152`, `528 frames drawn` in the first status line | With `RX_VK_DLSS_LOG=1`, NGX's own lines show both steps. Through the alias it finds the runtime (`Loading most recent 'libnvidia-ngx-dlss.so' snippet from '/proc/self/fd/135'... Found 1 snippets`). Then the driver cannot load it from the real path (`failed to load signed snippet - Unable to open fd`). Not tested: Windows end to end (no change there), and the U+FFFD fallback in a real run. That fallback only applies when a non-ASCII folder cannot be opened, and NGX would not find anything there anyway.