MagicBeanDev/Hookshot

Archived

Watch

Fork 0

feat(vk): neural rendering at the present, through NGX direct and OpenDLSS-NR, with a Neural tab #2820

Open

github-sync wants to merge 6 commits from github/pr/2820 into main

github-sync commented

2026-11-12 15:49:08 +00:00

Owner

@notquiteog opened this pull request on GitHub · 2026-11-12 15:27 UTC · original

Hookshot has no neural rendering in the tree: #1590 and #1833 are both open for it, and #1833 (which carries an earlier copy of this code from my fork) cannot be merged because it only builds together with two other PRs. This is the same work as one branch that builds and passes the checks alone on main, with the review points raised on #1833 and #1335 dealt with.

Two things to know before reviewing. NVIDIA ships no Linux libnvidia-ngx-dlssnr.so, so the NGX direct backend has not evaluated a frame on the PC this was prepared on: it needs a run on Windows with an RTX card before it merges (testing wanted). And it overlaps #1590, which is in review for the same NGX feature: I would rather the maintainers say how they want the two to relate (this as a follow-up for the Open backend and the Resolution scale, or parts of it folded into #1590) than have two PRs compete.

What it is

A model run over each of the game's finished frames at the present, before frame generation, with a Neural tab in the Insert menu (On/Off, Backend, Model, Style, Preset, Passes, Resolution scale, Intensity and the tone and structure strengths, Auto mask), every row read live and saved as nr_<name>.

NGX direct (vk_dlssnr_ngx.cpp, vk_ngx.cpp): NGX feature 18 on DLSS's own NGX session. Nothing of NVIDIA's is in the repository; the player's own nvngx_dlssnr.dll is looked for where DLSS's runtime is.
Open (vk_opennr.cpp, gpu/vk/opennr/, third_party/opendlss-nr, MIT): OpenDLSS-NR, the network run by compute shaders, from a model folder the player supplies. Hookshot ships no weights and fetches none.
Resolution scale and Passes (vk_nr_scale.cpp, nr_plan.h): the model on a smaller copy of the picture with its detail put back at full size, which is what makes it affordable (numbers below).
No Streamline. The review of #1335 asked for NGX rather than Streamline, so the Streamline backend of my fork is not here. Backend is 0 Auto, 1 NGX direct, 2 Open.

Review points from #1833, and what this does about them

"It only builds with #1831 and #1834." This builds alone on main: no vk_camera_motion.h, no scene_frame_inputs. The temporal path takes the scene's motion from vk_fsr3_inputs as FSR 3.1's interpolation does.
"VK_EXT_shader_float8 asked for with NR off on non-NVIDIA devices." The Open backend's extensions and features are requested only when the player has asked for it by the time the device is made: Neural rendering saved On with Backend at Open (or Auto on a GPU that is not NVIDIA's), a model folder already in place, or RX_VK_DLSS_NR_BACKEND=open. With the row off and no model folder, vkCreateDevice gets exactly what it gets today on every vendor; tests/test_opennr_backend.py pins that. A row turned on later without them says a restart is needed.
The ~29k lines of opennr_*_spv.h. They are compiled by glslang from their sources by scripts/opennr_kernels.py, and tests/test_opennr_kernels.py recompiles and compares. Size: of this PR's lines, the vendored third_party/opendlss-nr and the generated SPIR-V headers are the large majority; the hand-written part is about 7,400 lines across 56 files, tests and docs included.

How it was tested

tests/run_all.py (Linux, main at 65a4d874 merged in): 504 passed, 2 failed, 22 skipped. Two checks fail on this branch exactly as they fail on main at 65a4d874 without it: test_readme_menu_rows (the Skip opening row has no readme line; #2857 is open for it) and test_program_cache_one_variant (its fixture lacks the zstd cache's names since #1630). test_readme_menu_rows now reads the Neural tab too, and the readme's option table has a line for each of its rows.
The Vulkan back end builds from this branch with the NGX SDK and without it (the stubs link).
In the game with this branch's back end in a game built from main at 65a4d874 (run again after that merge) and a synthetic model (scripts/opennr_harness.py model, random weights of the right shapes: it proves the path, not the picture): RTX 5060 Ti 16 GB, driver 615.71.09, Linux (COSMIC, Wayland), Kokiri Forest, 2048x1152 output, DLSS, V-Sync Off.

Resolution scale	The model's frame	GPU time a frame	Game fps
Off			55 to 59
50%	1024x576	4.35 ms	51 to 52
100%	2048x1152	14.10 ms	37.5

Scale changed in play both ways, the row off and on again, Backend to NGX direct (refused with the reason below, the game's picture shown) and back to Auto (Open again), DLSS Frame Generation at 2x on top (600 of 600 presents made by DLSS-G): no device lost, no error.

NGX direct on Linux says why it does not run: NGX answers DLSSNR.Available = 0 ... NVIDIA has not shipped it for Linux yet.

Not tested

NGX direct evaluating a frame (Windows, an RTX card, the player's nvngx_dlssnr.dll). Compiled and read only.
A real model's picture with the Open backend: only the synthetic one ran. Screenshots of it would show noise shaped like a frame, so none are attached.
The Open backend on AMD (its khr route), Intel, and Windows at all.
HDR: with Native HDR the model is left out and the hint says so; that was read, not run on an HDR display.

Known and left

A failed backend is not always named in the row's hint (some refusals only reach the log).
Nothing is released while the row is off: the features, the network and their images stay until the game ends.
The Neural tab's texts are English in all 13 menu languages.

🤖 Generated with Claude Code

> [@notquiteog](https://github.com/notquiteog) opened this pull request on GitHub · 2026-11-12 15:27 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2820) Hookshot has no neural rendering in the tree: #1590 and #1833 are both open for it, and #1833 (which carries an earlier copy of this code from my fork) cannot be merged because it only builds together with two other PRs. This is the same work as one branch that builds and passes the checks alone on main, with the review points raised on #1833 and #1335 dealt with. **Two things to know before reviewing.** NVIDIA ships no Linux `libnvidia-ngx-dlssnr.so`, so the NGX direct backend has not evaluated a frame on the PC this was prepared on: it needs a run on Windows with an RTX card before it merges (`testing wanted`). And it overlaps #1590, which is in review for the same NGX feature: I would rather the maintainers say how they want the two to relate (this as a follow-up for the Open backend and the Resolution scale, or parts of it folded into #1590) than have two PRs compete. ## What it is A model run over each of the game's finished frames at the present, before frame generation, with a Neural tab in the Insert menu (On/Off, Backend, Model, Style, Preset, Passes, Resolution scale, Intensity and the tone and structure strengths, Auto mask), every row read live and saved as `nr_<name>`. - **NGX direct** (`vk_dlssnr_ngx.cpp`, `vk_ngx.cpp`): NGX feature 18 on DLSS's own NGX session. Nothing of NVIDIA's is in the repository; the player's own `nvngx_dlssnr.dll` is looked for where DLSS's runtime is. - **Open** (`vk_opennr.cpp`, `gpu/vk/opennr/`, `third_party/opendlss-nr`, MIT): OpenDLSS-NR, the network run by compute shaders, from a model folder the player supplies. Hookshot ships no weights and fetches none. - **Resolution scale and Passes** (`vk_nr_scale.cpp`, `nr_plan.h`): the model on a smaller copy of the picture with its detail put back at full size, which is what makes it affordable (numbers below). - **No Streamline.** The review of #1335 asked for NGX rather than Streamline, so the Streamline backend of my fork is not here. Backend is 0 Auto, 1 NGX direct, 2 Open. ## Review points from #1833, and what this does about them - *"It only builds with #1831 and #1834."* This builds alone on main: no `vk_camera_motion.h`, no `scene_frame_inputs`. The temporal path takes the scene's motion from `vk_fsr3_inputs` as FSR 3.1's interpolation does. - *"`VK_EXT_shader_float8` asked for with NR off on non-NVIDIA devices."* The Open backend's extensions and features are requested only when the player has asked for it by the time the device is made: Neural rendering saved On with Backend at Open (or Auto on a GPU that is not NVIDIA's), a model folder already in place, or `RX_VK_DLSS_NR_BACKEND=open`. With the row off and no model folder, `vkCreateDevice` gets exactly what it gets today on every vendor; `tests/test_opennr_backend.py` pins that. A row turned on later without them says a restart is needed. - *The ~29k lines of `opennr_*_spv.h`.* They are compiled by glslang from their sources by `scripts/opennr_kernels.py`, and `tests/test_opennr_kernels.py` recompiles and compares. Size: of this PR's lines, the vendored `third_party/opendlss-nr` and the generated SPIR-V headers are the large majority; the hand-written part is about 7,400 lines across 56 files, tests and docs included. ## How it was tested - `tests/run_all.py` (Linux, main at 65a4d874 merged in): **504 passed, 2 failed, 22 skipped**. Two checks fail on this branch exactly as they fail on main at 65a4d874 without it: `test_readme_menu_rows` (the Skip opening row has no readme line; #2857 is open for it) and `test_program_cache_one_variant` (its fixture lacks the zstd cache's names since #1630). `test_readme_menu_rows` now reads the Neural tab too, and the readme's option table has a line for each of its rows. - The Vulkan back end builds from this branch with the NGX SDK and without it (the stubs link). - In the game with this branch's back end in a game built from main at 65a4d874 (run again after that merge) and a **synthetic model** (`scripts/opennr_harness.py model`, random weights of the right shapes: it proves the path, not the picture): RTX 5060 Ti 16 GB, driver 615.71.09, Linux (COSMIC, Wayland), Kokiri Forest, 2048x1152 output, DLSS, V-Sync Off. | Resolution scale | The model's frame | GPU time a frame | Game fps | |---|---|---|---| | Off | | | 55 to 59 | | 50% | 1024x576 | 4.35 ms | 51 to 52 | | 100% | 2048x1152 | 14.10 ms | 37.5 | Scale changed in play both ways, the row off and on again, Backend to NGX direct (refused with the reason below, the game's picture shown) and back to Auto (Open again), DLSS Frame Generation at 2x on top (600 of 600 presents made by DLSS-G): no device lost, no error. - NGX direct on Linux says why it does not run: `NGX answers DLSSNR.Available = 0 ... NVIDIA has not shipped it for Linux yet`. ## Not tested - **NGX direct evaluating a frame** (Windows, an RTX card, the player's `nvngx_dlssnr.dll`). Compiled and read only. - **A real model's picture** with the Open backend: only the synthetic one ran. Screenshots of it would show noise shaped like a frame, so none are attached. - The Open backend on AMD (its `khr` route), Intel, and Windows at all. - HDR: with Native HDR the model is left out and the hint says so; that was read, not run on an HDR display. ## Known and left - A failed backend is not always named in the row's hint (some refusals only reach the log). - Nothing is released while the row is off: the features, the network and their images stay until the game ends. - The Neural tab's texts are English in all 13 menu languages. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

github-sync added 5 commits

2026-11-12 15:49:08 +00:00

build: vendor OpenDLSS-NR, the network behind open neural rendering 348999f38a

OpenDLSS-NR (https://github.com/maanHimself/OpenDLSS-NR, MIT, Copyright 2026 maan) is a Vulkan implementation of the
DLSS 5 neural rendering network. The Neural tab's Open backend, in the commits after this one, compiles its host
sources into the renderer and runs its kernels, so its files are here first, unmodified, at commit 9d08f418:

- src/: the compute context, the model directory loader, the kernel wrappers, the 71-block graph and the CPU
  reference of its arithmetic; and upstream's dlss5vk tool, which only the correctness harness links;
- shaders/: the GLSL kernels;
- scripts/ptx/: the Python generators of the PTX kernels;
- LICENSE, NOTICE and PROVENANCE.md, which says what was left out and how the files are built here.

No model weights are here and nothing here produces them: the network reads a model directory the player supplies.
Nothing is taken from any NVIDIA binary.

Co-Authored-By: Claude Fable 5.2 <[email protected]>

feat(vk): neural rendering at the present, through NGX and through OpenDLSS-NR bc4ad54ad2

DLSS 5 neural rendering is a model run over each finished frame: colour in, colour out, not an upscaler and not ray
reconstruction, so it works on a rasterized game. This adds the two backends that run it and the one call the present
makes; the menu's tab is the next commit, so here it is off unless RX_VK_DLSS_NR=1 is set.

Where it runs: do_present (vk_replay.cpp), once for each of the game's frames, on the picture's part of the window
image, before frame generation and before the menu is drawn, in the recording after the present's own so its GPU time
does not hold the game's next frame back. It returns an image of its own; a frame it cannot do goes on as before. It
is left out while the picture is the game's own HDR10.

The backends, behind nr:: (vk_dlssnr_ngx.cpp):

- NGX direct (vk_dlssnr_ngx.cpp, vk_ngx.cpp): NVIDIA's model as NGX feature 18, over the NGX session vk_dlss.cpp
already owns (dlss_ngx_session, dlss_ngx_runtime, dlss_ngx_data_path). Only in a build with the NGX SDK; the stubs
link without it. The parameter names are openNR's (MIT), written out as facts. A feature a pass (nr_plan.h).
- Open (vk_opennr.cpp, gpu/vk/opennr): OpenDLSS-NR on the renderer's own device, from a model folder the player
supplies (nr-model in the game folder): no weights are included and nothing here gets them. Upstream's kernels on
NVIDIA (VK_NV_cooperative_matrix2, with the PTX route where VK_NV_cuda_kernel_launch is), and the khr route
(gpu/vk/opennr/khr) on VK_KHR_cooperative_matrix alone. Its temporal path reads the scene's motion
(fsr3_inputs::latest).

Both are given the picture alone (vk_nr_scale.cpp), smaller under the Resolution scale, with the model's detail put
back at full size (shaders/nr_detail.comp).

What is asked of the device: nothing, unless the player asked for neural rendering by the start. NGX is asked for
feature 18's device extensions only with the row saved On on an NVIDIA GPU; the open network's extensions and feature
structs are requested only with the row saved On and Backend Open (or Auto on a GPU that is not NVIDIA's), a model
folder in place, or RX_VK_DLSS_NR_BACKEND=open. With it off, vkCreateDevice is given what it is given today on every
vendor. A row turned on later says when a restart is needed.

Build: nx2/build.py compiles OpenDLSS-NR's host sources from a patched copy in the build folder (third_party stays
upstream's), generates its PTX kernels into opennr_ptx.h, and gives the NGX files NVIDIA's headers; ngx_link adds
wintrust (Windows' own) for the refusal report. The SPIR-V headers are scripts/opennr_kernels.py's, checked against
their sources by tests/test_opennr_kernels.py with the pinned glslang. The AppImage carries the sources a player's
build compiles and OpenDLSS-NR's licence.

Checked on Linux, RTX 5060 Ti: the back end builds with and without the NGX SDK; scripts/opennr_harness.py runs every
route of the network bit for bit against the GLSL reference on a synthetic model (tests/test_opennr_harness.py,
test_opennr_khr.py). Not run: NVIDIA's model (NVIDIA ships no Linux libnvidia-ngx-dlssnr.so), anything on Windows, the
khr route on an AMD GPU, and real weights.

Co-Authored-By: Claude Fable 5.2 <[email protected]>

feat(menu): the Neural tab, neural rendering's switch and the model's settings 57fb5ed725

Neural rendering could only be switched on with RX_VK_DLSS_NR. The Insert menu gets a sixth tab, Neural, between
Controls and Debug, with the switch and the model's own settings:

- Neural rendering: Off by default. Its hint is the running backend's reason when it cannot run, and says when a row
turned on while the game runs needs a restart (the device is made for a backend only when it was asked for by the
start). With the row off nothing is asked of NGX or the network.
- Backend: Auto, NGX direct, Open (OpenDLSS-NR), saved as nr_backend 0, 1, 2; RX_VK_DLSS_NR_BACKEND=auto|ngx|open
over it for a run. Auto's hint says which one runs.
- Model, Style, Preset, Passes, Resolution scale, Intensity, Local tone, Local structure, Global tone, Skin
structure, Auto mask. Each takes effect on the next frame; the help says which ones are NGX direct's alone.

Every row but the switch is saved under an nr_ key (nr_model, nr_style, ...): "model", "preset" or "backend" alone
would say nothing in hookshot_options.ini beside quality_preset and dlss_model. None is a Quality preset's row. The
control bridge names the tab "neural". The texts are English: the 13 translation tables are not extended here.

Checks: tests/test_opennr_backend.py cuts opennr::selected() out of the source and runs it (off with no model folder
is "no" on AMD, Intel and NVIDIA, so the device is made as without the backend), and pins the device hooks' gates,
NGX's, and the hint's restart line; tests/test_nr_settings_live.py compiles nr_settings() and checks the rows' ranges,
the defaults, the Backend numbering and that nothing caches a setting; tests/test_nr_plan.py the Passes and
Resolution scale rows. The two fixtures that compile ini_key() get the fields it now names.

Co-Authored-By: Claude Fable 5.2 <[email protected]>

docs: neural rendering, its design, what was measured and the readme's entry 463b429ab5

docs/neural-rendering.md says what DLSS 5 neural rendering is and is not, where it runs in the frame, what is asked of
the device and when (nothing with the row off), NGX direct's parameter set and where it comes from, the open backend
(the model folder the player supplies, the frame's passes, the khr route), the Passes and Resolution scale rows, and
plainly what was measured and what was not: NVIDIA's model has not produced a frame here, and the open network was
measured on synthetic weights.

The readme gets the sixth tab, the Neural rows in the options table, a section on the two backends and what each
needs, and the credits for OpenDLSS-NR and openNR. CONTRIBUTING.md and docs/development.md name OpenDLSS-NR among
the third-party parts.

Co-Authored-By: Claude Fable 5.2 <[email protected]>

Merge main 7181e56858

Co-Authored-By: Claude Fable 5.2 <[email protected]>

github-sync referenced this pull request

2026-11-12 17:57:29 +00:00

ci: execute authored Vulkan fixtures on lavapipe #2866

github-sync added 1 commit

2026-11-12 19:01:00 +00:00

Merge upstream/main a4d8683b32

Co-Authored-By: Claude Fable 5.2 <[email protected]>

Commenting is not possible because the repository is archived.

No reviewers

No labels

No milestone

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Hookshot!2820

No description provided.

Rows
Columns