Idea: shareable pipeline manifest (no game content) so the first start is smooth #334

Closed

opened 2026-11-07 21:18:05 +00:00 by GuiFlam · 1 comment

GuiFlam commented

2026-11-07 21:18:05 +00:00

(Migrated from github.com)

The idea

Ship (or let the community share) a small pipeline manifest: a plain list of the pipelines the game is known to use (program hashes + fixed-function / render-pass state), with no shader text, no translated code and no game data. On first start, Hookshot would compile that list locally, in the background or on a "preparing" screen, so the first walk through Kokiri Forest is not a series of freezes.

Why

On a first start, or on any new area, every new pipeline is built on the fly. My log from a first lap of Kokiri Forest: 1029 pipelines compiled in about 34 s of wall time, and the freezes line up with that. With the default RX_VK_PIPE_SKIP, draws whose pipeline is not ready are left out, and that shows up on screen as grey or black-blob frames. Going through a screen recording frame by frame, 334 of 2761 frames (about 12 %) were affected during the first minutes. (I have not yet confirmed that RX_VK_PIPE_SKIP=0 removes them; I will report back.)

Setup: RTX 3050 Ti Laptop (4 GB), i9-12900HK, Fedora 43, Linux build. The same pattern is described for other GPUs in #62, #69 and in the cold-start vs second-start comments there.

What already exists

Hookshot already records the pipelines it sees (hookshot_pipelines.bin) and pre-warms them at the next start. So the second start is better than the first. The only thing missing is a seed for the first start.

Proposal

A manifest file (text or small binary) listing pipeline keys as the game produces them. It is generated by running the game and dumping the existing record in a stable format, then merged/deduplicated by the maintainers or by a script.
At startup, if the local hookshot_pipelines.bin is missing or smaller than the manifest, merge the manifest in and prewarm it (same code path that already exists), with a progress indicator.
Everything is built on the user's own machine, from the user's own game files. The manifest only says which pipelines to build, not what they contain.
Entries whose program hash is not found in the local game data are simply skipped, so a manifest from a different game version is harmless.

What must not be shared

Driver caches (~/.cache/nvidia, mesa_shader_cache, etc.): tied to the GPU and driver version, and not portable.
hookshot_shaders.bin, hookshot_programs.bin or any translated/generated shader or code: derived from the game, so it must not be redistributed.

Precedent

nxnative (Apple Silicon) ships a pso_manifest.tsv (9,625 rows) that is used the same way: a list of pipeline-state recipes, compiled locally.

Open questions for the maintainers

Is a program hash alone acceptable to publish? If not, the manifest could be distributed as a separate download or generated by a script from the user's own game files.
Do keys stay stable across releases? If they change often, the manifest has to be tied to a release.

Happy to help: I can produce a manifest from my own runs, test a prewarm-from-manifest build on NVIDIA/Linux, and report cold vs warm numbers.

### The idea Ship (or let the community share) a small **pipeline manifest**: a plain list of the pipelines the game is known to use (program hashes + fixed-function / render-pass state), with **no shader text, no translated code and no game data**. On first start, Hookshot would compile that list locally, in the background or on a "preparing" screen, so the first walk through Kokiri Forest is not a series of freezes. ### Why On a first start, or on any new area, every new pipeline is built on the fly. My log from a first lap of Kokiri Forest: **1029 pipelines compiled in about 34 s of wall time**, and the freezes line up with that. With the default `RX_VK_PIPE_SKIP`, draws whose pipeline is not ready are left out, and that shows up on screen as grey or black-blob frames. Going through a screen recording frame by frame, 334 of 2761 frames (about 12 %) were affected during the first minutes. (I have not yet confirmed that `RX_VK_PIPE_SKIP=0` removes them; I will report back.) Setup: RTX 3050 Ti Laptop (4 GB), i9-12900HK, Fedora 43, Linux build. The same pattern is described for other GPUs in #62, #69 and in the cold-start vs second-start comments there. ### What already exists Hookshot already records the pipelines it sees (`hookshot_pipelines.bin`) and pre-warms them at the next start. So the second start is better than the first. The only thing missing is a **seed for the first start**. ### Proposal 1. A manifest file (text or small binary) listing pipeline keys as the game produces them. It is generated by running the game and dumping the existing record in a stable format, then merged/deduplicated by the maintainers or by a script. 2. At startup, if the local `hookshot_pipelines.bin` is missing or smaller than the manifest, merge the manifest in and prewarm it (same code path that already exists), with a progress indicator. 3. Everything is built **on the user's own machine, from the user's own game files**. The manifest only says *which* pipelines to build, not *what* they contain. 4. Entries whose program hash is not found in the local game data are simply skipped, so a manifest from a different game version is harmless. ### What must not be shared - Driver caches (`~/.cache/nvidia`, `mesa_shader_cache`, etc.): tied to the GPU and driver version, and not portable. - `hookshot_shaders.bin`, `hookshot_programs.bin` or any translated/generated shader or code: derived from the game, so it must not be redistributed. ### Precedent nxnative (Apple Silicon) ships a `pso_manifest.tsv` (9,625 rows) that is used the same way: a list of pipeline-state recipes, compiled locally. ### Open questions for the maintainers - Is a program hash alone acceptable to publish? If not, the manifest could be distributed as a separate download or generated by a script from the user's own game files. - Do keys stay stable across releases? If they change often, the manifest has to be tied to a release. Happy to help: I can produce a manifest from my own runs, test a prewarm-from-manifest build on NVIDIA/Linux, and report cold vs warm numbers.

NotRexed commented

2026-11-09 05:46:51 +00:00

(Migrated from github.com)

Thanks for the idea. Feature requests are now collected in #1007, which lists this one with a link back here, so closing it in favour of that list (this is not a decision against the idea).

Commenting is not possible because the repository is archived.