Pipeline state list: the pipelines met in play, by hashes and state, built before the game on a new install or update #939

Merged

hckrman101 merged 2 commits from pipeline-state-list into main

2026-11-09 07:37:00 +00:00

hckrman101 commented

2026-11-09 02:07:58 +00:00

(Migrated from github.com)

Change

hookshot_pipeline_states.bin (beside the game, written as pipelines are made) holds each pipeline as:
- its stages' code keys;
- the vertex attribute formats and the target formats and types its program was translated for;
- the program variant's texture answers (by descriptor, and in the order the translation asked them);
- the state baked into the pipeline (topology, targets, blend, write masks, strides, divisors, polygon mode, depth clamp, alpha to coverage).
Each record is 528 bytes. It holds no game code: the stages come back from the player's own hookshot_shaders.bin. Unlike the notes, the list is good for any build.
prepare_run (once per back-end build) reads that list and hookshot_pipeline_states_shipped.bin:
- it translates any program the prepare did not already make (other input or target types);
- it makes texture variants by giving the recorded answers back in the order the translation asks them (nothing is bound before the game);
- it notes every pipeline, so precompile_run builds them before the title. The prediction then also has notes on a first start.
Shipping: nx2/make_game.py puts work/pipeline_states.bin, when there is one, beside the game as the shipped list. nx2/pipeline_states.py shows and merges lists, so a maintainer's play-through, or players' lists, become the shipped one.
Off switch: RX_VK_NO_STATE_LIST=1 turns both recording and reading off.
Refactor: the pipeline key is one function (pipeline_key) for draws and the list.
Test: tests/test_pipeline_states.py checks that the record size and magic agree between the back end and the tool, that merging keeps each pipeline once, and that make_game ships the list only when there is one.

Measured (RTX 3090, Linux, fresh first start, no caches, the same Kokiri Forest route each run with input timed in game frames, headless)

Main (232eecbd) against main plus this PR, with a list recorded on one earlier run of the same route (1,001 pipelines, 725 programs, 68 texture variants; all were made at prepare, none failed):

	main ×3	+ state list ×4
graphics pipelines built in play	314–352	22–89
programs translated in play	45–56	0–14
waits for a pipeline or program	0.3–1.9 s	0–0.04 s
draws left out (pop-in)	4,176–5,976	12–444
replay frames > 50 ms	18–46	2–11
worst replay frame	492–1,260 ms	54–564 ms
shader preparation	157–165 s	161–181 s (the list's part: ~4 s)
peak memory at play	11.1–11.2 GB	9.7–12.2 GB

The list came from the same route, so this is the best case. A list from a full play-through would cover the rest of the game the same way; one from a different route covers only what the two share.

Pipelines need draw state as well as shaders. A first start, of a new install or of a new build, still makes every graphics pipeline, and every texture variant of a program, the first time it is drawn. `hookshot_pipelines.bin` (the precompile's notes) only helps the next start of the *same* build: its stamp drops it on any back-end change. #772's prediction also reads those notes, so it has nothing to work from on a first start. ### Change - **`hookshot_pipeline_states.bin`** (beside the game, written as pipelines are made) holds each pipeline as: - its stages' code keys; - the vertex attribute formats and the target formats and types its program was translated for; - the program variant's texture answers (by descriptor, and in the order the translation asked them); - the state baked into the pipeline (topology, targets, blend, write masks, strides, divisors, polygon mode, depth clamp, alpha to coverage). Each record is 528 bytes. It holds **no game code**: the stages come back from the player's own `hookshot_shaders.bin`. Unlike the notes, the list is good for any build. - **`prepare_run`** (once per back-end build) reads that list and `hookshot_pipeline_states_shipped.bin`: - it translates any program the prepare did not already make (other input or target types); - it makes texture variants by giving the recorded answers back in the order the translation asks them (nothing is bound before the game); - it notes every pipeline, so `precompile_run` builds them before the title. The prediction then also has notes on a first start. - **Shipping:** `nx2/make_game.py` puts `work/pipeline_states.bin`, when there is one, beside the game as the shipped list. `nx2/pipeline_states.py` shows and merges lists, so a maintainer's play-through, or players' lists, become the shipped one. - **Off switch:** `RX_VK_NO_STATE_LIST=1` turns both recording and reading off. - **Refactor:** the pipeline key is one function (`pipeline_key`) for draws and the list. - **Test:** `tests/test_pipeline_states.py` checks that the record size and magic agree between the back end and the tool, that merging keeps each pipeline once, and that make_game ships the list only when there is one. ### Measured (RTX 3090, Linux, fresh first start, no caches, the same Kokiri Forest route each run with input timed in game frames, headless) Main (232eecbd) against main plus this PR, with a list recorded on one earlier run of the same route (1,001 pipelines, 725 programs, 68 texture variants; all were made at prepare, none failed): | | main ×3 | + state list ×4 | |---|---|---| | graphics pipelines built in play | 314–352 | 22–89 | | programs translated in play | 45–56 | 0–14 | | waits for a pipeline or program | 0.3–1.9 s | 0–0.04 s | | draws left out (pop-in) | 4,176–5,976 | 12–444 | | replay frames > 50 ms | 18–46 | 2–11 | | worst replay frame | 492–1,260 ms | 54–564 ms | | shader preparation | 157–165 s | 161–181 s (the list's part: ~4 s) | | peak memory at play | 11.1–11.2 GB | 9.7–12.2 GB | The list came from the same route, so this is the best case. A list from a full play-through would cover the rest of the game the same way; one from a different route covers only what the two share.

NotRexed commented

2026-11-09 05:31:07 +00:00

(Migrated from github.com)