build(release): ship the pipeline state list with every release #1918

Merged

hckrman101 merged 3 commits from ship-pipeline-list into main

2026-11-10 20:37:44 +00:00

hckrman101 commented

2026-11-10 20:23:20 +00:00

(Migrated from github.com)

Why it went missing

nx2/make_game.py (ship_state_list) and scripts/linux/appimage.sh both shipped work/pipeline_states.bin only when it happened to be there, and in CI it never is: work/ is ignored, and the "Play profile" steps fill work/ from the previous release's app part, which never had one. The Windows package (release_parts.py package) only ever takes tools/work from the previous release, so even a list in the checkout would not have reached it.

What changes

The list lives in assets/pipeline_states.bin.xz, which is tracked. work/ stays local and fully ignored, as CONTRIBUTING says. The file is xz-compressed to keep the repository's history small: 23,840 bytes for the 665,820-byte list. I chose xz over zstd because the AppImage's Python is 3.12, and its standard library has lzma but not compression.zstd.
What it holds for now: a list recorded from play on my PC, with 1261 pipelines (1186 graphics, 75 mesh) from 977 programs. It will be replaced by a fuller one later.
What the file contains (nx2/pipeline_states.py has the layout). For each pipeline:
- the hashes of its shaders' code and of the program variant;
- the state it was made with: target and depth formats, sample count, topology, vertex attribute formats, strides and divisors, blend state, channel mask and polygon mode;
- the texture-type answers the translation used.
It contains no game code and no assets. A player's build turns the hashes back into shaders from their own hookshot_shaders.bin, which is made from their own dump. Unpacked, strings finds nothing in it but the HKSTATE1 magic.
nx2/make_game.py unpacks the list beside the game as hookshot_pipeline_states_shipped.bin. The game side is unchanged.
- The file is only rewritten when its contents differ.
- A list that is missing or does not read is left out, with a line in the build log. The game works without it.
nx2/pipeline_states.py:
- reads xz lists (detected by their magic);
- writes xz when --out ends in .xz;
- --check is the release check: the list must be whole, of this build's format, and not empty.
- A list of another format now says so and says how to make a new one.
Every release path carries it:
- AppImage: assets/ was already copied. appimage.sh now stops if the list is missing. check_player_build.sh (which a tagged release needs to pass) reads the list back with the image's own Python, using --check.
- Windows builder: release_parts.py package copies the list to tools/assets/, where make_game.py (run from tools/) looks for it. It stops if the list is missing, damaged, of another format or empty. tools/assets ends up in the app part.
Release CI fails without it, with a message that says how to regenerate it. Both checks run on every build, so they run on tags too.
docs/development.md has a new section, "The pipeline state list":
- what the list is and where it ships;
- how to refresh it: python -I nx2/pipeline_states.py --out assets/pipeline_states.bin.xz assets/pipeline_states.bin.xz <game folder>/hookshot_pipeline_states.bin ...;
- when: before each release, and after a change to the shaders' translation or to the state a pipeline is made from;
- when the record format changes, a new list must be made without the old one among its inputs.
Tests:
- tests/test_ship_state_list.py checks that:
  - the committed list is xz and passes --check;
  - the AppImage carries assets/ and its check stops without the list;
  - the Windows package carries the list where make_game.py unpacks it from;
  - the package stops when the list is missing, empty, damaged or of another format.
- tests/test_pipeline_states.py adds the .xz round trip, make_game's unpacking, and a list that does not read being left out.

tests/run_all.py: 279 passed, 3 failed. The same 3 fail on main without this change (nx2/test_nup_read_name.py, tests/test_program_cache_cleanup.py, tests/test_wayland_libdecor_hint.py).

The first commit put the list in work/ with a .gitignore exception, and the second commit moves it out again. Please squash on merge, so that the uncompressed 666 KB file does not stay in main's history.

A question for you

Are you OK with shipping this file's contents? It is shader hashes and render state: no game code and nothing extracted from the dump. That is the same kind of data as hot_funcs.txt, hot_order.txt and the play profile (hookshot_pieces.prof), which every release already ships in tools/work. Nothing under work/ is committed.

A fresh install's first field load washes white for about 6 seconds, because the draws whose pipelines are still compiling are left out; with a pipeline state list beside the game, the first start builds those pipelines before the title and none are left out. Releases v2.0.8 to v2.0.14 shipped no list: I downloaded each release's Windows app part (`HookshotBuilder-app-*.zip`, named by its `hookshot-setup.txt`) and Linux tools part (`HookshotLinux-usr-share-hookshot-*.zip`, named by `hookshot-linux.txt`), and their `work/` folders hold only `hot_funcs.txt`, `hot_order.txt`, `hookshot_pieces.prof`, `image/chunks.txt` (and the icon on Windows), with no `pipeline_states.bin`. ## Why it went missing `nx2/make_game.py` (`ship_state_list`) and `scripts/linux/appimage.sh` both shipped `work/pipeline_states.bin` only when it happened to be there, and in CI it never is: `work/` is ignored, and the "Play profile" steps fill `work/` from the previous release's app part, which never had one. The Windows package (`release_parts.py package`) only ever takes `tools/work` from the previous release, so even a list in the checkout would not have reached it. ## What changes - **The list lives in `assets/pipeline_states.bin.xz`**, which is tracked. `work/` stays local and fully ignored, as CONTRIBUTING says. The file is xz-compressed to keep the repository's history small: 23,840 bytes for the 665,820-byte list. I chose xz over zstd because the AppImage's Python is 3.12, and its standard library has `lzma` but not `compression.zstd`. - **What it holds for now:** a list recorded from play on my PC, with 1261 pipelines (1186 graphics, 75 mesh) from 977 programs. It will be replaced by a fuller one later. - **What the file contains** (`nx2/pipeline_states.py` has the layout). For each pipeline: - the hashes of its shaders' code and of the program variant; - the state it was made with: target and depth formats, sample count, topology, vertex attribute formats, strides and divisors, blend state, channel mask and polygon mode; - the texture-type answers the translation used. It contains no game code and no assets. A player's build turns the hashes back into shaders from their own `hookshot_shaders.bin`, which is made from their own dump. Unpacked, `strings` finds nothing in it but the `HKSTATE1` magic. - **`nx2/make_game.py`** unpacks the list beside the game as `hookshot_pipeline_states_shipped.bin`. The game side is unchanged. - The file is only rewritten when its contents differ. - A list that is missing or does not read is left out, with a line in the build log. The game works without it. - **`nx2/pipeline_states.py`:** - reads xz lists (detected by their magic); - writes xz when `--out` ends in `.xz`; - `--check` is the release check: the list must be whole, of this build's format, and not empty. - A list of another format now says so and says how to make a new one. - **Every release path carries it:** - **AppImage:** `assets/` was already copied. `appimage.sh` now stops if the list is missing. `check_player_build.sh` (which a tagged release needs to pass) reads the list back with the image's own Python, using `--check`. - **Windows builder:** `release_parts.py package` copies the list to `tools/assets/`, where `make_game.py` (run from `tools/`) looks for it. It stops if the list is missing, damaged, of another format or empty. `tools/assets` ends up in the app part. - **Release CI fails without it**, with a message that says how to regenerate it. Both checks run on every build, so they run on tags too. - **docs/development.md** has a new section, "The pipeline state list": - what the list is and where it ships; - how to refresh it: `python -I nx2/pipeline_states.py --out assets/pipeline_states.bin.xz assets/pipeline_states.bin.xz <game folder>/hookshot_pipeline_states.bin ...`; - when: before each release, and after a change to the shaders' translation or to the state a pipeline is made from; - when the record format changes, a new list must be made without the old one among its inputs. - **Tests:** - `tests/test_ship_state_list.py` checks that: - the committed list is xz and passes `--check`; - the AppImage carries `assets/` and its check stops without the list; - the Windows package carries the list where `make_game.py` unpacks it from; - the package stops when the list is missing, empty, damaged or of another format. - `tests/test_pipeline_states.py` adds the `.xz` round trip, make_game's unpacking, and a list that does not read being left out. `tests/run_all.py`: 279 passed, 3 failed. The same 3 fail on main without this change (`nx2/test_nup_read_name.py`, `tests/test_program_cache_cleanup.py`, `tests/test_wayland_libdecor_hint.py`). The first commit put the list in `work/` with a `.gitignore` exception, and the second commit moves it out again. Please squash on merge, so that the uncompressed 666 KB file does not stay in main's history. ## A question for you Are you OK with shipping this file's contents? It is shader hashes and render state: no game code and nothing extracted from the dump. That is the same kind of data as `hot_funcs.txt`, `hot_order.txt` and the play profile (`hookshot_pieces.prof`), which every release already ships in `tools/work`. Nothing under `work/` is committed.

hckrman101 commented

2026-11-10 20:35:34 +00:00

(Migrated from github.com)

Self-review

What I checked, for the whole diff against the base:

AppImage:
- cp -a ... assets "$T/" carries the list, and appimage.sh now stops without it.
- check_player_build.sh reads the list back with the image's Python 3.12 (lzma is in its standard library) without writing into the read-only tree.
- The Linux update part usr/share/hookshot includes assets/.
Windows package:
- release_parts.py package copies only the tracked gpu/host/nx2/scripts/third_party folders, so assets/ was not in it before. The list is now copied explicitly to tools/assets/.
- make_setup_parts.py puts tools/assets in the app part.
- The builder runs make_game.py with ROOT = tools/, so tools/assets/pipeline_states.bin.xz is the path it reads.
- The builder's Python part (3.14) has _lzma.pyd.
Player build: make_game.py unpacks the list to hookshot_pipeline_states_shipped.bin with a .new file and os.replace, and only rewrites it when the contents differ. The game's reader (state_list_read) is unchanged.

Failure modes:

List is	Release checks	Player's build
missing	stop, with the regenerate command	leaves it out
empty	stop	—
xz-damaged	stop	leaves it out with a line
another format	stop ("found HKSTATE…, make it again with this build")	leaves it out with a line
cut short	stop	leaves it out

Player lists with a record cut short at the end are still merged; the partial record is dropped, as the game drops it.

Format change (HKSTATE2):
- Once MAGIC/SIZE change, the committed list stops reading and CI fails until it is regenerated with the new build. That PR has to replace assets/pipeline_states.bin.xz too.
- The old list cannot be merged into a new one; the docs say to make it fresh.
- The tests take the magic and size from nx2/pipeline_states.py, so they follow the change.
Windows paths: all paths are joined with os.path.join, and the file is binary.
Size: 23,840 bytes in git. An uncompressed list is about 520 bytes per pipeline, and compression brings that to roughly 20 bytes.
Main since the base: main has moved and changed nx2/make_game.py, but not near ship_state_list, so the merge is clean.

What I changed as a result:

lzma preset 9 became preset 6. The output is the same size, and readers need a 9 MB dictionary instead of 65 MB.
.gitattributes marks *.xz as binary, instead of relying on text=auto to guess.
make_game.py no longer crashes or ships a list that does not read: it leaves it out and says why.
The list's reader now rejects a list that is cut short when used as the release check, where it used to skip that silently.

Still open:

Squash on merge, so the first commit's uncompressed work/pipeline_states.bin (666 KB) is not kept in main's history.
The list covers only the start of the game. A fuller list from a play-through is planned, and should replace this one before a release.
A stale hookshot_pipeline_states_shipped.bin from an earlier build is kept when the new list is missing. It is harmless: it is either an earlier list of the same format, or one the game ignores.
Whether shipping the file's contents is acceptable under CONTRIBUTING is for the maintainer to say.

## Self-review **What I checked, for the whole diff against the base:** - **AppImage:** - `cp -a ... assets "$T/"` carries the list, and `appimage.sh` now stops without it. - `check_player_build.sh` reads the list back with the image's Python 3.12 (`lzma` is in its standard library) without writing into the read-only tree. - The Linux update part `usr/share/hookshot` includes `assets/`. - **Windows package:** - `release_parts.py package` copies only the tracked gpu/host/nx2/scripts/third_party folders, so `assets/` was not in it before. The list is now copied explicitly to `tools/assets/`. - `make_setup_parts.py` puts `tools/assets` in the app part. - The builder runs `make_game.py` with `ROOT = tools/`, so `tools/assets/pipeline_states.bin.xz` is the path it reads. - The builder's Python part (3.14) has `_lzma.pyd`. - **Player build:** `make_game.py` unpacks the list to `hookshot_pipeline_states_shipped.bin` with a `.new` file and `os.replace`, and only rewrites it when the contents differ. The game's reader (`state_list_read`) is unchanged. - **Failure modes:** | List is | Release checks | Player's build | |---|---|---| | missing | stop, with the regenerate command | leaves it out | | empty | stop | — | | xz-damaged | stop | leaves it out with a line | | another format | stop ("found HKSTATE…, make it again with this build") | leaves it out with a line | | cut short | stop | leaves it out | Player lists with a record cut short at the end are still merged; the partial record is dropped, as the game drops it. - **Format change (HKSTATE2):** - Once `MAGIC`/`SIZE` change, the committed list stops reading and CI fails until it is regenerated with the new build. That PR has to replace `assets/pipeline_states.bin.xz` too. - The old list cannot be merged into a new one; the docs say to make it fresh. - The tests take the magic and size from `nx2/pipeline_states.py`, so they follow the change. - **Windows paths:** all paths are joined with `os.path.join`, and the file is binary. - **Size:** 23,840 bytes in git. An uncompressed list is about 520 bytes per pipeline, and compression brings that to roughly 20 bytes. - **Main since the base:** main has moved and changed `nx2/make_game.py`, but not near `ship_state_list`, so the merge is clean. **What I changed as a result:** - lzma preset 9 became preset 6. The output is the same size, and readers need a 9 MB dictionary instead of 65 MB. - `.gitattributes` marks `*.xz` as binary, instead of relying on `text=auto` to guess. - `make_game.py` no longer crashes or ships a list that does not read: it leaves it out and says why. - The list's reader now rejects a list that is cut short when used as the release check, where it used to skip that silently. **Still open:** - Squash on merge, so the first commit's uncompressed `work/pipeline_states.bin` (666 KB) is not kept in main's history. - The list covers only the start of the game. A fuller list from a play-through is planned, and should replace this one before a release. - A stale `hookshot_pipeline_states_shipped.bin` from an earlier build is kept when the new list is missing. It is harmless: it is either an earlier list of the same format, or one the game ignores. - Whether shipping the file's contents is acceptable under CONTRIBUTING is for the maintainer to say.

Commenting is not possible because the repository is archived.

No reviewers