build(release): the shipped pipeline state list merged from separate sources (tour + contributed lists) #2040

Merged

hckrman101 merged 1 commit from pipeline-state-sources into main

2026-11-11 00:47:32 +00:00

hckrman101 commented

2026-11-11 00:44:29 +00:00

(Migrated from github.com)

Every list added so far meant re-merging one binary, assets/pipeline_states.bin.xz. So two contributors' PRs always conflicted, and the upcoming pipeline tour would have had to overwrite people's lists to regenerate its own. This keeps each source separate in the repository, and the build merges them deterministically.

The layout

assets/pipeline_states/
  README.md                         what the lists are, how to record and add one, where the current ones come from
  tour.bin.xz, tour.md              only the pipeline tour writes these, replaced whole (not here yet: the tour isn't on main)
  contrib/<name>.bin.xz             one per contributed list
  contrib/<name>.md                 its note: Who, GPU and driver, Build, Settings, Route, Pipelines (count)

One file per contribution. Two PRs that add lists never touch the same file. Each list has a small text note, so a review sees its provenance in the diff.
The merge keeps each pipeline once. It takes the tour's list first, then the contributed lists sorted by name; names are restricted to [a-z0-9-_.], so every file system gives the same order.
- The output depends only on the files: the same sources always give the same bytes.
- Lists are written xz with fixed settings (FORMAT_XZ, CRC64, preset 6 extreme; xz stores no time or name).
Format-neutral layout. Each file carries its own magic and the tool requires one format across all sources. A later record format (the GPU/settings-neutral HKSTATE2 in progress separately) only needs a converter run over each source, and the layout stays the same.

Where the merge happens

At build time: nx2/make_game.py (ship_state_list) merges the folder into the game folder's hookshot_pipeline_states_shipped.bin.

The game is unchanged. It still reads one file.
A source that does not read is left out with a line in the log, and the others are still shipped.
The release paths carry the folder:
- appimage.sh already copies assets/, and now stops when the folder holds no list.
- check_player_build.sh runs pipeline_states.py check with the image's Python.
- release_parts.py package checks the sources and copies the folder to tools/assets/pipeline_states.

Tooling (`nx2/pipeline_states.py`)

add <name> <lists>... --who --gpu --build --settings --route writes contrib/<name>.bin.xz and its note.
- It keeps only the pipelines that no other contributed list holds.
- It does not deduplicate against the tour's list on purpose: the tour's list is replaced whole, so a pipeline left out because the tour had it would be lost when a later tour doesn't find it.
- Running it again with the same name adds to that list.
tour <lists>... --gpu --build --settings --route replaces tour.bin.xz and tour.md. Nothing else is touched.
merge --out <file> writes the merged list, the same one make_game writes.
check verifies each source and the merge:
- each source is xz, whole, of this build's format, not empty, and holds no pipeline twice;
- its note has every field, and its count matches the list;
- nothing in contrib/ is a stray file;
- merging twice gives the same bytes, and the result holds exactly the union of the sources.
The old --out/--check <lists> form still works (the tour script uses it).

Migration: nothing lost

The old list grew by prefix-preserving merges, so its history splits it exactly. I checked this: the 1,261 records from 6643085 are a prefix of 5e4e211's 4,276, which are a prefix of #2029's 11,527.

List	Pipelines	From
`contrib/hckrman101-first-play`	1,261	#1918
`contrib/joey-magnus-linux-dlss-hd-pack`	3,015	#1935 (RTX 4070 Ti, Linux, DLSS, HD pack Lite)
`contrib/jarynclouatre-most-of-the-game`	7,251	#2029 (RTX 4070 Ti SUPER 621.14, Windows 11, no mods)
Merged	11,527 (10,944 graphics, 583 mesh) of 6,672 programs

Same set. The merged set equals the old file's set: 11,527 records, none missing, none added.
A lasting check. tests/test_pipeline_state_sources.py checks that the three lists still hold exactly the old file's pipelines, by a SHA-256 of the sorted records.
Not byte-identical. The merge order is by name, so the merged file's record order differs from the old file's. The game prepares every record whatever the order.
Size. 175 KB across three files, up from 150 KB in one.
Unknown provenance. The first list's GPU, driver, settings and route weren't recorded at the time, so its note says so.

Docs

assets/pipeline_states/README.md (it replaces assets/pipeline_states.md) and docs/development.md cover:

how to record a list in game and add it;
how the tour replaces its own list;
that HKSTATE1 records are settings-specific: a list recorded at 1440p with DLSS shares few pipelines with one recorded at 360p.

Checked

Passing: tests/test_pipeline_states.py, tests/test_ship_state_list.py and the new tests/test_pipeline_state_sources.py (8 tests). Also the related tests/test_appimage_*, tests/test_make_game_*, tests/test_linux_update.py and tests/test_pipeline_state_counts.py.
CI coverage: CI runs the new test through tests/run_all.py, so every source is checked and the merge is checked for reproducibility on each PR.
Not run: a game build, a game run, the full tests/run_all.py, and a real AppImage or Windows package build. The release scripts' changed blocks are exercised by the existing tests (the AppImage block is run, and release_parts.py package is run on a small checkout).

Every list added so far meant re-merging one binary, `assets/pipeline_states.bin.xz`. So two contributors' PRs always conflicted, and the upcoming pipeline tour would have had to overwrite people's lists to regenerate its own. This keeps each source separate in the repository, and the build merges them deterministically. ## The layout ``` assets/pipeline_states/ README.md what the lists are, how to record and add one, where the current ones come from tour.bin.xz, tour.md only the pipeline tour writes these, replaced whole (not here yet: the tour isn't on main) contrib/<name>.bin.xz one per contributed list contrib/<name>.md its note: Who, GPU and driver, Build, Settings, Route, Pipelines (count) ``` - **One file per contribution.** Two PRs that add lists never touch the same file. Each list has a small text note, so a review sees its provenance in the diff. - **The merge** keeps each pipeline once. It takes the tour's list first, then the contributed lists sorted by name; names are restricted to `[a-z0-9-_.]`, so every file system gives the same order. - The output depends only on the files: the same sources always give the same bytes. - Lists are written xz with fixed settings (`FORMAT_XZ`, CRC64, preset 6 extreme; xz stores no time or name). - **Format-neutral layout.** Each file carries its own magic and the tool requires one format across all sources. A later record format (the GPU/settings-neutral HKSTATE2 in progress separately) only needs a converter run over each source, and the layout stays the same. ## Where the merge happens At build time: `nx2/make_game.py` (`ship_state_list`) merges the folder into the game folder's `hookshot_pipeline_states_shipped.bin`. - **The game is unchanged.** It still reads one file. - **A source that does not read** is left out with a line in the log, and the others are still shipped. - **The release paths carry the folder:** - `appimage.sh` already copies `assets/`, and now stops when the folder holds no list. - `check_player_build.sh` runs `pipeline_states.py check` with the image's Python. - `release_parts.py package` checks the sources and copies the folder to `tools/assets/pipeline_states`. ## Tooling (`nx2/pipeline_states.py`) - **`add <name> <lists>... --who --gpu --build --settings --route`** writes `contrib/<name>.bin.xz` and its note. - It keeps only the pipelines that no other contributed list holds. - It does not deduplicate against the tour's list on purpose: the tour's list is replaced whole, so a pipeline left out because the tour had it would be lost when a later tour doesn't find it. - Running it again with the same name adds to that list. - **`tour <lists>... --gpu --build --settings --route`** replaces `tour.bin.xz` and `tour.md`. Nothing else is touched. - **`merge --out <file>`** writes the merged list, the same one `make_game` writes. - **`check`** verifies each source and the merge: - each source is xz, whole, of this build's format, not empty, and holds no pipeline twice; - its note has every field, and its count matches the list; - nothing in `contrib/` is a stray file; - merging twice gives the same bytes, and the result holds exactly the union of the sources. - The old `--out`/`--check <lists>` form still works (the tour script uses it). ## Migration: nothing lost The old list grew by prefix-preserving merges, so its history splits it exactly. I checked this: the 1,261 records from 6643085 are a prefix of 5e4e211's 4,276, which are a prefix of #2029's 11,527. | List | Pipelines | From | |---|---|---| | `contrib/hckrman101-first-play` | 1,261 | #1918 | | `contrib/joey-magnus-linux-dlss-hd-pack` | 3,015 | #1935 (RTX 4070 Ti, Linux, DLSS, HD pack Lite) | | `contrib/jarynclouatre-most-of-the-game` | 7,251 | #2029 (RTX 4070 Ti SUPER 621.14, Windows 11, no mods) | | **Merged** | **11,527** (10,944 graphics, 583 mesh) of 6,672 programs | | - **Same set.** The merged set equals the old file's set: 11,527 records, none missing, none added. - **A lasting check.** `tests/test_pipeline_state_sources.py` checks that the three lists still hold exactly the old file's pipelines, by a SHA-256 of the sorted records. - **Not byte-identical.** The merge order is by name, so the merged file's record order differs from the old file's. The game prepares every record whatever the order. - **Size.** 175 KB across three files, up from 150 KB in one. - **Unknown provenance.** The first list's GPU, driver, settings and route weren't recorded at the time, so its note says so. ## Docs `assets/pipeline_states/README.md` (it replaces `assets/pipeline_states.md`) and `docs/development.md` cover: - how to record a list in game and add it; - how the tour replaces its own list; - that HKSTATE1 records are settings-specific: a list recorded at 1440p with DLSS shares few pipelines with one recorded at 360p. ## Checked - **Passing:** `tests/test_pipeline_states.py`, `tests/test_ship_state_list.py` and the new `tests/test_pipeline_state_sources.py` (8 tests). Also the related `tests/test_appimage_*`, `tests/test_make_game_*`, `tests/test_linux_update.py` and `tests/test_pipeline_state_counts.py`. - **CI coverage:** CI runs the new test through `tests/run_all.py`, so every source is checked and the merge is checked for reproducibility on each PR. - **Not run:** a game build, a game run, the full `tests/run_all.py`, and a real AppImage or Windows package build. The release scripts' changed blocks are exercised by the existing tests (the AppImage block is run, and `release_parts.py package` is run on a small checkout).

Commenting is not possible because the repository is archived.

No reviewers