WIP: fix(build): bake the shaders with the back end's own prepare, opt-in at the build #2474

Draft

hckrman101 wants to merge 1 commit from hckrman101/fix/offline-bake into main

hckrman101 commented

2026-11-11 20:33:30 +00:00

(Migrated from github.com)

On main (after #2323: the program cache split into vk_progcache.cpp). Follows up #1707 and #1954.

Why the #1707 bake was refused

The game only reads a program cache (hookshot_programs.bin) whose stamp is its own. That stamp covers two things:

the translator's sources (HOOKSHOT_SHADER_STAMP from nx2/build.py), and
this PC's GPU and settings, read at runtime (translator_stamp in vk_shader.cpp): mesh shaders or the fallback, the subgroup
lanes and the stages pinned to 32, the cooperative-matrix layout measured on the device, the depth mode, the C++ library's
struct layouts, and the stamped RX_VK_* / EDEN_* switches.

nx2/bake_ds is a hand-made copy of the translator and the record writer. It can't compute the GPU half without the GPU. It had
also fallen behind main: no canonical_answer, no compute local memory from the control section, and the old record head
(kind/answers/workgroup/shared; main also writes the questions and five mesh-fallback words). The game refused its banks on the
stamp, and with a matching stamp it would have misread the records. Any copy will drift again the next time the translator or the
record format changes.

What changes

The bake now runs the back end's own code, so it writes the same stamp and the same bytes as the game.

hookshot_vk exports mkwvk_bake(folder, into, workers). It creates the device the game would, then runs prepare_run with
translation only: no pipelines, no pipeline cache, no state list. It works in a folder of its own: game_module_path takes
that folder through game_folder_override. If the game folder already holds a cache with the current stamp, the bake skips
the work and prints @@bake current.
nx2/bake.py live loads the build's back end with ctypes and calls it at idle priority (SCHED_IDLE on Linux, IDLE_PRIORITY_CLASS
on Windows). On Linux it also ends with the build. The live_* helpers stage <game>/.hookshot_bake: the shader list is
hard-linked, the settings and the back end are copied. Once the build is done they move the cache, its index and the prepare's
record into the game folder, with the record last.
nx2/make_game.py handles both builders' paths with start_bake / finish_bake. Before the lift it makes the shader list and
the back end (RX_VK_ONLY, objects shared with the full build; the AppImage's prebuilt one is used as is). It then starts the
bake beside the lift and the compile, and installs the result before --prepare-shaders. If the bake fails or finds nothing to
do, the prepare translates as before.
The prepare's record gets a new warmed value, kWarmedBaked. When the game's prepare (the build's --prepare-shaders, or a
first start) sees it:
- every program in the cache is kept;
- kept compute programs are not enumerated again (seed_compute's kept_whole), but their pipelines are still built in the
  prepare's worker pool, as before;
- only programs missing from the cache are translated;
- the state list is noted as before.
So a bank that was cut short, deleted, or made for another GPU or translator (a different stamp) falls back to translating, as
without a bake.
Off by default: HOOKSHOT_BAKE=1 turns it on, and only when the prepare runs too. The measurements below show it does not
shorten build plus first start on this PC.

Measured

RTX 3090, a 12-thread half of a Ryzen 9 7900X (oot-status timing CPUs, checked per CPU), cold build (no object cache, no lift
cache), empty driver cache, same source for both runs:

	old flow	with the bake
back end before the lift	–	23 s
lift	102 s	137 s (the bake ran beside it)
compile	692 s	716 s
prepare: translation + compute pipelines	235 s	179 s (0 programs translated)
prepare: predicted pipelines	288 s	294 s
prepare: precompile	182 s	161 s
build + prepare	1512 s	1521 s

Bake alone, 12 workers: 38,809 programs (52,464 records, 6.6 GB) in 60 s. Peak RSS 4.5 GB.
The game accepts the baked bank: vk: program cache …: 46666 program keys kept, 6592008 KB, then
vk: the shaders were translated at the build (baked)…, then vk: prepared 38809 shader programs (0 new, …) in 5.4 s. Without
the bake: 242.7 s.
Why it doesn't add up to a saving: translation is only ~60 s of 12-thread work (~720 CPU-seconds). The rest of the prepare is
~6,900 CPU-seconds of driver pipeline compiles (compute ~10k, predicted ~56k, listed ~25k). The build cost as much as the bake
saved. Even at idle priority, the bake slowed the mostly single-threaded lift through the SMT siblings of its cores. Building
the back end first added 23 s.
Idle CPU during the old build: ~880 CPU-seconds in the lift and ~920 in the compile. That is the most any work moved into the
build could save (~150 s at best).
An attempt in the same direction for the pipelines did not help on NVIDIA/Linux. A prepare run in a separate process, then the
game's precompile with that process's mkw_vk_pipelines.bin, took 415 s, the same as cold. On this driver the pipeline cache
file only points into the driver's disk cache. The driver picks the disk cache file by the executable's name: a copy of
python named hookshot shared the game's file, while the game run from another folder and python under its own name did not
share it. A pipeline bake would therefore have to run as an executable named hookshot.

Verified

Same bytes. All 52,464 baked records match records the game's own cold prepare wrote (same .so, same GPU), compared with
the cache's record layout and Shader::Info's container members zeroed. The only differences are uninitialized struct padding
bytes, checked against clang's record layout. The game's own run has 938 more records: the state list's texture variants,
which the prepare still makes after a bake.
Fallback. The game's --prepare-shaders on a baked folder, run up to its translation line:
- the bank as baked: 46666 program keys kept, baked, 0 new;
- the bank cut to half its size: 18383 program keys kept, 28283 new. The rest was translated and the total is the same
  46,666;
- the bank deleted, record kept: 0 program keys kept, 46666 new, 9925 compute variants;
- another stamp (the game run with RX_VK_MARKERS=1): the cache is made again from nothing, 46666 new, and the record is
  not taken as baked.
Game-free tests: tests/test_bake_live.py builds the record helpers and the baked decision from vk_draw.cpp, and
program_cache_file_stamp from vk_progcache.cpp, then checks:
- the stamp handshake: a baked record is taken; another stamp, another shader list, a missing record or the game's own record
  is not taken for a bake;
- the cache file stamp: a missing, short or foreign file reads as 0;
- the source invariants;
- the Python side: staging, install order, a cache without its record, result parsing, a library without mkwvk_bake, and
  make_game's switches and fallbacks.
tests/run_all.py bake prepare make_game program_cache shader_census passes, except test_bake_portability.py, which
needs zstandard (missing on this PC, unrelated). test_prepared_stamps.py and test_prepare_cache_save.py pass with the
prepare's code as changed here.

Not tested

A headless play run after a baked first start ("shaders built" / "draws left out" in play, compared with main), and
screenshots. The programs the game loads are byte-identical to the ones it makes itself, so no difference is expected.
Windows: the builder path is the same make_game.py, and the bake uses IDLE_PRIORITY_CLASS and ctypes with an ANSI path.
AMD and Intel GPUs.
16 GB PCs (the bake holds ~4.5 GB beside the lift).
Repeat runs: the e2e numbers are one run each.

On main (after #2323: the program cache split into vk_progcache.cpp). Follows up #1707 and #1954. ## Why the #1707 bake was refused The game only reads a program cache (`hookshot_programs.bin`) whose stamp is its own. That stamp covers two things: - the translator's sources (`HOOKSHOT_SHADER_STAMP` from `nx2/build.py`), and - this PC's GPU and settings, read at runtime (`translator_stamp` in `vk_shader.cpp`): mesh shaders or the fallback, the subgroup lanes and the stages pinned to 32, the cooperative-matrix layout measured on the device, the depth mode, the C++ library's struct layouts, and the stamped `RX_VK_*` / `EDEN_*` switches. `nx2/bake_ds` is a hand-made copy of the translator and the record writer. It can't compute the GPU half without the GPU. It had also fallen behind main: no `canonical_answer`, no compute local memory from the control section, and the old record head (`kind/answers/workgroup/shared`; main also writes the questions and five mesh-fallback words). The game refused its banks on the stamp, and with a matching stamp it would have misread the records. Any copy will drift again the next time the translator or the record format changes. ## What changes The bake now runs the back end's own code, so it writes the same stamp and the same bytes as the game. - `hookshot_vk` exports `mkwvk_bake(folder, into, workers)`. It creates the device the game would, then runs `prepare_run` with translation only: no pipelines, no pipeline cache, no state list. It works in a folder of its own: `game_module_path` takes that folder through `game_folder_override`. If the game folder already holds a cache with the current stamp, the bake skips the work and prints `@@bake current`. - `nx2/bake.py live` loads the build's back end with ctypes and calls it at idle priority (SCHED_IDLE on Linux, IDLE_PRIORITY_CLASS on Windows). On Linux it also ends with the build. The `live_*` helpers stage `<game>/.hookshot_bake`: the shader list is hard-linked, the settings and the back end are copied. Once the build is done they move the cache, its index and the prepare's record into the game folder, with the record last. - `nx2/make_game.py` handles both builders' paths with `start_bake` / `finish_bake`. Before the lift it makes the shader list and the back end (`RX_VK_ONLY`, objects shared with the full build; the AppImage's prebuilt one is used as is). It then starts the bake beside the lift and the compile, and installs the result before `--prepare-shaders`. If the bake fails or finds nothing to do, the prepare translates as before. - The prepare's record gets a new `warmed` value, `kWarmedBaked`. When the game's prepare (the build's `--prepare-shaders`, or a first start) sees it: - every program in the cache is kept; - kept compute programs are not enumerated again (`seed_compute`'s `kept_whole`), but their pipelines are still built in the prepare's worker pool, as before; - only programs missing from the cache are translated; - the state list is noted as before. So a bank that was cut short, deleted, or made for another GPU or translator (a different stamp) falls back to translating, as without a bake. - **Off by default: `HOOKSHOT_BAKE=1` turns it on**, and only when the prepare runs too. The measurements below show it does not shorten build plus first start on this PC. ## Measured RTX 3090, a 12-thread half of a Ryzen 9 7900X (oot-status timing CPUs, checked per CPU), cold build (no object cache, no lift cache), empty driver cache, same source for both runs: | | old flow | with the bake | |---|---|---| | back end before the lift | – | 23 s | | lift | 102 s | 137 s (the bake ran beside it) | | compile | 692 s | 716 s | | prepare: translation + compute pipelines | 235 s | 179 s (0 programs translated) | | prepare: predicted pipelines | 288 s | 294 s | | prepare: precompile | 182 s | 161 s | | **build + prepare** | **1512 s** | **1521 s** | - Bake alone, 12 workers: 38,809 programs (52,464 records, 6.6 GB) in 60 s. Peak RSS 4.5 GB. - The game accepts the baked bank: `vk: program cache …: 46666 program keys kept, 6592008 KB`, then `vk: the shaders were translated at the build (baked)…`, then `vk: prepared 38809 shader programs (0 new, …) in 5.4 s`. Without the bake: 242.7 s. - Why it doesn't add up to a saving: translation is only ~60 s of 12-thread work (~720 CPU-seconds). The rest of the prepare is ~6,900 CPU-seconds of driver pipeline compiles (compute ~10k, predicted ~56k, listed ~25k). The build cost as much as the bake saved. Even at idle priority, the bake slowed the mostly single-threaded lift through the SMT siblings of its cores. Building the back end first added 23 s. - Idle CPU during the old build: ~880 CPU-seconds in the lift and ~920 in the compile. That is the most any work moved into the build could save (~150 s at best). - An attempt in the same direction for the pipelines did not help on NVIDIA/Linux. A prepare run in a separate process, then the game's precompile with that process's `mkw_vk_pipelines.bin`, took 415 s, the same as cold. On this driver the pipeline cache file only points into the driver's disk cache. The driver picks the disk cache file by the executable's name: a copy of python named `hookshot` shared the game's file, while the game run from another folder and python under its own name did not share it. A pipeline bake would therefore have to run as an executable named `hookshot`. ## Verified - **Same bytes.** All 52,464 baked records match records the game's own cold prepare wrote (same .so, same GPU), compared with the cache's record layout and Shader::Info's container members zeroed. The only differences are uninitialized struct padding bytes, checked against clang's record layout. The game's own run has 938 more records: the state list's texture variants, which the prepare still makes after a bake. - **Fallback.** The game's `--prepare-shaders` on a baked folder, run up to its translation line: - the bank as baked: `46666 program keys kept`, `baked`, `0 new`; - the bank cut to half its size: `18383 program keys kept`, `28283 new`. The rest was translated and the total is the same 46,666; - the bank deleted, record kept: `0 program keys kept`, `46666 new, 9925 compute variants`; - another stamp (the game run with `RX_VK_MARKERS=1`): the cache is made again from nothing, `46666 new`, and the record is not taken as baked. - **Game-free tests:** `tests/test_bake_live.py` builds the record helpers and the `baked` decision from `vk_draw.cpp`, and `program_cache_file_stamp` from `vk_progcache.cpp`, then checks: - the stamp handshake: a baked record is taken; another stamp, another shader list, a missing record or the game's own record is not taken for a bake; - the cache file stamp: a missing, short or foreign file reads as 0; - the source invariants; - the Python side: staging, install order, a cache without its record, result parsing, a library without `mkwvk_bake`, and `make_game`'s switches and fallbacks. `tests/run_all.py bake prepare make_game program_cache shader_census` passes, except `test_bake_portability.py`, which needs `zstandard` (missing on this PC, unrelated). `test_prepared_stamps.py` and `test_prepare_cache_save.py` pass with the prepare's code as changed here. ## Not tested - A headless play run after a baked first start ("shaders built" / "draws left out" in play, compared with main), and screenshots. The programs the game loads are byte-identical to the ones it makes itself, so no difference is expected. - Windows: the builder path is the same `make_game.py`, and the bake uses `IDLE_PRIORITY_CLASS` and `ctypes` with an ANSI path. - AMD and Intel GPUs. - 16 GB PCs (the bake holds ~4.5 GB beside the lift). - Repeat runs: the e2e numbers are one run each.

Joey-Magnus (Migrated from github.com) reviewed

2026-11-11 21:27:00 +00:00

Joey-Magnus (Migrated from github.com) left a comment

On Windows, nx2/bake.py encodes the folder path as mbcs (the ANSI code page), which breaks the UTF-8 path rule (tests/test_utf8_paths.py) and fails for user folders with non-ASCII names. Also, by the description's own timing there's no gain (1512 s vs 1521 s), so is it worth merging even opt-in? BAKE.md at the repo root probably belongs under docs/.

On Windows, `nx2/bake.py` encodes the folder path as `mbcs` (the ANSI code page), which breaks the UTF-8 path rule (`tests/test_utf8_paths.py`) and fails for user folders with non-ASCII names. Also, by the description's own timing there's no gain (1512 s vs 1521 s), so is it worth merging even opt-in? `BAKE.md` at the repo root probably belongs under `docs/`.