WIP: feat(vk): Compile shaders (All / Most / Later) and When a shader isn't ready (Wait / Wait briefly / Skip) #2465

Draft

hckrman101 wants to merge 6 commits from hckrman101/feat/background-prepare into main

hckrman101 commented

2026-11-11 20:20:04 +00:00

(Migrated from github.com)

What this does

A first start today prepares every shader before the title: about 11 minutes on an RTX 3090 with a 12-core CPU, longer on smaller PCs. This adds two settings so a player can start sooner, and lets the rest of the work happen while they play.

Compile shaders (builders and the Insert menu's Graphics tab, applies after a restart):

All: everything before the game starts, as today.
Most (default): before the game, the shader programs are translated and the pipelines the shipped list and this folder's notes name are built, as today. Compute programs the game is known to dispatch get all their texture variants; the others get their base variant only. The predicted pipelines and the other compute variants are built in the background while you play.
Later: almost nothing before the game. Everything is built in the background while you play.

On Most and Later the background work runs until the folder has what All gives it. It saves where it got to, carries on at the next start, then stops until an update changes the shaders. While it runs, a small "Shaders: NN%" counter shows in the corner, and the log says when it is done.

When a shader isn't ready (Insert menu, changes at once):

Wait: every draw waits for its shader.
Wait briefly, then skip (default): a frame waits up to 15% of its time in all (2.5 ms at 60 fps, scaled with the fps cap); past that a draw is left out for a frame or two.
Skip: a draw whose shader is not ready is left out at once.

Under every option, a draw that writes the game's memory, or draws into a picture made in the same frame, always waits. Leaving one of those out would leave the game's state wrong or the picture empty. The menu help says so. This is the same rule vk_draw.cpp uses today (RX_VK_PIPE_SKIP=0 still means wait for everything).

How the background work stays out of the way

Its threads run at idle priority. The count is what the game leaves free: one on CPUs with 6 cores or fewer (Steam Deck), otherwise cores minus 3, at most 8. It drops to one or none when available memory is low or under memory pressure.
Between items it waits while the frame's own shader work is queued. What the frame needs is always built first, by the back end's normal workers.
It never takes the render thread's locks. Its pipelines go into the start's pipeline cache (mkw_vk_pipelines.bin), not play's tables.
The pipeline cache is not saved while the background is building into it. Reading that cache while pipelines were made into it crashed Intel's driver. Instead the background pauses once a minute, saves the cache, and writes where it got to (hookshot_prepare_play.txt).
The record that says "prepared" (hookshot_prepared.bin) is only written once everything is done. A start that ends halfway still knows it has work left.
After an area loads, the first frames are held (rendered but not shown, up to 3 s) until their pipelines are ready. That way an area does not appear with pieces missing. The area the save is in, and its neighbours, are built first.

The dispatched-compute list

assets/computes_dispatched.txt: 622 of the game's 4,127 compute programs, by hash only (no game code or data). They are the union of hookshot_computes.bin from 53 local play and test folders. make_game.py writes it beside the game as hookshot_computes_shipped.bin. The release packaging copies it with the pipeline state list.

Builders

The WPF, Qt and Tk builders get a Compile shaders box in place of a tick. make_game.py --compile-shaders all|most|later writes compile_shaders= into hookshot_options.ini, which is the same key the game's menu writes. Any launcher can pass the flag or write the key; without either, the game uses Most. On Later the build skips its own shader step.

Experimental, and the trade-off

Most and Later trade a shorter first start for work during play. Until the background finishes, some objects may appear a moment late, and an area's first frames may be held briefly. All stays available and behaves as before.

Measured

(RTX 3090, 12-core/24-thread CPU, Linux, headless, the game's own frames)

MEASURED_TABLE

Verified

Game-free: tests/test_prepare_play.py (the rules compiled and run: choices and defaults, thread counts, the wait decision for every option including memory-writing draws, the frame budget, the hold, the resume record, the percentage; and the plumbing in the back end, menu, make_game.py and the three builders). tests/test_linux_builder_jobs.py, and the updated fixtures of the tests that compile parts of vk_draw.cpp/vk_main.cpp/the menu. Full suite: VERIFIED_SUITE
VERIFIED_RUNS

Not tested

The WPF builder was not compiled (no .NET SDK on this PC). The Qt builder's box was not opened on screen.
Windows, AMD and Intel GPUs. The Intel cache-save crash is avoided by design but was not reproduced.
Steam Deck or other small CPUs (the one-thread rule is covered by the game-free test only).
NOT_TESTED_EXTRA

Overlaps with open PRs

#2382 changes the same skip check in draw() (may_skip becomes writes_memory(p)). Whichever lands second has a small, mechanical rebase; the rule stays the same.
#2309 removes Stage::module. The background's compute warm-up now goes through one function, warm_compute_one, shared with prepare_run, so #2309 only has one place to change.
#2373 (launcher): the Compile shaders choice is the --compile-shaders flag or the compile_shaders= key, so the launcher's Settings can offer it the same way.

## What this does A first start today prepares every shader before the title: about 11 minutes on an RTX 3090 with a 12-core CPU, longer on smaller PCs. This adds two settings so a player can start sooner, and lets the rest of the work happen while they play. **Compile shaders** (builders and the Insert menu's Graphics tab, applies after a restart): - **All**: everything before the game starts, as today. - **Most** (default): before the game, the shader programs are translated and the pipelines the shipped list and this folder's notes name are built, as today. Compute programs the game is known to dispatch get all their texture variants; the others get their base variant only. The predicted pipelines and the other compute variants are built in the background while you play. - **Later**: almost nothing before the game. Everything is built in the background while you play. On Most and Later the background work runs until the folder has what All gives it. It saves where it got to, carries on at the next start, then stops until an update changes the shaders. While it runs, a small "Shaders: NN%" counter shows in the corner, and the log says when it is done. **When a shader isn't ready** (Insert menu, changes at once): - **Wait**: every draw waits for its shader. - **Wait briefly, then skip** (default): a frame waits up to 15% of its time in all (2.5 ms at 60 fps, scaled with the fps cap); past that a draw is left out for a frame or two. - **Skip**: a draw whose shader is not ready is left out at once. Under every option, a draw that writes the game's memory, or draws into a picture made in the same frame, always waits. Leaving one of those out would leave the game's state wrong or the picture empty. The menu help says so. This is the same rule `vk_draw.cpp` uses today (`RX_VK_PIPE_SKIP=0` still means wait for everything). ### How the background work stays out of the way - Its threads run at idle priority. The count is what the game leaves free: one on CPUs with 6 cores or fewer (Steam Deck), otherwise cores minus 3, at most 8. It drops to one or none when available memory is low or under memory pressure. - Between items it waits while the frame's own shader work is queued. What the frame needs is always built first, by the back end's normal workers. - It never takes the render thread's locks. Its pipelines go into the start's pipeline cache (`mkw_vk_pipelines.bin`), not play's tables. - The pipeline cache is not saved while the background is building into it. Reading that cache while pipelines were made into it crashed Intel's driver. Instead the background pauses once a minute, saves the cache, and writes where it got to (`hookshot_prepare_play.txt`). - The record that says "prepared" (`hookshot_prepared.bin`) is only written once everything is done. A start that ends halfway still knows it has work left. - After an area loads, the first frames are held (rendered but not shown, up to 3 s) until their pipelines are ready. That way an area does not appear with pieces missing. The area the save is in, and its neighbours, are built first. ### The dispatched-compute list `assets/computes_dispatched.txt`: 622 of the game's 4,127 compute programs, by hash only (no game code or data). They are the union of `hookshot_computes.bin` from 53 local play and test folders. `make_game.py` writes it beside the game as `hookshot_computes_shipped.bin`. The release packaging copies it with the pipeline state list. ### Builders The WPF, Qt and Tk builders get a **Compile shaders** box in place of a tick. `make_game.py --compile-shaders all|most|later` writes `compile_shaders=` into `hookshot_options.ini`, which is the same key the game's menu writes. Any launcher can pass the flag or write the key; without either, the game uses Most. On Later the build skips its own shader step. ## Experimental, and the trade-off Most and Later trade a shorter first start for work during play. Until the background finishes, some objects may appear a moment late, and an area's first frames may be held briefly. All stays available and behaves as before. ## Measured (RTX 3090, 12-core/24-thread CPU, Linux, headless, the game's own frames) MEASURED_TABLE ## Verified - Game-free: `tests/test_prepare_play.py` (the rules compiled and run: choices and defaults, thread counts, the wait decision for every option including memory-writing draws, the frame budget, the hold, the resume record, the percentage; and the plumbing in the back end, menu, `make_game.py` and the three builders). `tests/test_linux_builder_jobs.py`, and the updated fixtures of the tests that compile parts of `vk_draw.cpp`/`vk_main.cpp`/the menu. Full suite: VERIFIED_SUITE - VERIFIED_RUNS ## Not tested - The WPF builder was not compiled (no .NET SDK on this PC). The Qt builder's box was not opened on screen. - Windows, AMD and Intel GPUs. The Intel cache-save crash is avoided by design but was not reproduced. - Steam Deck or other small CPUs (the one-thread rule is covered by the game-free test only). - NOT_TESTED_EXTRA ## Overlaps with open PRs - #2382 changes the same skip check in `draw()` (`may_skip` becomes `writes_memory(p)`). Whichever lands second has a small, mechanical rebase; the rule stays the same. - #2309 removes `Stage::module`. The background's compute warm-up now goes through one function, `warm_compute_one`, shared with `prepare_run`, so #2309 only has one place to change. - #2373 (launcher): the Compile shaders choice is the `--compile-shaders` flag or the `compile_shaders=` key, so the launcher's Settings can offer it the same way.

Joey-Magnus (Migrated from github.com) reviewed

2026-11-11 21:26:56 +00:00

Joey-Magnus (Migrated from github.com) left a comment

The description still has unfilled template placeholders (MEASURED_TABLE, VERIFIED_SUITE, VERIFIED_RUNS, NOT_TESTED_EXTRA). The new defaults (Most, Wait briefly) change behaviour for everyone across 42 files, so those numbers matter for review.

The description still has unfilled template placeholders (`MEASURED_TABLE`, `VERIFIED_SUITE`, `VERIFIED_RUNS`, `NOT_TESTED_EXTRA`). The new defaults (Most, Wait briefly) change behaviour for everyone across 42 files, so those numbers matter for review.