Start the game while the last precompiled pipelines still build #702

Closed

Palayop7239 wants to merge 5 commits from fix/precompile-hand-over into main

Palayop7239 commented

2026-11-08 17:47:40 +00:00

(Migrated from github.com)

Depends on #700 (it uses its pipeline entries' started flag and the quick build for a waiting draw). Until that one is merged, this PR's diff includes its changes too. Only the commit fix(vk): start the game while the last precompiled pipelines still build belongs here; the others come from #700 and from merging main.

What

At a start, once every noted pipeline has begun, the precompile waits RX_VK_PRECOMPILE_TAIL_MS (3 s by default; 0 waits for all, as before), then lets the game start with the rest still building:

Entries made up front. Each note's PipeEntry is made before the workers start and is marked begun. A draw that needs one meanwhile draws with a quick pipeline or waits (pipeline_build_waited) instead of starting a second full build. The workers never touch the pipelines map.
Workers own their state. They keep it alive themselves (a shared Run) and are detached instead of joined.
Compute builds claim their program. A compute build takes the program's compute_state as a dispatch's worker build does, so a dispatch meanwhile is left out rather than built twice. One that fails gives the program back to the dispatch.
The builder's prepare-only run still waits for all, since it has no game to start (hand_over defaults to false).

Why

The precompile builds every pipeline noted in earlier sessions before the game starts. On an RX 6650 XT (v2.0.2), AMD's optimizer took about 30 s each on seven of them, in parallel:

the console sat at 99%;
two starts in a row were closed as hung;
the second dropped the notes (the last two starts did not get through the precompile), so the next session compiled everything in play again.

Issues #615 and #643 describe the same "stuck at 99%, close and reopen several times".

Testing

tests/test_precompile_hand_over.py reads vk_draw.cpp, vk_main.cpp and vk_internal.h. It checks:
- entries are registered before the workers start, and the workers stay off the map;
- the compute-state claim;
- the hand-over happens only once all have begun and only when asked;
- detach versus join;
- the prepare-only run still waits.
tests/run_all.py picks it up; it only reads the source, so it needs no compiler.
hookshot_vk.dll compiles from this branch on main, and with the related warm-up PR merged in. A v2.0.2-based build of it is installed on the RX 6650 XT.
Played on the same RX 6650 XT (v2.0.2 with this DLL), at the start after the one that dropped the notes:
```
vk: precompile: 4799 programs, 1252 graphics and 3837 compute pipelines in 4941 ms; the game starts with 6 still building (they finish beside it)
vk: graphics pipeline of 6f2c76e46484ba4d: SLOW compile, 17009 ms
vk: graphics pipeline of 7637f6f4a052af15: SLOW compile, 17218 ms
vk: graphics pipeline of a936f393a2003184: SLOW compile, 17811 ms
vk: graphics pipeline of 939ce9134c8143d3: SLOW compile, 17817 ms
```
- The game started 4.9 s into the precompile. Without the hand-over it would have waited at least another 18 s for those four.
- They finished in the background while the intro ran at 59–60 fps, with no draw waiting for them (0.00 waits for shaders or pipelines).
- In the first minute and a half in game: no submit over 337 ms, no SLOW cache lookup line, at most 0.03 waits a frame (0.0 ms).

🤖 Generated with Claude Code

> **Depends on https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/700** (it uses its pipeline entries' `started` flag and the quick build for a waiting draw). Until that one is merged, this PR's diff includes its changes too. Only the commit `fix(vk): start the game while the last precompiled pipelines still build` belongs here; the others come from #700 and from merging `main`. ## What At a start, once every noted pipeline has **begun**, the precompile waits `RX_VK_PRECOMPILE_TAIL_MS` (3 s by default; `0` waits for all, as before), then lets the game start with the rest still building: - **Entries made up front.** Each note's `PipeEntry` is made before the workers start and is marked begun. A draw that needs one meanwhile draws with a quick pipeline or waits (`pipeline_build_waited`) instead of starting a second full build. The workers never touch the `pipelines` map. - **Workers own their state.** They keep it alive themselves (a shared `Run`) and are detached instead of joined. - **Compute builds claim their program.** A compute build takes the program's `compute_state` as a dispatch's worker build does, so a dispatch meanwhile is left out rather than built twice. One that fails gives the program back to the dispatch. - **The builder's prepare-only run still waits for all**, since it has no game to start (`hand_over` defaults to `false`). ## Why The precompile builds every pipeline noted in earlier sessions before the game starts. On an RX 6650 XT (v2.0.2), AMD's optimizer took about 30 s each on seven of them, in parallel: - the console sat at 99%; - two starts in a row were closed as hung; - the second dropped the notes (`the last two starts did not get through the precompile`), so the next session compiled everything in play again. Issues #615 and #643 describe the same "stuck at 99%, close and reopen several times". ## Testing - `tests/test_precompile_hand_over.py` reads `vk_draw.cpp`, `vk_main.cpp` and `vk_internal.h`. It checks: - entries are registered before the workers start, and the workers stay off the map; - the compute-state claim; - the hand-over happens only once all have begun and only when asked; - detach versus join; - the prepare-only run still waits. `tests/run_all.py` picks it up; it only reads the source, so it needs no compiler. - `hookshot_vk.dll` compiles from this branch on `main`, and with the related warm-up PR merged in. A v2.0.2-based build of it is installed on the RX 6650 XT. - **Played** on the same RX 6650 XT (v2.0.2 with this DLL), at the start after the one that dropped the notes: ``` vk: precompile: 4799 programs, 1252 graphics and 3837 compute pipelines in 4941 ms; the game starts with 6 still building (they finish beside it) vk: graphics pipeline of 6f2c76e46484ba4d: SLOW compile, 17009 ms vk: graphics pipeline of 7637f6f4a052af15: SLOW compile, 17218 ms vk: graphics pipeline of a936f393a2003184: SLOW compile, 17811 ms vk: graphics pipeline of 939ce9134c8143d3: SLOW compile, 17817 ms ``` - The game started 4.9 s into the precompile. Without the hand-over it would have waited at least another 18 s for those four. - They finished in the background while the intro ran at 59–60 fps, with no draw waiting for them (`0.00 waits for shaders or pipelines`). - In the first minute and a half in game: no submit over 337 ms, no `SLOW cache lookup` line, at most 0.03 waits a frame (0.0 ms). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

NotRexed commented

2026-11-09 05:30:05 +00:00

(Migrated from github.com)