perf(prepare): workers bench script and process WS in prepare log #2728

Open

github-sync wants to merge 4 commits from github/pr/2728 into main

github-sync commented

2026-11-12 10:34:10 +00:00

Owner

@yeetbot90 opened this pull request on GitHub · 2026-11-12 09:35 UTC · original

Summary - Add `scripts/prepare_workers_bench.py` to A/B prepare worker counts via `--prepare-shaders`, rank by predicted pipelines/s, and write `hookshot_prepare_workers.txt`. - Honour that file in `prepare_pool_workers` when no `HOOKSHOT_SHADER_WORKERS` / `RX_VK_PREPARE_WORKERS` is set (env still wins). - Log process working set / peak on `vk: prepare resources` for measurement. Stacked on #2725 / #2727. ## Test plan - [x] `python -I -B tests/test_prepare_pool_workers.py` / `test_prepare_cpus.py` - [x] CI: Python source checks green on Windows + Ubuntu (all four jobs). - [x] `python -I scripts/prepare_workers_bench.py --game D:\Hookshot_OOT --parse-only` (prior cold-ish log: ~38.7 pipe/s, peak WS 16048 MB) - [x] Live tip DLL on AMD RX 9070 XT: wrote `hookshot_prepare_workers.txt` with `workers=8`; prepare logged `prepare resources before predicted pipelines: ... process 10815 MB (peak 10815)` and `predicted pipelines: building 50623 on 8 workers (hookshot_prepare_workers.txt, usable 24)` — 50623 pipelines in 76.1 s on 8 threads (~666 pipe/s) with warm driver cache (~3.4 GB) - [x] Cold multi-count A/B was stopped short of a finished 4, 8, 12, 16, 24 table. On the measured DLL (AMD RX 9070 XT, empty VkCache, 50010 predicted pipelines, HOOKSHOT_SHADER_WORKERS=4) two long samples stayed near 11-13 pipelines/s and had not finished: 36440/50010 in 2777 s, then 30264/50010 in 2693 s. A separate cold run of the stock updated build, which compiled on 24 threads, did finish: 50043 pipelines in 1034.2 s (about 48/s). That is enough to leave hookshot_prepare_workers.txt at workers=8 from the warm run above. Four workers is the wrong cold default. The 8/12/16/24 cold ranking was not measured. Made with Cursor

> [@yeetbot90](https://github.com/yeetbot90) opened this pull request on GitHub · 2026-11-12 09:35 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2728) ## Summary - Add `scripts/prepare_workers_bench.py` to A/B prepare worker counts via `--prepare-shaders`, rank by predicted pipelines/s, and write `hookshot_prepare_workers.txt`. - Honour that file in `prepare_pool_workers` when no `HOOKSHOT_SHADER_WORKERS` / `RX_VK_PREPARE_WORKERS` is set (env still wins). - Log process working set / peak on `vk: prepare resources` for measurement. Stacked on #2725 / #2727. ## Test plan - [x] `python -I -B tests/test_prepare_pool_workers.py` / `test_prepare_cpus.py` - [x] CI: Python source checks green on Windows + Ubuntu (all four jobs). - [x] `python -I scripts/prepare_workers_bench.py --game D:\Hookshot_OOT --parse-only` (prior cold-ish log: ~38.7 pipe/s, peak WS 16048 MB) - [x] Live tip DLL on AMD RX 9070 XT: wrote `hookshot_prepare_workers.txt` with `workers=8`; prepare logged `prepare resources before predicted pipelines: ... process 10815 MB (peak 10815)` and `predicted pipelines: building 50623 on 8 workers (hookshot_prepare_workers.txt, usable 24)` — **50623 pipelines in 76.1 s on 8 threads** (~666 pipe/s) with warm driver cache (~3.4 GB) - [x] Cold multi-count A/B was stopped short of a finished 4, 8, 12, 16, 24 table. On the measured DLL (AMD RX 9070 XT, empty VkCache, 50010 predicted pipelines, HOOKSHOT_SHADER_WORKERS=4) two long samples stayed near 11-13 pipelines/s and had not finished: 36440/50010 in 2777 s, then 30264/50010 in 2693 s. A separate cold run of the stock updated build, which compiled on 24 threads, did finish: 50043 pipelines in 1034.2 s (about 48/s). That is enough to leave hookshot_prepare_workers.txt at workers=8 from the warm run above. Four workers is the wrong cold default. The 8/12/16/24 cold ranking was not measured. Made with [Cursor](https://cursor.com)

github-sync added 4 commits

2026-11-12 10:34:10 +00:00

perf(prepare): env override for prepare worker counts f29faf6382

Centralize pool sizing in prepare_pool_workers so HOOKSHOT_SHADER_WORKERS /
RX_VK_PREPARE_WORKERS can A/B fewer threads when RAM is tight, without changing
the default half/usable/background sizes. Log a one-line CPU and RAM snapshot
at prepare start.

Co-authored-by: Cursor <[email protected]>

perf(prepare): opt-in RAM guard for prepare worker counts 446dab5ead

Cap auto-sized prepare pools by available RAM when HOOKSHOT_PREPARE_RAM_GUARD
is on, and always force a tiny pool under ~2 GB so a prepare does not thrash.
Explicit worker overrides still win for A/B.

Co-authored-by: Cursor <[email protected]>

perf(prepare): workers bench script and process WS in prepare log 632595672d

Add scripts/prepare_workers_bench.py to A/B HOOKSHOT_SHADER_WORKERS via
--prepare-shaders and write hookshot_prepare_workers.txt. Honour that file
when no env override is set, and log process working-set / peak on the
prepare resources line for those runs.

Co-authored-by: Cursor <[email protected]>

fix(tests): stub prepare pool helpers in stall and CRT-safe pool check b8960d66cd

test_prepare_stall's predict harness needed PreparePool / prepare_pool_workers /
prepare_resources_log after the override wiring. The pool-workers check now
stubs prepare_ram_mb (shared fake avail on Windows and Linux) and compiles with
_CRT_SECURE_NO_WARNINGS so getenv is not an error under MSVC headers.

Co-authored-by: Cursor <[email protected]>

github-sync referenced this pull request

2026-11-12 14:44:42 +00:00

perf(prepare): env override for prepare worker counts #2725

github-sync referenced this pull request

2026-11-12 14:44:44 +00:00

perf(prepare): opt-in RAM guard for prepare worker counts #2727

github-sync commented

2026-11-12 14:44:47 +00:00

Author

Owner

@Joey-Magnus reviewed on GitHub · 2026-11-12 14:34 UTC · original

This is #2727 plus 63259567, and #2729 contains it unchanged, so I'd close this one and keep the combined PR. My notes on the bench and the workers file are on #2729. In short:

scripts/prepare_workers_bench.py:27 ranks by the last 10 s progress line ("N of M pipelines in S s"), not the final "predicted N pipelines ... building them on T threads" summary. So the last stretch is never counted, and a warm run that finishes inside 10 s gives no number at all (your own #2721 run did exactly that).
hookshot_prepare_workers.txt is honoured by the in-play background pool too (vk_progcache.cpp:775 comes before the Background branch). A bench that picks 24 would then run 24 translation threads during gameplay, next to the game's own.
The 76.1 s / ~666 pipe/s number is a warm-cache run, which says nothing about the cold first start the file is meant to tune.

> [@Joey-Magnus](https://github.com/Joey-Magnus) reviewed on GitHub · 2026-11-12 14:34 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2728#pullrequestreview-5876241160) This is #2727 plus 63259567, and #2729 contains it unchanged, so I'd close this one and keep the combined PR. My notes on the bench and the workers file are on #2729. In short: - scripts/prepare_workers_bench.py:27 ranks by the last 10 s progress line ("N of M pipelines in S s"), not the final "predicted N pipelines ... building them on T threads" summary. So the last stretch is never counted, and a warm run that finishes inside 10 s gives no number at all (your own #2721 run did exactly that). - `hookshot_prepare_workers.txt` is honoured by the in-play background pool too (vk_progcache.cpp:775 comes before the Background branch). A bench that picks 24 would then run 24 translation threads during gameplay, next to the game's own. - The 76.1 s / ~666 pipe/s number is a warm-cache run, which says nothing about the cold first start the file is meant to tune.

github-sync referenced this pull request

2026-11-12 14:44:48 +00:00

perf(vk): log GPL availability for stage-library A/B #2729

github-sync referenced this pull request

2026-11-12 19:01:05 +00:00

perf(prepare): the segment heap, so the driver's compiles run side by side on Windows #2895

Commenting is not possible because the repository is archived.

No reviewers