perf(build): start the Vulkan back end's longest files first #2772

Open

github-sync wants to merge 2 commits from github/pr/2772 into main

github-sync commented

2026-11-12 12:37:05 +00:00

Owner

@corentinbonjour opened this pull request on GitHub · 2026-11-12 12:22 UTC · original

The Vulkan back end's compile jobs were started in name order, so vk_draw.cpp, its longest file (30 to 45 s), started near the end and ran alone for the last seconds of every back-end build. Starting the longest files first removes that tail.

What changed

nx2/build.py: new start_order(jobs, vk_jobs, history) gives the order the compiles are submitted in. The back end's jobs go longest first, by this PC's build_times.json (per_file_seconds) when HOOKSHOT_TIMES is set, else by source size. Every other job keeps its place: the game's chunks are already sorted by compile_cost (#2324), and this does not touch them.
jobs and vk_jobs are not reordered, so objs.rsp and vk_objs.rsp, and the link, stay the same. Checked: vk_objs.rsp from a build before and after is byte-identical once the work folder name is normalised.
build_times.json is now read once (history) and used for both the countdown and the order.

Measurements

RX_VK_ONLY=1 python -I nx2/build.py --standalone --jobs N, HOOKSHOT_MEM_GATE=0, objects and object cache deleted before each run, so every run compiles all 213 files. Before and after were interleaved, 5 repetitions each at --jobs 8 and --jobs 24, on a 24-thread laptop (275HX), llvm-mingw clang 23. The PC was also running other builds and the game during the runs, so absolute times spread widely. The paired differences are the useful numbers.

--jobs 24	before	after
wall, median (min to max)	103.9 s (73 to 178)	77.5 s (56 to 159)
compile phase, median	96 s	75 s
tail after the last-but-one file ends, median (min to max)	9 s (5 to 13), always `vk_draw.cpp`	0 s (0 to 1)
`vk_draw.cpp` ends at	72 to 163 s	35 to 73 s

Paired by repetition, after minus before at 24 jobs: -20, -26, -18, -12, -17 s (median -18 s, faster in every repetition). A separate trial pair gave 85 s against 56 s.

At --jobs 8, before 134.7 s median (117 to 228), after 126.5 s (102 to 188). The paired differences (+2, -3, -22, +53, -127 s) are within the noise: with 8 compilers the rest of the work is much longer than vk_draw.cpp, and its tail was already 0 to 2 s. No change was expected there and none was measured.

How it was tested

tests/test_build_times.py: new StartOrderTests. The back end's jobs come out by size with no record and by measured seconds with one, the other jobs keep their place, jobs and vk_jobs are left unchanged, and main submits in that order. They fail on the current build.py.
tests/test_compile_retry_jobs.py runs main's compile and retry statements with stubs, and now gives them order.
python -I -B tests/run_all.py with llvm-mingw on PATH and RX_CPM set: 444 passed, 13 failed, 38 skipped. The failing checks are the same as on an untouched main checkout. They fail on this PC because tools are missing.

Notes

With no build_times.json, sizes decide. ui_fonts.cpp (3 MB of font bytes at -O0, 2 s) then starts first and takes one compiler for 2 s. The C++ files that matter come right after it: imgui.cpp, imgui_widgets.cpp, vk_draw.cpp, vk_texture.cpp, vk_menu.cpp and vk_replay.cpp.
In a full game build the back end's jobs still come after the game's, as before. Only their order among themselves changes.

> [@corentinbonjour](https://github.com/corentinbonjour) opened this pull request on GitHub · 2026-11-12 12:22 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2772) The Vulkan back end's compile jobs were started in name order, so `vk_draw.cpp`, its longest file (30 to 45 s), started near the end and ran alone for the last seconds of every back-end build. Starting the longest files first removes that tail. ## What changed - `nx2/build.py`: new `start_order(jobs, vk_jobs, history)` gives the order the compiles are submitted in. The back end's jobs go longest first, by this PC's `build_times.json` (`per_file_seconds`) when `HOOKSHOT_TIMES` is set, else by source size. Every other job keeps its place: the game's chunks are already sorted by `compile_cost` (#2324), and this does not touch them. - `jobs` and `vk_jobs` are not reordered, so `objs.rsp` and `vk_objs.rsp`, and the link, stay the same. Checked: `vk_objs.rsp` from a build before and after is byte-identical once the work folder name is normalised. - `build_times.json` is now read once (`history`) and used for both the countdown and the order. ## Measurements `RX_VK_ONLY=1 python -I nx2/build.py --standalone --jobs N`, `HOOKSHOT_MEM_GATE=0`, objects and object cache deleted before each run, so every run compiles all 213 files. Before and after were interleaved, 5 repetitions each at `--jobs 8` and `--jobs 24`, on a 24-thread laptop (275HX), llvm-mingw clang 23. The PC was also running other builds and the game during the runs, so absolute times spread widely. The paired differences are the useful numbers. | --jobs 24 | before | after | |---|---|---| | wall, median (min to max) | 103.9 s (73 to 178) | 77.5 s (56 to 159) | | compile phase, median | 96 s | 75 s | | tail after the last-but-one file ends, median (min to max) | 9 s (5 to 13), always `vk_draw.cpp` | 0 s (0 to 1) | | `vk_draw.cpp` ends at | 72 to 163 s | 35 to 73 s | Paired by repetition, after minus before at 24 jobs: -20, -26, -18, -12, -17 s (median -18 s, faster in every repetition). A separate trial pair gave 85 s against 56 s. At `--jobs 8`, before 134.7 s median (117 to 228), after 126.5 s (102 to 188). The paired differences (+2, -3, -22, +53, -127 s) are within the noise: with 8 compilers the rest of the work is much longer than `vk_draw.cpp`, and its tail was already 0 to 2 s. No change was expected there and none was measured. ## How it was tested - `tests/test_build_times.py`: new `StartOrderTests`. The back end's jobs come out by size with no record and by measured seconds with one, the other jobs keep their place, `jobs` and `vk_jobs` are left unchanged, and `main` submits in that order. They fail on the current `build.py`. - `tests/test_compile_retry_jobs.py` runs `main`'s compile and retry statements with stubs, and now gives them `order`. - `python -I -B tests/run_all.py` with llvm-mingw on PATH and `RX_CPM` set: 444 passed, 13 failed, 38 skipped. The failing checks are the same as on an untouched `main` checkout. They fail on this PC because tools are missing. ## Notes - With no `build_times.json`, sizes decide. `ui_fonts.cpp` (3 MB of font bytes at -O0, 2 s) then starts first and takes one compiler for 2 s. The C++ files that matter come right after it: `imgui.cpp`, `imgui_widgets.cpp`, `vk_draw.cpp`, `vk_texture.cpp`, `vk_menu.cpp` and `vk_replay.cpp`. - In a full game build the back end's jobs still come after the game's, as before. Only their order among themselves changes.

github-sync added 1 commit

2026-11-12 12:37:05 +00:00

perf(build): start the vulkan back end's longest files first c5398f4b8d

The back end's ~210 compile jobs were submitted in name order, so vk_draw.cpp
(the longest, 30-45 s) started near the end and ran alone. They now start
longest first, by build_times.json when there is one, else by size. The
jobs and vk_jobs lists, and so the link order, are unchanged.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

github-sync commented

2026-11-12 14:47:53 +00:00

Author

Owner

@Joey-Magnus approved on GitHub · 2026-11-12 14:35 UTC · original

I merged this into current main and ran tests/test_build_times.py and tests/test_compile_retry_jobs.py. Both pass. It also merges cleanly with #2775 and #2706, the other open build.py changes I checked. start_order only reorders the submits. jobs and vk_jobs keep their order, so objs.rsp and vk_objs.rsp are unchanged, and you checked the rsp is byte-identical. The game chunks keep their compile_cost order. history is read once and serves both the countdown and the order, which is a small cleanup. One edge, not a blocker: with HOOKSHOT_TIMES set but no measured back-end file in build_times.json, per_file_seconds gives every vk file the flat 1.3 s fallback, so sorted() keeps name order and nothing changes for that one build. After a build the files are recorded through @@file, so it only matters once. Falling back to src_size when no vk file is in history would cover it. The paired --jobs 24 differences are your own measurement, and they are consistent across repetitions.

> [@Joey-Magnus](https://github.com/Joey-Magnus) approved on GitHub · 2026-11-12 14:35 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2772#pullrequestreview-5876251968) I merged this into current main and ran tests/test_build_times.py and tests/test_compile_retry_jobs.py. Both pass. It also merges cleanly with #2775 and #2706, the other open build.py changes I checked. start_order only reorders the submits. jobs and vk_jobs keep their order, so objs.rsp and vk_objs.rsp are unchanged, and you checked the rsp is byte-identical. The game chunks keep their compile_cost order. history is read once and serves both the countdown and the order, which is a small cleanup. One edge, not a blocker: with HOOKSHOT_TIMES set but no measured back-end file in build_times.json, per_file_seconds gives every vk file the flat 1.3 s fallback, so sorted() keeps name order and nothing changes for that one build. After a build the files are recorded through @@file, so it only matters once. Falling back to src_size when no vk file is in history would cover it. The paired --jobs 24 differences are your own measurement, and they are consistent across repetitions.

github-sync added 1 commit

2026-11-12 15:48:59 +00:00

perf(build): order the back end by size when the history has none of it 5cf9fc55e4

With HOOKSHOT_TIMES set but no back-end file in build_times.json,
per_file_seconds gave every one the same flat guess and the order fell
back to names. The back end's files are then ordered by size.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

github-sync commented

2026-11-12 15:53:06 +00:00

Author

Owner

@corentinbonjour commented on GitHub · 2026-11-12 15:06 UTC · original

Thanks, done: start_order now uses the history only when it holds at least one of the back end's files, and orders them by size otherwise. A new test (a history with only a game chunk in it) fails without the change and passes with it.

> [@corentinbonjour](https://github.com/corentinbonjour) commented on GitHub · 2026-11-12 15:06 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2772#issuecomment-6827311242) Thanks, done: start_order now uses the history only when it holds at least one of the back end's files, and orders them by size otherwise. A new test (a history with only a game chunk in it) fails without the change and passes with it.

Commenting is not possible because the repository is archived.

No reviewers