Build speed: parallel lift and lift skip, compile order by cost, compilers started by the free memory (supersedes #428, #429, #561) #728

Merged

hckrman101 merged 5 commits from build-speed-2 into main

2026-11-08 18:57:50 +00:00

hckrman101 commented

2026-11-08 18:52:24 +00:00

(Migrated from github.com)

This combines three build-time changes into one branch, rebased on current main (66892e6b, with #673 and the build timer). Each change is still its own commit(s), so each part can be reviewed and bisected separately. Supersedes #428, #429 and #561.

What each part does

1. Lift: parallel output, and skipped when nothing it reads changed (was #428, 2 commits)

Several processes emit the functions' C. The files are byte-for-byte the same as a single-process lift. HOOKSHOT_LIFT_JOBS=<n> sets the process count (1 = the old single process). Where there is no fork (Windows), the lift stays one process.
make_game skips the lift (and split_hot) when its inputs match the last lift: the exefs, nx2/ and the vendored a64-kit, the set of HLE functions host/ defines, hot_funcs.txt, chunks.txt, the lift's A64_*/OOT_*/RX_* settings, and the Python version. Other host/ code is not part of the key, so a rebuild after a host/ change no longer translates the game again.
The key is deleted before a lift starts, and it also records the lifted C's names, sizes and times. A lift that stopped halfway, or another builder that lifted into the same work folder, is never trusted. To force a lift, delete <work>/sa/gen/.lift_key.

2. Compile order by cost (was #429, 2 commits)

Chunks that would cost more than 1.5× a normal chunk are cut at function starts into code_NNN_k.c parts of about equal cost. The cost model: a function's compile time grows with roughly the square of its size. No other chunk boundary moves.
The lift writes gen/chunk_costs.txt, and build.py compiles the longest files first, with hot pieces weighted ×4.
split_hot deals its call groups out to the 8 hot pieces by cost instead of bytes. The groups themselves, and so the inlining, are unchanged.
The lifted code is the same; only the files it is spread over change.

3. Compilers started by the free memory (was #561, 1 commit, adapted to #673)

#673 made both builder windows leave the job count to make_game, which picks free / 2 GB once, at the start. This branch keeps that change. On top of it, nx2/build.py's MemoryGate starts each compile only when the free memory, minus what the running compilers may still take, covers the file's estimated peak plus HOOKSHOT_MEM_RESERVE_GB (default 1.5).
Free memory is read again whenever a compile starts or ends. A program opened mid-build makes later compiles wait, and one that closes lets more start.
make_game's default becomes one compiler per CPU thread, and the gate decides how many actually run.
The gate's floor follows #673: up to 4 always run on Windows (which pages rather than fails), 1 on Linux.
tests/test_linux_builder_jobs.py is updated for the new default: one per thread, and the floor is checked on the gate.
The builder's compile countdown (the build timer on main) counts on the compilers the gate expects the free memory to hold, at 1.5 GB each, not on --jobs. With one compiler per thread, it would otherwise promise too short a time.
The objects are byte-for-byte the same; only the time each compile starts changes.

Turning each part off

Part	Off
Parallel lift	`HOOKSHOT_LIFT_JOBS=1`
Lift skip	delete `<work>/sa/gen/.lift_key` (the next build lifts)
Memory gate	`HOOKSHOT_MEM_GATE=0` (then `HOOKSHOT_JOBS` / `--jobs` / one per thread, as before)
Fewer compilers	`HOOKSHOT_JOBS=<n>` still caps it; `HOOKSHOT_MEM_RESERVE_GB=<gb>` keeps more memory free

Measured

Setup: Linux, Ryzen 9 7900X (24 threads), 32 GB, other programs holding 12-19 GB. Each build is make_game with the default job settings each side gives a user (no HOOKSHOT_JOBS), at nice 19 / SCHED_IDLE. The cold builds start from an empty work folder with the game already unpacked, with the hot list, play profile and chunk boundaries in work/. Before and after builds ran one at a time, in alternation, on the same PC.

Against main with #673 (d09dacd8, just before the build timer; the timer only adds timing output):

Cold build	Total	Lift	Compile	Compilers	Peak (all compilers)	Least memory free
main	721 s	111 s	604 s	9	15.4 GB	3.3 GB
this branch	451 s (−37%)	63 s	383 s	up to 15 at once	14.2 GB	4.0 GB

For a command-line build on this PC, #673 does not change the count: it chose 9 compilers here, the same as main before it. Its gain is in the Linux builder window, which chose fewer before. So the whole 37% here comes on top of #673.

Against the main before #673 (5a5c7e65), same setup: cold builds took 671 s and 730 s on main, 461 s on this branch (−31% / −37%).

Rebuilds (an earlier run on the older main without the play profile in work/; repeat runs on current main will follow):

Rebuild	main	this branch
Only a `host/*.c` file changed	92 s	6 s
Nothing changed	92 s	6 s expected (the lift is skipped, as in the row above)

More runs will be added here: a second cold pair, warm and one-host/-file rebuilds on current main, HOOKSHOT_JOBS=6 on both sides, and an in-game fps check.

In game: an earlier build of this combination (on the older main, both sides built without the play profile) reached play with normal draw counts, at the same fps as main's build (55.1 vs 54.2 fps over 40 s, with no flicker events).

Tests

All of this was run again on 66892e6b. A short build (the lift and the first objects) gives the timer's @@plan / @@eta / @@file lines, and the compile runs longest first.
New, game-free: tests/test_parallel_emit.py, tests/test_lift_cache.py, tests/test_chunk_split.py and tests/test_mem_gate.py all pass. tests/test_linux_builder_jobs.py (updated), tests/test_build_times.py and tests/test_builder_status.py pass.
I ran every tests/test_*.py on main and on this branch. The same two fail on both, and neither is about this change:
- test_cpp_preflight.py needs --cxx, which CI passes.
- test_make_game_cache.py fails only locally: a clang probe in its fixture.
The four new tests are not in source-regressions.yml yet. Each one runs as python -I -B tests/<name>.py if you want to add them.

This combines three build-time changes into one branch, rebased on current main (66892e6b, with #673 and the build timer). Each change is still its own commit(s), so each part can be reviewed and bisected separately. **Supersedes #428, #429 and #561.** ## What each part does **1. Lift: parallel output, and skipped when nothing it reads changed** (was #428, 2 commits) - Several processes emit the functions' C. The files are byte-for-byte the same as a single-process lift. `HOOKSHOT_LIFT_JOBS=<n>` sets the process count (1 = the old single process). Where there is no `fork` (Windows), the lift stays one process. - `make_game` skips the lift (and `split_hot`) when its inputs match the last lift: the exefs, `nx2/` and the vendored a64-kit, the set of HLE functions `host/` defines, `hot_funcs.txt`, `chunks.txt`, the lift's `A64_*`/`OOT_*`/`RX_*` settings, and the Python version. Other `host/` code is not part of the key, so a rebuild after a `host/` change no longer translates the game again. - The key is deleted before a lift starts, and it also records the lifted C's names, sizes and times. A lift that stopped halfway, or another builder that lifted into the same work folder, is never trusted. To force a lift, delete `<work>/sa/gen/.lift_key`. **2. Compile order by cost** (was #429, 2 commits) - Chunks that would cost more than 1.5× a normal chunk are cut at function starts into `code_NNN_k.c` parts of about equal cost. The cost model: a function's compile time grows with roughly the square of its size. No other chunk boundary moves. - The lift writes `gen/chunk_costs.txt`, and `build.py` compiles the longest files first, with hot pieces weighted ×4. - `split_hot` deals its call groups out to the 8 hot pieces by cost instead of bytes. The groups themselves, and so the inlining, are unchanged. - The lifted code is the same; only the files it is spread over change. **3. Compilers started by the free memory** (was #561, 1 commit, adapted to #673) - #673 made both builder windows leave the job count to `make_game`, which picks `free / 2 GB` once, at the start. This branch keeps that change. On top of it, `nx2/build.py`'s `MemoryGate` starts each compile only when the free memory, minus what the running compilers may still take, covers the file's estimated peak plus `HOOKSHOT_MEM_RESERVE_GB` (default 1.5). - Free memory is read again whenever a compile starts or ends. A program opened mid-build makes later compiles wait, and one that closes lets more start. - `make_game`'s default becomes one compiler per CPU thread, and the gate decides how many actually run. - The gate's floor follows #673: up to 4 always run on Windows (which pages rather than fails), 1 on Linux. - `tests/test_linux_builder_jobs.py` is updated for the new default: one per thread, and the floor is checked on the gate. - The builder's compile countdown (the build timer on main) counts on the compilers the gate expects the free memory to hold, at 1.5 GB each, not on `--jobs`. With one compiler per thread, it would otherwise promise too short a time. - The objects are byte-for-byte the same; only the time each compile starts changes. ## Turning each part off | Part | Off | |---|---| | Parallel lift | `HOOKSHOT_LIFT_JOBS=1` | | Lift skip | delete `<work>/sa/gen/.lift_key` (the next build lifts) | | Memory gate | `HOOKSHOT_MEM_GATE=0` (then `HOOKSHOT_JOBS` / `--jobs` / one per thread, as before) | | Fewer compilers | `HOOKSHOT_JOBS=<n>` still caps it; `HOOKSHOT_MEM_RESERVE_GB=<gb>` keeps more memory free | ## Measured Setup: Linux, Ryzen 9 7900X (24 threads), 32 GB, other programs holding 12-19 GB. Each build is `make_game` with the default job settings each side gives a user (no `HOOKSHOT_JOBS`), at nice 19 / SCHED_IDLE. The cold builds start from an empty work folder with the game already unpacked, with the hot list, play profile and chunk boundaries in `work/`. Before and after builds ran one at a time, in alternation, on the same PC. **Against main with #673 (d09dacd8, just before the build timer; the timer only adds timing output):** | Cold build | Total | Lift | Compile | Compilers | Peak (all compilers) | Least memory free | |---|---|---|---|---|---|---| | main | 721 s | 111 s | 604 s | 9 | 15.4 GB | 3.3 GB | | this branch | **451 s (−37%)** | 63 s | 383 s | up to 15 at once | 14.2 GB | 4.0 GB | For a command-line build on this PC, #673 does not change the count: it chose 9 compilers here, the same as main before it. Its gain is in the Linux builder window, which chose fewer before. So the whole 37% here comes on top of #673. **Against the main before #673 (5a5c7e65), same setup:** cold builds took 671 s and 730 s on main, 461 s on this branch (−31% / −37%). **Rebuilds** (an earlier run on the older main without the play profile in `work/`; repeat runs on current main will follow): | Rebuild | main | this branch | |---|---|---| | Only a `host/*.c` file changed | 92 s | **6 s** | | Nothing changed | 92 s | 6 s expected (the lift is skipped, as in the row above) | More runs will be added here: a second cold pair, warm and one-`host/`-file rebuilds on current main, `HOOKSHOT_JOBS=6` on both sides, and an in-game fps check. **In game:** an earlier build of this combination (on the older main, both sides built without the play profile) reached play with normal draw counts, at the same fps as main's build (55.1 vs 54.2 fps over 40 s, with no flicker events). ## Tests - All of this was run again on 66892e6b. A short build (the lift and the first objects) gives the timer's `@@plan` / `@@eta` / `@@file` lines, and the compile runs longest first. - New, game-free: `tests/test_parallel_emit.py`, `tests/test_lift_cache.py`, `tests/test_chunk_split.py` and `tests/test_mem_gate.py` all pass. `tests/test_linux_builder_jobs.py` (updated), `tests/test_build_times.py` and `tests/test_builder_status.py` pass. - I ran every `tests/test_*.py` on main and on this branch. The same two fail on both, and neither is about this change: - `test_cpp_preflight.py` needs `--cxx`, which CI passes. - `test_make_game_cache.py` fails only locally: a clang probe in its fixture. - The four new tests are not in `source-regressions.yml` yet. Each one runs as `python -I -B tests/<name>.py` if you want to add them.

shayderrr commented

2026-11-08 18:57:43 +00:00

(Migrated from github.com)