Builder: parallel lift output, and the lift skipped when nothing it reads changed #428

Closed

hckrman101 wants to merge 0 commits from build-lift-parallel-cache into main

hckrman101 commented

2026-11-08 00:59:51 +00:00

(Migrated from github.com)

Two builder changes that cut the time spent translating the game. Both are measured on Linux with a Ryzen 9 7900X and 12 jobs.

The lift emits the functions' C from several processes. The lift drops from about 95 s to about 67 s.

Output is byte-for-byte identical: the whole sa/ tree was diffed against a single-process run, and the printed stats match.
Lift counters are merged back from the workers.
Where there is no fork (Windows), it stays a single process. HOOKSHOT_LIFT_JOBS overrides the process count, capped at 16.
New test: tests/test_parallel_emit.py.

make_game skips the lift (and split_hot) when its inputs match the last lift.

Inputs that are hashed: the exefs, all of nx2/ and the vendored a64-kit, the set of HLE functions host/ defines, hot_funcs.txt, chunks.txt, the lift's A64_*/OOT_*/RX_* settings, and the Python version.
Other host/ code is not hashed. That is why a rebuild after a host/ change goes from ~102 s to ~7 s.
The key file is deleted before a lift starts, so a lift that stops halfway is never trusted. It also records the lifted C's file names, sizes and times. If another builder (an older version, or another branch) lifts into the same work folder, the key no longer matches and the next build lifts again.
New test: tests/test_lift_cache.py. tests/test_make_game_cache.py is unchanged; its 2 failures happen on plain main too, here.

Build	Before	After
Nothing changed	102 s	6 s
Only `host/` changed	102 s	7 s

The final build was tested in game and reaches play with normal draw counts. A separate PR balances the compile stage.

Two builder changes that cut the time spent translating the game. Both are measured on Linux with a Ryzen 9 7900X and 12 jobs. **The lift emits the functions' C from several processes.** The lift drops from about 95 s to about 67 s. - Output is byte-for-byte identical: the whole `sa/` tree was diffed against a single-process run, and the printed stats match. - Lift counters are merged back from the workers. - Where there is no `fork` (Windows), it stays a single process. `HOOKSHOT_LIFT_JOBS` overrides the process count, capped at 16. - New test: `tests/test_parallel_emit.py`. **make_game skips the lift (and split_hot) when its inputs match the last lift.** - Inputs that are hashed: the exefs, all of `nx2/` and the vendored a64-kit, the set of HLE functions `host/` defines, `hot_funcs.txt`, `chunks.txt`, the lift's `A64_*`/`OOT_*`/`RX_*` settings, and the Python version. - Other `host/` code is not hashed. That is why a rebuild after a `host/` change goes from ~102 s to ~7 s. - The key file is deleted before a lift starts, so a lift that stops halfway is never trusted. It also records the lifted C's file names, sizes and times. If another builder (an older version, or another branch) lifts into the same work folder, the key no longer matches and the next build lifts again. - New test: `tests/test_lift_cache.py`. `tests/test_make_game_cache.py` is unchanged; its 2 failures happen on plain main too, here. | Build | Before | After | |---|---|---| | Nothing changed | 102 s | 6 s | | Only `host/` changed | 102 s | 7 s | The final build was tested in game and reaches play with normal draw counts. A separate PR balances the compile stage.

hckrman101 commented

2026-11-08 18:51:08 +00:00

(Migrated from github.com)

Closing in favour of one combined build-speed PR (#428, #429 and #561 together, rebased on current main with #673), opening shortly. On main with #673 a cold Linux build went from 721 s to 451 s here. I'll link it here once it's up.