Build: compilers started by the memory free, up to one per CPU thread #561

Closed

hckrman101 wants to merge 0 commits from build-mem-gate into main

hckrman101 commented

2026-11-08 10:57:01 +00:00

(Migrated from github.com)

The compile stage starts compilers by how much memory is free right now, not by a count fixed at the start. This is a separate change from #428 and #429, and it merges cleanly with both.

Problem. make_game picks the number of compilers once, before the compile. First it gave each one 3.5 GB, now it uses free / 2 with a floor of 4. But what a file needs varies a lot. Peak RSS measured per file on a cold build (clang 23):

a 24 MB chunk of small functions: about 1.5 GB
a chunk with one giant function: up to 2.9 GB
a hot piece: 0.8–1.7 GB
the runtime's C files: 0.2 GB

The PC's other programs (a game, a browser) also come and go during a 10-minute build. So a fixed count either leaves most threads idle or runs the PC out of memory. Here, HOOKSHOT_JOBS=6 was the most that was safe, because more got compilers killed while other programs were running.

Change (nx2/build.py, MemoryGate):

A compile starts only when the free memory, minus what the running compilers may still take, covers the file's estimated peak plus HOOKSHOT_MEM_RESERVE_GB (default 1.5).
The estimate (mem_need) is fitted to the measured peaks: 0.4 GB + 50 MB per MB of C + 250 MB per MB of the file's largest function, and 1.5 GB for a hot piece. Its largest underestimate over the 81 chunks was 160 MB.
The running compilers' resident memory is read, including child processes on Linux. Only what they have yet to take is held back.
Free memory is read again whenever a compile starts or ends. A program opened mid-build makes later compiles wait; one that closes lets more start.
Up to 4 compilers (half the threads on smaller CPUs) always run, as make_game allowed before, so 16 GB PCs keep the floor from issues 502/503/511.
make_game's default is now one compiler per CPU thread, and the gate decides how many actually run. HOOKSHOT_JOBS still caps it, and HOOKSHOT_MEM_GATE=0 turns the gate off.
The objects are byte-for-byte the same. Only the time each compile starts changes.
The Linux builder app is unchanged: it still passes its own --jobs (toolchain.default_jobs, tested by tests/test_linux_builder_jobs.py). Passing none there would give it the same benefit; I left that for a separate change.

Measured on Linux (24 threads, 32 GB, other programs holding about 12 GB). All builds are cold: empty work folder, game already unpacked. The gated builds ran with HOOKSHOT_MEM_RESERVE_GB=6, which kept at least 6 GB free for the rest of the PC. With the default 1.5 GB more compilers would run.

Build	Compilers	Compile	Total
main (`f053f8f`), `HOOKSHOT_JOBS=6`	6	757 s	849 s
main + this	6–9 as memory allowed	608 s	700 s
main + #428 + #429, `HOOKSHOT_JOBS=6`	6	626 s	689 s
main + #428 + #429 + this (two runs)	8–12 as memory allowed	449 / 476 s	510 / 555 s

Free memory while compiling never went below 8.8 GB in the gated builds, with 9–12 GB held by the compilers. At a fixed 6 it dipped to 2.2 GB in one run, because of other programs.
On main alone the last 200 s are one giant chunk (code_021.c, 338 s) that starts late. #429 cuts it into parts and starts it first, so the two changes add up.
Warm builds (nothing to recompile, or only host/) are unchanged: the gate only waits when a compile actually runs.
Tested in game: builds from both bases reach play with normal draw counts.

New test: tests/test_mem_gate.py (game-free; it also reads the free memory on the OS it runs on). It could join the Python job in source-regressions.yml (Ubuntu and Windows): python -I -B tests/test_mem_gate.py. I can't push workflow changes from this account, so that line is left to you.

The compile stage starts compilers by how much memory is free right now, not by a count fixed at the start. This is a separate change from #428 and #429, and it merges cleanly with both. **Problem.** `make_game` picks the number of compilers once, before the compile. First it gave each one 3.5 GB, now it uses free / 2 with a floor of 4. But what a file needs varies a lot. Peak RSS measured per file on a cold build (clang 23): - a 24 MB chunk of small functions: about 1.5 GB - a chunk with one giant function: up to 2.9 GB - a hot piece: 0.8–1.7 GB - the runtime's C files: 0.2 GB The PC's other programs (a game, a browser) also come and go during a 10-minute build. So a fixed count either leaves most threads idle or runs the PC out of memory. Here, `HOOKSHOT_JOBS=6` was the most that was safe, because more got compilers killed while other programs were running. **Change** (`nx2/build.py`, `MemoryGate`): - A compile starts only when the free memory, minus what the running compilers may still take, covers the file's estimated peak plus `HOOKSHOT_MEM_RESERVE_GB` (default 1.5). - The estimate (`mem_need`) is fitted to the measured peaks: 0.4 GB + 50 MB per MB of C + 250 MB per MB of the file's largest function, and 1.5 GB for a hot piece. Its largest underestimate over the 81 chunks was 160 MB. - The running compilers' resident memory is read, including child processes on Linux. Only what they have yet to take is held back. - Free memory is read again whenever a compile starts or ends. A program opened mid-build makes later compiles wait; one that closes lets more start. - Up to 4 compilers (half the threads on smaller CPUs) always run, as `make_game` allowed before, so 16 GB PCs keep the floor from issues 502/503/511. - `make_game`'s default is now one compiler per CPU thread, and the gate decides how many actually run. `HOOKSHOT_JOBS` still caps it, and `HOOKSHOT_MEM_GATE=0` turns the gate off. - The objects are byte-for-byte the same. Only the time each compile starts changes. - The Linux builder app is unchanged: it still passes its own `--jobs` (`toolchain.default_jobs`, tested by `tests/test_linux_builder_jobs.py`). Passing none there would give it the same benefit; I left that for a separate change. **Measured** on Linux (24 threads, 32 GB, other programs holding about 12 GB). All builds are cold: empty work folder, game already unpacked. The gated builds ran with `HOOKSHOT_MEM_RESERVE_GB=6`, which kept at least 6 GB free for the rest of the PC. With the default 1.5 GB more compilers would run. | Build | Compilers | Compile | Total | |---|---|---|---| | main (f053f8f), `HOOKSHOT_JOBS=6` | 6 | 757 s | 849 s | | main + this | 6–9 as memory allowed | 608 s | 700 s | | main + #428 + #429, `HOOKSHOT_JOBS=6` | 6 | 626 s | 689 s | | main + #428 + #429 + this (two runs) | 8–12 as memory allowed | 449 / 476 s | 510 / 555 s | - **Free memory while compiling** never went below 8.8 GB in the gated builds, with 9–12 GB held by the compilers. At a fixed 6 it dipped to 2.2 GB in one run, because of other programs. - **On main alone** the last 200 s are one giant chunk (`code_021.c`, 338 s) that starts late. #429 cuts it into parts and starts it first, so the two changes add up. - **Warm builds** (nothing to recompile, or only `host/`) are unchanged: the gate only waits when a compile actually runs. - **Tested in game:** builds from both bases reach play with normal draw counts. New test: `tests/test_mem_gate.py` (game-free; it also reads the free memory on the OS it runs on). It could join the Python job in `source-regressions.yml` (Ubuntu and Windows): `python -I -B tests/test_mem_gate.py`. I can't push workflow changes from this account, so that line is left to you.

hckrman101 commented

2026-11-08 18:51:11 +00:00

(Migrated from github.com)

Closing in favour of one combined build-speed PR (#428, #429 and #561 together, rebased on current main with #673), opening shortly. On main with #673 a cold Linux build went from 721 s to 451 s here. I'll link it here once it's up.