Linux builder: as many compilers as on Windows, from the free memory (#605, #531) #673

Merged

amaralthalesalves merged 1 commit from fix/linux-builder-jobs into main

2026-11-08 17:03:30 +00:00

amaralthalesalves commented

2026-11-08 16:33:17 +00:00

(Migrated from github.com)

Closes #605. Closes #531.

This aligns the Linux builder with the current Windows policy. It adds no new rule: the Linux window kept an older, stricter one that the Windows builder had already tried and dropped.

What

Compilation on Linux was already parallel (nx2/build.py runs one clang process per worker), but the Linux builder window chose far fewer workers than Windows:

	Policy	16 threads, 32 GB (24 free)
Windows (`make_game.default_jobs`)	free memory / 2 GB, at least `min(4, threads / 2)`, at most one per thread	12
Linux window, before (`toolchain.default_jobs(3.5)`)	80% of total memory / 3.5 GB, at most one per thread	7
Linux window, after	same as Windows (without the floor, see below)	12

That fits the 29% CPU in #531 and the comment in #605.

builder/linux/hookshot_builder.py: without HOOKSHOT_JOBS or RX_JOBS the window passes no --jobs, so make_game chooses. HOOKSHOT_JOBS and RX_JOBS work as before; a bad RX_JOBS now shows the error dialog instead of an uncaught exception.
nx2/make_game.py:
- The floor of min(4, threads / 2) stays Windows' only. Its own comment says "Windows pages rather than fails"; Linux stops a compiler that does not fit, and build.py compiles that file again alone, so on Linux the free memory alone decides.
- The default is worked out only when neither --jobs nor HOOKSHOT_JOBS is given. The "build: N compilers at once" line used to be printed even with --jobs, with a number the build did not use (on Linux, always).
- A bad HOOKSHOT_JOBS or a negative --jobs stops with a message instead of a traceback.
readme.md: the Linux section describes the automatic choice.

Why the two rules differed

From the history, the Linux rule fell behind rather than being chosen:

0440738 (11-06): toolchain.default_jobs(3.5) arrives with the a64-kit (after SkyboundTotk PR 14), to avoid running 16 GB machines out of memory.
8301bf1 (11-07): the AppImage's window uses it. Windows then used threads / 2, so at that point Linux was the cautious one.
6036423 (11-07, #418): Windows moves to free memory / 2 GB with a floor. Only make_game.default_jobs changes.
fa22c2d (11-08): Windows tries 3.5 GB per compiler after an out-of-memory compile, and build.py starts compiling failed files again alone.
3adfb63 (11-08): Windows goes back to / 2 GB with the floor, because at 3.5 GB 16 GB PCs got 1-2 compilers and builds took hours (#502, #503, #511).

The Linux window calls a different function and never followed steps 3-5. It kept the 3.5 GB rule that step 5 dropped, on top of 80% of total memory.

Test

python -I -B tests/test_linux_builder_jobs.py: 15 tests OK. The window tests are updated (no --jobs when nothing is set, RX_JOBS still honoured, bad RX_JOBS stops before any side effect). New tests cover default_jobs on Linux and Windows (same count, floor Windows-only, one per thread at most) and the --jobs → HOOKSHOT_JOBS → automatic order. This file already runs in Game-free source regressions.
python -I -B tests/test_make_game_cache.py: OK.
nx2/make_game.py run directly: without --jobs it prints the count it uses; with --jobs 3 it prints none; --jobs -1 and HOOKSHOT_JOBS=n stop with a message.
Not tested: a full game build (I have no game files here), so the CPU use was not measured. On a PC short of free memory, more compilers can mean some are stopped and compiled again alone at the end, the same trade-off Windows made in 3adfb63.

Closes #605. Closes #531. > **This aligns the Linux builder with the current Windows policy.** It adds no new rule: the Linux window kept an older, stricter one that the Windows builder had already tried and dropped. ## What Compilation on Linux was already parallel (`nx2/build.py` runs one `clang` process per worker), but the Linux builder window chose far fewer workers than Windows: | | Policy | 16 threads, 32 GB (24 free) | |---|---|---| | Windows (`make_game.default_jobs`) | free memory / 2 GB, at least `min(4, threads / 2)`, at most one per thread | **12** | | Linux window, before (`toolchain.default_jobs(3.5)`) | 80% of **total** memory / 3.5 GB, at most one per thread | **7** | | Linux window, after | same as Windows (without the floor, see below) | **12** | That fits the 29% CPU in #531 and the comment in #605. - `builder/linux/hookshot_builder.py`: without `HOOKSHOT_JOBS` or `RX_JOBS` the window passes no `--jobs`, so `make_game` chooses. `HOOKSHOT_JOBS` and `RX_JOBS` work as before; a bad `RX_JOBS` now shows the error dialog instead of an uncaught exception. - `nx2/make_game.py`: - The floor of `min(4, threads / 2)` stays Windows' only. Its own comment says "Windows pages rather than fails"; Linux stops a compiler that does not fit, and `build.py` compiles that file again alone, so on Linux the free memory alone decides. - The default is worked out only when neither `--jobs` nor `HOOKSHOT_JOBS` is given. The "build: N compilers at once" line used to be printed even with `--jobs`, with a number the build did not use (on Linux, always). - A bad `HOOKSHOT_JOBS` or a negative `--jobs` stops with a message instead of a traceback. - `readme.md`: the Linux section describes the automatic choice. ## Why the two rules differed From the history, the Linux rule fell behind rather than being chosen: 1. `0440738` (11-06): `toolchain.default_jobs(3.5)` arrives with the a64-kit (after SkyboundTotk PR 14), to avoid running 16 GB machines out of memory. 2. `8301bf1` (11-07): the AppImage's window uses it. Windows then used `threads / 2`, so at that point Linux was the cautious one. 3. `6036423` (11-07, #418): Windows moves to free memory / 2 GB with a floor. Only `make_game.default_jobs` changes. 4. `fa22c2d` (11-08): Windows tries 3.5 GB per compiler after an out-of-memory compile, and `build.py` starts compiling failed files again alone. 5. `3adfb63` (11-08): Windows goes back to / 2 GB with the floor, because at 3.5 GB 16 GB PCs got 1-2 compilers and builds took hours (#502, #503, #511). The Linux window calls a different function and never followed steps 3-5. It kept the 3.5 GB rule that step 5 dropped, on top of 80% of total memory. ## Test - `python -I -B tests/test_linux_builder_jobs.py`: 15 tests OK. The window tests are updated (no `--jobs` when nothing is set, `RX_JOBS` still honoured, bad `RX_JOBS` stops before any side effect). New tests cover `default_jobs` on Linux and Windows (same count, floor Windows-only, one per thread at most) and the `--jobs` → `HOOKSHOT_JOBS` → automatic order. This file already runs in *Game-free source regressions*. - `python -I -B tests/test_make_game_cache.py`: OK. - `nx2/make_game.py` run directly: without `--jobs` it prints the count it uses; with `--jobs 3` it prints none; `--jobs -1` and `HOOKSHOT_JOBS=n` stop with a message. - Not tested: a full game build (I have no game files here), so the CPU use was not measured. On a PC short of free memory, more compilers can mean some are stopped and compiled again alone at the end, the same trade-off Windows made in `3adfb63`.

shayderrr commented

2026-11-08 17:03:25 +00:00

(Migrated from github.com)