fix(build): every step make_game.py runs side by side keeps to the workers it was given, not only the compile #2114

Merged
HumanBean merged 1 commit from HumanBean/Longshot:fix/build-jobs-every-step into main 2026-11-21 19:55:02 +00:00
Owner

The launcher's "Build workers" setting only reached the compile, so the translation, the button prompts and the shader preparation still used every CPU thread and the PC was unusable while an update ran (LongshotLauncher#148, LongshotLauncher#163). This makes every step make_game.py runs side by side keep to the count it is given, which LongshotLauncher#235 now always passes (Automatic leaves a quarter of the threads free).

Summary

make_game.py --jobs N (Linux launcher) and LONGSHOT_JOBS=N (Windows launcher) set how many files compile at once, and
nothing else. The other parallel steps sized themselves from the CPU count. Now one helper, nx2/build_jobs.py, holds the
rule, and each step uses it:

  • nx2/make_game.py main: when a count is given (--jobs, or LONGSHOT_JOBS already set; not 0), it is put in
    LONGSHOT_JOBS for every step it starts. When none is given nothing is set, so each step keeps today's count.
  • nx2/chunks.py file_jobs: the lift's emit processes, its per-file passes, its side process and nx2/discover.py's
    walks (lift.py passes file_jobs()) take at most that many. The memory rule still applies below it, and
    LONGSHOT_LIFT_JOBS still overrides it.
  • nx2/prompt_assets.py face_jobs: the prompts' face processes, the same way (LONGSHOT_PROMPT_JOBS still overrides it).
  • nx2/make_game.py prepare_shaders: the game's --prepare-shaders starts on N of the CPUs the builder may use
    (build_jobs.children_on_cpus: the calling thread's mask on Linux, the process's on Windows, set while the child is
    created and put back straight after). The game's environment drops the LONGSHOT_ build variables (game_env), so
    the affinity mask is how the count gets to it. Its preparation pools already follow the mask (usable_cpus in
    gpu/vk/vk_progcache.cpp, tests/test_prepare_cpus.py), so the game itself is unchanged. Without a count the game
    gets every CPU, as before.
  • nx2/build.py: already takes --jobs. Unchanged.

Left alone, and why:

  • nx2/nup.py (reading the .nup), unpack.py, text_assets.py and shader_census.py run on one thread. The overlapped
    package read (overlap_package_read) is one extra reader thread, and it is already off at --jobs 1.
  • third_party_objects.stamp's five threads are five short compiler probes at the start of the compile (a few seconds).
  • The linker: lld uses its own threads, but a link takes about 3 s (build.py notes it), so I didn't add a --threads
    flag. That flag would also change every link key.
  • third_party_objects.py make and scripts/validate_spirv.py are release and dev tools that make_game.py doesn't run.

Read first: nx2/build_jobs.py, then the calls in chunks.file_jobs, prompt_assets.face_jobs and
make_game.main / prepare_shaders.

Testing

Tested: Linux (CachyOS, 24 threads), game-free:

  • tests/test_build_jobs.py (new) checks four things. The count's parsing. That make_game.main exports a given count and
    exports none when no count is given. That the lift's per-file pool (chunks._each, sized by file_jobs) runs on at
    most 4 distinct worker processes with LONGSHOT_JOBS=4, and on 16 without it. That prepare_shaders starts its child
    on 2 CPUs with LONGSHOT_JOBS=2 and on all of them without, and that the builder gets its own mask back afterwards.
    The child is a stand-in that prints len(os.sched_getaffinity(0)), not the game.
  • tests/test_lift_workers.py and tests/test_prompt_speed.py (FaceJobs) gained the ceiling cases, and their setUp now
    clears LONGSHOT_JOBS so a value exported in the shell doesn't change them.
  • tests/test_lift_cache.py: LONGSHOT_JOBS is listed in LIFT_ENV_NOT_OUTPUT, so it is not part of the lift key.
  • python -I -B tests/run_all.py --jobs 8 (not --strict, in a jj workspace): 730 passed, 88 failed, 116 skipped. I ran
    the 88 failures again on main's sources (1bd93a48) in the same setup, and 87 of them fail there too: missing tools,
    checks that need a git checkout, and source checks that already fail on main. tests/test_lift_cache.py is one of
    them; its one failure, "...and the others still by their code", happens on main as well. The 88th was
    tests/test_prompt_skip.py, whose list of the prompts' local imports needed build_jobs. It is now on the list with
    kit and long_paths: it decides how many processes draw the faces, not what they draw. That test passes now.

No real build or game run: the game binary is unchanged, and the only change to the shader preparation is that it may
start on fewer CPUs, a case usable_cpus already handles (taskset, Process Lasso).

Still to test:

  • Windows (LongshotLauncher with #235, Build workers on Automatic or a small number): run an update. Expect Task
    Manager to show the translation and the shader preparation using about the chosen number of threads instead of all
    of them, and the log's discovery: ... in N processes to show at most that N (by hand).
  • Linux, the same through the launcher's --jobs (by hand).

Measurements

Verdict: a synthetic run (no game data) with LONGSHOT_JOBS=4 on a 24-thread PC uses 4 workers in each step, and
the same as before without it. This change limits CPU use on purpose. It doesn't make anything faster.

Setup: CachyOS, 24 CPU threads. A script calls the functions as make_game.py's steps do. chunks._available_gb is set
to 64 because this PC was busy and its free memory would otherwise have lowered the lift count further. 200 CPU-bound
items go through chunks._each, with distinct worker pids counted, and the stand-in game runs through
make_game.prepare_shaders.

Before: main @ 1bd93a48, no count Before: main @ 1bd93a48, LONGSHOT_JOBS=4 After: this PR @ 4ff025d4, no count After: this PR @ 4ff025d4, LONGSHOT_JOBS=4
lift processes (file_jobs, distinct worker pids over 200 items) 16 16 16 4
prompt face processes (face_jobs(5)) 5 5 5 4
CPUs the shader preparation's child may use 24 24 24 4

Risks

  • nx2/chunks.py is one of the lift's sources, so the first update after this merges translates the game again (a few
    minutes, once). LONGSHOT_JOBS itself isn't in the lift key.
  • prompt_assets.py is part of the prompts' key, so they are made again once (seconds).
  • A small count now really slows the translation and the shader preparation. That is the point of the setting, but a
    player who set 1 or 2 for memory reasons will see slower updates than before.
  • On Windows, if the affinity mask can't be read or set (more than 64 CPUs, or a process already spread over several
    processor groups), the preparation gets every CPU, as before.
  • To go back to the old behaviour: leave the count unset (Automatic before #235), or revert this PR.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says. (Game-free only: the game binary is unchanged, and no real build or prepare was run here.)
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • Docs updated.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

🤖 Generated with Claude Code

The launcher's "Build workers" setting only reached the compile, so the translation, the button prompts and the shader preparation still used every CPU thread and the PC was unusable while an update ran ([LongshotLauncher#148](https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/issues/148), [LongshotLauncher#163](https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/issues/163)). This makes every step `make_game.py` runs side by side keep to the count it is given, which [LongshotLauncher#235](https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/235) now always passes (Automatic leaves a quarter of the threads free). ## Summary `make_game.py --jobs N` (Linux launcher) and `LONGSHOT_JOBS=N` (Windows launcher) set how many files compile at once, and nothing else. The other parallel steps sized themselves from the CPU count. Now one helper, `nx2/build_jobs.py`, holds the rule, and each step uses it: - `nx2/make_game.py` `main`: when a count is given (`--jobs`, or `LONGSHOT_JOBS` already set; not 0), it is put in `LONGSHOT_JOBS` for every step it starts. When none is given nothing is set, so each step keeps today's count. - `nx2/chunks.py` `file_jobs`: the lift's emit processes, its per-file passes, its side process and `nx2/discover.py`'s walks (`lift.py` passes `file_jobs()`) take at most that many. The memory rule still applies below it, and `LONGSHOT_LIFT_JOBS` still overrides it. - `nx2/prompt_assets.py` `face_jobs`: the prompts' face processes, the same way (`LONGSHOT_PROMPT_JOBS` still overrides it). - `nx2/make_game.py` `prepare_shaders`: the game's `--prepare-shaders` starts on N of the CPUs the builder may use (`build_jobs.children_on_cpus`: the calling thread's mask on Linux, the process's on Windows, set while the child is created and put back straight after). The game's environment drops the `LONGSHOT_` build variables (`game_env`), so the affinity mask is how the count gets to it. Its preparation pools already follow the mask (`usable_cpus` in `gpu/vk/vk_progcache.cpp`, `tests/test_prepare_cpus.py`), so the game itself is unchanged. Without a count the game gets every CPU, as before. - `nx2/build.py`: already takes `--jobs`. Unchanged. Left alone, and why: - `nx2/nup.py` (reading the `.nup`), `unpack.py`, `text_assets.py` and `shader_census.py` run on one thread. The overlapped package read (`overlap_package_read`) is one extra reader thread, and it is already off at `--jobs 1`. - `third_party_objects.stamp`'s five threads are five short compiler probes at the start of the compile (a few seconds). - The linker: lld uses its own threads, but a link takes about 3 s (`build.py` notes it), so I didn't add a `--threads` flag. That flag would also change every link key. - `third_party_objects.py make` and `scripts/validate_spirv.py` are release and dev tools that `make_game.py` doesn't run. Read first: `nx2/build_jobs.py`, then the calls in `chunks.file_jobs`, `prompt_assets.face_jobs` and `make_game.main` / `prepare_shaders`. ## Testing **Tested:** Linux (CachyOS, 24 threads), game-free: - `tests/test_build_jobs.py` (new) checks four things. The count's parsing. That `make_game.main` exports a given count and exports none when no count is given. That the lift's per-file pool (`chunks._each`, sized by `file_jobs`) runs on at most 4 distinct worker processes with `LONGSHOT_JOBS=4`, and on 16 without it. That `prepare_shaders` starts its child on 2 CPUs with `LONGSHOT_JOBS=2` and on all of them without, and that the builder gets its own mask back afterwards. The child is a stand-in that prints `len(os.sched_getaffinity(0))`, not the game. - `tests/test_lift_workers.py` and `tests/test_prompt_speed.py` (FaceJobs) gained the ceiling cases, and their setUp now clears `LONGSHOT_JOBS` so a value exported in the shell doesn't change them. - `tests/test_lift_cache.py`: `LONGSHOT_JOBS` is listed in `LIFT_ENV_NOT_OUTPUT`, so it is not part of the lift key. - `python -I -B tests/run_all.py --jobs 8` (not `--strict`, in a jj workspace): 730 passed, 88 failed, 116 skipped. I ran the 88 failures again on main's sources (1bd93a48) in the same setup, and 87 of them fail there too: missing tools, checks that need a git checkout, and source checks that already fail on main. `tests/test_lift_cache.py` is one of them; its one failure, "...and the others still by their code", happens on main as well. The 88th was `tests/test_prompt_skip.py`, whose list of the prompts' local imports needed `build_jobs`. It is now on the list with `kit` and `long_paths`: it decides how many processes draw the faces, not what they draw. That test passes now. No real build or game run: the game binary is unchanged, and the only change to the shader preparation is that it may start on fewer CPUs, a case `usable_cpus` already handles (taskset, Process Lasso). **Still to test:** - Windows (LongshotLauncher with #235, Build workers on Automatic or a small number): run an update. Expect Task Manager to show the translation and the shader preparation using about the chosen number of threads instead of all of them, and the log's `discovery: ... in N processes` to show at most that N (by hand). - Linux, the same through the launcher's `--jobs` (by hand). ## Measurements **Verdict:** a synthetic run (no game data) with `LONGSHOT_JOBS=4` on a 24-thread PC uses 4 workers in each step, and the same as before without it. This change limits CPU use on purpose. It doesn't make anything faster. Setup: CachyOS, 24 CPU threads. A script calls the functions as `make_game.py`'s steps do. `chunks._available_gb` is set to 64 because this PC was busy and its free memory would otherwise have lowered the lift count further. 200 CPU-bound items go through `chunks._each`, with distinct worker pids counted, and the stand-in game runs through `make_game.prepare_shaders`. | | Before: main @ 1bd93a48, no count | Before: main @ 1bd93a48, `LONGSHOT_JOBS=4` | After: this PR @ 4ff025d4, no count | After: this PR @ 4ff025d4, `LONGSHOT_JOBS=4` | |---|---|---|---|---| | lift processes (`file_jobs`, distinct worker pids over 200 items) | 16 | 16 | 16 | 4 | | prompt face processes (`face_jobs(5)`) | 5 | 5 | 5 | 4 | | CPUs the shader preparation's child may use | 24 | 24 | 24 | 4 | ## Risks - `nx2/chunks.py` is one of the lift's sources, so the first update after this merges translates the game again (a few minutes, once). `LONGSHOT_JOBS` itself isn't in the lift key. - `prompt_assets.py` is part of the prompts' key, so they are made again once (seconds). - A small count now really slows the translation and the shader preparation. That is the point of the setting, but a player who set 1 or 2 for memory reasons will see slower updates than before. - On Windows, if the affinity mask can't be read or set (more than 64 CPUs, or a process already spread over several processor groups), the preparation gets every CPU, as before. - To go back to the old behaviour: leave the count unset (Automatic before #235), or revert this PR. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. (Game-free only: the game binary is unchanged, and no real build or prepare was run here.) - [x] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] Docs updated. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
fix(build): every step make_game.py runs side by side keeps to the workers it was given, not only the compile
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 12s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 25s
CI / Workflow lint (actionlint) (pull_request) Successful in 25s
Third-party parts / As their notes say (pull_request) Successful in 27s
PR description / Description follows the template (pull_request_target) Failing after 7s
CI / Website build and checks (pull_request) Failing after 38s
4ff025d491
--jobs / LONGSHOT_JOBS (the launcher's Build workers) reached nx2/build.py alone: the lift's processes and discovery
(chunks.file_jobs), the prompts' faces (prompt_assets.face_jobs) and the game's --prepare-shaders sized themselves from
the CPU count, so an update filled every thread whatever the player chose (LongshotLauncher #148, #163).

nx2/build_jobs.py holds the rule. make_game.py puts a count it was given in LONGSHOT_JOBS for its steps; file_jobs and
face_jobs take at most that many (their own variables and memory rules still apply); the shader preparation starts on
that many CPUs, which its pools follow (usable_cpus). With no count given nothing changes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
HumanBean changed title from WIP: fix(build): every step make_game.py runs side by side keeps to the workers it was given, not only the compile to fix(build): every step make_game.py runs side by side keeps to the workers it was given, not only the compile 2026-11-21 19:54:54 +00:00
HumanBean deleted branch fix/build-jobs-every-step 2026-11-21 19:55:03 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2114
No description provided.