fix(dev): a renderer-only build compiles LONGSHOT_JOBS files at once, not one #1147

Open
zeldaootr wants to merge 5 commits from zeldaootr/Longshot:dev-packs-renderer-jobs into main
Contributor

Type: fix · Risk: low · Tested on: Linux x86-64, 16 threads (a renderer-only pack timed before and after; the game-free checks) · Needs: none

Why

A change to the renderer alone, built with scripts/dev_packs.py pack or scripts/dev_build.py vk, compiled one file at a time whatever LONGSHOT_JOBS said, so a change to a header every renderer file includes took minutes longer than it needs to. Developers and agents testing a renderer change against a base wait on this every time.

The tracker was searched ("dev_packs jobs", "renderer-only pack", "LONGSHOT_JOBS", "dev_build vk"): no issue or open pull request covers it.

How to reproduce

Seen on: Linux x86-64, 16 threads, any GPU (no game is run)
Settings and switches: LONGSHOT_JOBS=14
Steps: on a base, python -I scripts/dev_packs.py pack <base> <name> --tree <checkout> with a checkout that changes a renderer header (gpu/vk/vk_internal.h), or python -I scripts/dev_build.py vk
Expected: nx2/build.py compiles 14 files at once
Actual: the command is nx2/build.py --standalone with no --jobs, and the build says 207 files, 1 jobs

What changed

The renderer-only build now passes --jobs to nx2/build.py: LONGSHOT_JOBS when it is set (0 or empty: automatic), else one compiler for every four CPU threads. nx2/build.py still starts each compiler only when the free memory holds it.

  • dev: renderer_build_cmd() in scripts/dev_packs.py builds the command, and cmd_pack runs it; a LONGSHOT_JOBS that is not a whole number is refused with the message make_game.py gives.
  • dev: cmd_vk in scripts/dev_build.py builds its renderer with the same helper, run by the build's own Python, in its light queue slot.
  • dev: environment() in scripts/dev_build.py keeps an exported LONGSHOT_JOBS. It dropped every name starting with LONGSHOT_JOB to keep the job queue's own LONGSHOT_JOB, LONGSHOT_JOB_NAME and LONGSHOT_JOB_MODE out of a build, which dropped LONGSHOT_JOBS too; it now drops LONGSHOT_JOB and the LONGSHOT_JOB_ prefix only (and an older queue's OOT_STATUS_*, as before). Found in review. This also cures the same loss on dev_build.py base, pack and build, which ran make_game.py and dev_packs.py without an exported LONGSHOT_JOBS since dev_build.py came in (#1085).
  • tests: test_renderer_build_jobs in tests/test_dev_packs.py (the count from LONGSHOT_JOBS, the default, never 0, bad values refused); test_vk_compiles_several_files in tests/test_dev_build.py (the command cmd_vk queues carries --jobs); test_vk_takes_the_exported_jobs in tests/test_dev_build.py, through the real environment() with LONGSHOT_JOBS exported (1, then 14, on 16 threads) and the queue's names set: the queued command carries that count, the build's environment keeps LONGSHOT_JOBS, and LONGSHOT_JOB, LONGSHOT_JOB_NAME, LONGSHOT_JOB_MODE and OOT_STATUS_LOCK are still dropped.
  • docs: docs/development.md, Testing a change against a base, says how many files a renderer-only pack compiles at once; dev_build.py's usage line says it for vk, and its notes say LONGSHOT_JOBS stays.

Measurements

Verdict: a renderer-only pack that compiles the whole renderer went from 402 s to 77 s with LONGSHOT_JOBS=14, the same 207 files planned and about 200 compiled each time. That is one pair of runs on a shared PC: evidence that the files now compile in parallel, not a repeatable 5.2x. The game-free checks pass, apart from two that fail on main too.

Hardware: Linux x86-64 (Ubuntu, kernel 6.8), 16 CPU threads, 62 GB RAM; other agents' work running beside it (load average 4-16)
Scene: no game; a renderer-only pack on a shared development base of main 9c9d69ed, from a checkout that changes 30 files under gpu/vk/ (renderer headers among them) and tests/
Method: one pack with each dev_packs.py, run one after the other from the same checkout and toolchain, LONGSHOT_JOBS=14; before each, the base's renderer work folder was emptied (linked again from the base's objects) and the object store pointed at an empty folder, so both compiled the same files from the same state. The time is the pack's own pack ... in N s line; the load average was read every 10 s
How to reproduce: LONGSHOT_JOBS=14 LONGSHOT_OBJ_STORE=<empty folder> python -I <dev_packs.py under test> pack <base> jobs-before --tree <checkout>, with the base's renderer-work folder moved aside first; then the same with this PR's dev_packs.py and jobs-after
Baseline: the base is an older main (9c9d69ed) because it was the shared base free to use; the dev_packs.py run is the file under test in each column, main's and this PR's, and nx2/build.py is the checkout's in both, so only the command line differs. The after column is cff4ef52: the commits since change only dev_build.py's environment() and its test, which a direct dev_packs.py pack does not go through, so the numbers still hold

Before: main @ 71b1edd8 After: this PR @ cff4ef52 (0856544b since the rebase)
the build's plan 207 files, 1 jobs 207 files, 14 jobs
files compiled (more than 0 s) 197 199
compile time 402 s 76 s
the pack, start to end 402 s 77 s
load average while it ran (median of 10 s samples; range) 7.2 (4.3-12.7) 14.2 (11.5-15.9)
before:
+ python3 -I nx2/build.py --standalone
207 files, 1 jobs, -O2 -D_GNU_SOURCE
compiled in 402s, 0 failed
pack jobs-before: longshot_vk.so; prepares, in 402 s

after:
+ python3 -I nx2/build.py --standalone --jobs 14
207 files, 14 jobs, -O2 -D_GNU_SOURCE
compiled in 76s, 0 failed
pack jobs-after: longshot_vk.so; prepares, in 77 s

One run each on a shared PC (the load average differs between the two), so the ratio is not a guarantee: how much faster a renderer-only build gets depends on the threads, the memory and what else runs. The plan line (1 jobs against 14 jobs) is the change itself. The two packs' longshot_vk.so differ in their bytes; they were not compared further (the renderer's version stamp is built in), and the code that runs in the game is not changed by this PR.

Flags and settings

No new switch: the renderer-only build now reads LONGSHOT_JOBS, which the game's build already reads, with the same rule for its values.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
LONGSHOT_JOBS How many compilers a build runs at once (development and the builder; no effect on the game) read by make_game.py's game build run directly; ignored by a renderer-only build, which compiled one file at a time; dropped by every dev_build.py command before its build ran also read by a renderer-only build (dev_packs.py pack, dev_build.py vk); unset, 0 or empty: one compiler for every four CPU threads; kept by dev_build.py, so base, pack, build and vk honour it A renderer-only build compiles that many files at once, as memory allows; a dev_build.py game build compiles the exported count Helps: 402 s to 77 s for a whole-renderer pack at 14 on 16 threads (one pair of runs on a shared PC); untested on Windows and on a PC short of memory As shipped: unset by default; set it lower on a busy or small PC

Testing

Tested:

  • The pack timing above, on Linux x86-64: both packs built and linked longshot_vk.so with 0 failures.
  • python -I -B tests/run_all.py --changed forge/main on this head (Linux, the Vulkan and SPIR-V headers found): 17 passed, 2 failed. Both, test_vk_occlusion_queries and test_walk_key, fail the same on main 31f8d8be run the same way. tests/test_dev_packs.py, tests/test_dev_build.py (16 tests) and tests/test_job_queue.py pass.
  • test_renderer_build_jobs fails at the test's commit (d18168d0, was 3513fcb7: dev_packs has no renderer_build_cmd) and passes after the fix; test_vk_compiles_several_files fails on the command cmd_vk had (no --jobs).
  • test_vk_takes_the_exported_jobs fails at its commit (62ab0afb, on 0856544b; before the rebase 9b6973cc on cff4ef52: LONGSHOT_JOBS=1 queued --jobs 4) and passes with the fix. Called directly with LONGSHOT_JOBS=1 and the queue's three names exported, environment() returned none of the four before the fix and LONGSHOT_JOBS=1 alone after it.

Not tested:

  • Windows: the helper has no platform code (os.cpu_count() and a command line), and tests/test_dev_build.py's cmd_vk checks were run on Linux only.
  • A machine short of memory: nx2/build.py starts each compiler only when the free memory holds it, as for a game build.
  • A real dev_build.py run with LONGSHOT_JOBS exported (the check goes through environment() and cmd_vk with the build itself stubbed).

Must be tested: None

Risks

What could break: a renderer-only build now runs several compilers at once, so it takes more CPU and memory beside the job queue's other jobs: at most a quarter of the CPU threads by default, LONGSHOT_JOBS when set, and nx2/build.py's memory gate still applies. A LONGSHOT_JOBS that is not a whole number now stops a renderer-only build with the message the game's build gives, where before it was ignored.

Anyone who exports LONGSHOT_JOBS and builds with dev_build.py (base, pack, build, vk) now gets that count: before, dev_build.py dropped it and make_game.py used its default (one compiler per CPU thread but one). A lower number now gives a slower, lighter build; a value that is not a whole number now stops the build with make_game.py's message.

Turning it off: LONGSHOT_JOBS=1 gives the old one-at-a-time build; or revert the commit.

Compatibility: Nothing changes for a player: development scripts only. The objects are the same (the count is in no object key), so a base's objects are reused as before.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. Onto main 31f8d8be (head 9cba4513) with no conflict.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.
**Type:** fix · **Risk:** low · **Tested on:** Linux x86-64, 16 threads (a renderer-only pack timed before and after; the game-free checks) · **Needs:** none ## Why A change to the renderer alone, built with `scripts/dev_packs.py pack` or `scripts/dev_build.py vk`, compiled one file at a time whatever `LONGSHOT_JOBS` said, so a change to a header every renderer file includes took minutes longer than it needs to. Developers and agents testing a renderer change against a base wait on this every time. The tracker was searched ("dev_packs jobs", "renderer-only pack", "LONGSHOT_JOBS", "dev_build vk"): no issue or open pull request covers it. ## How to reproduce **Seen on:** Linux x86-64, 16 threads, any GPU (no game is run) **Settings and switches:** `LONGSHOT_JOBS=14` **Steps:** on a base, `python -I scripts/dev_packs.py pack <base> <name> --tree <checkout>` with a checkout that changes a renderer header (`gpu/vk/vk_internal.h`), or `python -I scripts/dev_build.py vk` **Expected:** `nx2/build.py` compiles 14 files at once **Actual:** the command is `nx2/build.py --standalone` with no `--jobs`, and the build says `207 files, 1 jobs` ## What changed The renderer-only build now passes `--jobs` to `nx2/build.py`: `LONGSHOT_JOBS` when it is set (0 or empty: automatic), else one compiler for every four CPU threads. `nx2/build.py` still starts each compiler only when the free memory holds it. - dev: `renderer_build_cmd()` in `scripts/dev_packs.py` builds the command, and `cmd_pack` runs it; a `LONGSHOT_JOBS` that is not a whole number is refused with the message `make_game.py` gives. - dev: `cmd_vk` in `scripts/dev_build.py` builds its renderer with the same helper, run by the build's own Python, in its light queue slot. - dev: `environment()` in `scripts/dev_build.py` keeps an exported `LONGSHOT_JOBS`. It dropped every name starting with `LONGSHOT_JOB` to keep the job queue's own `LONGSHOT_JOB`, `LONGSHOT_JOB_NAME` and `LONGSHOT_JOB_MODE` out of a build, which dropped `LONGSHOT_JOBS` too; it now drops `LONGSHOT_JOB` and the `LONGSHOT_JOB_` prefix only (and an older queue's `OOT_STATUS_*`, as before). Found in review. This also cures the same loss on `dev_build.py base`, `pack` and `build`, which ran `make_game.py` and `dev_packs.py` without an exported `LONGSHOT_JOBS` since `dev_build.py` came in (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1085). - tests: `test_renderer_build_jobs` in `tests/test_dev_packs.py` (the count from `LONGSHOT_JOBS`, the default, never 0, bad values refused); `test_vk_compiles_several_files` in `tests/test_dev_build.py` (the command `cmd_vk` queues carries `--jobs`); `test_vk_takes_the_exported_jobs` in `tests/test_dev_build.py`, through the real `environment()` with `LONGSHOT_JOBS` exported (1, then 14, on 16 threads) and the queue's names set: the queued command carries that count, the build's environment keeps `LONGSHOT_JOBS`, and `LONGSHOT_JOB`, `LONGSHOT_JOB_NAME`, `LONGSHOT_JOB_MODE` and `OOT_STATUS_LOCK` are still dropped. - docs: `docs/development.md`, *Testing a change against a base*, says how many files a renderer-only pack compiles at once; `dev_build.py`'s usage line says it for `vk`, and its notes say `LONGSHOT_JOBS` stays. ## Measurements **Verdict:** a renderer-only pack that compiles the whole renderer went from 402 s to 77 s with `LONGSHOT_JOBS=14`, the same 207 files planned and about 200 compiled each time. That is one pair of runs on a shared PC: evidence that the files now compile in parallel, not a repeatable 5.2x. The game-free checks pass, apart from two that fail on `main` too. Hardware: Linux x86-64 (Ubuntu, kernel 6.8), 16 CPU threads, 62 GB RAM; other agents' work running beside it (load average 4-16) Scene: no game; a renderer-only pack on a shared development base of main 9c9d69ed, from a checkout that changes 30 files under `gpu/vk/` (renderer headers among them) and `tests/` Method: one pack with each `dev_packs.py`, run one after the other from the same checkout and toolchain, `LONGSHOT_JOBS=14`; before each, the base's renderer work folder was emptied (linked again from the base's objects) and the object store pointed at an empty folder, so both compiled the same files from the same state. The time is the pack's own `pack ... in N s` line; the load average was read every 10 s How to reproduce: `LONGSHOT_JOBS=14 LONGSHOT_OBJ_STORE=<empty folder> python -I <dev_packs.py under test> pack <base> jobs-before --tree <checkout>`, with the base's `renderer-work` folder moved aside first; then the same with this PR's `dev_packs.py` and `jobs-after` Baseline: the base is an older main (9c9d69ed) because it was the shared base free to use; the `dev_packs.py` run is the file under test in each column, main's and this PR's, and `nx2/build.py` is the checkout's in both, so only the command line differs. The after column is cff4ef52: the commits since change only `dev_build.py`'s `environment()` and its test, which a direct `dev_packs.py pack` does not go through, so the numbers still hold | | Before: main @ 71b1edd8 | After: this PR @ cff4ef52 (0856544b since the rebase) | |---|---|---| | the build's plan | 207 files, 1 jobs | 207 files, 14 jobs | | files compiled (more than 0 s) | 197 | 199 | | compile time | 402 s | 76 s | | the pack, start to end | 402 s | 77 s | | load average while it ran (median of 10 s samples; range) | 7.2 (4.3-12.7) | 14.2 (11.5-15.9) | ``` before: + python3 -I nx2/build.py --standalone 207 files, 1 jobs, -O2 -D_GNU_SOURCE compiled in 402s, 0 failed pack jobs-before: longshot_vk.so; prepares, in 402 s after: + python3 -I nx2/build.py --standalone --jobs 14 207 files, 14 jobs, -O2 -D_GNU_SOURCE compiled in 76s, 0 failed pack jobs-after: longshot_vk.so; prepares, in 77 s ``` One run each on a shared PC (the load average differs between the two), so the ratio is not a guarantee: how much faster a renderer-only build gets depends on the threads, the memory and what else runs. The plan line (`1 jobs` against `14 jobs`) is the change itself. The two packs' `longshot_vk.so` differ in their bytes; they were not compared further (the renderer's version stamp is built in), and the code that runs in the game is not changed by this PR. ### Flags and settings No new switch: the renderer-only build now reads `LONGSHOT_JOBS`, which the game's build already reads, with the same rule for its values. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `LONGSHOT_JOBS` | How many compilers a build runs at once (development and the builder; no effect on the game) | read by `make_game.py`'s game build run directly; ignored by a renderer-only build, which compiled one file at a time; dropped by every `dev_build.py` command before its build ran | also read by a renderer-only build (`dev_packs.py pack`, `dev_build.py vk`); unset, 0 or empty: one compiler for every four CPU threads; kept by `dev_build.py`, so `base`, `pack`, `build` and `vk` honour it | A renderer-only build compiles that many files at once, as memory allows; a `dev_build.py` game build compiles the exported count | Helps: 402 s to 77 s for a whole-renderer pack at 14 on 16 threads (one pair of runs on a shared PC); untested on Windows and on a PC short of memory | As shipped: unset by default; set it lower on a busy or small PC | ## Testing **Tested:** - The pack timing above, on Linux x86-64: both packs built and linked `longshot_vk.so` with 0 failures. - `python -I -B tests/run_all.py --changed forge/main` on this head (Linux, the Vulkan and SPIR-V headers found): 17 passed, 2 failed. Both, `test_vk_occlusion_queries` and `test_walk_key`, fail the same on `main` 31f8d8be run the same way. `tests/test_dev_packs.py`, `tests/test_dev_build.py` (16 tests) and `tests/test_job_queue.py` pass. - `test_renderer_build_jobs` fails at the test's commit (`d18168d0`, was 3513fcb7: `dev_packs` has no `renderer_build_cmd`) and passes after the fix; `test_vk_compiles_several_files` fails on the command `cmd_vk` had (no `--jobs`). - `test_vk_takes_the_exported_jobs` fails at its commit (62ab0afb, on 0856544b; before the rebase 9b6973cc on cff4ef52: `LONGSHOT_JOBS=1` queued `--jobs 4`) and passes with the fix. Called directly with `LONGSHOT_JOBS=1` and the queue's three names exported, `environment()` returned none of the four before the fix and `LONGSHOT_JOBS=1` alone after it. **Not tested:** - Windows: the helper has no platform code (`os.cpu_count()` and a command line), and `tests/test_dev_build.py`'s `cmd_vk` checks were run on Linux only. - A machine short of memory: `nx2/build.py` starts each compiler only when the free memory holds it, as for a game build. - A real `dev_build.py` run with `LONGSHOT_JOBS` exported (the check goes through `environment()` and `cmd_vk` with the build itself stubbed). **Must be tested:** None ## Risks **What could break:** a renderer-only build now runs several compilers at once, so it takes more CPU and memory beside the job queue's other jobs: at most a quarter of the CPU threads by default, `LONGSHOT_JOBS` when set, and `nx2/build.py`'s memory gate still applies. A `LONGSHOT_JOBS` that is not a whole number now stops a renderer-only build with the message the game's build gives, where before it was ignored. Anyone who exports `LONGSHOT_JOBS` and builds with `dev_build.py` (`base`, `pack`, `build`, `vk`) now gets that count: before, `dev_build.py` dropped it and `make_game.py` used its default (one compiler per CPU thread but one). A lower number now gives a slower, lighter build; a value that is not a whole number now stops the build with `make_game.py`'s message. **Turning it off:** `LONGSHOT_JOBS=1` gives the old one-at-a-time build; or revert the commit. **Compatibility:** Nothing changes for a player: development scripts only. The objects are the same (the count is in no object key), so a base's objects are reused as before. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. Onto main 31f8d8be (head 9cba4513) with no conflict. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated.
`dev_packs.py pack` builds a renderer-only change with `nx2/build.py --standalone` and
no `--jobs`, so it compiles one file at a time whatever `LONGSHOT_JOBS` says. This
check fails until the command carries the count.

Assisted-by: Claude
`dev_packs.py pack` ran `nx2/build.py --standalone` with no `--jobs`, and `build.py`
by itself compiles one file at a time: a change to a header every renderer file
includes compiled the 34 of them one after another. The command now carries
`LONGSHOT_JOBS`, or one compiler for every four CPU threads when that is unset
(a renderer-only build is one of the job queue's four light jobs).

Assisted-by: Claude
fix(dev): dev_build.py vk compiles LONGSHOT_JOBS files at once, not one
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 5s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
cff4ef52ec
It builds the renderer with the helper a renderer-only pack uses, run by the build's own Python, and the helper's note names the queue slot each caller holds.

Assisted-by: Claude
zeldaootr changed title from WIP: fix(dev): a renderer-only build compiles LONGSHOT_JOBS files at once, not one to fix(dev): a renderer-only build compiles LONGSHOT_JOBS files at once, not one 2026-11-16 15:12:25 +00:00
Member

Code-only review of cff4ef52ec; informational findings, with final approval left to another reviewer.

P2 — dev_build.py vk drops the exported worker limit before the new helper reads it (scripts/dev_build.py:621). environment() at line 359 excludes every name starting with LONGSHOT_JOB, which includes LONGSHOT_JOBS. Consequently, on a 16-thread PC, LONGSHOT_JOBS=1 python -I scripts/dev_build.py vk queues --jobs 4, not --jobs 1; an exported 14 likewise becomes 4. The new parallel default therefore cannot be reduced through the documented environment switch on this path. Direct dev_packs.py pack uses os.environ and does not have this problem. Preserve LONGSHOT_JOBS when removing queue bookkeeping, for example by removing only LONGSHOT_JOB and the LONGSHOT_JOB_ prefix. Add a test through the real environment assembly with an exported limit; test_vk_compiles_several_files currently replaces environment() with a mock that already contains the value, so it bypasses the loss. This is derived from source, not an observed build run.

The remaining change is confined to worker-count selection and its callers; the existing compiler memory gate and object keys remain in place. The author reports successful Linux renderer packs and relevant game-free tests. The 402 s versus 77 s result is one pair on a shared PC, useful evidence of parallel compilation but not a repeatable 5.2x guarantee. Native Windows and low-memory operation remain untested, as disclosed; no unrelated gameplay run is needed for this script-only change.

Read the full patch, environment/queue handoff, compiler scheduling and memory gate, new fixtures, and current feedback. The dev_packs.py overlaps with 1122, 737, and 28 concern cache lists, cache defaults, or documentation, not this worker-count path; no functional merge-order dependency found. No tests, builds, or PR code were run.

Code-only review of cff4ef52ec16f934bc2117a313db9a0cbe10d2ae; informational findings, with final approval left to another reviewer. **P2 — `dev_build.py vk` drops the exported worker limit before the new helper reads it** (`scripts/dev_build.py:621`). `environment()` at line 359 excludes every name starting with `LONGSHOT_JOB`, which includes `LONGSHOT_JOBS`. Consequently, on a 16-thread PC, `LONGSHOT_JOBS=1 python -I scripts/dev_build.py vk` queues `--jobs 4`, not `--jobs 1`; an exported `14` likewise becomes `4`. The new parallel default therefore cannot be reduced through the documented environment switch on this path. Direct `dev_packs.py pack` uses `os.environ` and does not have this problem. Preserve `LONGSHOT_JOBS` when removing queue bookkeeping, for example by removing only `LONGSHOT_JOB` and the `LONGSHOT_JOB_` prefix. Add a test through the real environment assembly with an exported limit; `test_vk_compiles_several_files` currently replaces `environment()` with a mock that already contains the value, so it bypasses the loss. This is derived from source, not an observed build run. The remaining change is confined to worker-count selection and its callers; the existing compiler memory gate and object keys remain in place. The author reports successful Linux renderer packs and relevant game-free tests. The 402 s versus 77 s result is one pair on a shared PC, useful evidence of parallel compilation but not a repeatable 5.2x guarantee. Native Windows and low-memory operation remain untested, as disclosed; no unrelated gameplay run is needed for this script-only change. Read the full patch, environment/queue handoff, compiler scheduling and memory gate, new fixtures, and current feedback. The `dev_packs.py` overlaps with [1122](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1122), [737](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/737), and [28](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/28) concern cache lists, cache defaults, or documentation, not this worker-count path; no functional merge-order dependency found. No tests, builds, or PR code were run.
zeldaootr force-pushed dev-packs-renderer-jobs from cff4ef52ec
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 5s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 9cba451380
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 8s
2026-11-16 21:23:37 +00:00
Compare
Author
Contributor

Two commits after bibiburl's review: dev_build.py's environment() dropped every name starting with LONGSHOT_JOB, LONGSHOT_JOBS among them, so dev_build.py vk queued --jobs 4 on 16 threads whatever was exported. It now drops only the queue's LONGSHOT_JOB and LONGSHOT_JOB_*. test_vk_takes_the_exported_jobs goes through the real environment() (fails before the fix, passes after). The same fix lets dev_build.py base, pack and build pass an exported LONGSHOT_JOBS on to make_game.py, which they had dropped since #1085. Description: What changed, Measurements (one pair of runs: evidence of parallel compilation, not a fixed ratio), Testing, Risks, checklist. Also rebased onto main 31f8d8be (head 9cba4513), with no conflict.

Two commits after `bibiburl`'s review: `dev_build.py`'s `environment()` dropped every name starting with `LONGSHOT_JOB`, `LONGSHOT_JOBS` among them, so `dev_build.py vk` queued `--jobs 4` on 16 threads whatever was exported. It now drops only the queue's `LONGSHOT_JOB` and `LONGSHOT_JOB_*`. `test_vk_takes_the_exported_jobs` goes through the real `environment()` (fails before the fix, passes after). The same fix lets `dev_build.py base`, `pack` and `build` pass an exported `LONGSHOT_JOBS` on to `make_game.py`, which they had dropped since https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1085. Description: What changed, Measurements (one pair of runs: evidence of parallel compilation, not a fixed ratio), Testing, Risks, checklist. Also rebased onto main 31f8d8be (head 9cba4513), with no conflict.
Author
Contributor

Thanks, bibiburl: confirmed. environment() in scripts/dev_build.py dropped every name starting with LONGSHOT_JOB to keep the job queue's own variables out of a build (LONGSHOT_JOB, LONGSHOT_JOB_NAME and LONGSHOT_JOB_MODE, the only ones job_queue.py's job_env sets). That dropped LONGSHOT_JOBS too. Called directly with LONGSHOT_JOBS=1 exported, it returned no LONGSHOT_JOBS, and cmd_vk queued --jobs 4 on 16 threads, as you said.

It was older than this PR: since #1085, dev_build.py base, pack and build also ran make_game.py and dev_packs.py without an exported LONGSHOT_JOBS, so make_game.py fell back to its own default.

Fixed as you suggested: it now drops LONGSHOT_JOB and the LONGSHOT_JOB_ prefix only (and an older queue's OOT_STATUS_*, as before). The new test_vk_takes_the_exported_jobs in tests/test_dev_build.py goes through the real environment() with only the tool lookup stubbed. It exports LONGSHOT_JOBS (1, then 14, on 16 threads) together with the queue's names, then checks three things: the command cmd_vk queues carries that count, the build's environment keeps LONGSHOT_JOBS, and LONGSHOT_JOB, LONGSHOT_JOB_NAME, LONGSHOT_JOB_MODE and OOT_STATUS_LOCK are still dropped. It fails on the previous head (--jobs 4 where 1 was asked) and passes with the fix.

I took your point on the measurement too: the description now calls the 402 s to 77 s pair evidence of parallel compilation on a shared PC, not a repeatable 5.2x.

Thanks, `bibiburl`: confirmed. `environment()` in `scripts/dev_build.py` dropped every name starting with `LONGSHOT_JOB` to keep the job queue's own variables out of a build (`LONGSHOT_JOB`, `LONGSHOT_JOB_NAME` and `LONGSHOT_JOB_MODE`, the only ones `job_queue.py`'s `job_env` sets). That dropped `LONGSHOT_JOBS` too. Called directly with `LONGSHOT_JOBS=1` exported, it returned no `LONGSHOT_JOBS`, and `cmd_vk` queued `--jobs 4` on 16 threads, as you said. It was older than this PR: since https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1085, `dev_build.py base`, `pack` and `build` also ran `make_game.py` and `dev_packs.py` without an exported `LONGSHOT_JOBS`, so `make_game.py` fell back to its own default. Fixed as you suggested: it now drops `LONGSHOT_JOB` and the `LONGSHOT_JOB_` prefix only (and an older queue's `OOT_STATUS_*`, as before). The new `test_vk_takes_the_exported_jobs` in `tests/test_dev_build.py` goes through the real `environment()` with only the tool lookup stubbed. It exports `LONGSHOT_JOBS` (1, then 14, on 16 threads) together with the queue's names, then checks three things: the command `cmd_vk` queues carries that count, the build's environment keeps `LONGSHOT_JOBS`, and `LONGSHOT_JOB`, `LONGSHOT_JOB_NAME`, `LONGSHOT_JOB_MODE` and `OOT_STATUS_LOCK` are still dropped. It fails on the previous head (`--jobs 4` where 1 was asked) and passes with the fix. I took your point on the measurement too: the description now calls the 402 s to 77 s pair evidence of parallel compilation on a shared PC, not a repeatable 5.2x.
TrashPanda approved these changes 2026-11-17 08:07:20 +00:00
TrashPanda left a comment

A renderer-only build now compiles several files at once. I read head 9cba4513. It applies onto current main. This can merge as it is.

renderer_build_cmd passes --jobs to nx2/build.py. LONGSHOT_JOBS is that count when it is a whole number. Unset, empty or 0 is one compiler for every four CPU threads, and never zero. A value that is not a whole number stops the build with the message make_game.py gives. dev_packs.py pack and dev_build.py vk both use it. nx2/build.py still starts each compiler only when the free memory holds it.

environment() used to drop every name starting with LONGSHOT_JOB, so dev_build.py threw away LONGSHOT_JOBS before the build ran. It now drops LONGSHOT_JOB and the LONGSHOT_JOB_ prefix, and an older queue's OOT_STATUS_ names. LONGSHOT_JOBS stays, so base, pack, build and vk pass an exported count on. The check goes through that real filter: an exported 1 or 14 is the --jobs on the command, and the queue's own names are still absent.

Must be tested is None. The objects do not depend on the count. One pair of packs on Linux, LONGSHOT_JOBS=14, planned 207 files at 14 jobs and finished in 77 s where the one-at-a-time pack took 402 s.

A renderer-only build now compiles several files at once. I read head `9cba4513`. It applies onto current main. This can merge as it is. `renderer_build_cmd` passes `--jobs` to `nx2/build.py`. `LONGSHOT_JOBS` is that count when it is a whole number. Unset, empty or 0 is one compiler for every four CPU threads, and never zero. A value that is not a whole number stops the build with the message `make_game.py` gives. `dev_packs.py pack` and `dev_build.py vk` both use it. `nx2/build.py` still starts each compiler only when the free memory holds it. `environment()` used to drop every name starting with `LONGSHOT_JOB`, so `dev_build.py` threw away `LONGSHOT_JOBS` before the build ran. It now drops `LONGSHOT_JOB` and the `LONGSHOT_JOB_` prefix, and an older queue's `OOT_STATUS_` names. `LONGSHOT_JOBS` stays, so `base`, `pack`, `build` and `vk` pass an exported count on. The check goes through that real filter: an exported 1 or 14 is the `--jobs` on the command, and the queue's own names are still absent. Must be tested is None. The objects do not depend on the count. One pair of packs on Linux, `LONGSHOT_JOBS=14`, planned 207 files at 14 jobs and finished in 77 s where the one-at-a-time pack took 402 s.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 8s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'dev-packs-renderer-jobs:zeldaootr-dev-packs-renderer-jobs'
git switch 'zeldaootr-dev-packs-renderer-jobs'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1147
No description provided.