MagicBeanDev/Longshot

Watch

Fork 71

perf(lift): a Windows lift runs one worker per 0.75 GB free, and makes the runtime tables beside the edits #357

Merged

HumanBean merged 2 commits from madebyjordan/Longshot:perf/lift-workers into main

2026-11-14 05:40:55 +00:00

madebyjordan commented

2026-11-14 04:23:00 +00:00

Contributor

Why

On Windows, the lift (the build's Translate step) used one or two processes on a typical 16 GB PC, so a first build sat in Translate for about 4 minutes. It now uses the PC's cores where memory allows, and the lifted C it writes is byte for byte the same.

What changed

lift: file_jobs() in nx2/chunks.py allows one process per 0.75 GB of available memory where the processes are spawned (Windows). Where they are forked it stays at one per 2 GB. A spawned worker holds only the emit's tables and its run's function bodies: at most 395 MB, measured over a whole lift with 4 workers on a Ryzen 5 5600 with 16 GB. The 2 GB rule is a forked worker's, which touches the lift's shared pages. With it, a 16 GB Windows PC with 4-8 GB free got one or two processes; that build's lift took 254 s.
lift: start_side() / end_side() in nx2/chunks.py. Where there is no fork, the runtime tables and standalone files (~11 s of Python) are made in a thread while the C is edited. This process only waits on the edit workers meanwhile; before, they were made after the edits. A side run that raises is done again in the lift's own thread, as a failed forked process already is.
docs: docs/development.md's LONGSHOT_LIFT_JOBS row, and file_jobs' docstring, say 0.75 GB on Windows.
tests: tests/test_lift_workers.py checks:
- the count, with forked and spawned workers, the core and 16 limits, unknown memory, and LONGSHOT_LIFT_JOBS first;
- the side run in a thread: its result, a failed run done again, and none for one process.
Against main's chunks.py, the spawned count and both thread cases fail.

Linux and macOS are unchanged: they fork, so file_jobs keeps 2 GB a process and the side work keeps its forked process.

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
`LONGSHOT_LIFT_JOBS=<n>` (existing)	The lift's processes	the cores, at most 16; one per 2 GB available (forked), now one per 0.75 GB (spawned: Windows)	Keep the default	Windows PCs with 16 GB or less	Linux, macOS (unchanged)	Windows PCs with 8 GB

Measurements

main: cf605c85 (2026-11-14)
this PR: perf/lift-workers @ d0564f17
Hardware: Ryzen 5 5600 (6 cores, 12 threads), 16 GB DDR4 at 2133 MT/s, NVMe, Windows 11
Scene: none (the build). nightly-34's exefs, a standalone lift as make_game.py runs it: nx2/lift.py --standalone --write, with the builder's tools (Python 3.13, llvm-mingw clang 23).
Method: main and this PR alternated, main / PR / main / PR, each from an empty work folder, with each checkout's a64emit.dll built beforehand. Wall time is the lift process's. The 90 files the lift writes (sa/gen) were compared by SHA-256 between the two.
How to reproduce: OOT_WORK=<empty folder> OOT_EXEFS=<exefs> RX_RELEASE=1 A64_HOT_CHUNK=1 python -I nx2/lift.py --standalone --write --exefs <exefs> --work <folder> on each checkout, alternated.

	main @ `cf605c85`	this PR @ `d0564f17`
lift, run 1	219.1 s	147.0 s
lift, run 2	274.3 s	154.2 s
median	246.7 s	150.6 s (-39%)
emit processes	2, 1	5, 3
files differing from main		none of 85 (SHA-256)

An earlier run on the same PC at dd02c6cc (2026-11-13) measured main at 232.3 / 223.3 / 171.3 s (1, 1 and 2 emit processes) and this change at 150.7 / 142.8 / 125.6 s (3, 3 and 5), with the outputs byte-identical. How many processes it gets depends on the memory free at the start.

Testing

Tested:

The lift on Windows, as above, on both checkouts, with the output compared.
tests/test_lift_workers.py.
python -I -B tests/run_all.py --jobs 4 on main (483 passed, 6 failed, 70 skipped) and on this PR merged with the other four build PRs of this batch (486 passed, 6 failed (the same 6 as main), 70 skipped).

Not tested:

In the game: the lifted C is byte-identical to main's, so the game built from it is the same.
A Windows PC with 8 GB. The rule gives it one process per 0.75 GB available, at least one.
Linux and macOS: their code path is unchanged.

Needs other testers:

Windows with 8 GB RAM: a first build's Translate step finishes, with no out-of-memory stop, expecting it no slower than main (T-HSYQ).

Screenshots

Nothing on screen.

Risk and rollback

Memory: more lift processes on Windows use more of it at once. The rule leaves each worker about 1.9 times its measured peak, and counts only the memory available when the lift starts. If a lift runs out of memory, LONGSHOT_LIFT_JOBS=1 (or any smaller number) sets the count.
Rollback: reverting the commit gives main's behaviour.

Reviewer notes

The side thread shares the process with the edits' waiting loop. tables() reads the image and writes the runtime tables, but no file the edits touch: the same split the forked process already had on Linux.

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags noted, and the switch reference updated (docs/development.md, the README).
Docs updated.
A check under tests/ that reads changed source updated in this PR, not weakened.

## Why On Windows, the lift (the build's Translate step) used one or two processes on a typical 16 GB PC, so a first build sat in Translate for about 4 minutes. It now uses the PC's cores where memory allows, and the lifted C it writes is byte for byte the same. ## What changed - lift: `file_jobs()` in `nx2/chunks.py` allows one process per 0.75 GB of available memory where the processes are spawned (Windows). Where they are forked it stays at one per 2 GB. A spawned worker holds only the emit's tables and its run's function bodies: at most 395 MB, measured over a whole lift with 4 workers on a Ryzen 5 5600 with 16 GB. The 2 GB rule is a forked worker's, which touches the lift's shared pages. With it, a 16 GB Windows PC with 4-8 GB free got one or two processes; that build's lift took 254 s. - lift: `start_side()` / `end_side()` in `nx2/chunks.py`. Where there is no fork, the runtime tables and standalone files (~11 s of Python) are made in a thread while the C is edited. This process only waits on the edit workers meanwhile; before, they were made after the edits. A side run that raises is done again in the lift's own thread, as a failed forked process already is. - docs: `docs/development.md`'s `LONGSHOT_LIFT_JOBS` row, and `file_jobs`' docstring, say 0.75 GB on Windows. - tests: `tests/test_lift_workers.py` checks: - the count, with forked and spawned workers, the core and 16 limits, unknown memory, and `LONGSHOT_LIFT_JOBS` first; - the side run in a thread: its result, a failed run done again, and none for one process. Against main's `chunks.py`, the spawned count and both thread cases fail. Linux and macOS are unchanged: they fork, so `file_jobs` keeps 2 GB a process and the side work keeps its forked process. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `LONGSHOT_LIFT_JOBS=<n>` (existing) | The lift's processes | the cores, at most 16; one per 2 GB available (forked), now one per 0.75 GB (spawned: Windows) | Keep the default | Windows PCs with 16 GB or less | Linux, macOS (unchanged) | Windows PCs with 8 GB | ## Measurements main: cf605c85 (2026-11-14) this PR: perf/lift-workers @ d0564f17 Hardware: Ryzen 5 5600 (6 cores, 12 threads), 16 GB DDR4 at 2133 MT/s, NVMe, Windows 11 Scene: none (the build). nightly-34's exefs, a standalone lift as `make_game.py` runs it: `nx2/lift.py --standalone --write`, with the builder's tools (Python 3.13, llvm-mingw clang 23). Method: main and this PR alternated, main / PR / main / PR, each from an empty work folder, with each checkout's `a64emit.dll` built beforehand. Wall time is the lift process's. The 90 files the lift writes (`sa/gen`) were compared by SHA-256 between the two. How to reproduce: `OOT_WORK=<empty folder> OOT_EXEFS=<exefs> RX_RELEASE=1 A64_HOT_CHUNK=1 python -I nx2/lift.py --standalone --write --exefs <exefs> --work <folder>` on each checkout, alternated. | | main @ cf605c85 | this PR @ d0564f17 | |---|---|---| | lift, run 1 | 219.1 s | 147.0 s | | lift, run 2 | 274.3 s | 154.2 s | | median | **246.7 s** | **150.6 s** (-39%) | | emit processes | 2, 1 | 5, 3 | | files differing from main | | none of 85 (SHA-256) | An earlier run on the same PC at dd02c6cc (2026-11-13) measured main at 232.3 / 223.3 / 171.3 s (1, 1 and 2 emit processes) and this change at 150.7 / 142.8 / 125.6 s (3, 3 and 5), with the outputs byte-identical. How many processes it gets depends on the memory free at the start. ## Testing **Tested:** - The lift on Windows, as above, on both checkouts, with the output compared. - `tests/test_lift_workers.py`. - `python -I -B tests/run_all.py --jobs 4` on main (483 passed, 6 failed, 70 skipped) and on this PR merged with the other four build PRs of this batch (486 passed, 6 failed (the same 6 as main), 70 skipped). **Not tested:** - In the game: the lifted C is byte-identical to main's, so the game built from it is the same. - A Windows PC with 8 GB. The rule gives it one process per 0.75 GB available, at least one. - Linux and macOS: their code path is unchanged. **Needs other testers:** - Windows with 8 GB RAM: a first build's Translate step finishes, with no out-of-memory stop, expecting it no slower than main (T-HSYQ). ## Screenshots Nothing on screen. ## Risk and rollback - **Memory:** more lift processes on Windows use more of it at once. The rule leaves each worker about 1.9 times its measured peak, and counts only the memory available when the lift starts. If a lift runs out of memory, `LONGSHOT_LIFT_JOBS=1` (or any smaller number) sets the count. - **Rollback:** reverting the commit gives main's behaviour. ## Reviewer notes The side thread shares the process with the edits' waiting loop. `tables()` reads the image and writes the runtime tables, but no file the edits touch: the same split the forked process already had on Linux. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [x] Docs updated. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.

madebyjordan added 2 commits

2026-11-14 04:23:00 +00:00

perf(lift): a Windows lift runs one worker per 0.75 GB free, and makes the runtime tables beside the edits 2fb11187a8

A spawned worker (no fork on Windows) peaks at 395 MB over a whole lift (Ryzen 5 5600, 16 GB, 4 workers); the 2 GB a
worker rule is a forked one's. With it a 16 GB Windows PC with 4-8 GB free ran one or two workers (that build's lift:
254 s). Where there is no fork the side work (the runtime tables, ~11 s) runs in a thread while this process waits on the
workers. Linux and macOS are unchanged.

Measured (standalone lift of nightly-34's exefs, Ryzen 5 5600, 16 GB, Windows 11, main and this branch alternated):
main 232.3 / 223.3 / 171.3 s (1, 1, 2 emit processes), this branch 150.7 / 142.8 / 125.6 s (3, 3, 5). The 90 files the
lift writes (81 lifted C files) are byte-identical between the two.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

test(lift): the worker count and the side run without fork; the docs say 0.75 GB on Windows

PR description / Description follows the template (pull_request_target) Successful in 5s

Details

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

d0564f1736

tests/test_lift_workers.py: one process per 2 GB available where forked, per 0.75 GB where spawned, never more than
the cores or 16, LONGSHOT_LIFT_JOBS first; without fork the side work runs in a thread, a failed one is done again in
the lift's thread, and none runs for one process. Against main's chunks.py the spawned count and both thread cases
fail. file_jobs' docstring and docs/development.md's LONGSHOT_LIFT_JOBS row said one per 2 GB on Windows too.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

madebyjordan changed title from ~~WIP: perf(lift): a Windows lift runs one worker per 0.75 GB free, and makes the runtime tables beside the edits~~ to perf(lift): a Windows lift runs one worker per 0.75 GB free, and makes the runtime tables beside the edits

2026-11-14 04:24:49 +00:00

madebyjordan referenced this pull request

2026-11-14 04:25:32 +00:00

perf(build): a first build in under 10 minutes: what was measured, and a proposal (a clang-free cold tier) #362

mooki added the

labels

2026-11-14 04:31:46 +00:00

mooki reviewed

2026-11-14 04:39:51 +00:00

mooki left a comment

Read head d0564f17 against main cf605c85: the complete diff and description, the empty comments and reviews, and the emit, edit and runtime-table callers; ran the new worker test and seven existing lift checks, including the spawned emitter with GCC and Clang. All pass here; the new test against main gives the expected two failures and one error. No game or full build was run.

What's good

The cost is real in main: spawned workers use the same 2 GB allowance as forked workers, and Windows makes the runtime files only after the edits finish. The change stays in those two decisions.
The new thread result reaches the existing caller correctly. nx2/lift.py, lines 200-214, starts the side work, edits code_*.c, then waits for its result before reporting completion. The side work writes different files. end_side joins the thread before reading its result and retries a failed callback only after that join.
The explicit worker override, minimum of one worker, CPU and 16-worker limits, and normal forked path remain intact. The new test fails against main for the changed behavior, not a missing import or symbol.

Must change

No functional defect found in this patch. It merges cleanly with main cf605c85.

Measurements and smaller points

Keep the gain tied to this Windows lift and show the spread. The named commits, same PC and empty-work-folder method, and main / PR / main / PR order satisfy the basic comparison. The displayed medians are correct: 246.7 s to 150.6 s is 39.0% less lift time. Main ranges from 219.1 to 274.3 s, versus 147.0 to 154.2 s here. Those ranges do not overlap, but their spreads are 22.4% and 4.8% of their medians, and the worker counts change between repetitions. The table supports the saving for these runs, not a fixed gain on every Windows PC. Add the per-run available-memory and stage-time lines, and say how warm-up was handled. It also does not separate the thread's contribution from the increased worker count.
Make the output comparison and test record unambiguous. Measurements says all 90 generated files were compared, while the table says none of 85 differ. Reconcile those counts and explicitly include the runtime and standalone files now generated by the thread. Testing reports a suite run on a five-PR combination, not this head alone, and says the six inherited failures are named above, but does not name them. Give the combined commit and the six test names and diagnostics, keeping that result separate from this head's focused checks. The source-check jobs currently say “Blocked by required conditions”, not passed: https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/4277.
Pin the overlap and wait behavior in the thread test. tests/test_lift_workers.py, lines 68-77, checks the callback's thread and result, but its immediate callback does not establish that start_side returns while work is pending or that end_side waits. A callback held by a threading.Event would make those two guarantees explicit. This is a coverage improvement, not an observed failure.

Not checked

No Windows or macOS execution, low-memory Windows build, whole suite, or reproduction of the author's timing and peak-memory measurements here. The passing Linux spawn fixture compares generated output, but is not Windows memory-pressure evidence. The outstanding 8 GB test should retain available memory, peak usage across emit and edit stages, and whether fallback ran. No old tested-configurations table rows are changed by this PR.

Read head `d0564f17` against main `cf605c85`: the complete diff and description, the empty comments and reviews, and the emit, edit and runtime-table callers; ran the new worker test and seven existing lift checks, including the spawned emitter with GCC and Clang. All pass here; the new test against main gives the expected two failures and one error. No game or full build was run. ### What's good - The cost is real in main: spawned workers use the same 2 GB allowance as forked workers, and Windows makes the runtime files only after the edits finish. The change stays in those two decisions. - The new thread result reaches the existing caller correctly. `nx2/lift.py`, lines 200-214, starts the side work, edits `code_*.c`, then waits for its result before reporting completion. The side work writes different files. `end_side` joins the thread before reading its result and retries a failed callback only after that join. - The explicit worker override, minimum of one worker, CPU and 16-worker limits, and normal forked path remain intact. The new test fails against main for the changed behavior, not a missing import or symbol. ### Must change No functional defect found in this patch. It merges cleanly with main `cf605c85`. ### Measurements and smaller points 1. **Keep the gain tied to this Windows lift and show the spread.** The named commits, same PC and empty-work-folder method, and main / PR / main / PR order satisfy the basic comparison. The displayed medians are correct: 246.7 s to 150.6 s is 39.0% less lift time. Main ranges from 219.1 to 274.3 s, versus 147.0 to 154.2 s here. Those ranges do not overlap, but their spreads are 22.4% and 4.8% of their medians, and the worker counts change between repetitions. The table supports the saving for these runs, not a fixed gain on every Windows PC. Add the per-run available-memory and stage-time lines, and say how warm-up was handled. It also does not separate the thread's contribution from the increased worker count. 2. **Make the output comparison and test record unambiguous.** Measurements says all 90 generated files were compared, while the table says none of 85 differ. Reconcile those counts and explicitly include the runtime and standalone files now generated by the thread. Testing reports a suite run on a five-PR combination, not this head alone, and says the six inherited failures are named above, but does not name them. Give the combined commit and the six test names and diagnostics, keeping that result separate from this head's focused checks. The source-check jobs currently say “Blocked by required conditions”, not passed: https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/4277. 3. **Pin the overlap and wait behavior in the thread test.** `tests/test_lift_workers.py`, lines 68-77, checks the callback's thread and result, but its immediate callback does not establish that `start_side` returns while work is pending or that `end_side` waits. A callback held by a `threading.Event` would make those two guarantees explicit. This is a coverage improvement, not an observed failure. ### Not checked No Windows or macOS execution, low-memory Windows build, whole suite, or reproduction of the author's timing and peak-memory measurements here. The passing Linux spawn fixture compares generated output, but is not Windows memory-pressure evidence. The outstanding 8 GB test should retain available memory, peak usage across emit and edit stages, and whether fallback ran. No old tested-configurations table rows are changed by this PR.