fix(builder): a compile that raises stops build.py at once, not after every queued compile #540

Open
zeldaootr wants to merge 3 commits from zeldaootr/Longshot:build-compile-stops-on-exception into main
Contributor

Type: fix · Risk: low · Tested on: Linux, no GPU (game-free check) · Needs: none

Why

When one compile in a build fails with an error the Builder did not plan for (the compiler gone, an object or its stamp that cannot be written, a full disk), or when a source build is stopped with Ctrl-C, build.py first runs every compile still waiting, with nothing printed, and only then stops. The player sees a frozen progress bar and gets the first file's error up to an hour later.

How to reproduce

Seen on: Linux (Ubuntu 24.04), no GPU; any OS, it is the build script
Settings and switches: none
Steps: on main aa95d35f with this PR's test added (1fe16e7d), run python -I -B tests/test_compile_stops_on_exception.py: it runs the real compile loop of nx2/build.py with stand-in compilers whose first job raises FileNotFoundError (1 worker, 5 jobs) or PermissionError (2 workers)
Expected: the build stops once the running compiles finish; at most one queued job per worker starts after the failure
Actual: all 4 queued compiles start and finish before the exception leaves main()

What changed

A compile that raises now stops build.py at once: the compiles not yet started are cancelled instead of run, and the same exception leaves main() as before.

  • build: the compile loop in main() of nx2/build.py calls future.result() inside with ThreadPoolExecutor. When a call raises, leaving the with waits for the pool, which starts and finishes every queued job first. The loop is now in a try: on any exception the queued jobs are cancelled (ex.shutdown(wait=False, cancel_futures=True)) and the exception goes on as before. The loop body is only indented one level; git diff -w shows the six added lines.
  • build: nx2/build.py ends with one newline (CI's lint runs the end-of-file hook on the files a PR changes).
  • tests: tests/test_compile_stops_on_exception.py runs the real compile_one and the real loop, cut out of the source, with stand-in compilers: 1 worker and 5 jobs where the first raises FileNotFoundError, and 2 workers where one raises PermissionError. It allows one queued job per worker to start, since a worker can take its next job before the main thread hears of the failure.

Measurements

Verdict: fixed: on main 4 queued compiles still start after the failure in both cases; with this PR at most one per worker, in two interleaved rounds.

Hardware: Linux x86-64 PC (Ubuntu 24.04), no GPU used; Python 3.15
Scene: no game: the game-free check
Method: python3 -I -B tests/test_compile_stops_on_exception.py at the test-only commit 1fe16e7d (main aa95d35f plus the test) and at 335e9461, twice each, alternating
How to reproduce: check out each commit and run the command above

Before: main @ aa95d35f (with the test, 1fe16e7d) After: this PR @ 335e9461
queued compiles started after the first raised, 1 worker and 5 jobs (both rounds the same) 4 at most 1
queued compiles started after one raised, 2 workers (both rounds the same) 4 at most 2
test result FAILED (failures=2) OK
1fe16e7d r1: AssertionError: 4 not less than or equal to 1 : the queued compiles ran after the first raised
1fe16e7d r1: AssertionError: 4 not less than or equal to 2 : the queued compiles ran after one raised
1fe16e7d r1: FAILED (failures=2)
335e9461 r1: OK
(round 2 the same on both commits)

Flags and settings

None

Testing

Tested: Linux, game-free. The new test fails on the test-only commit 1fe16e7d and passes at 335e9461 (this branch rebased onto main aa95d35f, which changed nx2/build.py in five merges since the old base 1be60eae, none in the lines this PR changes), twice each, alternating, and under tests/run_all.py with g++ 13.3 and with clang 23.1.5 as CXX at both commits. At the tip, the 41 checks that read nx2/build.py or the new test: 32 passed, 3 skipped, 6 failed, each of the 6 also failing or skipped by OS on main aa95d35f (test_appimage_player_build, test_cpm_discovery, test_lift_cache, test_pipeline_tour, test_vk_pgo, and test_dll, a Windows-only check run here by name). tests/test_compile_retry_jobs.py, which cuts the same part of main() out of the source, passes. Before the rebase the same at ba05f7b7 and 1c84a150 on main 1be60eae.

Not tested: A real build stopped this way (it needs a game dump): the Builder's own handling of the exit is unchanged, since the same exception leaves main() as before. Windows, where an antivirus taking the compiler or a locked object is the likelier cause: the change is plain concurrent.futures, the same on both.

Must be tested: None

Risks

What could break: compiles already running when one raises still finish (a thread cannot be stopped), so the stop takes as long as the slowest of them; only the ones not started are dropped. A compile that fails in the ordinary way (the compiler's own error) does not raise and is handled as before, by the retry that follows the loop.

Turning it off: revert the commit

Compatibility: Nothing changes: the same objects and the same error messages; only the queued compiles after an unexpected error are not run.

Issues

None filed; found by reading nx2/make_game.py and nx2/build.py whole. Not touched here, for a decision: a full disk in the finish step still gets the "an error it did not expect" message, because tests/test_builder_drive_error.py asserts that ENOSPC is not a drive error.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. Onto main 94abdd57 (head 70e2343a) with no conflict, with the commit subjects made Conventional Commits, which CI's lint job now checks for every commit, on a base that has CI's config (#417); each commit's change is the same as before by git patch-id, plus one new commit, chore(build), that drops the blank last line of nx2/build.py: CI's lint runs the end-of-file hook over the files a PR changes, and this PR changes that one. The commits named in this description are from before this rebase: 1fe16e7d is now f8011e0f, 335e9461 is now a32abce2, 70e2343a is new. Re-run on 94abdd57: test_compile_stops_on_exception fails at the test-only commit f8011e0f and passes at the head, with g++ 13.3 and clang 23.1.5; of the 42 checks that are this PR's or read its files (g++ 13.3), 34 pass, 3 skip (a tool this PC lacks) and the 5 that fail fail on main 94abdd57 too (test_dll, test_appimage_player_build, test_lift_cache, test_pipeline_tour, test_vk_pgo).
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). The full suite on main aa95d35f (Linux, g++ 13.3, the Vulkan headers set): 528 passed, 15 failed, 50 skipped; on this branch the new test and the 41 checks that read nx2/build.py ran, with no failure that main does not have (see Testing). The 15 that fail on main: test_appimage_player_build, test_bake_portability, test_cpm_discovery, test_crash_dialog, test_env_on, test_lift_cache, test_lift_cleanup_error, test_make_game_cache, test_mods_size_table, test_options_budget_once, test_pipeline_tour, test_readback_late, test_vk_pgo, test_walk_key and nx2/test_texture_init_barrier.py.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • A game-free test added for the fix, failing on main and passing here.
**Type:** fix · **Risk:** low · **Tested on:** Linux, no GPU (game-free check) · **Needs:** none ## Why When one compile in a build fails with an error the Builder did not plan for (the compiler gone, an object or its stamp that cannot be written, a full disk), or when a source build is stopped with Ctrl-C, `build.py` first runs every compile still waiting, with nothing printed, and only then stops. The player sees a frozen progress bar and gets the first file's error up to an hour later. ## How to reproduce **Seen on:** Linux (Ubuntu 24.04), no GPU; any OS, it is the build script **Settings and switches:** none **Steps:** on `main` aa95d35f with this PR's test added (1fe16e7d), run `python -I -B tests/test_compile_stops_on_exception.py`: it runs the real compile loop of `nx2/build.py` with stand-in compilers whose first job raises `FileNotFoundError` (1 worker, 5 jobs) or `PermissionError` (2 workers) **Expected:** the build stops once the running compiles finish; at most one queued job per worker starts after the failure **Actual:** all 4 queued compiles start and finish before the exception leaves `main()` ## What changed A compile that raises now stops `build.py` at once: the compiles not yet started are cancelled instead of run, and the same exception leaves `main()` as before. - build: the compile loop in `main()` of `nx2/build.py` calls `future.result()` inside `with ThreadPoolExecutor`. When a call raises, leaving the `with` waits for the pool, which starts and finishes every queued job first. The loop is now in a `try`: on any exception the queued jobs are cancelled (`ex.shutdown(wait=False, cancel_futures=True)`) and the exception goes on as before. The loop body is only indented one level; `git diff -w` shows the six added lines. - build: `nx2/build.py` ends with one newline (CI's lint runs the end-of-file hook on the files a PR changes). - tests: `tests/test_compile_stops_on_exception.py` runs the real `compile_one` and the real loop, cut out of the source, with stand-in compilers: 1 worker and 5 jobs where the first raises `FileNotFoundError`, and 2 workers where one raises `PermissionError`. It allows one queued job per worker to start, since a worker can take its next job before the main thread hears of the failure. ## Measurements **Verdict:** fixed: on `main` 4 queued compiles still start after the failure in both cases; with this PR at most one per worker, in two interleaved rounds. Hardware: Linux x86-64 PC (Ubuntu 24.04), no GPU used; Python 3.15 Scene: no game: the game-free check Method: `python3 -I -B tests/test_compile_stops_on_exception.py` at the test-only commit 1fe16e7d (main aa95d35f plus the test) and at 335e9461, twice each, alternating How to reproduce: check out each commit and run the command above | | Before: main @ aa95d35f (with the test, 1fe16e7d) | After: this PR @ 335e9461 | |---|---|---| | queued compiles started after the first raised, 1 worker and 5 jobs (both rounds the same) | 4 | at most 1 | | queued compiles started after one raised, 2 workers (both rounds the same) | 4 | at most 2 | | test result | FAILED (failures=2) | OK | ``` 1fe16e7d r1: AssertionError: 4 not less than or equal to 1 : the queued compiles ran after the first raised 1fe16e7d r1: AssertionError: 4 not less than or equal to 2 : the queued compiles ran after one raised 1fe16e7d r1: FAILED (failures=2) 335e9461 r1: OK (round 2 the same on both commits) ``` ### Flags and settings None ## Testing **Tested:** Linux, game-free. The new test fails on the test-only commit 1fe16e7d and passes at 335e9461 (this branch rebased onto `main` aa95d35f, which changed `nx2/build.py` in five merges since the old base 1be60eae, none in the lines this PR changes), twice each, alternating, and under `tests/run_all.py` with g++ 13.3 and with clang 23.1.5 as `CXX` at both commits. At the tip, the 41 checks that read `nx2/build.py` or the new test: 32 passed, 3 skipped, 6 failed, each of the 6 also failing or skipped by OS on `main` aa95d35f (`test_appimage_player_build`, `test_cpm_discovery`, `test_lift_cache`, `test_pipeline_tour`, `test_vk_pgo`, and `test_dll`, a Windows-only check run here by name). `tests/test_compile_retry_jobs.py`, which cuts the same part of `main()` out of the source, passes. Before the rebase the same at ba05f7b7 and 1c84a150 on main 1be60eae. **Not tested:** A real build stopped this way (it needs a game dump): the Builder's own handling of the exit is unchanged, since the same exception leaves `main()` as before. Windows, where an antivirus taking the compiler or a locked object is the likelier cause: the change is plain `concurrent.futures`, the same on both. **Must be tested:** None ## Risks **What could break:** compiles already running when one raises still finish (a thread cannot be stopped), so the stop takes as long as the slowest of them; only the ones not started are dropped. A compile that fails in the ordinary way (the compiler's own error) does not raise and is handled as before, by the retry that follows the loop. **Turning it off:** revert the commit **Compatibility:** Nothing changes: the same objects and the same error messages; only the queued compiles after an unexpected error are not run. ## Issues None filed; found by reading `nx2/make_game.py` and `nx2/build.py` whole. Not touched here, for a decision: a full disk in the finish step still gets the "an error it did not expect" message, because `tests/test_builder_drive_error.py` asserts that `ENOSPC` is not a drive error. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. Onto main 94abdd57 (head 70e2343a) with no conflict, with the commit subjects made Conventional Commits, which CI's lint job now checks for every commit, on a base that has CI's config (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/417); each commit's change is the same as before by `git patch-id`, plus one new commit, `chore(build)`, that drops the blank last line of `nx2/build.py`: CI's lint runs the end-of-file hook over the files a PR changes, and this PR changes that one. The commits named in this description are from before this rebase: 1fe16e7d is now f8011e0f, 335e9461 is now a32abce2, 70e2343a is new. Re-run on 94abdd57: `test_compile_stops_on_exception` fails at the test-only commit f8011e0f and passes at the head, with g++ 13.3 and clang 23.1.5; of the 42 checks that are this PR's or read its files (g++ 13.3), 34 pass, 3 skip (a tool this PC lacks) and the 5 that fail fail on main 94abdd57 too (`test_dll`, `test_appimage_player_build`, `test_lift_cache`, `test_pipeline_tour`, `test_vk_pgo`). - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). The full suite on main aa95d35f (Linux, g++ 13.3, the Vulkan headers set): 528 passed, 15 failed, 50 skipped; on this branch the new test and the 41 checks that read `nx2/build.py` ran, with no failure that `main` does not have (see Testing). The 15 that fail on main: `test_appimage_player_build`, `test_bake_portability`, `test_cpm_discovery`, `test_crash_dialog`, `test_env_on`, `test_lift_cache`, `test_lift_cleanup_error`, `test_make_game_cache`, `test_mods_size_table`, `test_options_budget_once`, `test_pipeline_tour`, `test_readback_late`, `test_vk_pgo`, `test_walk_key` and `nx2/test_texture_init_barrier.py`. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] A game-free test added for the fix, failing on `main` and passing here.
Assisted-by: Claude
fix: a compile that raises stops build.py at once instead of after every queued compile
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
1c84a1509d
Assisted-by: Claude
Author
Contributor

Rebased onto main aa95d35f with no conflict (the same changes, git patch-id; commits now under the noreply identity) and the description moved to the current template.
The test was run again on the new base: it fails at the test-only commit and passes at the tip, two interleaved rounds, g++ and clang, with the same numbers as before, so the Measurements are now from aa95d35f.

Rebased onto `main` aa95d35f with no conflict (the same changes, `git patch-id`; commits now under the noreply identity) and the description moved to the current template. The test was run again on the new base: it fails at the test-only commit and passes at the tip, two interleaved rounds, g++ and clang, with the same numbers as before, so the Measurements are now from aa95d35f.
zeldaootr force-pushed build-compile-stops-on-exception from 1c84a1509d
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to 335e9461fb
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-15 16:43:40 +00:00
Compare
Author
Contributor

Rebased onto main 94abdd57 (head 70e2343a), with no conflict; the commit subjects are now Conventional Commits, which CircleCI's lint job checks for every commit of a pull request (#417), and the base has the CI's config, which a fork's pull request needs at its own head, plus one commit that drops nx2/build.py's blank last line, which CI's end-of-file hook would otherwise fail this PR on. No other change: each commit's change is the same as before by git patch-id.
Re-run on 94abdd57: test_compile_stops_on_exception fails at the test-only commit f8011e0f and passes at the head, with g++ 13.3 and clang 23.1.5; of the 42 checks that are this PR's or read its files (g++ 13.3), 34 pass, 3 skip (a tool this PC lacks) and the 5 that fail fail on main 94abdd57 too (test_dll, test_appimage_player_build, test_lift_cache, test_pipeline_tour, test_vk_pgo). The description names the new commits.

Rebased onto `main` 94abdd57 (head 70e2343a), with no conflict; the commit subjects are now Conventional Commits, which CircleCI's lint job checks for every commit of a pull request (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/417), and the base has the CI's config, which a fork's pull request needs at its own head, plus one commit that drops `nx2/build.py`'s blank last line, which CI's end-of-file hook would otherwise fail this PR on. No other change: each commit's change is the same as before by `git patch-id`. Re-run on 94abdd57: `test_compile_stops_on_exception` fails at the test-only commit f8011e0f and passes at the head, with g++ 13.3 and clang 23.1.5; of the 42 checks that are this PR's or read its files (g++ 13.3), 34 pass, 3 skip (a tool this PC lacks) and the 5 that fail fail on main 94abdd57 too (`test_dll`, `test_appimage_player_build`, `test_lift_cache`, `test_pipeline_tour`, `test_vk_pgo`). The description names the new commits.
zeldaootr force-pushed build-compile-stops-on-exception from 335e9461fb
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 4s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 70e2343a16
Some checks are pending
PR description / Description follows the template (pull_request_target) Waiting to run
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
2026-11-15 19:14:40 +00:00
Compare
Some checks are pending
PR description / Description follows the template (pull_request_target) Waiting to run
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'build-compile-stops-on-exception:zeldaootr-build-compile-stops-on-exception'
git switch 'zeldaootr-build-compile-stops-on-exception'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!540
No description provided.