fix(builder): a compile that raises stops build.py at once, not after every queued compile #540
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8015c9a
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!540
Loading…
Reference in a new issue
No description provided.
Delete branch "zeldaootr/Longshot:build-compile-stops-on-exception"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux, no GPU (game-free check) · Needs: none
Why
When one compile in a build fails with an error the Builder did not plan for (the compiler gone, an object or its stamp that cannot be written, a full disk), or when a source build is stopped with Ctrl-C,
build.pyfirst runs every compile still waiting, with nothing printed, and only then stops. The player sees a frozen progress bar and gets the first file's error up to an hour later.How to reproduce
Seen on: Linux (Ubuntu 24.04), no GPU; any OS, it is the build script
Settings and switches: none
Steps: on
mainaa95d35fwith this PR's test added (1fe16e7d), runpython -I -B tests/test_compile_stops_on_exception.py: it runs the real compile loop ofnx2/build.pywith stand-in compilers whose first job raisesFileNotFoundError(1 worker, 5 jobs) orPermissionError(2 workers)Expected: the build stops once the running compiles finish; at most one queued job per worker starts after the failure
Actual: all 4 queued compiles start and finish before the exception leaves
main()What changed
A compile that raises now stops
build.pyat once: the compiles not yet started are cancelled instead of run, and the same exception leavesmain()as before.main()ofnx2/build.pycallsfuture.result()insidewith ThreadPoolExecutor. When a call raises, leaving thewithwaits for the pool, which starts and finishes every queued job first. The loop is now in atry: on any exception the queued jobs are cancelled (ex.shutdown(wait=False, cancel_futures=True)) and the exception goes on as before. The loop body is only indented one level;git diff -wshows the six added lines.nx2/build.pyends with one newline (CI's lint runs the end-of-file hook on the files a PR changes).tests/test_compile_stops_on_exception.pyruns the realcompile_oneand the real loop, cut out of the source, with stand-in compilers: 1 worker and 5 jobs where the first raisesFileNotFoundError, and 2 workers where one raisesPermissionError. It allows one queued job per worker to start, since a worker can take its next job before the main thread hears of the failure.Measurements
Verdict: fixed: on
main4 queued compiles still start after the failure in both cases; with this PR at most one per worker, in two interleaved rounds.Hardware: Linux x86-64 PC (Ubuntu 24.04), no GPU used; Python 3.15
Scene: no game: the game-free check
Method:
python3 -I -B tests/test_compile_stops_on_exception.pyat the test-only commit1fe16e7d(mainaa95d35fplus the test) and at335e9461, twice each, alternatingHow to reproduce: check out each commit and run the command above
aa95d35f(with the test,1fe16e7d)335e9461Flags and settings
None
Testing
Tested: Linux, game-free. The new test fails on the test-only commit
1fe16e7dand passes at335e9461(this branch rebased ontomainaa95d35f, which changednx2/build.pyin five merges since the old base1be60eae, none in the lines this PR changes), twice each, alternating, and undertests/run_all.pywith g++ 13.3 and with clang 23.1.5 asCXXat both commits. At the tip, the 41 checks that readnx2/build.pyor the new test: 32 passed, 3 skipped, 6 failed, each of the 6 also failing or skipped by OS onmainaa95d35f(test_appimage_player_build,test_cpm_discovery,test_lift_cache,test_pipeline_tour,test_vk_pgo, andtest_dll, a Windows-only check run here by name).tests/test_compile_retry_jobs.py, which cuts the same part ofmain()out of the source, passes. Before the rebase the same atba05f7b7and1c84a150on main1be60eae.Not tested: A real build stopped this way (it needs a game dump): the Builder's own handling of the exit is unchanged, since the same exception leaves
main()as before. Windows, where an antivirus taking the compiler or a locked object is the likelier cause: the change is plainconcurrent.futures, the same on both.Must be tested: None
Risks
What could break: compiles already running when one raises still finish (a thread cannot be stopped), so the stop takes as long as the slowest of them; only the ones not started are dropped. A compile that fails in the ordinary way (the compiler's own error) does not raise and is handled as before, by the retry that follows the loop.
Turning it off: revert the commit
Compatibility: Nothing changes: the same objects and the same error messages; only the queued compiles after an unexpected error are not run.
Issues
None filed; found by reading
nx2/make_game.pyandnx2/build.pywhole. Not touched here, for a decision: a full disk in the finish step still gets the "an error it did not expect" message, becausetests/test_builder_drive_error.pyasserts thatENOSPCis not a drive error.Checklist
Everyone
type(scope): description) of the status line's type.main. Onto main94abdd57(head70e2343a) with no conflict, with the commit subjects made Conventional Commits, which CI's lint job now checks for every commit, on a base that has CI's config (#417); each commit's change is the same as before bygit patch-id, plus one new commit,chore(build), that drops the blank last line ofnx2/build.py: CI's lint runs the end-of-file hook over the files a PR changes, and this PR changes that one. The commits named in this description are from before this rebase:1fe16e7dis nowf8011e0f,335e9461is nowa32abce2,70e2343ais new. Re-run on94abdd57:test_compile_stops_on_exceptionfails at the test-only commitf8011e0fand passes at the head, with g++ 13.3 and clang 23.1.5; of the 42 checks that are this PR's or read its files (g++ 13.3), 34 pass, 3 skip (a tool this PC lacks) and the 5 that fail fail on main94abdd57too (test_dll,test_appimage_player_build,test_lift_cache,test_pipeline_tour,test_vk_pgo).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). The full suite on mainaa95d35f(Linux, g++ 13.3, the Vulkan headers set): 528 passed, 15 failed, 50 skipped; on this branch the new test and the 41 checks that readnx2/build.pyran, with no failure thatmaindoes not have (see Testing). The 15 that fail on main:test_appimage_player_build,test_bake_portability,test_cpm_discovery,test_crash_dialog,test_env_on,test_lift_cache,test_lift_cleanup_error,test_make_game_cache,test_mods_size_table,test_options_budget_once,test_pipeline_tour,test_readback_late,test_vk_pgo,test_walk_keyandnx2/test_texture_init_barrier.py.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.build.pymain) ba05f7b7b8build.pyat once instead of after every queued compileRebased onto
mainaa95d35fwith no conflict (the same changes,git patch-id; commits now under the noreply identity) and the description moved to the current template.The test was run again on the new base: it fails at the test-only commit and passes at the tip, two interleaved rounds, g++ and clang, with the same numbers as before, so the Measurements are now from
aa95d35f.1c84a1509d335e9461fbRebased onto
main94abdd57(head70e2343a), with no conflict; the commit subjects are now Conventional Commits, which CircleCI's lint job checks for every commit of a pull request (#417), and the base has the CI's config, which a fork's pull request needs at its own head, plus one commit that dropsnx2/build.py's blank last line, which CI's end-of-file hook would otherwise fail this PR on. No other change: each commit's change is the same as before bygit patch-id.Re-run on
94abdd57:test_compile_stops_on_exceptionfails at the test-only commitf8011e0fand passes at the head, with g++ 13.3 and clang 23.1.5; of the 42 checks that are this PR's or read its files (g++ 13.3), 34 pass, 3 skip (a tool this PC lacks) and the 5 that fail fail on main94abdd57too (test_dll,test_appimage_player_build,test_lift_cache,test_pipeline_tour,test_vk_pgo). The description names the new commits.335e9461fb70e2343a16View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.