fix(builder): a compile that raises stops build.py at once, not after every queued compile #540
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!540
Loading…
Reference in a new issue
No description provided.
Delete branch "zeldaootr/Longshot:build-compile-stops-on-exception"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Linux, no GPU (game-free check) · Needs: none
Why
When one compile in a build fails with an error the Builder did not plan for (the compiler gone, an object or its stamp that cannot be written, a full disk), or when a source build is stopped with Ctrl-C,
build.pyfirst runs every compile still waiting, with nothing printed, and only then stops. The player sees a frozen progress bar and gets the first file's error up to an hour later.How to reproduce
Seen on: Linux (Ubuntu 24.04), no GPU; any OS, it is the build script
Settings and switches: none
Steps: on
mainaa95d35fwith this PR's test added (1fe16e7d), runpython -I -B tests/test_compile_stops_on_exception.py: it runs the real compile loop ofnx2/build.pywith stand-in compilers whose first job raisesFileNotFoundError(1 worker, 5 jobs) orPermissionError(2 workers)Expected: the build stops once the running compiles finish; at most one queued job per worker starts after the failure
Actual: all 4 queued compiles start and finish before the exception leaves
main()What changed
A compile that raises now stops
build.pyat once: the compiles not yet started are cancelled instead of run, and the same exception leavesmain()as before.main()ofnx2/build.pycallsfuture.result()insidewith ThreadPoolExecutor. When a call raises, leaving thewithwaits for the pool, which starts and finishes every queued job first. The loop is now in atry: on any exception the queued jobs are cancelled (ex.shutdown(wait=False, cancel_futures=True)) and the exception goes on as before. The loop body is only indented one level;git diff -wshows the six added lines.nx2/build.pyends with one newline (CI's lint runs the end-of-file hook on the files a PR changes).tests/test_compile_stops_on_exception.pyruns the realcompile_oneand the real loop, cut out of the source, with stand-in compilers: 1 worker and 5 jobs where the first raisesFileNotFoundError, and 2 workers where one raisesPermissionError. It allows one queued job per worker to start, since a worker can take its next job before the main thread hears of the failure.Measurements
Verdict: fixed: on
main4 queued compiles still start after the failure in both cases; with this PR at most one per worker, in two interleaved rounds.Hardware: Linux x86-64 PC (Ubuntu 24.04), no GPU used; Python 3.15
Scene: no game: the game-free check
Method:
python3 -I -B tests/test_compile_stops_on_exception.pyat the test-only commit1fe16e7d(mainaa95d35fplus the test) and at335e9461, twice each, alternatingHow to reproduce: check out each commit and run the command above
aa95d35f(with the test,1fe16e7d)335e9461Flags and settings
None
Testing
Tested: Linux, game-free. The new test fails on the test-only commit
1fe16e7dand passes at335e9461(this branch rebased ontomainaa95d35f, which changednx2/build.pyin five merges since the old base1be60eae, none in the lines this PR changes), twice each, alternating, and undertests/run_all.pywith g++ 13.3 and with clang 23.1.5 asCXXat both commits. At the tip, the 41 checks that readnx2/build.pyor the new test: 32 passed, 3 skipped, 6 failed, each of the 6 also failing or skipped by OS onmainaa95d35f(test_appimage_player_build,test_cpm_discovery,test_lift_cache,test_pipeline_tour,test_vk_pgo, andtest_dll, a Windows-only check run here by name).tests/test_compile_retry_jobs.py, which cuts the same part ofmain()out of the source, passes. Before the rebase the same atba05f7b7and1c84a150on main1be60eae.Not tested: A real build stopped this way (it needs a game dump): the Builder's own handling of the exit is unchanged, since the same exception leaves
main()as before. Windows, where an antivirus taking the compiler or a locked object is the likelier cause: the change is plainconcurrent.futures, the same on both.Must be tested: None
Risks
What could break: compiles already running when one raises still finish (a thread cannot be stopped), so the stop takes as long as the slowest of them; only the ones not started are dropped. A compile that fails in the ordinary way (the compiler's own error) does not raise and is handled as before, by the retry that follows the loop.
Turning it off: revert the commit
Compatibility: Nothing changes: the same objects and the same error messages; only the queued compiles after an unexpected error are not run.
Issues
None filed; found by reading
nx2/make_game.pyandnx2/build.pywhole. Not touched here, for a decision: a full disk in the finish step still gets the "an error it did not expect" message, becausetests/test_builder_drive_error.pyasserts thatENOSPCis not a drive error.Checklist
Everyone
type(scope): description) of the status line's type.main. Onto main94abdd57(head70e2343a) with no conflict, with the commit subjects made Conventional Commits, which CI's lint job now checks for every commit, on a base that has CI's config (#417); each commit's change is the same as before bygit patch-id, plus one new commit,chore(build), that drops the blank last line ofnx2/build.py: CI's lint runs the end-of-file hook over the files a PR changes, and this PR changes that one. The commits named in this description are from before this rebase:1fe16e7dis nowf8011e0f,335e9461is nowa32abce2,70e2343ais new. Re-run on94abdd57:test_compile_stops_on_exceptionfails at the test-only commitf8011e0fand passes at the head, with g++ 13.3 and clang 23.1.5; of the 42 checks that are this PR's or read its files (g++ 13.3), 34 pass, 3 skip (a tool this PC lacks) and the 5 that fail fail on main94abdd57too (test_dll,test_appimage_player_build,test_lift_cache,test_pipeline_tour,test_vk_pgo).python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). The full suite on mainaa95d35f(Linux, g++ 13.3, the Vulkan headers set): 528 passed, 15 failed, 50 skipped; on this branch the new test and the 41 checks that readnx2/build.pyran, with no failure thatmaindoes not have (see Testing). The 15 that fail on main:test_appimage_player_build,test_bake_portability,test_cpm_discovery,test_crash_dialog,test_env_on,test_lift_cache,test_lift_cleanup_error,test_make_game_cache,test_mods_size_table,test_options_budget_once,test_pipeline_tour,test_readback_late,test_vk_pgo,test_walk_keyandnx2/test_texture_init_barrier.py.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.build.pymain) ba05f7b7b8build.pyat once instead of after every queued compileRebased onto
mainaa95d35fwith no conflict (the same changes,git patch-id; commits now under the noreply identity) and the description moved to the current template.The test was run again on the new base: it fails at the test-only commit and passes at the tip, two interleaved rounds, g++ and clang, with the same numbers as before, so the Measurements are now from
aa95d35f.1c84a1509d335e9461fbRebased onto
main94abdd57(head70e2343a), with no conflict; the commit subjects are now Conventional Commits, which CircleCI's lint job checks for every commit of a pull request (#417), and the base has the CI's config, which a fork's pull request needs at its own head, plus one commit that dropsnx2/build.py's blank last line, which CI's end-of-file hook would otherwise fail this PR on. No other change: each commit's change is the same as before bygit patch-id.Re-run on
94abdd57:test_compile_stops_on_exceptionfails at the test-only commitf8011e0fand passes at the head, with g++ 13.3 and clang 23.1.5; of the 42 checks that are this PR's or read its files (g++ 13.3), 34 pass, 3 skip (a tool this PC lacks) and the 5 that fail fail on main94abdd57too (test_dll,test_appimage_player_build,test_lift_cache,test_pipeline_tour,test_vk_pgo). The description names the new commits.335e9461fb70e2343a16Read the diff (code only, nothing run).
The change is right.
future.result()raising insidewith ThreadPoolExecutorused to let the pool's exit run every queued compile before the exception leftmain(). Nowex.shutdown(wait=False, cancel_futures=True)drops the unstarted jobs, thenraisere-raises the same exception. Thewithexit still waits for the compiles already running, which the description says.BaseExceptionalso covers Ctrl-C. The ordinary compiler-error path (errreturned, not raised) is untouched. The loop body is only re-indented, and the last-line newline change is harmless.The test cuts the real
compile_oneand the loop out ofbuild.pyby AST and uses stand-in compilers, with a bound of one queued job per worker. That bound is honest about the race. It is a game-free test that fails on main, as the description shows.Overlap: #425 rewrites
printtoannouncein this same loop. That is a textual conflict with the re-indent here and is trivial to resolve. Merging this one first is simplest, and then 425 rebases onto the extratry. #660, #791 and #673 touch other parts ofbuild.py, so there is no overlap with them in the lines changed.Approving: low risk, mergeable, and tested game-free by the author.