fix(tests): run_all uses the clone's llvm-mingw folder, as make_game does #1032

Open
enum wants to merge 3 commits from enum/Longshot:fix/run-all-clone-mingw into main
First-time contributor

Type: fix · Risk: low · Tested on: Windows 11, llvm-mingw 23.1.5 (the clone's llvm-mingw folder) · Needs: none

Why

CONTRIBUTING.md and docs/development.md give two ways to set up the compiler on Windows: set LONGSHOT_MINGW, or put
llvm-mingw in an llvm-mingw folder in the clone. The build (nx2/make_game.py's bundled_mingw()) takes either, but
tests/run_all.py took only the variable. With the folder alone, three game-free checks fail on Windows for a reason
that has nothing to do with the code they test: test_spawn_emit, test_fpx_sticky and test_fastfp_fma. A
contributor who set up the second way sees red checks on a clean main.

How to reproduce

Seen on: Windows 11, llvm-mingw 23.1.5 in <clone>\llvm-mingw, its bin on PATH, LONGSHOT_MINGW unset
Settings and switches: defaults
Steps: python -I -B tests/run_all.py
Expected: the three checks pass, as they do with LONGSHOT_MINGW set to the same folder
Actual: they fail with fatal error: 'algorithm' file not found. They find the folder's clang++ on PATH, but
toolchain.mingw() (third_party/a64-kit/tools/toolchain.py) reads only LONGSHOT_MINGW, so they compile for
x86_64-pc-windows-msvc, where llvm-mingw has no C++ library.

What changed

run_all now does what make_game does: on Windows, when LONGSHOT_MINGW is unset and the clone's
llvm-mingw/bin/clang.exe exists, it sets LONGSHOT_MINGW to that folder for the checks it starts.

  • tests: bundled_mingw() in tests/run_all.py, called in main() before find_tools(). A LONGSHOT_MINGW that is
    already set wins. LONGSHOT_MSVC=1 (the Visual Studio route) and a folder without its clang leave it unset. It does
    nothing on Linux.
  • tests: tests/test_run_all_mingw.py (new, the first commit) runs a copy of run_all.py in a scratch repository with
    a stand-in llvm-mingw/bin/clang.exe and a probe check that records the LONGSHOT_MINGW it was started with. Its 4
    cases: the folder with the variable unset, a variable already set, LONGSHOT_MSVC=1, and a folder without clang.

third_party/ is untouched: the fix sits outside toolchain.py.

Measurements

Verdict: passes. The new test fails on main and passes here; the full suite on Windows goes from 23 failures to
19: the three checks above and the new test, and nothing else differs.

Hardware: Windows 11 Pro (26300), llvm-mingw 23.1.5 in the clone's folder, Python 3.12.10
Scene: none (game-free checks)
Method: python -I -B tests/test_run_all_mingw.py, and python -I -B tests/run_all.py with LONGSHOT_MINGW unset, on
the same PC and shell
How to reproduce: the commands above, with llvm-mingw in <clone>\llvm-mingw and LONGSHOT_MINGW unset

Before: main @ 5f3bb44b (run_all.py from main, the new test added) After: this PR @ e1ab75fe
test_run_all_mingw.py 1 of 4 cases fails 4 of 4 pass
run_all.py 598 passed, 23 failed, 42 skipped 602 passed, 19 failed, 42 skipped
== main's run_all.py
FAIL: test_the_clone_folder_when_the_variable_is_unset
    self.assertEqual(seen, folder if WINDOWS else "<unset>")
AssertionError: '<unset>' != 'C:\\Users\\...\\Temp\\tmpt5bylcq6\\llvm-mingw'
Ran 4 tests in 0.475s
FAILED (failures=1)

== this PR
Ran 4 tests in 0.475s
OK

== run_all.py, this PR
pass      12.4s  tests/test_spawn_emit.py
pass      55.9s  tests/test_fpx_sticky.py
pass       7.3s  tests/test_fastfp_fma.py
pass       1.7s  tests/test_run_all_mingw.py
602 passed, 19 failed, 42 skipped in 126 s (windows, 24 at once)

Flags and settings

None. No switch is added. run_all now sets the existing LONGSHOT_MINGW for its own checks, the way make_game does.

Testing

Tested: On Windows 11, the new test (fails on main, passes here) and the full run_all.py with LONGSHOT_MINGW
unset, before and after. I also ran each check that reads run_all.py on its own (test_run_all_*, test_bake_*,
test_check_pr_description, test_cpm_discovery, test_portable_builder_startup, test_program_cache_cleanup,
test_relocations, test_split_giants, test_spawn_emit). All pass except test_run_all_changed, which fails on
main too (#954). The new test also passes in WSL Linux, where the function does nothing. To check that the test
catches a broken fix: removing the assignment makes it fail, and dropping the LONGSHOT_MSVC condition makes
test_the_visual_studio_route_leaves_it_unset fail.

Still failing here, the same on main: test_auto_handheld, test_bug_report_defaults (#773), test_crash_dialog,
test_crt_options, test_fsr_uniform_ownership, test_guest_imported, test_import_reach, test_linux_preflight
(#999, #592), test_make_game_cache (#592), test_missing_nup (#999), test_pads_after_menu,
test_parallel_init_order, test_pending_guest_import, test_replay_write_order (#937), test_run_all_changed
(#954), test_tested_configurations (#493), test_vk_occlusion_queries, test_walk_key (#537). test_gpu_hang_watch
fails on main too and is flaky on this PC: run on its own 3 times each, it passed and failed about equally with
LONGSHOT_MINGW set or unset.

Not tested: a full run_all on Linux, where the new function returns before doing anything; the Visual Studio route
end to end (the test covers its condition).

Must be tested: None. This is a game-free change to the test runner: no in-game run applies.

Risks

What could break: only run_all on Windows with the folder and no variable, which was the failing setup. A check
that expected LONGSHOT_MINGW unset there now sees it set, the same as with the first setup route.

Turning it off: set LONGSHOT_MINGW or LONGSHOT_MSVC=1, or revert the commit.

Compatibility: Nothing changes for players.

Out of scope: a check run on its own, outside run_all, still needs LONGSHOT_MINGW with the folder setup; and run_all
doesn't fall back to llvm-mingw\bin\clang++ when it isn't on PATH.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.

🤖 Generated with Claude Code

**Type:** fix · **Risk:** low · **Tested on:** Windows 11, llvm-mingw 23.1.5 (the clone's `llvm-mingw` folder) · **Needs:** none ## Why CONTRIBUTING.md and docs/development.md give two ways to set up the compiler on Windows: set `LONGSHOT_MINGW`, or put llvm-mingw in an `llvm-mingw` folder in the clone. The build (`nx2/make_game.py`'s `bundled_mingw()`) takes either, but `tests/run_all.py` took only the variable. With the folder alone, three game-free checks fail on Windows for a reason that has nothing to do with the code they test: `test_spawn_emit`, `test_fpx_sticky` and `test_fastfp_fma`. A contributor who set up the second way sees red checks on a clean `main`. ## How to reproduce **Seen on:** Windows 11, llvm-mingw 23.1.5 in `<clone>\llvm-mingw`, its `bin` on PATH, `LONGSHOT_MINGW` unset **Settings and switches:** defaults **Steps:** `python -I -B tests/run_all.py` **Expected:** the three checks pass, as they do with `LONGSHOT_MINGW` set to the same folder **Actual:** they fail with `fatal error: 'algorithm' file not found`. They find the folder's `clang++` on PATH, but `toolchain.mingw()` (`third_party/a64-kit/tools/toolchain.py`) reads only `LONGSHOT_MINGW`, so they compile for `x86_64-pc-windows-msvc`, where llvm-mingw has no C++ library. ## What changed run_all now does what make_game does: on Windows, when `LONGSHOT_MINGW` is unset and the clone's `llvm-mingw/bin/clang.exe` exists, it sets `LONGSHOT_MINGW` to that folder for the checks it starts. - tests: `bundled_mingw()` in `tests/run_all.py`, called in `main()` before `find_tools()`. A `LONGSHOT_MINGW` that is already set wins. `LONGSHOT_MSVC=1` (the Visual Studio route) and a folder without its clang leave it unset. It does nothing on Linux. - tests: `tests/test_run_all_mingw.py` (new, the first commit) runs a copy of run_all.py in a scratch repository with a stand-in `llvm-mingw/bin/clang.exe` and a probe check that records the `LONGSHOT_MINGW` it was started with. Its 4 cases: the folder with the variable unset, a variable already set, `LONGSHOT_MSVC=1`, and a folder without clang. `third_party/` is untouched: the fix sits outside `toolchain.py`. ## Measurements **Verdict:** passes. The new test fails on `main` and passes here; the full suite on Windows goes from 23 failures to 19: the three checks above and the new test, and nothing else differs. Hardware: Windows 11 Pro (26300), llvm-mingw 23.1.5 in the clone's folder, Python 3.12.10 Scene: none (game-free checks) Method: `python -I -B tests/test_run_all_mingw.py`, and `python -I -B tests/run_all.py` with `LONGSHOT_MINGW` unset, on the same PC and shell How to reproduce: the commands above, with llvm-mingw in `<clone>\llvm-mingw` and `LONGSHOT_MINGW` unset | | Before: main @ 5f3bb44b (run_all.py from main, the new test added) | After: this PR @ e1ab75fe | |---|---|---| | `test_run_all_mingw.py` | 1 of 4 cases fails | 4 of 4 pass | | `run_all.py` | 598 passed, 23 failed, 42 skipped | 602 passed, 19 failed, 42 skipped | ``` == main's run_all.py FAIL: test_the_clone_folder_when_the_variable_is_unset self.assertEqual(seen, folder if WINDOWS else "<unset>") AssertionError: '<unset>' != 'C:\\Users\\...\\Temp\\tmpt5bylcq6\\llvm-mingw' Ran 4 tests in 0.475s FAILED (failures=1) == this PR Ran 4 tests in 0.475s OK == run_all.py, this PR pass 12.4s tests/test_spawn_emit.py pass 55.9s tests/test_fpx_sticky.py pass 7.3s tests/test_fastfp_fma.py pass 1.7s tests/test_run_all_mingw.py 602 passed, 19 failed, 42 skipped in 126 s (windows, 24 at once) ``` ### Flags and settings None. No switch is added. run_all now sets the existing `LONGSHOT_MINGW` for its own checks, the way make_game does. ## Testing **Tested:** On Windows 11, the new test (fails on main, passes here) and the full `run_all.py` with `LONGSHOT_MINGW` unset, before and after. I also ran each check that reads run_all.py on its own (`test_run_all_*`, `test_bake_*`, `test_check_pr_description`, `test_cpm_discovery`, `test_portable_builder_startup`, `test_program_cache_cleanup`, `test_relocations`, `test_split_giants`, `test_spawn_emit`). All pass except `test_run_all_changed`, which fails on main too (#954). The new test also passes in WSL Linux, where the function does nothing. To check that the test catches a broken fix: removing the assignment makes it fail, and dropping the `LONGSHOT_MSVC` condition makes `test_the_visual_studio_route_leaves_it_unset` fail. Still failing here, the same on main: `test_auto_handheld`, `test_bug_report_defaults` (#773), `test_crash_dialog`, `test_crt_options`, `test_fsr_uniform_ownership`, `test_guest_imported`, `test_import_reach`, `test_linux_preflight` (#999, #592), `test_make_game_cache` (#592), `test_missing_nup` (#999), `test_pads_after_menu`, `test_parallel_init_order`, `test_pending_guest_import`, `test_replay_write_order` (#937), `test_run_all_changed` (#954), `test_tested_configurations` (#493), `test_vk_occlusion_queries`, `test_walk_key` (#537). `test_gpu_hang_watch` fails on main too and is flaky on this PC: run on its own 3 times each, it passed and failed about equally with `LONGSHOT_MINGW` set or unset. **Not tested:** a full run_all on Linux, where the new function returns before doing anything; the Visual Studio route end to end (the test covers its condition). **Must be tested:** None. This is a game-free change to the test runner: no in-game run applies. ## Risks **What could break:** only run_all on Windows with the folder and no variable, which was the failing setup. A check that expected `LONGSHOT_MINGW` unset there now sees it set, the same as with the first setup route. **Turning it off:** set `LONGSHOT_MINGW` or `LONGSHOT_MSVC=1`, or revert the commit. **Compatibility:** Nothing changes for players. Out of scope: a check run on its own, outside run_all, still needs `LONGSHOT_MINGW` with the folder setup; and run_all doesn't fall back to `llvm-mingw\bin\clang++` when it isn't on PATH. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
A copy of run_all.py runs in a scratch repository with a stand-in llvm-mingw/bin/clang and a check that writes down the
LONGSHOT_MINGW it was started with. On Windows, with the variable unset, the check must see the folder, as
nx2/make_game.py's bundled_mingw() gives it to the build; a variable already set wins, and LONGSHOT_MSVC=1 or a folder
without its compiler leave it unset. On main the first case fails on Windows: the check sees it unset, which is why
test_spawn_emit, test_fpx_sticky and test_fastfp_fma compile for the MSVC target there.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(tests): run_all uses the clone's llvm-mingw folder, as make_game does
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
e1ab75fe59
CONTRIBUTING.md and docs/development.md give two ways to set up the compiler on Windows: LONGSHOT_MINGW, or an
llvm-mingw folder in the clone. nx2/make_game.py's bundled_mingw() handles the folder; run_all.py did not. Its checks
found that folder's clang++ on PATH, but toolchain.mingw() reads only the variable, so test_spawn_emit, test_fpx_sticky
and test_fastfp_fma compiled for x86_64-pc-windows-msvc, where llvm-mingw has no C++ library: "'algorithm' file not
found". run_all now sets LONGSHOT_MINGW to the folder for its checks on Windows when the variable is unset, LONGSHOT_MSVC=1
is not, and the folder has its clang.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Collaborator

Possibly related, for context:

  • PR #893 perf(tests): cache the checks' compiles by what they compile — Overlapping changes to tests/run_all.py and check compiler setup
Possibly related, for context: - PR #893 perf(tests): cache the checks' compiles by what they compile — Overlapping changes to tests/run_all.py and check compiler setup
Author
First-time contributor

Re the possibly related #893: checked, no overlap in what they do. #893 caches the checks' compiles; this PR picks which llvm-mingw they compile with (LONGSHOT_MINGW from the clone's folder). They touch different parts of tests/run_all.py and the branches merge cleanly (git merge-tree). With both in, the checks still inherit the LONGSHOT_MINGW set here, and #893 keys a compile by its compiler and flags, so the target this fixes gets its own entries.

Also since opening: Before is now measured on main @ 5f3bb44b, the PR's base (598 passed / 23 failed / 42 skipped; this PR 602 / 19 / 42; only the three checks and the new test differ).

Re the possibly related #893: checked, no overlap in what they do. #893 caches the checks' compiles; this PR picks which llvm-mingw they compile with (LONGSHOT_MINGW from the clone's folder). They touch different parts of tests/run_all.py and the branches merge cleanly (git merge-tree). With both in, the checks still inherit the LONGSHOT_MINGW set here, and #893 keys a compile by its compiler and flags, so the target this fixes gets its own entries. Also since opening: Before is now measured on main @ 5f3bb44b, the PR's base (598 passed / 23 failed / 42 skipped; this PR 602 / 19 / 42; only the three checks and the new test differ).
enum changed title from WIP: fix(tests): run_all uses the clone's llvm-mingw folder, as make_game does to fix(tests): run_all uses the clone's llvm-mingw folder, as make_game does 2026-11-16 03:44:46 +00:00
Merge branch 'main' into fix/run-all-clone-mingw
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
e68b073045
Member

Code-only review of e68b073045; informational findings, with final approval left to another reviewer.

No introduced correctness issue found in the complete diff. The runner sets the clone's compiler folder before starting checks, preserves an existing LONGSHOT_MINGW, and leaves Linux and the explicit Visual Studio route alone. The environment value is passed directly to child processes, without shell interpolation.

The new regression checks what a real child check inherits, rather than matching source text. Its four cases cover discovery, an explicit override, Visual Studio opt-out, and a missing compiler. The PR records Windows before/after runs of the three affected checks and a WSL pass of the new regression; I did not rerun them. The fixture verifies selection and propagation, not whether a real compiler can link, which the author's Windows runs cover for the reported setup.

Read the runner, regression, affected checks, make_game.py discovery and the shared toolchain's compiler/target selection. No tests, builds, generators or game code were executed, and no source changes were made. Final approval remains with another reviewer.

Code-only review of e68b073045822b8fc62c0b1ec5990ec6ec70ee2a; informational findings, with final approval left to another reviewer. No introduced correctness issue found in the complete diff. The runner sets the clone's compiler folder before starting checks, preserves an existing `LONGSHOT_MINGW`, and leaves Linux and the explicit Visual Studio route alone. The environment value is passed directly to child processes, without shell interpolation. The new regression checks what a real child check inherits, rather than matching source text. Its four cases cover discovery, an explicit override, Visual Studio opt-out, and a missing compiler. The PR records Windows before/after runs of the three affected checks and a WSL pass of the new regression; I did not rerun them. The fixture verifies selection and propagation, not whether a real compiler can link, which the author's Windows runs cover for the reported setup. Read the runner, regression, affected checks, `make_game.py` discovery and the shared toolchain's compiler/target selection. No tests, builds, generators or game code were executed, and no source changes were made. Final approval remains with another reviewer.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/run-all-clone-mingw:enum-fix/run-all-clone-mingw'
git switch 'enum-fix/run-all-clone-mingw'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1032
No description provided.