WIP: test: pytest runs the game-free checks; requirements by machine; main's known failures #1818
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1818
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:test/pytest-runner"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: test · Risk: low · Tested on: Windows 11, Ryzen 9 3900X (game-free checks only) · Needs: Linux
TL;DR
Our game-free checks are 860 separate scripts with no test library: a check that cannot run on a PC prints "skipped" and counts as passed, the runner takes whatever compiler is first on PATH (on Windows often an old gcc that fails checks the game's own compiler passes), and main always shows about 45 failures, so a new one is easy to miss. This PR lets pytest run every check unchanged, picks the Builder's compiler, skips a check the machine cannot run with the reason, and lists main's known failures so a red run means something new broke. On Windows, main went from 87 failures to 38 known ones and none new.
Breakdown
tests/known_failures.txtlists main's failures.mise run teststill call run_all.py with the same flags; nothing in the game.Why
The checks are the only safety net we have (Forgejo runs none and CircleCI has been stuck since Nov 15, #1412), and today their results can't be trusted at a glance:
RUN_ALL_SKIPis by OS; 16 checks build with-march, AVX or intrinsics and only 9 of them look at the CPU first.The longer plan is for features to move into folders of their own with their tests beside them (
gpu/vk/menu/since #1774); pytest finds a feature'stests/there.What changed
pytest runs every check
tests/run_all.pyruns, without a check changing: a test file that imports pytest is a pytest module, every other one a script check run in its own process exactly as run_all runs it. What both runners need from the machine is in one place,tests/check_env.py.tests/pytest_longshot.py(the plugin: how a script check and a pytest module are run, the requires marker, the fixturescxx,compile_cxx,read_source,machine,repo), the rootconftest.pyandpytest.ini(tests/, nx2/ and the tests/ of gpu/ and host/).tests/requirements.txtpins pytest 9.2.0, pytest-xdist 3.8.0 and pytest-timeout 2.4.0;mise run test-deps,mise run pytest. run_all itself still needs no package.check_env.find_toolchain():$CXX, thenLONGSHOT_MINGW's clang++, then the Builder's own llvm-mingw (scripts/dev_build.py's builder folder), then PATH. An llvm-mingw it picks goes first on PATH (its programs load its libc++ DLLs from there) and becomesLONGSHOT_MINGWwhen that is unset (the a64-kit's extension builds through it; without it six checks stopped on'algorithm' file not found). Every run's first line says what it found:here: windows x86_64 x86-64-v3, clang 23.1 (mingw) at ....RUN_ALL_REQUIRES = {...}in a script check,@pytest.mark.requires(...)on a pytest test:os,arch,cpu(features or a level,x86-64-v3),cxx(clang>=17,gcc>=13) andcxx_flags. A machine that lacks an OS, CPU or feature always skips (even when the check is named;--no-skipstill runs it); a missing or too old compiler skips, or fails with--strict. CPU features come from a small CPUID and XGETBV program (any gcc or clang) or/proc/cpuinfo.check_env.skip(why)ends a script check with exit 77, shown as skipped with the reason.tests/test_fastfp_fma.pyis converted as the example.tests/known_failures.txt, one line a check:<check> <os> <why>. Listed and failing:known, not a failure of the run. Listed and passing:FIXED, a failure until its line goes. A pytest module's entries are applied by pytest as strict xfails (by file or by node id).RUN_ALL_REQUIRES, runs a pytest module withpython -m pytest, honours exit 77 and the known failures, prints the machine, and looks up each tool once (which()cached: collection went from 18 s to 10 s under pytest).tests/test_game_menu_mouse_buttons.py's fixture kept three counters nothing read, which clang 21+ refuses under-Werror(its own commit).tests/test_vbytes.py, which never ran on Windows before, would take about 25 minutes there (6 minutes a-march; stopped after 22) at its default 1,000,000 random states per guest word (130 words, four-marchbuilds): the runners now pass--random 50000, which still gives every index value of every word about 3,000 hits (its own commit, with the reasoning; for #1576's author: a full-depth run is stillpython -I -B tests/test_vbytes.py, and a stronger check would feed every one of the 256 index values to every lane).tests/test_check_env.py(the first pytest module, 33 tests: requirements, the machine, the known failures file, and both runners end to end over a scratch repository);test_run_all_declarations.pycovers the new declaration;test_run_all_output.pyandtest_run_all_reap.pycopycheck_env.pybeside run_all.py.Measurements
Verdict: on Windows, main's 87 failures under the old runner become 38 known failures and 0 new ones under run_all; pytest agrees on every check but two timeouts under load (below).
Hardware: Windows 11 Pro, Ryzen 9 3900X (x86-64-v3, no AVX-512), Python 3.11.8 in a venv with the repository's pins (numpy 2.4.6: 2.5.4 needs Python 3.12)
Method: a full run of each, 24 at once, one after another (the last pytest run overlapped another session's runtime build)
a433db32, old run_all8ffe5053, run_all8ffe5053, pytest -n 24(main's 978 s was with the old gcc and other work on the PC: not a speed comparison. pytest counts the 33 tests of test_check_env.py one by one, run_all as one check.)
Of main's 87: 50 pass here (the compiler and
LONGSHOT_MINGWfailed them, not the code; one more after its fixture fix, and test_vbytes at its runner depth), and 37 are the known failures, each with why. The "Before" column is main @a433db32; this PR was rebased on0de25c3e(#1774) since, and both full runs of the rebased branch are the "After" columns.The 2 pytest failures were timeouts inside the checks while another runtime build ran on this PC (the CPU at 99 %):
test_fallback_buffer_cleanup(a 10 s limit on its fixture) andtest_jobs_wait_runs_queued(60 s on its race case). Both passed in the run_all run of the same commit, three times each alone under pytest (5 to 7 s), and both were among main's 87: timing-sensitive checks, not this PR. One check that was skipped before (no clang on PATH),test_linux_preflight, now runs and fails as it does wherever clang is found: it is in the list.Flags and settings
None for the game. For the checks:
RUN_ALL_REQUIRES(a declaration), and pytest's--strict-tools,--no-skip,--check-timeout,--skip-exit,--known-exit(the last two are how run_all runs a pytest module).Testing
Tested: full runs on Windows 11 (Ryzen 9 3900X) of main and of this branch, under run_all.py and under pytest with xdist: the same failures, all in the known list; the runners' own tests pass (test_check_env, test_run_all_declarations, _output, _junit, _changed, _schedule, _reap); test_vbytes and test_seed_reader timed alone.
Not tested: Linux (no Linux PC here): the run_all and pytest paths are the same code, the Linux process reaping is unchanged, and
/proc/cpuinfogives the CPU features there.Must be tested:
python -I -B tests/run_all.pyandpython -I -B -m pytest -n autoafterpython -m pip install -r tests/requirements.txt, expecting the same failures under both; the failing checks go intotests/known_failures.txtaslinuxlines (by hand: anyone on Linux).Risks
What could break: on Windows the checks get the Builder's llvm-mingw where it is unpacked, not the first compiler on PATH; a check that only passed with another compiler shows up (
test_game_menu_mouse_buttonswas one, fixed here).tests/test_seed_reader.pywrites its seed file 20,000 times and takes 99 s alone on Windows, near its own 300 s limit under a full parallel run: it timed out in one of the six full runs of this branch here, and failed on main under the old runner (not changed by this PR).Turning it off:
CXX=<compiler>picks another compiler; reverting the commits removes the rest.Compatibility: nothing changes for a player.
Issues
See #1412 (CI not running).
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Possibly related, for context:
View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.