WIP: test: teach the game-free runner about macOS #454
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!454
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:mac/run-all"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is for: anyone running the game-free checks on a Mac. Today
tests/run_all.pycalls macOSlinux, so aMac runs Linux-only checks and skips nothing for its own reasons, and the summary says
linux. This PR makes the runnerknow macOS as its own system, part of the macOS port (#420). What is measured: on an Apple M4 Max with macOS 26.7.1,
the declaration tests fail against
main's runner (9 failures and 1 error in 27 tests) and pass against this PR's (27 of27), two interleaved runs each. Every one of the 556 checks keeps its Linux and Windows arguments, modules and skips. Two
fixtures fail to compile on
mainand pass here, arm64 and x86_64. And the whole suite is compared by test name withmain. Not measured: Windows and Linux runs, macOS 14.4, an Intel Mac, a game.Why
On a Mac,
run_all.pysetsOS = "linux". Checks that cannot run on a Mac (Windows setup, the Linux AppImage bootstrap,ELF symbols) run and fail there, and a test file cannot declare a macOS reason for a skip.
What changed
tests/run_all.py:sys.platform == "darwin"ismacos, andRUN_ALL_SKIPaccepts amacoskey next tolinuxandwindows;macos, and macOS gets its own timing entry and shard estimates (no timing data is added);pkg-config --variable=prefix dav1d, after the existing roots.structured exceptions), and
test_shader_module_guard, whichmainadded since (structured exceptions as well);nx2/test_bc_storage_views,nx2/test_dll,nx2/test_insn,nx2/test_leaf,tests/test_present_aspect_vulkan);tests/test_appimage_package_retry.py(the Linux AppImage's apt/PPA bootstrap with GNUtimeout) andtests/test_prof_symbols.py(Linux ELF,link.h,dl_iterate_phdr). These two test Linux subjects, not macOS ones.files, memory gate, startup notice).
#if defined(__APPLE__)only, with every assertion kept:tests/test_collect_wide.pyandtests/test_oot_state_tap.py: undo the SDK's fortifiedmemcpyorvsnprintfmacrobefore the fixture defines its own counting one;
tests/test_mirror_trim.cpp: callsmirror_trim::wait, not the SDK'swait(int *).tests/test_dav1d_av1.py: on macOS, the found prefix'slibgoes on the library path and rpath.tests/test_run_all_declarations.py,tests/test_run_all_schedule.py: the runner's own functions on each system:detection, tool choices, declarations, skip overrides (
--no-skip, a named check,--strict), the summary and thestep summary, and per-system timing and shards. One case lists the 18 macOS exclusions exactly.
docs/development.md: running the game-free checks on macOS 14.4 or newer.host/orgpu/, no build script, nothing inthird_party/, no Windows or Linux timingrecord. No check was removed or weakened.
Flags and settings
None.
RUN_ALL_SKIP's newmacoskey is test metadata.--jobs,--strict,--no-skip, named selection,--shardand--write-timeswork on macOS as elsewhere.Measurements
main:
5e072fc8c2adadba5ea610148be1c47800ba0cca(2026-11-14)this PR:
mac/run-all@9ebd6ae863a70c9123c5a33904880318c24d162aHardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1 (25G241); Apple clang 21.0.0, Python 3.14.10. No GPU.
Scene: none. Game-free checks only.
Method: this PR's
tests/test_run_all_declarations.pyrun againstmain'srun_all.pyand tree, then against thisPR's (a driver rebinds the module and the root before the tests load), interleaved, two runs each. Each check's
declarations read through both runners. The collection and mirror fixtures compiled with
-arch arm64and-arch x86_64from both trees, and run (x86_64 witharch -x86_64). The whole suite on this PR, compared by test namewith a
mainrun on the same Mac.How to reproduce:
python -I -B tests/test_run_all_declarations.py,python -I -B tests/test_run_all_schedule.py, thenpython -I -B tests/run_all.py --jobs 6.5e072fc89ebd6ae8tests=27 failures=9 errors=1/ the sametests=27 failures=0 errors=0/ the sametest_collect_widefixture, arm64 / x86_64'memcpy' macro redefined/ the sametest_mirror_trim.cpp, arm64 / x86_64call to 'wait' is ambiguous/ the samemirror_trim: ok/ the same(linux, 6 at once)(macos, 6 at once)Testing
Tested: the above, and the whole suite,
tests/run_all.py, 6 at once, on the same Mac:By test name, five checks change, as intended:
test_collect_wideandtest_mirror_trim: FAIL → pass (the Apple fixture repairs);test_dav1d_av1: skip → pass (dav1d found throughpkg-config);test_appimage_package_retryandtest_prof_symbols: FAIL → skip, with their macOS reasons.No check fails here that passes on
main.test_shader_module_guardis skipped on both, now for its macOS reason.The 15 remaining failures also fail on
main:test_control,test_framegen_resources,test_game_layout,test_low_memory_notice,test_mem_gate,test_mem_gate_children,test_module_ids,test_oot_state_tap,test_pad_layout,test_pcache_header_read,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_program_cache_cleanup,test_program_cache_one_variant. The port's other PRs fixall but the three below that are stale on
main.Crash reports: eight from
MTLCompilerServiceappeared during this run. No game-free check uses Metal; otherwork on this Mac was compiling Metal shaders at the time [inference].
Main's other Mac failures, classified (none is this PR's to fix):
main, on every platform:test_module_ids: its source markerbool complete_from(Program& p, FILE* in, std::mutex* mu) {no longer matchesvk_progcache.cpp, whose function gained abool* busyargument. A text search, the same everywhere.test_framegen_resources: the cutmake_pipeusesIntelDriverGateandextern_guarded_shader_module(#252),which the fixture template does not define;
test_pad_layout: the cuthid.ctable usesHoldToggle, whoseheader the fixture does not include. Neither use sits under a platform guard [inference: they fail on Windows and
Linux too; not run there].
test_game_layout: compileswin32_posix.c, which needs the port's macOS host PRs.test_oot_state_tap: past thevsnprintfmacro (fixed here), it stops athost/control.c's Linux socket calls,which the control-bridge PR ports. With both PRs it passes on this Mac (checked on a scratch tree, not on a branch).
Not tested:
outside
#if defined(__APPLE__)orsys.platform == "darwin".Needs other testers:
main(request to open).main(request to open).
Screenshots
Nothing on screen.
Risk and rollback
edit there.
linuxskip now runs on a Mac. Before this PR a Mac took it as Linux and skipped it.--shardruns less balanced until times are written.Reviewer notes
tests/run_all.py's five lines, then the two runner tests.9ebd6ae8is new since the first version (2b992d6f):mainaddedtest_shader_module_guardandtest_oot_state_tapafter the port's base, so the first gets its macOS key and the second its Apple macro repair.
6e666d70and9ebd6ae8: the whole branch is this PR's own (it depends on no other PR).Issues
See #420.
Checklist
For everyone
type(scope): description).main. (Branched from5e072fc8.)python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above). (528 passed,18 skipped, and 15 failures that also fail on
main, named under Testing.)requests are still to open.
testchange: the test's output onmainand here, two interleaved runs each.)third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.