perf(tests): cache the checks' compiles by what they compile #893
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!893
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:perf/check-compile-cache"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: low · Tested on: Windows 11, llvm-mingw (v2.0.14 tools); Linux, Ubuntu 24.04 (Docker), gcc 13.3 and clang 18.1 · Needs: none
Why
Many game-free checks compile a small program from the source every time they run, in every worktree, even when nothing they read has changed. Those compiles were a third of a full run's check time on Windows, and they come back on every PR's run. The build already keeps objects by what they compile (
nx2/object_store.py); the checks now do the same.What changed
tests/run_all.pystarts each check through a small boot script that sends the check's compiler calls (and run_all's own C++ checks) through a compile cache. A compile is keyed by what it compiles, not by its paths, so a second run, or a run in another worktree, copies the program instead of compiling it.LONGSHOT_CHECK_CACHE=0turns it off.tests/check_cache.py(new):key_of/plankey a compile by the compiler's real path, size and time, its flags (the paths of its inputs, output and include folders left out), each source as-E -Ppreprocesses it (with the source's own path, asassert's__FILE__spells it, left out, so a fixture cut into a new temporary folder each run still hits) and each other input's bytes.compile_cachedcopies a hit's program and replays what the compiler printed; it keeps only compiles that succeed, writing each entry whole (os.replace). A compile it can't key that way goes to the compiler untouched:-x,-E,-M*,-save-temps, profiling, coverage,@file, stdin,-L,-B, a sysroot, a linker named by its path, or a second output (an import library, map, PDB,-gsplit-dwarf). On Windows,-o fixturewritesfixture.exe:writtenkeeps and gives back that file.prunedrops entries unused for 14 days and the oldest over 4 GB.installpatchessubprocess.Popenin the check's process for a list command whose first word is a C or C++ compiler.tests/check_cache_boot.py(new): runs a check aspython -I -B <check>would (runpy, the samesys.path), aftercheck_cache.install().tests/run_all.py:python_checkstarts the check through the boot script andcpp_checkscompiles throughcached(), both only whilecache_root()is on (and the two files are there:test_run_all_output.pycopiesrun_all.pyalone).mainprunes the cache and ends with acompile cache: H hits, M misses, P passed throughline before the summary.tests/test_check_cache.py(new): a hit gives the same program; a program named without an extension hits; a changed header or flag misses; a source in a new folder hits; a failed compile isn't kept; warnings are replayed on a hit; what it can't key passes through;=0turns it off; a check started through the boot script has its compiles cached.run_all.py's docstring say what the cache does and how to move or turn it off.Measurements
Verdict: a full warm run takes 36% less check time on Windows (2970 s → 1886 s) and 74% less on Linux (11543 s → 2948 s; wall 564 s → 205 s), with most compiles from the cache (425 of 434 on Windows, 1083 of 1098 on Linux) and the same failures as without it. On Windows wall time fell less (308 s → 228-246 s) because one check,
test_seed_reader, runs its sanitized program for about 245 s and is now the run's long pole. A cold cache costs a little over no cache (Linux: 11543 s vs 10852 s, wall 564 s vs 539 s), the preprocessing for the key.Hardware: Windows 11 Pro, the PC that runs the bench (other sessions' checks ran at the same time, so the times are noisy); Linux: a Docker container on the same PC (Ubuntu 24.04.5, gcc 13.3.0, clang 18.1.3, Python 3.12.3, 24 cores, nothing else running), set up as CI's
pythonjob is (its packages, pinned Python packages, Vulkan and SPIR-V headers throughscripts/fetch_commit.sh, glslang 16.6.0), the tree copied in as a clean checkout, the checks run as a user that is not rootScene: none (game-free checks)
Method:
python -I -B tests/run_all.py --jobs 12(Windows) or--jobs 24(Linux) on this branch, withLONGSHOT_CHECK_CACHEset to an empty folder: a cold run, then a warm run (Windows also a run to fill the cache between them); on Linux also one withLONGSHOT_CHECK_CACHE=0. Check time is the sum of each check's own time from run_all's lines; wall time is run_all's summary. One run each, so the times are a guide, not a median.How to reproduce:
LONGSHOT_CHECK_CACHE=<an empty folder>, thenpython -I -B tests/run_all.pytwice; the second run'scompile cache:line.e36ebb6ee36ebb6eLinux, on
main@02903b04with this PR @c3275d77, in the container above:c3275d77c3275d77=0), this PR @c3275d77Baseline: a cold cache rather than main, so both columns run the same checks; a cold run compiles every program, as main does, plus the preprocessing for the key. The cache-off column is main's way of compiling.
The Linux cold column is the second of two cold runs. The first (
/cache-a, the folder the warm run then used: 12580 s, 594 s, 76 hits, 1022 misses, 79 passed through) failed one check more,test_flash_sampling: its fixture program ran past its own 40 s timeout under the run's load. That program takes 3 s alone, from a cold cache, a warm one and with the cache off, and passed in the other three full runs (25-68 s for the check), so it is the load, not the cache.Checks that gained the most (cold → warm): Windows
test_system_header_cache53.7 → 13.4 s,test_pipeline_cache_file40.5 → 9.1 s,test_upscaler_round_trip36.7 → 11.6 s,test_split_giants43.1 → 18.0 s,test_vk_occlusion_queries22.9 → 4.6 s; Linuxtest_recorder_sync564 → 205 s,test_fpx_sticky541 → 184 s,nx2/test_draw_parameters360 → 29 s,test_mesh_unreachable_outputs355 → 26 s,nx2/test_spirv_struct_identity353 → 25 s.Flags and settings
One new environment variable for the test runner; nothing in the game or the build changes.
LONGSHOT_CHECK_CACHEtests/run_all.pykeeps the checks' compiles%LOCALAPPDATA%\Longshot\check-cache, else$XDG_CACHE_HOME/longshot-checksor~/.cache/longshot-checks;=<dir>moves it,=0turns it off0to rule the cache out when a check behaves oddlyTesting
Tested:
tests/run_all.pyon this branch (cold, fill, warm), each 513 passed, 19 failed, 81 skipped. The same 19 fail withLONGSHOT_CHECK_CACHE=0: the 16 that fail onmain@1310ecc6(the baseline), andtest_auto_handheld,test_crt_optionsandtest_vk_occlusion_queries, which fail on this branch's main with the cache off too.main@02903b04, the checks that read this diff:tests/run_all.py check_cache run_all byte_compile utf8_paths(8 checks): 7 pass (test_check_cache,test_run_all_output,test_run_all_junit,test_run_all_declarations,test_run_all_schedule,test_utf8_paths, the byte-compile);test_run_all_changedfails, as it does onmain@02903b04(it expectstests/test_team_chat.py, which moved to MagicBeanDev/Tooling).pythonjob): four full runs oftests/run_all.py --jobs 24on this PR @c3275d77(Measurements): cold, warm (1083 of 1098 compiles from the cache) andLONGSHOT_CHECK_CACHE=0each 628 passed, 20 failed, 17 skipped, the same 20 checks in each; a first cold run failedtest_flash_samplingtoo, on its fixture's 40 s run timeout under load (it passes alone, cold, warm and off). The 20 fail with the cache off as well.Not tested: CI's Linux run, which starts with a cold cache each time (the container's cold run took 5% longer than with the cache off).
Must be tested: None
Risks
What could break: a check whose result depends on something its compile reads that the key leaves out would pass on an old program. The key covers the preprocessed source, flags, compiler and input files, and refuses library folders,
-B, sysroots and linker paths. What it leaves out on purpose is the source file's own path: a hit's program names the path of the run that made it, in anassertmessage only. A check that compares its compiler's timing or counts its compiler starts would see fewer.Turning it off:
LONGSHOT_CHECK_CACHE=0, or revert the commit.Compatibility: Nothing changes for a player. Developers get a cache folder of up to 4 GB (a full run's compiles came to 48 MB on Windows), pruned by each run.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.🤖 Generated with Claude Code
WIP: perf(tests): cache the checks' compiles by what they compileto perf(tests): cache the checks' compiles by what they compilee36ebb6e23c3275d775bRebased onto main @
02903b04(headc3275d77). Linux runs in an Ubuntu 24.04 container (gcc 13.3, clang 18.1, 24 cores): cold 11543 s / 564 s, warm 2948 s / 205 s with 1083 of 1098 compiles from the cache, cache off 10852 s / 539 s; the same 20 checks fail in all three. Measurements and Testing updated; nothing left to test.Review of
c3275d77(base02903b04, the parent of your commit).Thanks, this is a useful change and it does what the description says. I read the diff, the description and the thread (one comment from you, no other review), and ran the whole suite 14 times (the base, and this head cold, warm and with the cache off), and tried to make the key serve a stale program. I would fix points 1 and 2 before it merges (neither breaks anything in the suite today, both are small); point 3 is a decision about what the description promises.
What I ran
Linux (Arch, i9-13900K with 32 threads, clang 23.1.3, gcc 16.2.1, Python 3.12.16),
python -I -B tests/run_all.py --jobs 6 --strict, a fresh HOME and an explicit cache folder. The PC was shared with other work, so each run is listed with its wall time, the CPU seconds of everything it started, and the load when it began.=0)Wall is
run_all.py's own "in N s"; CPU is the user and system seconds of everything the run started. Run 1 was made while other work kept the load at 40 to 47, so it says little; the rest are comparable only with the load in mind.tests/test_check_cache.pypasses. The 17 skips are the same.tests/test_recorder_sync.py(87 to 97 s warm).What I tried to break
All of these behaved:
-I,-isystem,-includeand-imacros: each recompiles and the program shows the new value; a changed-D, a changed object file input, an edited compiler script: all miss;run_all.pyon a copy with your warm cache: a comment-only edit ofgpu/vk/vk_present_rect.hhits (same code, as designed), a rounding change in it misses andpresent_rect_test.cppfollows the edit, an#erroradded to a header reached through-Imakeslower_test.cppfail with the compiler's message (a stale hit would have passed), reverting gives hits again;prune: an entry unused for 15 days goes, a half-written folder older than an hour goes, a newer one stays, over the cap the oldest go first;x86_64-w64-mingw32-clang,LONGSHOT_MINGWset): 6 pass cold, warm and with the cache off; 18 compiles cached.Please change
1. The key does not tell compilers apart that are the same file (
tests/check_cache.py:84-88). It takes the resolved path, size and time of the compiler, so a name that is a symlink is lost. In llvm-mingw the per-target compilers are all symlinks to one script that picks its target from the name it is started as:With real headers the three miss, because the preprocessed text differs per target, and the Windows-target checks only use the x86-64 wrapper, so nothing in the suite breaks today. It breaks the first time one does not. The same collapse covers
clangandclang++(one file) and the ccache links that.forgejo/workflows/ci.yml:106-108puts first in PATH (all six names resolve to the ccache binary, so the key names ccache and not the compiler). Addingos.path.basename(real)(the name as it was started) to the key fixes the wrapper case and costs nothing; I tried it and your test and my probes pass.2. A cache folder that cannot be listed stops the whole run (
tests/check_cache.py:269and:273, called fromtests/run_all.py:591-592).prune()callsos.listdiroutside anytry, and the runner calls it before the first check. With a folder of mode 000 (left by a run as root, another user, a restrictive mount)run_all.pyends withPermissionError: [Errno 13]and a traceback, no check runs. Every other bad folder I tried only turned the cache off. CatchingOSErrorinprune()(and treating the folder as empty) is enough; I tried that too.3. What the description says about CI. It says these compiles "come back on every PR's run". Nothing in this PR makes CI keep the folder, and the CI I can see would not use it anyway: the Python checks run by hand only ("off for now",
.forgejo/workflows/ci.yml:61, which is why their statuses on this PR say skipped), when they do run they already go through ccache with a cache restored from main (:99-123), and there is no Windows runner (:8-9), so the Windows numbers are a gain for people running the suite on their own PC, not for CI. A cold cache costs a little (see Cold above), so if someone does run the suite in CI without the folder kept, it is a little slower. Either say in the description that it is for local runs, or keep the folder the way ccache's is kept (restore in pull requests, save on a push to main, keyed by the compiler versions).Smaller
LONGSHOT_CHECK_CACHE=cache(a relative path) is used as given (tests/check_cache.py:61-62), so each compile resolves it against its own folder: two caches (miss miss) and acache/folder in the checkout.os.path.abspathinroot()fixes it.Command '[python3, -I, -B, .../check_cache.py, --cache, ..., --, clang++, ...instead of the compile, plus four extra traceback frames (tests/check_cache.py:253-257,tests/check_cache_boot.py:19-20). Settingself.argsback to the original aftersuper().__init__fixes it.os.utime(entry)fails (a cache on a read-only mount, or owned by someone else) the finished hit is compiled again and the warning is printed twice (tests/check_cache.py:212-216). It needs its owntry.-llibraries (rebuildinglibfoo.afound throughLIBRARY_PATHstill hits and links the old one), a second output other than the ones on the list (-Xlinker -Map=xhits and leaves no map;-Wl,-Mapis refused), a C++ source named.C(keyed by its bytes, so a header edit still hits),CCC_ADD_ARGS, the output's own name (-shared -o b.dllafter-o a.dllhits andb.dllnames itselfa.dll), and line numbers the front end writes itself (__builtin_LINE()printed 2 after three comment lines were added on top; a fresh compile prints 5). Refusing what it cannot see, instead of listing what it can, would close all of them; at least the docstring could list them.prunehas no test, and an entry is trusted as found. If a directory renamed in without a flush can survive a power loss with a short program (I did not reproduce this), every hit serves it until it is 14 days old; a size inmeta.jsonchecked on a hit would cover it.Co-Authored-Bybut noAssisted-byline (CONTRIBUTING.md asks for both). CONTRIBUTING.md does not say where the folder is, that it keeps 14 days and 4 GB, or thatoff,falseandnoalso turn it off; the docstring does.I have a patch for points 1 and 2 and the first three of the smaller ones (about 20 lines; your test passes with it, and so do my probes). Say if you want it as a suggestion.
Not tested
Windows (not run for this review, so the 36% and
os.replaceof a directory under a virus scanner are untested), CI (the suite has not run in CI on this head), a cache shared by two users, a full disk, a power loss, and the head merged into main at71b1edd8(21 commits later; it merges cleanly, I did not run it).View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.