MagicBeanDev/Longshot

Watch

Fork 102

perf(tests): cache the checks' compiles by what they compile #893

Open

dyst wants to merge 1 commit from dyst/Longshot:perf/check-compile-cache into main

dyst commented

2026-11-15 19:31:44 +00:00

Member

Type: perf · Risk: low · Tested on: Windows 11, llvm-mingw (v2.0.14 tools) · Needs: Linux, a run of tests/run_all.py twice

Why

Many game-free checks compile a small program from the source every time they run, in every worktree, even when nothing they read has changed. Those compiles were a third of a full run's check time on Windows, and they come back on every PR's run. The build already keeps objects by what they compile (nx2/object_store.py); the checks now do the same.

What changed

tests/run_all.py starts each check through a small boot script that sends the check's compiler calls (and run_all's own C++ checks) through a compile cache. A compile is keyed by what it compiles, not by its paths, so a second run, or a run in another worktree, copies the program instead of compiling it. LONGSHOT_CHECK_CACHE=0 turns it off.

tests: tests/check_cache.py (new): key_of/plan key a compile by the compiler's real path, size and time, its flags (the paths of its inputs, output and include folders left out), each source as -E -P preprocesses it (with the source's own path, as assert's __FILE__ spells it, left out, so a fixture cut into a new temporary folder each run still hits) and each other input's bytes. compile_cached copies a hit's program and replays what the compiler printed; it keeps only compiles that succeed, writing each entry whole (os.replace). A compile it can't key that way goes to the compiler untouched: -x, -E, -M*, -save-temps, profiling, coverage, @file, stdin, -L, -B, a sysroot, a linker named by its path, or a second output (an import library, map, PDB, -gsplit-dwarf). On Windows, -o fixture writes fixture.exe: written keeps and gives back that file. prune drops entries unused for 14 days and the oldest over 4 GB. install patches subprocess.Popen in the check's process for a list command whose first word is a C or C++ compiler.
tests: tests/check_cache_boot.py (new): runs a check as python -I -B <check> would (runpy, the same sys.path), after check_cache.install().
tests: tests/run_all.py: python_check starts the check through the boot script and cpp_checks compiles through cached(), both only while cache_root() is on (and the two files are there: test_run_all_output.py copies run_all.py alone). main prunes the cache and ends with a compile cache: H hits, M misses, P passed through line before the summary.
tests: tests/test_check_cache.py (new): a hit gives the same program; a program named without an extension hits; a changed header or flag misses; a source in a new folder hits; a failed compile isn't kept; warnings are replayed on a hit; what it can't key passes through; =0 turns it off; a check started through the boot script has its compiles cached.
docs: CONTRIBUTING.md, "Game-free checks", and run_all.py's docstring say what the cache does and how to move or turn it off.

Measurements

Verdict: a full warm run takes 36% less check time (2970 s → 1886 s), 425 of its 434 compiles from the cache, with the same failures as without it. Wall time fell less (308 s → 228-246 s) because one check, test_seed_reader, runs its sanitized program for about 245 s and is now the run's long pole.

Hardware: Windows 11 Pro, the PC that runs the bench (other sessions' checks ran at the same time, so the times are noisy)
Scene: none (game-free checks)
Method: python -I -B tests/run_all.py --jobs 12 on this branch, with LONGSHOT_CHECK_CACHE set to an empty folder: a cold run, a run to fill the cache, then a warm run. Check time is the sum of each check's own time from run_all's lines; wall time is run_all's summary. One run each, so the times are a guide, not a median.
How to reproduce: LONGSHOT_CHECK_CACHE=<an empty folder>, then python -I -B tests/run_all.py twice; the second run's compile cache: line.

	Before: cold cache, this PR @ `e36ebb6e`	After: warm cache, this PR @ `e36ebb6e`
Sum of check times	2970 s	1886 s
Wall time	308 s	246 s
Compiles from the cache	1 of 434	425 of 434
Checks that fail	19	the same 19

Baseline: a cold cache rather than main, so both columns run the same checks; a cold run compiles every program, as main does, plus the preprocessing for the key.

cold:  compile cache: 1 hits, 433 misses, 47 passed through
       513 passed, 19 failed, 81 skipped in 308 s (windows, 12 at once)
warm:  compile cache: 425 hits, 9 misses, 47 passed through
       513 passed, 19 failed, 81 skipped in 246 s (windows, 12 at once)

Checks that gained the most (cold → warm): test_system_header_cache 53.7 → 13.4 s, test_pipeline_cache_file 40.5 → 9.1 s, test_upscaler_round_trip 36.7 → 11.6 s, test_split_giants 43.1 → 18.0 s, test_vk_occlusion_queries 22.9 → 4.6 s.

Flags and settings

One new environment variable for the test runner; nothing in the game or the build changes.

Flag	What it is	Before this PR	After this PR	Effect	Helps / doesn't help / untested	Recommendation
`LONGSHOT_CHECK_CACHE`	Where `tests/run_all.py` keeps the checks' compiles	(none)	`%LOCALAPPDATA%\Longshot\check-cache`, else `$XDG_CACHE_HOME/longshot-checks` or `~/.cache/longshot-checks`; `=<dir>` moves it, `=0` turns it off	A compile done before, in any worktree, is copied instead of run	Helps on Windows (measured above); untested on Linux	On for everyone; `0` to rule the cache out when a check behaves oddly

Testing

Tested:

Windows 11, llvm-mingw: three full runs of tests/run_all.py on this branch (cold, fill, warm), each 513 passed, 19 failed, 81 skipped. The same 19 fail with LONGSHOT_CHECK_CACHE=0: the 16 that fail on main @ 1310ecc6 (the baseline), and test_auto_handheld, test_crt_options and test_vk_occlusion_queries, which fail on this branch's main with the cache off too.
After the rebase onto main @ b8d2b04e, the checks that read this diff: tests/run_all.py check_cache run_all byte_compile utf8_paths (7 checks, test_check_cache, test_run_all_output, test_run_all_junit, test_run_all_declarations, test_run_all_schedule, test_utf8_paths, the byte-compile), all pass.

Not tested: Linux (gcc and clang): the keying is the same code, but nobody has run it there; CI's Linux run with a cold cache each time.

Must be tested:

Linux, any compiler: python -I -B tests/run_all.py twice, expecting the second run's compile cache: line to show most compiles as hits, and the same failures as LONGSHOT_CHECK_CACHE=0 python -I -B tests/run_all.py (Test Farm: @Beanstalk linux, run the game-free checks twice and compare with the cache off)

Risks

What could break: a check whose result depends on something its compile reads that the key leaves out would pass on an old program. The key covers the preprocessed source, flags, compiler and input files, and refuses library folders, -B, sysroots and linker paths. What it leaves out on purpose is the source file's own path: a hit's program names the path of the run that made it, in an assert message only. A check that compares its compiler's timing or counts its compiler starts would see fewer.

Turning it off: LONGSHOT_CHECK_CACHE=0, or revert the commit.

Compatibility: Nothing changes for a player. Developers get a cache folder of up to 4 GB (a full run's compiles came to 48 MB on Windows), pruned by each run.

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main.
python -I -B tests/run_all.py passes (any failure that also fails on main is named).
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
Docs updated.

🤖 Generated with Claude Code

**Type:** perf · **Risk:** low · **Tested on:** Windows 11, llvm-mingw (v2.0.14 tools) · **Needs:** Linux, a run of `tests/run_all.py` twice ## Why Many game-free checks compile a small program from the source every time they run, in every worktree, even when nothing they read has changed. Those compiles were a third of a full run's check time on Windows, and they come back on every PR's run. The build already keeps objects by what they compile (`nx2/object_store.py`); the checks now do the same. ## What changed `tests/run_all.py` starts each check through a small boot script that sends the check's compiler calls (and run_all's own C++ checks) through a compile cache. A compile is keyed by what it compiles, not by its paths, so a second run, or a run in another worktree, copies the program instead of compiling it. `LONGSHOT_CHECK_CACHE=0` turns it off. - tests: `tests/check_cache.py` (new): `key_of`/`plan` key a compile by the compiler's real path, size and time, its flags (the paths of its inputs, output and include folders left out), each source as `-E -P` preprocesses it (with the source's own path, as `assert`'s `__FILE__` spells it, left out, so a fixture cut into a new temporary folder each run still hits) and each other input's bytes. `compile_cached` copies a hit's program and replays what the compiler printed; it keeps only compiles that succeed, writing each entry whole (`os.replace`). A compile it can't key that way goes to the compiler untouched: `-x`, `-E`, `-M*`, `-save-temps`, profiling, coverage, `@file`, stdin, `-L`, `-B`, a sysroot, a linker named by its path, or a second output (an import library, map, PDB, `-gsplit-dwarf`). On Windows, `-o fixture` writes `fixture.exe`: `written` keeps and gives back that file. `prune` drops entries unused for 14 days and the oldest over 4 GB. `install` patches `subprocess.Popen` in the check's process for a list command whose first word is a C or C++ compiler. - tests: `tests/check_cache_boot.py` (new): runs a check as `python -I -B <check>` would (`runpy`, the same `sys.path`), after `check_cache.install()`. - tests: `tests/run_all.py`: `python_check` starts the check through the boot script and `cpp_checks` compiles through `cached()`, both only while `cache_root()` is on (and the two files are there: `test_run_all_output.py` copies `run_all.py` alone). `main` prunes the cache and ends with a `compile cache: H hits, M misses, P passed through` line before the summary. - tests: `tests/test_check_cache.py` (new): a hit gives the same program; a program named without an extension hits; a changed header or flag misses; a source in a new folder hits; a failed compile isn't kept; warnings are replayed on a hit; what it can't key passes through; `=0` turns it off; a check started through the boot script has its compiles cached. - docs: CONTRIBUTING.md, "Game-free checks", and `run_all.py`'s docstring say what the cache does and how to move or turn it off. ## Measurements **Verdict:** a full warm run takes 36% less check time (2970 s → 1886 s), 425 of its 434 compiles from the cache, with the same failures as without it. Wall time fell less (308 s → 228-246 s) because one check, `test_seed_reader`, runs its sanitized program for about 245 s and is now the run's long pole. Hardware: Windows 11 Pro, the PC that runs the bench (other sessions' checks ran at the same time, so the times are noisy) Scene: none (game-free checks) Method: `python -I -B tests/run_all.py --jobs 12` on this branch, with `LONGSHOT_CHECK_CACHE` set to an empty folder: a cold run, a run to fill the cache, then a warm run. Check time is the sum of each check's own time from run_all's lines; wall time is run_all's summary. One run each, so the times are a guide, not a median. How to reproduce: `LONGSHOT_CHECK_CACHE=<an empty folder>`, then `python -I -B tests/run_all.py` twice; the second run's `compile cache:` line. | | Before: cold cache, this PR @ e36ebb6e | After: warm cache, this PR @ e36ebb6e | |---|---|---| | Sum of check times | 2970 s | 1886 s | | Wall time | 308 s | 246 s | | Compiles from the cache | 1 of 434 | 425 of 434 | | Checks that fail | 19 | the same 19 | Baseline: a cold cache rather than main, so both columns run the same checks; a cold run compiles every program, as main does, plus the preprocessing for the key. ``` cold: compile cache: 1 hits, 433 misses, 47 passed through 513 passed, 19 failed, 81 skipped in 308 s (windows, 12 at once) warm: compile cache: 425 hits, 9 misses, 47 passed through 513 passed, 19 failed, 81 skipped in 246 s (windows, 12 at once) ``` Checks that gained the most (cold → warm): `test_system_header_cache` 53.7 → 13.4 s, `test_pipeline_cache_file` 40.5 → 9.1 s, `test_upscaler_round_trip` 36.7 → 11.6 s, `test_split_giants` 43.1 → 18.0 s, `test_vk_occlusion_queries` 22.9 → 4.6 s. ### Flags and settings One new environment variable for the test runner; nothing in the game or the build changes. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `LONGSHOT_CHECK_CACHE` | Where `tests/run_all.py` keeps the checks' compiles | (none) | `%LOCALAPPDATA%\Longshot\check-cache`, else `$XDG_CACHE_HOME/longshot-checks` or `~/.cache/longshot-checks`; `=<dir>` moves it, `=0` turns it off | A compile done before, in any worktree, is copied instead of run | Helps on Windows (measured above); untested on Linux | On for everyone; `0` to rule the cache out when a check behaves oddly | ## Testing **Tested:** - Windows 11, llvm-mingw: three full runs of `tests/run_all.py` on this branch (cold, fill, warm), each 513 passed, 19 failed, 81 skipped. The same 19 fail with `LONGSHOT_CHECK_CACHE=0`: the 16 that fail on `main` @ 1310ecc6 (the baseline), and `test_auto_handheld`, `test_crt_options` and `test_vk_occlusion_queries`, which fail on this branch's main with the cache off too. - After the rebase onto `main` @ b8d2b04e, the checks that read this diff: `tests/run_all.py check_cache run_all byte_compile utf8_paths` (7 checks, `test_check_cache`, `test_run_all_output`, `test_run_all_junit`, `test_run_all_declarations`, `test_run_all_schedule`, `test_utf8_paths`, the byte-compile), all pass. **Not tested:** Linux (gcc and clang): the keying is the same code, but nobody has run it there; CI's Linux run with a cold cache each time. **Must be tested:** - Linux, any compiler: `python -I -B tests/run_all.py` twice, expecting the second run's `compile cache:` line to show most compiles as hits, and the same failures as `LONGSHOT_CHECK_CACHE=0 python -I -B tests/run_all.py` (Test Farm: @Beanstalk linux, run the game-free checks twice and compare with the cache off) ## Risks **What could break:** a check whose result depends on something its compile reads that the key leaves out would pass on an old program. The key covers the preprocessed source, flags, compiler and input files, and refuses library folders, `-B`, sysroots and linker paths. What it leaves out on purpose is the source file's own path: a hit's program names the path of the run that made it, in an `assert` message only. A check that compares its compiler's timing or counts its compiler starts would see fewer. **Turning it off:** `LONGSHOT_CHECK_CACHE=0`, or revert the commit. **Compatibility:** Nothing changes for a player. Developers get a cache folder of up to 4 GB (a full run's compiles came to 48 MB on Windows), pruned by each run. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit

2026-11-15 19:31:44 +00:00

perf(tests): cache the checks' compiles by what they compile

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 4s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Successful in 7s

Details

e36ebb6e23

Many checks compile a small program from the source on every run, in
every worktree. run_all.py now starts each check through
tests/check_cache_boot.py, which routes its compiler calls (and the C++
checks' own) through tests/check_cache.py: a compile keyed by its
preprocessed source, flags and compiler, kept in one folder every
worktree on the PC shares. LONGSHOT_CHECK_CACHE=0 turns it off.

Windows, full run: 2970 s of check time cold, 1886 s warm (425 of 434
compiles from the cache), the same 19 failures as with it off.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

BeanIssues added the

kind

performance

label

2026-11-15 19:31:51 +00:00

dyst changed title from ~~WIP: perf(tests): cache the checks' compiles by what they compile~~ to perf(tests): cache the checks' compiles by what they compile

2026-11-15 20:12:47 +00:00

BeanIssues added the

risk

low

label

2026-11-15 20:27:10 +00:00