perf(tests): cache the checks' compiles by what they compile #893

Open
dyst wants to merge 1 commit from dyst/Longshot:perf/check-compile-cache into main
Member

Type: perf · Risk: low · Tested on: Windows 11, llvm-mingw (v2.0.14 tools); Linux, Ubuntu 24.04 (Docker), gcc 13.3 and clang 18.1 · Needs: none

Why

Many game-free checks compile a small program from the source every time they run, in every worktree, even when nothing they read has changed. Those compiles were a third of a full run's check time on Windows, and they come back on every PR's run. The build already keeps objects by what they compile (nx2/object_store.py); the checks now do the same.

What changed

tests/run_all.py starts each check through a small boot script that sends the check's compiler calls (and run_all's own C++ checks) through a compile cache. A compile is keyed by what it compiles, not by its paths, so a second run, or a run in another worktree, copies the program instead of compiling it. LONGSHOT_CHECK_CACHE=0 turns it off.

  • tests: tests/check_cache.py (new): key_of/plan key a compile by the compiler's real path, size and time, its flags (the paths of its inputs, output and include folders left out), each source as -E -P preprocesses it (with the source's own path, as assert's __FILE__ spells it, left out, so a fixture cut into a new temporary folder each run still hits) and each other input's bytes. compile_cached copies a hit's program and replays what the compiler printed; it keeps only compiles that succeed, writing each entry whole (os.replace). A compile it can't key that way goes to the compiler untouched: -x, -E, -M*, -save-temps, profiling, coverage, @file, stdin, -L, -B, a sysroot, a linker named by its path, or a second output (an import library, map, PDB, -gsplit-dwarf). On Windows, -o fixture writes fixture.exe: written keeps and gives back that file. prune drops entries unused for 14 days and the oldest over 4 GB. install patches subprocess.Popen in the check's process for a list command whose first word is a C or C++ compiler.
  • tests: tests/check_cache_boot.py (new): runs a check as python -I -B <check> would (runpy, the same sys.path), after check_cache.install().
  • tests: tests/run_all.py: python_check starts the check through the boot script and cpp_checks compiles through cached(), both only while cache_root() is on (and the two files are there: test_run_all_output.py copies run_all.py alone). main prunes the cache and ends with a compile cache: H hits, M misses, P passed through line before the summary.
  • tests: tests/test_check_cache.py (new): a hit gives the same program; a program named without an extension hits; a changed header or flag misses; a source in a new folder hits; a failed compile isn't kept; warnings are replayed on a hit; what it can't key passes through; =0 turns it off; a check started through the boot script has its compiles cached.
  • docs: CONTRIBUTING.md, "Game-free checks", and run_all.py's docstring say what the cache does and how to move or turn it off.

Measurements

Verdict: a full warm run takes 36% less check time on Windows (2970 s → 1886 s) and 74% less on Linux (11543 s → 2948 s; wall 564 s → 205 s), with most compiles from the cache (425 of 434 on Windows, 1083 of 1098 on Linux) and the same failures as without it. On Windows wall time fell less (308 s → 228-246 s) because one check, test_seed_reader, runs its sanitized program for about 245 s and is now the run's long pole. A cold cache costs a little over no cache (Linux: 11543 s vs 10852 s, wall 564 s vs 539 s), the preprocessing for the key.

Hardware: Windows 11 Pro, the PC that runs the bench (other sessions' checks ran at the same time, so the times are noisy); Linux: a Docker container on the same PC (Ubuntu 24.04.5, gcc 13.3.0, clang 18.1.3, Python 3.12.3, 24 cores, nothing else running), set up as CI's python job is (its packages, pinned Python packages, Vulkan and SPIR-V headers through scripts/fetch_commit.sh, glslang 16.6.0), the tree copied in as a clean checkout, the checks run as a user that is not root
Scene: none (game-free checks)
Method: python -I -B tests/run_all.py --jobs 12 (Windows) or --jobs 24 (Linux) on this branch, with LONGSHOT_CHECK_CACHE set to an empty folder: a cold run, then a warm run (Windows also a run to fill the cache between them); on Linux also one with LONGSHOT_CHECK_CACHE=0. Check time is the sum of each check's own time from run_all's lines; wall time is run_all's summary. One run each, so the times are a guide, not a median.
How to reproduce: LONGSHOT_CHECK_CACHE=<an empty folder>, then python -I -B tests/run_all.py twice; the second run's compile cache: line.

Before: cold cache, this PR @ e36ebb6e After: warm cache, this PR @ e36ebb6e
Sum of check times 2970 s 1886 s
Wall time 308 s 246 s
Compiles from the cache 1 of 434 425 of 434
Checks that fail 19 the same 19

Linux, on main @ 02903b04 with this PR @ c3275d77, in the container above:

Before: cold cache, this PR @ c3275d77 After: warm cache, this PR @ c3275d77 Cache off (=0), this PR @ c3275d77
Sum of check times 11543 s 2948 s 10852 s
Wall time 564 s 205 s 539 s
Compiles from the cache 76 of 1098 1083 of 1098 (off)
Checks that fail 20 the same 20 the same 20

Baseline: a cold cache rather than main, so both columns run the same checks; a cold run compiles every program, as main does, plus the preprocessing for the key. The cache-off column is main's way of compiling.

Windows cold:  compile cache: 1 hits, 433 misses, 47 passed through
               513 passed, 19 failed, 81 skipped in 308 s (windows, 12 at once)
Windows warm:  compile cache: 425 hits, 9 misses, 47 passed through
               513 passed, 19 failed, 81 skipped in 246 s (windows, 12 at once)
Linux cold:    compile cache: 76 hits, 1022 misses, 78 passed through (/cache-b)
               628 passed, 20 failed, 17 skipped in 564 s (linux, 24 at once)
Linux warm:    compile cache: 1083 hits, 15 misses, 78 passed through (/cache-a)
               628 passed, 20 failed, 17 skipped in 205 s (linux, 24 at once)
Linux off:     628 passed, 20 failed, 17 skipped in 539 s (linux, 24 at once)

The Linux cold column is the second of two cold runs. The first (/cache-a, the folder the warm run then used: 12580 s, 594 s, 76 hits, 1022 misses, 79 passed through) failed one check more, test_flash_sampling: its fixture program ran past its own 40 s timeout under the run's load. That program takes 3 s alone, from a cold cache, a warm one and with the cache off, and passed in the other three full runs (25-68 s for the check), so it is the load, not the cache.

Checks that gained the most (cold → warm): Windows test_system_header_cache 53.7 → 13.4 s, test_pipeline_cache_file 40.5 → 9.1 s, test_upscaler_round_trip 36.7 → 11.6 s, test_split_giants 43.1 → 18.0 s, test_vk_occlusion_queries 22.9 → 4.6 s; Linux test_recorder_sync 564 → 205 s, test_fpx_sticky 541 → 184 s, nx2/test_draw_parameters 360 → 29 s, test_mesh_unreachable_outputs 355 → 26 s, nx2/test_spirv_struct_identity 353 → 25 s.

Flags and settings

One new environment variable for the test runner; nothing in the game or the build changes.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
LONGSHOT_CHECK_CACHE Where tests/run_all.py keeps the checks' compiles (none) %LOCALAPPDATA%\Longshot\check-cache, else $XDG_CACHE_HOME/longshot-checks or ~/.cache/longshot-checks; =<dir> moves it, =0 turns it off A compile done before, in any worktree, is copied instead of run Helps on Windows and Linux (measured above) On for everyone; 0 to rule the cache out when a check behaves oddly

Testing

Tested:

  • Windows 11, llvm-mingw: three full runs of tests/run_all.py on this branch (cold, fill, warm), each 513 passed, 19 failed, 81 skipped. The same 19 fail with LONGSHOT_CHECK_CACHE=0: the 16 that fail on main @ 1310ecc6 (the baseline), and test_auto_handheld, test_crt_options and test_vk_occlusion_queries, which fail on this branch's main with the cache off too.
  • After the rebase onto main @ 02903b04, the checks that read this diff: tests/run_all.py check_cache run_all byte_compile utf8_paths (8 checks): 7 pass (test_check_cache, test_run_all_output, test_run_all_junit, test_run_all_declarations, test_run_all_schedule, test_utf8_paths, the byte-compile); test_run_all_changed fails, as it does on main @ 02903b04 (it expects tests/test_team_chat.py, which moved to MagicBeanDev/Tooling).
  • Linux, Ubuntu 24.04 in Docker (gcc 13.3, clang 18.1, 24 cores, set up as CI's python job): four full runs of tests/run_all.py --jobs 24 on this PR @ c3275d77 (Measurements): cold, warm (1083 of 1098 compiles from the cache) and LONGSHOT_CHECK_CACHE=0 each 628 passed, 20 failed, 17 skipped, the same 20 checks in each; a first cold run failed test_flash_sampling too, on its fixture's 40 s run timeout under load (it passes alone, cold, warm and off). The 20 fail with the cache off as well.

Not tested: CI's Linux run, which starts with a cold cache each time (the container's cold run took 5% longer than with the cache off).

Must be tested: None

Risks

What could break: a check whose result depends on something its compile reads that the key leaves out would pass on an old program. The key covers the preprocessed source, flags, compiler and input files, and refuses library folders, -B, sysroots and linker paths. What it leaves out on purpose is the source file's own path: a hit's program names the path of the run that made it, in an assert message only. A check that compares its compiler's timing or counts its compiler starts would see fewer.

Turning it off: LONGSHOT_CHECK_CACHE=0, or revert the commit.

Compatibility: Nothing changes for a player. Developers get a cache folder of up to 4 GB (a full run's compiles came to 48 MB on Windows), pruned by each run.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** perf · **Risk:** low · **Tested on:** Windows 11, llvm-mingw (v2.0.14 tools); Linux, Ubuntu 24.04 (Docker), gcc 13.3 and clang 18.1 · **Needs:** none ## Why Many game-free checks compile a small program from the source every time they run, in every worktree, even when nothing they read has changed. Those compiles were a third of a full run's check time on Windows, and they come back on every PR's run. The build already keeps objects by what they compile (`nx2/object_store.py`); the checks now do the same. ## What changed `tests/run_all.py` starts each check through a small boot script that sends the check's compiler calls (and run_all's own C++ checks) through a compile cache. A compile is keyed by what it compiles, not by its paths, so a second run, or a run in another worktree, copies the program instead of compiling it. `LONGSHOT_CHECK_CACHE=0` turns it off. - tests: `tests/check_cache.py` (new): `key_of`/`plan` key a compile by the compiler's real path, size and time, its flags (the paths of its inputs, output and include folders left out), each source as `-E -P` preprocesses it (with the source's own path, as `assert`'s `__FILE__` spells it, left out, so a fixture cut into a new temporary folder each run still hits) and each other input's bytes. `compile_cached` copies a hit's program and replays what the compiler printed; it keeps only compiles that succeed, writing each entry whole (`os.replace`). A compile it can't key that way goes to the compiler untouched: `-x`, `-E`, `-M*`, `-save-temps`, profiling, coverage, `@file`, stdin, `-L`, `-B`, a sysroot, a linker named by its path, or a second output (an import library, map, PDB, `-gsplit-dwarf`). On Windows, `-o fixture` writes `fixture.exe`: `written` keeps and gives back that file. `prune` drops entries unused for 14 days and the oldest over 4 GB. `install` patches `subprocess.Popen` in the check's process for a list command whose first word is a C or C++ compiler. - tests: `tests/check_cache_boot.py` (new): runs a check as `python -I -B <check>` would (`runpy`, the same `sys.path`), after `check_cache.install()`. - tests: `tests/run_all.py`: `python_check` starts the check through the boot script and `cpp_checks` compiles through `cached()`, both only while `cache_root()` is on (and the two files are there: `test_run_all_output.py` copies `run_all.py` alone). `main` prunes the cache and ends with a `compile cache: H hits, M misses, P passed through` line before the summary. - tests: `tests/test_check_cache.py` (new): a hit gives the same program; a program named without an extension hits; a changed header or flag misses; a source in a new folder hits; a failed compile isn't kept; warnings are replayed on a hit; what it can't key passes through; `=0` turns it off; a check started through the boot script has its compiles cached. - docs: CONTRIBUTING.md, "Game-free checks", and `run_all.py`'s docstring say what the cache does and how to move or turn it off. ## Measurements **Verdict:** a full warm run takes 36% less check time on Windows (2970 s → 1886 s) and 74% less on Linux (11543 s → 2948 s; wall 564 s → 205 s), with most compiles from the cache (425 of 434 on Windows, 1083 of 1098 on Linux) and the same failures as without it. On Windows wall time fell less (308 s → 228-246 s) because one check, `test_seed_reader`, runs its sanitized program for about 245 s and is now the run's long pole. A cold cache costs a little over no cache (Linux: 11543 s vs 10852 s, wall 564 s vs 539 s), the preprocessing for the key. Hardware: Windows 11 Pro, the PC that runs the bench (other sessions' checks ran at the same time, so the times are noisy); Linux: a Docker container on the same PC (Ubuntu 24.04.5, gcc 13.3.0, clang 18.1.3, Python 3.12.3, 24 cores, nothing else running), set up as CI's `python` job is (its packages, pinned Python packages, Vulkan and SPIR-V headers through `scripts/fetch_commit.sh`, glslang 16.6.0), the tree copied in as a clean checkout, the checks run as a user that is not root Scene: none (game-free checks) Method: `python -I -B tests/run_all.py --jobs 12` (Windows) or `--jobs 24` (Linux) on this branch, with `LONGSHOT_CHECK_CACHE` set to an empty folder: a cold run, then a warm run (Windows also a run to fill the cache between them); on Linux also one with `LONGSHOT_CHECK_CACHE=0`. Check time is the sum of each check's own time from run_all's lines; wall time is run_all's summary. One run each, so the times are a guide, not a median. How to reproduce: `LONGSHOT_CHECK_CACHE=<an empty folder>`, then `python -I -B tests/run_all.py` twice; the second run's `compile cache:` line. | | Before: cold cache, this PR @ e36ebb6e | After: warm cache, this PR @ e36ebb6e | |---|---|---| | Sum of check times | 2970 s | 1886 s | | Wall time | 308 s | 246 s | | Compiles from the cache | 1 of 434 | 425 of 434 | | Checks that fail | 19 | the same 19 | Linux, on `main` @ 02903b04 with this PR @ c3275d77, in the container above: | | Before: cold cache, this PR @ c3275d77 | After: warm cache, this PR @ c3275d77 | Cache off (`=0`), this PR @ c3275d77 | |---|---|---|---| | Sum of check times | 11543 s | 2948 s | 10852 s | | Wall time | 564 s | 205 s | 539 s | | Compiles from the cache | 76 of 1098 | 1083 of 1098 | (off) | | Checks that fail | 20 | the same 20 | the same 20 | Baseline: a cold cache rather than main, so both columns run the same checks; a cold run compiles every program, as main does, plus the preprocessing for the key. The cache-off column is main's way of compiling. ``` Windows cold: compile cache: 1 hits, 433 misses, 47 passed through 513 passed, 19 failed, 81 skipped in 308 s (windows, 12 at once) Windows warm: compile cache: 425 hits, 9 misses, 47 passed through 513 passed, 19 failed, 81 skipped in 246 s (windows, 12 at once) Linux cold: compile cache: 76 hits, 1022 misses, 78 passed through (/cache-b) 628 passed, 20 failed, 17 skipped in 564 s (linux, 24 at once) Linux warm: compile cache: 1083 hits, 15 misses, 78 passed through (/cache-a) 628 passed, 20 failed, 17 skipped in 205 s (linux, 24 at once) Linux off: 628 passed, 20 failed, 17 skipped in 539 s (linux, 24 at once) ``` The Linux cold column is the second of two cold runs. The first (`/cache-a`, the folder the warm run then used: 12580 s, 594 s, 76 hits, 1022 misses, 79 passed through) failed one check more, `test_flash_sampling`: its fixture program ran past its own 40 s timeout under the run's load. That program takes 3 s alone, from a cold cache, a warm one and with the cache off, and passed in the other three full runs (25-68 s for the check), so it is the load, not the cache. Checks that gained the most (cold → warm): Windows `test_system_header_cache` 53.7 → 13.4 s, `test_pipeline_cache_file` 40.5 → 9.1 s, `test_upscaler_round_trip` 36.7 → 11.6 s, `test_split_giants` 43.1 → 18.0 s, `test_vk_occlusion_queries` 22.9 → 4.6 s; Linux `test_recorder_sync` 564 → 205 s, `test_fpx_sticky` 541 → 184 s, `nx2/test_draw_parameters` 360 → 29 s, `test_mesh_unreachable_outputs` 355 → 26 s, `nx2/test_spirv_struct_identity` 353 → 25 s. ### Flags and settings One new environment variable for the test runner; nothing in the game or the build changes. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `LONGSHOT_CHECK_CACHE` | Where `tests/run_all.py` keeps the checks' compiles | (none) | `%LOCALAPPDATA%\Longshot\check-cache`, else `$XDG_CACHE_HOME/longshot-checks` or `~/.cache/longshot-checks`; `=<dir>` moves it, `=0` turns it off | A compile done before, in any worktree, is copied instead of run | Helps on Windows and Linux (measured above) | On for everyone; `0` to rule the cache out when a check behaves oddly | ## Testing **Tested:** - Windows 11, llvm-mingw: three full runs of `tests/run_all.py` on this branch (cold, fill, warm), each 513 passed, 19 failed, 81 skipped. The same 19 fail with `LONGSHOT_CHECK_CACHE=0`: the 16 that fail on `main` @ 1310ecc6 (the baseline), and `test_auto_handheld`, `test_crt_options` and `test_vk_occlusion_queries`, which fail on this branch's main with the cache off too. - After the rebase onto `main` @ 02903b04, the checks that read this diff: `tests/run_all.py check_cache run_all byte_compile utf8_paths` (8 checks): 7 pass (`test_check_cache`, `test_run_all_output`, `test_run_all_junit`, `test_run_all_declarations`, `test_run_all_schedule`, `test_utf8_paths`, the byte-compile); `test_run_all_changed` fails, as it does on `main` @ 02903b04 (it expects `tests/test_team_chat.py`, which moved to MagicBeanDev/Tooling). - Linux, Ubuntu 24.04 in Docker (gcc 13.3, clang 18.1, 24 cores, set up as CI's `python` job): four full runs of `tests/run_all.py --jobs 24` on this PR @ c3275d77 (Measurements): cold, warm (1083 of 1098 compiles from the cache) and `LONGSHOT_CHECK_CACHE=0` each 628 passed, 20 failed, 17 skipped, the same 20 checks in each; a first cold run failed `test_flash_sampling` too, on its fixture's 40 s run timeout under load (it passes alone, cold, warm and off). The 20 fail with the cache off as well. **Not tested:** CI's Linux run, which starts with a cold cache each time (the container's cold run took 5% longer than with the cache off). **Must be tested:** None ## Risks **What could break:** a check whose result depends on something its compile reads that the key leaves out would pass on an old program. The key covers the preprocessed source, flags, compiler and input files, and refuses library folders, `-B`, sysroots and linker paths. What it leaves out on purpose is the source file's own path: a hit's program names the path of the run that made it, in an `assert` message only. A check that compares its compiler's timing or counts its compiler starts would see fewer. **Turning it off:** `LONGSHOT_CHECK_CACHE=0`, or revert the commit. **Compatibility:** Nothing changes for a player. Developers get a cache folder of up to 4 GB (a full run's compiles came to 48 MB on Windows), pruned by each run. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
perf(tests): cache the checks' compiles by what they compile
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s
Third-party parts / As their notes say (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 7s
e36ebb6e23
Many checks compile a small program from the source on every run, in
every worktree. run_all.py now starts each check through
tests/check_cache_boot.py, which routes its compiler calls (and the C++
checks' own) through tests/check_cache.py: a compile keyed by its
preprocessed source, flags and compiler, kept in one folder every
worktree on the PC shares. LONGSHOT_CHECK_CACHE=0 turns it off.

Windows, full run: 2970 s of check time cold, 1886 s warm (425 of 434
compiles from the cache), the same 19 failures as with it off.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dyst changed title from WIP: perf(tests): cache the checks' compiles by what they compile to perf(tests): cache the checks' compiles by what they compile 2026-11-15 20:12:47 +00:00
dyst force-pushed perf/check-compile-cache from e36ebb6e23
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 6s
Third-party parts / As their notes say (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 7s
to c3275d775b
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 14s
PR description / Description follows the template (pull_request_target) Successful in 3s
2026-11-16 06:35:24 +00:00
Compare
Author
Member

Rebased onto main @ 02903b04 (head c3275d77). Linux runs in an Ubuntu 24.04 container (gcc 13.3, clang 18.1, 24 cores): cold 11543 s / 564 s, warm 2948 s / 205 s with 1083 of 1098 compiles from the cache, cache off 10852 s / 539 s; the same 20 checks fail in all three. Measurements and Testing updated; nothing left to test.

Rebased onto main @ 02903b04 (head c3275d77). Linux runs in an Ubuntu 24.04 container (gcc 13.3, clang 18.1, 24 cores): cold 11543 s / 564 s, warm 2948 s / 205 s with 1083 of 1098 compiles from the cache, cache off 10852 s / 539 s; the same 20 checks fail in all three. Measurements and Testing updated; nothing left to test.
mooki left a comment

Review of c3275d77 (base 02903b04, the parent of your commit).

Thanks, this is a useful change and it does what the description says. I read the diff, the description and the thread (one comment from you, no other review), and ran the whole suite 14 times (the base, and this head cold, warm and with the cache off), and tried to make the key serve a stale program. I would fix points 1 and 2 before it merges (neither breaks anything in the suite today, both are small); point 3 is a decision about what the description promises.

What I ran

Linux (Arch, i9-13900K with 32 threads, clang 23.1.3, gcc 16.2.1, Python 3.12.16), python -I -B tests/run_all.py --jobs 6 --strict, a fresh HOME and an explicit cache folder. The PC was shared with other work, so each run is listed with its wall time, the CPU seconds of everything it started, and the load when it began.

# tree cache wall CPU compile cache line passed / failed / skipped load at start
1 base none 314 s 2366 s 629 / 18 / 17 40
2 head cold 246 s 2263 s 77 hits, 1022 misses, 79 passed through 630 / 18 / 17 47
3 head warm 112 s 828 s 1084 hits, 15 misses, 78 passed through 630 / 18 / 17 23
4 head off (=0) 189 s 2034 s 630 / 18 / 17 17
5 base none 178 s 2036 s 629 / 18 / 17 10
6 head cold 222 s 2287 s 80 hits, 1019 misses, 78 passed through 630 / 18 / 17 22
7 head warm 128 s 873 s 1084 hits, 15 misses, 78 passed through 630 / 18 / 17 30
8 head cold (logging what it passes through) 218 s 2262 s 76 hits, 1023 misses, 79 passed through 630 / 18 / 17 37
9 head off 178 s 2024 s 630 / 18 / 17 25
10 head warm 104 s 797 s 1084 hits, 15 misses, 78 passed through 630 / 18 / 17 16
11 head off 195 s 2160 s 630 / 18 / 17 14
12 head cold 209 s 2202 s 77 hits, 1022 misses, 78 passed through 630 / 18 / 17 19
13 head off 200 s 2229 s 630 / 18 / 17 28
14 head cold 198 s 2140 s 76 hits, 1023 misses, 78 passed through 630 / 18 / 17 19

Wall is run_all.py's own "in N s"; CPU is the user and system seconds of everything the run started. Run 1 was made while other work kept the load at 40 to 47, so it says little; the rest are comparable only with the load in mind.

  • Failures: the same 18 checks fail in every run (base, cold, warm, off), by name, and the failure text is the same line for line. They fail on the base here too (clang 23 and gcc 16, no game dump). tests/test_check_cache.py passes. The 17 skips are the same.
  • Hits: 1084 of 1099 compiles come from the cache in each warm run (you: 1083 of 1098), and 76 to 80 in a cold run (you: 76).
  • Warm: median 113 s (104 to 128 s) and 828 CPU seconds, against 189 s and 2036 CPU seconds with no cache: 40% less wall and 59% less CPU. Your 62% less wall is more than I get at 6 jobs; the long pole here is tests/test_recorder_sync.py (87 to 97 s warm).
  • Cold: median 219 s and 2262 CPU seconds, 16% more wall and 11% more CPU than no cache. The single pairs run from 1% less to 30% more wall, and the cache-off runs alone vary from 178 to 200 s, so I would call it a small cost and not say more. Your 5% is inside that range.

What I tried to break

All of these behaved:

  • a header edited and reached through a quote include, -I, -isystem, -include and -imacros: each recompiles and the program shows the new value; a changed -D, a changed object file input, an edited compiler script: all miss;
  • through run_all.py on a copy with your warm cache: a comment-only edit of gpu/vk/vk_present_rect.h hits (same code, as designed), a rounding change in it misses and present_rect_test.cpp follows the edit, an #error added to a header reached through -I makes lower_test.cpp fail with the compiler's message (a stale hit would have passed), reverting gives hits again;
  • 32 processes compiling one key at once on a cold cache: one whole entry, no leftover folder, no failure; a pruner emptying the cache 796 times while 16 workers compiled 400 programs: no failure;
  • prune: an entry unused for 15 days goes, a half-written folder older than an hour goes, a newer one stays, over the cap the oldest go first;
  • a cache folder that is a file, sits under a file, is mode 000 or cannot be created: the compile still works (see point 2 for the one that stops the run);
  • six Windows-target checks (llvm-mingw's x86_64-w64-mingw32-clang, LONGSHOT_MINGW set): 6 pass cold, warm and with the cache off; 18 compiles cached.

Please change

1. The key does not tell compilers apart that are the same file (tests/check_cache.py:84-88). It takes the resolved path, size and time of the compiler, so a name that is a symlink is lost. In llvm-mingw the per-target compilers are all symlinks to one script that picks its target from the name it is started as:

$ cat bare.c
int main(void){return 0;}
$ python tests/check_cache.py --cache C -- x86_64-w64-mingw32-clang -O1 bare.c -o a.exe    (miss) makes an x86-64 program
$ python tests/check_cache.py --cache C -- aarch64-w64-mingw32-clang -O1 bare.c -o b.exe   (hit)  gives the x86-64 program; compiled directly: ARM64
$ python tests/check_cache.py --cache C -- i686-w64-mingw32-clang -O1 bare.c -o c.exe      (hit)  gives the x86-64 program; compiled directly: Intel i386

With real headers the three miss, because the preprocessed text differs per target, and the Windows-target checks only use the x86-64 wrapper, so nothing in the suite breaks today. It breaks the first time one does not. The same collapse covers clang and clang++ (one file) and the ccache links that .forgejo/workflows/ci.yml:106-108 puts first in PATH (all six names resolve to the ccache binary, so the key names ccache and not the compiler). Adding os.path.basename(real) (the name as it was started) to the key fixes the wrapper case and costs nothing; I tried it and your test and my probes pass.

2. A cache folder that cannot be listed stops the whole run (tests/check_cache.py:269 and :273, called from tests/run_all.py:591-592). prune() calls os.listdir outside any try, and the runner calls it before the first check. With a folder of mode 000 (left by a run as root, another user, a restrictive mount) run_all.py ends with PermissionError: [Errno 13] and a traceback, no check runs. Every other bad folder I tried only turned the cache off. Catching OSError in prune() (and treating the folder as empty) is enough; I tried that too.

3. What the description says about CI. It says these compiles "come back on every PR's run". Nothing in this PR makes CI keep the folder, and the CI I can see would not use it anyway: the Python checks run by hand only ("off for now", .forgejo/workflows/ci.yml:61, which is why their statuses on this PR say skipped), when they do run they already go through ccache with a cache restored from main (:99-123), and there is no Windows runner (:8-9), so the Windows numbers are a gain for people running the suite on their own PC, not for CI. A cold cache costs a little (see Cold above), so if someone does run the suite in CI without the folder kept, it is a little slower. Either say in the description that it is for local runs, or keep the folder the way ccache's is kept (restore in pull requests, save on a push to main, keyed by the compiler versions).

Smaller

  • LONGSHOT_CHECK_CACHE=cache (a relative path) is used as given (tests/check_cache.py:61-62), so each compile resolves it against its own folder: two caches (miss miss) and a cache/ folder in the checkout. os.path.abspath in root() fixes it.
  • A failed compile now shows the launcher in the error: Command '[python3, -I, -B, .../check_cache.py, --cache, ..., --, clang++, ... instead of the compile, plus four extra traceback frames (tests/check_cache.py:253-257, tests/check_cache_boot.py:19-20). Setting self.args back to the original after super().__init__ fixes it.
  • If os.utime(entry) fails (a cache on a read-only mount, or owned by someone else) the finished hit is compiled again and the warning is printed twice (tests/check_cache.py:212-216). It needs its own try.
  • The key does not see a few things a compile reads, none of them used by a check today (I listed the flags of all 872 compiles of a cold run): -l libraries (rebuilding libfoo.a found through LIBRARY_PATH still hits and links the old one), a second output other than the ones on the list (-Xlinker -Map=x hits and leaves no map; -Wl,-Map is refused), a C++ source named .C (keyed by its bytes, so a header edit still hits), CCC_ADD_ARGS, the output's own name (-shared -o b.dll after -o a.dll hits and b.dll names itself a.dll), and line numbers the front end writes itself (__builtin_LINE() printed 2 after three comment lines were added on top; a fresh compile prints 5). Refusing what it cannot see, instead of listing what it can, would close all of them; at least the docstring could list them.
  • prune has no test, and an entry is trusted as found. If a directory renamed in without a flush can survive a power loss with a short program (I did not reproduce this), every hit serves it until it is 14 days old; a size in meta.json checked on a hit would cover it.
  • The headline compares the warm run with the cold column. Against the cache-off column your Linux numbers are 72.8% less check time and 62% less wall (539 s to 205 s), not 74% and 64%. The Windows 36% has no cache-off run. The sum of check times at 24 jobs on 24 cores counts every wait for the CPU, so wall and CPU seconds are the better figures. Your hit counts reproduce here to within one or two compiles.
  • The commit has Co-Authored-By but no Assisted-by line (CONTRIBUTING.md asks for both). CONTRIBUTING.md does not say where the folder is, that it keeps 14 days and 4 GB, or that off, false and no also turn it off; the docstring does.

I have a patch for points 1 and 2 and the first three of the smaller ones (about 20 lines; your test passes with it, and so do my probes). Say if you want it as a suggestion.

Not tested

Windows (not run for this review, so the 36% and os.replace of a directory under a virus scanner are untested), CI (the suite has not run in CI on this head), a cache shared by two users, a full disk, a power loss, and the head merged into main at 71b1edd8 (21 commits later; it merges cleanly, I did not run it).

Review of c3275d77 (base 02903b04, the parent of your commit). Thanks, this is a useful change and it does what the description says. I read the diff, the description and the thread (one comment from you, no other review), and ran the whole suite 14 times (the base, and this head cold, warm and with the cache off), and tried to make the key serve a stale program. **I would fix points 1 and 2 before it merges** (neither breaks anything in the suite today, both are small); point 3 is a decision about what the description promises. ## What I ran Linux (Arch, i9-13900K with 32 threads, clang 23.1.3, gcc 16.2.1, Python 3.12.16), `python -I -B tests/run_all.py --jobs 6 --strict`, a fresh HOME and an explicit cache folder. The PC was shared with other work, so each run is listed with its wall time, the CPU seconds of everything it started, and the load when it began. | # | tree | cache | wall | CPU | compile cache line | passed / failed / skipped | load at start | |---|---|---|---|---|---|---|---| | 1 | base | none | 314 s | 2366 s | | 629 / 18 / 17 | 40 | | 2 | head | cold | 246 s | 2263 s | 77 hits, 1022 misses, 79 passed through | 630 / 18 / 17 | 47 | | 3 | head | warm | 112 s | 828 s | 1084 hits, 15 misses, 78 passed through | 630 / 18 / 17 | 23 | | 4 | head | off (`=0`) | 189 s | 2034 s | | 630 / 18 / 17 | 17 | | 5 | base | none | 178 s | 2036 s | | 629 / 18 / 17 | 10 | | 6 | head | cold | 222 s | 2287 s | 80 hits, 1019 misses, 78 passed through | 630 / 18 / 17 | 22 | | 7 | head | warm | 128 s | 873 s | 1084 hits, 15 misses, 78 passed through | 630 / 18 / 17 | 30 | | 8 | head | cold (logging what it passes through) | 218 s | 2262 s | 76 hits, 1023 misses, 79 passed through | 630 / 18 / 17 | 37 | | 9 | head | off | 178 s | 2024 s | | 630 / 18 / 17 | 25 | | 10 | head | warm | 104 s | 797 s | 1084 hits, 15 misses, 78 passed through | 630 / 18 / 17 | 16 | | 11 | head | off | 195 s | 2160 s | | 630 / 18 / 17 | 14 | | 12 | head | cold | 209 s | 2202 s | 77 hits, 1022 misses, 78 passed through | 630 / 18 / 17 | 19 | | 13 | head | off | 200 s | 2229 s | | 630 / 18 / 17 | 28 | | 14 | head | cold | 198 s | 2140 s | 76 hits, 1023 misses, 78 passed through | 630 / 18 / 17 | 19 | Wall is `run_all.py`'s own "in N s"; CPU is the user and system seconds of everything the run started. Run 1 was made while other work kept the load at 40 to 47, so it says little; the rest are comparable only with the load in mind. - **Failures:** the same 18 checks fail in every run (base, cold, warm, off), by name, and the failure text is the same line for line. They fail on the base here too (clang 23 and gcc 16, no game dump). `tests/test_check_cache.py` passes. The 17 skips are the same. - **Hits:** 1084 of 1099 compiles come from the cache in each warm run (you: 1083 of 1098), and 76 to 80 in a cold run (you: 76). - **Warm:** median 113 s (104 to 128 s) and 828 CPU seconds, against 189 s and 2036 CPU seconds with no cache: 40% less wall and 59% less CPU. Your 62% less wall is more than I get at 6 jobs; the long pole here is `tests/test_recorder_sync.py` (87 to 97 s warm). - **Cold:** median 219 s and 2262 CPU seconds, 16% more wall and 11% more CPU than no cache. The single pairs run from 1% less to 30% more wall, and the cache-off runs alone vary from 178 to 200 s, so I would call it a small cost and not say more. Your 5% is inside that range. ## What I tried to break All of these behaved: - a header edited and reached through a quote include, `-I`, `-isystem`, `-include` and `-imacros`: each recompiles and the program shows the new value; a changed `-D`, a changed object file input, an edited compiler script: all miss; - through `run_all.py` on a copy with your warm cache: a comment-only edit of `gpu/vk/vk_present_rect.h` hits (same code, as designed), a rounding change in it misses and `present_rect_test.cpp` follows the edit, an `#error` added to a header reached through `-I` makes `lower_test.cpp` fail with the compiler's message (a stale hit would have passed), reverting gives hits again; - 32 processes compiling one key at once on a cold cache: one whole entry, no leftover folder, no failure; a pruner emptying the cache 796 times while 16 workers compiled 400 programs: no failure; - `prune`: an entry unused for 15 days goes, a half-written folder older than an hour goes, a newer one stays, over the cap the oldest go first; - a cache folder that is a file, sits under a file, is mode 000 or cannot be created: the compile still works (see point 2 for the one that stops the run); - six Windows-target checks (llvm-mingw's `x86_64-w64-mingw32-clang`, `LONGSHOT_MINGW` set): 6 pass cold, warm and with the cache off; 18 compiles cached. ## Please change **1. The key does not tell compilers apart that are the same file** (`tests/check_cache.py:84-88`). It takes the resolved path, size and time of the compiler, so a name that is a symlink is lost. In llvm-mingw the per-target compilers are all symlinks to one script that picks its target from the name it is started as: ``` $ cat bare.c int main(void){return 0;} $ python tests/check_cache.py --cache C -- x86_64-w64-mingw32-clang -O1 bare.c -o a.exe (miss) makes an x86-64 program $ python tests/check_cache.py --cache C -- aarch64-w64-mingw32-clang -O1 bare.c -o b.exe (hit) gives the x86-64 program; compiled directly: ARM64 $ python tests/check_cache.py --cache C -- i686-w64-mingw32-clang -O1 bare.c -o c.exe (hit) gives the x86-64 program; compiled directly: Intel i386 ``` With real headers the three miss, because the preprocessed text differs per target, and the Windows-target checks only use the x86-64 wrapper, so nothing in the suite breaks today. It breaks the first time one does not. The same collapse covers `clang` and `clang++` (one file) and the ccache links that `.forgejo/workflows/ci.yml:106-108` puts first in PATH (all six names resolve to the ccache binary, so the key names ccache and not the compiler). Adding `os.path.basename(real)` (the name as it was started) to the key fixes the wrapper case and costs nothing; I tried it and your test and my probes pass. **2. A cache folder that cannot be listed stops the whole run** (`tests/check_cache.py:269` and `:273`, called from `tests/run_all.py:591-592`). `prune()` calls `os.listdir` outside any `try`, and the runner calls it before the first check. With a folder of mode 000 (left by a run as root, another user, a restrictive mount) `run_all.py` ends with `PermissionError: [Errno 13]` and a traceback, no check runs. Every other bad folder I tried only turned the cache off. Catching `OSError` in `prune()` (and treating the folder as empty) is enough; I tried that too. **3. What the description says about CI.** It says these compiles "come back on every PR's run". Nothing in this PR makes CI keep the folder, and the CI I can see would not use it anyway: the Python checks run by hand only ("off for now", `.forgejo/workflows/ci.yml:61`, which is why their statuses on this PR say skipped), when they do run they already go through ccache with a cache restored from main (`:99-123`), and there is no Windows runner (`:8-9`), so the Windows numbers are a gain for people running the suite on their own PC, not for CI. A cold cache costs a little (see Cold above), so if someone does run the suite in CI without the folder kept, it is a little slower. Either say in the description that it is for local runs, or keep the folder the way ccache's is kept (restore in pull requests, save on a push to main, keyed by the compiler versions). ## Smaller - `LONGSHOT_CHECK_CACHE=cache` (a relative path) is used as given (`tests/check_cache.py:61-62`), so each compile resolves it against its own folder: two caches (`miss miss`) and a `cache/` folder in the checkout. `os.path.abspath` in `root()` fixes it. - A failed compile now shows the launcher in the error: `Command '[python3, -I, -B, .../check_cache.py, --cache, ..., --, clang++, ...` instead of the compile, plus four extra traceback frames (`tests/check_cache.py:253-257`, `tests/check_cache_boot.py:19-20`). Setting `self.args` back to the original after `super().__init__` fixes it. - If `os.utime(entry)` fails (a cache on a read-only mount, or owned by someone else) the finished hit is compiled again and the warning is printed twice (`tests/check_cache.py:212-216`). It needs its own `try`. - The key does not see a few things a compile reads, none of them used by a check today (I listed the flags of all 872 compiles of a cold run): `-l` libraries (rebuilding `libfoo.a` found through `LIBRARY_PATH` still hits and links the old one), a second output other than the ones on the list (`-Xlinker -Map=x` hits and leaves no map; `-Wl,-Map` is refused), a C++ source named `.C` (keyed by its bytes, so a header edit still hits), `CCC_ADD_ARGS`, the output's own name (`-shared -o b.dll` after `-o a.dll` hits and `b.dll` names itself `a.dll`), and line numbers the front end writes itself (`__builtin_LINE()` printed 2 after three comment lines were added on top; a fresh compile prints 5). Refusing what it cannot see, instead of listing what it can, would close all of them; at least the docstring could list them. - `prune` has no test, and an entry is trusted as found. If a directory renamed in without a flush can survive a power loss with a short program (I did not reproduce this), every hit serves it until it is 14 days old; a size in `meta.json` checked on a hit would cover it. - The headline compares the warm run with the cold column. Against the cache-off column your Linux numbers are 72.8% less check time and 62% less wall (539 s to 205 s), not 74% and 64%. The Windows 36% has no cache-off run. The sum of check times at 24 jobs on 24 cores counts every wait for the CPU, so wall and CPU seconds are the better figures. Your hit counts reproduce here to within one or two compiles. - The commit has `Co-Authored-By` but no `Assisted-by` line (CONTRIBUTING.md asks for both). CONTRIBUTING.md does not say where the folder is, that it keeps 14 days and 4 GB, or that `off`, `false` and `no` also turn it off; the docstring does. I have a patch for points 1 and 2 and the first three of the smaller ones (about 20 lines; your test passes with it, and so do my probes). Say if you want it as a suggestion. ## Not tested Windows (not run for this review, so the 36% and `os.replace` of a directory under a virus scanner are untested), CI (the suite has not run in CI on this head), a cache shared by two users, a full disk, a power loss, and the head merged into main at 71b1edd8 (21 commits later; it merges cleanly, I did not run it).
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 14s
PR description / Description follows the template (pull_request_target) Successful in 3s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/check-compile-cache:dyst-perf/check-compile-cache'
git switch 'dyst-perf/check-compile-cache'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!893
No description provided.