WIP: feat(scripts): coverage_gaps.py, what no game-free check names #1819

Draft
dyst wants to merge 4 commits from dyst/Longshot:test/test-gaps into main
Member

Type: feat · Risk: low · Tested on: Windows 11 (a script and its test; nothing the game runs) · Needs: none

Builds on #1818 (its test is a pytest module, which run_all runs only with #1818): review that one first. Only the last commit is this PR's.

TL;DR

We can't easily see what has no test. This adds a script that lists the switches, source folders and files no game-free check names, and each feature folder without a tests/ folder of its own, so the next test to write can be picked from a list instead of a guess. It is a map, not a gate: nothing fails on it. On main today: 362 of 712 switches and 387 of 444 source files are named by a check, and gpu/vk/menu, the first feature folder, has 19 files no check names.

Breakdown

  • Who it helps: anyone choosing what to test next, and reviewers asking whether an area has tests.
  • What was wrong: nothing showed which switches, files or feature folders no check covers.
  • What changes: scripts/coverage_gaps.py prints that list (and --json for the dashboard); CONTRIBUTING.md says where to find it.
  • What does not change: the checks, the runners, the game.
  • How it was checked: tests/test_coverage_gaps.py over a made-up checkout (named by path, by file name, by import, in a feature's own tests/; vendored and generated files left out; GONE rows not counted) and over this one; the report run on main.
  • What is still open: nothing for this PR; what it lists is for later PRs.
  • Risk: low: a new script and its test.

Why

The plan is for features to move into folders of their own with their tests beside them (gpu/vk/menu/ since #1774). To do that methodically we need to know where tests are missing: which player switches no check sets, which files no check reads, which feature folder has no tests/ yet.

What changed

  • scripts/coverage_gaps.py: reads every test file (tests/, nx2/test_*.py, any tests/ folder under gpu/, host/, nx2/) for the source paths, file names, imported modules and switch names it mentions, as tests/run_all.py --changed matches them. Reports:
    • the switches of gpu/vk/switches.def by kind, and the player, attempt and kill switches no check names (--all: every one);
    • each source folder (a feature folder is a row of its own): files, lines, how much of it a check names, and the test files in its own tests/;
    • the large files (300 lines or more) only one check names, and the files none does, largest first.
      Vendored code (gpu/compat, gpu/vk/compat, gpu/vk/eden, gpu/vk/sirit, nx2/prompt_faces) and generated *_spv.h are left out. --json FILE writes all of it.
  • tests/test_coverage_gaps.py: the cases above over a made-up checkout, and the real tree (every switch row read).
  • CONTRIBUTING.md: one line under "When you add a fix or a check".

Measurements

Nothing to measure: a report script and its test. Its output on main @ 0de25c3e, for the record: switches named by a check 362 of 712 (player 40 of 53, attempt 44 of 84, kill 59 of 113); source files 387 of 444 (57 unnamed, 4,489 lines; 11 large files named by one check); gpu/vk/menu 22 of 41 files, no tests/ of its own. It runs in about 7 s.

Flags and settings

None.

Testing

Tested: python -I -B -m pytest tests/test_coverage_gaps.py (3 passed) and python -I -B tests/run_all.py --exact tests/test_coverage_gaps.py (passed) on Windows 11; the report on main.

Not tested: Linux: standard library only, and paths are read with forward slashes.

Must be tested: None.

Risks

What could break: nothing runs it automatically; a wrong count misleads only whoever reads it (the docstring and the header say it matches by text).

Turning it off: revert the commit.

Compatibility: nothing changes for a player.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** low · **Tested on:** Windows 11 (a script and its test; nothing the game runs) · **Needs:** none **Builds on #1818** (its test is a pytest module, which run_all runs only with #1818): review that one first. Only the last commit is this PR's. ## TL;DR We can't easily see what has no test. This adds a script that lists the switches, source folders and files no game-free check names, and each feature folder without a tests/ folder of its own, so the next test to write can be picked from a list instead of a guess. It is a map, not a gate: nothing fails on it. On main today: 362 of 712 switches and 387 of 444 source files are named by a check, and gpu/vk/menu, the first feature folder, has 19 files no check names. ## Breakdown - **Who it helps:** anyone choosing what to test next, and reviewers asking whether an area has tests. - **What was wrong:** nothing showed which switches, files or feature folders no check covers. - **What changes:** `scripts/coverage_gaps.py` prints that list (and `--json` for the dashboard); CONTRIBUTING.md says where to find it. - **What does not change:** the checks, the runners, the game. - **How it was checked:** `tests/test_coverage_gaps.py` over a made-up checkout (named by path, by file name, by import, in a feature's own tests/; vendored and generated files left out; GONE rows not counted) and over this one; the report run on main. - **What is still open:** nothing for this PR; what it lists is for later PRs. - **Risk:** low: a new script and its test. ## Why The plan is for features to move into folders of their own with their tests beside them (`gpu/vk/menu/` since #1774). To do that methodically we need to know where tests are missing: which player switches no check sets, which files no check reads, which feature folder has no tests/ yet. ## What changed - `scripts/coverage_gaps.py`: reads every test file (tests/, nx2/test_*.py, any tests/ folder under gpu/, host/, nx2/) for the source paths, file names, imported modules and switch names it mentions, as `tests/run_all.py --changed` matches them. Reports: - the switches of `gpu/vk/switches.def` by kind, and the player, attempt and kill switches no check names (`--all`: every one); - each source folder (a feature folder is a row of its own): files, lines, how much of it a check names, and the test files in its own `tests/`; - the large files (300 lines or more) only one check names, and the files none does, largest first. Vendored code (gpu/compat, gpu/vk/compat, gpu/vk/eden, gpu/vk/sirit, nx2/prompt_faces) and generated `*_spv.h` are left out. `--json FILE` writes all of it. - `tests/test_coverage_gaps.py`: the cases above over a made-up checkout, and the real tree (every switch row read). - CONTRIBUTING.md: one line under "When you add a fix or a check". ## Measurements Nothing to measure: a report script and its test. Its output on main @ 0de25c3e, for the record: switches named by a check 362 of 712 (player 40 of 53, attempt 44 of 84, kill 59 of 113); source files 387 of 444 (57 unnamed, 4,489 lines; 11 large files named by one check); gpu/vk/menu 22 of 41 files, no tests/ of its own. It runs in about 7 s. ### Flags and settings None. ## Testing **Tested:** `python -I -B -m pytest tests/test_coverage_gaps.py` (3 passed) and `python -I -B tests/run_all.py --exact tests/test_coverage_gaps.py` (passed) on Windows 11; the report on main. **Not tested:** Linux: standard library only, and paths are read with forward slashes. **Must be tested:** None. ## Risks **What could break:** nothing runs it automatically; a wrong count misleads only whoever reads it (the docstring and the header say it matches by text). **Turning it off:** revert the commit. **Compatibility:** nothing changes for a player. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Its stand-ins counted toggle clears, captures and releases, and nothing
read them: clang 21 and later refuse that under -Werror
(-Wunused-but-set-global), so the check failed with the Builder's
llvm-mingw, whose gcc is clang. Passes with clang 23.1 and gcc 10.3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
--random counts per guest word, and there are 130 of them: the default
1,000,000 is 130M checks per -march, four times, about 25 minutes on a
Ryzen 9 3900X (6 minutes a -march). TBL and EXT work lane by lane, so
50,000 states give each word 800,000 index bytes, every value 0-255
about 3,000 times and the edge values far more: about 2 minutes. The
check's own default stays 1,000,000 for a run by hand.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
test: pytest runs the game-free checks; requirements by machine; known failures
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 13s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 16s
8ffe5053e1
pytest collects every check tests/run_all.py runs, unchanged: a test file
that imports pytest is a pytest module, every other one a script check run
in its own process as run_all runs it (tests/pytest_longshot.py, the root
conftest.py and pytest.ini). Feature folders' tests/ are found too.

tests/check_env.py, shared by both runners:
- the C++ compiler is $CXX, then LONGSHOT_MINGW's, then the Builder's own
  llvm-mingw, and only then PATH (which here found an old TDM gcc 10.3);
  its folder goes first on PATH and becomes LONGSHOT_MINGW, so the
  programs it builds find its DLLs and the a64-kit's extension builds;
- requirements by OS, CPU architecture, CPU features or level
  (x86-64-v3), compiler family and version, and flags it must take
  (RUN_ALL_REQUIRES, the requires marker); the features from CPUID and
  XGETBV, or /proc/cpuinfo;
- a check that cannot run here ends with skip(why), exit 77: shown as
  skipped with why, where a print and exit 0 counted as a pass;
- tests/known_failures.txt, main's failures per OS with why: known, not
  a failure of the run; one that passes fails it until its line goes.

run_all runs a pytest module with pytest, prints what it found here, and
reads which() once per name. tests/test_check_env.py covers all of it,
end to end under both runners; test_fastfp_fma declares x86-64-v3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
feat(scripts): coverage_gaps.py, what no game-free check names
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 12s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 15s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 15s
1429e77525
The switches of gpu/vk/switches.def by kind, the source folders (each
feature folder a row, with its own tests/ counted), the large files only
one check names and the files none does, by text as run_all --changed
picks checks: a map of where tests are missing, not a measurement, and
nothing fails on it. --json for the dashboard, --all for every gap.
tests/test_coverage_gaps.py over a made-up checkout and this one.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 12s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 15s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 15s
This pull request has changes conflicting with the target branch.
  • CONTRIBUTING.md
  • mise.toml
  • tests/run_all.py
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'test/test-gaps:dyst-test/test-gaps'
git switch 'dyst-test/test-gaps'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1819
No description provided.