WIP: feat(scripts): coverage_gaps.py, what no game-free check names #1819
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1819
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:test/test-gaps"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Windows 11 (a script and its test; nothing the game runs) · Needs: none
Builds on #1818 (its test is a pytest module, which run_all runs only with #1818): review that one first. Only the last commit is this PR's.
TL;DR
We can't easily see what has no test. This adds a script that lists the switches, source folders and files no game-free check names, and each feature folder without a tests/ folder of its own, so the next test to write can be picked from a list instead of a guess. It is a map, not a gate: nothing fails on it. On main today: 362 of 712 switches and 387 of 444 source files are named by a check, and gpu/vk/menu, the first feature folder, has 19 files no check names.
Breakdown
scripts/coverage_gaps.pyprints that list (and--jsonfor the dashboard); CONTRIBUTING.md says where to find it.tests/test_coverage_gaps.pyover a made-up checkout (named by path, by file name, by import, in a feature's own tests/; vendored and generated files left out; GONE rows not counted) and over this one; the report run on main.Why
The plan is for features to move into folders of their own with their tests beside them (
gpu/vk/menu/since #1774). To do that methodically we need to know where tests are missing: which player switches no check sets, which files no check reads, which feature folder has no tests/ yet.What changed
scripts/coverage_gaps.py: reads every test file (tests/, nx2/test_*.py, any tests/ folder under gpu/, host/, nx2/) for the source paths, file names, imported modules and switch names it mentions, astests/run_all.py --changedmatches them. Reports:gpu/vk/switches.defby kind, and the player, attempt and kill switches no check names (--all: every one);tests/;Vendored code (gpu/compat, gpu/vk/compat, gpu/vk/eden, gpu/vk/sirit, nx2/prompt_faces) and generated
*_spv.hare left out.--json FILEwrites all of it.tests/test_coverage_gaps.py: the cases above over a made-up checkout, and the real tree (every switch row read).Measurements
Nothing to measure: a report script and its test. Its output on main @
0de25c3e, for the record: switches named by a check 362 of 712 (player 40 of 53, attempt 44 of 84, kill 59 of 113); source files 387 of 444 (57 unnamed, 4,489 lines; 11 large files named by one check); gpu/vk/menu 22 of 41 files, no tests/ of its own. It runs in about 7 s.Flags and settings
None.
Testing
Tested:
python -I -B -m pytest tests/test_coverage_gaps.py(3 passed) andpython -I -B tests/run_all.py --exact tests/test_coverage_gaps.py(passed) on Windows 11; the report on main.Not tested: Linux: standard library only, and paths are read with forward slashes.
Must be tested: None.
Risks
What could break: nothing runs it automatically; a wrong count misleads only whoever reads it (the docstring and the header say it matches by text).
Turning it off: revert the commit.
Compatibility: nothing changes for a player.
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.🤖 Generated with Claude Code
View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.