test: each test names its own run_all arguments, skips and modules #1822

Merged

hckrman101 merged 2 commits from runall-declarations into main

2026-11-10 17:47:49 +00:00

hckrman101 commented

2026-11-10 17:37:20 +00:00

(Migrated from github.com)

Every PR that adds a check also adds a line at the top of ARGS (and often SKIP) in tests/run_all.py, so two such PRs open at the same time conflict there even when their tests have nothing to do with each other. This moves those lines into the tests themselves, so adding a test no longer touches a shared file.

What changed

A test declares what the runner has to give it in its own file, as top-level literals after its docstring and imports. tests/run_all.py reads them with ast and never imports the test.
Every entry in ARGS, MODULES and SKIP, and the spirv-val and glslang skip loops, moved into the 125 test files they named. The group comments moved with them ((a GPU), (Linux only), the spirv-val and glslang notes, the game note).
The runner checks the declarations before it runs anything. A value that is not a plain literal of the right shape, an unknown {placeholder}, an unknown OS, a declaration assigned twice or changed after it is set, or a misspelt RUN_ALL_ name stops the run with exit code 2 and prints the file and line. A test that does not parse just declares nothing: it fails when it runs, and so does the byte-compile, as before.
tests/test_run_all_declarations.py covers the reader: valid declarations, malformed ones, unknown placeholders, unknown OSes, the retired tables, and a pass over every test in the repository.
CONTRIBUTING.md's When you add a fix or a check and the workflow's header comment describe the new way.

Declaring what a new test needs

"""What the test checks"""
import argparse

RUN_ALL_ARGS = ["--cxx", "{cxx}", "--headers", "{vulkan_headers}"]   # its arguments
RUN_ALL_MODULES = ["zstandard"]                                       # packages it imports
RUN_ALL_SKIP = {"windows": "POSIX signals: the Linux guard"}          # OS -> why it is skipped there

{name} is a tool the runner finds: cxx, vulkan_headers, spirv_headers, spirv_val, glslang, dotnet. If the tool is missing, the check is skipped, or fails with --strict. The same goes for the compiler after --cc/--cxx and for a missing module, as before.
RUN_ALL_SKIP maps "linux" and/or "windows" to a reason. A check that can run on neither (a GPU, the game) names both. --no-skip, or naming the check, still runs it.
A test that needs none of these declares nothing.

PRs opened before this one

They will conflict in run_all.py once, on the ARGS lines this PR removes. That can't be avoided, but it is the last such conflict. ARGS, MODULES and SKIP stay in run_all.py, empty, with a comment saying they are retired. If a resolution (or a merge) keeps the old line in one of them, run_all.py refuses to run anything and prints the declaration to put in the test, for example:

tests/run_all.py: ARGS, MODULES and SKIP are retired: each test declares what it needs in its own file (see the docstring at the top of tests/run_all.py)
  tests/test_new.py: declare it in the test: RUN_ALL_ARGS = ['--cxx', '{cxx}']

I chose this over honouring and merging the old tables for a while. Merging would let new entries keep landing in the shared file, which keeps the conflicts this PR is meant to stop. Failing loudly costs a stale PR one small move. Dropping the old tables without a check would have been worse: a test whose arguments all have defaults (--cxx g++ and the like) would then quietly run with the defaults instead of the arguments the PR asked for. test_run_all_declarations.py also fails while any of the three tables has an entry.

Same checks, same arguments

I ran a throwaway script against main's run_all.py and this one, with subprocess.run, the tool lookup and the module lookup replaced by recorders. It ran main() for both OSes (OS forced to windows for the second), with every tool present and with none present, each plainly, with --no-skip and with --strict. That is 12 runs, which recorded every command line a check would get, every skip and its reason, and every "not found here" list. The two outputs (9,896 lines, 2,758 command lines) are identical. The only differences are the new test and the check counts that include it. A second script compared the resolved test -> (arguments, modules, skip reason) table for all 295 test files on both OSes: 0 differences.

Testing

python -I -B tests/run_all.py on Linux runs the same checks as main, with the same failures and the same 9 skips with the same reasons, plus the new test, which passes.

Every PR that adds a check also adds a line at the top of `ARGS` (and often `SKIP`) in `tests/run_all.py`, so two such PRs open at the same time conflict there even when their tests have nothing to do with each other. This moves those lines into the tests themselves, so adding a test no longer touches a shared file. ## What changed - A test declares what the runner has to give it in its own file, as top-level literals after its docstring and imports. `tests/run_all.py` reads them with `ast` and never imports the test. - Every entry in `ARGS`, `MODULES` and `SKIP`, and the spirv-val and glslang skip loops, moved into the 125 test files they named. The group comments moved with them (`(a GPU)`, `(Linux only)`, the spirv-val and glslang notes, the game note). - The runner checks the declarations before it runs anything. A value that is not a plain literal of the right shape, an unknown `{placeholder}`, an unknown OS, a declaration assigned twice or changed after it is set, or a misspelt `RUN_ALL_` name stops the run with exit code 2 and prints the file and line. A test that does not parse just declares nothing: it fails when it runs, and so does the byte-compile, as before. - `tests/test_run_all_declarations.py` covers the reader: valid declarations, malformed ones, unknown placeholders, unknown OSes, the retired tables, and a pass over every test in the repository. - CONTRIBUTING.md's *When you add a fix or a check* and the workflow's header comment describe the new way. ## Declaring what a new test needs ```python """What the test checks""" import argparse RUN_ALL_ARGS = ["--cxx", "{cxx}", "--headers", "{vulkan_headers}"] # its arguments RUN_ALL_MODULES = ["zstandard"] # packages it imports RUN_ALL_SKIP = {"windows": "POSIX signals: the Linux guard"} # OS -> why it is skipped there ``` - `{name}` is a tool the runner finds: `cxx`, `vulkan_headers`, `spirv_headers`, `spirv_val`, `glslang`, `dotnet`. If the tool is missing, the check is skipped, or fails with `--strict`. The same goes for the compiler after `--cc`/`--cxx` and for a missing module, as before. - `RUN_ALL_SKIP` maps `"linux"` and/or `"windows"` to a reason. A check that can run on neither (a GPU, the game) names both. `--no-skip`, or naming the check, still runs it. - A test that needs none of these declares nothing. ## PRs opened before this one They will conflict in `run_all.py` once, on the `ARGS` lines this PR removes. That can't be avoided, but it is the last such conflict. `ARGS`, `MODULES` and `SKIP` stay in `run_all.py`, empty, with a comment saying they are retired. If a resolution (or a merge) keeps the old line in one of them, `run_all.py` refuses to run anything and prints the declaration to put in the test, for example: ``` tests/run_all.py: ARGS, MODULES and SKIP are retired: each test declares what it needs in its own file (see the docstring at the top of tests/run_all.py) tests/test_new.py: declare it in the test: RUN_ALL_ARGS = ['--cxx', '{cxx}'] ``` I chose this over honouring and merging the old tables for a while. Merging would let new entries keep landing in the shared file, which keeps the conflicts this PR is meant to stop. Failing loudly costs a stale PR one small move. Dropping the old tables without a check would have been worse: a test whose arguments all have defaults (`--cxx g++` and the like) would then quietly run with the defaults instead of the arguments the PR asked for. `test_run_all_declarations.py` also fails while any of the three tables has an entry. ## Same checks, same arguments I ran a throwaway script against `main`'s `run_all.py` and this one, with `subprocess.run`, the tool lookup and the module lookup replaced by recorders. It ran `main()` for both OSes (`OS` forced to `windows` for the second), with every tool present and with none present, each plainly, with `--no-skip` and with `--strict`. That is 12 runs, which recorded every command line a check would get, every skip and its reason, and every "not found here" list. The two outputs (9,896 lines, 2,758 command lines) are identical. The only differences are the new test and the check counts that include it. A second script compared the resolved test -> (arguments, modules, skip reason) table for all 295 test files on both OSes: 0 differences. ## Testing `python -I -B tests/run_all.py` on Linux runs the same checks as `main`, with the same failures and the same 9 skips with the same reasons, plus the new test, which passes.

hckrman101 (Migrated from github.com) reviewed

2026-11-10 17:42:36 +00:00

hckrman101 (Migrated from github.com) left a comment

Self-review pass before merge.

Equivalence: I checked every RUN_ALL_* declaration against main's ARGS / MODULES / SKIP tables and the spirv-val / glslang skip loops. 125 test files, 0 differences: the same arguments, placeholders, modules, per-OS skips and reason strings. Main's tests/run_all.py is byte-identical to this branch's base, so there's nothing to carry over yet.

Parser: read_declarations uses ast.parse + literal_eval and never imports a test, so discovery has no side effects. It rejects misspelt RUN_ALL_ names, unknown placeholders/OSes, duplicate assignments, +=, subscripts and non-literals, all with file:line. BOM/CRLF are fine, and --strict / --no-skip behave as before.

For the ~40 open PRs that add a line to run_all.py: each will get one conflict where the ARGS block is gone. The fix is to move that line into the test file as RUN_ALL_ARGS = [...] (or RUN_ALL_SKIP / RUN_ALL_MODULES). If a resolution keeps the old table line, stale_tables() stops the run and prints the line to add. If it just drops it, the test silently runs with its defaults (e.g. --cxx g++), so watch for that when rebasing.

Small follow-ups, not blocking:

An assignment inside a top-level if/try, or RUN_ALL_ARGS.append(...), is ignored without a message. It could be rejected the same way += is.
A test file that doesn't parse declares nothing, so on Windows its skip turns into a failure (the byte-compile check already fails it anyway).
One bad declaration anywhere blocks running even a single named test. Scoping the check to the selected tests would be friendlier.

Self-review pass before merge. **Equivalence:** I checked every `RUN_ALL_*` declaration against main's `ARGS` / `MODULES` / `SKIP` tables and the spirv-val / glslang skip loops. 125 test files, 0 differences: the same arguments, placeholders, modules, per-OS skips and reason strings. Main's `tests/run_all.py` is byte-identical to this branch's base, so there's nothing to carry over yet. **Parser:** `read_declarations` uses `ast.parse` + `literal_eval` and never imports a test, so discovery has no side effects. It rejects misspelt `RUN_ALL_` names, unknown placeholders/OSes, duplicate assignments, `+=`, subscripts and non-literals, all with file:line. BOM/CRLF are fine, and `--strict` / `--no-skip` behave as before. **For the ~40 open PRs that add a line to `run_all.py`:** each will get one conflict where the `ARGS` block is gone. The fix is to move that line into the test file as `RUN_ALL_ARGS = [...]` (or `RUN_ALL_SKIP` / `RUN_ALL_MODULES`). If a resolution keeps the old table line, `stale_tables()` stops the run and prints the line to add. If it just drops it, the test silently runs with its defaults (e.g. `--cxx g++`), so watch for that when rebasing. Small follow-ups, not blocking: - An assignment inside a top-level `if`/`try`, or `RUN_ALL_ARGS.append(...)`, is ignored without a message. It could be rejected the same way `+=` is. - A test file that doesn't parse declares nothing, so on Windows its skip turns into a failure (the byte-compile check already fails it anyway). - One bad declaration anywhere blocks running even a single named test. Scoping the check to the selected tests would be friendlier.

hckrman101 commented

2026-11-10 17:46:01 +00:00

(Migrated from github.com)

All three follow-ups are in 6b54b58:

A RUN_ALL_ name set inside a top-level if or try, or changed with .append(...), .update(...) or del, now stops the run with the file and line, like += and subscripts already did. A function's or class's own names are still left alone.
A run of named tests (run_all.py test_foo) reads only those tests' declarations, so a bad declaration elsewhere no longer blocks it.
A test that doesn't parse now fails on every OS with does not parse (...), without being run, instead of losing its skip on Windows.

tests/test_run_all_declarations.py has a test for each (17 pass). The full run is unchanged: the same failures as main, plus the new tests passing.

All three follow-ups are in 6b54b58: - A `RUN_ALL_` name set inside a top-level `if` or `try`, or changed with `.append(...)`, `.update(...)` or `del`, now stops the run with the file and line, like `+=` and subscripts already did. A function's or class's own names are still left alone. - A run of named tests (`run_all.py test_foo`) reads only those tests' declarations, so a bad declaration elsewhere no longer blocks it. - A test that doesn't parse now fails on every OS with `does not parse (...)`, without being run, instead of losing its skip on Windows. `tests/test_run_all_declarations.py` has a test for each (17 pass). The full run is unchanged: the same failures as `main`, plus the new tests passing.

Commenting is not possible because the repository is archived.

No reviewers