feat(tests): run_all.py --changed runs the checks that read what changed #32

Open

dyst wants to merge 2 commits from dyst/Longshot:feat/run-all-changed into main

pull from: dyst/Longshot:feat/run-all-changed

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 5 Commits 2 Files changed 2 +148

dyst commented 2026-11-12 23:38:49 +00:00

Member

• Copy link
• Copy Markdown

Why

A full tests/run_all.py is about 490 checks and several minutes, and the ones an edit touches had to be picked by hand, which is how a check that cuts a function far from the edit gets missed (AGENTS.md: "Tests read the source").

What changed

• tests: tests/run_all.py --changed [BASE]: changed_files takes the files changed since BASE's merge base with HEAD (origin/main by default), committed or not, plus the untracked ones, from git. reads_changed picks:
  • the checks whose text names one of them: by path, by file name standing alone, or as a module they import (import chunks, from chunks import);
  • the changed checks themselves (and the byte-compile check, as before);
  • every check when the runner itself or a shared .py under tests/ that is not a test changed.
• tests: with --changed, the C++ unit tests run only when something under gpu/ changed or every check runs. It prints how many files changed and how many checks read them, and that a pull request still needs the whole run. --changed can't be combined with names or --write-times. Nothing else in the runner changes.
• tests: with no BASE, --changed takes main on the remote that points at Longshot (forge/main where scripts/forge.py added it, as CONTRIBUTING.md has it), else origin/main (default_base): in a clone carried over from Hookshot, origin is GitHub (the second commit).
• tests: tests/test_run_all_changed.py (new): which checks a changed file picks (by path, by name and not inside a longer name, by import), that a changed check runs itself, that the runner or a shared test file runs everything, that an unnamed file runs none, and the files from git (committed, uncommitted and untracked), with git's answers stood in; and the default base in three remote layouts (it fails with the old default).

Example, on the GitHub branch (the runner changed): --changed: 2 files changed since origin/main, every check (the runner or a shared test file changed). For a change to nx2/chunks.py alone it picks the chunk, emit and lift-cache checks (under a quarter of all), not the renderer's.

Flags and settings

None: no environment switch, menu row or ini key. The new tests/run_all.py --changed [BASE] option is the test runner's, described above.

Measurements

main: 000f19fb (2026-11-13)
this PR: feat/run-all-changed @ f0982a1e
Hardware: Ryzen 9 3900X, 64 GB, Windows 11 (RTX 3080; no game run: these are game-free checks)
Scene: none: game-free. A one-line change to nx2/chunks.py (a comment, uncommitted), checked
Method: two interleaved rounds (main, PR, main, PR), each run with the PC otherwise quiet and the bench lock held. main: the whole suite, the only run it has (run_all.py --jobs 12). This PR: run_all.py --jobs 12 --changed HEAD (the branch's own commits as the base, so only the edit counts; with this branch against main the runner itself changed, and --changed runs every check, as it should). Times by the script's clock
How to reproduce: append a comment line to nx2/chunks.py, then python -I -B tests/run_all.py --changed HEAD; against python -I -B tests/run_all.py

	main @ 000f19fb	this PR @ f0982a1e
checks run for the change	521 (all)	9
time to the result (s), median of 2	201.1	17.8

main 1: the whole suite: 205.0 s, 483 passed, 5 failed, 33 skipped
pr 1:   --changed: 17.7 s, 1 files changed since HEAD, 9 checks read them; 9 passed, 1 failed
main 2: the whole suite: 197.2 s, 483 passed, 5 failed, 33 skipped
pr 2:   --changed: 17.9 s, 1 files changed since HEAD, 9 checks read them; 9 passed, 1 failed

The 9: test_chunk_split, test_chunks_save_folder, test_lift_cache, test_mark_groups, test_parallel_emit, test_spawn_emit, test_stale_hot_pieces, test_camera_speed and test_run_all_changed (which names nx2/chunks.py), with the byte-compile check. The failure both times is test_spawn_emit, which fails the same way on main (it is one of main's five).

Rebased since onto main 0c74253d: its four commits since 000f19fb are #177 (a log line's level, in the renderer's logging), #192, #193 and #194 (the README, the website, the Discord status), none in the runner or the checks measured here, so the numbers hold.

Testing

Tested: the runs above; tests/run_all.py, the whole suite, on this branch and on main 0c74253d, Windows (483 passed and 6 failed on main, 484 passed and 6 failed here, no failure main doesn't have: test_check_third_party, test_display_pointer, test_fpx_sticky, test_release, test_spawn_emit and test_texture_staging_publication fail on main too (the last since #177; #202 fixes it)); tests/test_run_all_changed.py (8 cases, the default base's with the old default put back: it fails).

Not tested: on Linux (the runner's --changed is git and Python only); a check that reads a changed file without naming it (by path, name or import) is left out of a --changed run by design, which is why it says a pull request still needs the whole run.

Needs other testers: None

Screenshots

Nothing on screen.

Risk and rollback

Only --changed picks checks; a run without it is unchanged. A check that reads a file without naming it (by path, name or import) would be left out of a --changed run, which is why it says a pull request still needs the whole run. Reverting the two commits is enough.

Reviewer notes

Carried over from https://github.com/NotRexed/Hookshot/pull/2887 with the move to Longshot; the second commit makes the default base Longshot's main. The dev-loop proposal's D1 (https://github.com/NotRexed/Hookshot/pull/2636).

Issues

None on Longshot. See https://github.com/NotRexed/Hookshot/pull/2636 (D1).

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
• Tested in the game, or what wasn't is named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• A game-free test added for the fix, failing on main and passing here.

🤖 Generated with Claude Code

## Why A full `tests/run_all.py` is about 490 checks and several minutes, and the ones an edit touches had to be picked by hand, which is how a check that cuts a function far from the edit gets missed (AGENTS.md: "Tests read the source"). ## What changed - tests: `tests/run_all.py --changed [BASE]`: `changed_files` takes the files changed since BASE's merge base with HEAD (`origin/main` by default), committed or not, plus the untracked ones, from git. `reads_changed` picks: - the checks whose text names one of them: by path, by file name standing alone, or as a module they import (`import chunks`, `from chunks import`); - the changed checks themselves (and the byte-compile check, as before); - every check when the runner itself or a shared `.py` under `tests/` that is not a test changed. - tests: with `--changed`, the C++ unit tests run only when something under `gpu/` changed or every check runs. It prints how many files changed and how many checks read them, and that a pull request still needs the whole run. `--changed` can't be combined with names or `--write-times`. Nothing else in the runner changes. - tests: with no BASE, `--changed` takes main on the remote that points at Longshot (`forge/main` where `scripts/forge.py` added it, as CONTRIBUTING.md has it), else `origin/main` (`default_base`): in a clone carried over from Hookshot, `origin` is GitHub (the second commit). - tests: `tests/test_run_all_changed.py` (new): which checks a changed file picks (by path, by name and not inside a longer name, by import), that a changed check runs itself, that the runner or a shared test file runs everything, that an unnamed file runs none, and the files from git (committed, uncommitted and untracked), with git's answers stood in; and the default base in three remote layouts (it fails with the old default). Example, on the GitHub branch (the runner changed): `--changed: 2 files changed since origin/main, every check (the runner or a shared test file changed)`. For a change to `nx2/chunks.py` alone it picks the chunk, emit and lift-cache checks (under a quarter of all), not the renderer's. ## Flags and settings None: no environment switch, menu row or ini key. The new `tests/run_all.py --changed [BASE]` option is the test runner's, described above. ## Measurements main: 000f19fb (2026-11-13) this PR: feat/run-all-changed @ f0982a1e Hardware: Ryzen 9 3900X, 64 GB, Windows 11 (RTX 3080; no game run: these are game-free checks) Scene: none: game-free. A one-line change to `nx2/chunks.py` (a comment, uncommitted), checked Method: two interleaved rounds (main, PR, main, PR), each run with the PC otherwise quiet and the bench lock held. main: the whole suite, the only run it has (`run_all.py --jobs 12`). This PR: `run_all.py --jobs 12 --changed HEAD` (the branch's own commits as the base, so only the edit counts; with this branch against main the runner itself changed, and `--changed` runs every check, as it should). Times by the script's clock How to reproduce: append a comment line to `nx2/chunks.py`, then `python -I -B tests/run_all.py --changed HEAD`; against `python -I -B tests/run_all.py` | | main @ 000f19fb | this PR @ f0982a1e | |---|---|---| | checks run for the change | 521 (all) | 9 | | time to the result (s), median of 2 | 201.1 | 17.8 | ``` main 1: the whole suite: 205.0 s, 483 passed, 5 failed, 33 skipped pr 1: --changed: 17.7 s, 1 files changed since HEAD, 9 checks read them; 9 passed, 1 failed main 2: the whole suite: 197.2 s, 483 passed, 5 failed, 33 skipped pr 2: --changed: 17.9 s, 1 files changed since HEAD, 9 checks read them; 9 passed, 1 failed ``` The 9: `test_chunk_split`, `test_chunks_save_folder`, `test_lift_cache`, `test_mark_groups`, `test_parallel_emit`, `test_spawn_emit`, `test_stale_hot_pieces`, `test_camera_speed` and `test_run_all_changed` (which names `nx2/chunks.py`), with the byte-compile check. The failure both times is `test_spawn_emit`, which fails the same way on main (it is one of main's five). Rebased since onto main 0c74253d: its four commits since 000f19fb are #177 (a log line's level, in the renderer's logging), #192, #193 and #194 (the README, the website, the Discord status), none in the runner or the checks measured here, so the numbers hold. ## Testing **Tested:** the runs above; `tests/run_all.py`, the whole suite, on this branch and on main 0c74253d, Windows (483 passed and 6 failed on main, 484 passed and 6 failed here, no failure main doesn't have: test_check_third_party, test_display_pointer, test_fpx_sticky, test_release, test_spawn_emit and test_texture_staging_publication fail on main too (the last since #177; #202 fixes it)); `tests/test_run_all_changed.py` (8 cases, the default base's with the old default put back: it fails). **Not tested:** on Linux (the runner's `--changed` is git and Python only); a check that reads a changed file without naming it (by path, name or import) is left out of a `--changed` run by design, which is why it says a pull request still needs the whole run. **Needs other testers:** None ## Screenshots Nothing on screen. ## Risk and rollback Only `--changed` picks checks; a run without it is unchanged. A check that reads a file without naming it (by path, name or import) would be left out of a `--changed` run, which is why it says a pull request still needs the whole run. Reverting the two commits is enough. ## Reviewer notes Carried over from https://github.com/NotRexed/Hookshot/pull/2887 with the move to Longshot; the second commit makes the default base Longshot's main. The dev-loop proposal's D1 (https://github.com/NotRexed/Hookshot/pull/2636). ## Issues None on Longshot. See https://github.com/NotRexed/Hookshot/pull/2636 (D1). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst changed title from feat(tests): run_all.py --changed runs the checks that read what changed to WIP: feat(tests): run_all.py --changed runs the checks that read what changed 2026-11-13 01:13:55 +00:00

dyst commented 2026-11-13 01:13:56 +00:00

Author

Member

• Copy link
• Copy Markdown

Back to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.

Back to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.

dyst commented 2026-11-13 01:58:21 +00:00

Author

Member

• Copy link
• Copy Markdown

Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready.

Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready.

dyst commented 2026-11-13 02:00:36 +00:00

Author

Member

• Copy link
• Copy Markdown

Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).

Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).

dyst commented 2026-11-13 11:45:33 +00:00

Author

Member

• Copy link
• Copy Markdown

The description follows the template after #76 and #106: If applicable keeps only the boxes that apply (one that applies and isn't done yet stays unticked while this is WIP), the Flags table has the template's seven columns (Goal is now Recommendation), and GitHub references are full URLs. Nothing else changed.

The description follows the template after #76 and #106: If applicable keeps only the boxes that apply (one that applies and isn't done yet stays unticked while this is WIP), the Flags table has the template's seven columns (Goal is now Recommendation), and GitHub references are full URLs. Nothing else changed.

dyst force-pushed feat/run-all-changed from b1936abba7 to 51dca071a9 2026-11-13 13:56:45 +00:00 Compare

dyst commented 2026-11-13 13:56:47 +00:00

Author

Member

• Copy link
• Copy Markdown

Rebased onto main 0c74253d, and a second commit: with no BASE, --changed takes main on the remote that points at Longshot (forge/main as forge.py adds it), not origin/main, which is GitHub in a clone carried over from Hookshot. Measured on Longshot: a one-line change to nx2/chunks.py runs 9 checks in 17.8 s against the whole suite's 521 in 201.1 s (medians of 2). Whole suite on this head and on main: no failure main doesn't have. Ready for review.

Rebased onto main 0c74253d, and a second commit: with no BASE, --changed takes main on the remote that points at Longshot (forge/main as forge.py adds it), not origin/main, which is GitHub in a clone carried over from Hookshot. Measured on Longshot: a one-line change to nx2/chunks.py runs 9 checks in 17.8 s against the whole suite's 521 in 201.1 s (medians of 2). Whole suite on this head and on main: no failure main doesn't have. Ready for review.

dyst changed title from WIP: feat(tests): run_all.py --changed runs the checks that read what changed to feat(tests): run_all.py --changed runs the checks that read what changed 2026-11-13 13:57:21 +00:00

mooki added the

kind

feature

label 2026-11-14 01:44:03 +00:00

mooki added the

priority

low

label 2026-11-14 04:17:11 +00:00

APusija04 referenced this pull request 2026-11-14 05:05:21 +00:00

[RX 6600] Game crashes with GPU timeout #365

Austin referenced this pull request 2026-11-14 17:33:36 +00:00

Game: #503

Austin referenced this pull request 2026-11-14 17:33:51 +00:00

Game: #504

Austin referenced this pull request 2026-11-14 17:34:07 +00:00

Game: #509

Austin referenced this pull request 2026-11-14 17:34:31 +00:00

Game: Longshot #510

Austin referenced this pull request 2026-11-14 17:34:39 +00:00

Game: #512

Austin referenced this pull request 2026-11-14 18:19:08 +00:00

Game: Switch Pro Controller issues when gamestreaming #535

mooki added the

gpu/nvidia

platform/linux

platform/windows

labels 2026-11-14 18:21:12 +00:00

Some checks failed

Hide checks

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

PR description / Description follows the template (pull_request_target) Has been cancelled

Details

Third-party parts / As their notes say (pull_request) Has been cancelled

Details

CI / Conventional Commit title (pull_request) Has been cancelled

Details

CI / Workflow lint (actionlint) (pull_request) Has been cancelled

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'feat/run-all-changed:dyst-feat/run-all-changed'

git switch 'dyst-feat/run-all-changed'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Affects Linux (a change for every system carries both Windows and Linux)

  

platform/macos

macOS (Apple silicon or Intel Macs)

  

platform/steam-deck

Steam Deck

  

platform/windows

Affects Windows (a change for every system carries both Windows and Linux)

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

  

ci: approved

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/macos

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

ci: approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!32

No description provided.