test: the two vector rewrite checks run 10,000 random states by default #2139

Merged
HumanBean merged 1 commit from Veix/Longshot:test/vector-tests-random-10000 into main 2026-11-21 22:27:07 +00:00
Contributor

Summary

The two vector rewrite checks from #1576 run 10,000 random states by default instead of 1,000,000 and 2,000,000: with the old counts the suite's run took about 28 minutes on my PC because of them.

tests/test_vbytes.py ran 1,000,000 random states for each of 130 guest words in four builds, and tests/test_vfpx_elem.py 2,000,000 for each of 6 guest words in three builds, every time tests/run_all.py ran. I chose those counts in #1576 and they are far more than the checks need. The default of --random is now 10,000 in both; a longer run is still one flag away (--random 1000000) for someone changing the helpers in nx2/fastvec.py. The fixed sweep of test_vfpx_elem.py and both timing parts are unchanged.

Read first: the --random default in the two files; nothing else changes.

See #1576

Testing

Tested: Windows 11, Ryzen 9 9900X, llvm-mingw clang. Both checks pass at the new default: test_vbytes.py in 28 s (1,300,000 checks a build, 0 mismatches), test_vfpx_elem.py in 5 s.

That 10,000 states still catch a fault: nine deliberate faults put into the helpers in nx2/fastvec.py, one at a time, each followed by the check at its new default. Eight are caught: TBL reading the table for an index of 16 to 31 (53,898 mismatches of 1,300,000), TBL's 8-byte form leaving the top half (49,164), the fallback's rule at index 16 exactly (10,659), EXT's sources swapped, EXT's 8-byte shift a byte short (189,923), the multiply's smallest-normal constant off by one bit (1 in the random part, 88 in the sweep), a tiny product under FZ keeping its value (873 and 326), a non-finite product of a zero kept (273 and 1,201). The ninth (a product exactly the smallest normal no longer sent to the scalar helper) passes at 10,000 and passes the same way at the old 2,000,000, so the lower count loses nothing there.

The checks: tests/run_all.py --changed (the checks that read the changed files) passes, 3 of 3. The whole suite does not run to its end on my PC (its runner is interrupted partway, on main as well): of the 364 checks it reached, 323 pass and 41 fail, and those 41 fail the same way run by name on main @ c7bd8f80; none of them reads these two files.

Still to test: None

Measurements

Nothing to measure: two checks' default counts, no code the game runs.

Risks

A fault in the helpers that shows only on an input rarer than about one in 10,000 random states is no longer found by the default run; --random with a larger count finds it as before. Nothing changes for players. A revert is enough.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named).
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

Written with AI assistance (Claude Code); run and checked on my own PC.

🤖 Generated with Claude Code

## Summary **The two vector rewrite checks from #1576 run 10,000 random states by default instead of 1,000,000 and 2,000,000: with the old counts the suite's run took about 28 minutes on my PC because of them.** `tests/test_vbytes.py` ran 1,000,000 random states for each of 130 guest words in four builds, and `tests/test_vfpx_elem.py` 2,000,000 for each of 6 guest words in three builds, every time `tests/run_all.py` ran. I chose those counts in #1576 and they are far more than the checks need. The default of `--random` is now 10,000 in both; a longer run is still one flag away (`--random 1000000`) for someone changing the helpers in `nx2/fastvec.py`. The fixed sweep of `test_vfpx_elem.py` and both timing parts are unchanged. Read first: the `--random` default in the two files; nothing else changes. See #1576 ## Testing **Tested:** Windows 11, Ryzen 9 9900X, llvm-mingw clang. Both checks pass at the new default: `test_vbytes.py` in 28 s (1,300,000 checks a build, 0 mismatches), `test_vfpx_elem.py` in 5 s. That 10,000 states still catch a fault: nine deliberate faults put into the helpers in `nx2/fastvec.py`, one at a time, each followed by the check at its new default. Eight are caught: TBL reading the table for an index of 16 to 31 (53,898 mismatches of 1,300,000), TBL's 8-byte form leaving the top half (49,164), the fallback's rule at index 16 exactly (10,659), EXT's sources swapped, EXT's 8-byte shift a byte short (189,923), the multiply's smallest-normal constant off by one bit (1 in the random part, 88 in the sweep), a tiny product under FZ keeping its value (873 and 326), a non-finite product of a zero kept (273 and 1,201). The ninth (a product exactly the smallest normal no longer sent to the scalar helper) passes at 10,000 and passes the same way at the old 2,000,000, so the lower count loses nothing there. The checks: `tests/run_all.py --changed` (the checks that read the changed files) passes, 3 of 3. The whole suite does not run to its end on my PC (its runner is interrupted partway, on `main` as well): of the 364 checks it reached, 323 pass and 41 fail, and those 41 fail the same way run by name on `main` @ c7bd8f80; none of them reads these two files. **Still to test:** None ## Measurements Nothing to measure: two checks' default counts, no code the game runs. ## Risks A fault in the helpers that shows only on an input rarer than about one in 10,000 random states is no longer found by the default run; `--random` with a larger count finds it as before. Nothing changes for players. A revert is enough. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [x] `mise run test` passes (any failure that also fails on `main` is named). - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. Written with AI assistance (Claude Code); run and checked on my own PC. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
test: the two vector rewrite checks run 10,000 random states by default, not 1,000,000 and 2,000,000
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
1a72e7cf03
tests/test_vbytes.py ran 1,000,000 random states for each of 130 guest words in four builds, tests/test_vfpx_elem.py 2,000,000 for each of 6 in three, every time the suite ran. The default is now 10,000; --random gives the long run by hand. The fixed sweep of test_vfpx_elem.py and both timing parts are unchanged.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
Member

Possibly related, for context:

  • PR #1876 test(lift): test_vbytes and test_vfpx_elem run in seconds, not minutes (fewer random states) — Open overlapping change lowers the same test defaults
Possibly related, for context: - PR #1876 test(lift): test_vbytes and test_vfpx_elem run in seconds, not minutes (fewer random states) — Open overlapping change lowers the same test defaults
HumanBean deleted branch test/vector-tests-random-10000 2026-11-21 22:27:07 +00:00
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Some workflows are waiting to be reviewed.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!2139
No description provided.