test(lift): test_vbytes and test_vfpx_elem run in seconds, not minutes (fewer random states) #1876

Open
Veix wants to merge 1 commit from Veix/Longshot:fix/faster-vector-checks into main
Contributor

Summary

tests/test_vbytes.py takes about 25 minutes, so it times out in tests/run_all.py (600 s a check). This brings it down to 28 s, and tests/test_vfpx_elem.py from 67 s to 11 s, by running fewer random states. What they check is unchanged.

test_vbytes ran 1,000,000 random states for each of its 130 guest words, on four builds. Now it runs 10,000, which is still 1.3 million checks a build, with the index bytes biased to the edges as before. A planted off-by-one in a64_tbl1 (0x70 as 0x71, so index 15 reads 0) is caught at once at 10,000: 91,240 mismatches. test_vfpx_elem goes from 2,000,000 to 200,000 states; its sweep of every pair of special values is unchanged.

Both checks came with #1576 (mine); the defaults were never meant to take this long.

See #1576

Testing

Tested: Windows 11, Ryzen 9 9900X: python -I -B tests/run_all.py test_vbytes test_vfpx_elem test_fpx_sticky passes (28.1, 10.8 and 18.3 s). With a64_tbl1's 0x70 changed to 0x71, test_vbytes fails with 91,240 mismatches. No game run: only the checks change.

Still to test:

  • linux: python -I -B tests/run_all.py test_vbytes test_vfpx_elem, expecting both to pass well under 600 s (@Beanstalk, or by hand)

Measurements

Baseline: main @ 89c3d21e, the same checks at their old defaults on the same PC.

Verdict: test_vbytes 25 min (it timed out at 600 s in the suite) to 28 s, test_vfpx_elem 67 to 11 s.

Before: main @ 89c3d21e After: this PR @ 138723ce
test_vbytes timed out at 600 s in the full suite (about 25 min alone: 13 s fixed, 15 s per 10,000 states) 28.1 s
test_vfpx_elem 67.0 s in the full suite 10.8 s

Risks

What could break: a bug that shows only in a rare random state the smaller count misses; the edges are biased in, a planted off-by-one fails at once, and --random still runs any count by hand. Turning it off: --random 1000000 (or 2000000) as before. After updating: nothing changes for players.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says.
  • mise run test passes (any failure that also fails on main is named). [on Windows: the checks this changes pass; the 50 others that fail on this PC fail the same on main @ 89c3d21e]
  • Measured against main, or Measurements says why there is nothing to measure.
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

Written with help from Claude Code (Claude Opus 5.5).

## Summary `tests/test_vbytes.py` takes about 25 minutes, so it times out in `tests/run_all.py` (600 s a check). This brings it down to 28 s, and `tests/test_vfpx_elem.py` from 67 s to 11 s, by running fewer random states. What they check is unchanged. `test_vbytes` ran 1,000,000 random states for each of its 130 guest words, on four builds. Now it runs 10,000, which is still 1.3 million checks a build, with the index bytes biased to the edges as before. A planted off-by-one in `a64_tbl1` (`0x70` as `0x71`, so index 15 reads 0) is caught at once at 10,000: 91,240 mismatches. `test_vfpx_elem` goes from 2,000,000 to 200,000 states; its sweep of every pair of special values is unchanged. Both checks came with #1576 (mine); the defaults were never meant to take this long. See #1576 ## Testing **Tested:** Windows 11, Ryzen 9 9900X: `python -I -B tests/run_all.py test_vbytes test_vfpx_elem test_fpx_sticky` passes (28.1, 10.8 and 18.3 s). With `a64_tbl1`'s `0x70` changed to `0x71`, `test_vbytes` fails with 91,240 mismatches. No game run: only the checks change. **Still to test:** - linux: `python -I -B tests/run_all.py test_vbytes test_vfpx_elem`, expecting both to pass well under 600 s (@Beanstalk, or by hand) ## Measurements Baseline: main @ 89c3d21e, the same checks at their old defaults on the same PC. **Verdict:** `test_vbytes` 25 min (it timed out at 600 s in the suite) to 28 s, `test_vfpx_elem` 67 to 11 s. | | Before: main @ 89c3d21e | After: this PR @ 138723ce | |---|---|---| | `test_vbytes` | timed out at 600 s in the full suite (about 25 min alone: 13 s fixed, 15 s per 10,000 states) | 28.1 s | | `test_vfpx_elem` | 67.0 s in the full suite | 10.8 s | ## Risks What could break: a bug that shows only in a rare random state the smaller count misses; the edges are biased in, a planted off-by-one fails at once, and `--random` still runs any count by hand. Turning it off: `--random 1000000` (or `2000000`) as before. After updating: nothing changes for players. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [x] Tested on my own PC, in the game for a change the game runs, as Testing says. - [x] `mise run test` passes (any failure that also fails on `main` is named). [on Windows: the checks this changes pass; the 50 others that fail on this PC fail the same on `main` @ 89c3d21e] - [x] Measured against `main`, or Measurements says why there is nothing to measure. - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. Written with help from Claude Code (Claude Opus 5.5).
test(lift): test_vbytes and test_vfpx_elem run in seconds, not minutes (fewer random states)
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
138723ce7e
test_vbytes ran 1,000,000 random states for each of its 130 guest words on four builds: about 25 minutes, past
run_all.py's 600 s limit. 10,000 states (1.3 million checks a build) take 28 s and still catch a planted off-by-one in
a64_tbl1 at once. test_vfpx_elem goes from 2,000,000 to 200,000 states (67 s to 11 s); its sweep of special values is
unchanged. Both came with #1576.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • PR #1818 WIP: test: pytest runs the game-free checks; requirements by machine; main's known failures — Touches test_vbytes runtime; its runner already lowers random count
Possibly related, for context: - PR #1818 WIP: test: pytest runs the game-free checks; requirements by machine; main's known failures — Touches test_vbytes runtime; its runner already lowers random count
Author
Contributor

#1818 also lowers test_vbytes' count, for its runner (50,000 states); this is the small fix that can land first. If both land, #1818's RUN_ALL_ARGS line could go or use 10,000: a planted off-by-one in a64_tbl1 fails at once at 10,000.

**#1818 also lowers `test_vbytes`' count, for its runner (50,000 states); this is the small fix that can land first.** If both land, #1818's `RUN_ALL_ARGS` line could go or use 10,000: a planted off-by-one in `a64_tbl1` fails at once at 10,000.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/faster-vector-checks:Veix-fix/faster-vector-checks'
git switch 'Veix-fix/faster-vector-checks'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1876
No description provided.