WIP: test(lift): the instruction check without the game, over encodings drawn from the instruction set #1570

Draft
dyst wants to merge 1 commit from dyst/Longshot:test/insn-free into main
Member

Type: test · Risk: low · Tested on: Windows 11, Ryzen 9 3900X · Needs: a run on Linux (Capstone's form names there)

TL;DR

The check that runs every instruction form in an emulator and as translated C, and compares them, only works with the game's own files, so the automatic checks always skip it. This adds a version that draws its instructions from the instruction set instead, with nothing from the game: 1,629 forms and 16,998 cases in about 50 seconds. Forms that already fail on main are listed, so only a new failure fails it. On the way it showed that the emulator, not our translation, gets the signed atomic min and max wrong.

Breakdown

  • Who it helps: anyone changing the translator, and every PC that runs tests/run_all.py: the instruction check now runs there.
  • What was wrong: nx2/test_insn.py samples the game's code, so run_all.py skips it on every system; translator changes were checked by hand on a PC with the game, or not at all.
  • What changes: tests/test_insn_free.py draws seeded random encodings in the load/store, data-processing and SIMD/FP classes and runs a64-kit's insn_test unchanged over them, with nx2/test_insn.py's fixes.
  • What does not change: the game, the translator, nx2/test_insn.py.
  • How it was checked: it passes on main with 263 forms listed as known failures; a translator broken by hand (LDEOR written as an OR) fails it with 12 new failing forms.
  • What is still open: a run on Linux, where Capstone's form names may differ from Windows'.
  • Risk: low: a new check only.

Why

The translator is changed often (#1469, #1507, #1202, #1185 and others are open), and its only instruction-level check needs a dump, so tests/run_all.py never runs it. The wiki's CPU rework plan makes a game-free version its first tool, so that every later change to what the lifter emits is checked on every PC and on the forge.

What changed

  • tests/test_insn_free.py (new): draws up to --per (2) encodings of each form, grouped by mnemonic and vector arrangement as insn_test.sample groups the game's, from --tries (400,000) seeded random words kept when a64_lift.classify takes them as plain ops, Capstone decodes them and they are not one of nx2/insn.py's identity forms (pointer authentication, MTE); then insn_test.main over them with --states (6) register states each, on Unicorn's ARMv8 "max" CPU with nx2/test_insn.py's memory-operand fix. A failing form not in the known list fails the check; a listed form that passes is named. --write-known writes the list.
  • tests/insn_free_known.txt (new): the 263 forms that fail on main. Most are forms the translator leaves unhandled (halted 7), which the game does not use. The signed LSE min/max forms (ldsmax*, ldsmin*, stsmax*, stsmin*) are Unicorn's: version 2.1.4 compares them unsigned (memory -1, register 1: its LDSMAX writes -1, where the answer is 1); nx2/insn.py's C is right.

Measurements

Verdict: nothing to measure in the game (a check only); it runs in about 50 s on a Ryzen 9 3900X.

instruction check without the game: 16998 cases, 1366 forms pass, 263 forms with failures; 263 forms fail, 263 of them listed as known
instruction check without the game: ok

Flags and settings

None.

Testing

Tested: on Windows 11 with the Builder's Python 3.14, Unicorn 2.1.4 and Capstone 5.0.7: passes on 310e7c9d. LDEOR's lowering in nx2/insn.py changed by hand to an OR: FAIL (12 new failing forms); put back: passes. Unicorn's signed min/max checked on its own (LDSMAX with memory -1 and register 1 writes -1).

Not tested: Linux; tests/run_all.py in full.

Must be tested:

  • Linux, any PC: python -I -B tests/test_insn_free.py, expecting ok (Capstone's names for a form may differ: then the known list needs those names too).

Risks

What could break: nothing in the game. The check could fail on a PC whose Capstone or Unicorn names or runs a form differently.

Turning it off: revert the commit.

Compatibility: nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. On 310e7c9d.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). This check passes; the full suite not run yet.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

🤖 Generated with Claude Code

**Type:** test · **Risk:** low · **Tested on:** Windows 11, Ryzen 9 3900X · **Needs:** a run on Linux (Capstone's form names there) ## TL;DR The check that runs every instruction form in an emulator and as translated C, and compares them, only works with the game's own files, so the automatic checks always skip it. This adds a version that draws its instructions from the instruction set instead, with nothing from the game: 1,629 forms and 16,998 cases in about 50 seconds. Forms that already fail on main are listed, so only a new failure fails it. On the way it showed that the emulator, not our translation, gets the signed atomic min and max wrong. ## Breakdown - **Who it helps:** anyone changing the translator, and every PC that runs `tests/run_all.py`: the instruction check now runs there. - **What was wrong:** `nx2/test_insn.py` samples the game's code, so `run_all.py` skips it on every system; translator changes were checked by hand on a PC with the game, or not at all. - **What changes:** `tests/test_insn_free.py` draws seeded random encodings in the load/store, data-processing and SIMD/FP classes and runs a64-kit's `insn_test` unchanged over them, with `nx2/test_insn.py`'s fixes. - **What does not change:** the game, the translator, `nx2/test_insn.py`. - **How it was checked:** it passes on main with 263 forms listed as known failures; a translator broken by hand (LDEOR written as an OR) fails it with 12 new failing forms. - **What is still open:** a run on Linux, where Capstone's form names may differ from Windows'. - **Risk:** low: a new check only. ## Why The translator is changed often (#1469, #1507, #1202, #1185 and others are open), and its only instruction-level check needs a dump, so `tests/run_all.py` never runs it. The wiki's CPU rework plan makes a game-free version its first tool, so that every later change to what the lifter emits is checked on every PC and on the forge. ## What changed - `tests/test_insn_free.py` (new): draws up to `--per` (2) encodings of each form, grouped by mnemonic and vector arrangement as `insn_test.sample` groups the game's, from `--tries` (400,000) seeded random words kept when `a64_lift.classify` takes them as plain ops, Capstone decodes them and they are not one of `nx2/insn.py`'s identity forms (pointer authentication, MTE); then `insn_test.main` over them with `--states` (6) register states each, on Unicorn's ARMv8 "max" CPU with `nx2/test_insn.py`'s memory-operand fix. A failing form not in the known list fails the check; a listed form that passes is named. `--write-known` writes the list. - `tests/insn_free_known.txt` (new): the 263 forms that fail on main. Most are forms the translator leaves unhandled (halted 7), which the game does not use. The signed LSE min/max forms (`ldsmax*`, `ldsmin*`, `stsmax*`, `stsmin*`) are Unicorn's: version 2.1.4 compares them unsigned (memory -1, register 1: its `LDSMAX` writes -1, where the answer is 1); `nx2/insn.py`'s C is right. ## Measurements **Verdict:** nothing to measure in the game (a check only); it runs in about 50 s on a Ryzen 9 3900X. ``` instruction check without the game: 16998 cases, 1366 forms pass, 263 forms with failures; 263 forms fail, 263 of them listed as known instruction check without the game: ok ``` ### Flags and settings None. ## Testing **Tested:** on Windows 11 with the Builder's Python 3.14, Unicorn 2.1.4 and Capstone 5.0.7: passes on `310e7c9d`. LDEOR's lowering in `nx2/insn.py` changed by hand to an OR: `FAIL (12 new failing forms)`; put back: passes. Unicorn's signed min/max checked on its own (`LDSMAX` with memory -1 and register 1 writes -1). **Not tested:** Linux; `tests/run_all.py` in full. **Must be tested:** - Linux, any PC: `python -I -B tests/test_insn_free.py`, expecting `ok` (Capstone's names for a form may differ: then the known list needs those names too). ## Risks **What could break:** nothing in the game. The check could fail on a PC whose Capstone or Unicorn names or runs a form differently. **Turning it off:** revert the commit. **Compatibility:** nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [ ] Rebased on current `main`. *On `310e7c9d`.* - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). *This check passes; the full suite not run yet.* - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
test(lift): the instruction check without the game, over encodings drawn from the instruction set
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 10s
af006df03b
nx2/test_insn.py runs each instruction form the game uses in Unicorn and as lifted C, but it
samples the game's code, so tests/run_all.py skips it everywhere. tests/test_insn_free.py draws
its encodings instead (seeded random words in the load/store, data-processing and SIMD/FP
classes, kept when classify takes them as plain ops and Capstone decodes them, grouped by form
as insn_test groups the game's) and runs insn_test unchanged over them, with nx2/test_insn.py's
fixes: 1,629 forms, 16,998 cases in about 50 s on a Ryzen 9 3900X.

263 forms fail on main and are listed in tests/insn_free_known.txt: forms the translator leaves
unhandled, which the game does not use, and the signed LSE min/max forms, where Unicorn 2.1.4
compares unsigned (nx2/insn.py's C is right). A form that fails and is not listed fails the check
(LDEOR written as an OR: 12 new failing forms); a listed form that passes is named.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 10s
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'test/insn-free:dyst-test/insn-free'
git switch 'dyst-test/insn-free'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1570
No description provided.