WIP: test(lift): the whole-function check without the game, over made-up functions #1592

Draft
dyst wants to merge 2 commits from dyst/Longshot:test/leaf-free into main
Member

Type: test · Risk: low · Tested on: Windows 11, Ryzen 9 3900X · Needs: a Linux run

Stacked on #1570 (it reuses that check's instruction draw and known list); review that first.

TL;DR

nx2/test_leaf.py checks whole translated functions against Unicorn, an AArch64 emulator, but it takes its functions from the game's own code, so the checks skip it everywhere. This makes the functions up instead, from a seeded generator: branches, loops, frames, calls, tail calls and float runs. It translates them through the same steps a real build uses, then compares every register and the memory they wrote. It runs with no game in about 40 s and catches five translator breaks made by hand.

Breakdown

  • Who it helps: anyone changing the translator. The CPU rework plan restructures it (milestones C4 and C6), and #1560 and #1469 already change it.
  • What was wrong: no game-free check ran whole functions. The instruction check (#1570) runs one instruction at a time, and the whole-function one needs the game.
  • What changes: tests/test_leaf_free.py (new).
  • What does not change: the translator, the game.
  • How it was checked: passes on this branch (300 functions, 4 states each); five breaks each make it fail.
  • What is still open: a Linux run (the harness has an mmap path, not run yet).
  • Risk: low: a check only.

Why

The CPU rework plan's milestone C0: before the translator's passes are restructured, a game-free check must run whole functions through the production emitter. Two passes now go through it that nx2/test_leaf.py leaves out: the fast float regions and the entry hooks.

What changed

tests/test_leaf_free.py (new):

  • Lifting: the passes nx2/lift.py installs, in its order. It compiles with the translated code's flags from nx2/build.py (A64_POLLING, A64_WRITE_TRACK) on the 64-bit guest window.

  • The image: a synthetic EdenImage, made the way tests/test_walk.py makes one, which the translator discovers as it would the game's.

  • The functions: straight runs of instructions drawn as #1570 draws them, leaving out the forms its known list names and Unicorn's signed LSE min/max bug, plus control-flow shapes:

    Shape In the default run
    Forward B.cond / CBZ / TBZ 387 / 192 / 204
    If/else 158
    Counted loop 299
    Block below the entry (the entry goto) 54
    Frame 156
    BL / BL in a loop / tail call 65 / 25 / 28
    Single-precision float run (fast float regions) 157
    Entry hook 79

    Loads and stores address a scratch area only. Flags are read only where every path has set them (the AAPCS64, as the translator's dead-flag pass assumes).

  • Compared, as nx2/test_leaf.py compares: X0-X30, SP, V0-V31 and every scratch page that changed. FPSR is reported apart (--fpsr makes it fail). It also checks that every written page outside the stack has its write-watch mark, that the entry hooks' call counts equal Unicorn's entries, and that no function halted.

  • A failure prints the function's disassembly, the state and the differing registers (--show-c adds the translated C), keeps its build folder and exits 1.

Measurements

Verdict: nothing to measure: a check only.

Flags and settings

None.

Testing

Tested: on Windows 11 with the Builder's Python 3.14 and clang (llvm-mingw). The default run (300 roots, 94 callees, 11,739 instructions) passes: 1,176 comparable cases, 0 differences, about 40 s on an idle PC. Each of five breaks made by hand fails it, each restored afterwards:

Break Roots failing (of 49-59)
ADD (immediate) emitted as SUB in nx2/insn.py 11
B.GE emitted as B.LT in the fast float emit 8
nx2/lift_fixes.py's entry goto left out 11-13
The LSE ops' write-watch mark left out 19 (written pages without their mark)
The fast float regions' AVX operands swapped 10

Not tested: Linux; tests/run_all.py in full.

Must be tested: Linux, any PC: python -I -B tests/test_leaf_free.py (Test Farm: linux).

Risks

What could break: nothing in the game. A slow PC takes longer (--count sets the size; --jobs compiles in parallel).

Turning it off: revert the commit.

Compatibility: nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. On #1570, on 310e7c9d.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). This check passes; the full suite not run yet.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

🤖 Generated with Claude Code

**Type:** test · **Risk:** low · **Tested on:** Windows 11, Ryzen 9 3900X · **Needs:** a Linux run Stacked on #1570 (it reuses that check's instruction draw and known list); review that first. ## TL;DR `nx2/test_leaf.py` checks whole translated functions against Unicorn, an AArch64 emulator, but it takes its functions from the game's own code, so the checks skip it everywhere. This makes the functions up instead, from a seeded generator: branches, loops, frames, calls, tail calls and float runs. It translates them through the same steps a real build uses, then compares every register and the memory they wrote. It runs with no game in about 40 s and catches five translator breaks made by hand. ## Breakdown - **Who it helps:** anyone changing the translator. The CPU rework plan restructures it (milestones C4 and C6), and #1560 and #1469 already change it. - **What was wrong:** no game-free check ran whole functions. The instruction check (#1570) runs one instruction at a time, and the whole-function one needs the game. - **What changes:** `tests/test_leaf_free.py` (new). - **What does not change:** the translator, the game. - **How it was checked:** passes on this branch (300 functions, 4 states each); five breaks each make it fail. - **What is still open:** a Linux run (the harness has an mmap path, not run yet). - **Risk:** low: a check only. ## Why The CPU rework plan's milestone C0: before the translator's passes are restructured, a game-free check must run whole functions through the production emitter. Two passes now go through it that `nx2/test_leaf.py` leaves out: the fast float regions and the entry hooks. ## What changed `tests/test_leaf_free.py` (new): - **Lifting:** the passes `nx2/lift.py` installs, in its order. It compiles with the translated code's flags from `nx2/build.py` (`A64_POLLING`, `A64_WRITE_TRACK`) on the 64-bit guest window. - **The image:** a synthetic `EdenImage`, made the way `tests/test_walk.py` makes one, which the translator discovers as it would the game's. - **The functions:** straight runs of instructions drawn as #1570 draws them, leaving out the forms its known list names and Unicorn's signed LSE min/max bug, plus control-flow shapes: | Shape | In the default run | |---|---| | Forward B.cond / CBZ / TBZ | 387 / 192 / 204 | | If/else | 158 | | Counted loop | 299 | | Block below the entry (the entry goto) | 54 | | Frame | 156 | | BL / BL in a loop / tail call | 65 / 25 / 28 | | Single-precision float run (fast float regions) | 157 | | Entry hook | 79 | Loads and stores address a scratch area only. Flags are read only where every path has set them (the AAPCS64, as the translator's dead-flag pass assumes). - **Compared, as `nx2/test_leaf.py` compares:** X0-X30, SP, V0-V31 and every scratch page that changed. FPSR is reported apart (`--fpsr` makes it fail). It also checks that every written page outside the stack has its write-watch mark, that the entry hooks' call counts equal Unicorn's entries, and that no function halted. - **A failure** prints the function's disassembly, the state and the differing registers (`--show-c` adds the translated C), keeps its build folder and exits 1. ## Measurements **Verdict:** nothing to measure: a check only. ### Flags and settings None. ## Testing **Tested:** on Windows 11 with the Builder's Python 3.14 and clang (llvm-mingw). The default run (300 roots, 94 callees, 11,739 instructions) passes: 1,176 comparable cases, 0 differences, about 40 s on an idle PC. Each of five breaks made by hand fails it, each restored afterwards: | Break | Roots failing (of 49-59) | |---|---| | ADD (immediate) emitted as SUB in `nx2/insn.py` | 11 | | B.GE emitted as B.LT in the fast float emit | 8 | | `nx2/lift_fixes.py`'s entry goto left out | 11-13 | | The LSE ops' write-watch mark left out | 19 (written pages without their mark) | | The fast float regions' AVX operands swapped | 10 | **Not tested:** Linux; `tests/run_all.py` in full. **Must be tested:** Linux, any PC: `python -I -B tests/test_leaf_free.py` (Test Farm: linux). ## Risks **What could break:** nothing in the game. A slow PC takes longer (`--count` sets the size; `--jobs` compiles in parallel). **Turning it off:** revert the commit. **Compatibility:** nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [ ] Rebased on current `main`. *On #1570, on `310e7c9d`.* - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). *This check passes; the full suite not run yet.* - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
test(lift): the instruction check without the game, over encodings drawn from the instruction set
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 10s
af006df03b
nx2/test_insn.py runs each instruction form the game uses in Unicorn and as lifted C, but it
samples the game's code, so tests/run_all.py skips it everywhere. tests/test_insn_free.py draws
its encodings instead (seeded random words in the load/store, data-processing and SIMD/FP
classes, kept when classify takes them as plain ops and Capstone decodes them, grouped by form
as insn_test groups the game's) and runs insn_test unchanged over them, with nx2/test_insn.py's
fixes: 1,629 forms, 16,998 cases in about 50 s on a Ryzen 9 3900X.

263 forms fail on main and are listed in tests/insn_free_known.txt: forms the translator leaves
unhandled, which the game does not use, and the signed LSE min/max forms, where Unicorn 2.1.4
compares unsigned (nx2/insn.py's C is right). A form that fails and is not listed fails the check
(LDEOR written as an OR: 12 new failing forms); a listed form that passes is named.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
test(lift): the whole-function check without the game, over made-up functions
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 10s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 14s
Third-party parts / As their notes say (pull_request) Successful in 15s
f7d558ee40
nx2/test_leaf.py compares whole lifted functions against Unicorn but lifts them out of the game's modules, so
tests/run_all.py skips it everywhere. tests/test_leaf_free.py makes the functions from a seeded generator and lifts them
through the production emitter (the passes nx2/lift.py installs, in its order, fast float regions and entry hooks
included), on a synthetic image, then compares X0-X30, SP, V0-V31 and the changed scratch pages, and checks write-watch
marks and the hooks' call counts.

Shapes: forward B.cond / CBZ / TBZ, if/else, counted loops, a block below the entry (the entry goto), frames, BL and BL in
a loop, tail calls, single-precision float runs. The default run (300 roots, 4 states) passes in about 40 s. Five breaks
made by hand each fail it: ADD emitted as SUB, B.GE as B.LT in the fast float emit, the entry goto dropped, the LSE ops'
mark dropped, the fast float AVX operands swapped.

The CPU rework plan's milestone C0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 10s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 14s
Third-party parts / As their notes say (pull_request) Successful in 15s
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'test/leaf-free:dyst-test/leaf-free'
git switch 'dyst-test/leaf-free'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1592
No description provided.