test(lift): scripts/lift_parity.py, the translation of two checkouts compared byte for byte #1798

Open
dyst wants to merge 2 commits from dyst/Longshot:test/lift-parity into main
Member

Type: test · Risk: low · Tested on: Windows 11, RTX 3080, Ryzen 9 3900X · Needs: a run on Linux

TL;DR

A refactor of the translator is only safe when it can be shown not to change the translated game. This adds a tool that translates the game from two checkouts the way a real build does, and compares the two byte for byte, naming any functions that differ without printing their code. It is the Modding API Plan's first parity gate (M1, gate 1), and it is what the hook compiler's pull requests use to show which functions they change. Two translations of main came out identical (91 files, 1.73 GB), so the lift is stable enough to gate on.

Breakdown

  • Who it helps: anyone changing nx2/, the hook compiler first (milestone M2): a refactor shows "the same", a converted feature shows exactly the functions it changes.
  • What was wrong: the comparison existed only as local scripts, lifting without the hot list and the hot pieces a real build has.
  • What changes: scripts/lift_parity.py (lift, compare) and its game-free test.
  • What does not change: the game, the build and the lift.
  • How it was checked: the new test; main lifted twice and compared (the same); #1587 against a branch with an empty hook-site table (the same); a branch converting one feature (exactly its one function named).
  • What is still open: a run on Linux.
  • Risk: low: a tool and a test, nothing the game or the build runs.

Why

The Modding API Plan (wiki, Modding-API-Plan, section 4, M1) asks for parity gates before the first refactor of the core: gate 1 is code parity, the translation unchanged with a hook-site table present and empty. The plan names nx2/test_leaf.py for it, which checks a sample of functions against Unicorn; a byte-for-byte comparison of the whole translation is a stronger check for a change that must change nothing, and the only one that says which functions a deliberate change touched.

What changed

A script with two commands, and a test of the second.

  • scripts/lift_parity.py lift [--out DIR] [--exefs DIR] [--checkout DIR]: a checkout's translation, written as nx2/make_game.py writes it: nx2/lift.py with make_game's environment (RX_RELEASE=1, A64_HOT_CHUNK=1, the hot list hot_list_for gives), then nx2/split_hot.py; through the job queue as a build, with scripts/dev_build.py's Python and tools (on Windows the Builder's Python and llvm-mingw: what a base or a pack lifts with). It writes its settings into the folder (lift_parity.txt: the Python, and a file setting by its contents' hash).
  • scripts/lift_parity.py compare A B: every file under sa/ and image/ byte for byte (keys, timings and logs left out); a differing C file compared function by function (nx2/lifted_c.py's definitions), the functions named; different settings said. Exit 0: the same.
  • tests/test_lift_parity.py (new): two made-up translations: equal ones the same; a changed function named, and only it; files only on one side, a changed image file and different settings said; keys and logs not compared; no function's code printed.

Measurements

Verdict: nothing to measure: a tool; what it found is in Testing.

Flags and settings

None.

Testing

Tested:

  • python -I -B tests/test_lift_parity.py passes.
  • On the game (rc.8's ExeFS, the Builder's Python and llvm-mingw): main ad9fa48e lifted twice, ~4 minutes each: 91 files, 1.73 GB, the same.
  • #1587's head (ab68fc1a) against a branch adding the hook compiler with OoT's site table present and empty: the same.
  • That branch's next commit, text speed moved from a text patch to a hook site: 1 of 2,994 functions in its file named (f_80b1e140), plus the new site table file and the chunk cost estimates.
  • python -I -B tests/run_all.py (the Builder's llvm-mingw first on PATH, LONGSHOT_MINGW set): 693 passed / 41 failed on the branch before its rebase onto main a433db32; the same 41 checks fail on a433db32, rerun there, and none fails on the branch only. After the rebase (which touches none of its files) its own test passes again.

Not tested: Linux.

Must be tested:

  • Linux, any PC with the game: lift twice from one checkout and compare, expecting "the same" (by hand, or the Test Farm on a Linux machine).

Risks

What could break: nothing the game runs; a wrong comparison would show in its test.

Turning it off: delete the script.

Compatibility: none.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

🤖 Generated with Claude Code

**Type:** test · **Risk:** low · **Tested on:** Windows 11, RTX 3080, Ryzen 9 3900X · **Needs:** a run on Linux ## TL;DR A refactor of the translator is only safe when it can be shown not to change the translated game. This adds a tool that translates the game from two checkouts the way a real build does, and compares the two byte for byte, naming any functions that differ without printing their code. It is the Modding API Plan's first parity gate (M1, gate 1), and it is what the hook compiler's pull requests use to show which functions they change. Two translations of main came out identical (91 files, 1.73 GB), so the lift is stable enough to gate on. ## Breakdown - **Who it helps:** anyone changing `nx2/`, the hook compiler first (milestone M2): a refactor shows "the same", a converted feature shows exactly the functions it changes. - **What was wrong:** the comparison existed only as local scripts, lifting without the hot list and the hot pieces a real build has. - **What changes:** `scripts/lift_parity.py` (`lift`, `compare`) and its game-free test. - **What does not change:** the game, the build and the lift. - **How it was checked:** the new test; main lifted twice and compared (the same); #1587 against a branch with an empty hook-site table (the same); a branch converting one feature (exactly its one function named). - **What is still open:** a run on Linux. - **Risk:** low: a tool and a test, nothing the game or the build runs. ## Why The Modding API Plan (wiki, Modding-API-Plan, section 4, M1) asks for parity gates before the first refactor of the core: gate 1 is code parity, the translation unchanged with a hook-site table present and empty. The plan names `nx2/test_leaf.py` for it, which checks a sample of functions against Unicorn; a byte-for-byte comparison of the whole translation is a stronger check for a change that must change nothing, and the only one that says which functions a deliberate change touched. ## What changed A script with two commands, and a test of the second. - `scripts/lift_parity.py lift [--out DIR] [--exefs DIR] [--checkout DIR]`: a checkout's translation, written as `nx2/make_game.py` writes it: `nx2/lift.py` with make_game's environment (`RX_RELEASE=1`, `A64_HOT_CHUNK=1`, the hot list `hot_list_for` gives), then `nx2/split_hot.py`; through the job queue as a build, with `scripts/dev_build.py`'s Python and tools (on Windows the Builder's Python and llvm-mingw: what a base or a pack lifts with). It writes its settings into the folder (`lift_parity.txt`: the Python, and a file setting by its contents' hash). - `scripts/lift_parity.py compare A B`: every file under `sa/` and `image/` byte for byte (keys, timings and logs left out); a differing C file compared function by function (`nx2/lifted_c.py`'s definitions), the functions named; different settings said. Exit 0: the same. - `tests/test_lift_parity.py` (new): two made-up translations: equal ones the same; a changed function named, and only it; files only on one side, a changed image file and different settings said; keys and logs not compared; no function's code printed. ## Measurements **Verdict:** nothing to measure: a tool; what it found is in Testing. ### Flags and settings None. ## Testing **Tested:** - `python -I -B tests/test_lift_parity.py` passes. - On the game (rc.8's ExeFS, the Builder's Python and llvm-mingw): main ad9fa48e lifted twice, ~4 minutes each: 91 files, 1.73 GB, the same. - #1587's head (ab68fc1a) against a branch adding the hook compiler with OoT's site table present and empty: the same. - That branch's next commit, text speed moved from a text patch to a hook site: 1 of 2,994 functions in its file named (`f_80b1e140`), plus the new site table file and the chunk cost estimates. - `python -I -B tests/run_all.py` (the Builder's llvm-mingw first on `PATH`, `LONGSHOT_MINGW` set): 693 passed / 41 failed on the branch before its rebase onto main a433db32; the same 41 checks fail on a433db32, rerun there, and none fails on the branch only. After the rebase (which touches none of its files) its own test passes again. **Not tested:** Linux. **Must be tested:** - Linux, any PC with the game: `lift` twice from one checkout and `compare`, expecting "the same" (by hand, or the Test Farm on a Linux machine). ## Risks **What could break:** nothing the game runs; a wrong comparison would show in its test. **Turning it off:** delete the script. **Compatibility:** none. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
test(lift): scripts/lift_parity.py, the translation of two checkouts compared byte for byte
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 10s
PR description / Description follows the template (pull_request_target) Successful in 12s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 14s
CI / Workflow lint (actionlint) (pull_request) Successful in 16s
Third-party parts / As their notes say (pull_request) Successful in 17s
Website / Website on Cloudflare (pull_request_target) Has been skipped
74f67e767b
The Modding API Plan's gate 1 (code parity): `lift` writes a checkout's translation of the player's game as
nx2/make_game.py writes it (its environment, the hot list it would choose, split_hot's pieces), through the job queue;
`compare` checks two of them byte for byte and names the functions that differ, never their code.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member

Possibly related, for context:

  • PR #1560 refactor(lift): one emit_function with hook points, in place of fastfp's copy and three wrappers — Whole-game byte comparison already used for the overlapping lift parity gate
  • PR #1592 WIP: test(lift): the whole-function check without the game, over made-up functions — Adjacent translator-testing effort; game-free whole-function equivalence checks
Possibly related, for context: - PR #1560 refactor(lift): one emit_function with hook points, in place of fastfp's copy and three wrappers — Whole-game byte comparison already used for the overlapping lift parity gate - PR #1592 WIP: test(lift): the whole-function check without the game, over made-up functions — Adjacent translator-testing effort; game-free whole-function equivalence checks
fix(scripts): lift_parity lifts with dev_build.py's Python and tools
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 15s
PR description / Description follows the template (pull_request_target) Successful in 5s
c271dd2bfc
A checkout with no translator extension built yet stopped at its build (lld not found: LONGSHOT_MINGW unset), and a
Python without capstone stopped at the lift's import: lift now takes scripts/dev_build.py's environment, the one a base
or a pack lifts with (on Windows the Builder's Python and llvm-mingw), and records the Python in lift_parity.txt, so a
comparison of two lifts made with different ones says so.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
futondestroyer left a comment

Verdict: Request changes. “Parity” can currently be reported for incomplete or non-independent outputs or for lifts with different effective inputs.

Reviewed c271dd2b against a433db32.

  • scripts/lift_parity.py:82–114,198: require both sides to produce the complete expected output set and independently own their outputs before comparing. Missing artifacts on both sides, stale outputs and shared output roots need to fail explicitly.
  • scripts/lift_parity.py:57–77: preserve or validate the effective lift inputs on both sides, including settings and inherited overrides. Report the mismatch before claiming byte parity.
  • scripts/lift_parity.py:129,169: retain the file identity in function reporting. A function moving between files needs a correct per-file report even when its body text is unchanged.

Add game-free fixtures for those failure modes and a positive comparison from two independent complete runs. Existing comparison evidence applies to the author’s tested configuration.

**Verdict: Request changes.** “Parity” can currently be reported for incomplete or non-independent outputs or for lifts with different effective inputs. Reviewed `c271dd2b` against `a433db32`. - `scripts/lift_parity.py:82–114,198`: require both sides to produce the complete expected output set and independently own their outputs before comparing. Missing artifacts on both sides, stale outputs and shared output roots need to fail explicitly. - `scripts/lift_parity.py:57–77`: preserve or validate the effective lift inputs on both sides, including settings and inherited overrides. Report the mismatch before claiming byte parity. - `scripts/lift_parity.py:129,169`: retain the file identity in function reporting. A function moving between files needs a correct per-file report even when its body text is unchanged. Add game-free fixtures for those failure modes and a positive comparison from two independent complete runs. Existing comparison evidence applies to the author’s tested configuration.
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Third-party parts / As their notes say (pull_request) Successful in 15s
PR description / Description follows the template (pull_request_target) Successful in 5s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'test/lift-parity:dyst-test/lift-parity'
git switch 'dyst-test/lift-parity'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1798
No description provided.