WIP: fix(lift): with local registers, a function with code below its entry loads them before its entry #1587

Draft
dyst wants to merge 3 commits from dyst/Longshot:fix/lift-entry-local-regs into main
Member

Type: fix · Risk: low · Tested on: Windows 11, RTX 3080, Ryzen 9 3900X (game-free; the default lift compared byte for byte) · Needs: none for the default build; a game run with local registers on, any PC

TL;DR

With the translator's local registers turned on, the 26 game functions whose code starts below their entry point began by jumping over the lines that load their registers, so they ran on garbage values. It never happened in a normal build, where local registers are off, but pull request #1469 turns them on by default. This makes those functions load their registers first. A normal build's translated game is byte for byte the same; a new game-free test fails on main and passes here.

Breakdown

  • Who it helps: any build with A64_LOCAL_REGS or A64_LOCAL_FLAGS set, and #1469, which turns them on by default.
  • What was wrong: the goto that starts such a function at its entry was put in after the head, before the register locals' declarations, so it jumped over their loads.
  • What changes: the head lines go in before the kit declares the locals, so the locals are loaded before the goto.
  • What does not change: a default build (local registers off): its translated C is byte-identical.
  • How it was checked: tests/test_lift_entry_local_regs.py fails on main and passes here, and fails again with the fix's line taken out; the default lift compared byte for byte with main's.
  • What is still open: Linux.
  • Risk: low: in a default build nothing changes; with local registers, 26 functions start correctly.

Why

nx2/lift_fixes.py starts a function whose code lies below its entry with goto L_<entry>; right after the head (its fix 4; 26 functions in a lift of OoT). With A64_LOCAL_REGS or A64_LOCAL_FLAGS the kit's localize_regs declares the function's register locals right after the head too, each loaded from the context (uint64_t x0_ = c->x[0], ...;). The goto came first, so it jumped over those declarations: C allows that, but the locals are never loaded. With #1469's defaults each of those 26 functions ran on whatever the stack held.

Stacked on #1560, whose nx2/emit.py holds the head lines this moves.

How to reproduce

Seen on: any PC, game-free
Settings and switches: A64_LOCAL_REGS=6 at lift time
Steps: python -I -B tests/test_lift_entry_local_regs.py on main; or lift with A64_LOCAL_REGS=6 and read any function nx2/lift_fixes.py gives an entry goto
Expected: the locals' declarations, then the goto
Actual: the goto, then the declarations it jumps over

What changed

The head lines (entry hooks' call, the entry goto) now go into the function's lines before the kit's localize_regs runs, instead of being spliced into its finished text after it; localize_regs then declares the locals right after the head, ahead of them. A PLT stub, which the method writes and returns before that point, still gets them in its text.

  • nx2/emit.py: emit hands the head lines to emit_function, which puts them after the head before its last passes (two tagged lines); a PLT stub keeps the text splice.
  • tests/test_lift_entry_local_regs.py (new): a made-up function with code below its entry, emitted with local registers, local flags and both: every declaration that loads a local comes before the goto; without them the goto still follows the head.

Measurements

Verdict: nothing to measure for a default build: its translated game is byte-identical to main's. With local registers on (this fix and #1469's change, registers only), the game plays, and the frame rate is within noise of main's on this PC: +0.2% in Hyrule Field and +1.6% in the Market, over 3 interleaved pairs.

Before: main @ 310e7c9d (the base's lift) After: this PR (default lift)
translated files (sa/gen, sa/image), byte for byte 107 107, the same

Flags and settings

None.

Testing

Tested:

  • tests/test_lift_entry_local_regs.py fails on main and on #1560 (the goto on line 1, the declarations on line 2) and passes here; with the fix's line taken out it fails again.

  • The default lift of this branch and the base's lift of main at 310e7c9d: byte-identical.

  • tests/test_lift_emit.py, test_lift_entry_below.py, test_spawn_emit.py, test_split_giants.py, test_mark_groups.py, test_fastfp_regions.py, test_parallel_emit.py pass on Windows 11. tests/test_libc_hooks.py fails here and on main alike (it looks for L.discover( in nx2/lift.py).

  • In the game (Windows 11, RTX 3080, Ryzen 9 3900X, NVIDIA's driver): a pack over 310e7c9d with this fix and #1469's change (A64_LOCAL_REGS=6, flags left in the context), 4 runs of the A/B kit's route from a Hyrule Field save (a walk and a look round, then the Market): no crash or hang, and every game patch applied.

Not tested: tests/run_all.py in full; Linux.

Must be tested: None beyond Linux, any PC, with A64_LOCAL_REGS=6 (Test Farm: linux, a few minutes of play).

Risks

What could break: in a default build, nothing (byte-identical). With local registers, where a function's head lines go.

Turning it off: revert the commit.

Compatibility: a player's next build translates the game again once (the lift key hashes nx2/); nothing recompiles.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why. Default build: byte-identical; the run with local registers on is pending.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. Stacked on #1560, on 310e7c9d.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). Not run in full yet.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.

🤖 Generated with Claude Code

**Type:** fix · **Risk:** low · **Tested on:** Windows 11, RTX 3080, Ryzen 9 3900X (game-free; the default lift compared byte for byte) · **Needs:** none for the default build; a game run with local registers on, any PC ## TL;DR With the translator's local registers turned on, the 26 game functions whose code starts below their entry point began by jumping over the lines that load their registers, so they ran on garbage values. It never happened in a normal build, where local registers are off, but pull request #1469 turns them on by default. This makes those functions load their registers first. A normal build's translated game is byte for byte the same; a new game-free test fails on main and passes here. ## Breakdown - **Who it helps:** any build with `A64_LOCAL_REGS` or `A64_LOCAL_FLAGS` set, and #1469, which turns them on by default. - **What was wrong:** the goto that starts such a function at its entry was put in after the head, before the register locals' declarations, so it jumped over their loads. - **What changes:** the head lines go in before the kit declares the locals, so the locals are loaded before the goto. - **What does not change:** a default build (local registers off): its translated C is byte-identical. - **How it was checked:** `tests/test_lift_entry_local_regs.py` fails on main and passes here, and fails again with the fix's line taken out; the default lift compared byte for byte with main's. - **What is still open:** Linux. - **Risk:** low: in a default build nothing changes; with local registers, 26 functions start correctly. ## Why `nx2/lift_fixes.py` starts a function whose code lies below its entry with `goto L_<entry>;` right after the head (its fix 4; 26 functions in a lift of OoT). With `A64_LOCAL_REGS` or `A64_LOCAL_FLAGS` the kit's `localize_regs` declares the function's register locals right after the head too, each loaded from the context (`uint64_t x0_ = c->x[0], ...;`). The goto came first, so it jumped over those declarations: C allows that, but the locals are never loaded. With #1469's defaults each of those 26 functions ran on whatever the stack held. Stacked on #1560, whose `nx2/emit.py` holds the head lines this moves. ## How to reproduce **Seen on:** any PC, game-free **Settings and switches:** `A64_LOCAL_REGS=6` at lift time **Steps:** `python -I -B tests/test_lift_entry_local_regs.py` on main; or lift with `A64_LOCAL_REGS=6` and read any function `nx2/lift_fixes.py` gives an entry goto **Expected:** the locals' declarations, then the goto **Actual:** the goto, then the declarations it jumps over ## What changed The head lines (entry hooks' call, the entry goto) now go into the function's lines before the kit's `localize_regs` runs, instead of being spliced into its finished text after it; `localize_regs` then declares the locals right after the head, ahead of them. A PLT stub, which the method writes and returns before that point, still gets them in its text. - `nx2/emit.py`: `emit` hands the head lines to `emit_function`, which puts them after the head before its last passes (two tagged lines); a PLT stub keeps the text splice. - `tests/test_lift_entry_local_regs.py` (new): a made-up function with code below its entry, emitted with local registers, local flags and both: every declaration that loads a local comes before the goto; without them the goto still follows the head. ## Measurements **Verdict:** nothing to measure for a default build: its translated game is byte-identical to main's. With local registers on (this fix and #1469's change, registers only), the game plays, and the frame rate is within noise of main's on this PC: +0.2% in Hyrule Field and +1.6% in the Market, over 3 interleaved pairs. | | Before: main @ 310e7c9d (the base's lift) | After: this PR (default lift) | |---|---|---| | translated files (`sa/gen`, `sa/image`), byte for byte | 107 | 107, the same | ### Flags and settings None. ## Testing **Tested:** - `tests/test_lift_entry_local_regs.py` fails on main and on #1560 (the goto on line 1, the declarations on line 2) and passes here; with the fix's line taken out it fails again. - The default lift of this branch and the base's lift of main at `310e7c9d`: byte-identical. - `tests/test_lift_emit.py`, `test_lift_entry_below.py`, `test_spawn_emit.py`, `test_split_giants.py`, `test_mark_groups.py`, `test_fastfp_regions.py`, `test_parallel_emit.py` pass on Windows 11. `tests/test_libc_hooks.py` fails here and on main alike (it looks for `L.discover(` in `nx2/lift.py`). - In the game (Windows 11, RTX 3080, Ryzen 9 3900X, NVIDIA's driver): a pack over `310e7c9d` with this fix and #1469's change (`A64_LOCAL_REGS=6`, flags left in the context), 4 runs of the A/B kit's route from a Hyrule Field save (a walk and a look round, then the Market): no crash or hang, and every game patch applied. **Not tested:** `tests/run_all.py` in full; Linux. **Must be tested:** None beyond Linux, any PC, with `A64_LOCAL_REGS=6` (Test Farm: linux, a few minutes of play). ## Risks **What could break:** in a default build, nothing (byte-identical). With local registers, where a function's head lines go. **Turning it off:** revert the commit. **Compatibility:** a player's next build translates the game again once (the lift key hashes `nx2/`); nothing recompiles. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [ ] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. *Default build: byte-identical; the run with local registers on is pending.* - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [ ] Rebased on current `main`. *Stacked on #1560, on `310e7c9d`.* - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). *Not run in full yet.* - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
refactor(lift): one emit_function with hook points, in place of fastfp's copy and three wrappers
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
4dd0e507f8
Longshot adds to a64-kit's Lifter.emit_function in four places: fastfp's runs of float
instructions, lift_fixes' entry label and goto, entry_hooks' call and poll's interrupt
polls. Each replaced or wrapped the method when nx2/lift.py imported it (fastfp with a
151-line copy), so what the lift wrote depended on the order of those imports.

nx2/emit.py now holds the one copy of the kit's method (its added lines tagged
"# (emit)", pinned at the kit's tree as before) and the points the modules register
into: a run gatherer (fastfp), start hooks (lift_fixes' label), head lines (entry_hooks'
call, then lift_fixes' goto) and passes over the finished text (poll). The order of
what they add is written there, not left to install order. nx2/chunks.py's new-process
check compares the registered hooks too.

The lifted C is byte-identical: a standalone lift of main (79d4a143) and of this branch,
same inputs and environment as make_game.py's, wrote the same 106 files (1.69 GB).
tests/test_fastfp_override.py becomes tests/test_lift_emit.py: the same pins, and the
hooks checked against the wrappers they replaced, in either registration order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
nx2/lift_fixes.py starts such a function with a goto to its entry right after the head,
and with A64_LOCAL_REGS or A64_LOCAL_FLAGS the kit declares the register locals there too,
loaded from the context. The goto came first and jumped over the declarations, so the
function ran on locals that were never loaded. A made-up function emitted as the lift does
shows it; this fails here and on main.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(lift): with local registers, a function with code below its entry loads them before its entry
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 11s
Website / Website on Cloudflare (pull_request_target) Has been skipped
6870be0dc7
nx2/emit.py put its head lines (entry_hooks' call, lift_fixes' goto to the entry) into
a function's finished text, after the kit's localize_regs had declared the register
locals right after the head, so the goto came first and jumped over the loads: with
A64_LOCAL_REGS or A64_LOCAL_FLAGS, each function with code below its entry (26 in a lift
of OoT) ran on locals never loaded from the context. The head lines now go into the
function's lines before localize_regs, which then declares the locals ahead of them.
A PLT stub, written before that point, still gets them in its text.

Without local registers (the default) the C is unchanged; tests/test_lift_entry_local_regs.py
passes now.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

Tested in game: Windows 11, RTX 3080 (NVIDIA driver), Ryzen 9 3900X: a pack over 310e7c9d with this fix and #1469's change (A64_LOCAL_REGS=6, flags left in the context), 4 runs of a Hyrule Field walk and look round then the Market from a save: no crash or hang, every game patch applied; frame rate within noise of main (+0.2% Hyrule Field, +1.6% Market over 3 interleaved pairs). A default build's translated C is byte-identical to main's.

Tested in game: Windows 11, RTX 3080 (NVIDIA driver), Ryzen 9 3900X: a pack over 310e7c9d with this fix and #1469's change (A64_LOCAL_REGS=6, flags left in the context), 4 runs of a Hyrule Field walk and look round then the Market from a save: no crash or hang, every game patch applied; frame rate within noise of main (+0.2% Hyrule Field, +1.6% Market over 3 interleaved pairs). A default build's translated C is byte-identical to main's.
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 11s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/lift-entry-local-regs:dyst-fix/lift-entry-local-regs'
git switch 'dyst-fix/lift-entry-local-regs'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1587
No description provided.