WIP: refactor(lift): one emit_function with hook points, in place of fastfp's copy and three wrappers #1560

Draft
dyst wants to merge 1 commit from dyst/Longshot:refactor/lift-emit-hooks into main
Member

Type: refactor · Risk: low · Tested on: Windows 11, RTX 3080, Ryzen 9 3900X (game-free: the translated game compared byte for byte; not yet run in the game) · Needs: a run in the game, any PC; a lift on Linux

TL;DR

The part of the translator that writes each game function as C was changed in four places by modules that each replaced or wrapped it as they loaded, one of them with a full copy of it, so what it wrote depended on the order they loaded in. This PR gives it one copy with named places to add to, in an order written down in one file. The translated game comes out byte for byte the same as main's, compared over the whole game twice. It has not run in the game yet.

Breakdown

  • Who it helps: anyone changing the translator; a player sees no difference.
  • What was wrong: four modules patched the kit's emit_function when nx2/lift.py imported them, one with a 151-line copy, so the output depended on import order and each new addition had to wrap it again.
  • What changes: one emit_function in nx2/emit.py with hook points; the four modules register into it, and the order of what they add is written there.
  • What does not change: the translated C (byte-identical), so the game, its switches and saves; a player's next build translates the game again once (the lift key covers nx2/) but compiles nothing new.
  • How it was checked: two whole-game translations compared byte for byte with main's (106 files on 79d4a143; 108 files through dev_build.py pack on 310e7c9d), and the lifter's checks, one of them rewritten to pin the new layout.
  • What is still open: the run in the game, tests/run_all.py, and a lift on Linux (the fork path of the parallel emit).
  • Risk: low: the output is identical, and an addition placed in the wrong order would show as a difference in the comparison.

Why

a64-kit's Lifter.emit_function writes one game function's C. Longshot adds to it in four places: fast float regions (nx2/fastfp.py), the entry label and goto of a function with code below its entry (nx2/lift_fixes.py), a host hook's call at a function's head (nx2/entry_hooks.py) and interrupt polls before backward gotos (nx2/poll.py). Because third_party/ is not edited, each did it by replacing or wrapping the method at import: fastfp with a copy of the whole method, the other three by wrapping whatever was installed before them and editing its finished text. So the order of the install() calls in nx2/lift.py decided what the lift wrote (the entry hook's call comes before the entry goto only because entry_hooks is installed after lift_fixes), the emit's new processes had to repeat every wrapping, and the next addition would have been a fifth wrapper. This puts the additions in one place, named and ordered, so later changes to the emit can be made and checked one at a time.

No issue covers it (the tracker searched for emit_function, lift_fixes, entry_hooks and the fastfp copy).

What changed

One copy of the kit's method now lives in nx2/emit.py, with the run gathering built in and every added line tagged # (emit); the modules register hooks into it instead of replacing it. The order of what they add is three fixed lists in that file, not the order of their installs. Nothing in what the lift writes changes.

  • nx2/emit.py (new): emit_function, the kit's at tree 6164061e with its added lines tagged; emit, what Lifter.emit_function now is, which runs the hooks around it; add(point, name, fn) and set_runs(gatherer) to register; START_ORDER, HEAD_ORDER (the entry hook's call, then the entry goto) and TEXT_ORDER (the polls); fingerprint() for the new-process check.
  • nx2/fastfp.py: its copy of the method is gone; Runs (begin, joins, region) is the run gatherer, called exactly where the copy called its own code, so the per-instruction work is the same.
  • nx2/lift_fixes.py: entry_label (start) and entry_goto (head line) in place of its wrapper.
  • nx2/entry_hooks.py: entry_call (head line) in place of its wrapper.
  • nx2/poll.py: polls, its unchanged pass over the finished text, as a text hook.
  • nx2/chunks.py: _identities adds emit.fingerprint(), so a new emit process with other hooks than the lift's still stops.
  • nx2/lift.py: comments only; the install() calls stay where they were.
  • tests/test_fastfp_override.py → tests/test_lift_emit.py: the same pins on the copy (the kit's method line for line once the tagged lines are out, the kit's tree hash, the made-up fast float runs), plus a function with code below its entry, a host hook and loops, checked against the old wrappers' logic with the hooks registered in either order, and a hook name in no order refused. Each of its four tamper cases (a copied line changed, a tag dropped, KIT_TREE off by a digit, HEAD_ORDER swapped) fails it.
  • docs/design.md: the lift step of the pipeline names nx2/emit.py.

Measurements

Verdict: nothing to measure: the translated game is byte-identical to main's, so the game that runs is the same.

Hardware: Windows 11 Pro, RTX 3080, Ryzen 9 3900X, 64 GB
Method: (1) nx2/lift.py --standalone --write of main at 79d4a143 and of this branch on it, with the environment make_game.py gives the lift (RX_RELEASE=1, A64_HOT_CHUNK=1, no hot list), both through the job queue, every file they wrote compared byte for byte. (2) After the rebase onto 310e7c9d: dev_build.py pack of this branch over the shared base of 310e7c9d, its translation compared with the base's own translation of main.

Before: main @ 79d4a143 After: this PR @ 3060a340 (this commit before its rebase, on 79d4a143)
files written by the lift, byte for byte 106 (1.69 GB) 106, the same
fast float regions / entry gotos / entry hooks / interrupt polls 17,068 / 26 / 6 / 683,145 17,068 / 26 / 6 / 683,145
Before: base of main @ 310e7c9d After: pack of this PR @ 4dd0e507 (on 310e7c9d)
translated files (sa/gen, sa/image), byte for byte 108 108, the same; only .lift_key and lifted_objects.txt differ (they hold the lift key)
106 files the same (1.69 GB), 0 differ, 0 only in A, 0 only in B
same: sa/image/functions.json
same: sa/image/image.json
translation: 108 files the same, 0 differ

Flags and settings

None.

Testing

Tested:

  • The whole game translated on main and on this branch, the output compared byte for byte, twice (Measurements).
  • On Windows 11 with the Builder's Python 3.14: tests/test_lift_emit.py, test_lift_entry_below.py, test_mark_groups.py, test_parallel_emit.py, test_spawn_emit.py (the new-process emit), test_split_giants.py, test_stale_hot_pieces.py, test_fastfp_regions.py, test_dead_flags.py, test_walk.py, test_discover.py, test_lift_cache.py pass. tests/test_libc_hooks.py fails here and on main alike (it looks for L.discover( in nx2/lift.py, which main no longer has).
  • The runtime builds: dev_build.py pack and rt over the base of 310e7c9d.

Not tested:

  • Not run in the game. Opened as WIP before that run, at the request of the person I work for; the game it builds is the same C as main's, but the rule stands and the run comes next.
  • tests/run_all.py in full, and the 39 checks that read these files on main and here: not run yet.
  • Linux: the parallel emit forks there (test_parallel_emit.py has nothing to compare on Windows), and no lift has run on Linux.

Must be tested:

  • This PC (Windows 11, RTX 3080): the runtime from this branch, a few minutes of play from a save, expecting the game as on main (by hand: me).
  • Linux, any PC: a lift of this branch and of main compared byte for byte, or at least tests/test_parallel_emit.py and tests/test_spawn_emit.py, expecting the same files (by hand, or the Test Farm: @Beanstalk linux, a lift of this branch).

Risks

What could break: what the lift writes, if an addition ran in another order or a hook missed a case the wrappers had; the comparison above would show it. The emit's new processes, if they registered other hooks than the lift's: nx2/chunks.py's check stops them, as before.

Turning it off: revert the commit.

Compatibility: a player's next build translates the game again once (about 3 to 4 minutes; the lift key hashes nx2/), and compiles nothing new, since the C is the same. Nothing else changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why. Not yet: opened before the in-game run, at the request of the person I work for.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. On 310e7c9d, the base #1553 builds on, to build over that shared base; rebased onto main before it is ready.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). Not run yet; tests/test_libc_hooks.py fails on main too.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** refactor · **Risk:** low · **Tested on:** Windows 11, RTX 3080, Ryzen 9 3900X (game-free: the translated game compared byte for byte; not yet run in the game) · **Needs:** a run in the game, any PC; a lift on Linux ## TL;DR The part of the translator that writes each game function as C was changed in four places by modules that each replaced or wrapped it as they loaded, one of them with a full copy of it, so what it wrote depended on the order they loaded in. This PR gives it one copy with named places to add to, in an order written down in one file. The translated game comes out byte for byte the same as main's, compared over the whole game twice. It has not run in the game yet. ## Breakdown - **Who it helps:** anyone changing the translator; a player sees no difference. - **What was wrong:** four modules patched the kit's `emit_function` when `nx2/lift.py` imported them, one with a 151-line copy, so the output depended on import order and each new addition had to wrap it again. - **What changes:** one `emit_function` in `nx2/emit.py` with hook points; the four modules register into it, and the order of what they add is written there. - **What does not change:** the translated C (byte-identical), so the game, its switches and saves; a player's next build translates the game again once (the lift key covers `nx2/`) but compiles nothing new. - **How it was checked:** two whole-game translations compared byte for byte with main's (106 files on `79d4a143`; 108 files through `dev_build.py pack` on `310e7c9d`), and the lifter's checks, one of them rewritten to pin the new layout. - **What is still open:** the run in the game, `tests/run_all.py`, and a lift on Linux (the fork path of the parallel emit). - **Risk:** low: the output is identical, and an addition placed in the wrong order would show as a difference in the comparison. ## Why a64-kit's `Lifter.emit_function` writes one game function's C. Longshot adds to it in four places: fast float regions (`nx2/fastfp.py`), the entry label and goto of a function with code below its entry (`nx2/lift_fixes.py`), a host hook's call at a function's head (`nx2/entry_hooks.py`) and interrupt polls before backward gotos (`nx2/poll.py`). Because `third_party/` is not edited, each did it by replacing or wrapping the method at import: `fastfp` with a copy of the whole method, the other three by wrapping whatever was installed before them and editing its finished text. So the order of the `install()` calls in `nx2/lift.py` decided what the lift wrote (the entry hook's call comes before the entry goto only because `entry_hooks` is installed after `lift_fixes`), the emit's new processes had to repeat every wrapping, and the next addition would have been a fifth wrapper. This puts the additions in one place, named and ordered, so later changes to the emit can be made and checked one at a time. No issue covers it (the tracker searched for emit_function, lift_fixes, entry_hooks and the fastfp copy). ## What changed One copy of the kit's method now lives in `nx2/emit.py`, with the run gathering built in and every added line tagged `# (emit)`; the modules register hooks into it instead of replacing it. The order of what they add is three fixed lists in that file, not the order of their installs. Nothing in what the lift writes changes. - **`nx2/emit.py`** (new): `emit_function`, the kit's at tree `6164061e` with its added lines tagged; `emit`, what `Lifter.emit_function` now is, which runs the hooks around it; `add(point, name, fn)` and `set_runs(gatherer)` to register; `START_ORDER`, `HEAD_ORDER` (the entry hook's call, then the entry goto) and `TEXT_ORDER` (the polls); `fingerprint()` for the new-process check. - **`nx2/fastfp.py`**: its copy of the method is gone; `Runs` (begin, joins, region) is the run gatherer, called exactly where the copy called its own code, so the per-instruction work is the same. - **`nx2/lift_fixes.py`**: `entry_label` (start) and `entry_goto` (head line) in place of its wrapper. - **`nx2/entry_hooks.py`**: `entry_call` (head line) in place of its wrapper. - **`nx2/poll.py`**: `polls`, its unchanged pass over the finished text, as a text hook. - **`nx2/chunks.py`**: `_identities` adds `emit.fingerprint()`, so a new emit process with other hooks than the lift's still stops. - **`nx2/lift.py`**: comments only; the `install()` calls stay where they were. - **`tests/test_fastfp_override.py` → `tests/test_lift_emit.py`**: the same pins on the copy (the kit's method line for line once the tagged lines are out, the kit's tree hash, the made-up fast float runs), plus a function with code below its entry, a host hook and loops, checked against the old wrappers' logic with the hooks registered in either order, and a hook name in no order refused. Each of its four tamper cases (a copied line changed, a tag dropped, `KIT_TREE` off by a digit, `HEAD_ORDER` swapped) fails it. - **`docs/design.md`**: the lift step of the pipeline names `nx2/emit.py`. ## Measurements **Verdict:** nothing to measure: the translated game is byte-identical to main's, so the game that runs is the same. Hardware: Windows 11 Pro, RTX 3080, Ryzen 9 3900X, 64 GB Method: (1) `nx2/lift.py --standalone --write` of main at `79d4a143` and of this branch on it, with the environment `make_game.py` gives the lift (`RX_RELEASE=1`, `A64_HOT_CHUNK=1`, no hot list), both through the job queue, every file they wrote compared byte for byte. (2) After the rebase onto `310e7c9d`: `dev_build.py pack` of this branch over the shared base of `310e7c9d`, its translation compared with the base's own translation of main. | | Before: main @ 79d4a143 | After: this PR @ 3060a340 (this commit before its rebase, on 79d4a143) | |---|---|---| | files written by the lift, byte for byte | 106 (1.69 GB) | 106, the same | | fast float regions / entry gotos / entry hooks / interrupt polls | 17,068 / 26 / 6 / 683,145 | 17,068 / 26 / 6 / 683,145 | | | Before: base of main @ 310e7c9d | After: pack of this PR @ 4dd0e507 (on 310e7c9d) | |---|---|---| | translated files (`sa/gen`, `sa/image`), byte for byte | 108 | 108, the same; only `.lift_key` and `lifted_objects.txt` differ (they hold the lift key) | ``` 106 files the same (1.69 GB), 0 differ, 0 only in A, 0 only in B same: sa/image/functions.json same: sa/image/image.json translation: 108 files the same, 0 differ ``` ### Flags and settings None. ## Testing **Tested:** - The whole game translated on main and on this branch, the output compared byte for byte, twice (Measurements). - On Windows 11 with the Builder's Python 3.14: `tests/test_lift_emit.py`, `test_lift_entry_below.py`, `test_mark_groups.py`, `test_parallel_emit.py`, `test_spawn_emit.py` (the new-process emit), `test_split_giants.py`, `test_stale_hot_pieces.py`, `test_fastfp_regions.py`, `test_dead_flags.py`, `test_walk.py`, `test_discover.py`, `test_lift_cache.py` pass. `tests/test_libc_hooks.py` fails here and on main alike (it looks for `L.discover(` in `nx2/lift.py`, which main no longer has). - The runtime builds: `dev_build.py pack` and `rt` over the base of `310e7c9d`. **Not tested:** - **Not run in the game.** Opened as WIP before that run, at the request of the person I work for; the game it builds is the same C as main's, but the rule stands and the run comes next. - `tests/run_all.py` in full, and the 39 checks that read these files on main and here: not run yet. - Linux: the parallel emit forks there (`test_parallel_emit.py` has nothing to compare on Windows), and no lift has run on Linux. **Must be tested:** - This PC (Windows 11, RTX 3080): the runtime from this branch, a few minutes of play from a save, expecting the game as on main (by hand: me). - Linux, any PC: a lift of this branch and of main compared byte for byte, or at least `tests/test_parallel_emit.py` and `tests/test_spawn_emit.py`, expecting the same files (by hand, or the Test Farm: @Beanstalk linux, a lift of this branch). ## Risks **What could break:** what the lift writes, if an addition ran in another order or a hook missed a case the wrappers had; the comparison above would show it. The emit's new processes, if they registered other hooks than the lift's: `nx2/chunks.py`'s check stops them, as before. **Turning it off:** revert the commit. **Compatibility:** a player's next build translates the game again once (about 3 to 4 minutes; the lift key hashes `nx2/`), and compiles nothing new, since the C is the same. Nothing else changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [ ] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. *Not yet: opened before the in-game run, at the request of the person I work for.* - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [ ] Rebased on current `main`. *On `310e7c9d`, the base #1553 builds on, to build over that shared base; rebased onto main before it is ready.* - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). *Not run yet; `tests/test_libc_hooks.py` fails on main too.* - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
refactor(lift): one emit_function with hook points, in place of fastfp's copy and three wrappers
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
4dd0e507f8
Longshot adds to a64-kit's Lifter.emit_function in four places: fastfp's runs of float
instructions, lift_fixes' entry label and goto, entry_hooks' call and poll's interrupt
polls. Each replaced or wrapped the method when nx2/lift.py imported it (fastfp with a
151-line copy), so what the lift wrote depended on the order of those imports.

nx2/emit.py now holds the one copy of the kit's method (its added lines tagged
"# (emit)", pinned at the kit's tree as before) and the points the modules register
into: a run gatherer (fastfp), start hooks (lift_fixes' label), head lines (entry_hooks'
call, then lift_fixes' goto) and passes over the finished text (poll). The order of
what they add is written there, not left to install order. nx2/chunks.py's new-process
check compares the registered hooks too.

The lifted C is byte-identical: a standalone lift of main (79d4a143) and of this branch,
same inputs and environment as make_game.py's, wrote the same 106 files (1.69 GB).
tests/test_fastfp_override.py becomes tests/test_lift_emit.py: the same pins, and the
hooks checked against the wrappers they replaced, in either registration order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Collaborator

Possibly related, for context:

  • PR #38 perf(lift): fast float regions, plain SSE checked once by MXCSR with the exact copy as fallback — Introduced the fastfp emit_function copy this refactor replaces
  • issue #765 Install: Windows build fails during translation with corrupted chunks or TypeError — Open Windows translation failure involves emit workers; possible relevance to lift changes
Possibly related, for context: - PR #38 perf(lift): fast float regions, plain SSE checked once by MXCSR with the exact copy as fallback — Introduced the fastfp emit_function copy this refactor replaces - issue #765 Install: Windows build fails during translation with corrupted chunks or TypeError — Open Windows translation failure involves emit workers; possible relevance to lift changes
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'refactor/lift-emit-hooks:dyst-refactor/lift-emit-hooks'
git switch 'dyst-refactor/lift-emit-hooks'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1560
No description provided.