WIP: refactor(lift): one emit_function with hook points, in place of fastfp's copy and three wrappers #1560
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1560
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:refactor/lift-emit-hooks"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: refactor · Risk: low · Tested on: Windows 11, RTX 3080, Ryzen 9 3900X (game-free: the translated game compared byte for byte; not yet run in the game) · Needs: a run in the game, any PC; a lift on Linux
TL;DR
The part of the translator that writes each game function as C was changed in four places by modules that each replaced or wrapped it as they loaded, one of them with a full copy of it, so what it wrote depended on the order they loaded in. This PR gives it one copy with named places to add to, in an order written down in one file. The translated game comes out byte for byte the same as main's, compared over the whole game twice. It has not run in the game yet.
Breakdown
emit_functionwhennx2/lift.pyimported them, one with a 151-line copy, so the output depended on import order and each new addition had to wrap it again.emit_functioninnx2/emit.pywith hook points; the four modules register into it, and the order of what they add is written there.nx2/) but compiles nothing new.79d4a143; 108 files throughdev_build.py packon310e7c9d), and the lifter's checks, one of them rewritten to pin the new layout.tests/run_all.py, and a lift on Linux (the fork path of the parallel emit).Why
a64-kit's
Lifter.emit_functionwrites one game function's C. Longshot adds to it in four places: fast float regions (nx2/fastfp.py), the entry label and goto of a function with code below its entry (nx2/lift_fixes.py), a host hook's call at a function's head (nx2/entry_hooks.py) and interrupt polls before backward gotos (nx2/poll.py). Becausethird_party/is not edited, each did it by replacing or wrapping the method at import:fastfpwith a copy of the whole method, the other three by wrapping whatever was installed before them and editing its finished text. So the order of theinstall()calls innx2/lift.pydecided what the lift wrote (the entry hook's call comes before the entry goto only becauseentry_hooksis installed afterlift_fixes), the emit's new processes had to repeat every wrapping, and the next addition would have been a fifth wrapper. This puts the additions in one place, named and ordered, so later changes to the emit can be made and checked one at a time.No issue covers it (the tracker searched for emit_function, lift_fixes, entry_hooks and the fastfp copy).
What changed
One copy of the kit's method now lives in
nx2/emit.py, with the run gathering built in and every added line tagged# (emit); the modules register hooks into it instead of replacing it. The order of what they add is three fixed lists in that file, not the order of their installs. Nothing in what the lift writes changes.nx2/emit.py(new):emit_function, the kit's at tree6164061ewith its added lines tagged;emit, whatLifter.emit_functionnow is, which runs the hooks around it;add(point, name, fn)andset_runs(gatherer)to register;START_ORDER,HEAD_ORDER(the entry hook's call, then the entry goto) andTEXT_ORDER(the polls);fingerprint()for the new-process check.nx2/fastfp.py: its copy of the method is gone;Runs(begin, joins, region) is the run gatherer, called exactly where the copy called its own code, so the per-instruction work is the same.nx2/lift_fixes.py:entry_label(start) andentry_goto(head line) in place of its wrapper.nx2/entry_hooks.py:entry_call(head line) in place of its wrapper.nx2/poll.py:polls, its unchanged pass over the finished text, as a text hook.nx2/chunks.py:_identitiesaddsemit.fingerprint(), so a new emit process with other hooks than the lift's still stops.nx2/lift.py: comments only; theinstall()calls stay where they were.tests/test_fastfp_override.py→tests/test_lift_emit.py: the same pins on the copy (the kit's method line for line once the tagged lines are out, the kit's tree hash, the made-up fast float runs), plus a function with code below its entry, a host hook and loops, checked against the old wrappers' logic with the hooks registered in either order, and a hook name in no order refused. Each of its four tamper cases (a copied line changed, a tag dropped,KIT_TREEoff by a digit,HEAD_ORDERswapped) fails it.docs/design.md: the lift step of the pipeline namesnx2/emit.py.Measurements
Verdict: nothing to measure: the translated game is byte-identical to main's, so the game that runs is the same.
Hardware: Windows 11 Pro, RTX 3080, Ryzen 9 3900X, 64 GB
Method: (1)
nx2/lift.py --standalone --writeof main at79d4a143and of this branch on it, with the environmentmake_game.pygives the lift (RX_RELEASE=1,A64_HOT_CHUNK=1, no hot list), both through the job queue, every file they wrote compared byte for byte. (2) After the rebase onto310e7c9d:dev_build.py packof this branch over the shared base of310e7c9d, its translation compared with the base's own translation of main.79d4a14379d4a143)310e7c9d4dd0e507(on310e7c9d)sa/gen,sa/image), byte for byte.lift_keyandlifted_objects.txtdiffer (they hold the lift key)Flags and settings
None.
Testing
Tested:
tests/test_lift_emit.py,test_lift_entry_below.py,test_mark_groups.py,test_parallel_emit.py,test_spawn_emit.py(the new-process emit),test_split_giants.py,test_stale_hot_pieces.py,test_fastfp_regions.py,test_dead_flags.py,test_walk.py,test_discover.py,test_lift_cache.pypass.tests/test_libc_hooks.pyfails here and on main alike (it looks forL.discover(innx2/lift.py, which main no longer has).dev_build.py packandrtover the base of310e7c9d.Not tested:
tests/run_all.pyin full, and the 39 checks that read these files on main and here: not run yet.test_parallel_emit.pyhas nothing to compare on Windows), and no lift has run on Linux.Must be tested:
tests/test_parallel_emit.pyandtests/test_spawn_emit.py, expecting the same files (by hand, or the Test Farm: @Beanstalk linux, a lift of this branch).Risks
What could break: what the lift writes, if an addition ran in another order or a hook missed a case the wrappers had; the comparison above would show it. The emit's new processes, if they registered other hooks than the lift's:
nx2/chunks.py's check stops them, as before.Turning it off: revert the commit.
Compatibility: a player's next build translates the game again once (about 3 to 4 minutes; the lift key hashes
nx2/), and compiles nothing new, since the C is the same. Nothing else changes.Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why. Not yet: opened before the in-game run, at the request of the person I work for.
in Risks.
main. On310e7c9d, the base #1553 builds on, to build over that shared base; rebased onto main before it is ready.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Not run yet;tests/test_libc_hooks.pyfails on main too.third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Possibly related, for context:
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.