WIP: feat(lift): hook sites, named places in the game's code the host steps in, compiled into the translated game; Text speed and size the first #1821
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1821
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:feat/hook-sites"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: low · Tested on: Windows 11, RTX 3080, Ryzen 9 3900X · Needs: a run on Linux
TL;DR
The first piece of the Modding API's core: hook sites. A site is a named place in the game's code, found by the instruction words around it, where the host can step in; the lift compiles each one into the translated game as a check of one byte, so a site nothing uses costs a load and a predicted branch, and its function keeps its fast local registers. Text speed and Text size, which were a text patch on the translated C, are the first users: their patch is deleted, and the game behaves as before. The translation changes in exactly one function.
Breakdown
nx2/oot_patches.py), found by exact text that any change to the translator can break, with no list of what they touch.textStep, their text patch deleted.tests/run_all.pyagainst main; a run in the game.Why
The Modding API Spec (wiki, Modding-API-Spec, section 8.1) and its plan (Modding-API-Plan, milestone M2, pieces 1 and 2) start the core with the hook compiler: named sites, compiled into the lifted code and always present, which the engine's own features use first and mods later. Text speed is the plan's first conversion (small and serial). Relates to #1097 (the text patches have no list of their addresses). The plan's gate 1 (the translation unchanged with a site table present and empty) is #1798's tool.
Stacked on #1587 (and #1560 under it), whose head hook point the entry sites use.
What changed
A site table, a resolver and a compiler of sites into the lifted code, the host's side of it, and the first feature moved onto a site.
nx2/hooks.py(new): the site table of a game build (nx2/sites/<build id>.toml; the next PR makes it a file a site); a site's signature must match exactly once in its module, at the address given, or the lift stops naming every bad site. An "after" site runs once its instruction falls through (not at a branch that never does); an "entry" site is the first thing its function does, and its handler may finish the function. What a site compiles to:if (hos_sites[n]) hos_site_call(c, n). Inside that branch the function's local registers that changed since the last sync are written back before the call and read again after it, so a site keeps its function's locals and a dormant one touches only its byte. The lift counts every site it writes and stops on one written into no function.a64_sites.cnames the lift's sites and holds their bytes.python -I nx2/hooks.py <exefs>resolves the sites without lifting.nx2/emit.py: C spliced after an instruction that is not an op (a conditional branch, a call) where it falls through; the sites' own table,Lifter.site_after, with the instruction's sync marks (tagged lines).nx2/chunks.pysends the site tables to the emit's worker processes.nx2/lift.py,nx2/make_game.py(.tomlin the lift's sources),nx2/build.py(a64_sites.c).host/hook_sites.c,host/hook_sites.h(new): handlers attached by a site's name, called in order until one says the function is done; a site's byte set while it has a handler.host/oot_text_step.c(new): Text speed and Text size's handler ontextStep, after the letter count's check inf_80b1e140(a text pane's LetterAnimControl step).nx2/oot_patches.py: the Text speed text patch deleted.A64_NOESC, as every call lifted code makes (without it the call captures the context in the whole function, its dormant path too: a64-kit'sa64.h); an after site runs every handler, an entry site stops at the first that did the function's work (a64_site_entry);BLRAAandBLRABare calls, not branches away; a signature is found by an exact anchor word, not a regex at every byte; a lift with more sites than the host's table holds stops;host/hook_sites.hsays what a handler may rely on.tests/test_hook_sites.py(new),tests/test_text_speed.pyrewritten,tests/test_text_size.py.Measurements
Verdict: not measured in frames: the translation changes in one function,
f_80b1e140, which now keeps its local registers (it kept them in the context before, for the text patch), and a dormant site costs a byte load and a predicted branch. A timed run of dormant sites in hot functions is planned with the next pieces.Baseline: #1587's head, which this PR is stacked on: main differs from it by #1560 and #1587, which have their own comparisons with main (byte for byte the same).
Hardware: Windows 11 Pro N (build 22631), RTX 3080, Ryzen 9 3900X, 64 GB
Scene: none: the translation, game-free (the in-game run is in Testing)
Method:
scripts/lift_parity.py lift(#1798) of each commit, rc.8's ExeFS, the Builder's Python and llvm-mingw, thencompareHow to reproduce:
python -I scripts/lift_parity.py lift --checkout <a checkout>for #1587's head and for this branch, thenpython -I scripts/lift_parity.py compare <one> <the other>ab68fc1aa64_sites.cnew, the chunk cost estimatesf_80b1e140(the patch's two lines replaced by the site; its locals kept), at every step: the conversion, the locals, the review's fixesAfter the rebase onto main
0de25c3e(which brought #134's ExeFS patches, #1740's text patch and #1774's menu modules), lifted again:0de25c3e725cc27a725cc27a957a28b8(this PR with the feature modules on top)a64_sites.cnew, the chunk cost estimatesf_80b1e140This PR's own head was not lifted again after the rebase: its translation lies between those two (the site's byte by number here, by name in #1822).
Flags and settings
None.
Testing
Tested:
tests/test_hook_sites.py(new): the table's format and 16 kinds of mistake; the resolver over made-up code; install; a made-up function emitted through the lift's own emitter with an entry and an after site (the locals kept, a dormant site touching only its byte, the write-back and the reads after the handler, a patch and a site on one instruction, a stand-in run gatherer); that function compiled and run, dormant and with a handler; the host's handler table compiled and run. Seven deliberate breaks of the site's code, each caught.tests/test_text_speed.py,tests/test_text_size.py,tests/test_spawn_emit.py,tests/test_lift_emit.py,tests/test_menu_hooks.py,tests/test_entry_below_locals.pypass.The whole translation compared byte for byte (#1798) at each commit: #1587's head against a table present and empty, the same; the conversion of Text speed, its one function named; the site keeping its function's locals, the same one function; the review's fixes, the same one function and
a64_sites.c.python -I -B tests/run_all.py(the Builder's llvm-mingw first onPATH,LONGSHOT_MINGWset), on main0de25c3e: 706 passed / 45 failed at4cdd901d. The one failing here only,tests/test_a64_helper_build_flags.py(#756's: its stand-in lift folder lackeda64_sites.c), is fixed in the commit that adds the file and passes; the other 44 fail on main0de25c3etoo, rerun there.A cold review of the branch by a second agent (code read, nothing run); its findings fixed in 6eeedb76, each with a test, and every test mutation it listed as uncaught now caught.
In the game (Windows 11, RTX 3080, Ryzen 9 3900X), by the person I work for: a pack of the next PR's branch at 95fbe486 (this branch at ada6d3fc with the feature modules on top; their translations byte for byte the same) over main 47430e81's base, with #1729's header fix applied for the build only; the repository's
kokiri-startsave, Kokiri Forest's signs and dialogue. Text speed: Normal types the letters out, the fastest setting shows each page at once. Text size: larger, as before, including #1779's first showing at the default size, which main has too.Rebased onto main
0de25c3e: the conflicts were the menu rows' reader (#1774'smenu_text()) and an import oftests/test_text_size.py; the checks of the files it touches pass; the comparisons andrun_allabove are on it.Not tested: Linux. In the game, this branch's last commit (6eeedb76, the review's fixes: the site's call declared
A64_NOESC, an after site running every handler): its translation differs from ada6d3fc's inf_80b1e140alone (the declaration) and ina64_sites.c(the sites' kinds), and its host code is covered bytests/test_hook_sites.py's compiled runs.Must be tested:
f_80b1e140; and Text speed in a dialogue, expecting the letters faster (by hand, or the Test Farm on a Linux machine).Risks
What could break: Text speed and Text size, whose code moved (the in-game run covers both); a site whose signature stops matching after a game update stops the lift with the site named, by design.
Turning it off: revert; a site with no handler costs a byte load.
Compatibility:
nx2/sites/is part of the lift's sources, so the translation is made again once.Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Tested in game: Windows 11 Pro N 22631, RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, by the person I work for: a pack of the next PR's branch at 95fbe486 (this branch's code at ada6d3fc with the feature modules on top, their translations byte for byte the same) over main 47430e81's base, #1729's header fix applied for the build only. The repository's kokiri-start save, Kokiri Forest's signs and dialogue: Text speed at Normal types the letters out, at its fastest shows each page at once; Text size larger as before, including #1779's first showing at the default size, which main has too. Not run in the game since: the review's fixes (A64_NOESC, after sites running every handler) and the rebase onto
0de25c3e.4cdd901d75ba76c36161View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.