WIP: docs(design): the lifted code's contract with the host, as rules (section 14) #1626

Draft
dyst wants to merge 4 commits from dyst/Longshot:docs/lift-contract-rules into main
Member

Type: docs · Risk: low · Tested on: Windows 11, Ryzen 9 3900X · Needs: none

Stacked on #1587 (and through it on #1560): rule 1 anchors to nx2/emit.py, which #1560 adds. Review those first.

TL;DR

The translated game code and the host rely on each other in ways the code shows only in pieces: the guest context's layout, where a guest register lives, when a thread gives up its core, how a store and its page mark are ordered, what each barrier and atomic becomes, and how the host replaces SDK functions. This writes those down in docs/design.md as six rules, each naming the function that keeps it, and gives those functions a doc: comment pointing at the rule. tests/test_doc_anchors.py checks that every anchor resolves and that the functions keep theirs.

Breakdown

  • Who it helps: anyone changing the translator or the host's side of it. The CPU rework plan reworks both, and #1469, #1576 and #1617 change what these rules describe.
  • What was wrong: the rules lived in comments spread over six files, and two changes this week broke one without a test noticing: the entry goto above the locals (#1587), and DMB as lighter fences (#1507).
  • What changes: docs/design.md section 14, seven doc: comment lines, and seven rows in tests/test_doc_anchors.py.
  • What does not change: any code. The added lines in host/ and nx2/ are comments.
  • How it was checked: tests/test_doc_anchors.py passes: 19 anchors, each resolves.
  • What is still open: nothing.
  • Risk: low: documentation and a check.

Why

The wiki's CPU rework plan, milestone C2: before the lift is restructured, the contract between the lifted code and the host is written as rules, each with the function that keeps it, so that a change can be checked against them.

What changed

docs/design.md, section 14, "The lifted code's contract with the host, as rules":

  1. The context: one GuestContext (Ctx) for each guest thread, written by nx2/rtgen.py from the kit's rt/a64.h, read by the host through host/hos.h, checked field by field by the emulator shim.
  2. Registers: a guest register is in the context, or, with local registers, in a C local that localize_regs writes back at every context mark. A function whose C something edits by its text keeps its registers in the context (Lifter.no_local). Lines put first in a function (an entry hook's call, the goto to an entry with code below it) come after the locals' declarations (nx2/emit.py, the rule #1587 fixes).
  3. Polls and cores: A64_POLL before each backward goto (nx2/poll.py). A thread waiting for a virtual core sets the holder's poll; the holder gives the core up at its next poll or nn::os call (host/mkw_glue.c's a64_poll_cur, host/hos_thread.c). A host call that runs long checkpoints between its pieces, and never while a host lock is held.
  4. Page marks: every guest store outside the stack and frame marks its pages after its data, with release stores (A64_MARK, nx2/rtgen.py); host writes mark after their last write (HOS_WRITTEN). The renderer takes the marks with an acquire exchange (host/rx_backend.c's collect_in). The marks' grouping joins them only in straight-line runs.
  5. Barriers and atomics: LDAR/STLR sequentially consistent, LDAPR an acquire load; the LSE forms, SWP, CAS and CASP sequentially consistent read-modify-writes (nx2/insn.py). Every DMB, a DSB and an ISB are recomp_barrier(), a sequentially consistent fence. The load-only and store-only DMBs lifted as an acquire and a release fence alone made OoT crash at its start (#1507).
  6. Replaced SDK functions and entry hooks: an HLE(sym) function replaces the SDK's sym whole; an ENTRY_HOOK_SYM(sym) hook is called first in the lifted sym and may decline. The lift finds both by reading host/*.c, so their names are in the lift key.

doc: comments above emit(), add_polls(), MARK_NATIVE_64, ldar_stlr(), collect_in(), a64_poll_cur() and vc_acquire(); their rows in tests/test_doc_anchors.py's REQUIRED.

Measurements

Verdict: nothing to measure: documentation and comment lines.

Flags and settings

None.

Testing

Tested: tests/test_doc_anchors.py on Windows 11 (the Builder's Python 3.14): 19 anchors, each resolves; the seven functions keep theirs. The lines added to host/ and nx2/ are comments, so the game builds and runs as #1587's tree, which played 4 runs of PR #801's route in the game here.

Not tested: tests/run_all.py in full.

Must be tested: None.

Risks

What could break: nothing that runs. A later change that moves one of the seven functions fails tests/test_doc_anchors.py until its anchor moves with it, which is the point.

Turning it off: revert the commit.

Compatibility: nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. On #1587, on #1560, on 310e7c9d.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). The anchor check only.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A doc: comment names a rule written in docs/design.md, and tests/test_doc_anchors.py checks it.

🤖 Generated with Claude Code

**Type:** docs · **Risk:** low · **Tested on:** Windows 11, Ryzen 9 3900X · **Needs:** none Stacked on #1587 (and through it on #1560): rule 1 anchors to `nx2/emit.py`, which #1560 adds. Review those first. ## TL;DR The translated game code and the host rely on each other in ways the code shows only in pieces: the guest context's layout, where a guest register lives, when a thread gives up its core, how a store and its page mark are ordered, what each barrier and atomic becomes, and how the host replaces SDK functions. This writes those down in `docs/design.md` as six rules, each naming the function that keeps it, and gives those functions a `doc:` comment pointing at the rule. `tests/test_doc_anchors.py` checks that every anchor resolves and that the functions keep theirs. ## Breakdown - **Who it helps:** anyone changing the translator or the host's side of it. The CPU rework plan reworks both, and #1469, #1576 and #1617 change what these rules describe. - **What was wrong:** the rules lived in comments spread over six files, and two changes this week broke one without a test noticing: the entry goto above the locals (#1587), and DMB as lighter fences (#1507). - **What changes:** `docs/design.md` section 14, seven `doc:` comment lines, and seven rows in `tests/test_doc_anchors.py`. - **What does not change:** any code. The added lines in `host/` and `nx2/` are comments. - **How it was checked:** `tests/test_doc_anchors.py` passes: 19 anchors, each resolves. - **What is still open:** nothing. - **Risk:** low: documentation and a check. ## Why The wiki's CPU rework plan, milestone C2: before the lift is restructured, the contract between the lifted code and the host is written as rules, each with the function that keeps it, so that a change can be checked against them. ## What changed `docs/design.md`, section 14, "The lifted code's contract with the host, as rules": 1. **The context:** one `GuestContext` (`Ctx`) for each guest thread, written by `nx2/rtgen.py` from the kit's `rt/a64.h`, read by the host through `host/hos.h`, checked field by field by the emulator shim. 2. **Registers:** a guest register is in the context, or, with local registers, in a C local that `localize_regs` writes back at every context mark. A function whose C something edits by its text keeps its registers in the context (`Lifter.no_local`). Lines put first in a function (an entry hook's call, the goto to an entry with code below it) come after the locals' declarations (`nx2/emit.py`, the rule #1587 fixes). 3. **Polls and cores:** `A64_POLL` before each backward goto (`nx2/poll.py`). A thread waiting for a virtual core sets the holder's `poll`; the holder gives the core up at its next poll or `nn::os` call (`host/mkw_glue.c`'s `a64_poll_cur`, `host/hos_thread.c`). A host call that runs long checkpoints between its pieces, and never while a host lock is held. 4. **Page marks:** every guest store outside the stack and frame marks its pages after its data, with release stores (`A64_MARK`, `nx2/rtgen.py`); host writes mark after their last write (`HOS_WRITTEN`). The renderer takes the marks with an acquire exchange (`host/rx_backend.c`'s `collect_in`). The marks' grouping joins them only in straight-line runs. 5. **Barriers and atomics:** `LDAR`/`STLR` sequentially consistent, `LDAPR` an acquire load; the LSE forms, `SWP`, `CAS` and `CASP` sequentially consistent read-modify-writes (`nx2/insn.py`). Every `DMB`, a `DSB` and an `ISB` are `recomp_barrier()`, a sequentially consistent fence. The load-only and store-only `DMB`s lifted as an acquire and a release fence alone made OoT crash at its start (#1507). 6. **Replaced SDK functions and entry hooks:** an `HLE(sym)` function replaces the SDK's `sym` whole; an `ENTRY_HOOK_SYM(sym)` hook is called first in the lifted `sym` and may decline. The lift finds both by reading `host/*.c`, so their names are in the lift key. `doc:` comments above `emit()`, `add_polls()`, `MARK_NATIVE_64`, `ldar_stlr()`, `collect_in()`, `a64_poll_cur()` and `vc_acquire()`; their rows in `tests/test_doc_anchors.py`'s `REQUIRED`. ## Measurements **Verdict:** nothing to measure: documentation and comment lines. ### Flags and settings None. ## Testing **Tested:** `tests/test_doc_anchors.py` on Windows 11 (the Builder's Python 3.14): 19 anchors, each resolves; the seven functions keep theirs. The lines added to `host/` and `nx2/` are comments, so the game builds and runs as #1587's tree, which played 4 runs of PR #801's route in the game here. **Not tested:** `tests/run_all.py` in full. **Must be tested:** None. ## Risks **What could break:** nothing that runs. A later change that moves one of the seven functions fails `tests/test_doc_anchors.py` until its anchor moves with it, which is the point. **Turning it off:** revert the commit. **Compatibility:** nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [ ] Rebased on current `main`. *On #1587, on #1560, on `310e7c9d`.* - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). *The anchor check only.* - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A `doc:` comment names a rule written in docs/design.md, and tests/test_doc_anchors.py checks it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
refactor(lift): one emit_function with hook points, in place of fastfp's copy and three wrappers
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
4dd0e507f8
Longshot adds to a64-kit's Lifter.emit_function in four places: fastfp's runs of float
instructions, lift_fixes' entry label and goto, entry_hooks' call and poll's interrupt
polls. Each replaced or wrapped the method when nx2/lift.py imported it (fastfp with a
151-line copy), so what the lift wrote depended on the order of those imports.

nx2/emit.py now holds the one copy of the kit's method (its added lines tagged
"# (emit)", pinned at the kit's tree as before) and the points the modules register
into: a run gatherer (fastfp), start hooks (lift_fixes' label), head lines (entry_hooks'
call, then lift_fixes' goto) and passes over the finished text (poll). The order of
what they add is written there, not left to install order. nx2/chunks.py's new-process
check compares the registered hooks too.

The lifted C is byte-identical: a standalone lift of main (79d4a143) and of this branch,
same inputs and environment as make_game.py's, wrote the same 106 files (1.69 GB).
tests/test_fastfp_override.py becomes tests/test_lift_emit.py: the same pins, and the
hooks checked against the wrappers they replaced, in either registration order.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
nx2/lift_fixes.py starts such a function with a goto to its entry right after the head,
and with A64_LOCAL_REGS or A64_LOCAL_FLAGS the kit declares the register locals there too,
loaded from the context. The goto came first and jumped over the declarations, so the
function ran on locals that were never loaded. A made-up function emitted as the lift does
shows it; this fails here and on main.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(lift): with local registers, a function with code below its entry loads them before its entry
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
PR description / Description follows the template (pull_request_target) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 11s
Website / Website on Cloudflare (pull_request_target) Has been skipped
6870be0dc7
nx2/emit.py put its head lines (entry_hooks' call, lift_fixes' goto to the entry) into
a function's finished text, after the kit's localize_regs had declared the register
locals right after the head, so the goto came first and jumped over the loads: with
A64_LOCAL_REGS or A64_LOCAL_FLAGS, each function with code below its entry (26 in a lift
of OoT) ran on locals never loaded from the context. The head lines now go into the
function's lines before localize_regs, which then declares the locals ahead of them.
A PLT stub, written before that point, still gets them in its text.

Without local registers (the default) the C is unchanged; tests/test_lift_entry_local_regs.py
passes now.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs(design): the lifted code's contract with the host, as rules (section 14)
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 12s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 15s
CI / Workflow lint (actionlint) (pull_request) Successful in 16s
Third-party parts / As their notes say (pull_request) Successful in 18s
Website / Website on Cloudflare (pull_request_target) Has been skipped
77a4922a9b
design.md's section 14 writes down, as six rules, what the lifted code and the host rely on of each other: the guest
context, where a guest register lives (the context or a local, and what goes first in a function), the polls and the
virtual cores, the page marks' order, the barriers and atomics, and the replaced SDK functions and entry hooks. The
functions that keep a rule carry a `doc:` anchor to it; tests/test_doc_anchors.py checks every anchor resolves and that
the seven functions keep theirs.

Rule 5 says every DMB is a full fence, as main lifts it: the load-only and store-only kinds as an acquire and a release
fence alone made OoT crash at its start (#1507).

The CPU rework plan's milestone C2, the docs half.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

Tested in game: Windows 11, RTX 3080, Ryzen 9 3900X: the lines this adds to host/ and nx2/ are comments only (doc: anchors), so the game builds and runs as #1587's tree, which played 4 runs of PR #801's route (Hyrule Field, the Market) here with no crash or hang.

Tested in game: Windows 11, RTX 3080, Ryzen 9 3900X: the lines this adds to host/ and nx2/ are comments only (doc: anchors), so the game builds and runs as #1587's tree, which played 4 runs of PR #801's route (Hyrule Field, the Market) here with no crash or hang.
Collaborator

Possibly related, for context:

  • PR #1572 WIP: test(lift): the guest context's two definitions are checked to agree on every PC — Related guest-context consistency test
  • PR #25 test(docs): anchors from the functions behind design.md's rules to their docs, and a check they resolve — Merged precedent for doc anchors and anchor check
Possibly related, for context: - PR #1572 WIP: test(lift): the guest context's two definitions are checked to agree on every PC — Related guest-context consistency test - PR #25 test(docs): anchors from the functions behind design.md's rules to their docs, and a check they resolve — Merged precedent for doc anchors and anchor check
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
PR description / Description follows the template (pull_request_target) Successful in 12s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 15s
CI / Workflow lint (actionlint) (pull_request) Successful in 16s
Third-party parts / As their notes say (pull_request) Successful in 18s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request has changes conflicting with the target branch.
  • docs/design.md
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'docs/lift-contract-rules:dyst-docs/lift-contract-rules'
git switch 'dyst-docs/lift-contract-rules'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1626
No description provided.