perf(lift): a Windows lift about a third faster, the same C: fast float checks kept by text, discovery lookups, emit processes started early #1116

Merged
HumanBean merged 3 commits from madebyjordan/Longshot:perf/lift-speed into main 2026-11-16 12:02:56 +00:00
Member

Type: perf · Risk: low · Tested on: Windows 11, RX 6700 XT · Needs: Linux, any GPU (a lift compared with main's)

Why

On Windows the launcher's Translate step is now most of an update: today's updates on a Ryzen 5 5600 spent 235-261 s
translating and 20-56 s compiling. Its "emit and write" part took 131-165 s in those builds, where an 11-14 build
took 46 s: since #38 the fast float regions ran a regex 13 times per instruction, for every one of OoT's 13 million. And on Windows the emit's worker
processes waited ~25 s for work the lift's own process did that they never used.

What changed

The lift writes the same C, byte for byte, in about two thirds of the time on Windows. Three commits:

  • fastfp (nx2/fastfp.py): the per-instruction checks of #38 are kept by the instruction's text (ff_joins,
    ff_convs, a cached fp_is_frame): instruction texts repeat, so 13 million checks become a few hundred thousand.
    ff_eligible also runs its substitution once, not once per word of FF_BAD.
  • discovery (nx2/image.py EdenImage.lookups/word/in_code, nx2/insn.py classify): word reads a 32-bit
    view of the image instead of struct.unpack_from, in_code a flat tuple of ranges, and a word that is a plain op at
    one pc is remembered as one at every pc (neither classify reads the pc for those). New check
    tests/test_lift_lookups.py compares each with the kit's own answers.
  • the Windows emit (nx2/chunks.py spawn_emit, write, no_texts, fill_fpx, lifter_texts; nx2/lift.py):
    • spawn_emit starts its processes when called, and chunks.write calls it first, so they work while the lift's
      process makes the coverage report (write(beside=report); its lines and its lift: coverage step print as before).
    • The lift's process no longer translates every instruction for itself on that path (no_texts: ~15 s and 1-2 GB,
      used only for a64_fpx.h's list of FPX shapes). The processes send their runs' shapes back and fill_fpx writes
      the header from them. If a process stops, the texts are made in the lift's process as before (lifter_texts).
    • The translator DLL is loaded before the processes start, so they never race to build it.

Linux (fork) keeps its emit path; it gets the first two commits.

Measurements

Verdict: faster, same output. The Windows lift's median fell from 258 s to 174 s (-33%) at 4 workers on a busy
PC, and the coverage and emit steps together from 150 s to 80 s (-47%). Every run's lifted C is identical to main's
(sha1 of all 102 files in sa/gen).

Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB (about 6 GB free at idle), Crucial P3 NVMe, RX 6700 XT
Scene: none (the lift alone)
Method: nx2/lift.py --standalone --write as make_game.py runs it (RX_RELEASE=1 A64_HOT_CHUNK=1, the launcher's
image/chunks.txt, the nightly-336 tools' Python 3.14 and llvm-mingw), LONGSHOT_LIFT_JOBS=4 so both sides get the
same workers, through scripts/job_queue.py --mode timing-cpu; 2 interleaved pairs (main, this PR, main, this PR),
the median. Other apps were busy (about 47% CPU in the background), so single runs vary by ±25%; the pairs vary together.
How to reproduce: python -I nx2/lift.py --standalone --write --work NEW_FOLDER with OOT_EXEFS set, on each side, then
compare sha1sum NEW_FOLDER/sa/gen/*.

Before: main @ 8a4aa282 After: this PR @ aaa96fda (now 464286e5)
whole lift (run 1 / run 2) 236.5 s / 279.1 s 204.2 s / 142.7 s
whole lift, median 257.8 s 173.5 s (-33%)
image and discovery (run 1 / run 2) 57 s / 82 s 64 s / 40 s
coverage + emit and write (run 1 / run 2) 141.2 s / 159.4 s 93.6 s / 65.6 s
coverage + emit and write, median 150.3 s 79.6 s (-47%)
lifted C 102 files the same 102 files, byte for byte

Baseline: the table's pairs ran on main @ 8a4aa282 and this PR's commits on it (aaa96fda). Rebased since onto
6c8d6e6a, whose only lift change is #1002's always_inline in a64_fpx.h's helpers (header text, not lift work): one
more pair there, at the default worker count as the launcher runs it, gave main @ 6c8d6e6a 306.8 s (the gate allowed 2
workers; emit and write 206.2 s) and this PR @ 464286e5 162.2 s (3 workers; emit and write 60.9 s), the lifted C again
identical.
On a quieter PC earlier the same day (not interleaved): main 206 s at 4 workers; this PR 158 s at the default's 3
workers and 126 s at 6. The default count is the Windows memory gate (one per 0.75 GB free): the lift's own process
holds 4.3 GB after discovery, so a 16 GB PC usually gets 3-4.

main, run 1:
discovered 294522 functions (57s)
lift: coverage in 23.2s (80s)
lift: emit and write (4 processes) in 118.0s (198s)
lift: big chunks split in 8.1s (206s)
lift: patches (4 processes) in 8.8s (215s)
lift: output size and page-mark groups in 17.5s (233s)
this PR, run 2:
discovered 294522 functions (40s)
lift: coverage in 22.0s (62s)
lift: emit and write (4 processes) in 43.6s (106s)
lift: big chunks split in 7.8s (114s)
lift: patches (4 processes) in 12.0s (126s)
lift: output size and page-mark groups in 14.2s (140s)

Flags and settings

None.

Testing

Tested:

  • The lifted C of every lift above, both sides, compared file by file (sha1 of all 102 files in sa/gen): identical,
    on 8a4aa282 and again on 6c8d6e6a. It
    also matches the launcher's own nightly-336 build on this PC (09:33 today): the same size for each of the 92 code
    files, and the same sha1 for its 10 other files, a64_fpx.h (which fill_fpx now writes) among them. That build
    then ran about 22 minutes of play (80,813 frames presented, Windows 11, RX 6700 XT, AMD 26.10.1). This PR changes no
    byte of the game it builds.
  • tests/test_spawn_emit.py (one process against three spawned ones over a real lifter, the fallback included)
    passes with llvm-mingw's clang++. It fails if fill_fpx is given the wrong shapes (checked by hand).
    tests/test_lift_lookups.py fails if word reads one word off, in_code takes a range's end, or the classify cache
    keeps a branch (each checked by hand).
  • After the rebase onto 6c8d6e6a: test_spawn_emit, test_fastfp_override, test_fastfp_regions, test_lift_lookups,
    test_lift_workers, test_mark_groups, test_stale_hot_pieces, test_light_passes pass.
  • python -I -B tests/run_all.py on Windows (on 8a4aa282): 567 passed, 21 failed, 86 skipped. main @ 8a4aa282: 566 passed and the
    same 21 failed (test_vk_occlusion_queries, test_crash_dialog, test_linux_preflight, test_job_queue,
    test_make_game_cache, test_bug_report_defaults, test_run_all_changed, test_tested_configurations,
    test_portable_builder_home, test_pending_guest_import, test_missing_nup, test_guest_imported,
    test_walk_key, test_pads_after_menu, test_auto_handheld, test_failed_compile_keeps_objects,
    test_portable_builder_startup, test_bug_report, test_crt_options, test_audio_output_devices,
    test_hardware_report). The one extra pass is the new check.

Not tested: a lift on Linux or macOS (forked emit processes: their path is unchanged, but the first two commits run
there too), and the lift's time inside a whole make_game.py build (the steps are the same ones).

Must be tested:

  • Linux, any GPU (the fork path; I'm on Windows): a lift of the same game on main and on this PR
    (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, or two make_game.py builds), expecting the same
    sha1 for every file of the work folder's sa/gen (Test Farm: @Beanstalk linux, the lift on main and on this PR, comparing
    the files of sa/gen)

Risks

What could break: the lift, where it would show as a failed lift or a build that differs from main's. On Windows
no_texts swaps a64emit.translate and a64_lift.op_text for stand-ins while the lifter writes, restored after
(also on an error), and fill_fpx stops the lift if a64_fpx.h doesn't begin as expected. The caches only hold
answers that depend on the text alone.

Turning it off: revert the commits (LONGSHOT_LIFT_JOBS=1 also takes the Windows emit off the new path).

Compatibility: the lift key changes (translator files changed), so the next update translates the game once, as
for any translator change. The C it writes is the same as before, so the compile finds the same object keys; shader
caches are not touched.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.

🤖 Generated with Claude Code

**Type:** perf · **Risk:** low · **Tested on:** Windows 11, RX 6700 XT · **Needs:** Linux, any GPU (a lift compared with main's) ## Why On Windows the launcher's Translate step is now most of an update: today's updates on a Ryzen 5 5600 spent 235-261 s translating and 20-56 s compiling. Its "emit and write" part took 131-165 s in those builds, where an 11-14 build took 46 s: since #38 the fast float regions ran a regex 13 times per instruction, for every one of OoT's 13 million. And on Windows the emit's worker processes waited ~25 s for work the lift's own process did that they never used. ## What changed The lift writes the same C, byte for byte, in about two thirds of the time on Windows. Three commits: - fastfp (`nx2/fastfp.py`): the per-instruction checks of #38 are kept by the instruction's text (`ff_joins`, `ff_convs`, a cached `fp_is_frame`): instruction texts repeat, so 13 million checks become a few hundred thousand. `ff_eligible` also runs its substitution once, not once per word of `FF_BAD`. - discovery (`nx2/image.py` `EdenImage.lookups`/`word`/`in_code`, `nx2/insn.py` `classify`): `word` reads a 32-bit view of the image instead of `struct.unpack_from`, `in_code` a flat tuple of ranges, and a word that is a plain op at one pc is remembered as one at every pc (neither classify reads the pc for those). New check `tests/test_lift_lookups.py` compares each with the kit's own answers. - the Windows emit (`nx2/chunks.py` `spawn_emit`, `write`, `no_texts`, `fill_fpx`, `lifter_texts`; `nx2/lift.py`): - `spawn_emit` starts its processes when called, and `chunks.write` calls it first, so they work while the lift's process makes the coverage report (`write(beside=report)`; its lines and its `lift: coverage` step print as before). - The lift's process no longer translates every instruction for itself on that path (`no_texts`: ~15 s and 1-2 GB, used only for `a64_fpx.h`'s list of FPX shapes). The processes send their runs' shapes back and `fill_fpx` writes the header from them. If a process stops, the texts are made in the lift's process as before (`lifter_texts`). - The translator DLL is loaded before the processes start, so they never race to build it. Linux (fork) keeps its emit path; it gets the first two commits. ## Measurements **Verdict:** faster, same output. The Windows lift's median fell from 258 s to 174 s (-33%) at 4 workers on a busy PC, and the coverage and emit steps together from 150 s to 80 s (-47%). Every run's lifted C is identical to main's (sha1 of all 102 files in `sa/gen`). Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB (about 6 GB free at idle), Crucial P3 NVMe, RX 6700 XT Scene: none (the lift alone) Method: `nx2/lift.py --standalone --write` as `make_game.py` runs it (`RX_RELEASE=1 A64_HOT_CHUNK=1`, the launcher's `image/chunks.txt`, the nightly-336 tools' Python 3.14 and llvm-mingw), `LONGSHOT_LIFT_JOBS=4` so both sides get the same workers, through `scripts/job_queue.py --mode timing-cpu`; 2 interleaved pairs (main, this PR, main, this PR), the median. Other apps were busy (about 47% CPU in the background), so single runs vary by ±25%; the pairs vary together. How to reproduce: `python -I nx2/lift.py --standalone --write --work NEW_FOLDER` with `OOT_EXEFS` set, on each side, then compare `sha1sum NEW_FOLDER/sa/gen/*`. | | Before: main @ 8a4aa282 | After: this PR @ aaa96fda (now 464286e5) | |---|---|---| | whole lift (run 1 / run 2) | 236.5 s / 279.1 s | 204.2 s / 142.7 s | | whole lift, median | 257.8 s | 173.5 s (-33%) | | image and discovery (run 1 / run 2) | 57 s / 82 s | 64 s / 40 s | | coverage + emit and write (run 1 / run 2) | 141.2 s / 159.4 s | 93.6 s / 65.6 s | | coverage + emit and write, median | 150.3 s | 79.6 s (-47%) | | lifted C | 102 files | the same 102 files, byte for byte | Baseline: the table's pairs ran on main @ 8a4aa282 and this PR's commits on it (aaa96fda). Rebased since onto 6c8d6e6a, whose only lift change is #1002's `always_inline` in `a64_fpx.h`'s helpers (header text, not lift work): one more pair there, at the default worker count as the launcher runs it, gave main @ 6c8d6e6a 306.8 s (the gate allowed 2 workers; emit and write 206.2 s) and this PR @ 464286e5 162.2 s (3 workers; emit and write 60.9 s), the lifted C again identical. On a quieter PC earlier the same day (not interleaved): main 206 s at 4 workers; this PR 158 s at the default's 3 workers and 126 s at 6. The default count is the Windows memory gate (one per 0.75 GB free): the lift's own process holds 4.3 GB after discovery, so a 16 GB PC usually gets 3-4. ``` main, run 1: discovered 294522 functions (57s) lift: coverage in 23.2s (80s) lift: emit and write (4 processes) in 118.0s (198s) lift: big chunks split in 8.1s (206s) lift: patches (4 processes) in 8.8s (215s) lift: output size and page-mark groups in 17.5s (233s) this PR, run 2: discovered 294522 functions (40s) lift: coverage in 22.0s (62s) lift: emit and write (4 processes) in 43.6s (106s) lift: big chunks split in 7.8s (114s) lift: patches (4 processes) in 12.0s (126s) lift: output size and page-mark groups in 14.2s (140s) ``` ### Flags and settings None. ## Testing **Tested:** - The lifted C of every lift above, both sides, compared file by file (sha1 of all 102 files in `sa/gen`): identical, on 8a4aa282 and again on 6c8d6e6a. It also matches the launcher's own nightly-336 build on this PC (09:33 today): the same size for each of the 92 code files, and the same sha1 for its 10 other files, `a64_fpx.h` (which `fill_fpx` now writes) among them. That build then ran about 22 minutes of play (80,813 frames presented, Windows 11, RX 6700 XT, AMD 26.10.1). This PR changes no byte of the game it builds. - `tests/test_spawn_emit.py` (one process against three spawned ones over a real lifter, the fallback included) passes with llvm-mingw's clang++. It fails if `fill_fpx` is given the wrong shapes (checked by hand). `tests/test_lift_lookups.py` fails if `word` reads one word off, `in_code` takes a range's end, or the classify cache keeps a branch (each checked by hand). - After the rebase onto 6c8d6e6a: `test_spawn_emit`, `test_fastfp_override`, `test_fastfp_regions`, `test_lift_lookups`, `test_lift_workers`, `test_mark_groups`, `test_stale_hot_pieces`, `test_light_passes` pass. - `python -I -B tests/run_all.py` on Windows (on 8a4aa282): 567 passed, 21 failed, 86 skipped. main @ 8a4aa282: 566 passed and the same 21 failed (`test_vk_occlusion_queries`, `test_crash_dialog`, `test_linux_preflight`, `test_job_queue`, `test_make_game_cache`, `test_bug_report_defaults`, `test_run_all_changed`, `test_tested_configurations`, `test_portable_builder_home`, `test_pending_guest_import`, `test_missing_nup`, `test_guest_imported`, `test_walk_key`, `test_pads_after_menu`, `test_auto_handheld`, `test_failed_compile_keeps_objects`, `test_portable_builder_startup`, `test_bug_report`, `test_crt_options`, `test_audio_output_devices`, `test_hardware_report`). The one extra pass is the new check. **Not tested:** a lift on Linux or macOS (forked emit processes: their path is unchanged, but the first two commits run there too), and the lift's time inside a whole `make_game.py` build (the steps are the same ones). **Must be tested:** - Linux, any GPU (the fork path; I'm on Windows): a lift of the same game on main and on this PR (`python -I nx2/lift.py --standalone --write --work NEW_FOLDER`, or two `make_game.py` builds), expecting the same sha1 for every file of the work folder's `sa/gen` (Test Farm: @Beanstalk linux, the lift on main and on this PR, comparing the files of sa/gen) ## Risks **What could break:** the lift, where it would show as a failed lift or a build that differs from main's. On Windows `no_texts` swaps `a64emit.translate` and `a64_lift.op_text` for stand-ins while the lifter writes, restored after (also on an error), and `fill_fpx` stops the lift if `a64_fpx.h` doesn't begin as expected. The caches only hold answers that depend on the text alone. **Turning it off:** revert the commits (`LONGSHOT_LIFT_JOBS=1` also takes the Windows emit off the new path). **Compatibility:** the lift key changes (translator files changed), so the next update translates the game once, as for any translator change. The C it writes is the same as before, so the compile finds the same object keys; shader caches are not touched. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
ff_eligible ran its FF_FP_OK.sub once for each of FF_BAD's 13 words, on every instruction the emit
walked (13 million for OoT, 172 million regex substitutions), and ff_region joined and searched
every run however short. Since #38 the emit and write took 106 s where it took ~46 s. The answers
are now kept by the instruction's text (ff_joins, ff_convs, fp_is_frame), which repeat: the lifted
C is byte for byte the same.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The discovery reads 40 million words through EdenImage.word and in_code and classifies 18 million:
word is now a 32-bit view of the image, in_code a flat tuple of ranges, and a word known to be a
plain op at one pc is one at every pc (nx2/insn.py classify). tests/test_lift_lookups.py checks
each against the kit's own answers.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
perf(lift): on Windows the emit's processes start before the lifter's own texts, which it no longer makes
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 16s
PR description / Description follows the template (pull_request_target) Successful in 4s
beanstalk/approval needs a second approval (sensitive paths)
464286e5f3
Where there is no fork, each emit process translates its own run, yet they started only once the
lift's process had translated every instruction itself (~15 s for OoT, 1-2 GB, used by nothing but
a64_fpx.h's list of FPX shapes) and had made the coverage report (~12 s). spawn_emit now starts
them when chunks.write begins; the coverage report runs beside them (write's beside), and the
lifter writes with no texts of its own (no_texts). The processes send their runs' FPX shapes back
and fill_fpx writes a64_fpx.h from them. Should a process stop, the texts are made in the lift's
process as before (lifter_texts). The lifted C is byte for byte the same (tests/test_spawn_emit.py
compares one process and three, the fallback too).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the game it builds. Its lifted C is identical to main's (sha1 of all 102 files of sa/gen, on 8a4aa282 and 6c8d6e6a) and to the launcher's nightly-336 build on this PC from 09:33 today (sizes of the 92 code files, sha1 of the other 10, a64_fpx.h among them). That build then ran about 22 minutes of play (80,813 frames presented, logs/longshot_log.txt) with no problem.

Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the game it builds. Its lifted C is identical to main's (sha1 of all 102 files of sa/gen, on 8a4aa282 and 6c8d6e6a) and to the launcher's nightly-336 build on this PC from 09:33 today (sizes of the 92 code files, sha1 of the other 10, a64_fpx.h among them). That build then ran about 22 minutes of play (80,813 frames presented, logs/longshot_log.txt) with no problem.
madebyjordan changed title from WIP: perf(lift): a Windows lift about a third faster, the same C: fast float checks kept by text, discovery lookups, emit processes started early to perf(lift): a Windows lift about a third faster, the same C: fast float checks kept by text, discovery lookups, emit processes started early 2026-11-16 11:43:43 +00:00
Author
Member

@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical, and the PR's lift no slower than main's (the lift: lines of each log).

@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical, and the PR's lift no slower than main's (the lift: lines of each log).
Member

@madebyjordan approval of 464286e recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@madebyjordan approval of `464286e` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
Member

Beanstalk test farm

Commit 464286e · tier untrusted · approved by madebyjordan · dashboard

This diff touches sensitive paths, so it needs two members of BigBeans: nx2/chunks.py, nx2/fastfp.py, nx2/image.py, nx2/insn.py, nx2/lift.py

A member of BigBeans can review this commit and comment @Beanstalk to test it.

<!-- beanstalk:sticky --> ### Beanstalk test farm Commit `464286e` · tier **untrusted** · approved by madebyjordan · [dashboard](https://beanstalk.magicbean.dev/prs/1116) This diff touches sensitive paths, so it needs two members of `BigBeans`: `nx2/chunks.py`, `nx2/fastfp.py`, `nx2/image.py`, `nx2/insn.py`, `nx2/lift.py` A member of `BigBeans` can review this commit and comment `@Beanstalk` to test it.
HumanBean deleted branch perf/lift-speed 2026-11-16 12:02:56 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1116
No description provided.