perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster #1131

Merged
HumanBean merged 2 commits from madebyjordan/Longshot:perf/lift-discover into main 2026-11-16 22:35:50 +00:00
Member

Type: perf · Risk: medium · Tested on: Windows 11, RX 6700 XT · Needs: Linux, any GPU (a lift compared with main's)

Why

After #1116 and #1128 the biggest fixed part of a Windows lift is the discovery: the kit walks all 294,000 of OoT's
entries on one core, ~31 s on a quiet Ryzen 5 5600 and ~60 s when other programs keep the PC busy, while the other cores
wait.

What changed

The discovery's walks run in new processes, in parts of consecutive addresses, and the result is the kit's own.

  • nx2/discover.py (new), called from nx2/lift.py in place of L.discover(...):
    • parts_of cuts the sorted entries into about 3 parts a process, each cut where the unwind range changes.
    • Each part is walked in a new process by a lifter of its own (walk_part, fresh funcs, jt, jt_missing,
      stats and predecessor cache), over the image the lift already wrote (image/image.bin) and the lifter's entries,
      hooks and unwind starts. A worker whose walk is not the lift's (identities) stops.
    • Then the kit's discover runs as it is, its first loop given those bodies (merge); the jump-table retry, PLT
      stubs and setjmp owners walk as before.
  • Why the parts give the kit's result: one walk sees another's only through the predecessors of an unwind range
    (preds, read by reaching_constants), which hold the jump tables found so far in that range. Each part reports the
    ranges it found tables in and the ranges whose predecessors it read; when two parts meet in a range that was read
    (conflicts), the parts are dropped and the kit walks everything in order, as it does when a worker stops or raises.
    OoT: 1,475 jump tables, 435 of them through the predecessors, no range shared.
  • Not used with LONGSHOT_LIFT_JOBS=1, where there is fork (Linux, macOS: unchanged), for another image layout, a
    Lifter subclass or fewer than MIN_ENTRIES (4,000) entries.
  • tests/test_discover.py (new: the parts against the kit in one process and through 2 spawned processes, the workers'
    checks, the fallbacks); tests/test_oot_fov.py and tests/test_oot_shadow.py look for the new call in
    nx2/lift.py (discover.discover(L) where they looked for L.discover(: they check the patches are installed
    before the discovery.

Measurements

Verdict: faster, same output. Against main @ 983d607b (with #1128): the Windows lift's median fell from 158.5 s to
110.8 s (-30%), the image and discovery from 36 s to 20 s; like for like (both sides' emit in 4-5 processes) 131.8 s
against 106.9-114.6 s (-13 to -19%). The lifted C is main's, byte for byte.

Baseline: main @ 983d607b (rebased; #1128 merged). The first measurement, against #1128 before the rebase, is below.
Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB, Crucial P3 NVMe, RX 6700 XT
Scene: none (the lift alone)
Method: nx2/lift.py --standalone --write as make_game.py runs it (RX_RELEASE=1 A64_HOT_CHUNK=1, the launcher's
image/chunks.txt, the nightly-351 tools' Python 3.14 and llvm-mingw), the default worker count, through
scripts/job_queue.py --mode timing-cpu; 2 interleaved pairs, the median.
How to reproduce: python -I nx2/lift.py --standalone --write --work NEW_FOLDER with OOT_EXEFS set, on each side, then
compare sha1sum NEW_FOLDER/sa/gen/*.

Before: main @ 983d607b After: this PR @ 27909cb6
whole lift (run 1 / run 2) 185.1 s / 131.8 s 114.6 s / 106.9 s
whole lift, median 158.5 s 110.8 s (-30%)
image and discovery (run 1 / run 2) 36 s / 35 s 21 s / 19 s
discovery's processes 1 7 / 8 (21 / 24 parts)
emit processes (the memory gate) 2 / 4 4 / 5
lifted C 102 files the same 102 files, byte for byte

Main's first run had memory for only 2 emit processes (89.8 s of emit); its second, with 4, is the like-for-like one.

The first measurement, against #1128 before the rebase (the same method with the nightly-336 tools; other programs used
about 45% of the CPU throughout, which slows a one-core discovery most):

Before: #1128 @ 68f5e131 After: this PR @ 2a65abcb
whole lift (run 1 / run 2) 195.5 s / 178.3 s 150.5 s / 126.5 s
whole lift, median 186.9 s 138.5 s (-26%)
image and discovery (run 1 / run 2) 58 s / 62 s 37 s / 31 s
discovery's processes 1 6 / 8 (18 / 24 parts)
lifted C 102 files the same 102 files, byte for byte (and main @ 71b1edd8's)

On its own (a script running only the discovery, 6 processes): 20.5 s against ~31 s for the kit's, and the lift's
process held 2.72 GB after it (3.02 GB with #1128's walk alone).

main @ 983d607b, run 2:
discovered 294522 functions (35s)
lift: coverage in 22.8s (58s)
lift: emit and write (4 processes) in 37.3s (95s)
this PR @ 27909cb6, run 2:
discovery: 24 parts walked in 8 processes
discovered 294522 functions (19s)
lift: coverage in 23.2s (43s)
lift: emit and write (5 processes) in 29.4s (72s)

Flags and settings

None (LONGSHOT_LIFT_JOBS=1 already turns the processes off).

Testing

Tested:

  • Every lift above, and two more, compared file by file (sha1 of all 102 files of sa/gen): after the rebase identical
    to main @ 983d607b's, before it to #1128's and main @ 71b1edd8's. Every Windows lift of this PR printed
    discovery: N parts walked in M processes (no fallback). A script also split the discovery into 6 lifters of its own in one process and compared all of
    their state with the kit's: functions, tables, misses and counters all equal.
  • In the game: the lifted C is main's, byte for byte; 101 of its 102 files are those of the launcher's nightly-336 build
    on this PC, which ran about 22 minutes of play today (Windows 11, RX 6700 XT, AMD 26.10.1; the other file is
    a64_fpx.h, changed by main's #1002).
  • tests/test_discover.py runs the discovery through 2 real spawned processes over a made-up image written to an
    image.bin, and in one process: both give the kit's state. It fails if merge drops the parts' counters, parts_of
    cuts inside an unwind range, conflicts misses a range read by one part where another found a table, a worker skips
    the image's base address, the workers' walk check is dropped, or parts that meet are merged anyway (each checked by
    hand).
  • python -I -B tests/run_all.py on Windows @ 27909cb6: 582 passed, 27 failed, 88 skipped; main @ 983d607b on the same
    PC at the same time: 581 passed and the same 27 failed (test_auto_handheld, test_bug_report,
    test_bug_report_defaults, test_crash_dialog, test_crt_options, test_damage_debug_menu, test_dlss_feature_size,
    test_failed_compile_keeps_objects, test_framegen_preset, test_guest_imported, test_hardware_report,
    test_job_queue, test_linux_preflight, test_make_game_cache, test_menu_i18n, test_missing_nup,
    test_pads_after_menu, test_pending_guest_import, test_portable_builder_home, test_portable_builder_startup,
    test_rumble_menu, test_run_all_changed, test_tested_configurations, test_upscaler_round_trip,
    test_upscaler_rows, test_vk_occlusion_queries, test_walk_key).
  • Test Farm @ 2a65abcb (before the rebase): on Windows (RX 9070 XT) the game was built and played with no crash; its lift
    took 98.9 s against main's 115.5 s. It failed on one field picture (0.15 against a 0.10 limit) and on the Deku Tree's
    p99 frame time (+30%, from one 164 ms hitch; p50 -0.5%), with one run a side. The lifted C is main's byte for byte, so
    the game code is the same, and I read these as run-to-run variation; a rerun is asked for. Linux: the route's inputs
    timed out (an error, not a result).

Not tested: Linux and macOS (unchanged: the processes are only started where there is no fork; a Linux lift does not
exercise them), and a game
whose parts would meet in a searched range (the fallback is the kit's discover; tests/test_discover.py runs it).

Must be tested:

  • Linux, any GPU (I'm on Windows): a lift of the same game on main and on this PR
    (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, or a make_game.py build of each), expecting the same
    sha1 for every file of the work folder's sa/gen (Test Farm: @Beanstalk linux, the lift on main and on this PR,
    comparing the files of sa/gen)

Risks

What could break: the discovery on Windows: a different result would show as lifted C that differs from main's (the
conflict rule exists so it can't; a range the rule misses would be a bug in it). A worker that can't start, stops or
raises gives the kit's discover, with a discovery: ... walked here line. The processes take memory beside the lift's
own while they run (each its image, 100 MB, and its part's bodies); they end before the emit's start.

Turning it off: LONGSHOT_LIFT_JOBS=1 (which also keeps the emit in one process), or revert the commit.

Compatibility: the lift key changes (translator files changed), so the next update translates the game once. The C
is the same as before, so the compile finds the same object keys; shader caches are not touched.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • A check under tests/ that reads changed source updated in this PR, not weakened.

🤖 Generated with Claude Code

**Type:** perf · **Risk:** medium · **Tested on:** Windows 11, RX 6700 XT · **Needs:** Linux, any GPU (a lift compared with main's) ## Why After #1116 and #1128 the biggest fixed part of a Windows lift is the discovery: the kit walks all 294,000 of OoT's entries on one core, ~31 s on a quiet Ryzen 5 5600 and ~60 s when other programs keep the PC busy, while the other cores wait. ## What changed The discovery's walks run in new processes, in parts of consecutive addresses, and the result is the kit's own. - `nx2/discover.py` (new), called from `nx2/lift.py` in place of `L.discover(...)`: - `parts_of` cuts the sorted entries into about 3 parts a process, each cut where the unwind range changes. - Each part is walked in a new process by a lifter of its own (`walk_part`, fresh `funcs`, `jt`, `jt_missing`, `stats` and predecessor cache), over the image the lift already wrote (`image/image.bin`) and the lifter's entries, hooks and unwind starts. A worker whose walk is not the lift's (`identities`) stops. - Then the kit's `discover` runs as it is, its first loop given those bodies (`merge`); the jump-table retry, PLT stubs and setjmp owners walk as before. - Why the parts give the kit's result: one walk sees another's only through the predecessors of an unwind range (`preds`, read by `reaching_constants`), which hold the jump tables found so far in that range. Each part reports the ranges it found tables in and the ranges whose predecessors it read; when two parts meet in a range that was read (`conflicts`), the parts are dropped and the kit walks everything in order, as it does when a worker stops or raises. OoT: 1,475 jump tables, 435 of them through the predecessors, no range shared. - Not used with `LONGSHOT_LIFT_JOBS=1`, where there is fork (Linux, macOS: unchanged), for another image layout, a Lifter subclass or fewer than `MIN_ENTRIES` (4,000) entries. - `tests/test_discover.py` (new: the parts against the kit in one process and through 2 spawned processes, the workers' checks, the fallbacks); `tests/test_oot_fov.py` and `tests/test_oot_shadow.py` look for the new call in `nx2/lift.py` (`discover.discover(L`) where they looked for `L.discover(`: they check the patches are installed before the discovery. ## Measurements **Verdict:** faster, same output. Against main @ 983d607b (with #1128): the Windows lift's median fell from 158.5 s to 110.8 s (-30%), the image and discovery from 36 s to 20 s; like for like (both sides' emit in 4-5 processes) 131.8 s against 106.9-114.6 s (-13 to -19%). The lifted C is main's, byte for byte. Baseline: main @ 983d607b (rebased; #1128 merged). The first measurement, against #1128 before the rebase, is below. Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB, Crucial P3 NVMe, RX 6700 XT Scene: none (the lift alone) Method: `nx2/lift.py --standalone --write` as `make_game.py` runs it (`RX_RELEASE=1 A64_HOT_CHUNK=1`, the launcher's `image/chunks.txt`, the nightly-351 tools' Python 3.14 and llvm-mingw), the default worker count, through `scripts/job_queue.py --mode timing-cpu`; 2 interleaved pairs, the median. How to reproduce: `python -I nx2/lift.py --standalone --write --work NEW_FOLDER` with `OOT_EXEFS` set, on each side, then compare `sha1sum NEW_FOLDER/sa/gen/*`. | | Before: main @ 983d607b | After: this PR @ 27909cb6 | |---|---|---| | whole lift (run 1 / run 2) | 185.1 s / 131.8 s | 114.6 s / 106.9 s | | whole lift, median | 158.5 s | 110.8 s (-30%) | | image and discovery (run 1 / run 2) | 36 s / 35 s | 21 s / 19 s | | discovery's processes | 1 | 7 / 8 (21 / 24 parts) | | emit processes (the memory gate) | 2 / 4 | 4 / 5 | | lifted C | 102 files | the same 102 files, byte for byte | Main's first run had memory for only 2 emit processes (89.8 s of emit); its second, with 4, is the like-for-like one. The first measurement, against #1128 before the rebase (the same method with the nightly-336 tools; other programs used about 45% of the CPU throughout, which slows a one-core discovery most): | | Before: #1128 @ 68f5e131 | After: this PR @ 2a65abcb | |---|---|---| | whole lift (run 1 / run 2) | 195.5 s / 178.3 s | 150.5 s / 126.5 s | | whole lift, median | 186.9 s | 138.5 s (-26%) | | image and discovery (run 1 / run 2) | 58 s / 62 s | 37 s / 31 s | | discovery's processes | 1 | 6 / 8 (18 / 24 parts) | | lifted C | 102 files | the same 102 files, byte for byte (and main @ 71b1edd8's) | On its own (a script running only the discovery, 6 processes): 20.5 s against ~31 s for the kit's, and the lift's process held 2.72 GB after it (3.02 GB with #1128's walk alone). ``` main @ 983d607b, run 2: discovered 294522 functions (35s) lift: coverage in 22.8s (58s) lift: emit and write (4 processes) in 37.3s (95s) this PR @ 27909cb6, run 2: discovery: 24 parts walked in 8 processes discovered 294522 functions (19s) lift: coverage in 23.2s (43s) lift: emit and write (5 processes) in 29.4s (72s) ``` ### Flags and settings None (`LONGSHOT_LIFT_JOBS=1` already turns the processes off). ## Testing **Tested:** - Every lift above, and two more, compared file by file (sha1 of all 102 files of `sa/gen`): after the rebase identical to main @ 983d607b's, before it to #1128's and main @ 71b1edd8's. Every Windows lift of this PR printed `discovery: N parts walked in M processes` (no fallback). A script also split the discovery into 6 lifters of its own in one process and compared all of their state with the kit's: functions, tables, misses and counters all equal. - In the game: the lifted C is main's, byte for byte; 101 of its 102 files are those of the launcher's nightly-336 build on this PC, which ran about 22 minutes of play today (Windows 11, RX 6700 XT, AMD 26.10.1; the other file is `a64_fpx.h`, changed by main's #1002). - `tests/test_discover.py` runs the discovery through 2 real spawned processes over a made-up image written to an `image.bin`, and in one process: both give the kit's state. It fails if `merge` drops the parts' counters, `parts_of` cuts inside an unwind range, `conflicts` misses a range read by one part where another found a table, a worker skips the image's base address, the workers' walk check is dropped, or parts that meet are merged anyway (each checked by hand). - `python -I -B tests/run_all.py` on Windows @ 27909cb6: 582 passed, 27 failed, 88 skipped; main @ 983d607b on the same PC at the same time: 581 passed and the same 27 failed (`test_auto_handheld`, `test_bug_report`, `test_bug_report_defaults`, `test_crash_dialog`, `test_crt_options`, `test_damage_debug_menu`, `test_dlss_feature_size`, `test_failed_compile_keeps_objects`, `test_framegen_preset`, `test_guest_imported`, `test_hardware_report`, `test_job_queue`, `test_linux_preflight`, `test_make_game_cache`, `test_menu_i18n`, `test_missing_nup`, `test_pads_after_menu`, `test_pending_guest_import`, `test_portable_builder_home`, `test_portable_builder_startup`, `test_rumble_menu`, `test_run_all_changed`, `test_tested_configurations`, `test_upscaler_round_trip`, `test_upscaler_rows`, `test_vk_occlusion_queries`, `test_walk_key`). - Test Farm @ 2a65abcb (before the rebase): on Windows (RX 9070 XT) the game was built and played with no crash; its lift took 98.9 s against main's 115.5 s. It failed on one field picture (0.15 against a 0.10 limit) and on the Deku Tree's p99 frame time (+30%, from one 164 ms hitch; p50 -0.5%), with one run a side. The lifted C is main's byte for byte, so the game code is the same, and I read these as run-to-run variation; a rerun is asked for. Linux: the route's inputs timed out (an error, not a result). **Not tested:** Linux and macOS (unchanged: the processes are only started where there is no fork; a Linux lift does not exercise them), and a game whose parts would meet in a searched range (the fallback is the kit's discover; `tests/test_discover.py` runs it). **Must be tested:** - Linux, any GPU (I'm on Windows): a lift of the same game on main and on this PR (`python -I nx2/lift.py --standalone --write --work NEW_FOLDER`, or a `make_game.py` build of each), expecting the same sha1 for every file of the work folder's `sa/gen` (Test Farm: @Beanstalk linux, the lift on main and on this PR, comparing the files of sa/gen) ## Risks **What could break:** the discovery on Windows: a different result would show as lifted C that differs from main's (the conflict rule exists so it can't; a range the rule misses would be a bug in it). A worker that can't start, stops or raises gives the kit's discover, with a `discovery: ... walked here` line. The processes take memory beside the lift's own while they run (each its image, 100 MB, and its part's bodies); they end before the emit's start. **Turning it off:** `LONGSHOT_LIFT_JOBS=1` (which also keeps the emit in one process), or revert the commit. **Compatibility:** the lift key changes (translator files changed), so the next update translates the game once. The C is the same as before, so the compile finds the same object keys; shader caches are not touched. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The kit's walk stores every instruction as its own (kind, word) tuple: 13.1 million for OoT, of which 12.7
million plain ops of 1.16 million distinct words. The lift's process held about 1.2 GB of duplicates
through the emit, and Windows starts one emit process per 0.75 GB free, so a 16 GB PC got 2-3 where it
now gets 5. nx2/walk.py is the kit's walk with the image's lookups inline and one shared tuple per plain
op's word (kept after the fact, the memory stays with Python's allocator, so it is done as the walk makes
them). tests/test_walk.py compares it with the kit's walk over a made-up image and pins the kit's tree.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
perf(lift): the emit asks dead_flags only of functions that write a flag, and reads A64_FASTFP_MIN once a function
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 15s
PR description / Description follows the template (pull_request_target) Successful in 6s
beanstalk/approval approved by futondestroyer, madebyjordan (double-reviewed)
beanstalk/linux 1/1 pass
68f5e131e9
The kit's dead_flags (which flag stores nothing reads) can only find a dead store where an instruction
writes a flag: 58% of OoT's functions have none, and its pass and fixpoint over them returned {} after
all that work (nx2/dead_flags.py answers them at once). ff_region read A64_FASTFP_MIN from the
environment for each of 3.8 million runs; emit_function now reads it once and passes it on. The lifted C
is the same.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the lifted C. It is identical to #1128's and main @ 71b1edd8's (sha1 of all 102 files of sa/gen, four lifts), and 101 of those 102 files are the launcher's nightly-336 build on this PC from 09:33 today, which then ran about 22 minutes of play (80,813 frames presented) with no problem; the other, a64_fpx.h, differs by main's own #1002.

Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the lifted C. It is identical to #1128's and main @ 71b1edd8's (sha1 of all 102 files of sa/gen, four lifts), and 101 of those 102 files are the launcher's nightly-336 build on this PC from 09:33 today, which then ran about 22 minutes of play (80,813 frames presented) with no problem; the other, a64_fpx.h, differs by main's own #1002.
madebyjordan changed title from WIP: perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster to perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster 2026-11-16 13:27:12 +00:00
Author
Member

@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical (Linux keeps the one-process discovery; the walk and dead_flags changes from #1128 run there).

@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical (Linux keeps the one-process discovery; the walk and dead_flags changes from #1128 run there).
Member

@madebyjordan approval of 2a65abc recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@madebyjordan approval of `2a65abc` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
Member

Beanstalk test farm

Commit 27909cb · tier untrusted · approved by madebyjordan · dashboard

This diff touches sensitive paths, so it needs two members of BigBeans: nx2/discover.py, nx2/lift.py

A member of BigBeans can review this commit and comment @Beanstalk to test it.

<!-- beanstalk:sticky --> ### Beanstalk test farm Commit `27909cb` · tier **untrusted** · approved by madebyjordan · [dashboard](https://beanstalk.magicbean.dev/prs/1131) This diff touches sensitive paths, so it needs two members of `BigBeans`: `nx2/discover.py`, `nx2/lift.py` A member of `BigBeans` can review this commit and comment `@Beanstalk` to test it.
futondestroyer left a comment

Code-and-evidence review of 2a65abcb95c3efb849ec9d784f1fd74e98b60e84, with this PR's own discovery change compared against its dependency #1128 at 68f5e131. No tests, builds, generators or gameplay run for this review.

No blocking code defect found. The workers receive the written image, entries/hooks and unwind starts; each part resets discovery state and the predecessor cache. The partition preserves address order within a range. The conflict check conservatively discards candidate results when another part could affect a searched predecessor range, before merging anything into the parent. On success the original discover still owns retry, splitting, PLT and setjmp processing, and consumed bodies fall back to the real walk on subsequent visits. The temporary walk override is removed in finally. Existing serial/fork and single-worker paths remain available.

Evidence limits and merge prerequisites:

  • This is stacked on #1128. Land that first, then verify the resulting diff/head; the reported performance baseline here is that dependency, not bare main. The claimed 26% improvement is from a busy Windows PC and should not be generalized to a quiet PC or Linux.
  • The new authored test compares serial/partitioned state through an in-process stand-in, and checks conflicts and raised-worker fallback. It does not exercise actual spawn initialization, image reconstruction or callable-identity rejection. A useful additional game-free check would spawn over a synthetic image and compare full discovery state; an integration test of the conflict-triggered fallback would also strengthen coverage. These are coverage suggestions, not a demonstrated production defect.
  • The author's repeated Windows lifts and file comparisons are meaningful evidence, but the stated Linux comparison remains outstanding. Linux normally takes the serial path, so a Linux pass does not validate the new Windows multiprocessing path. Include another Windows build when farm policy/workers support it, and check whether discovery really used processes or fell back.
  • The current forge statuses show Guide screenshots / Guide images and local UI source review failing; Python source-check jobs are skipped. Clear or explain that failure before merge. This review did not independently reproduce the author's full-suite results; the description says its full run preceded the two wiring-test corrections, which were rerun individually.
  • The farm's documented probes cannot compare every generated sa/gen file or establish lift-stage speed. A farm build/gameplay pass is supplemental validation, not proof of byte-identical generated C or the claimed lift speed. Keep all generated files local; an independent lift comparison should report only the equality verdict/counts and timings.

Verdict: ready for farm validation; merge approval deferred pending the outstanding evidence and CI disposition. I will supply the second sensitive-path farm authorization separately.

Code-and-evidence review of `2a65abcb95c3efb849ec9d784f1fd74e98b60e84`, with this PR's own discovery change compared against its dependency https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1128 at `68f5e131`. No tests, builds, generators or gameplay run for this review. No blocking code defect found. The workers receive the written image, entries/hooks and unwind starts; each part resets discovery state and the predecessor cache. The partition preserves address order within a range. The conflict check conservatively discards candidate results when another part could affect a searched predecessor range, before merging anything into the parent. On success the original discover still owns retry, splitting, PLT and setjmp processing, and consumed bodies fall back to the real walk on subsequent visits. The temporary walk override is removed in finally. Existing serial/fork and single-worker paths remain available. Evidence limits and merge prerequisites: - This is stacked on https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1128. Land that first, then verify the resulting diff/head; the reported performance baseline here is that dependency, not bare main. The claimed 26% improvement is from a busy Windows PC and should not be generalized to a quiet PC or Linux. - The new authored test compares serial/partitioned state through an in-process stand-in, and checks conflicts and raised-worker fallback. It does not exercise actual spawn initialization, image reconstruction or callable-identity rejection. A useful additional game-free check would spawn over a synthetic image and compare full discovery state; an integration test of the conflict-triggered fallback would also strengthen coverage. These are coverage suggestions, not a demonstrated production defect. - The author's repeated Windows lifts and file comparisons are meaningful evidence, but the stated Linux comparison remains outstanding. Linux normally takes the serial path, so a Linux pass does not validate the new Windows multiprocessing path. Include another Windows build when farm policy/workers support it, and check whether discovery really used processes or fell back. - The current forge statuses show `Guide screenshots / Guide images and local UI source review` failing; Python source-check jobs are skipped. Clear or explain that failure before merge. This review did not independently reproduce the author's full-suite results; the description says its full run preceded the two wiring-test corrections, which were rerun individually. - The farm's documented probes cannot compare every generated `sa/gen` file or establish lift-stage speed. A farm build/gameplay pass is supplemental validation, not proof of byte-identical generated C or the claimed lift speed. Keep all generated files local; an independent lift comparison should report only the equality verdict/counts and timings. Verdict: ready for farm validation; merge approval deferred pending the outstanding evidence and CI disposition. I will supply the second sensitive-path farm authorization separately.
Member

/beanstalk 2a65abcb95c3efb849ec9d784f1fd74e98b60e84

/beanstalk 2a65abcb95c3efb849ec9d784f1fd74e98b60e84
madebyjordan force-pushed perf/lift-discover from 2a65abcb95
Some checks reported errors
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Successful in 4s
beanstalk/approval approved by futondestroyer, madebyjordan (double-reviewed)
beanstalk/linux 0/1 pass, 1 error
beanstalk/windows 1/1 fail
to 27909cb68b
Some checks are pending
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Successful in 14s
CI / Workflow lint (actionlint) (pull_request) Successful in 15s
Third-party parts / As their notes say (pull_request) Successful in 19s
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval needs a second approval (sensitive paths)
2026-11-16 22:18:27 +00:00
Compare
Author
Member

Rebased on main @ 983d607b now that #1128 is merged: this PR is its own commit plus a test commit. Answering @futondestroyer's review and the farm's run:

  • Real processes in a test: tests/test_discover.py now starts 2 spawned workers over a made-up image written to an image.bin and compares the whole discovery state with the kit's. It also checks that a worker whose walk differs from the lift's stops (and the discovery then walks in one process with the same result), that _init rejects another walk or an image.bin of the wrong size, and that two parts meeting in a searched range fall back end to end. Breaking the worker's base address, the walk check or the meeting check makes it fail (each tried by hand). The test lowers the new MIN_ENTRIES gate (4,000) for its ~4,000-entry image.
  • Measured against main now, not #1128: main @ 983d607b against this PR @ 27909cb6, 2 interleaved pairs on Windows. The whole lift took 185.1 / 131.8 s against 114.6 / 106.9 s, and discovery took 35-36 s against 19-21 s. Main's first run had memory for only 2 emit processes, so the like-for-like comparison is its second (131.8 s), which puts the gain at 13 to 19%. All 102 sa/gen files are identical to main's, and both lifts printed discovery: 21 parts walked in 7 processes / 24 parts walked in 8 processes, so the processes were really used, with no fallback.
  • Full suite: this run includes the updated test_oot_fov / test_oot_shadow. It gave 582 passed and 27 failed, against 581 passed on main at the same time with the same 27 failures (all listed in the description).
  • Guide screenshots: scripts/check_guide_screenshots.py --review-sources passes on the rebased head here. This PR touches no guide or UI file; the push reruns the check.
  • Farm @ 2a65abcb:
    • Windows: no crash, and its lift was faster (98.9 s against main's 115.5 s). It failed on one field picture (0.15 against 0.10) and on the Deku Tree's p99 frame time (+30%), which came from a single 164 ms hitch; p50 was -0.5%. There was one run a side. The lifted C is byte for byte main's, so the game code is the same, and I read both as run-to-run variation.
    • Linux: an input timeout, so no result. Linux keeps the one-process discovery, so it only checks that nothing changed there.
    • Asking for a rerun below.
Rebased on main @ 983d607b now that #1128 is merged: this PR is its own commit plus a test commit. Answering @futondestroyer's review and the farm's run: - **Real processes in a test:** `tests/test_discover.py` now starts 2 spawned workers over a made-up image written to an `image.bin` and compares the whole discovery state with the kit's. It also checks that a worker whose walk differs from the lift's stops (and the discovery then walks in one process with the same result), that `_init` rejects another walk or an `image.bin` of the wrong size, and that two parts meeting in a searched range fall back end to end. Breaking the worker's base address, the walk check or the meeting check makes it fail (each tried by hand). The test lowers the new `MIN_ENTRIES` gate (4,000) for its ~4,000-entry image. - **Measured against main now, not #1128:** main @ 983d607b against this PR @ 27909cb6, 2 interleaved pairs on Windows. The whole lift took 185.1 / 131.8 s against 114.6 / 106.9 s, and discovery took 35-36 s against 19-21 s. Main's first run had memory for only 2 emit processes, so the like-for-like comparison is its second (131.8 s), which puts the gain at 13 to 19%. All 102 `sa/gen` files are identical to main's, and both lifts printed `discovery: 21 parts walked in 7 processes` / `24 parts walked in 8 processes`, so the processes were really used, with no fallback. - **Full suite:** this run includes the updated `test_oot_fov` / `test_oot_shadow`. It gave 582 passed and 27 failed, against 581 passed on main at the same time with the same 27 failures (all listed in the description). - **Guide screenshots:** `scripts/check_guide_screenshots.py --review-sources` passes on the rebased head here. This PR touches no guide or UI file; the push reruns the check. - **Farm @ 2a65abcb:** - Windows: no crash, and its lift was faster (98.9 s against main's 115.5 s). It failed on one field picture (0.15 against 0.10) and on the Deku Tree's p99 frame time (+30%), which came from a single 164 ms hitch; p50 was -0.5%. There was one run a side. The lifted C is byte for byte main's, so the game code is the same, and I read both as run-to-run variation. - Linux: an input timeout, so no result. Linux keeps the one-process discovery, so it only checks that nothing changed there. - Asking for a rerun below.
Author
Member

@Beanstalk windows and linux: run it again on 27909cb6. On Windows, the build log's lift should print 'discovery: N parts walked in M processes' (the new path) and the game should run as main's (the lifted C is main's byte for byte); Linux keeps the one-process discovery.

@Beanstalk windows and linux: run it again on 27909cb6. On Windows, the build log's lift should print 'discovery: N parts walked in M processes' (the new path) and the game should run as main's (the lifted C is main's byte for byte); Linux keeps the one-process discovery.
Member

@madebyjordan approval of 27909cb recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@madebyjordan approval of `27909cb` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
HumanBean deleted branch perf/lift-discover 2026-11-16 22:35:50 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1131
No description provided.