perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster #1131

Open
madebyjordan wants to merge 3 commits from madebyjordan/Longshot:perf/lift-discover into main
Member

Type: perf · Risk: medium · Tested on: Windows 11, RX 6700 XT · Needs: Linux, any GPU (a lift compared with main's)

Builds on #1128 (its two commits are below this one until it merges; this PR's own change is the last commit,
perf(lift): on Windows the discovery's walks run in new processes). Review #1128 first.

Why

After #1116 and #1128 the biggest fixed part of a Windows lift is the discovery: the kit walks all 294,000 of OoT's
entries on one core, ~31 s on a quiet Ryzen 5 5600 and ~60 s when other programs keep the PC busy, while the other cores
wait.

What changed

The discovery's walks run in new processes, in parts of consecutive addresses, and the result is the kit's own.

  • nx2/discover.py (new), called from nx2/lift.py in place of L.discover(...):
    • parts_of cuts the sorted entries into about 3 parts a process, each cut where the unwind range changes.
    • Each part is walked in a new process by a lifter of its own (walk_part, fresh funcs, jt, jt_missing,
      stats and predecessor cache), over the image the lift already wrote (image/image.bin) and the lifter's entries,
      hooks and unwind starts. A worker whose walk is not the lift's (identities) stops.
    • Then the kit's discover runs as it is, its first loop given those bodies (merge); the jump-table retry, PLT
      stubs and setjmp owners walk as before.
  • Why the parts give the kit's result: one walk sees another's only through the predecessors of an unwind range
    (preds, read by reaching_constants), which hold the jump tables found so far in that range. Each part reports the
    ranges it found tables in and the ranges whose predecessors it read; when two parts meet in a range that was read
    (conflicts), the parts are dropped and the kit walks everything in order, as it does when a worker stops or raises.
    OoT: 1,475 jump tables, 435 of them through the predecessors, no range shared.
  • Not used with LONGSHOT_LIFT_JOBS=1, where there is fork (Linux, macOS: unchanged), for another image layout or a
    Lifter subclass.
  • tests/test_discover.py (new); tests/test_oot_fov.py and tests/test_oot_shadow.py look for the new call in
    nx2/lift.py (discover.discover(L) where they looked for L.discover(: they check the patches are installed
    before the discovery.

Measurements

Verdict: faster, same output. The Windows lift's median fell from 186.9 s to 138.5 s (-26%) on a busy PC, the
image and discovery from 60 s to 34 s. The lifted C is the same as main's.

Baseline: #1128 @ 68f5e131, not main: this PR is built on it, and #1128 against main is measured on #1128.
Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB, Crucial P3 NVMe, RX 6700 XT
Scene: none (the lift alone)
Method: nx2/lift.py --standalone --write as make_game.py runs it (RX_RELEASE=1 A64_HOT_CHUNK=1, the launcher's
image/chunks.txt, the nightly-336 tools' Python 3.14 and llvm-mingw), the default worker count, through
scripts/job_queue.py --mode timing-cpu; 2 interleaved pairs, the median. Other programs used about 45% of the CPU
throughout, which slows a one-core discovery most.
How to reproduce: python -I nx2/lift.py --standalone --write --work NEW_FOLDER with OOT_EXEFS set, on each side, then
compare sha1sum NEW_FOLDER/sa/gen/*.

Before: #1128 @ 68f5e131 After: this PR @ 2a65abcb
whole lift (run 1 / run 2) 195.5 s / 178.3 s 150.5 s / 126.5 s
whole lift, median 186.9 s 138.5 s (-26%)
image and discovery (run 1 / run 2) 58 s / 62 s 37 s / 31 s
discovery's processes 1 6 / 8 (18 / 24 parts)
lifted C 102 files the same 102 files, byte for byte (and main @ 71b1edd8's)

On its own (a script running only the discovery, 6 processes): 20.5 s against ~31 s for the kit's, and the lift's
process held 2.72 GB after it (3.02 GB with #1128's walk alone).

#1128, run 2:
discovered 294522 functions (62s)
lift: coverage in 34.7s (97s)
lift: emit and write (5 processes) in 38.9s (136s)
this PR, run 2:
discovery: 24 parts walked in 8 processes
discovered 294522 functions (31s)
lift: coverage in 28.2s (59s)
lift: emit and write (5 processes) in 32.2s (91s)

Flags and settings

None (LONGSHOT_LIFT_JOBS=1 already turns the processes off).

Testing

Tested:

  • Every lift above, and two more, compared file by file (sha1 of all 102 files of sa/gen): identical to #1128's and
    to main @ 71b1edd8's. A script also split the discovery into 6 lifters of its own in one process and compared all of
    their state with the kit's: functions, tables, misses and counters all equal.
  • In the game: the lifted C is main's, byte for byte; 101 of its 102 files are those of the launcher's nightly-336 build
    on this PC, which ran about 22 minutes of play today (Windows 11, RX 6700 XT, AMD 26.10.1; the other file is
    a64_fpx.h, changed by main's #1002).
  • tests/test_discover.py fails if merge drops the parts' counters, parts_of cuts inside an unwind range, or
    conflicts misses a range read by one part where another found a table (each checked by hand).
  • python -I -B tests/run_all.py on Windows: 570 passed, 24 failed, 86 skipped, before test_oot_fov and
    test_oot_shadow were updated for the new call (both pass since, run alone); #1128 @ 68f5e131: 571 passed and the
    other 22 failed here too (test_audio_output_devices, test_auto_handheld, test_bug_report, test_bug_report_defaults,
    test_crash_dialog, test_crt_options, test_dlss_feature_size, test_failed_compile_keeps_objects,
    test_guest_imported, test_hardware_report, test_job_queue, test_linux_preflight, test_make_game_cache,
    test_missing_nup, test_pads_after_menu, test_pending_guest_import, test_portable_builder_home,
    test_portable_builder_startup, test_run_all_changed, test_tested_configurations, test_vk_occlusion_queries,
    test_walk_key), all failing on main too.

Not tested: Linux and macOS (unchanged: the processes are only started where there is no fork), and a game
whose parts would meet in a searched range (the fallback is the kit's discover; tests/test_discover.py runs it).

Must be tested:

  • Linux, any GPU (I'm on Windows): a lift of the same game on main and on this PR
    (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, or a make_game.py build of each), expecting the same
    sha1 for every file of the work folder's sa/gen (Test Farm: @Beanstalk linux, the lift on main and on this PR,
    comparing the files of sa/gen)

Risks

What could break: the discovery on Windows: a different result would show as lifted C that differs from main's (the
conflict rule exists so it can't; a range the rule misses would be a bug in it). A worker that can't start, stops or
raises gives the kit's discover, with a discovery: ... walked here line. The processes take memory beside the lift's
own while they run (each its image, 100 MB, and its part's bodies); they end before the emit's start.

Turning it off: LONGSHOT_LIFT_JOBS=1 (which also keeps the emit in one process), or revert the commit.

Compatibility: the lift key changes (translator files changed), so the next update translates the game once. The C
is the same as before, so the compile finds the same object keys; shader caches are not touched.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • A check under tests/ that reads changed source updated in this PR, not weakened.

🤖 Generated with Claude Code

**Type:** perf · **Risk:** medium · **Tested on:** Windows 11, RX 6700 XT · **Needs:** Linux, any GPU (a lift compared with main's) **Builds on #1128** (its two commits are below this one until it merges; this PR's own change is the last commit, `perf(lift): on Windows the discovery's walks run in new processes`). Review #1128 first. ## Why After #1116 and #1128 the biggest fixed part of a Windows lift is the discovery: the kit walks all 294,000 of OoT's entries on one core, ~31 s on a quiet Ryzen 5 5600 and ~60 s when other programs keep the PC busy, while the other cores wait. ## What changed The discovery's walks run in new processes, in parts of consecutive addresses, and the result is the kit's own. - `nx2/discover.py` (new), called from `nx2/lift.py` in place of `L.discover(...)`: - `parts_of` cuts the sorted entries into about 3 parts a process, each cut where the unwind range changes. - Each part is walked in a new process by a lifter of its own (`walk_part`, fresh `funcs`, `jt`, `jt_missing`, `stats` and predecessor cache), over the image the lift already wrote (`image/image.bin`) and the lifter's entries, hooks and unwind starts. A worker whose walk is not the lift's (`identities`) stops. - Then the kit's `discover` runs as it is, its first loop given those bodies (`merge`); the jump-table retry, PLT stubs and setjmp owners walk as before. - Why the parts give the kit's result: one walk sees another's only through the predecessors of an unwind range (`preds`, read by `reaching_constants`), which hold the jump tables found so far in that range. Each part reports the ranges it found tables in and the ranges whose predecessors it read; when two parts meet in a range that was read (`conflicts`), the parts are dropped and the kit walks everything in order, as it does when a worker stops or raises. OoT: 1,475 jump tables, 435 of them through the predecessors, no range shared. - Not used with `LONGSHOT_LIFT_JOBS=1`, where there is fork (Linux, macOS: unchanged), for another image layout or a Lifter subclass. - `tests/test_discover.py` (new); `tests/test_oot_fov.py` and `tests/test_oot_shadow.py` look for the new call in `nx2/lift.py` (`discover.discover(L`) where they looked for `L.discover(`: they check the patches are installed before the discovery. ## Measurements **Verdict:** faster, same output. The Windows lift's median fell from 186.9 s to 138.5 s (-26%) on a busy PC, the image and discovery from 60 s to 34 s. The lifted C is the same as main's. Baseline: #1128 @ 68f5e131, not main: this PR is built on it, and #1128 against main is measured on #1128. Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB, Crucial P3 NVMe, RX 6700 XT Scene: none (the lift alone) Method: `nx2/lift.py --standalone --write` as `make_game.py` runs it (`RX_RELEASE=1 A64_HOT_CHUNK=1`, the launcher's `image/chunks.txt`, the nightly-336 tools' Python 3.14 and llvm-mingw), the default worker count, through `scripts/job_queue.py --mode timing-cpu`; 2 interleaved pairs, the median. Other programs used about 45% of the CPU throughout, which slows a one-core discovery most. How to reproduce: `python -I nx2/lift.py --standalone --write --work NEW_FOLDER` with `OOT_EXEFS` set, on each side, then compare `sha1sum NEW_FOLDER/sa/gen/*`. | | Before: #1128 @ 68f5e131 | After: this PR @ 2a65abcb | |---|---|---| | whole lift (run 1 / run 2) | 195.5 s / 178.3 s | 150.5 s / 126.5 s | | whole lift, median | 186.9 s | 138.5 s (-26%) | | image and discovery (run 1 / run 2) | 58 s / 62 s | 37 s / 31 s | | discovery's processes | 1 | 6 / 8 (18 / 24 parts) | | lifted C | 102 files | the same 102 files, byte for byte (and main @ 71b1edd8's) | On its own (a script running only the discovery, 6 processes): 20.5 s against ~31 s for the kit's, and the lift's process held 2.72 GB after it (3.02 GB with #1128's walk alone). ``` #1128, run 2: discovered 294522 functions (62s) lift: coverage in 34.7s (97s) lift: emit and write (5 processes) in 38.9s (136s) this PR, run 2: discovery: 24 parts walked in 8 processes discovered 294522 functions (31s) lift: coverage in 28.2s (59s) lift: emit and write (5 processes) in 32.2s (91s) ``` ### Flags and settings None (`LONGSHOT_LIFT_JOBS=1` already turns the processes off). ## Testing **Tested:** - Every lift above, and two more, compared file by file (sha1 of all 102 files of `sa/gen`): identical to #1128's and to main @ 71b1edd8's. A script also split the discovery into 6 lifters of its own in one process and compared all of their state with the kit's: functions, tables, misses and counters all equal. - In the game: the lifted C is main's, byte for byte; 101 of its 102 files are those of the launcher's nightly-336 build on this PC, which ran about 22 minutes of play today (Windows 11, RX 6700 XT, AMD 26.10.1; the other file is `a64_fpx.h`, changed by main's #1002). - `tests/test_discover.py` fails if `merge` drops the parts' counters, `parts_of` cuts inside an unwind range, or `conflicts` misses a range read by one part where another found a table (each checked by hand). - `python -I -B tests/run_all.py` on Windows: 570 passed, 24 failed, 86 skipped, before `test_oot_fov` and `test_oot_shadow` were updated for the new call (both pass since, run alone); #1128 @ 68f5e131: 571 passed and the other 22 failed here too (`test_audio_output_devices`, `test_auto_handheld`, `test_bug_report`, `test_bug_report_defaults`, `test_crash_dialog`, `test_crt_options`, `test_dlss_feature_size`, `test_failed_compile_keeps_objects`, `test_guest_imported`, `test_hardware_report`, `test_job_queue`, `test_linux_preflight`, `test_make_game_cache`, `test_missing_nup`, `test_pads_after_menu`, `test_pending_guest_import`, `test_portable_builder_home`, `test_portable_builder_startup`, `test_run_all_changed`, `test_tested_configurations`, `test_vk_occlusion_queries`, `test_walk_key`), all failing on main too. **Not tested:** Linux and macOS (unchanged: the processes are only started where there is no fork), and a game whose parts would meet in a searched range (the fallback is the kit's discover; `tests/test_discover.py` runs it). **Must be tested:** - Linux, any GPU (I'm on Windows): a lift of the same game on main and on this PR (`python -I nx2/lift.py --standalone --write --work NEW_FOLDER`, or a `make_game.py` build of each), expecting the same sha1 for every file of the work folder's `sa/gen` (Test Farm: @Beanstalk linux, the lift on main and on this PR, comparing the files of sa/gen) ## Risks **What could break:** the discovery on Windows: a different result would show as lifted C that differs from main's (the conflict rule exists so it can't; a range the rule misses would be a bug in it). A worker that can't start, stops or raises gives the kit's discover, with a `discovery: ... walked here` line. The processes take memory beside the lift's own while they run (each its image, 100 MB, and its part's bodies); they end before the emit's start. **Turning it off:** `LONGSHOT_LIFT_JOBS=1` (which also keeps the emit in one process), or revert the commit. **Compatibility:** the lift key changes (translator files changed), so the next update translates the game once. The C is the same as before, so the compile finds the same object keys; shader caches are not touched. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The kit's walk stores every instruction as its own (kind, word) tuple: 13.1 million for OoT, of which 12.7
million plain ops of 1.16 million distinct words. The lift's process held about 1.2 GB of duplicates
through the emit, and Windows starts one emit process per 0.75 GB free, so a 16 GB PC got 2-3 where it
now gets 5. nx2/walk.py is the kit's walk with the image's lookups inline and one shared tuple per plain
op's word (kept after the fact, the memory stays with Python's allocator, so it is done as the walk makes
them). tests/test_walk.py compares it with the kit's walk over a made-up image and pins the kit's tree.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
perf(lift): the emit asks dead_flags only of functions that write a flag, and reads A64_FASTFP_MIN once a function
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Third-party parts / As their notes say (pull_request) Successful in 15s
PR description / Description follows the template (pull_request_target) Successful in 6s
beanstalk/linux 1/1 pass
beanstalk/approval approved by futondestroyer, madebyjordan (double-reviewed)
68f5e131e9
The kit's dead_flags (which flag stores nothing reads) can only find a dead store where an instruction
writes a flag: 58% of OoT's functions have none, and its pass and fixpoint over them returned {} after
all that work (nx2/dead_flags.py answers them at once). ff_region read A64_FASTFP_MIN from the
environment for each of 3.8 million runs; emit_function now reads it once and passes it on. The lifted C
is the same.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
perf(lift): on Windows the discovery's walks run in new processes, cut at unwind ranges, with the kit's result
Some checks reported errors
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Successful in 4s
beanstalk/approval approved by futondestroyer, madebyjordan (double-reviewed)
beanstalk/linux 0/1 pass, 1 error
beanstalk/windows 1/1 fail
2a65abcb95
a64_lift.Lifter.discover walked OoT's 294,000 entries on one core (~31 s on a Ryzen 5 5600). nx2/discover.py
cuts the entries into parts of consecutive addresses at unwind-range starts and has new processes walk them,
each part with a lifter of its own, then runs the kit's discover with its first loop given those bodies (the
retry, PLT stubs and setjmp owners as before). One walk sees another's work only through the predecessors of
an unwind range (reaching_constants): when two parts meet in a range whose predecessors were read, the parts
are dropped and the kit walks everything, as it does when a worker stops. tests/test_discover.py compares the
parts with the kit's discover over a made-up image, and checks the conflict rule and the fallback.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Member

Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the lifted C. It is identical to #1128's and main @ 71b1edd8's (sha1 of all 102 files of sa/gen, four lifts), and 101 of those 102 files are the launcher's nightly-336 build on this PC from 09:33 today, which then ran about 22 minutes of play (80,813 frames presented) with no problem; the other, a64_fpx.h, differs by main's own #1002.

Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the lifted C. It is identical to #1128's and main @ 71b1edd8's (sha1 of all 102 files of sa/gen, four lifts), and 101 of those 102 files are the launcher's nightly-336 build on this PC from 09:33 today, which then ran about 22 minutes of play (80,813 frames presented) with no problem; the other, a64_fpx.h, differs by main's own #1002.
madebyjordan changed title from WIP: perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster to perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster 2026-11-16 13:27:12 +00:00
Author
Member

@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical (Linux keeps the one-process discovery; the walk and dead_flags changes from #1128 run there).

@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical (Linux keeps the one-process discovery; the walk and dead_flags changes from #1128 run there).
Member

@madebyjordan approval of 2a65abc recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@madebyjordan approval of `2a65abc` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
Member

Beanstalk test farm

Commit 2a65abc · tier double-reviewed · approved by futondestroyer, madebyjordan · dashboard

What will be tested · AI plan (gpt-6.1-terra)

On Linux and Windows, main and the PR will each be built and run through a short gameplay route from Kokiri Forest to Hyrule Field and the Deku Tree, taking about 13 minutes per tester excluding builds. The plan compares screenshots and checks for crashes, failed transitions, and substantial frame-time regressions. This is supplemental validation: the farm cannot compare generated C files or measure lift-stage speed, and gameplay on Linux does not exercise Windows worker spawning.

  • Where: linux (1 report), windows (1 report).
  • lifted-gameplay: from a new save, then take the picture kokiri to compare with main, teleport to Hyrule Field, move the stick, take the picture field to compare with main, measure field for 20 s (after 5 s warm-up), teleport to Inside the Deku Tree, take the picture deku to compare with main, measure deku for 20 s (after 5 s warm-up); 1 run on each side.
  • Measured: crashes, frame times, stutters and hitches.
  • The PR fails if: it crashes; or a picture differs from main by more than 10%; or frame time (p99) is more than 20% above main.
  • Time: about 14 min of game runs per tester (2 game starts), plus the two builds.

Done. Comment @Beanstalk to run it again.

This diff touches sensitive paths, so it needs two members of BigBeans: nx2/dead_flags.py, nx2/discover.py, nx2/fastfp.py, nx2/lift.py, nx2/walk.py

Target Reports Crash Frame p99 (ms) vs main VRAM peak (MB) vs main Result
linux · 1: linux+amd+radv+dgpu AMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7 1/1 1 none — — error
windows · 1: windows+amd+amd-proprietary+dgpu AMD Radeon RX 9070 XT, 2.0.416 1/1 1 none 24.7 vs 18.9 (+30%) — fail

Measurements · linux

Verdict: could not be judged on linux (1/1 report).

Hardware: 1: linux+amd+radv+dgpu AMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7 AMD Ryzen 7 5800X3D 8-Core Processor, 64 GB RAM
Scene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report
How to reproduce: the farm's plan above (@Beanstalk on this PR runs it again)

Nothing was measured: no metric came back.

Flags and settings

None: the plan varied no setting, switch or build option.

Markdown for the PR description
## Measurements

**Verdict:** could not be judged on linux (1/1 report).

Hardware: 1: linux+amd+radv+dgpu `AMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7` `AMD Ryzen 7 5800X3D 8-Core Processor, 64 GB RAM`  
Scene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them  
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report  
How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again)

Nothing was measured: no metric came back.

### Flags and settings

None: the plan varied no setting, switch or build option.

Measurements · windows

Verdict: fail on windows (1/1 report): lifted-gameplay screenshot.max_diff; lifted-gameplay deku.frametime_p99_ms.max_delta_pct (+30.4%) failed.

Hardware: 1: windows+amd+amd-proprietary+dgpu AMD Radeon RX 9070 XT, 2.0.416 AMD Ryzen 7 5800X3D 8-Core Processor, 32 GB RAM
Scene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report
How to reproduce: the farm's plan above (@Beanstalk on this PR runs it again)

Before: main @ 71b1edd After: this PR @ 2a65abc
lifted-gameplay · deku · frame time (p99) (ms) 18.9 24.7 (+30%)
lifted-gameplay · field · frame time (p99) (ms) 19.5 20.2 (+4%)

Flags and settings

None: the plan varied no setting, switch or build option.

Markdown for the PR description
## Measurements

**Verdict:** fail on windows (1/1 report): `lifted-gameplay screenshot.max_diff`; `lifted-gameplay deku.frametime_p99_ms.max_delta_pct (+30.4%)` failed.

Hardware: 1: windows+amd+amd-proprietary+dgpu `AMD Radeon RX 9070 XT, 2.0.416` `AMD Ryzen 7 5800X3D 8-Core Processor, 32 GB RAM`  
Scene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them  
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report  
How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again)

| | Before: main @ 71b1edd | After: this PR @ 2a65abc |
|---|---|---|
| lifted-gameplay · deku · frame time (p99) (ms) | 18.9 | 24.7 (+30%) |
| lifted-gameplay · field · frame time (p99) (ms) | 19.5 | 20.2 (+4%) |

### Flags and settings

None: the plan varied no setting, switch or build option.
<!-- beanstalk:sticky --> ### Beanstalk test farm Commit `2a65abc` · tier **double-reviewed** · approved by futondestroyer, madebyjordan · [dashboard](https://beanstalk.magicbean.dev/prs/1131) **What will be tested** · AI plan (`gpt-6.1-terra`) On Linux and Windows, main and the PR will each be built and run through a short gameplay route from Kokiri Forest to Hyrule Field and the Deku Tree, taking about 13 minutes per tester excluding builds. The plan compares screenshots and checks for crashes, failed transitions, and substantial frame-time regressions. This is supplemental validation: the farm cannot compare generated C files or measure lift-stage speed, and gameplay on Linux does not exercise Windows worker spawning. - **Where:** linux (1 report), windows (1 report). - **lifted-gameplay:** from a new save, then take the picture kokiri to compare with main, teleport to Hyrule Field, move the stick, take the picture field to compare with main, measure field for 20 s (after 5 s warm-up), teleport to Inside the Deku Tree, take the picture deku to compare with main, measure deku for 20 s (after 5 s warm-up); 1 run on each side. - **Measured:** crashes, frame times, stutters and hitches. - **The PR fails if:** it crashes; or a picture differs from main by more than 10%; or frame time (p99) is more than 20% above main. - **Time:** about 14 min of game runs per tester (2 game starts), plus the two builds. Done. Comment `@Beanstalk` to run it again. This diff touches sensitive paths, so it needs two members of `BigBeans`: `nx2/dead_flags.py`, `nx2/discover.py`, `nx2/fastfp.py`, `nx2/lift.py`, `nx2/walk.py` | Target | Reports | Crash | Frame p99 (ms) vs main | VRAM peak (MB) vs main | Result | | --- | --- | --- | --- | --- | --- | | linux · 1: linux+amd+radv+dgpu `AMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7` | 1/1 [1](https://beanstalk.magicbean.dev/reports/rp_064e0dc85dd3fe2d) | none | — | — | error | | windows · 1: windows+amd+amd-proprietary+dgpu `AMD Radeon RX 9070 XT, 2.0.416` | 1/1 [1](https://beanstalk.magicbean.dev/reports/rp_e1b0cef198c4c0e7) | none | 24.7 vs 18.9 (+30%) | — | fail | #### Measurements · linux **Verdict:** could not be judged on linux (1/1 report). Hardware: 1: linux+amd+radv+dgpu `AMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7` `AMD Ryzen 7 5800X3D 8-Core Processor, 64 GB RAM` Scene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) Nothing was measured: no metric came back. ### Flags and settings None: the plan varied no setting, switch or build option. <details><summary>Markdown for the PR description</summary> ```markdown ## Measurements **Verdict:** could not be judged on linux (1/1 report). Hardware: 1: linux+amd+radv+dgpu `AMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7` `AMD Ryzen 7 5800X3D 8-Core Processor, 64 GB RAM` Scene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) Nothing was measured: no metric came back. ### Flags and settings None: the plan varied no setting, switch or build option. ``` </details> #### Measurements · windows **Verdict:** fail on windows (1/1 report): `lifted-gameplay screenshot.max_diff`; `lifted-gameplay deku.frametime_p99_ms.max_delta_pct (+30.4%)` failed. Hardware: 1: windows+amd+amd-proprietary+dgpu `AMD Radeon RX 9070 XT, 2.0.416` `AMD Ryzen 7 5800X3D 8-Core Processor, 32 GB RAM` Scene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) | | Before: main @ 71b1edd | After: this PR @ 2a65abc | |---|---|---| | lifted-gameplay · deku · frame time (p99) (ms) | 18.9 | 24.7 (+30%) | | lifted-gameplay · field · frame time (p99) (ms) | 19.5 | 20.2 (+4%) | ### Flags and settings None: the plan varied no setting, switch or build option. <details><summary>Markdown for the PR description</summary> ```markdown ## Measurements **Verdict:** fail on windows (1/1 report): `lifted-gameplay screenshot.max_diff`; `lifted-gameplay deku.frametime_p99_ms.max_delta_pct (+30.4%)` failed. Hardware: 1: windows+amd+amd-proprietary+dgpu `AMD Radeon RX 9070 XT, 2.0.416` `AMD Ryzen 7 5800X3D 8-Core Processor, 32 GB RAM` Scene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) | | Before: main @ 71b1edd | After: this PR @ 2a65abc | |---|---|---| | lifted-gameplay · deku · frame time (p99) (ms) | 18.9 | 24.7 (+30%) | | lifted-gameplay · field · frame time (p99) (ms) | 19.5 | 20.2 (+4%) | ### Flags and settings None: the plan varied no setting, switch or build option. ``` </details>
futondestroyer left a comment

Code-and-evidence review of 2a65abcb95c3efb849ec9d784f1fd74e98b60e84, with this PR's own discovery change compared against its dependency #1128 at 68f5e131. No tests, builds, generators or gameplay run for this review.

No blocking code defect found. The workers receive the written image, entries/hooks and unwind starts; each part resets discovery state and the predecessor cache. The partition preserves address order within a range. The conflict check conservatively discards candidate results when another part could affect a searched predecessor range, before merging anything into the parent. On success the original discover still owns retry, splitting, PLT and setjmp processing, and consumed bodies fall back to the real walk on subsequent visits. The temporary walk override is removed in finally. Existing serial/fork and single-worker paths remain available.

Evidence limits and merge prerequisites:

  • This is stacked on #1128. Land that first, then verify the resulting diff/head; the reported performance baseline here is that dependency, not bare main. The claimed 26% improvement is from a busy Windows PC and should not be generalized to a quiet PC or Linux.
  • The new authored test compares serial/partitioned state through an in-process stand-in, and checks conflicts and raised-worker fallback. It does not exercise actual spawn initialization, image reconstruction or callable-identity rejection. A useful additional game-free check would spawn over a synthetic image and compare full discovery state; an integration test of the conflict-triggered fallback would also strengthen coverage. These are coverage suggestions, not a demonstrated production defect.
  • The author's repeated Windows lifts and file comparisons are meaningful evidence, but the stated Linux comparison remains outstanding. Linux normally takes the serial path, so a Linux pass does not validate the new Windows multiprocessing path. Include another Windows build when farm policy/workers support it, and check whether discovery really used processes or fell back.
  • The current forge statuses show Guide screenshots / Guide images and local UI source review failing; Python source-check jobs are skipped. Clear or explain that failure before merge. This review did not independently reproduce the author's full-suite results; the description says its full run preceded the two wiring-test corrections, which were rerun individually.
  • The farm's documented probes cannot compare every generated sa/gen file or establish lift-stage speed. A farm build/gameplay pass is supplemental validation, not proof of byte-identical generated C or the claimed lift speed. Keep all generated files local; an independent lift comparison should report only the equality verdict/counts and timings.

Verdict: ready for farm validation; merge approval deferred pending the outstanding evidence and CI disposition. I will supply the second sensitive-path farm authorization separately.

Code-and-evidence review of `2a65abcb95c3efb849ec9d784f1fd74e98b60e84`, with this PR's own discovery change compared against its dependency https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1128 at `68f5e131`. No tests, builds, generators or gameplay run for this review. No blocking code defect found. The workers receive the written image, entries/hooks and unwind starts; each part resets discovery state and the predecessor cache. The partition preserves address order within a range. The conflict check conservatively discards candidate results when another part could affect a searched predecessor range, before merging anything into the parent. On success the original discover still owns retry, splitting, PLT and setjmp processing, and consumed bodies fall back to the real walk on subsequent visits. The temporary walk override is removed in finally. Existing serial/fork and single-worker paths remain available. Evidence limits and merge prerequisites: - This is stacked on https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1128. Land that first, then verify the resulting diff/head; the reported performance baseline here is that dependency, not bare main. The claimed 26% improvement is from a busy Windows PC and should not be generalized to a quiet PC or Linux. - The new authored test compares serial/partitioned state through an in-process stand-in, and checks conflicts and raised-worker fallback. It does not exercise actual spawn initialization, image reconstruction or callable-identity rejection. A useful additional game-free check would spawn over a synthetic image and compare full discovery state; an integration test of the conflict-triggered fallback would also strengthen coverage. These are coverage suggestions, not a demonstrated production defect. - The author's repeated Windows lifts and file comparisons are meaningful evidence, but the stated Linux comparison remains outstanding. Linux normally takes the serial path, so a Linux pass does not validate the new Windows multiprocessing path. Include another Windows build when farm policy/workers support it, and check whether discovery really used processes or fell back. - The current forge statuses show `Guide screenshots / Guide images and local UI source review` failing; Python source-check jobs are skipped. Clear or explain that failure before merge. This review did not independently reproduce the author's full-suite results; the description says its full run preceded the two wiring-test corrections, which were rerun individually. - The farm's documented probes cannot compare every generated `sa/gen` file or establish lift-stage speed. A farm build/gameplay pass is supplemental validation, not proof of byte-identical generated C or the claimed lift speed. Keep all generated files local; an independent lift comparison should report only the equality verdict/counts and timings. Verdict: ready for farm validation; merge approval deferred pending the outstanding evidence and CI disposition. I will supply the second sensitive-path farm authorization separately.
Member

/beanstalk 2a65abcb95

/beanstalk 2a65abcb95c3efb849ec9d784f1fd74e98b60e84
Some checks reported errors
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Successful in 4s
beanstalk/approval approved by futondestroyer, madebyjordan (double-reviewed)
beanstalk/linux 0/1 pass, 1 error
beanstalk/windows 1/1 fail
This pull request has changes conflicting with the target branch.
  • nx2/lift.py
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/lift-discover:madebyjordan-perf/lift-discover'
git switch 'madebyjordan-perf/lift-discover'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1131
No description provided.