perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster #1131
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1131
Loading…
Reference in a new issue
No description provided.
Delete branch "madebyjordan/Longshot:perf/lift-discover"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: medium · Tested on: Windows 11, RX 6700 XT · Needs: Linux, any GPU (a lift compared with main's)
Builds on #1128 (its two commits are below this one until it merges; this PR's own change is the last commit,
perf(lift): on Windows the discovery's walks run in new processes). Review #1128 first.Why
After #1116 and #1128 the biggest fixed part of a Windows lift is the discovery: the kit walks all 294,000 of OoT's
entries on one core, ~31 s on a quiet Ryzen 5 5600 and ~60 s when other programs keep the PC busy, while the other cores
wait.
What changed
The discovery's walks run in new processes, in parts of consecutive addresses, and the result is the kit's own.
nx2/discover.py(new), called fromnx2/lift.pyin place ofL.discover(...):parts_ofcuts the sorted entries into about 3 parts a process, each cut where the unwind range changes.walk_part, freshfuncs,jt,jt_missing,statsand predecessor cache), over the image the lift already wrote (image/image.bin) and the lifter's entries,hooks and unwind starts. A worker whose walk is not the lift's (
identities) stops.discoverruns as it is, its first loop given those bodies (merge); the jump-table retry, PLTstubs and setjmp owners walk as before.
(
preds, read byreaching_constants), which hold the jump tables found so far in that range. Each part reports theranges it found tables in and the ranges whose predecessors it read; when two parts meet in a range that was read
(
conflicts), the parts are dropped and the kit walks everything in order, as it does when a worker stops or raises.OoT: 1,475 jump tables, 435 of them through the predecessors, no range shared.
LONGSHOT_LIFT_JOBS=1, where there is fork (Linux, macOS: unchanged), for another image layout or aLifter subclass.
tests/test_discover.py(new);tests/test_oot_fov.pyandtests/test_oot_shadow.pylook for the new call innx2/lift.py(discover.discover(L) where they looked forL.discover(: they check the patches are installedbefore the discovery.
Measurements
Verdict: faster, same output. The Windows lift's median fell from 186.9 s to 138.5 s (-26%) on a busy PC, the
image and discovery from 60 s to 34 s. The lifted C is the same as main's.
Baseline: #1128 @
68f5e131, not main: this PR is built on it, and #1128 against main is measured on #1128.Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB, Crucial P3 NVMe, RX 6700 XT
Scene: none (the lift alone)
Method:
nx2/lift.py --standalone --writeasmake_game.pyruns it (RX_RELEASE=1 A64_HOT_CHUNK=1, the launcher'simage/chunks.txt, the nightly-336 tools' Python 3.14 and llvm-mingw), the default worker count, throughscripts/job_queue.py --mode timing-cpu; 2 interleaved pairs, the median. Other programs used about 45% of the CPUthroughout, which slows a one-core discovery most.
How to reproduce:
python -I nx2/lift.py --standalone --write --work NEW_FOLDERwithOOT_EXEFSset, on each side, thencompare
sha1sum NEW_FOLDER/sa/gen/*.68f5e1312a65abcbOn its own (a script running only the discovery, 6 processes): 20.5 s against ~31 s for the kit's, and the lift's
process held 2.72 GB after it (3.02 GB with #1128's walk alone).
Flags and settings
None (
LONGSHOT_LIFT_JOBS=1already turns the processes off).Testing
Tested:
sa/gen): identical to #1128's andto main @ 71b1edd8's. A script also split the discovery into 6 lifters of its own in one process and compared all of
their state with the kit's: functions, tables, misses and counters all equal.
on this PC, which ran about 22 minutes of play today (Windows 11, RX 6700 XT, AMD 26.10.1; the other file is
a64_fpx.h, changed by main's #1002).tests/test_discover.pyfails ifmergedrops the parts' counters,parts_ofcuts inside an unwind range, orconflictsmisses a range read by one part where another found a table (each checked by hand).python -I -B tests/run_all.pyon Windows: 570 passed, 24 failed, 86 skipped, beforetest_oot_fovandtest_oot_shadowwere updated for the new call (both pass since, run alone); #1128 @68f5e131: 571 passed and theother 22 failed here too (
test_audio_output_devices,test_auto_handheld,test_bug_report,test_bug_report_defaults,test_crash_dialog,test_crt_options,test_dlss_feature_size,test_failed_compile_keeps_objects,test_guest_imported,test_hardware_report,test_job_queue,test_linux_preflight,test_make_game_cache,test_missing_nup,test_pads_after_menu,test_pending_guest_import,test_portable_builder_home,test_portable_builder_startup,test_run_all_changed,test_tested_configurations,test_vk_occlusion_queries,test_walk_key), all failing on main too.Not tested: Linux and macOS (unchanged: the processes are only started where there is no fork), and a game
whose parts would meet in a searched range (the fallback is the kit's discover;
tests/test_discover.pyruns it).Must be tested:
(
python -I nx2/lift.py --standalone --write --work NEW_FOLDER, or amake_game.pybuild of each), expecting the samesha1 for every file of the work folder's
sa/gen(Test Farm: @Beanstalk linux, the lift on main and on this PR,comparing the files of sa/gen)
Risks
What could break: the discovery on Windows: a different result would show as lifted C that differs from main's (the
conflict rule exists so it can't; a range the rule misses would be a bug in it). A worker that can't start, stops or
raises gives the kit's discover, with a
discovery: ... walked hereline. The processes take memory beside the lift'sown while they run (each its image, 100 MB, and its part's bodies); they end before the emit's start.
Turning it off:
LONGSHOT_LIFT_JOBS=1(which also keeps the emit in one process), or revert the commit.Compatibility: the lift key changes (translator files changed), so the next update translates the game once. The C
is the same as before, so the compile finds the same object keys; shader caches are not touched.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
The kit's dead_flags (which flag stores nothing reads) can only find a dead store where an instruction writes a flag: 58% of OoT's functions have none, and its pass and fixpoint over them returned {} after all that work (nx2/dead_flags.py answers them at once). ff_region read A64_FASTFP_MIN from the environment for each of 3.8 million runs; emit_function now reads it once and passes it on. The lifted C is the same. Co-Authored-By: Claude Opus 5.5 <[email protected]>Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the lifted C. It is identical to #1128's and main @ 71b1edd8's (sha1 of all 102 files of sa/gen, four lifts), and 101 of those 102 files are the launcher's nightly-336 build on this PC from 09:33 today, which then ran about 22 minutes of play (80,813 frames presented) with no problem; the other, a64_fpx.h, differs by main's own #1002.
WIP: perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter fasterto perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical (Linux keeps the one-process discovery; the walk and dead_flags changes from #1128 run there).
@madebyjordan approval of
2a65abcrecorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.Beanstalk test farm
Commit
2a65abc· tier double-reviewed · approved by futondestroyer, madebyjordan · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)On Linux and Windows, main and the PR will each be built and run through a short gameplay route from Kokiri Forest to Hyrule Field and the Deku Tree, taking about 13 minutes per tester excluding builds. The plan compares screenshots and checks for crashes, failed transitions, and substantial frame-time regressions. This is supplemental validation: the farm cannot compare generated C files or measure lift-stage speed, and gameplay on Linux does not exercise Windows worker spawning.
Done. Comment
@Beanstalkto run it again.This diff touches sensitive paths, so it needs two members of
BigBeans:nx2/dead_flags.py,nx2/discover.py,nx2/fastfp.py,nx2/lift.py,nx2/walk.pyAMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7AMD Radeon RX 9070 XT, 2.0.416Measurements · linux
Verdict: could not be judged on linux (1/1 report).
Hardware: 1: linux+amd+radv+dgpu
AMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7AMD Ryzen 7 5800X3D 8-Core Processor, 64 GB RAMScene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)Nothing was measured: no metric came back.
Flags and settings
None: the plan varied no setting, switch or build option.
Markdown for the PR description
Measurements · windows
Verdict: fail on windows (1/1 report):
lifted-gameplay screenshot.max_diff;lifted-gameplay deku.frametime_p99_ms.max_delta_pct (+30.4%)failed.Hardware: 1: windows+amd+amd-proprietary+dgpu
AMD Radeon RX 9070 XT, 2.0.416AMD Ryzen 7 5800X3D 8-Core Processor, 32 GB RAMScene: lifted-gameplay, from a new save; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); lifted-gameplay: 1 game start a side, field 20 s after 5 s warm-up, deku 20 s after 5 s warm-up; each number the median of the runs, then of 1 report
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)71b1edd2a65abcFlags and settings
None: the plan varied no setting, switch or build option.
Markdown for the PR description
Code-and-evidence review of
2a65abcb95c3efb849ec9d784f1fd74e98b60e84, with this PR's own discovery change compared against its dependency #1128 at68f5e131. No tests, builds, generators or gameplay run for this review.No blocking code defect found. The workers receive the written image, entries/hooks and unwind starts; each part resets discovery state and the predecessor cache. The partition preserves address order within a range. The conflict check conservatively discards candidate results when another part could affect a searched predecessor range, before merging anything into the parent. On success the original discover still owns retry, splitting, PLT and setjmp processing, and consumed bodies fall back to the real walk on subsequent visits. The temporary walk override is removed in finally. Existing serial/fork and single-worker paths remain available.
Evidence limits and merge prerequisites:
Guide screenshots / Guide images and local UI source reviewfailing; Python source-check jobs are skipped. Clear or explain that failure before merge. This review did not independently reproduce the author's full-suite results; the description says its full run preceded the two wiring-test corrections, which were rerun individually.sa/genfile or establish lift-stage speed. A farm build/gameplay pass is supplemental validation, not proof of byte-identical generated C or the claimed lift speed. Keep all generated files local; an independent lift comparison should report only the equality verdict/counts and timings.Verdict: ready for farm validation; merge approval deferred pending the outstanding evidence and CI disposition. I will supply the second sensitive-path farm authorization separately.
/beanstalk
2a65abcb95View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.