perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster #1131
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1131
Loading…
Reference in a new issue
No description provided.
Delete branch "madebyjordan/Longshot:perf/lift-discover"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: medium · Tested on: Windows 11, RX 6700 XT · Needs: Linux, any GPU (a lift compared with main's)
Why
After #1116 and #1128 the biggest fixed part of a Windows lift is the discovery: the kit walks all 294,000 of OoT's
entries on one core, ~31 s on a quiet Ryzen 5 5600 and ~60 s when other programs keep the PC busy, while the other cores
wait.
What changed
The discovery's walks run in new processes, in parts of consecutive addresses, and the result is the kit's own.
nx2/discover.py(new), called fromnx2/lift.pyin place ofL.discover(...):parts_ofcuts the sorted entries into about 3 parts a process, each cut where the unwind range changes.walk_part, freshfuncs,jt,jt_missing,statsand predecessor cache), over the image the lift already wrote (image/image.bin) and the lifter's entries,hooks and unwind starts. A worker whose walk is not the lift's (
identities) stops.discoverruns as it is, its first loop given those bodies (merge); the jump-table retry, PLTstubs and setjmp owners walk as before.
(
preds, read byreaching_constants), which hold the jump tables found so far in that range. Each part reports theranges it found tables in and the ranges whose predecessors it read; when two parts meet in a range that was read
(
conflicts), the parts are dropped and the kit walks everything in order, as it does when a worker stops or raises.OoT: 1,475 jump tables, 435 of them through the predecessors, no range shared.
LONGSHOT_LIFT_JOBS=1, where there is fork (Linux, macOS: unchanged), for another image layout, aLifter subclass or fewer than
MIN_ENTRIES(4,000) entries.tests/test_discover.py(new: the parts against the kit in one process and through 2 spawned processes, the workers'checks, the fallbacks);
tests/test_oot_fov.pyandtests/test_oot_shadow.pylook for the new call innx2/lift.py(discover.discover(L) where they looked forL.discover(: they check the patches are installedbefore the discovery.
Measurements
Verdict: faster, same output. Against main @
983d607b(with #1128): the Windows lift's median fell from 158.5 s to110.8 s (-30%), the image and discovery from 36 s to 20 s; like for like (both sides' emit in 4-5 processes) 131.8 s
against 106.9-114.6 s (-13 to -19%). The lifted C is main's, byte for byte.
Baseline: main @
983d607b(rebased; #1128 merged). The first measurement, against #1128 before the rebase, is below.Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB, Crucial P3 NVMe, RX 6700 XT
Scene: none (the lift alone)
Method:
nx2/lift.py --standalone --writeasmake_game.pyruns it (RX_RELEASE=1 A64_HOT_CHUNK=1, the launcher'simage/chunks.txt, the nightly-351 tools' Python 3.14 and llvm-mingw), the default worker count, throughscripts/job_queue.py --mode timing-cpu; 2 interleaved pairs, the median.How to reproduce:
python -I nx2/lift.py --standalone --write --work NEW_FOLDERwithOOT_EXEFSset, on each side, thencompare
sha1sum NEW_FOLDER/sa/gen/*.983d607b27909cb6Main's first run had memory for only 2 emit processes (89.8 s of emit); its second, with 4, is the like-for-like one.
The first measurement, against #1128 before the rebase (the same method with the nightly-336 tools; other programs used
about 45% of the CPU throughout, which slows a one-core discovery most):
68f5e131On its own (a script running only the discovery, 6 processes): 20.5 s against ~31 s for the kit's, and the lift's
process held 2.72 GB after it (3.02 GB with #1128's walk alone).
Flags and settings
None (
LONGSHOT_LIFT_JOBS=1already turns the processes off).Testing
Tested:
sa/gen): after the rebase identicalto main @ 983d607b's, before it to #1128's and main @ 71b1edd8's. Every Windows lift of this PR printed
discovery: N parts walked in M processes(no fallback). A script also split the discovery into 6 lifters of its own in one process and compared all oftheir state with the kit's: functions, tables, misses and counters all equal.
on this PC, which ran about 22 minutes of play today (Windows 11, RX 6700 XT, AMD 26.10.1; the other file is
a64_fpx.h, changed by main's #1002).tests/test_discover.pyruns the discovery through 2 real spawned processes over a made-up image written to animage.bin, and in one process: both give the kit's state. It fails ifmergedrops the parts' counters,parts_ofcuts inside an unwind range,
conflictsmisses a range read by one part where another found a table, a worker skipsthe image's base address, the workers' walk check is dropped, or parts that meet are merged anyway (each checked by
hand).
python -I -B tests/run_all.pyon Windows @27909cb6: 582 passed, 27 failed, 88 skipped; main @983d607bon the samePC at the same time: 581 passed and the same 27 failed (
test_auto_handheld,test_bug_report,test_bug_report_defaults,test_crash_dialog,test_crt_options,test_damage_debug_menu,test_dlss_feature_size,test_failed_compile_keeps_objects,test_framegen_preset,test_guest_imported,test_hardware_report,test_job_queue,test_linux_preflight,test_make_game_cache,test_menu_i18n,test_missing_nup,test_pads_after_menu,test_pending_guest_import,test_portable_builder_home,test_portable_builder_startup,test_rumble_menu,test_run_all_changed,test_tested_configurations,test_upscaler_round_trip,test_upscaler_rows,test_vk_occlusion_queries,test_walk_key).took 98.9 s against main's 115.5 s. It failed on one field picture (0.15 against a 0.10 limit) and on the Deku Tree's
p99 frame time (+30%, from one 164 ms hitch; p50 -0.5%), with one run a side. The lifted C is main's byte for byte, so
the game code is the same, and I read these as run-to-run variation; a rerun is asked for. Linux: the route's inputs
timed out (an error, not a result).
Not tested: Linux and macOS (unchanged: the processes are only started where there is no fork; a Linux lift does not
exercise them), and a game
whose parts would meet in a searched range (the fallback is the kit's discover;
tests/test_discover.pyruns it).Must be tested:
(
python -I nx2/lift.py --standalone --write --work NEW_FOLDER, or amake_game.pybuild of each), expecting the samesha1 for every file of the work folder's
sa/gen(Test Farm: @Beanstalk linux, the lift on main and on this PR,comparing the files of sa/gen)
Risks
What could break: the discovery on Windows: a different result would show as lifted C that differs from main's (the
conflict rule exists so it can't; a range the rule misses would be a bug in it). A worker that can't start, stops or
raises gives the kit's discover, with a
discovery: ... walked hereline. The processes take memory beside the lift'sown while they run (each its image, 100 MB, and its part's bodies); they end before the emit's start.
Turning it off:
LONGSHOT_LIFT_JOBS=1(which also keeps the emit in one process), or revert the commit.Compatibility: the lift key changes (translator files changed), so the next update translates the game once. The C
is the same as before, so the compile finds the same object keys; shader caches are not touched.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
The kit's dead_flags (which flag stores nothing reads) can only find a dead store where an instruction writes a flag: 58% of OoT's functions have none, and its pass and fixpoint over them returned {} after all that work (nx2/dead_flags.py answers them at once). ff_region read A64_FASTFP_MIN from the environment for each of 3.8 million runs; emit_function now reads it once and passes it on. The lifted C is the same. Co-Authored-By: Claude Opus 5.5 <[email protected]>Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the lifted C. It is identical to #1128's and main @ 71b1edd8's (sha1 of all 102 files of sa/gen, four lifts), and 101 of those 102 files are the launcher's nightly-336 build on this PC from 09:33 today, which then ran about 22 minutes of play (80,813 frames presented) with no problem; the other, a64_fpx.h, differs by main's own #1002.
WIP: perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter fasterto perf(lift): on Windows the discovery's walks run in new processes, with the kit's result: a busy PC's lift about a quarter faster@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical (Linux keeps the one-process discovery; the walk and dead_flags changes from #1128 run there).
@madebyjordan approval of
2a65abcrecorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.Beanstalk test farm
Commit
27909cb· tier untrusted · approved by madebyjordan · dashboardThis diff touches sensitive paths, so it needs two members of
BigBeans:nx2/discover.py,nx2/lift.pyA member of
BigBeanscan review this commit and comment@Beanstalkto test it.Code-and-evidence review of
2a65abcb95c3efb849ec9d784f1fd74e98b60e84, with this PR's own discovery change compared against its dependency #1128 at68f5e131. No tests, builds, generators or gameplay run for this review.No blocking code defect found. The workers receive the written image, entries/hooks and unwind starts; each part resets discovery state and the predecessor cache. The partition preserves address order within a range. The conflict check conservatively discards candidate results when another part could affect a searched predecessor range, before merging anything into the parent. On success the original discover still owns retry, splitting, PLT and setjmp processing, and consumed bodies fall back to the real walk on subsequent visits. The temporary walk override is removed in finally. Existing serial/fork and single-worker paths remain available.
Evidence limits and merge prerequisites:
Guide screenshots / Guide images and local UI source reviewfailing; Python source-check jobs are skipped. Clear or explain that failure before merge. This review did not independently reproduce the author's full-suite results; the description says its full run preceded the two wiring-test corrections, which were rerun individually.sa/genfile or establish lift-stage speed. A farm build/gameplay pass is supplemental validation, not proof of byte-identical generated C or the claimed lift speed. Keep all generated files local; an independent lift comparison should report only the equality verdict/counts and timings.Verdict: ready for farm validation; merge approval deferred pending the outstanding evidence and CI disposition. I will supply the second sensitive-path farm authorization separately.
/beanstalk 2a65abcb95c3efb849ec9d784f1fd74e98b60e84
2a65abcb9527909cb68bRebased on main @
983d607bnow that #1128 is merged: this PR is its own commit plus a test commit. Answering @futondestroyer's review and the farm's run:tests/test_discover.pynow starts 2 spawned workers over a made-up image written to animage.binand compares the whole discovery state with the kit's. It also checks that a worker whose walk differs from the lift's stops (and the discovery then walks in one process with the same result), that_initrejects another walk or animage.binof the wrong size, and that two parts meeting in a searched range fall back end to end. Breaking the worker's base address, the walk check or the meeting check makes it fail (each tried by hand). The test lowers the newMIN_ENTRIESgate (4,000) for its ~4,000-entry image.983d607bagainst this PR @27909cb6, 2 interleaved pairs on Windows. The whole lift took 185.1 / 131.8 s against 114.6 / 106.9 s, and discovery took 35-36 s against 19-21 s. Main's first run had memory for only 2 emit processes, so the like-for-like comparison is its second (131.8 s), which puts the gain at 13 to 19%. All 102sa/genfiles are identical to main's, and both lifts printeddiscovery: 21 parts walked in 7 processes/24 parts walked in 8 processes, so the processes were really used, with no fallback.test_oot_fov/test_oot_shadow. It gave 582 passed and 27 failed, against 581 passed on main at the same time with the same 27 failures (all listed in the description).scripts/check_guide_screenshots.py --review-sourcespasses on the rebased head here. This PR touches no guide or UI file; the push reruns the check.@Beanstalk windows and linux: run it again on
27909cb6. On Windows, the build log's lift should print 'discovery: N parts walked in M processes' (the new path) and the game should run as main's (the lifted C is main's byte for byte); Linux keeps the one-process discovery.@madebyjordan approval of
27909cbrecorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.