perf(lift): 1.2 GB less in a Windows lift's own process, so more emit workers start: about a fifth faster, the same C #1128
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1128
Loading…
Reference in a new issue
No description provided.
Delete branch "madebyjordan/Longshot:perf/lift-memory"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: low · Tested on: Windows 11, RX 6700 XT · Needs: Linux, any GPU (a lift compared with main's)
Why
After #1116, a Windows lift's emit is limited by how many worker processes it may start: one per 0.75 GB free, and the
lift's own process held 4.2 GB after discovery, so a 16 GB PC got 2-4 of them. About 1.2 GB of that was duplicates:
each of OoT's 12.7 million plain instructions stored as a tuple of its own, though there are only 1.16 million distinct
ones. The workers also ran the kit's flag liveness pass on functions that write no flag, where it always finds nothing.
What changed
The lift's process needs 1.2 GB less, so the same PC starts more emit workers; the workers do a little less work each. The
lifted C is the same, byte for byte.
nx2/walk.py, new; installed bynx2/lift.py): the kit'sLifter.walk, visiting the same instructions inthe same order, with every plain op of one word stored as one shared
(("op",), word)tuple (made as the walk goes:sharing them afterwards frees nothing back to Windows) and the image's lookups (#1116's
EdenImageview and ranges)read inline.
tests/test_walk.pycompares it with the kit's walk over a made-up image of random words, through boththe fast and the plain image lookups, and pins the kit's tree-sha256 so an a64-kit refresh fails until it is compared
again.
nx2/dead_flags.py, new;nx2/fastfp.py):Lifter.dead_flagsanswers{}at once for a function with noinstruction that writes a flag (58% of OoT's functions) and calls the kit's method otherwise;
tests/test_dead_flags.pycompares the two over 3,000 made-up functions.
ff_regiontakesA64_FASTFP_MINfromemit_function, which reads itonce a function instead of once for each of 3.8 million runs.
Measurements
Verdict: faster, same output. At the default worker count, as the launcher runs it, the Windows lift's median fell
from 154.8 s to 125.7 s (-19%): the memory gate allowed 5-6 emit workers where main got 3-4. The lift's process holds
4.19 GB after discovery on main and 3.02 GB here.
Hardware: Windows 11, Ryzen 5 5600 (6 cores, 12 threads), 16 GB, Crucial P3 NVMe, RX 6700 XT
Scene: none (the lift alone)
Method:
nx2/lift.py --standalone --writeasmake_game.pyruns it (RX_RELEASE=1 A64_HOT_CHUNK=1, the launcher'simage/chunks.txt, the nightly-336 tools' Python 3.14 and llvm-mingw), the defaultLONGSHOT_LIFT_JOBS(each side's ownmemory gate decides), through
scripts/job_queue.py --mode timing-cpu; 2 interleaved pairs, the median. Memory: theprocess's working set right after
discover(), measured twice each side in a separate script.How to reproduce:
python -I nx2/lift.py --standalone --write --work NEW_FOLDERwithOOT_EXEFSset, on each side, thencompare
sha1sum NEW_FOLDER/sa/gen/*.71b1edd868f5e131At the same worker count the workers' own time fell about 6% (profiled in each worker: 105 s busy each before the
dead_flags and A64_FASTFP_MIN change, 99 s after, 4 workers), so most of the gain is the extra workers.
Flags and settings
None.
Testing
Tested:
sa/gen): identical.Against the launcher's nightly-336 build on this PC (09:33 today), which then ran about 22 minutes of play (80,813
frames presented, Windows 11, RX 6700 XT, AMD 26.10.1), 101 of the 102 files are the same; the other is
a64_fpx.h,changed on main by #1002 (
always_inlinehelpers), not here.tests/test_walk.pyfails if the walk doesn't stop at another entry, leaves a conditional branch's target unvisited,or shares the entry of a word that classifies as a branch;
tests/test_dead_flags.pyfails if the shortcut looks at thefirst op only or at the flags an op reads (each checked by hand).
python -I -B tests/run_all.pyon Windows: 570 passed, 23 failed, 86 skipped; main @71b1edd8: 569 passed, 22 failed.The failures are the same 22 (
test_vk_occlusion_queries,test_crash_dialog,test_linux_preflight,test_job_queue,test_make_game_cache,test_bug_report_defaults,test_run_all_changed,test_tested_configurations,test_portable_builder_home,test_pending_guest_import,test_missing_nup,test_guest_imported,test_walk_key,test_pads_after_menu,test_auto_handheld,test_failed_compile_keeps_objects,test_portable_builder_startup,test_bug_report,test_crt_options,test_audio_output_devices,test_hardware_report,test_dlss_feature_size) plustest_longshot_ctlhere, which isflaky on main too (
test_the_session_follows_a_relaunch: run alone twice each, it failed once on main and passedtwice here). The two extra passes are the new checks.
Not tested: a lift on Linux or macOS (the walk and dead_flags run there too; the forked emit's worker count is
gated at 2 GB each, so the memory saved matters less there), and the lift inside a whole
make_game.pybuild.Must be tested:
(
python -I nx2/lift.py --standalone --write --work NEW_FOLDER, or amake_game.pybuild of each), expecting the samesha1 for every file of the work folder's
sa/gen(Test Farm: @Beanstalk linux, the lift on main and on this PR,comparing the files of sa/gen)
Risks
What could break: the lift's discovery or emit, where it would show as a failed lift or C that differs from main's.
The walk relies on a plain op's classification not depending on its pc (true of the kit's and
nx2/insn.py'sclassify;
tests/test_lift_lookups.pyfrom #1116 checks it). More emit workers on Windows means more memory in use duringthe emit; free memory stayed above 1.6 GB on this PC at 5 workers, where main at 3 went down to 0.7 GB.
Turning it off: revert the commits.
Compatibility: the lift key changes (translator files changed), so the next update translates the game once. The C
is the same as before, so the compile finds the same object keys; shader caches are not touched.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.🤖 Generated with Claude Code
The kit's dead_flags (which flag stores nothing reads) can only find a dead store where an instruction writes a flag: 58% of OoT's functions have none, and its pass and fixpoint over them returned {} after all that work (nx2/dead_flags.py answers them at once). ff_region read A64_FASTFP_MIN from the environment for each of 3.8 million runs; emit_function now reads it once and passes it on. The lifted C is the same. Co-Authored-By: Claude Opus 5.5 <[email protected]>Tested in game: Windows 11, RX 6700 XT, AMD 26.10.1, Ryzen 5 5600: this PR changes no byte of the lifted C. It is identical to main @ 71b1edd8's (sha1 of all 102 files of sa/gen, two lifts each side), and 101 of those 102 files are the same as the launcher's nightly-336 build on this PC from 09:33 today, which then ran about 22 minutes of play (80,813 frames presented) with no problem; the other file, a64_fpx.h, differs by main's own #1002, not by this PR.
WIP: perf(lift): 1.2 GB less in a Windows lift's own process, so more emit workers start: about a fifth faster, the same Cto perf(lift): 1.2 GB less in a Windows lift's own process, so more emit workers start: about a fifth faster, the same C@Beanstalk linux: translate the game on main and on this PR (python -I nx2/lift.py --standalone --write --work NEW_FOLDER, a new folder for each, or a make_game.py build of each), then compare the sha1 of every file in each work folder's sa/gen. Expected: every file identical, and the PR's lift no slower than main's (the lift: lines of each log).
@madebyjordan approval of
68f5e13recorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.Beanstalk test farm
Commit
68f5e13· tier double-reviewed · approved by futondestroyer, madebyjordan · dashboardWhat will be tested · AI plan (
gpt-6.1-terra)On Linux, this plan builds main and the PR, then runs a short gameplay smoke test in Kokiri Forest and Hyrule Field, taking comparison pictures and measuring frame time. It takes about 13 minutes per tester, excluding build time; crashes, picture differences or a frame-time regression fail the plan. Farm probes cannot compare generated C files or measure lift-stage timing, so this plan does not establish the requested lift-output equivalence or speed.
Done. Run 2: the farm's reports were cleared after its Windows and Linux fixes: started again. Comment
@Beanstalkto run it again.This diff touches sensitive paths, so it needs two members of
BigBeans:nx2/dead_flags.py,nx2/fastfp.py,nx2/lift.py,nx2/walk.pyAMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7Measurements · linux
Verdict: pass on linux (1/1 report): no rule failed and no PR run crashed.
Hardware: 1: linux+amd+radv+dgpu
AMD Radeon RX 7900 XTX (RADV NAVI31), 26.2.7AMD Ryzen 7 5800X3D 8-Core Processor, 64 GB RAMScene: linux-gameplay, from a new save; each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); linux-gameplay: 1 game start a side, field 20 s after 5 s warm-up; each number the median of the runs, then of 1 report
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)71b1edd68f5e13Flags and settings
None: the plan varied no setting, switch or build option.
Markdown for the PR description
Reviewed head
68f5e131against main71b1edd8.No blocking code finding. The replacement walk preserves the kit's traversal order, entry boundaries, labels and jump-table handling. It shares only immutable plain-op entries; PC-relative control flow is still classified per address. Its fast image lookup is reached only for aligned code addresses, and the kit provenance pin makes a later kit refresh require reconsideration. The dead-flags shortcut uses the same text lookup, UNHANDLED exclusion and flag-write mask as the kit; functions with a flag writer retain the original analysis. Reading A64_FASTFP_MIN once per function preserves the configured threshold.
Independent Windows game-free validation: 13 focused checks passed, 0 failed, 0 skipped (walk, 3,000-function dead-flags differential, fastfp regions/override/FMA, image lookups, workers, lift cache/store, cleanup, entry and chunk boundaries). Diff whitespace check passed; third_party is unchanged. I did not rerun the entire suite or perform a real-game lift; the PR documents its full-suite failures and baseline comparison.
Ready for Linux farm validation, not a claim that Linux output equivalence has already been established. Please compare builds of the same base/head on Linux, checking lift/build success and recording lift-stage timings. The requested byte-for-byte equality of every sa/gen file remains an explicit requirement: the documented farm probes do not expose a generated-file equality metric, so a normal build/game pass alone must not be presented as that proof. Generated C and other game-derived files must remain on the tester's PC; report only equality verdict/counts and timings.
Farm authorization will be a separate SHA-pinned command; merge approval is deferred pending the outstanding Linux evidence.
/beanstalk
68f5e131e9