perf(build): updates skip the lift, a faster lift, and hot pieces that compile sooner and stay put #2324
No reviewers
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!2324
Loading…
Reference in a new issue
No description provided.
Delete branch "perf/faster-build"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Rebuilding the game spends most of a warm build in the lift (translating the game to C), and most of a cold build in the eight hot pieces. This PR makes both shorter. The game's code comes out the same.
What changes
make_game.keep_translation/kept_before,build.pyKEPT).sa/, including the 1.5 GB of lifted C. So every update lifted again (1-3 minutes), even when nothing the lift reads had changed.code_*.cfiles and the build outputs. It keeps:gen/lifted_objects.txt). Its first line names the lift key it was written for and a format number; a new lift deletes the list of the one before..lift_keygets a third line,kept.build.py --kept-lift. build.py then takes the lifted files' objects from the object cache by the keys in that list. These are the same objects as before, so nothing is compiled again.KEPT_STALE; not 3, which isabort()'s on Windows). make_game then lifts again and compiles as usual.sa/go as far as it can, and the next update lifts again.Storage.Free(afterBuild)/core.free(after_build=True)) also deletedsa/. It now leaves a kept lift in place. "Free up space" still deletes it.chunks.split_big, using the same worker pool as the edits). Before, it read all 1.5 GB of chunks on one core.a64_rtgen,hos_image.c, the import stubs andnvn2_exports.c. They read the image, not the lifted C, so a forked process now makes them while the C is being edited. Without fork (Windows), or if that process fails, they are made after the edits, as before.make_game.shader_list_current).shader_census.py(about 4 s) and rewrote the 514 MBhookshot_shaders.bin.scripts/make_setup_parts.py), one thread each, as the Linuxmake_parts.pyhas done since #2079. The zips come out byte for byte the same.split_hot.py).<work>/image/hot_pieces.txt, next to the chunk boundaries:hot_pieces.txtis part of the lift key, aschunks.txtis: the pieces' C follows it.chunks.file_jobs,_each). Before, they used forked workers only, so without fork (Windows) they ran in one process. Their workers take a file's path and read the file themselves, so without fork they now start as new processes (spawn). The emit still runs in one process there.HOOKSHOT_LIFT_NO_FORK=1takes this path on Linux, for testing.Measured
Setup: Linux, Ryzen 9 7900X (24 threads), 30 GB RAM, other jobs running on the PC.
oot-status --lock --timing, which gives the run its own core complex (6 cores / 12 threads).Player update, same tools (
make_game.py --nup, noHOOKSHOT_KEEP_BUILD_FILES, objects cached, shader prepare skipped). Times are from start to@@done:Caveat: these four runs went through
make_game.pyunderoot-status --timing. A queue bug that night movedmake_game.py/build.pyprocesses onto the busy build CPUs during timed runs, so the absolute seconds of both rows are inflated by load. The gap does not come from load, though: the "before" runs spent 83 / 175 s in a lift that the "after" runs skip entirely. The lift-step, hot-piece and game-speed numbers below ran lift.py, clang or the game directly, and the bug did not affect them.A second pair after the queue bug was fixed (
--timing --cpu-only, one round, the PC heavily loaded and the builds at idle priority, so both absolute times are high again): make_game took 640 s before (lift 614 s: discovery alone 162 s, against 26 s on a quiet PC) and 42 s after (lift skipped; the rest is the finish stage's file copies).hookshot,hookshot_vk.soandimage.bin(sha1).code_010) and ran make_game again. It printed "the translation kept from the last build does not fit this one (1 of 80 files: code_010.c): translating the game again", lifted, compiled that one file, and produced the samehookshotsha1 as the full build.Lift steps (lift.py + split_hot.py run alone, two rounds each). All 95 output files and
chunks.txtwere sha1-identical to the old lift:Shader list: the finish stage took 1 s when the list was kept, against 4-7 s when it was made.
Windows setup parts: I could not run a real package here. On a 104 MB stand-in package (three parts), three rounds took 1.81 / 1.50 / 1.50 s before and 1.05 / 1.15 / 1.04 s after. The zips and
hookshot-setup.txtwere identical.Windows path for the per-file passes: measured on Linux with
HOOKSHOT_LIFT_NO_FORK=1, against the old one-process lift (HOOKSHOT_LIFT_JOBS=1), two rounds. The three per-file passes (big chunk split, patches, output size + page-mark groups) took 9.9 / 16.1 s, against 40.6 / 35.6 s. The non-hot lifted files were byte-identical. The hot pieces differ only because of item 7. The whole lift's time varied more than that, because the one-process emit took 56-84 s from run to run.Hot pieces kept: this was measured without a game run, on the game's real
code_hot.c(849 functions). In 12 trials, each adding 1-60 lines to one random hot function, the next split changed 5.42 of the 8 pieces on average before (1-6 per trial) and exactly 1 after. A fresh split put the same functions in each piece as the old code.Tried and dropped:
Full build of the branch (before merging main, still with the dropped item 6): lift 94 s, 9 of 264 files compiled (the eight hot pieces, 165-363 s each, and one more), then a kept update from it as above.
Merge with main: main removed
oot_native_resolution.py(#2042) and added patches tooot_patches.apply(the text hooks of #2366, Camera speed of #1945). They are in the per-filepatch_file/reportsplit now; the ripple fix of #2409 goes through the frame-time table as before. A full build of the merge booted from the test save into Kokiri Forest and played 20 s of scripted running and sword swings (headless) without an error. It lifted with no patch warnings: every patch found once, the camera speed at its 4 functions, the three text hooks, both step divides and both frame counters (the ripple one included). 3 files compiled.Tests:
tests/run_all.pyat the head (the merge and the review fixes): 402 passed, 0 failed, 12 skipped. New or changed for the fixes: a kept list of another lift or format is not used, a clean-up that cannot remove a file is no failed build, the kept hot pieces change the lift key, and no module of the lift imports by a name made at run time. Before the merge: 364 passed, 1 failed (test_wayland_libdecor_hint, a harness-Werrorwarning that also fails without this PR). New tests:test_kept_translation.py,test_shader_list_skip.pyandtest_sticky_hot_pieces.py. Updated tests that cut functions out of build.py and make_game.py:test_lift_cleanup_error.pyandtest_compile_retry_jobs.py.Not tested
lz4, and were fixed):Storage.cs): no .NET SDK here, only the Python builder's twin was run;code_hot_1.c(-Wprofile-instr-out-of-date). This happens without this PR too.Not in this PR (measured or written, for later)
Compiling the hot pieces without loop unrolling (
-fno-unroll-loops). It cut the hot pieces' compile a lot (code_hot_4.c261 → 216 s alone; all eight 4547 → 2807 s of CPU in a loaded build), and Kokiri Forest ran as fast (62.8/64.8 fps before, 65.1/65.2 after). But a review found that it also stops clang from fully unrolling the lifter's ~25,000 fixed-count lane loops (for (_i < 4)and the like): on six lane-heavy functions ofcode_hot_5.cthe loops left in the code went from 65 to 263 and the stack accesses from 744 to 1165. The scenes measured barely run those functions, so they could not show it. Reverted. A narrower set (-mllvm -unroll-runtime=false -mllvm -unroll-allow-partial=false, perhaps-mllvm -unroll-max-upperbound=0) gave code byte-identical to today's on that sample; its compile-time gain with the play profile is not measured yet.Smaller chunks of ordinary code: three 24 MB chunks compiled whole took 145 s of CPU and 1.4-1.6 GB each; the same code cut in thirds (at function starts, as
split_bigcuts) took 117 s (-19%) and 0.52-0.70 GB each. More compilers would fit on a 16 GB PC, and less CPU on any PC. It changes every ordinary chunk's object (one full compile for everyone) and needs a game-speed check, since fewer callees share a file.Unpacking the romfs while the game is translated and compiled (about as long as the unpack, ~1-2 minutes, off a first build): written and tested game-free, not measured end to end.
There is a cleanup integration point with #2367 (portable Builder): both touch
Storage.cs, and your kept-translation marker changes what after-build cleanup may remove. When combining them,Storage.Free(afterBuild)should keep the marked lift in the selected portable work home; explicit Free up space should retain your current semantics. My #2250 separately scopes stale-process cleanup to proven work/output paths, so retaining a translation must not broaden which processes cleanup may terminate. These look complementary rather than duplicate work; could we preserve those two contracts when rebasing whichever lands second?github-sync referenced this pull request2026-11-12 06:22:54 +00:00