perf(build): overlap package extraction and translation on cold installs #1569
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1569
Loading…
Reference in a new issue
No description provided.
Delete branch "HumanBean/Longshot:perf/cold-lifted-source-scan"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: medium · Tested on: CachyOS Linux, Ryzen 9 7900X, RTX 3090 · Needs: Windows cold-build timing
TL;DR
The complete cold player build took 897.19 s on main and 802.35 s with this PR. That saves 94.84 s (10.6%). Compared with the revision before #1567, the two PRs together save 169.42 s (17.4%). Large first installs start translating the executable while the rest of the package is still being read. The compiler also spends less time scanning generated C. The timed builds finished preparing the same 38,809 programs and 32,655 noted graphics pipelines.
Breakdown
Why
The control waited for all the assets before starting translation. Compiler planning and chunk splitting then searched 1.56 GB of generated C for column-zero function signatures. #1567 had already reduced cold compute pipeline preparation. This PR addresses two earlier stages of the same first build.
How to reproduce
Seen on: CachyOS Linux 7.3.3, Ryzen 9 7900X, 32 GB, NVIDIA RTX 3090 with driver 620.70.06.
Settings and switches: Most shader preparation, timing queue with 12 logical CPUs, pinned release toolchain already installed. All build and driver caches start empty apart from the third-party objects bundled for players. The OS file cache was shared between runs.
Steps: On each revision, run the release builder through the timing queue with the same private package and
--compile-shaders most. Give each invocation new output, work, XDG and driver-cache directories. Disable the shared lift and object stores and provide the 316 game-independent third-party objects shipped by the Linux release. Start the timer at builder process launch and stop it on successful exit after the game reports shader preparation complete. Check the preparation marker and pipeline counts locally.Expected: Translation can proceed as soon as the executable has been extracted and validated.
Actual: Main waits for the full package before starting translation.
What changed
The builder starts translating as soon as it has the executable. A narrower search for function definitions also shortens translation and compiler setup.
nx2/make_game.pyextracts and validates the executable first, then uses the existing asset-only extractor while translating. It waits for that reader before compiling and checks that the input package identity still matches. This applies to first installs from packages of at least 1 GiB, with multiple build jobs, at least 8 GiB of available RAM, and no known spinning work or output disk.nx2/lifted_c.pychecks candidate column-zero definitions using the complete signature. The chunk splitters keep leading notes with their functions, and the compiler reuses the same search for memory estimates.Measurements
Verdict: The complete cold player build took 897.19 s on main and 802.35 s with this PR. That saves 94.84 s (10.6%). Compared with the revision before #1567, the two PRs together save 169.42 s (17.4%).
Hardware: CachyOS Linux 7.3.3, NVIDIA RTX 3090 / 620.70.06, Ryzen 9 7900X, 32 GB.
Scene: Fresh player build of the same local package, through completed shader preparation.
Method: Two full builds per revision, interleaved on the same PC. Execution order was main, pre-#1567, branch, branch, pre-#1567, main. The first row lists each complete build in seconds. The remaining rows give the median for each revision. Compilation includes compiler setup and linking. The last column includes saving the pipeline caches and exiting the builder.
How to reproduce: On each revision, run the release builder through the timing queue with the same private package and
--compile-shaders most. Give each invocation new output, work, XDG and driver-cache directories. Disable the shared lift and object stores and provide the 316 game-independent third-party objects shipped by the Linux release. Start the timer at builder process launch and stop it on successful exit after the game reports shader preparation complete. Check the preparation marker and pipeline counts locally.7daafa7a430eef4f736b82fdEach run built 208 of 524 native inputs locally, used all 316 shipped objects, translated 9,927 compute variants and finished with zero failed shaders or missing noted pipelines. #1567 reduced the number of compute pipelines sent to the driver from 9,927 to 3,656. The queue flagged CPU pressure during all six runs. The new PR saved 131.92 s in the first comparison and 57.75 s in the reverse comparison. Extraction through the start of native compilation fell from a median of 149.74 s to 79.95 s. The 25.11 s difference in native compilation and linking includes normal run-to-run variation as well as the faster compiler setup.
For the compiler scan in isolation, fresh processes ran in main / change / change / main order on the same PC. The compile planner still listed all 524 inputs. All four translations generated the same 93 C files.
430eef4fdc293287The scan timers overlap with translation and compiler setup. The reverse-order translation pair took 73.91 s on main and 74.18 s on the branch.
Flags and settings
None.
Testing
Tested: Cold build through actual shader preparation on Linux. The builder and driver both reported completion and the preparation marker was present. Of 43 cache tests, 42 passed and one was skipped on Linux. Chunk splitting, compiler memory planning and LF/CRLF fixtures passed. The full game-free suite on the branch and on the control produced the same result: 720 passed, 41 failed and 19 skipped. The 41 common failures are listed below.
Checks failing on both revisions
tests/test_bug_report_defaults.py,tests/test_capture_readback_visibility.py,tests/test_cpu_report.py,tests/test_crt_options.py,tests/test_damage_debug_menu.py,tests/test_dev_packs.py,tests/test_display_startup_failure.py,tests/test_dlss_feature_size.py,tests/test_draw_stream_visibility.py,tests/test_failed_compile_keeps_objects.py,tests/test_fsr_image_retirement.py,tests/test_fsr_uniform_ownership.py,tests/test_guest_imported.py,tests/test_hud_layout_row.py,tests/test_invalidate_quick.py,tests/test_invalidate_vcore.py,tests/test_libc_hooks.py,tests/test_linux_preflight.py,tests/test_linux_pressure.py,tests/test_linux_update.py,tests/test_log_open_failure.py,tests/test_mirror_budget.py,tests/test_missing_nup.py,tests/test_mods_size_table.py,tests/test_nonuniform_image_arrays.py,tests/test_parallel_init_order.py,tests/test_pending_guest_import.py,tests/test_pipeline_play_cache.py,tests/test_present_source_crop.py,tests/test_present_thread.py,tests/test_profiling_mode.py,tests/test_psychov35_fsr3_fg.py,tests/test_readback_timeout_source.py,tests/test_replay_write_order.py,tests/test_run_all_changed.py,tests/test_slow_frames.py,tests/test_staging_burst.py,tests/test_switch_table.py,tests/test_upscaler_rows.py,tests/test_vk_occlusion_queries.py,tests/test_walk_key.pyNot tested: Windows wall time, spinning-disk performance and desktop play after preparation.
Must be tested:
Risks
What could break: Parsing the package directory twice adds work. The reader and translator share CPU, disk bandwidth and RAM during the overlap. During extraction and translation, sampled peak aggregate resident memory was 20521 / 23948 MiB on the branch and 23980 MiB on main (two-second samples across the builder process tree, including shared pages). Small packages, a single build job, low available memory and recognized spinning work or output disks retain sequential extraction. If the package changes between reads, the builder rejects the result and the next attempt can extract again.
Turning it off: Revert the two build changes to restore serial extraction and the previous function scanner.
Compatibility: A previously saved translation built with the old scanner regenerates once after upgrading. A full translation took 74.08 s at the median on this PC with three workers. Its generated C matched byte for byte, so existing native objects with matching keys remain reusable.
Issues
Related work: #362, #1002 and #1567.
Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each new measured run (format indocs/tested-configurations.md).tests/that reads changed source updated in this PR, not weakened.Tested in game: Build-time Python change. On CachyOS Linux, Ryzen 9 7900X and RTX 3090, ran both scanners on real local game-generated C from the development base. All 4,281 function signatures and note offsets matched across 44,593,210 bytes. Full cold translation runs are queued.
Possibly related, for context:
Measured both build scans across 1.56 GB of generated C in four fresh processes. The combined median fell from 11.49 s to 1.49 s with identical chunk contents, costs and maximum function sizes. The full cold translation is running.
Added the four complete cold translation runs, four cold compiler-planning runs, and all timing receipts. Compiler startup was 10.57 s → 5.25 s; the real translation’s chunk split was 4.35 s → 1.15 s, with byte-identical generated C. The full suite exposed 41 failures shared with the control and three environment-only failures which passed on rerun.
WIP: perf(build): skip function bodies when scanning generated Cto WIP: perf(build): overlap package extraction and translation on cold installsAdded overlapping asset extraction and translation, plus full cold package-to-shader-ready measurements. First runs took 983.99 s before #1567, 935.96 s on main, and 804.04 s on this branch. Each completed with the same program and noted graphics pipeline counts. Reverse-order repeats and the full suite are running.
The second optimized build completed in 800.66 s, close to the first 804.04 s. The full suite now matches the control exactly: 720 passed, the same 41 failed, 19 skipped. The lift-cache test now selects the translator function by its Python syntax so it checks both callers.
Six full Linux cold builds completed. Median package-to-ready time was 971.77 s before both PRs, 897.19 s on main and 802.35 s here. This PR saved 94.84 s (10.6%) over main and both PRs saved 169.42 s (17.4%). All runs finished with 38,809 programs, 32,655 noted graphics pipelines and zero failures. @Beanstalk please compare first-time package builds on Windows with an SSD and Most shaders. Use fresh build and driver caches and report elapsed time through successful shader preparation on main and this branch.
@HumanBean approval of
6e3a41arecorded. This diff touches sensitive paths, so a second member ofBigBeansmust approve it too.WIP: perf(build): overlap package extraction and translation on cold installsto perf(build): overlap package extraction and translation on cold installsBeanstalk test farm
Commit
6e3a41a· tier untrusted · approved by HumanBean · dashboardThis diff touches sensitive paths, so it needs two members of
BigBeans:nx2/build.py,nx2/chunks.py,nx2/lifted_c.py,nx2/make_game.py,nx2/split_hot.pyA member of
BigBeanscan review this commit and comment@Beanstalkto test it.Farm readiness review for
6e3a41a4: the requested Windows metric is first-time package extraction through completed shader preparation. Beanstalk’s timed builds start from the base ROMFS, copy it on Windows, and setLONGSHOT_NO_PREPARE=1. A cold timing run here would miss both important parts of the requested endpoint. Please record a Windows SSD package-to-ready A/B using fresh build and driver caches, or narrow the farm request to the named build stages it can measure. Timing methodVeix referenced this pull request2026-11-18 16:37:01 +00:00