perf(build): overlap package extraction and translation on cold installs #1569

Merged
HumanBean merged 7 commits from HumanBean/Longshot:perf/cold-lifted-source-scan into main 2026-11-18 14:56:19 +00:00
Owner

Type: perf · Risk: medium · Tested on: CachyOS Linux, Ryzen 9 7900X, RTX 3090 · Needs: Windows cold-build timing

TL;DR

The complete cold player build took 897.19 s on main and 802.35 s with this PR. That saves 94.84 s (10.6%). Compared with the revision before #1567, the two PRs together save 169.42 s (17.4%). Large first installs start translating the executable while the rest of the package is still being read. The compiler also spends less time scanning generated C. The timed builds finished preparing the same 38,809 programs and 32,655 noted graphics pipelines.

Breakdown

  • Who it helps: Players building a game from scratch, especially from a large package on an SSD.
  • What was wrong: The translator waited for every game asset, although it only reads the executable. Compiler planning also searched through every generated function body.
  • What changes: A fresh install can extract assets alongside translation. The function scanner jumps straight to candidate definitions.
  • What does not change: The normal builder validates the assets before starting native compilation. All 93 generated C files compared byte for byte.
  • How it was checked: Complete local package-to-ready builds and focused checks for output equivalence, concurrency, package replacement and incomplete extraction.
  • What is still open: Windows timing.
  • Risk: The archive reader and translator can use memory concurrently. Existing installs, known spinning output or work drives, and constrained machines use the sequential path.

Why

The control waited for all the assets before starting translation. Compiler planning and chunk splitting then searched 1.56 GB of generated C for column-zero function signatures. #1567 had already reduced cold compute pipeline preparation. This PR addresses two earlier stages of the same first build.

How to reproduce

Seen on: CachyOS Linux 7.3.3, Ryzen 9 7900X, 32 GB, NVIDIA RTX 3090 with driver 620.70.06.

Settings and switches: Most shader preparation, timing queue with 12 logical CPUs, pinned release toolchain already installed. All build and driver caches start empty apart from the third-party objects bundled for players. The OS file cache was shared between runs.

Steps: On each revision, run the release builder through the timing queue with the same private package and --compile-shaders most. Give each invocation new output, work, XDG and driver-cache directories. Disable the shared lift and object stores and provide the 316 game-independent third-party objects shipped by the Linux release. Start the timer at builder process launch and stop it on successful exit after the game reports shader preparation complete. Check the preparation marker and pipeline counts locally.

Expected: Translation can proceed as soon as the executable has been extracted and validated.

Actual: Main waits for the full package before starting translation.

What changed

The builder starts translating as soon as it has the executable. A narrower search for function definitions also shortens translation and compiler setup.

  • nx2/make_game.py extracts and validates the executable first, then uses the existing asset-only extractor while translating. It waits for that reader before compiling and checks that the input package identity still matches. This applies to first installs from packages of at least 1 GiB, with multiple build jobs, at least 8 GiB of available RAM, and no known spinning work or output disk.
  • nx2/lifted_c.py checks candidate column-zero definitions using the complete signature. The chunk splitters keep leading notes with their functions, and the compiler reuses the same search for memory estimates.
  • Synthetic fixtures cover both extraction orders, a swapped input and missing assets. Existing cache and update tests exercise the sequential path.

Measurements

Verdict: The complete cold player build took 897.19 s on main and 802.35 s with this PR. That saves 94.84 s (10.6%). Compared with the revision before #1567, the two PRs together save 169.42 s (17.4%).

Hardware: CachyOS Linux 7.3.3, NVIDIA RTX 3090 / 620.70.06, Ryzen 9 7900X, 32 GB.

Scene: Fresh player build of the same local package, through completed shader preparation.

Method: Two full builds per revision, interleaved on the same PC. Execution order was main, pre-#1567, branch, branch, pre-#1567, main. The first row lists each complete build in seconds. The remaining rows give the median for each revision. Compilation includes compiler setup and linking. The last column includes saving the pipeline caches and exiting the builder.

How to reproduce: On each revision, run the release builder through the timing queue with the same private package and --compile-shaders most. Give each invocation new output, work, XDG and driver-cache directories. Disable the shared lift and object stores and provide the 316 game-independent third-party objects shipped by the Linux release. Start the timer at builder process launch and stop it on successful exit after the game reports shader preparation complete. Check the preparation marker and pipeline counts locally.

Elapsed seconds Before both PRs: main @ 7daafa7a Before: main @ 430eef4f After: this PR @ 736b82fd
Individual full builds 983.99, 959.54 935.96, 858.41 804.04, 800.66
Full build, median 971.77 897.19 802.35
Read and translate 130.74 149.74 79.95
Compile and link 349.16 364.50 339.39
Assemble folder 6.33 6.51 6.60
Prepare shaders and exit 485.36 376.27 376.24

Each run built 208 of 524 native inputs locally, used all 316 shipped objects, translated 9,927 compute variants and finished with zero failed shaders or missing noted pipelines. #1567 reduced the number of compute pipelines sent to the driver from 9,927 to 3,656. The queue flagged CPU pressure during all six runs. The new PR saved 131.92 s in the first comparison and 57.75 s in the reverse comparison. Extraction through the start of native compilation fell from a median of 149.74 s to 79.95 s. The 25.11 s difference in native compilation and linking includes normal run-to-run variation as well as the faster compiler setup.

For the compiler scan in isolation, fresh processes ran in main / change / change / main order on the same PC. The compile planner still listed all 524 inputs. All four translations generated the same 93 C files.

Cold phase Before: main @ 430eef4f After: this PR @ dc293287
Two function scans over 1.56 GB 11.49 s 1.49 s
Compiler setup through worker pool creation 10.57 s 5.25 s
Splitting large chunks during translation 4.35 s 1.15 s
Whole translation, three workers 79.25 s 74.08 s

The scan timers overlap with translation and compiler setup. The reverse-order translation pair took 73.91 s on main and 74.18 s on the branch.

Flags and settings

None.

Testing

Tested: Cold build through actual shader preparation on Linux. The builder and driver both reported completion and the preparation marker was present. Of 43 cache tests, 42 passed and one was skipped on Linux. Chunk splitting, compiler memory planning and LF/CRLF fixtures passed. The full game-free suite on the branch and on the control produced the same result: 720 passed, 41 failed and 19 skipped. The 41 common failures are listed below.

Checks failing on both revisions

tests/test_bug_report_defaults.py, tests/test_capture_readback_visibility.py, tests/test_cpu_report.py, tests/test_crt_options.py, tests/test_damage_debug_menu.py, tests/test_dev_packs.py, tests/test_display_startup_failure.py, tests/test_dlss_feature_size.py, tests/test_draw_stream_visibility.py, tests/test_failed_compile_keeps_objects.py, tests/test_fsr_image_retirement.py, tests/test_fsr_uniform_ownership.py, tests/test_guest_imported.py, tests/test_hud_layout_row.py, tests/test_invalidate_quick.py, tests/test_invalidate_vcore.py, tests/test_libc_hooks.py, tests/test_linux_preflight.py, tests/test_linux_pressure.py, tests/test_linux_update.py, tests/test_log_open_failure.py, tests/test_mirror_budget.py, tests/test_missing_nup.py, tests/test_mods_size_table.py, tests/test_nonuniform_image_arrays.py, tests/test_parallel_init_order.py, tests/test_pending_guest_import.py, tests/test_pipeline_play_cache.py, tests/test_present_source_crop.py, tests/test_present_thread.py, tests/test_profiling_mode.py, tests/test_psychov35_fsr3_fg.py, tests/test_readback_timeout_source.py, tests/test_replay_write_order.py, tests/test_run_all_changed.py, tests/test_slow_frames.py, tests/test_staging_burst.py, tests/test_switch_table.py, tests/test_upscaler_rows.py, tests/test_vk_occlusion_queries.py, tests/test_walk_key.py

Not tested: Windows wall time, spinning-disk performance and desktop play after preparation.

Must be tested:

  • Windows SSD, Test Farm: repeat fresh package builds on both revisions with the same disk and driver, expecting all shader and pipeline checks to pass and extraction and translation to overlap on an eligible machine.

Risks

What could break: Parsing the package directory twice adds work. The reader and translator share CPU, disk bandwidth and RAM during the overlap. During extraction and translation, sampled peak aggregate resident memory was 20521 / 23948 MiB on the branch and 23980 MiB on main (two-second samples across the builder process tree, including shared pages). Small packages, a single build job, low available memory and recognized spinning work or output disks retain sequential extraction. If the package changes between reads, the builder rejects the result and the next attempt can extract again.

Turning it off: Revert the two build changes to restore serial extraction and the previous function scanner.

Compatibility: A previously saved translation built with the old scanner regenerates once after upgrading. A full translation took 74.08 s at the median on this PC with three workers. Its generated C matched byte for byte, so existing native objects with matching keys remain reusable.

Issues

Related work: #362, #1002 and #1567.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • Docs updated with six end-to-end timing receipts.
  • A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md).
  • A check under tests/ that reads changed source updated in this PR, not weakened.
**Type:** perf · **Risk:** medium · **Tested on:** CachyOS Linux, Ryzen 9 7900X, RTX 3090 · **Needs:** Windows cold-build timing ## TL;DR The complete cold player build took **897.19 s on main and 802.35 s with this PR**. That saves 94.84 s (10.6%). Compared with the revision before #1567, the two PRs together save 169.42 s (17.4%). Large first installs start translating the executable while the rest of the package is still being read. The compiler also spends less time scanning generated C. The timed builds finished preparing the same 38,809 programs and 32,655 noted graphics pipelines. ## Breakdown - **Who it helps:** Players building a game from scratch, especially from a large package on an SSD. - **What was wrong:** The translator waited for every game asset, although it only reads the executable. Compiler planning also searched through every generated function body. - **What changes:** A fresh install can extract assets alongside translation. The function scanner jumps straight to candidate definitions. - **What does not change:** The normal builder validates the assets before starting native compilation. All 93 generated C files compared byte for byte. - **How it was checked:** Complete local package-to-ready builds and focused checks for output equivalence, concurrency, package replacement and incomplete extraction. - **What is still open:** Windows timing. - **Risk:** The archive reader and translator can use memory concurrently. Existing installs, known spinning output or work drives, and constrained machines use the sequential path. ## Why The control waited for all the assets before starting translation. Compiler planning and chunk splitting then searched 1.56 GB of generated C for column-zero function signatures. #1567 had already reduced cold compute pipeline preparation. This PR addresses two earlier stages of the same first build. ## How to reproduce **Seen on:** CachyOS Linux 7.3.3, Ryzen 9 7900X, 32 GB, NVIDIA RTX 3090 with driver 620.70.06. **Settings and switches:** Most shader preparation, timing queue with 12 logical CPUs, pinned release toolchain already installed. All build and driver caches start empty apart from the third-party objects bundled for players. The OS file cache was shared between runs. **Steps:** On each revision, run the release builder through the timing queue with the same private package and `--compile-shaders most`. Give each invocation new output, work, XDG and driver-cache directories. Disable the shared lift and object stores and provide the 316 game-independent third-party objects shipped by the Linux release. Start the timer at builder process launch and stop it on successful exit after the game reports shader preparation complete. Check the preparation marker and pipeline counts locally. **Expected:** Translation can proceed as soon as the executable has been extracted and validated. **Actual:** Main waits for the full package before starting translation. ## What changed The builder starts translating as soon as it has the executable. A narrower search for function definitions also shortens translation and compiler setup. - `nx2/make_game.py` extracts and validates the executable first, then uses the existing asset-only extractor while translating. It waits for that reader before compiling and checks that the input package identity still matches. This applies to first installs from packages of at least 1 GiB, with multiple build jobs, at least 8 GiB of available RAM, and no known spinning work or output disk. - `nx2/lifted_c.py` checks candidate column-zero definitions using the complete signature. The chunk splitters keep leading notes with their functions, and the compiler reuses the same search for memory estimates. - Synthetic fixtures cover both extraction orders, a swapped input and missing assets. Existing cache and update tests exercise the sequential path. ## Measurements **Verdict:** The complete cold player build took **897.19 s on main and 802.35 s with this PR**. That saves 94.84 s (10.6%). Compared with the revision before #1567, the two PRs together save 169.42 s (17.4%). Hardware: CachyOS Linux 7.3.3, NVIDIA RTX 3090 / 620.70.06, Ryzen 9 7900X, 32 GB. Scene: Fresh player build of the same local package, through completed shader preparation. Method: Two full builds per revision, interleaved on the same PC. Execution order was main, pre-#1567, branch, branch, pre-#1567, main. The first row lists each complete build in seconds. The remaining rows give the median for each revision. Compilation includes compiler setup and linking. The last column includes saving the pipeline caches and exiting the builder. How to reproduce: On each revision, run the release builder through the timing queue with the same private package and `--compile-shaders most`. Give each invocation new output, work, XDG and driver-cache directories. Disable the shared lift and object stores and provide the 316 game-independent third-party objects shipped by the Linux release. Start the timer at builder process launch and stop it on successful exit after the game reports shader preparation complete. Check the preparation marker and pipeline counts locally. | Elapsed seconds | Before both PRs: main @ 7daafa7a | Before: main @ 430eef4f | After: this PR @ 736b82fd | |---|---:|---:|---:| | Individual full builds | 983.99, 959.54 | 935.96, 858.41 | 804.04, 800.66 | | Full build, median | 971.77 | 897.19 | 802.35 | | Read and translate | 130.74 | 149.74 | 79.95 | | Compile and link | 349.16 | 364.50 | 339.39 | | Assemble folder | 6.33 | 6.51 | 6.60 | | Prepare shaders and exit | 485.36 | 376.27 | 376.24 | Each run built 208 of 524 native inputs locally, used all 316 shipped objects, translated 9,927 compute variants and finished with zero failed shaders or missing noted pipelines. #1567 reduced the number of compute pipelines sent to the driver from 9,927 to 3,656. The queue flagged CPU pressure during all six runs. The new PR saved 131.92 s in the first comparison and 57.75 s in the reverse comparison. Extraction through the start of native compilation fell from a median of 149.74 s to 79.95 s. The 25.11 s difference in native compilation and linking includes normal run-to-run variation as well as the faster compiler setup. For the compiler scan in isolation, fresh processes ran in main / change / change / main order on the same PC. The compile planner still listed all 524 inputs. All four translations generated the same 93 C files. | Cold phase | Before: main @ 430eef4f | After: this PR @ dc293287 | |---|---:|---:| | Two function scans over 1.56 GB | 11.49 s | 1.49 s | | Compiler setup through worker pool creation | 10.57 s | 5.25 s | | Splitting large chunks during translation | 4.35 s | 1.15 s | | Whole translation, three workers | 79.25 s | 74.08 s | The scan timers overlap with translation and compiler setup. The reverse-order translation pair took 73.91 s on main and 74.18 s on the branch. ### Flags and settings None. ## Testing **Tested:** Cold build through actual shader preparation on Linux. The builder and driver both reported completion and the preparation marker was present. Of 43 cache tests, 42 passed and one was skipped on Linux. Chunk splitting, compiler memory planning and LF/CRLF fixtures passed. The full game-free suite on the branch and on the control produced the same result: 720 passed, 41 failed and 19 skipped. The 41 common failures are listed below. <details><summary>Checks failing on both revisions</summary> `tests/test_bug_report_defaults.py`, `tests/test_capture_readback_visibility.py`, `tests/test_cpu_report.py`, `tests/test_crt_options.py`, `tests/test_damage_debug_menu.py`, `tests/test_dev_packs.py`, `tests/test_display_startup_failure.py`, `tests/test_dlss_feature_size.py`, `tests/test_draw_stream_visibility.py`, `tests/test_failed_compile_keeps_objects.py`, `tests/test_fsr_image_retirement.py`, `tests/test_fsr_uniform_ownership.py`, `tests/test_guest_imported.py`, `tests/test_hud_layout_row.py`, `tests/test_invalidate_quick.py`, `tests/test_invalidate_vcore.py`, `tests/test_libc_hooks.py`, `tests/test_linux_preflight.py`, `tests/test_linux_pressure.py`, `tests/test_linux_update.py`, `tests/test_log_open_failure.py`, `tests/test_mirror_budget.py`, `tests/test_missing_nup.py`, `tests/test_mods_size_table.py`, `tests/test_nonuniform_image_arrays.py`, `tests/test_parallel_init_order.py`, `tests/test_pending_guest_import.py`, `tests/test_pipeline_play_cache.py`, `tests/test_present_source_crop.py`, `tests/test_present_thread.py`, `tests/test_profiling_mode.py`, `tests/test_psychov35_fsr3_fg.py`, `tests/test_readback_timeout_source.py`, `tests/test_replay_write_order.py`, `tests/test_run_all_changed.py`, `tests/test_slow_frames.py`, `tests/test_staging_burst.py`, `tests/test_switch_table.py`, `tests/test_upscaler_rows.py`, `tests/test_vk_occlusion_queries.py`, `tests/test_walk_key.py` </details> **Not tested:** Windows wall time, spinning-disk performance and desktop play after preparation. **Must be tested:** - Windows SSD, Test Farm: repeat fresh package builds on both revisions with the same disk and driver, expecting all shader and pipeline checks to pass and extraction and translation to overlap on an eligible machine. ## Risks **What could break:** Parsing the package directory twice adds work. The reader and translator share CPU, disk bandwidth and RAM during the overlap. During extraction and translation, sampled peak aggregate resident memory was 20521 / 23948 MiB on the branch and 23980 MiB on main (two-second samples across the builder process tree, including shared pages). Small packages, a single build job, low available memory and recognized spinning work or output disks retain sequential extraction. If the package changes between reads, the builder rejects the result and the next attempt can extract again. **Turning it off:** Revert the two build changes to restore serial extraction and the previous function scanner. **Compatibility:** A previously saved translation built with the old scanner regenerates once after upgrading. A full translation took 74.08 s at the median on this PC with three workers. Its generated C matched byte for byte, so existing native objects with matching keys remain reusable. ## Issues Related work: #362, #1002 and #1567. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] Docs updated with six end-to-end timing receipts. - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`). - [x] A check under `tests/` that reads changed source updated in this PR, not weakened.
perf(build): speed up function boundary scans across lifted C
Some checks failed
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 11s
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 4s
dc29328744
Assisted-by: OpenAI Codex
Co-Authored-By: OpenAI Codex <[email protected]>
Author
Owner

Tested in game: Build-time Python change. On CachyOS Linux, Ryzen 9 7900X and RTX 3090, ran both scanners on real local game-generated C from the development base. All 4,281 function signatures and note offsets matched across 44,593,210 bytes. Full cold translation runs are queued.

Tested in game: Build-time Python change. On CachyOS Linux, Ryzen 9 7900X and RTX 3090, ran both scanners on real local game-generated C from the development base. All 4,281 function signatures and note offsets matched across 44,593,210 bytes. Full cold translation runs are queued.
Collaborator

Possibly related, for context:

  • issue #362 perf(build): a first build in under 10 minutes: what was measured, and a proposal (a clang-free cold tier) — Open first-build performance proposal covering build-time reductions
Possibly related, for context: - issue #362 perf(build): a first build in under 10 minutes: what was measured, and a proposal (a clang-free cold tier) — Open first-build performance proposal covering build-time reductions
Author
Owner

Measured both build scans across 1.56 GB of generated C in four fresh processes. The combined median fell from 11.49 s to 1.49 s with identical chunk contents, costs and maximum function sizes. The full cold translation is running.

Measured both build scans across 1.56 GB of generated C in four fresh processes. The combined median fell from 11.49 s to 1.49 s with identical chunk contents, costs and maximum function sizes. The full cold translation is running.
docs: record cold lifted-source scan measurements
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
Third-party parts / As their notes say (pull_request) Successful in 14s
PR description / Description follows the template (pull_request_target) Successful in 4s
7a6e3e368d
Assisted-by: OpenAI Codex
Co-Authored-By: OpenAI Codex <[email protected]>
Author
Owner

Added the four complete cold translation runs, four cold compiler-planning runs, and all timing receipts. Compiler startup was 10.57 s → 5.25 s; the real translation’s chunk split was 4.35 s → 1.15 s, with byte-identical generated C. The full suite exposed 41 failures shared with the control and three environment-only failures which passed on rerun.

Added the four complete cold translation runs, four cold compiler-planning runs, and all timing receipts. Compiler startup was 10.57 s → 5.25 s; the real translation’s chunk split was 4.35 s → 1.15 s, with byte-identical generated C. The full suite exposed 41 failures shared with the control and three environment-only failures which passed on rerun.
Assisted-by: OpenAI Codex
Co-Authored-By: OpenAI Codex <[email protected]>
Assisted-by: OpenAI Codex
Co-Authored-By: OpenAI Codex <[email protected]>
test(build): cover cold install guards and record full builds
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 13s
PR description / Description follows the template (pull_request_target) Successful in 5s
3a3d82ba4f
Assisted-by: OpenAI Codex
HumanBean changed title from WIP: perf(build): skip function bodies when scanning generated C to WIP: perf(build): overlap package extraction and translation on cold installs 2026-11-18 10:17:41 +00:00
Author
Owner

Added overlapping asset extraction and translation, plus full cold package-to-shader-ready measurements. First runs took 983.99 s before #1567, 935.96 s on main, and 804.04 s on this branch. Each completed with the same program and noted graphics pipeline counts. Reverse-order repeats and the full suite are running.

Added overlapping asset extraction and translation, plus full cold package-to-shader-ready measurements. First runs took 983.99 s before #1567, 935.96 s on main, and 804.04 s on this branch. Each completed with the same program and noted graphics pipeline counts. Reverse-order repeats and the full suite are running.
test(build): check the lift cache across both extraction paths
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 7s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 11s
Third-party parts / As their notes say (pull_request) Successful in 11s
PR description / Description follows the template (pull_request_target) Successful in 4s
6f9abc8e63
Assisted-by: OpenAI Codex
Author
Owner

The second optimized build completed in 800.66 s, close to the first 804.04 s. The full suite now matches the control exactly: 720 passed, the same 41 failed, 19 skipped. The lift-cache test now selects the translator function by its Python syntax so it checks both callers.

The second optimized build completed in 800.66 s, close to the first 804.04 s. The full suite now matches the control exactly: 720 passed, the same 41 failed, 19 skipped. The lift-cache test now selects the translator function by its Python syntax so it checks both callers.
docs: record cold build control repeats
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 10s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 12s
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 4s
beanstalk/approval needs a second approval (sensitive paths)
6e3a41a4cb
Assisted-by: OpenAI Codex
Author
Owner

Six full Linux cold builds completed. Median package-to-ready time was 971.77 s before both PRs, 897.19 s on main and 802.35 s here. This PR saved 94.84 s (10.6%) over main and both PRs saved 169.42 s (17.4%). All runs finished with 38,809 programs, 32,655 noted graphics pipelines and zero failures. @Beanstalk please compare first-time package builds on Windows with an SSD and Most shaders. Use fresh build and driver caches and report elapsed time through successful shader preparation on main and this branch.

Six full Linux cold builds completed. Median package-to-ready time was 971.77 s before both PRs, 897.19 s on main and 802.35 s here. This PR saved 94.84 s (10.6%) over main and both PRs saved 169.42 s (17.4%). All runs finished with 38,809 programs, 32,655 noted graphics pipelines and zero failures. @Beanstalk please compare first-time package builds on Windows with an SSD and Most shaders. Use fresh build and driver caches and report elapsed time through successful shader preparation on main and this branch.
Member

@HumanBean approval of 6e3a41a recorded. This diff touches sensitive paths, so a second member of BigBeans must approve it too.

@HumanBean approval of `6e3a41a` recorded. This diff touches sensitive paths, so a second member of `BigBeans` must approve it too.
HumanBean changed title from WIP: perf(build): overlap package extraction and translation on cold installs to perf(build): overlap package extraction and translation on cold installs 2026-11-18 11:09:29 +00:00
Member

Beanstalk test farm

Commit 6e3a41a · tier untrusted · approved by HumanBean · dashboard

This diff touches sensitive paths, so it needs two members of BigBeans: nx2/build.py, nx2/chunks.py, nx2/lifted_c.py, nx2/make_game.py, nx2/split_hot.py

A member of BigBeans can review this commit and comment @Beanstalk to test it.

<!-- beanstalk:sticky --> ### Beanstalk test farm Commit `6e3a41a` · tier **untrusted** · approved by HumanBean · [dashboard](https://beanstalk.magicbean.dev/prs/1569) This diff touches sensitive paths, so it needs two members of `BigBeans`: `nx2/build.py`, `nx2/chunks.py`, `nx2/lifted_c.py`, `nx2/make_game.py`, `nx2/split_hot.py` A member of `BigBeans` can review this commit and comment `@Beanstalk` to test it.
Member

Farm readiness review for 6e3a41a4: the requested Windows metric is first-time package extraction through completed shader preparation. Beanstalk’s timed builds start from the base ROMFS, copy it on Windows, and set LONGSHOT_NO_PREPARE=1. A cold timing run here would miss both important parts of the requested endpoint. Please record a Windows SSD package-to-ready A/B using fresh build and driver caches, or narrow the farm request to the named build stages it can measure. Timing method

Farm readiness review for `6e3a41a4`: the requested Windows metric is first-time package extraction through completed shader preparation. Beanstalk’s timed builds start from the base ROMFS, copy it on Windows, and set `LONGSHOT_NO_PREPARE=1`. A cold timing run here would miss both important parts of the requested endpoint. Please record a Windows SSD package-to-ready A/B using fresh build and driver caches, or narrow the farm request to the named build stages it can measure. [Timing method](https://git.magicbean.dev/MagicBeanDev/Beanstalk/src/commit/e431624320e261f63a070a52415fc464f1d459e2/docs/test-plan.md#timing)
HumanBean deleted branch perf/cold-lifted-source-scan 2026-11-18 14:56:19 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1569
No description provided.