MagicBeanDev/Hookshot

Archived

Watch

Fork 0

perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries #2890

Open

github-sync wants to merge 1 commit from github/pr/2890 into main

github-sync commented

2026-11-12 19:01:03 +00:00

Owner

@jordanella opened this pull request on GitHub · 2026-11-12 18:33 UTC · original

A small update that changes a few of the game's functions still recompiles every lifted chunk that holds one, and a chunk is ~24 MB of C: a typical one takes ~227 s of compiler time, and the few heavy ones far more. Splitting the lifted C into ~4 MB pieces by guest address makes that ~41 s, with no cost in play or in a full build. Part of the dev-loop proposal's work on small updates (#2636).

Change

nx2/chunks.py:

Sub-boundaries: besides image/chunks.txt's stable boundaries, sub_starts adds one every SUB_STRIDE (0x40000) of guest address, at the first function at or after it. They are guest addresses, so they stay put when a function grows: a change moves no boundary but its own chunk's.
Not saved: chunks.txt keeps only the boundaries it had (the sub-boundaries are worked out again each lift from the same functions), so dropping this change later is clean.
A guard: more than 1000 chunks stops the lift with a message, since split_big, split_hot and build.py match three-digit chunk names.

The first build after this compiles every lifted chunk once (all boundaries are new): for a player, that update takes about as long as a full build, once. Updates after it recompile only the ~4 MB pieces that changed.

Measured

Windows 11, Ryzen 9 3900X:

One chunk's compile: a typical 24 MB chunk ~227 s of compiler time against ~41 s for a 4 MB piece. The heavy-function cluster (old chunks 021-023) still takes 300-900 s per piece.
A full build with my local build changes together (this, the lift key's environment, link skips and others), each in a fresh player-like work folder: 36:17 on main against 32:30, at the same total compiler time (25,700 against 25,500 s) over 698 files instead of 486. The smaller pieces spread better over the cores.
A two-PR update (#2093 and #1930, the same setup): 6:06 against 4:36; compiling 2:16 against 1:00.

In the game

No change in play: the 4 MB-chunk game against the 24 MB one, Kokiri Forest from a save (the game logic bounds its frame on the CPU there), RTX 3080 (NVIDIA 621.14), Windows 11, two interleaved pairs of 30 samples: median 29.6 / 30.0 fps against 31.0 / 30.0. No errors in either.

Testing

tests/test_sub_chunks.py (new): the first function past each stride; a long function spans strides without a boundary inside it; a function found since moves one boundary at most; chunks end at them as well as at the saved boundaries; write uses them and saves only chunks.txt's; the stride makes chunks of a few MB. (The 1000-chunk guard isn't tested: the game's code makes ~330 pieces.)
Every check that reads the changed files (9), through run_all.py, rebased on today's main (with #2748's spawned emit), Windows: 8 pass; test_spawn_emit fails the same way on main here.
Not tested: Linux.

See #2636.

🤖 Generated with Claude Code

> [@jordanella](https://github.com/jordanella) opened this pull request on GitHub · 2026-11-12 18:33 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2890) A small update that changes a few of the game's functions still recompiles every lifted chunk that holds one, and a chunk is ~24 MB of C: a typical one takes ~227 s of compiler time, and the few heavy ones far more. Splitting the lifted C into ~4 MB pieces by guest address makes that ~41 s, with no cost in play or in a full build. Part of the dev-loop proposal's work on small updates (#2636). ## Change `nx2/chunks.py`: - **Sub-boundaries:** besides `image/chunks.txt`'s stable boundaries, `sub_starts` adds one every `SUB_STRIDE` (0x40000) of guest address, at the first function at or after it. They are guest addresses, so they stay put when a function grows: a change moves no boundary but its own chunk's. - **Not saved:** `chunks.txt` keeps only the boundaries it had (the sub-boundaries are worked out again each lift from the same functions), so dropping this change later is clean. - **A guard:** more than 1000 chunks stops the lift with a message, since `split_big`, `split_hot` and `build.py` match three-digit chunk names. **The first build after this compiles every lifted chunk once** (all boundaries are new): for a player, that update takes about as long as a full build, once. Updates after it recompile only the ~4 MB pieces that changed. ## Measured Windows 11, Ryzen 9 3900X: - **One chunk's compile:** a typical 24 MB chunk ~227 s of compiler time against ~41 s for a 4 MB piece. The heavy-function cluster (old chunks 021-023) still takes 300-900 s per piece. - **A full build** with my local build changes together (this, the lift key's environment, link skips and others), each in a fresh player-like work folder: 36:17 on `main` against 32:30, at the same total compiler time (25,700 against 25,500 s) over 698 files instead of 486. The smaller pieces spread better over the cores. - **A two-PR update** (#2093 and #1930, the same setup): 6:06 against 4:36; compiling 2:16 against 1:00. ## In the game No change in play: the 4 MB-chunk game against the 24 MB one, Kokiri Forest from a save (the game logic bounds its frame on the CPU there), RTX 3080 (NVIDIA 621.14), Windows 11, two interleaved pairs of 30 samples: median 29.6 / 30.0 fps against 31.0 / 30.0. No errors in either. ## Testing - **`tests/test_sub_chunks.py` (new):** the first function past each stride; a long function spans strides without a boundary inside it; a function found since moves one boundary at most; chunks end at them as well as at the saved boundaries; `write` uses them and saves only `chunks.txt`'s; the stride makes chunks of a few MB. (The 1000-chunk guard isn't tested: the game's code makes ~330 pieces.) - **Every check that reads the changed files (9), through `run_all.py`, rebased on today's `main` (with #2748's spawned emit), Windows:** 8 pass; `test_spawn_emit` fails the same way on `main` here. - **Not tested:** Linux. See #2636. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

github-sync added 1 commit

2026-11-12 19:01:03 +00:00

perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries fc0a4363b4

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Commenting is not possible because the repository is archived.

No reviewers

No labels

No milestone

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Hookshot!2890

No description provided.

Rows
Columns