WIP: perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries

dyst commented

2026-11-12 23:38:46 +00:00

First-time contributor

Why

A small update that changes a few of the game's functions still recompiles every lifted chunk that holds one, and a chunk is ~24 MB of C: a typical one takes ~227 s of compiler time. Splitting the lifted C into ~4 MB pieces by guest address makes that ~41 s, with no cost in play or in a full build.

What changed

lift: sub_starts in nx2/chunks.py adds a chunk boundary every SUB_STRIDE (0x40000) of guest address, at the first function at or after it, besides image/chunks.txt's stable boundaries; write builds its Boundaries from both. They are guest addresses, so they stay put when a function grows: a change moves no boundary but its own chunk's.
lift: the sub-boundaries are not saved: chunks.txt keeps only the boundaries it had, and they are worked out again each lift from the same functions, so dropping this change later is clean.
lift: a guard in write: more than 1000 chunks (a code_NNNN.c) stops the lift with a message, since split_big, split_hot and build.py match three-digit chunk names.
tests: tests/test_sub_chunks.py (new): the first function past each stride; a long function spans strides without a boundary inside it; a function found since moves one boundary at most; chunks end at them as well as at the saved boundaries; write uses them and saves only chunks.txt's; the stride makes chunks of a few MB.

The first build after this compiles every lifted chunk once (all boundaries are new): for a player, that update takes about as long as a full build, once. Updates after it recompile only the ~4 MB pieces that changed.

Flags and settings

None.

Measurements

Measured on the GitHub version (NotRexed/Hookshot#2890), not yet on Longshot: this PR stays WIP until it is.

Compiles: Ryzen 9 3900X, Windows 11 (as the GitHub description states it).
In the game: RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11; Kokiri Forest from a save (the game logic bounds its frame on the CPU there); two interleaved pairs of 30 samples, the median of each.

	GitHub main	this PR (GitHub)
One chunk's compile (compiler time)	~227 s (a typical 24 MB chunk)	~41 s (a 4 MB piece)
In play, Kokiri Forest, median fps (pair 1 / pair 2)	31.0 / 30.0	29.6 / 30.0

The heavy-function cluster (old chunks 021-023) still takes 300-900 s per piece on this PR. No errors in either game run.

These two were measured with the author's other local build changes on the same side (this, the lift key's environment, link skips and others), each in a fresh player-like work folder, so they are not this PR alone:

	GitHub main	this PR with other local build changes (GitHub)
A full build (wall time)	36:17	32:30
Its total compiler time	25,700 s	25,500 s
Files compiled	486	698
A two-PR update (NotRexed/Hookshot#2093 and NotRexed/Hookshot#1930)	6:06	4:36
Of it, compiling	2:16	1:00

The smaller pieces spread better over the cores.

Testing

Tested: the runs under Measurements (on the GitHub version) and the game-free checks this description names.

Not tested: Linux. The 1000-chunk guard (the game's code makes ~330 pieces).

Needs other testers:

Linux: a whole build, expecting the game to start and play as on main with the ~4 MB chunks (request to open)

Screenshots

Nothing on screen.

Risk and rollback

The first build after this lands compiles every lifted chunk once, for players too. A stride that made more than 1000 chunks would stop the lift with a message rather than leave a chunk out. Nothing is saved to chunks.txt, so reverting the commit is enough.

Reviewer notes

Carried over from NotRexed/Hookshot#2890 with the move to Longshot: the same change on Longshot's main, with its rename applied. Its review and discussion are there.

Checks run on the GitHub version: every check that reads the changed files (9), through run_all.py, rebased on GitHub's main of that day (with NotRexed/Hookshot#2748's spawned emit), Windows: 8 pass; test_spawn_emit fails the same way on main there. Only those checks were run, not the whole suite. The checks and the game runs were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks; they were not run again here. Part of the dev-loop proposal's work on small updates (NotRexed/Hookshot#2636).

Issues

None on Longshot. See NotRexed/Hookshot#2636.

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags table filled in; README / docs/development.md switch reference updated.
Screenshots for a change that shows on screen.
A row in docs/tested-configurations.md for each measured run.
MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed).
The SPIR-V headers regenerated with glslang (a shader changed).
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
Docs updated.

🤖 Generated with Claude Code

## Why A small update that changes a few of the game's functions still recompiles every lifted chunk that holds one, and a chunk is ~24 MB of C: a typical one takes ~227 s of compiler time. Splitting the lifted C into ~4 MB pieces by guest address makes that ~41 s, with no cost in play or in a full build. ## What changed - lift: `sub_starts` in `nx2/chunks.py` adds a chunk boundary every `SUB_STRIDE` (0x40000) of guest address, at the first function at or after it, besides `image/chunks.txt`'s stable boundaries; `write` builds its `Boundaries` from both. They are guest addresses, so they stay put when a function grows: a change moves no boundary but its own chunk's. - lift: the sub-boundaries are not saved: `chunks.txt` keeps only the boundaries it had, and they are worked out again each lift from the same functions, so dropping this change later is clean. - lift: a guard in `write`: more than 1000 chunks (a `code_NNNN.c`) stops the lift with a message, since `split_big`, `split_hot` and `build.py` match three-digit chunk names. - tests: `tests/test_sub_chunks.py` (new): the first function past each stride; a long function spans strides without a boundary inside it; a function found since moves one boundary at most; chunks end at them as well as at the saved boundaries; `write` uses them and saves only `chunks.txt`'s; the stride makes chunks of a few MB. The first build after this compiles every lifted chunk once (all boundaries are new): for a player, that update takes about as long as a full build, once. Updates after it recompile only the ~4 MB pieces that changed. ## Flags and settings None. ## Measurements Measured on the GitHub version (NotRexed/Hookshot#2890), not yet on Longshot: this PR stays WIP until it is. Compiles: Ryzen 9 3900X, Windows 11 (as the GitHub description states it). In the game: RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11; Kokiri Forest from a save (the game logic bounds its frame on the CPU there); two interleaved pairs of 30 samples, the median of each. | | GitHub main | this PR (GitHub) | |---|---|---| | One chunk's compile (compiler time) | ~227 s (a typical 24 MB chunk) | ~41 s (a 4 MB piece) | | In play, Kokiri Forest, median fps (pair 1 / pair 2) | 31.0 / 30.0 | 29.6 / 30.0 | The heavy-function cluster (old chunks 021-023) still takes 300-900 s per piece on this PR. No errors in either game run. These two were measured with the author's other local build changes on the same side (this, the lift key's environment, link skips and others), each in a fresh player-like work folder, so they are not this PR alone: | | GitHub main | this PR with other local build changes (GitHub) | |---|---|---| | A full build (wall time) | 36:17 | 32:30 | | Its total compiler time | 25,700 s | 25,500 s | | Files compiled | 486 | 698 | | A two-PR update (NotRexed/Hookshot#2093 and NotRexed/Hookshot#1930) | 6:06 | 4:36 | | Of it, compiling | 2:16 | 1:00 | The smaller pieces spread better over the cores. ## Testing **Tested:** the runs under Measurements (on the GitHub version) and the game-free checks this description names. **Not tested:** Linux. The 1000-chunk guard (the game's code makes ~330 pieces). **Needs other testers:** - Linux: a whole build, expecting the game to start and play as on `main` with the ~4 MB chunks (request to open) ## Screenshots Nothing on screen. ## Risk and rollback The first build after this lands compiles every lifted chunk once, for players too. A stride that made more than 1000 chunks would stop the lift with a message rather than leave a chunk out. Nothing is saved to `chunks.txt`, so reverting the commit is enough. ## Reviewer notes Carried over from [NotRexed/Hookshot#2890](https://github.com/NotRexed/Hookshot/pull/2890) with the move to Longshot: the same change on Longshot's `main`, with its rename applied. Its review and discussion are there. Checks run on the GitHub version: every check that reads the changed files (9), through `run_all.py`, rebased on GitHub's `main` of that day (with [NotRexed/Hookshot#2748](https://github.com/NotRexed/Hookshot/pull/2748)'s spawned emit), Windows: 8 pass; `test_spawn_emit` fails the same way on `main` there. Only those checks were run, not the whole suite. The checks and the game runs were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks; they were not run again here. Part of the dev-loop proposal's work on small updates ([NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636)). ## Issues None on Longshot. See [NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] Flags table filled in; README / docs/development.md switch reference updated. - [ ] Screenshots for a change that shows on screen. - [ ] A row in `docs/tested-configurations.md` for each measured run. - [ ] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). - [ ] The SPIR-V headers regenerated with glslang (a shader changed). - [x] A game-free test added for the fix, failing on `main` and passing here. - [ ] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [ ] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit

2026-11-12 23:38:46 +00:00

perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries a16d681a7f

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst changed title from ~~perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries~~ to WIP: perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries

2026-11-13 01:13:54 +00:00

dyst commented

2026-11-13 01:13:55 +00:00

Author

First-time contributor

Back to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.

dyst commented

2026-11-13 01:58:20 +00:00

Author

First-time contributor

Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready.

dyst commented

2026-11-13 02:00:35 +00:00

Author

First-time contributor

Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'perf/smaller-chunks:dyst-perf/smaller-chunks'

git switch 'dyst-perf/smaller-chunks'

Rows
Columns

WIP: perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries #31

Why

What changed

Flags and settings

Measurements

Testing

Screenshots

Risk and rollback

Reviewer notes

Issues

Checklist

For everyone

If applicable

Checkout