WIP: perf(build): skip unchanged links, write compile_commands.json, key the hot pieces by the profile's contents

dyst commented

2026-11-12 23:38:44 +00:00

First-time contributor

Why

A build with nothing to compile still relinked the 440 MB executable and the renderer, a build from a second checkout of the same tree recompiled all eight hot pieces (~10-20 minutes) because their keys named the play profile by its path, and clangd had no compile database.

What changed

build: run_link in nx2/build.py skips a link whose inputs are the last successful one's. link_key hashes the command, the compiler (its size and time) and the work folder's system_headers.txt; each object's key (its .o.sha1, or its size and time without one); and the contents of the other files the command names, inside options too (link_inputs: the @ response files, the hot symbol order, the libraries, the resources; not the output or a .map). The key is kept beside the output as <output>.linkkey with the output's size; a failed link, or an output of another size or gone, links again. The three links in main go through it (the executable's, the renderer's and the game DLL's). (A4)
build: write_compdb writes every compile as compile_one runs it to the build folder's compile_commands.json, for clangd and clang-tidy; a failure to write it does not fail the build. .gitignore ignores /compile_commands.json at the checkout's top, for a copy there. (A8)
build: flag_text leaves the path of -fprofile-instr-use= out of the hot pieces' object keys, since -DLONGSHOT_PROFILE_SHA1 beside it already says what the key needs of the profile. This changes their keys once.
build: nx2/build.py imports re (GitHub's main didn't; NotRexed/Hookshot#2706 adds it too).
tests: tests/test_link_skip.py (new) runs a stand-in linker through the real run_link: the same inputs aren't linked again; a changed object, library or symbol order is; so is an output of another size or gone; a failed link forgets its key; the files inside options are inputs; the compile commands are the compiles; the build's links go through it.
tests: tests/test_profile_path_key.py (new): two paths, one profile give one key; another profile, another key; every use of the profile's path names its contents too.

Flags and settings

None.

Measurements

Measured on the GitHub version (NotRexed/Hookshot#2893), not yet on Longshot: this PR stays WIP until it is.

Hardware: Ryzen 9 3900X, Windows 11 (as the GitHub description states it), a dev work folder.
Method: the GitHub description gives one figure for each side and does not say how many runs.

	GitHub main	this PR (GitHub)
A build with nothing to do	16 s	6 s, both links skipped
A second checkout of the same tree, building in the same work folder	the eight hot pieces compiled again	the eight hot pieces from the cache

One renderer edit, on this PR: only the renderer's link runs (no figure for main).

Testing

Tested: the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured: every build the author ran in the game that day used this build.py. The latest: Windows 11, RTX 3080 (NVIDIA 621.14), a dev-profile build, from a save, headless through the control bridge: in play in Kokiri Forest, 28-31 fps, nothing in the log about a lost device or a crash. A skipped link leaves the same binary in place.

Not tested: Linux.

Needs other testers:

Linux: a second build with nothing changed, expecting the links skipped and compile_commands.json written (request to open)

Screenshots

Nothing on screen.

Risk and rollback

A link that should have run but was skipped would leave an old binary in place: the key covers the command, the compiler, each object's key and the files the command names, and deleting <output>.linkkey makes the next build link again. The hot pieces' new keys compile the eight hot pieces once after this lands. Reverting the commit is enough.

Reviewer notes

Carried over from NotRexed/Hookshot#2893 with the move to Longshot: the same change on Longshot's main, with its rename applied. Its review and discussion are there.

Three small changes to nx2/build.py from the dev-loop proposal (NotRexed/Hookshot#2636, A4 and A8), in one PR because they touch the same lines; they can be split if preferred.

Checks run on the GitHub version: every check that reads the changed files (37), through run_all.py, rebased on GitHub's main of that day, Windows: pass, apart from four that fail the same way on main there (test_driver_cache_folder, test_fpx_sticky, test_linux_update, nx2/test_dll.py). Only those checks were run, not the whole suite. The checks and the game runs were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks; they were not run again here.

Issues

None on Longshot. See NotRexed/Hookshot#2636 (A4, A8).

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags table filled in; README / docs/development.md switch reference updated.
Screenshots for a change that shows on screen.
A row in docs/tested-configurations.md for each measured run.
MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed).
The SPIR-V headers regenerated with glslang (a shader changed).
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
Docs updated.

🤖 Generated with Claude Code

## Why A build with nothing to compile still relinked the 440 MB executable and the renderer, a build from a second checkout of the same tree recompiled all eight hot pieces (~10-20 minutes) because their keys named the play profile by its path, and clangd had no compile database. ## What changed - build: `run_link` in `nx2/build.py` skips a link whose inputs are the last successful one's. `link_key` hashes the command, the compiler (its size and time) and the work folder's `system_headers.txt`; each object's key (its `.o.sha1`, or its size and time without one); and the contents of the other files the command names, inside options too (`link_inputs`: the `@` response files, the hot symbol order, the libraries, the resources; not the output or a `.map`). The key is kept beside the output as `<output>.linkkey` with the output's size; a failed link, or an output of another size or gone, links again. The three links in `main` go through it (the executable's, the renderer's and the game DLL's). (A4) - build: `write_compdb` writes every compile as `compile_one` runs it to the build folder's `compile_commands.json`, for clangd and clang-tidy; a failure to write it does not fail the build. `.gitignore` ignores `/compile_commands.json` at the checkout's top, for a copy there. (A8) - build: `flag_text` leaves the path of `-fprofile-instr-use=` out of the hot pieces' object keys, since `-DLONGSHOT_PROFILE_SHA1` beside it already says what the key needs of the profile. This changes their keys once. - build: `nx2/build.py` imports `re` (GitHub's `main` didn't; NotRexed/Hookshot#2706 adds it too). - tests: `tests/test_link_skip.py` (new) runs a stand-in linker through the real `run_link`: the same inputs aren't linked again; a changed object, library or symbol order is; so is an output of another size or gone; a failed link forgets its key; the files inside options are inputs; the compile commands are the compiles; the build's links go through it. - tests: `tests/test_profile_path_key.py` (new): two paths, one profile give one key; another profile, another key; every use of the profile's path names its contents too. ## Flags and settings None. ## Measurements Measured on the GitHub version (NotRexed/Hookshot#2893), not yet on Longshot: this PR stays WIP until it is. Hardware: Ryzen 9 3900X, Windows 11 (as the GitHub description states it), a dev work folder. Method: the GitHub description gives one figure for each side and does not say how many runs. | | GitHub main | this PR (GitHub) | |---|---|---| | A build with nothing to do | 16 s | 6 s, both links skipped | | A second checkout of the same tree, building in the same work folder | the eight hot pieces compiled again | the eight hot pieces from the cache | One renderer edit, on this PR: only the renderer's link runs (no figure for `main`). ## Testing **Tested:** the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured: every build the author ran in the game that day used this `build.py`. The latest: Windows 11, RTX 3080 (NVIDIA 621.14), a dev-profile build, from a save, headless through the control bridge: in play in Kokiri Forest, 28-31 fps, nothing in the log about a lost device or a crash. A skipped link leaves the same binary in place. **Not tested:** Linux. **Needs other testers:** - Linux: a second build with nothing changed, expecting the links skipped and `compile_commands.json` written (request to open) ## Screenshots Nothing on screen. ## Risk and rollback A link that should have run but was skipped would leave an old binary in place: the key covers the command, the compiler, each object's key and the files the command names, and deleting `<output>.linkkey` makes the next build link again. The hot pieces' new keys compile the eight hot pieces once after this lands. Reverting the commit is enough. ## Reviewer notes Carried over from [NotRexed/Hookshot#2893](https://github.com/NotRexed/Hookshot/pull/2893) with the move to Longshot: the same change on Longshot's `main`, with its rename applied. Its review and discussion are there. Three small changes to `nx2/build.py` from the dev-loop proposal ([NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636), A4 and A8), in one PR because they touch the same lines; they can be split if preferred. Checks run on the GitHub version: every check that reads the changed files (37), through `run_all.py`, rebased on GitHub's `main` of that day, Windows: pass, apart from four that fail the same way on `main` there (`test_driver_cache_folder`, `test_fpx_sticky`, `test_linux_update`, `nx2/test_dll.py`). Only those checks were run, not the whole suite. The checks and the game runs were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks; they were not run again here. ## Issues None on Longshot. See [NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636) (A4, A8). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] Flags table filled in; README / docs/development.md switch reference updated. - [ ] Screenshots for a change that shows on screen. - [ ] A row in `docs/tested-configurations.md` for each measured run. - [ ] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). - [ ] The SPIR-V headers regenerated with glslang (a shader changed). - [x] A game-free test added for the fix, failing on `main` and passing here. - [ ] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [ ] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit

2026-11-12 23:38:44 +00:00

perf(build): skip unchanged links, write compile_commands.json, key the hot pieces by the profile's contents eb8ca08fce

Three changes to nx2/build.py:

- A4: a link whose inputs are the last successful one's is not run again
  (run_link). Its key, beside the output as <output>.linkkey: the command,
  the compiler and the work folder's system-headers stamp, each object's
  key (.o.sha1), and the contents of the other files it names, inside
  options too (the hot symbol order). A build with nothing to do went from
  16 s to 6 s on Windows. Players' builds let sa/build go, so it helps a
  work folder that keeps its files (the checkout's work/,
  LONGSHOT_KEEP_BUILD_FILES=1).
- A8: <build>/compile_commands.json, every compile as compile_one runs it,
  for clangd and clang-tidy (git-ignored at the checkout's top).
- The hot pieces' keys name the play profile by its contents
  (-DLONGSHOT_PROFILE_SHA1), not its path: a build from a checkout other
  than the work folder's compiled all eight again (~10-20 minutes). This
  changes their keys once.

tests/test_link_skip.py (a stand-in linker: skipped, relinked on a changed
object, library, symbol order or output, a failed link forgets the key;
the compile commands) and tests/test_profile_path_key.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst changed title from ~~perf(build): skip unchanged links, write compile_commands.json, key the hot pieces by the profile's contents~~ to WIP: perf(build): skip unchanged links, write compile_commands.json, key the hot pieces by the profile's contents

2026-11-13 01:13:53 +00:00

dyst commented

2026-11-13 01:13:53 +00:00

Author

First-time contributor

Back to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.

dyst commented

2026-11-13 01:58:19 +00:00

Author

First-time contributor

Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready.

dyst commented

2026-11-13 02:00:34 +00:00

Author

First-time contributor

Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'perf/link-skip:dyst-perf/link-skip'

git switch 'dyst-perf/link-skip'

Rows
Columns

WIP: perf(build): skip unchanged links, write compile_commands.json, key the hot pieces by the profile's contents #30

Why

What changed

Flags and settings

Measurements

Testing

Screenshots

Risk and rollback

Reviewer notes

Issues

Checklist

For everyone

If applicable

Checkout