WIP: perf(build): a renderer header edit compiles the files that read it, not the whole back end

dyst commented

2026-11-12 23:39:03 +00:00

First-time contributor

Why

Editing any renderer header recompiles the whole back end: on Windows a comment in vk_present_rect.h, which two files include, compiled 35 files again (417 s of compiler time; a 44 s build against 26 s for a .cpp edit, and minutes on a 4–8 thread PC).

What changed

build: nx2/build.py compiles the back end's own files (gpu/vk/*.cpp, not Eden's or sirit's) with -MD, and clang writes a depfile beside each object (vk_x.d for vk_x.o, depfile_of).
build: depfile_paths reads a make-style depfile (the target left out, line continuations joined, \ , $$ and \#). renderer_deps keys each object by the headers its depfile lists under the same folders as before (vdeps_roots: gpu/vk, gpu/render, gpu/compat, suyu's texture decoders), by name and content (_HEADER_DIGESTS). A header the file doesn't read no longer changes its key; one it reads, or one that's gone, does.
build: the folders' blob (tree_headers) stays as the fallback, for a file with no depfile yet or one whose depfile names nothing under those folders.
build: flag_text leaves -MD out of the key, so no key changes and nothing compiles again for this change itself; depfiles appear as files are compiled.
build: compile_one neither caches nor stamps an object under a key that came from the last compile's depfile (it hashes the new source with the old include list): it stamps it DEPFILE_PENDING. rekey_depfiles, run in main after the compile, stamps and caches each back-end object of this build (DEPFILE_JOBS) by the key its own new depfile gives, also when that is the planned key; an object stamped otherwise is another build's and is left alone. A build that stops before re-keying leaves the object pending, and the next build compiles it again. This answers hckrman101's review (an include added, removed, the header edited, the include back used to get the first object back from the cache).
build: unchanged: Eden's and sirit's files keep their own headers' blob, and the host keeps its blob (it mixes in non-header inputs such as the FFmpeg version key, so it is left for a separate change).
tests: tests/test_renderer_depfiles.py (new, 11 cases). tests/test_compiler_failure.py, tests/test_mem_gate_leases.py and tests/test_compile_retry_jobs.py take DEPFILE_DEPS and DEPFILE_PENDING with compile_one; tests/test_dlss_sdk_cache.py and tests/test_shader_stamp_cache.py run the job loop with its new helpers.

Flags and settings

None. (The back end's files are compiled with -MD now: a compiler flag inside the build, not a switch, and left out of object keys.)

Measurements

Measured on the GitHub version (NotRexed/Hookshot#2706), not yet on Longshot: this PR stays WIP until it is.

Hardware: Ryzen 9 3900X, NVMe, Windows
Scene: none (a build measurement)
Method: one build after a comment was added to gpu/vk/vk_present_rect.h, without depfiles and with them; then the header reverted

One comment in `vk_present_rect.h`	GitHub main	this PR (GitHub)
files compiled	35	4
compiler time	417 s	20 s
build	44 s	27 s

With the header reverted again (this PR only): 0 files, 0 s of compiler time, an 18 s build.
The four are the files that read it: vk_framegen.cpp, vk_menu.cpp, vk_replay.cpp, and host/hos_main.c, through the host's own blob.
Getting depfiles took one compile of each back-end file, done there by commenting them all. The re-keying then kept the reverted files from compiling a third time.
The "GitHub main" column is the original description's "before"; neither column's commit is named there.

In the game (a check, not a comparison): RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11, this branch's build.py (with the review fix) on GitHub's main ee29dd55. A build with depfiles, then one with a comment added to vk_present_rect.h: 5 files compiled (the four above, and vk_hangdump.cpp, which reads no repo header, so its depfile names nothing under those folders and it keeps the blob), no object left pending; with the comment taken out again, 0 files. That game, launched headless through the control bridge from a save: in play in Kokiri Forest, 27–34 fps over five samples, nothing in the log about a lost device or a crash.

Testing

Tested: the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured: the game on Windows (above): the change is build-only, and nothing in play was compared with main.

Not tested: Linux. gcc and clang write the same make-style depfile.

Needs other testers:

Linux: an edit to one gpu/vk header, expecting only the renderer files that include it compiled again (request to open)

Screenshots

Nothing on screen.

Risk and rollback

A back-end object keyed by too few headers would be taken from the cache though a header it reads changed: a stale object in the renderer. The pending stamp and rekey_depfiles cover a header read for the first time; the remaining limit is make's and ninja's: a newly added header that shadows another earlier in the include path isn't seen until something the file reads changes.
A file without a readable depfile falls back to the folders' blob, as before.
No switch: reverting the commit is enough (objects keyed by depfiles are only cache entries; the blob keys come back).

Reviewer notes

Carried over from NotRexed/Hookshot#2706 with the move to Longshot: the same change on Longshot's main, with its rename applied. Its review and discussion are there (reviewed by Joey-Magnus, jordanella).

The checks and the game run above were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks, and they were not run again here.
Checks run (on GitHub, not the whole run_all.py): every check that reads build.py (39), on Windows, passes apart from test_driver_cache_folder, test_fpx_sticky and test_linux_update, which fail the same way on main there.
tests/test_renderer_depfiles.py: the depfile read as clang 23 writes it on Windows (/ and \ in paths with spaces, CRLF continuations), plus $$ and \#; -MD in no key; no depfile keeps the blob; an unread header or one outside the folders changes nothing, while a read header or a gone one changes the key; the re-keying, and that it leaves alone an object not of this build; the wiring; and hckrman101's four steps, with a stand-in compiler that writes the headers it read into the object and a real depfile (on the previous commit step 4 got back step 2's object, built with the old b.h). It fails against main's build.py.
test_dlss_sdk_cache.py and test_shader_stamp_cache.py still take an older build.py (--build-script), and test_shader_stamp_cache passes against main's.
This is the dev-loop proposal's A6 (NotRexed/Hookshot#2636), validated by hckrman101's Linux numbers there (16 objects for one header).

Issues

None.

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags table filled in; README / docs/development.md switch reference updated.
Screenshots for a change that shows on screen.
A row in docs/tested-configurations.md for each measured run.
MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed).
The SPIR-V headers regenerated with glslang (a shader changed).
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
Docs updated.

🤖 Generated with Claude Code

## Why Editing any renderer header recompiles the whole back end: on Windows a comment in `vk_present_rect.h`, which two files include, compiled 35 files again (417 s of compiler time; a 44 s build against 26 s for a `.cpp` edit, and minutes on a 4–8 thread PC). ## What changed - build: `nx2/build.py` compiles the back end's own files (`gpu/vk/*.cpp`, not Eden's or sirit's) with `-MD`, and clang writes a depfile beside each object (`vk_x.d` for `vk_x.o`, `depfile_of`). - build: `depfile_paths` reads a make-style depfile (the target left out, line continuations joined, `\ `, `$$` and `\#`). `renderer_deps` keys each object by the headers its depfile lists under the same folders as before (`vdeps_roots`: `gpu/vk`, `gpu/render`, `gpu/compat`, suyu's texture decoders), by name and content (`_HEADER_DIGESTS`). A header the file doesn't read no longer changes its key; one it reads, or one that's gone, does. - build: the folders' blob (`tree_headers`) stays as the fallback, for a file with no depfile yet or one whose depfile names nothing under those folders. - build: `flag_text` leaves `-MD` out of the key, so no key changes and nothing compiles again for this change itself; depfiles appear as files are compiled. - build: `compile_one` neither caches nor stamps an object under a key that came from the last compile's depfile (it hashes the new source with the old include list): it stamps it `DEPFILE_PENDING`. `rekey_depfiles`, run in `main` after the compile, stamps and caches each back-end object of this build (`DEPFILE_JOBS`) by the key its own new depfile gives, also when that is the planned key; an object stamped otherwise is another build's and is left alone. A build that stops before re-keying leaves the object pending, and the next build compiles it again. This answers hckrman101's review (an include added, removed, the header edited, the include back used to get the first object back from the cache). - build: unchanged: Eden's and sirit's files keep their own headers' blob, and the host keeps its blob (it mixes in non-header inputs such as the FFmpeg version key, so it is left for a separate change). - tests: `tests/test_renderer_depfiles.py` (new, 11 cases). `tests/test_compiler_failure.py`, `tests/test_mem_gate_leases.py` and `tests/test_compile_retry_jobs.py` take `DEPFILE_DEPS` and `DEPFILE_PENDING` with `compile_one`; `tests/test_dlss_sdk_cache.py` and `tests/test_shader_stamp_cache.py` run the job loop with its new helpers. ## Flags and settings None. (The back end's files are compiled with `-MD` now: a compiler flag inside the build, not a switch, and left out of object keys.) ## Measurements Measured on the GitHub version (NotRexed/Hookshot#2706), not yet on Longshot: this PR stays WIP until it is. Hardware: Ryzen 9 3900X, NVMe, Windows Scene: none (a build measurement) Method: one build after a comment was added to `gpu/vk/vk_present_rect.h`, without depfiles and with them; then the header reverted | One comment in `vk_present_rect.h` | GitHub main | this PR (GitHub) | |---|---|---| | files compiled | 35 | 4 | | compiler time | 417 s | 20 s | | build | 44 s | 27 s | - With the header reverted again (this PR only): 0 files, 0 s of compiler time, an 18 s build. - The four are the files that read it: `vk_framegen.cpp`, `vk_menu.cpp`, `vk_replay.cpp`, and `host/hos_main.c`, through the host's own blob. - Getting depfiles took one compile of each back-end file, done there by commenting them all. The re-keying then kept the reverted files from compiling a third time. - The "GitHub main" column is the original description's "before"; neither column's commit is named there. In the game (a check, not a comparison): RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11, this branch's `build.py` (with the review fix) on GitHub's `main` ee29dd55. A build with depfiles, then one with a comment added to `vk_present_rect.h`: 5 files compiled (the four above, and `vk_hangdump.cpp`, which reads no repo header, so its depfile names nothing under those folders and it keeps the blob), no object left pending; with the comment taken out again, 0 files. That game, launched headless through the control bridge from a save: in play in Kokiri Forest, 27–34 fps over five samples, nothing in the log about a lost device or a crash. ## Testing **Tested:** the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured: the game on Windows (above): the change is build-only, and nothing in play was compared with `main`. **Not tested:** Linux. gcc and clang write the same make-style depfile. **Needs other testers:** - Linux: an edit to one `gpu/vk` header, expecting only the renderer files that include it compiled again (request to open) ## Screenshots Nothing on screen. ## Risk and rollback - A back-end object keyed by too few headers would be taken from the cache though a header it reads changed: a stale object in the renderer. The pending stamp and `rekey_depfiles` cover a header read for the first time; the remaining limit is make's and ninja's: a newly added header that shadows another earlier in the include path isn't seen until something the file reads changes. - A file without a readable depfile falls back to the folders' blob, as before. - No switch: reverting the commit is enough (objects keyed by depfiles are only cache entries; the blob keys come back). ## Reviewer notes Carried over from [NotRexed/Hookshot#2706](https://github.com/NotRexed/Hookshot/pull/2706) with the move to Longshot: the same change on Longshot's `main`, with its rename applied. Its review and discussion are there (reviewed by Joey-Magnus, jordanella). - The checks and the game run above were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks, and they were not run again here. - Checks run (on GitHub, not the whole `run_all.py`): every check that reads `build.py` (39), on Windows, passes apart from `test_driver_cache_folder`, `test_fpx_sticky` and `test_linux_update`, which fail the same way on `main` there. - `tests/test_renderer_depfiles.py`: the depfile read as clang 23 writes it on Windows (`/` and `\ ` in paths with spaces, CRLF continuations), plus `$$` and `\#`; `-MD` in no key; no depfile keeps the blob; an unread header or one outside the folders changes nothing, while a read header or a gone one changes the key; the re-keying, and that it leaves alone an object not of this build; the wiring; and hckrman101's four steps, with a stand-in compiler that writes the headers it read into the object and a real depfile (on the previous commit step 4 got back step 2's object, built with the old `b.h`). It fails against `main`'s `build.py`. - `test_dlss_sdk_cache.py` and `test_shader_stamp_cache.py` still take an older `build.py` (`--build-script`), and `test_shader_stamp_cache` passes against `main`'s. - This is the dev-loop proposal's A6 ([NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636)), validated by hckrman101's Linux numbers there (16 objects for one header). ## Issues None. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] Flags table filled in; README / docs/development.md switch reference updated. - [ ] Screenshots for a change that shows on screen. - [ ] A row in `docs/tested-configurations.md` for each measured run. - [ ] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). - [ ] The SPIR-V headers regenerated with glslang (a shader changed). - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [ ] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit

2026-11-12 23:39:03 +00:00

perf(build): a renderer header edit compiles the files that read it, not the whole back end 13d7223935

Carried over from github.com/jordanella/Hookshot perf/renderer-depfiles (its commits: perf(build): a renderer header edit compiles the files that read it, not the whole back end; fix(build): an object keyed by the last depfile is cached by its own depfile's key only). Merged with Longshot's
zstd object for the program cache (nx2/build.py), kept as it is after the renderer's jobs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst changed title from ~~perf(build): a renderer header edit compiles the files that read it, not the whole back end~~ to WIP: perf(build): a renderer header edit compiles the files that read it, not the whole back end

2026-11-13 01:14:00 +00:00

dyst commented

2026-11-13 01:14:01 +00:00

Author

First-time contributor

Back to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.

dyst commented

2026-11-13 01:58:27 +00:00

Author

First-time contributor

Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready.

dyst commented

2026-11-13 02:00:42 +00:00

Author

First-time contributor

Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'perf/renderer-depfiles:dyst-perf/renderer-depfiles'

git switch 'dyst-perf/renderer-depfiles'

Rows
Columns

WIP: perf(build): a renderer header edit compiles the files that read it, not the whole back end #37

Why

What changed

Flags and settings

Measurements

Testing

Screenshots

Risk and rollback

Reviewer notes

Issues

Checklist

For everyone

If applicable

Checkout