MagicBeanDev/Longshot

Watch

Fork 98

perf(build): a host header edit compiles the files that read it, not the whole host #673

Open

dyst wants to merge 5 commits from dyst/Longshot:perf/host-depfiles into main

dyst commented

2026-11-15 03:41:19 +00:00

Member

Type: perf · Risk: medium · Tested on: Windows 11, game-free checks · Needs: a timed build after a host header edit (Windows and Linux, any GPU)

Why

Every host object shares one header key, so an edit to any host/*.h compiles all ~55 host files again, even a comment in a header two of them include. The same key also leaves out gpu headers the host does read: it names only vk_api.h, dlss_math.h and longshot_env.h from gpu/, while host/nn_libc.c, nvn_layout.c and nvn_native.c include gpu/render/rx_nvn.h (and through it nvn_api.h), and host/display.c includes gpu/render/rx.h. An edit to one of those keeps the host objects built with the old header.

Builds on #37 (the renderer's half of the same change): review that first. This PR's own commit is the top one, 301586e8. It is the dev-loop proposal's A6 for the host (https://github.com/NotRexed/Hookshot/pull/2636). No Longshot issue fits (the tracker searched).

What changed

The host's files are keyed the way #37 keys the renderer's own files: each is compiled with -MD, and the next build names its object by the headers its depfile lists, so a host header edit compiles only the files that include it. A file with no depfile yet falls back to a wider key than before, one that now covers every header under gpu/ the host could read.

build: nx2/build.py main(): the host jobs get -MD, an entry in DEPFILE_JOBS, and renderer_deps(host_blob, host_roots, obj) as their headers. host_roots is host/, the lifted headers (gen, rt64), gpu/, opus's include folder and zstd's. host_blob, for a file with no depfile, is the lifted headers, every header under host/ (with host/posix on both systems), and every header under gpu/ but eden's, sirit's and gpu/third_party's. FFmpeg's version, dav1d's version.h and zstd's API for prompts.c and hos_zstd.c stay in the key as before.
build: rekey_depfiles(jobs) in place of rekey_depfiles(vk_jobs), after the compile and before the object store's share: jobs holds the back end's jobs too, and only the objects in DEPFILE_JOBS are keyed again.
docs: docs/development.md, Building, "What a build compiles again": the host's files are named by their depfiles too, and what the fallback covers for each.
tests: tests/test_host_depfiles.py (new): a host header compiles only its readers; a gpu header (rx_nvn.h) or a lifted header a file reads changes its key; the blob covers gpu/'s own headers and host/posix's but not eden's, sirit's or gpu/third_party's; no depfile keeps the blob; and the wiring in main(). tests/test_renderer_depfiles.py follows the rekey over jobs.

Measurements

Verdict: game-free: with this PR a host header edit changes the key of only the files that read it, and a gpu header a host file reads changes that file's key, where on #37's tree (and main) one key covers every host file and leaves those gpu headers out. The build times are not taken yet: they will be added here when the build under Must be tested is run, by me or by anyone who runs it first.

Hardware: Windows 11 Pro, Ryzen 9 3900X, 64 GB (the key needs no GPU)
Scene: none (a build change, game-free, no dump)
Method: tests/test_host_depfiles.py on a made-up host/ and gpu/ tree (two host files with clang-style depfiles: one reads hos.h, gpu/render/rx_nvn.h and a lifted header, the other other.h), each edit against the key before it; main's and #37's host key read in nx2/build.py main() (host_deps). A key is the same or not, so there is nothing to take a median of
How to reproduce: python -I -B tests/test_host_depfiles.py; for the Before column, host_deps in nx2/build.py at main @ 53f96d78

A build after	Before: main @ `53f96d78`	After: this PR @ `301586e8`
a comment in a `host/*.h` one file reads	every host file compiled again (one key for all)	only the files that read it
an edit to `gpu/render/rx_nvn.h`	no host file compiled again (not in the key)	the host files that read it (`nn_libc.c`, `nvn_layout.c`, `nvn_native.c`)
an edit to a lifted header a host file reads	every host file compiled again	the files that read it
a host file with no depfile yet (a first build)	the one key	every header under `host/`, the lifted ones and `gpu/`'s own

test_a_gpu_header_the_host_reads_changes_its_key ... ok
test_a_host_header_compiles_only_the_files_that_read_it ... ok
test_a_lifted_header_it_reads_changes_its_key ... ok
test_no_depfile_yet_keeps_the_blob ... ok
test_the_blob_covers_gpus_own_headers_but_not_eden_sirit_or_third_party ... ok
test_the_build_uses_them ... ok
Ran 6 tests
OK

Against #37's build.py (this PR's parent, 6631b629) the wiring case fails: the host's jobs have no -MD, no DEPFILE_JOBS entry and no renderer_deps.

Flags and settings

None: no switch, menu row, ini key or build flag changes for players (-MD writes a depfile beside each host object and changes nothing in it; flag_text leaves it out of the key, as #37 does).

Testing

Tested: tests/test_host_depfiles.py (6 cases) and tests/test_renderer_depfiles.py (11) pass, Windows 11. python -I -B tests/run_all.py on this branch and on main @ 53f96d78, with the Builder's llvm-mingw clang (12 at once): this branch 542 passed, 18 failed, 35 skipped; main 539 passed, 19 failed, 35 skipped; the 18 fail on both (test_appimage_player_build, test_cpm_discovery, test_cpp_preflight, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_readback_late, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_vram_budget_row, test_walk_key), and main's nineteenth was test_dlss_sdk_cache, which fails on this PC only some of the time. tests/test_third_party_objects.py (#629) passes here too.

Not tested: a real build: that clang writes the host's depfiles where renderer_deps reads them (it does for the back end's, #37) and that the second build compiles only the includers; Linux; a build after one of the host's object-store fetches (an object from the store has no depfile, so it keeps the blob until it is compiled in that folder, as #37's do).

Must be tested:

Windows, any GPU, by hand: anyone with a build tree and their own game files (I will run it too and add the times here). Build this PR twice (the first writes the depfiles), then add a comment to a host/*.h few files include and build again, expecting the build's @@file ... new lines to name only that header's includers, host/hos_main.c and gpu/vk/vk_menu.cpp (both carry the version), where main compiles every host file; then edit gpu/render/rx_nvn.h and build, expecting nn_libc.c, nvn_layout.c, nvn_native.c, hos_main.c and the renderer files that read it compiled again, where main compiles no host file. Then start the game once from a save, expecting it to reach play.
Linux, any GPU, gcc or clang, by hand: the same steps, expecting the same files (the host/posix files read their own headers too).

Risks

What could break: a host file that reads a header outside host_roots and outside the extras (a header the build adds to its include path that neither the depfile's folders nor the old key covered) would keep an old object after that header changes. The roots cover every folder our host code includes from (host/, the lifted headers, gpu/, opus, zstd), the extras keep FFmpeg's and dav1d's version, and the system's headers are covered by the flags and the system headers' stamp, as for every object. An include added to a source changes the source, so its key changes anyway, and an object keyed by an old depfile is compiled again before it is trusted (DEPFILE_PENDING, #37).

Turning it off: revert the commit (the host goes back to one key).

Compatibility: the host objects' keys change once (the blob is wider, and once a file has a depfile its key comes from it), so the next build compiles the host's ~55 files again: a minute or two, no translation. Saves, the options file, the shader and pipeline caches and the vk ABI are unchanged.

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main.
python -I -B tests/run_all.py passes (any failure that also fails on main is named).
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Docs updated.

🤖 Generated with Claude Code

**Type:** perf · **Risk:** medium · **Tested on:** Windows 11, game-free checks · **Needs:** a timed build after a host header edit (Windows and Linux, any GPU) ## Why Every host object shares one header key, so an edit to any `host/*.h` compiles all ~55 host files again, even a comment in a header two of them include. The same key also leaves out gpu headers the host does read: it names only `vk_api.h`, `dlss_math.h` and `longshot_env.h` from `gpu/`, while `host/nn_libc.c`, `nvn_layout.c` and `nvn_native.c` include `gpu/render/rx_nvn.h` (and through it `nvn_api.h`), and `host/display.c` includes `gpu/render/rx.h`. An edit to one of those keeps the host objects built with the old header. Builds on https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/37 (the renderer's half of the same change): review that first. This PR's own commit is the top one, 301586e8. It is the dev-loop proposal's A6 for the host (https://github.com/NotRexed/Hookshot/pull/2636). No Longshot issue fits (the tracker searched). ## What changed The host's files are keyed the way #37 keys the renderer's own files: each is compiled with `-MD`, and the next build names its object by the headers its depfile lists, so a host header edit compiles only the files that include it. A file with no depfile yet falls back to a wider key than before, one that now covers every header under `gpu/` the host could read. - build: `nx2/build.py` `main()`: the host jobs get `-MD`, an entry in `DEPFILE_JOBS`, and `renderer_deps(host_blob, host_roots, obj)` as their headers. `host_roots` is `host/`, the lifted headers (`gen`, `rt64`), `gpu/`, opus's include folder and zstd's. `host_blob`, for a file with no depfile, is the lifted headers, every header under `host/` (with `host/posix` on both systems), and every header under `gpu/` but eden's, sirit's and `gpu/third_party`'s. FFmpeg's version, dav1d's `version.h` and zstd's API for `prompts.c` and `hos_zstd.c` stay in the key as before. - build: `rekey_depfiles(jobs)` in place of `rekey_depfiles(vk_jobs)`, after the compile and before the object store's share: `jobs` holds the back end's jobs too, and only the objects in `DEPFILE_JOBS` are keyed again. - docs: `docs/development.md`, *Building*, "What a build compiles again": the host's files are named by their depfiles too, and what the fallback covers for each. - tests: `tests/test_host_depfiles.py` (new): a host header compiles only its readers; a gpu header (`rx_nvn.h`) or a lifted header a file reads changes its key; the blob covers `gpu/`'s own headers and `host/posix`'s but not eden's, sirit's or `gpu/third_party`'s; no depfile keeps the blob; and the wiring in `main()`. `tests/test_renderer_depfiles.py` follows the rekey over `jobs`. ## Measurements **Verdict:** game-free: with this PR a host header edit changes the key of only the files that read it, and a gpu header a host file reads changes that file's key, where on #37's tree (and main) one key covers every host file and leaves those gpu headers out. The build times are not taken yet: they will be added here when the build under Must be tested is run, by me or by anyone who runs it first. Hardware: Windows 11 Pro, Ryzen 9 3900X, 64 GB (the key needs no GPU) Scene: none (a build change, game-free, no dump) Method: `tests/test_host_depfiles.py` on a made-up `host/` and `gpu/` tree (two host files with clang-style depfiles: one reads `hos.h`, `gpu/render/rx_nvn.h` and a lifted header, the other `other.h`), each edit against the key before it; main's and #37's host key read in `nx2/build.py` `main()` (`host_deps`). A key is the same or not, so there is nothing to take a median of How to reproduce: `python -I -B tests/test_host_depfiles.py`; for the Before column, `host_deps` in `nx2/build.py` at main @ 53f96d78 | A build after | Before: main @ 53f96d78 | After: this PR @ 301586e8 | |---|---|---| | a comment in a `host/*.h` one file reads | every host file compiled again (one key for all) | only the files that read it | | an edit to `gpu/render/rx_nvn.h` | no host file compiled again (not in the key) | the host files that read it (`nn_libc.c`, `nvn_layout.c`, `nvn_native.c`) | | an edit to a lifted header a host file reads | every host file compiled again | the files that read it | | a host file with no depfile yet (a first build) | the one key | every header under `host/`, the lifted ones and `gpu/`'s own | ``` test_a_gpu_header_the_host_reads_changes_its_key ... ok test_a_host_header_compiles_only_the_files_that_read_it ... ok test_a_lifted_header_it_reads_changes_its_key ... ok test_no_depfile_yet_keeps_the_blob ... ok test_the_blob_covers_gpus_own_headers_but_not_eden_sirit_or_third_party ... ok test_the_build_uses_them ... ok Ran 6 tests OK ``` Against #37's `build.py` (this PR's parent, 6631b629) the wiring case fails: the host's jobs have no `-MD`, no `DEPFILE_JOBS` entry and no `renderer_deps`. ### Flags and settings None: no switch, menu row, ini key or build flag changes for players (`-MD` writes a depfile beside each host object and changes nothing in it; `flag_text` leaves it out of the key, as #37 does). ## Testing **Tested:** `tests/test_host_depfiles.py` (6 cases) and `tests/test_renderer_depfiles.py` (11) pass, Windows 11. `python -I -B tests/run_all.py` on this branch and on main @ 53f96d78, with the Builder's llvm-mingw clang (12 at once): this branch 542 passed, 18 failed, 35 skipped; main 539 passed, 19 failed, 35 skipped; the 18 fail on both (test_appimage_player_build, test_cpm_discovery, test_cpp_preflight, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_readback_late, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_vram_budget_row, test_walk_key), and main's nineteenth was test_dlss_sdk_cache, which fails on this PC only some of the time. `tests/test_third_party_objects.py` (#629) passes here too. **Not tested:** a real build: that clang writes the host's depfiles where `renderer_deps` reads them (it does for the back end's, #37) and that the second build compiles only the includers; Linux; a build after one of the host's object-store fetches (an object from the store has no depfile, so it keeps the blob until it is compiled in that folder, as #37's do). **Must be tested:** - Windows, any GPU, by hand: anyone with a build tree and their own game files (I will run it too and add the times here). Build this PR twice (the first writes the depfiles), then add a comment to a `host/*.h` few files include and build again, expecting the build's `@@file ... new` lines to name only that header's includers, `host/hos_main.c` and `gpu/vk/vk_menu.cpp` (both carry the version), where main compiles every host file; then edit `gpu/render/rx_nvn.h` and build, expecting `nn_libc.c`, `nvn_layout.c`, `nvn_native.c`, `hos_main.c` and the renderer files that read it compiled again, where main compiles no host file. Then start the game once from a save, expecting it to reach play. - Linux, any GPU, gcc or clang, by hand: the same steps, expecting the same files (the `host/posix` files read their own headers too). ## Risks **What could break:** a host file that reads a header outside `host_roots` and outside the extras (a header the build adds to its include path that neither the depfile's folders nor the old key covered) would keep an old object after that header changes. The roots cover every folder our host code includes from (`host/`, the lifted headers, `gpu/`, opus, zstd), the extras keep FFmpeg's and dav1d's version, and the system's headers are covered by the flags and the system headers' stamp, as for every object. An include added to a source changes the source, so its key changes anyway, and an object keyed by an old depfile is compiled again before it is trusted (`DEPFILE_PENDING`, #37). **Turning it off:** revert the commit (the host goes back to one key). **Compatibility:** the host objects' keys change once (the blob is wider, and once a file has a depfile its key comes from it), so the next build compiles the host's ~55 files again: a minute or two, no translation. Saves, the options file, the shader and pipeline caches and the vk ABI are unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 5 commits

2026-11-15 03:41:19 +00:00

perf(build): a renderer header edit compiles the files that read it, not the whole back end 1e830adcd1

Carried over from github.com/jordanella/Hookshot perf/renderer-depfiles (its commits: perf(build): a renderer header edit compiles the files that read it, not the whole back end; fix(build): an object keyed by the last depfile is cached by its own depfile's key only). Merged with Longshot's
zstd object for the program cache (nx2/build.py), kept as it is after the renderer's jobs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

test(build): the depfile check reads the renderer's flags with main's play profile, and the store after the re-keying 61d6597113

Rebased onto main: the back end's own files now take vk_pgo_flags (#569)
in the same "if lib else" as -MD, so the check matches that branch
instead of its old text. The objects rekey_depfiles caches by their
depfile's key go to the object store (#371) with the rest: the check
holds that order.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs(development): what a build compiles again, a renderer header's includers by their depfiles ec2d2c5849

Building says how objects are named, that the back end's own files are named by the headers their depfile read, the whole-folder fallback (the first build, an object from the store), and why vk_menu.cpp and host/hos_main.c compile again on any source change.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

build: -MD beside #629 's third-party flags, so its check keeps reading them

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 13s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 20s

Details

Third-party parts / As their notes say (pull_request) Successful in 33s

Details

PR description / Description follows the template (pull_request_target) Successful in 8s

Details

6de3811742

Rebased onto main d55e3bf7: since #629 eden and sirit take tp_vflags and tp_deps. The back end's own files add -MD as a term of their own after (tp_vflags if lib else vflags + vk_pgo_flags), the text tests/test_third_party_objects.py reads, and test_renderer_depfiles.py checks both lines as they now read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

perf(build): a host header edit compiles the files that read it, not the whole host

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 5s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

64bd6b9789

The host's objects all shared one header key: every host/*.h (and
host/posix's on Linux), the lifted headers, and three gpu headers. So a
host/*.h edit compiled every host file again, and an edit to a gpu header
the host reads that was not one of the three (gpu/render/rx.h, rx_nvn.h,
gpu/vk/oot_cheat_items.h) compiled none of them, keeping objects built
with the old header.

The host's files now go the way the back end's own files went in #37:
compiled with -MD, named by the headers their depfile lists under host/,
the lifted headers, gpu/, opus's and zstd's (renderer_deps), and keyed
again after the compile (rekey_depfiles, over all of jobs now). A file
with no depfile yet is named by host_blob: every header under host/, the
lifted ones, and gpu/'s own (not eden's, sirit's or gpu/third_party's).
FFmpeg's, dav1d's and zstd's API stay in the key as before.

tests/test_host_depfiles.py (new): a host header compiles only its
readers, a gpu or lifted header the file reads changes its key, the
blob's reach, and the wiring in main(). test_renderer_depfiles.py follows
the rekey over jobs. docs/development.md's "What a build compiles again"
covers the host.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst commented

2026-11-15 03:41:26 +00:00

Author

Member

Ready for review and independent testing now: the measurement here is game-free (Measurements), and the timed builds under Must be tested are not run yet. I will run them and edit this description with the numbers when they are done; anyone with a build tree and their own game files is welcome to run them first and post the results here. Stacked on #37: its commits are the first four here, this PR's own is the top one, 64bd6b97.

dyst changed title from ~~WIP: perf(build): a host header edit compiles the files that read it, not the whole host~~ to perf(build): a host header edit compiles the files that read it, not the whole host

2026-11-15 03:41:27 +00:00

dyst force-pushed perf/host-depfiles from 64bd6b9789

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 5s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

to 301586e8d1

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 13s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 18s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 19s

Details

Third-party parts / As their notes say (pull_request) Successful in 30s

Details

PR description / Description follows the template (pull_request_target) Successful in 14s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

2026-11-15 04:23:50 +00:00

Compare

dyst referenced this pull request

2026-11-15 04:23:50 +00:00

perf(build): a renderer header edit compiles the files that read it, not the whole back end #37

dyst commented

2026-11-15 04:23:53 +00:00

Author

Member

Moved onto #37's rebase (main 53f96d78); this PR's own commit is now 301586e8, unchanged. The full suite on this branch and on main again: no failure main doesn't have (Testing).

Moved onto #37's rebase (main 53f96d78); this PR's own commit is now 301586e8, unchanged. The full suite on this branch and on main again: no failure main doesn't have (Testing).