WIP: feat(build): LONGSHOT_OBJ_STORE, one object cache for every work folder

dyst commented

2026-11-12 23:38:28 +00:00

First-time contributor

Why

A worktree for a PR compiled the whole game again (about 600 files, most of a 30-minute build on Windows), although another work folder on the same PC had compiled the same sources with the same flags already. With LONGSHOT_OBJ_STORE=<folder>, every work folder shares one object cache, so each object is compiled once on the PC.

What changed

Opt-in: unset, nothing changes (<work>/objcache as before).

build: object_cache() in nx2/make_game.py points build.py at <store>/objects/<toolchain stamp>:
- an object is already named by its source, flags and headers (object_key), with no work folder's path in it;
- the compiler and the system headers are not part of the key: a work folder throws its own cache away when they change (drop_stale_objects), so the store is split by that stamp (system_headers.txt) instead, one part per toolchain. Without a stamp, the work folder's own cache is used;
- only build.py's runs get the folder (LONGSHOT_OBJ_CACHE, in build_env in main), so the lift's environment, and its key, are unchanged. nx2/build.py's OBJ_CACHE reads it, else <work>/objcache.
build: cache_put() in nx2/build.py puts an object in whole or not at all: it links or copies the object to a temporary of its own (<key>.o.<pid>-<thread>.tmp), then renames it onto the key. Another build reading the store never finds part of one, a build killed meanwhile leaves no entry, and two builds putting the same object leave one. This also helps without a store: a copy that failed halfway used to leave a partial object in the cache, and raised.
build: the work folder's clean-ups (prune_object_cache, forget_failed_build, the system-header check) touch only <work>/objcache, never the store.
build: adopt_objects(): the first build with the store puts the work folder's own objects into it, so switching it on compiles nothing again.
build: seed_work() (in place of main's copy loop): the git-ignored lists a lift reads (hot_funcs.txt, hot_order.txt, longshot_pieces.prof, longshot.res, image/chunks.txt) come from <store>/seed when this checkout's work/ has none, so a new worktree's chunk boundaries and hot pieces match, and so do its objects' keys. The store's copy follows the checkout that has them.
build: the end-of-build message says the objects are in the shared store when one is set.
tests: new tests/test_obj_store.py (no compiler or game); tests/test_compiler_failure.py (the cut cache_put's names: shutil, threading) and tests/test_failed_compile_keeps_objects.py (the pinned build.py run now takes build_env) updated.

Not shared yet: the translation itself. A new worktree still translates the game (about 2:40, below); its C is the same, so its objects come from the store. The store is never pruned; it can be deleted at any time.

Flags and settings

Flag	New or changed	What it does	Default now	Goal: on by default?	Helps on	Doesn't help on	Untested on	Untested, but should help on
`LONGSHOT_OBJ_STORE`	new	`<folder>`: one compiled-object cache shared by every work folder (split by toolchain stamp), plus the lift's seed lists	unset (`<work>/objcache`, as before)	no (a developer switch)	Windows 11, llvm-mingw, store and worktrees on one NTFS drive: a new worktree compiled 0 of 484 files	-	Linux	Linux (the same code; hard links work there as on NTFS)
`LONGSHOT_OBJ_CACHE`	new	the folder `build.py` keeps its compiled objects in; `make_game.py` sets it for `build.py`'s runs only (from `LONGSHOT_OBJ_STORE`, else `<work>/objcache`)	unset when `build.py` runs alone (`<work>/objcache`)	no (set by `make_game.py`, not by hand)	-	-	Linux	-
`LONGSHOT_OBJ`	new (test only)	test only: `tests/test_obj_store.py` checks that `lift_key` names no `LONGSHOT_OBJ*` variable	-	no (a developer switch)	-	-	-	-
`RX_RELEASE`	unchanged (its line in `main()` is rewritten as it was)	the release build's switch, as before	as before	no (a developer switch)	-	-	-	-

Measurements

Measured on the GitHub version (NotRexed/Hookshot#2974), not yet on Longshot: this PR stays WIP until it is.

Hardware: RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11; the Builder's tools (llvm-mingw), store and worktrees on one NTFS drive.
Method: the main figure is what such a build took before this change, a full build on the same PC, not a run made beside these.

	GitHub main	this PR (GitHub)
A brand-new worktree, work folder empty: files compiled	a full build	0 (484 of 484 objects from the store; compile step 15 s)
A brand-new worktree, work folder empty: total	32-36 minutes	4:17
A worktree with a work folder, the store switched on (its 1,942 objects put in): files compiled, total	-	3 files (what changed), 4:15

The rest of the new worktree's 4:17 is the translation (about 2:40), not shared yet.
Its hot lists and chunk boundaries came from the store's seed, so its lifted files' keys matched.
Its game plays: Kokiri Forest from a save, 27-38 fps, no errors in the log.

Checks: every check that reads the changed files (76), through run_all.py with the Builder's tools, Windows: 71 pass. nx2/test_dll.py, test_driver_cache_folder, test_fpx_sticky and test_linux_update fail the same way on main there. test_dlss_sdk_cache failed once in the parallel run and passes alone, twice (Defender holds the exe it compiles, now and then).

Testing

Tested: the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured: the new worktree's game in play (Kokiri Forest, above; no main run beside it).

Not tested: Linux (the same code; hard links work there as on NTFS).

Needs other testers:

Linux: a second worktree built with LONGSHOT_OBJ_STORE set, expecting none of the objects the first compiled compiled again (request to open)

Screenshots

Nothing on screen.

Risk and rollback

Unset, nothing changes except cache_put, which now renames a temporary onto the key: a failed put is skipped where it used to raise. With the store set, every work folder links the store's object for a key, so a bad object there reaches each of them, and a new worktree takes the seed lists another checkout left there; the store is never pruned, so it grows. Unsetting LONGSHOT_OBJ_STORE turns it off (the store folder can be deleted at any time); reverting the commit is enough.

Reviewer notes

Carried over from NotRexed/Hookshot#2974 with the move to Longshot: the same change on Longshot's main, with its rename applied. Its review and discussion are there.

The dev-loop proposal's A10 (NotRexed/Hookshot#2636); hckrman101 offered it in the review and is away, so this takes it on.

tests/test_obj_store.py covers: two work folders over one store compile the same sources through build.py's compile_one with a stand-in compiler (the second compiles none, and its objects are the store's, the same file); a changed source compiles alone; a copy that stops halfway leaves no entry and no temporary (main's cache_put fails this); eight threads putting one object leave one; the store's part by stamp, and the fallbacks; adoption, and the seeds both ways; the lift's environment never gets the store's folder.

The checks and the in-game runs above were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks, and they were not run again here. Only the checks that read the changed files were run, not the whole run_all.py.

Issues

See NotRexed/Hookshot#2636 (A10).

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags table filled in; README / docs/development.md switch reference updated.
Screenshots for a change that shows on screen.
A row in docs/tested-configurations.md for each measured run.
MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed).
The SPIR-V headers regenerated with glslang (a shader changed).
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
Docs updated.

🤖 Generated with Claude Code

## Why A worktree for a PR compiled the whole game again (about 600 files, most of a 30-minute build on Windows), although another work folder on the same PC had compiled the same sources with the same flags already. With `LONGSHOT_OBJ_STORE=<folder>`, every work folder shares one object cache, so each object is compiled once on the PC. ## What changed Opt-in: unset, nothing changes (`<work>/objcache` as before). - build: `object_cache()` in `nx2/make_game.py` points `build.py` at `<store>/objects/<toolchain stamp>`: - an object is already named by its source, flags and headers (`object_key`), with no work folder's path in it; - the compiler and the system headers are not part of the key: a work folder throws its own cache away when they change (`drop_stale_objects`), so the store is split by that stamp (`system_headers.txt`) instead, one part per toolchain. Without a stamp, the work folder's own cache is used; - only `build.py`'s runs get the folder (`LONGSHOT_OBJ_CACHE`, in `build_env` in `main`), so the lift's environment, and its key, are unchanged. `nx2/build.py`'s `OBJ_CACHE` reads it, else `<work>/objcache`. - build: `cache_put()` in `nx2/build.py` puts an object in whole or not at all: it links or copies the object to a temporary of its own (`<key>.o.<pid>-<thread>.tmp`), then renames it onto the key. Another build reading the store never finds part of one, a build killed meanwhile leaves no entry, and two builds putting the same object leave one. This also helps without a store: a copy that failed halfway used to leave a partial object in the cache, and raised. - build: the work folder's clean-ups (`prune_object_cache`, `forget_failed_build`, the system-header check) touch only `<work>/objcache`, never the store. - build: `adopt_objects()`: the first build with the store puts the work folder's own objects into it, so switching it on compiles nothing again. - build: `seed_work()` (in place of `main`'s copy loop): the git-ignored lists a lift reads (`hot_funcs.txt`, `hot_order.txt`, `longshot_pieces.prof`, `longshot.res`, `image/chunks.txt`) come from `<store>/seed` when this checkout's `work/` has none, so a new worktree's chunk boundaries and hot pieces match, and so do its objects' keys. The store's copy follows the checkout that has them. - build: the end-of-build message says the objects are in the shared store when one is set. - tests: new `tests/test_obj_store.py` (no compiler or game); `tests/test_compiler_failure.py` (the cut `cache_put`'s names: `shutil`, `threading`) and `tests/test_failed_compile_keeps_objects.py` (the pinned `build.py` run now takes `build_env`) updated. Not shared yet: the translation itself. A new worktree still translates the game (about 2:40, below); its C is the same, so its objects come from the store. The store is never pruned; it can be deleted at any time. ## Flags and settings | Flag | New or changed | What it does | Default now | Goal: on by default? | Helps on | Doesn't help on | Untested on | Untested, but should help on | |---|---|---|---|---|---|---|---|---| | `LONGSHOT_OBJ_STORE` | new | `<folder>`: one compiled-object cache shared by every work folder (split by toolchain stamp), plus the lift's seed lists | unset (`<work>/objcache`, as before) | no (a developer switch) | Windows 11, llvm-mingw, store and worktrees on one NTFS drive: a new worktree compiled 0 of 484 files | - | Linux | Linux (the same code; hard links work there as on NTFS) | | `LONGSHOT_OBJ_CACHE` | new | the folder `build.py` keeps its compiled objects in; `make_game.py` sets it for `build.py`'s runs only (from `LONGSHOT_OBJ_STORE`, else `<work>/objcache`) | unset when `build.py` runs alone (`<work>/objcache`) | no (set by `make_game.py`, not by hand) | - | - | Linux | - | | `LONGSHOT_OBJ` | new (test only) | test only: `tests/test_obj_store.py` checks that `lift_key` names no `LONGSHOT_OBJ*` variable | - | no (a developer switch) | - | - | - | - | | `RX_RELEASE` | unchanged (its line in `main()` is rewritten as it was) | the release build's switch, as before | as before | no (a developer switch) | - | - | - | - | ## Measurements Measured on the GitHub version (NotRexed/Hookshot#2974), not yet on Longshot: this PR stays WIP until it is. Hardware: RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11; the Builder's tools (llvm-mingw), store and worktrees on one NTFS drive. Method: the `main` figure is what such a build took before this change, a full build on the same PC, not a run made beside these. | | GitHub main | this PR (GitHub) | |---|---|---| | A brand-new worktree, work folder empty: files compiled | a full build | 0 (484 of 484 objects from the store; compile step 15 s) | | A brand-new worktree, work folder empty: total | 32-36 minutes | 4:17 | | A worktree with a work folder, the store switched on (its 1,942 objects put in): files compiled, total | - | 3 files (what changed), 4:15 | - The rest of the new worktree's 4:17 is the translation (about 2:40), not shared yet. - Its hot lists and chunk boundaries came from the store's seed, so its lifted files' keys matched. - Its game plays: Kokiri Forest from a save, 27-38 fps, no errors in the log. Checks: every check that reads the changed files (76), through `run_all.py` with the Builder's tools, Windows: 71 pass. `nx2/test_dll.py`, `test_driver_cache_folder`, `test_fpx_sticky` and `test_linux_update` fail the same way on `main` there. `test_dlss_sdk_cache` failed once in the parallel run and passes alone, twice (Defender holds the exe it compiles, now and then). ## Testing **Tested:** the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured: the new worktree's game in play (Kokiri Forest, above; no `main` run beside it). **Not tested:** Linux (the same code; hard links work there as on NTFS). **Needs other testers:** - Linux: a second worktree built with `LONGSHOT_OBJ_STORE` set, expecting none of the objects the first compiled compiled again (request to open) ## Screenshots Nothing on screen. ## Risk and rollback Unset, nothing changes except `cache_put`, which now renames a temporary onto the key: a failed put is skipped where it used to raise. With the store set, every work folder links the store's object for a key, so a bad object there reaches each of them, and a new worktree takes the seed lists another checkout left there; the store is never pruned, so it grows. Unsetting `LONGSHOT_OBJ_STORE` turns it off (the store folder can be deleted at any time); reverting the commit is enough. ## Reviewer notes Carried over from [NotRexed/Hookshot#2974](https://github.com/NotRexed/Hookshot/pull/2974) with the move to Longshot: the same change on Longshot's `main`, with its rename applied. Its review and discussion are there. The dev-loop proposal's A10 ([NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636)); hckrman101 offered it in the review and is away, so this takes it on. `tests/test_obj_store.py` covers: two work folders over one store compile the same sources through `build.py`'s `compile_one` with a stand-in compiler (the second compiles none, and its objects are the store's, the same file); a changed source compiles alone; a copy that stops halfway leaves no entry and no temporary (`main`'s `cache_put` fails this); eight threads putting one object leave one; the store's part by stamp, and the fallbacks; adoption, and the seeds both ways; the lift's environment never gets the store's folder. The checks and the in-game runs above were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks, and they were not run again here. Only the checks that read the changed files were run, not the whole `run_all.py`. ## Issues See [NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636) (A10). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] Flags table filled in; README / docs/development.md switch reference updated. - [ ] Screenshots for a change that shows on screen. - [ ] A row in `docs/tested-configurations.md` for each measured run. - [ ] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). - [ ] The SPIR-V headers regenerated with glslang (a shader changed). - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [ ] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 3 commits

2026-11-12 23:38:28 +00:00

feat(build): LONGSHOT_OBJ_STORE, one object cache for every work folder e2a5463623

A worktree for a PR compiled the whole game again (~600 files) although another work folder had compiled the same
sources with the same flags already. With LONGSHOT_OBJ_STORE=<folder>, build.py keeps its objects in the store's part for
this toolchain (make_game.object_cache: <store>/objects/<system_headers.txt's stamp>), so each object is compiled once
across work folders; unset, <work>/objcache as before. Objects go in whole (cache_put: linked or copied to a temporary,
then renamed onto the key); the work folder's clean-ups never touch the store; the git-ignored lists a lift reads come
from <store>/seed when this checkout's work folder has none (seed_work), so a new worktree's chunks and hot pieces
match. Only build.py's runs get the folder, so the lift's key is unchanged. The dev-loop proposal's A10 (#2636).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

test: the checks that cut cache_put and pin build.py's run take the object store's names 3d23dc2ed1

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

feat(build): a work folder's own objects go into the store when it is first used 819cd1555e

Switching LONGSHOT_OBJ_STORE on compiled the whole game once more, though the work folder held every object already. Its
<work>/objcache (the same toolchain: its stamp names the store's part) is linked into the store first, each in whole and
only where the store has none.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst changed title from ~~feat(build): LONGSHOT_OBJ_STORE, one object cache for every work folder~~ to WIP: feat(build): LONGSHOT_OBJ_STORE, one object cache for every work folder

2026-11-13 01:13:46 +00:00

dyst commented

2026-11-13 01:13:46 +00:00

Author

First-time contributor

Back to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.

dyst added 1 commit

2026-11-13 01:54:32 +00:00

style(build): the space make_game.py's env line lost 7c39708d1c

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst commented

2026-11-13 01:58:13 +00:00

Author

First-time contributor

Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready. Also: 7c39708d puts back a space make_game.py's env line had lost.

Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready. Also: 7c39708d puts back a space make_game.py's env line had lost.

dyst commented

2026-11-13 02:00:29 +00:00

Author

First-time contributor

Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'feat/obj-store:dyst-feat/obj-store'

git switch 'dyst-feat/obj-store'

Rows
Columns

WIP: feat(build): LONGSHOT_OBJ_STORE, one object cache for every work folder #24

Why

What changed

Flags and settings

Measurements

Testing

Screenshots

Risk and rollback

Reviewer notes

Issues

Checklist

For everyone

If applicable

Checkout