MagicBeanDev/Longshot

Watch

Fork 71

perf(build): one object store for every work folder, so a new one compiles only what no other built #371

Merged

HumanBean merged 1 commit from HumanBean/Longshot:perf/object-store into main

2026-11-14 15:15:13 +00:00

HumanBean commented

2026-11-14 05:56:40 +00:00

Owner

Why

Each new work folder (a second checkout, a new dev_packs.py base, a fresh --work) compiles every one of the ~480 files again, although another folder on the same PC already built the same objects: 8 min of compile on a 12-thread share of a Ryzen 9 7900X, 23 min for a whole base on an 8-core lab PC. This shares one object store across work folders, so a new folder compiles only what none built before.

What changed

build: nx2/object_store.py (new): the store, off unless LONGSHOT_OBJ_STORE=<dir>. One sub-folder per compiler and system headers (the first 16 characters of the work folder's system_headers.txt, which make_game.py drop_stale_objects writes; a folder without it uses no store). fetch hard-links the keys a build needs from the store into <work>/objcache, share links the folder's objects the store lacks back; across volumes or without hard links, a copy under a temporary name renamed in (no torn object, no overwrite on Windows). prune: the store's own clean-up. forget: objects out of the store.
build: nx2/build.py main: before the compile, shares the folder's own objects, works out every job's key (into PLANNED, so nothing is hashed twice) and fetches those keys; the kept-translation check fetches the kept keys first. After a compile with no failures, shares the new objects. compile_one, cache_put, plan_jobs and the object keys are unchanged: an existing folder compiles nothing again.
build: nx2/make_game.py: forget_failed_build (a link that failed) also takes that link's objects (object_stamps, from sa/build/obj/*.sha1) out of the store, or the next try would take the same objects back. After each build, object_store.prune(): an object only the store holds (one link; every object where hard links don't exist) goes once unused for LONGSHOT_OBJ_STORE_DAYS (14), and oldest first while those exceed LONGSHOT_OBJ_STORE_GB (8); nothing used in the last hour. prune_object_cache and the rmtrees of <work>/objcache only ever remove the folder's own links, so they never take an object another folder uses.
dev: scripts/dev_packs.py run: its builds use ~/.cache/longshot-objects (%LOCALAPPDATA%\Longshot\objects) unless LONGSHOT_OBJ_STORE is set (0: none).
docs: docs/development.md (Building's switch table, Testing a change against a base).
tests: tests/test_object_store.py (new): fetch/share, the folder's prune leaves the store whole, forget, the store's prune by links, age and size.

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
`LONGSHOT_OBJ_STORE=<dir>`	One object store for every work folder	off (`make_game.py`); on in `dev_packs.py`	Off for players: the Longshot Launcher and the Builders keep one work folder for every version (`LauncherHome.Work`, `make_game.py`'s default `--work`), so there is nothing to share. On for development	developers with several checkouts or bases	one work folder (players)	Windows (the copy fallback, NTFS hard links)
`LONGSHOT_OBJ_STORE_GB=<n>`	The store's own objects (held by no work folder) above this go, oldest first	8	keep
`LONGSHOT_OBJ_STORE_DAYS=<n>`	The store's own objects unused this long go	14	keep

Measurements

main: 0caf0618 (2026-11-14); measured on cf605c85 with this change (34b7e787), then rebased: the three commits since touch neither nx2/build.py, nx2/make_game.py nor scripts/dev_packs.py (the Windows lift's worker count, the button prompts, a docs file), so the compile numbers hold.
this PR: perf/object-store @ 8a7bf09d (34b7e787 before the rebase)
Hardware: Ryzen 9 7900X (24 threads), 30 GB, NVMe SSD (btrfs), CachyOS Linux; the builds pinned to 12 threads (oot-status --timing --cpu-only), the main session's builds running on the other 12
Scene: no game run: python -I scripts/dev_packs.py base <new folder> --romfs ... --exefs ... of the same commit, LONGSHOT_NO_PREPARE=1, NGX SDK and FSR as in the regression env, into a new folder each time
Method: the build's own "where the time went" table and its @@file ... new|cached lines. Baseline: this PR @ 34b7e787, LONGSHOT_OBJ_STORE=0, which is main's behaviour (the store's code does nothing then), on the same commit so the binaries can be compared byte for byte (the version fingerprint covers nx2/). Not main itself because main's binaries carry another fingerprint.
How to reproduce: two dev_packs.py base runs of one commit into two new folders, the first with an empty LONGSHOT_OBJ_STORE, the second with the same store; compare the second's compile with a run at LONGSHOT_OBJ_STORE=0.

	this PR @ 34b7e787, LONGSHOT_OBJ_STORE=0 (baseline, cold)	this PR @ 34b7e787, store filled by an earlier folder
read	9 s	not run yet
lift	76 s	not run yet
compile	498 s (483 of 483 files compiled)	not run yet
total	590 s	not run yet
`longshot`, `longshot_vk.so` byte-identical to the baseline		not run yet

The run with the store (an empty store, then a second folder over it) was stopped by Claude Code because the PC ran low on memory while other builds ran; it is to be run again. Until then this stays WIP.

where the time went:   (this PR @ 34b7e787, LONGSHOT_OBJ_STORE=0)
  read                                    9 s
  lift                                   76 s
  compile                               498 s
    planning                              3 s
    compiling                           494 s
    linking longshot                      1 s
    linking longshot_vk.so                0 s
  finish                                  7 s
the build took 0:09:50, compiled in 0:08:18

Testing

Tested: a cold dev_packs.py base with LONGSHOT_OBJ_STORE=0 on Linux (above): builds and links as before. No checks were run on the author's PC (tests/run_all.py, including the new tests/test_object_store.py, is left to CI). The checks that cut compile_one, cache_put, plan_jobs, prune_object_cache and forget_failed_build out of the source were read: those functions are unchanged or keep their names and signatures, so no check needed updating.

Not tested: the build with the store on (stopped, above); Windows (the copy fallback across drives, NTFS hard links, a file held open during the store's prune); two builds sharing a store at the same time; a failed link taking its objects out of the store.

Needs other testers:

Windows: two dev_packs.py base runs of one commit into two new folders (LONGSHOT_OBJ_STORE on the bases' drive, then on another drive): the second compiles only what the first didn't, and its game starts (request to open)

Screenshots

Nothing on screen.

Risk and rollback

A wrong object in the store would reach every folder that uses it, where today it stays in one: the keys are the per-folder cache's (path-free since issue 565), partitioned by the same compiler and headers stamp that guards a folder's cache, and a failed link drops its objects from the store. Off unless LONGSHOT_OBJ_STORE is set (dev_packs: LONGSHOT_OBJ_STORE=0). Reverting the commit is enough; a store folder left behind can be deleted.

Reviewer notes

The hot pieces' flags carry -fprofile-instr-use=<work>/longshot_pieces.prof (make_game.py sets RX_PGO to the work folder's copy), so in a checkout with a play profile the 8 hot pieces still compile once per work folder. Fixing that changes those keys (one recompile of them in every folder), so it is left for a follow-up.
The lift is still repeated in each new folder: the next PR (perf/lift-store) shares it.

Issues

None.

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above). (no tests run on the author's PC; CI runs them)
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (the run with the store was stopped by low memory; one cold run so far)
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags noted, and the switch reference updated (docs/development.md, the README).
A game-free test added for the fix, failing on main and passing here. (added, tests/test_object_store.py; not run on the author's PC, CI runs it)
Docs updated.

🤖 Generated with Claude Code

https://claude.ai/code/session_017vw8e34avQ5cTCWow16MXn

## Why Each new work folder (a second checkout, a new `dev_packs.py` base, a fresh `--work`) compiles every one of the ~480 files again, although another folder on the same PC already built the same objects: 8 min of compile on a 12-thread share of a Ryzen 9 7900X, 23 min for a whole base on an 8-core lab PC. This shares one object store across work folders, so a new folder compiles only what none built before. ## What changed - build: `nx2/object_store.py` (new): the store, off unless `LONGSHOT_OBJ_STORE=<dir>`. One sub-folder per compiler and system headers (the first 16 characters of the work folder's `system_headers.txt`, which `make_game.py drop_stale_objects` writes; a folder without it uses no store). `fetch` hard-links the keys a build needs from the store into `<work>/objcache`, `share` links the folder's objects the store lacks back; across volumes or without hard links, a copy under a temporary name renamed in (no torn object, no overwrite on Windows). `prune`: the store's own clean-up. `forget`: objects out of the store. - build: `nx2/build.py main`: before the compile, shares the folder's own objects, works out every job's key (into `PLANNED`, so nothing is hashed twice) and fetches those keys; the kept-translation check fetches the kept keys first. After a compile with no failures, shares the new objects. `compile_one`, `cache_put`, `plan_jobs` and the object keys are unchanged: an existing folder compiles nothing again. - build: `nx2/make_game.py`: `forget_failed_build` (a link that failed) also takes that link's objects (`object_stamps`, from `sa/build/obj/*.sha1`) out of the store, or the next try would take the same objects back. After each build, `object_store.prune()`: an object only the store holds (one link; every object where hard links don't exist) goes once unused for `LONGSHOT_OBJ_STORE_DAYS` (14), and oldest first while those exceed `LONGSHOT_OBJ_STORE_GB` (8); nothing used in the last hour. `prune_object_cache` and the `rmtree`s of `<work>/objcache` only ever remove the folder's own links, so they never take an object another folder uses. - dev: `scripts/dev_packs.py run`: its builds use `~/.cache/longshot-objects` (`%LOCALAPPDATA%\Longshot\objects`) unless `LONGSHOT_OBJ_STORE` is set (`0`: none). - docs: `docs/development.md` (Building's switch table, *Testing a change against a base*). - tests: `tests/test_object_store.py` (new): fetch/share, the folder's prune leaves the store whole, `forget`, the store's prune by links, age and size. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `LONGSHOT_OBJ_STORE=<dir>` | One object store for every work folder | off (`make_game.py`); on in `dev_packs.py` | Off for players: the Longshot Launcher and the Builders keep one work folder for every version (`LauncherHome.Work`, `make_game.py`'s default `--work`), so there is nothing to share. On for development | developers with several checkouts or bases | one work folder (players) | Windows (the copy fallback, NTFS hard links) | | `LONGSHOT_OBJ_STORE_GB=<n>` | The store's own objects (held by no work folder) above this go, oldest first | 8 | keep | | | | | `LONGSHOT_OBJ_STORE_DAYS=<n>` | The store's own objects unused this long go | 14 | keep | | | | ## Measurements main: 0caf0618 (2026-11-14); measured on cf605c85 with this change (34b7e787), then rebased: the three commits since touch neither `nx2/build.py`, `nx2/make_game.py` nor `scripts/dev_packs.py` (the Windows lift's worker count, the button prompts, a docs file), so the compile numbers hold. this PR: perf/object-store @ 8a7bf09d (34b7e787 before the rebase) Hardware: Ryzen 9 7900X (24 threads), 30 GB, NVMe SSD (btrfs), CachyOS Linux; the builds pinned to 12 threads (`oot-status --timing --cpu-only`), the main session's builds running on the other 12 Scene: no game run: `python -I scripts/dev_packs.py base <new folder> --romfs ... --exefs ...` of the same commit, `LONGSHOT_NO_PREPARE=1`, NGX SDK and FSR as in the regression env, into a new folder each time Method: the build's own "where the time went" table and its `@@file ... new|cached` lines. Baseline: `this PR @ 34b7e787, LONGSHOT_OBJ_STORE=0`, which is main's behaviour (the store's code does nothing then), on the same commit so the binaries can be compared byte for byte (the version fingerprint covers `nx2/`). Not main itself because main's binaries carry another fingerprint. How to reproduce: two `dev_packs.py base` runs of one commit into two new folders, the first with an empty `LONGSHOT_OBJ_STORE`, the second with the same store; compare the second's compile with a run at `LONGSHOT_OBJ_STORE=0`. | | this PR @ 34b7e787, LONGSHOT_OBJ_STORE=0 (baseline, cold) | this PR @ 34b7e787, store filled by an earlier folder | |---|---|---| | read | 9 s | not run yet | | lift | 76 s | not run yet | | compile | 498 s (483 of 483 files compiled) | not run yet | | total | 590 s | not run yet | | `longshot`, `longshot_vk.so` byte-identical to the baseline | | not run yet | The run with the store (an empty store, then a second folder over it) was stopped by Claude Code because the PC ran low on memory while other builds ran; it is to be run again. Until then this stays WIP. ``` where the time went: (this PR @ 34b7e787, LONGSHOT_OBJ_STORE=0) read 9 s lift 76 s compile 498 s planning 3 s compiling 494 s linking longshot 1 s linking longshot_vk.so 0 s finish 7 s the build took 0:09:50, compiled in 0:08:18 ``` ## Testing **Tested:** a cold `dev_packs.py base` with `LONGSHOT_OBJ_STORE=0` on Linux (above): builds and links as before. No checks were run on the author's PC (`tests/run_all.py`, including the new `tests/test_object_store.py`, is left to CI). The checks that cut `compile_one`, `cache_put`, `plan_jobs`, `prune_object_cache` and `forget_failed_build` out of the source were read: those functions are unchanged or keep their names and signatures, so no check needed updating. **Not tested:** the build with the store on (stopped, above); Windows (the copy fallback across drives, NTFS hard links, a file held open during the store's prune); two builds sharing a store at the same time; a failed link taking its objects out of the store. **Needs other testers:** - Windows: two `dev_packs.py base` runs of one commit into two new folders (`LONGSHOT_OBJ_STORE` on the bases' drive, then on another drive): the second compiles only what the first didn't, and its game starts (request to open) ## Screenshots Nothing on screen. ## Risk and rollback A wrong object in the store would reach every folder that uses it, where today it stays in one: the keys are the per-folder cache's (path-free since issue 565), partitioned by the same compiler and headers stamp that guards a folder's cache, and a failed link drops its objects from the store. Off unless `LONGSHOT_OBJ_STORE` is set (dev_packs: `LONGSHOT_OBJ_STORE=0`). Reverting the commit is enough; a store folder left behind can be deleted. ## Reviewer notes - The hot pieces' flags carry `-fprofile-instr-use=<work>/longshot_pieces.prof` (`make_game.py` sets `RX_PGO` to the work folder's copy), so in a checkout with a play profile the 8 hot pieces still compile once per work folder. Fixing that changes those keys (one recompile of them in every folder), so it is left for a follow-up. - The lift is still repeated in each new folder: the next PR (perf/lift-store) shares it. ## Issues None. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). (no tests run on the author's PC; CI runs them) - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (the run with the store was stopped by low memory; one cold run so far) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [ ] A game-free test added for the fix, failing on `main` and passing here. (added, tests/test_object_store.py; not run on the author's PC, CI runs it) - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017vw8e34avQ5cTCWow16MXn

HumanBean added 1 commit

2026-11-14 05:56:40 +00:00

perf(build): one object store for every work folder

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 7s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 8s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Failing after 10s

Details

8a7bf09d45

LONGSHOT_OBJ_STORE=<dir> (nx2/object_store.py): nx2/build.py takes the
objects a build needs and its work folder lacks from the store, and gives
the store the folder's own, by hard links (a copy renamed in across
drives). The store has a folder per compiler and system headers (the work
folder's system_headers.txt). The folder's clean-up only removes its own
links; the store has its own (make_game.py, after each build): objects no
folder holds, unused for 14 days or oldest first over 8 GB. A link that
fails takes its objects out of the store. scripts/dev_packs.py turns it on
for its bases and packs. The object keys are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017vw8e34avQ5cTCWow16MXn

HumanBean referenced this pull request

2026-11-14 05:57:13 +00:00

perf(build): a lift reused across work folders, so a new one skips translating the same game #372

mooki added the

labels

2026-11-14 06:29:03 +00:00

mooki requested changes

2026-11-14 06:45:01 +00:00

mooki left a comment

Read head 8a7bf09d against main 0caf0618: the complete six-file diff, description, empty comment and review lists, object writers and consumers, recovery paths and related build changes; ran 10 game-free check invocations across 9 files, all passed, including the header-dependency fixture with Clang and GCC. No game, full build or timing run.

What's good

The repeated work is real: main's object cache belongs to one work folder, so an unrelated new folder cannot use its objects. Keeping that local cache also keeps the ordinary cleanup from deleting another folder's links.
The store's own copy path publishes through a process-specific temporary file. Two publishers of a complete object do not expose either partial copy: the hard-link creation is exclusive, and the copy fallback renames only after copying. Windows refusing an existing destination is caught.
The fetched keys reach both the kept-translation check and the normal planner, and compile_one actually consumes the populated local cache. The WIP description clearly admits that no store-enabled build comparison completed.

Must change

Include the selected external headers in shared-object identity. In nx2/build.py, lines 1290-1293, the new fetch trusts keys which omit some real inputs. For a concrete case, build two fresh work folders with the same compiler and source but different VMA packages selected by RX_CPM. flag_text, lines 666-678, drops their -I paths, and the dependency roots at lines 1218-1222 contain neither package. system_headers.txt does not distinguish them either. Yet gpu/vk/vk_vma.cpp, lines 4-8, compiles the allocator implementation from vk_mem_alloc.h. The second folder therefore takes the first package's allocator object rather than compiling its selected one. Main would compile in that fresh folder. Cover the selected external headers and their contents before enabling cross-folder reuse; the same audit is needed for Vulkan, SPIR-V and the C/C++ system headers. A two-folder test that changes an implementation-bearing header, then checks both the miss and the resulting object, would settle this. Traced through the source, not reproduced with a native build.
Make local-cache publication atomic before exporting interrupted builds. In nx2/build.py, lines 1288-1289, share publishes every existing local .o, explicitly including a stopped build's entries. However, cache_put, lines 323-333, falls back to shutil.copyfile(obj, dst) directly at the final cache name when hard links are unavailable. Killing that copy, or running out of space during it, leaves an incomplete key.o. The next build now exports it, and a different work folder accepts it by existence in compile_one, lines 962-972. The store's temporary-file copy cannot repair an already partial source. Make the producer's fallback atomic and ensure incomplete entries left by older producers are not imported. Add an interrupted-copy test which restarts publication and tries a fetch from a second folder, not just a successful share/fetch test. Traced through the source; no process-kill or disk-full reproduction run.
Keep failed-link invalidation from being undone by another folder. In nx2/object_store.py, lines 123-127, forget only unlinks the store pathname. A second folder that already fetched the rejected object keeps its local link, and its next share recreates that same entry. The first folder's retry can then fetch the object it just rejected instead of recompiling. On Windows, an open copy source can also make the unlink fail; that exception is discarded, leaving the same entry immediately reusable. Preserve a rejected-entry record or otherwise make the retry bypass that object even if another publisher retains it. Test two folders through fetch, failed-link eviction, re-sharing and retry, plus a refused unlink. The current test checks only that one removal leaves the first folder's local copy intact. Traced through the source, not exercised with concurrent builds or Windows file handles.

Measurements and smaller points

There is no measured gain yet. The only completed row is 34b7e787 with the store off: 483 files compiled, 498 s compiling and 590 s total. The store-enabled run was stopped, so there is no comparison, output-equivalence result, median or spread. An on/off comparison at one named commit can isolate this feature, but needs at least two alternating runs per condition on the same PC, controlled store state and retained individual results. Record the warm-up that fills the store separately and keep other builds out of the comparison. The current WIP qualification is appropriate.
The size setting is not a total disk cap. The 8 GiB limit counts only objects with no other hard link, and recent objects have an hour's grace. Retained development folders keep their referenced entries outside both age and size eviction, deliberately. Renderer-only packs call build.py directly, so they do not run the cleanup added to make_game.py. Explain those limits when recommending a shared store, especially the copy fallback across drives.
The passing fixture does not cover the critical failures. tests/test_object_store.py uses authored bytes and prewritten stamps, not the real compiler or header-note producer. It has no simultaneous publishers, interrupted copy, forced copy fallback, denied Windows deletion or failed-link retry through another folder. These passing checks are useful but do not settle the findings above. The description also leaves checks to CI, while both source-check shards are skipped: https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/4408.

Overlaps

#30 complements this change by removing the hot profile's work-folder path from its key. The pair merges cleanly. #37 conflicts in nx2/build.py immediately after compilation: its rekey_depfiles(vk_jobs) and this change's share occupy the same position. Re-key before sharing; that change's restricted dependency roots do not by themselves fix the external-header omission above.
#24 is an alternative shared-store design, not a prerequisite. It conflicts at the OBJ_CACHE declaration and changes the local cache's publication method. Choose one storage layout rather than combining both defaults.
#370 merges cleanly and changes shader-stamp and pack-preparation decisions. #372 is directly stacked on this head, adding shared lift snapshots; it depends on this change, not the reverse.
#31 and #38 affect the lift and merge pairwise without conflicts. #360 is separate game-file reuse; its code merges, but its new documentation row conflicts with the Windows worker wording inherited here. #357 and #358 are already in this base. This head itself merges cleanly with main 0caf0618.

Not checked

No Windows execution, actual concurrent publishers, interrupted-copy recovery, changed-header native reproduction, full suite, clean/incremental game build, game start or independent speed measurement. The 10 passing invocations were test_object_store, test_object_cache_prune, test_build_hash_cache, test_kept_translation, test_compiler_failure, test_compile_retry_jobs, test_failed_compile_keeps_objects, test_dev_packs, and test_system_header_cache with each compiler. The latter preprocesses authored headers; it is not a Windows runtime test. The requested failure tests and clean/incremental Linux and Windows builds, on both same-drive and cross-drive stores, would decide the remaining safety questions.

Read head `8a7bf09d` against main `0caf0618`: the complete six-file diff, description, empty comment and review lists, object writers and consumers, recovery paths and related build changes; ran 10 game-free check invocations across 9 files, all passed, including the header-dependency fixture with Clang and GCC. No game, full build or timing run. ### What's good - The repeated work is real: main's object cache belongs to one work folder, so an unrelated new folder cannot use its objects. Keeping that local cache also keeps the ordinary cleanup from deleting another folder's links. - The store's own copy path publishes through a process-specific temporary file. Two publishers of a complete object do not expose either partial copy: the hard-link creation is exclusive, and the copy fallback renames only after copying. Windows refusing an existing destination is caught. - The fetched keys reach both the kept-translation check and the normal planner, and `compile_one` actually consumes the populated local cache. The WIP description clearly admits that no store-enabled build comparison completed. ### Must change 1. **Include the selected external headers in shared-object identity.** In `nx2/build.py`, lines 1290-1293, the new fetch trusts keys which omit some real inputs. For a concrete case, build two fresh work folders with the same compiler and source but different VMA packages selected by `RX_CPM`. `flag_text`, lines 666-678, drops their `-I` paths, and the dependency roots at lines 1218-1222 contain neither package. `system_headers.txt` does not distinguish them either. Yet `gpu/vk/vk_vma.cpp`, lines 4-8, compiles the allocator implementation from `vk_mem_alloc.h`. The second folder therefore takes the first package's allocator object rather than compiling its selected one. Main would compile in that fresh folder. Cover the selected external headers and their contents before enabling cross-folder reuse; the same audit is needed for Vulkan, SPIR-V and the C/C++ system headers. A two-folder test that changes an implementation-bearing header, then checks both the miss and the resulting object, would settle this. **Traced through the source, not reproduced with a native build.** 2. **Make local-cache publication atomic before exporting interrupted builds.** In `nx2/build.py`, lines 1288-1289, `share` publishes every existing local `.o`, explicitly including a stopped build's entries. However, `cache_put`, lines 323-333, falls back to `shutil.copyfile(obj, dst)` directly at the final cache name when hard links are unavailable. Killing that copy, or running out of space during it, leaves an incomplete `key.o`. The next build now exports it, and a different work folder accepts it by existence in `compile_one`, lines 962-972. The store's temporary-file copy cannot repair an already partial source. Make the producer's fallback atomic and ensure incomplete entries left by older producers are not imported. Add an interrupted-copy test which restarts publication and tries a fetch from a second folder, not just a successful `share`/`fetch` test. **Traced through the source; no process-kill or disk-full reproduction run.** 3. **Keep failed-link invalidation from being undone by another folder.** In `nx2/object_store.py`, lines 123-127, `forget` only unlinks the store pathname. A second folder that already fetched the rejected object keeps its local link, and its next `share` recreates that same entry. The first folder's retry can then fetch the object it just rejected instead of recompiling. On Windows, an open copy source can also make the unlink fail; that exception is discarded, leaving the same entry immediately reusable. Preserve a rejected-entry record or otherwise make the retry bypass that object even if another publisher retains it. Test two folders through fetch, failed-link eviction, re-sharing and retry, plus a refused unlink. The current test checks only that one removal leaves the first folder's local copy intact. **Traced through the source, not exercised with concurrent builds or Windows file handles.** ### Measurements and smaller points - **There is no measured gain yet.** The only completed row is `34b7e787` with the store off: 483 files compiled, 498 s compiling and 590 s total. The store-enabled run was stopped, so there is no comparison, output-equivalence result, median or spread. An on/off comparison at one named commit can isolate this feature, but needs at least two alternating runs per condition on the same PC, controlled store state and retained individual results. Record the warm-up that fills the store separately and keep other builds out of the comparison. The current WIP qualification is appropriate. - **The size setting is not a total disk cap.** The 8 GiB limit counts only objects with no other hard link, and recent objects have an hour's grace. Retained development folders keep their referenced entries outside both age and size eviction, deliberately. Renderer-only packs call `build.py` directly, so they do not run the cleanup added to `make_game.py`. Explain those limits when recommending a shared store, especially the copy fallback across drives. - **The passing fixture does not cover the critical failures.** `tests/test_object_store.py` uses authored bytes and prewritten stamps, not the real compiler or header-note producer. It has no simultaneous publishers, interrupted copy, forced copy fallback, denied Windows deletion or failed-link retry through another folder. These passing checks are useful but do not settle the findings above. The description also leaves checks to CI, while both source-check shards are skipped: https://git.magicbean.dev/MagicBeanDev/Longshot/actions/runs/4408. ### Overlaps - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/30 complements this change by removing the hot profile's work-folder path from its key. The pair merges cleanly. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/37 conflicts in `nx2/build.py` immediately after compilation: its `rekey_depfiles(vk_jobs)` and this change's `share` occupy the same position. Re-key before sharing; that change's restricted dependency roots do not by themselves fix the external-header omission above. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/24 is an alternative shared-store design, not a prerequisite. It conflicts at the `OBJ_CACHE` declaration and changes the local cache's publication method. Choose one storage layout rather than combining both defaults. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/370 merges cleanly and changes shader-stamp and pack-preparation decisions. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/372 is directly stacked on this head, adding shared lift snapshots; it depends on this change, not the reverse. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/31 and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/38 affect the lift and merge pairwise without conflicts. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/360 is separate game-file reuse; its code merges, but its new documentation row conflicts with the Windows worker wording inherited here. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/357 and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/358 are already in this base. This head itself merges cleanly with main `0caf0618`. ### Not checked No Windows execution, actual concurrent publishers, interrupted-copy recovery, changed-header native reproduction, full suite, clean/incremental game build, game start or independent speed measurement. The 10 passing invocations were `test_object_store`, `test_object_cache_prune`, `test_build_hash_cache`, `test_kept_translation`, `test_compiler_failure`, `test_compile_retry_jobs`, `test_failed_compile_keeps_objects`, `test_dev_packs`, and `test_system_header_cache` with each compiler. The latter preprocesses authored headers; it is not a Windows runtime test. The requested failure tests and clean/incremental Linux and Windows builds, on both same-drive and cross-drive stores, would decide the remaining safety questions.

HumanBean changed title from ~~WIP: perf(build): one object store for every work folder, so a new one compiles only what no other built~~ to perf(build): one object store for every work folder, so a new one compiles only what no other built

2026-11-14 15:15:04 +00:00

HumanBean merged commit 7111d1b102 into main

2026-11-14 15:15:13 +00:00

HumanBean deleted branch perf/object-store

2026-11-14 15:15:14 +00:00

HumanBean referenced this pull request from a commit

2026-11-14 15:15:15 +00:00

perf(build): one object store for every work folder, so a new one compiles only what no other built (#371)

mooki referenced this pull request

2026-11-14 16:11:54 +00:00

WIP: build(macos): classify Darwin helpers and package a relocatable macOS 14.4 dylib set #479