perf(build): a lift reused across work folders, so a new one skips translating the same game #372

Open

HumanBean wants to merge 2 commits from HumanBean/Longshot:perf/lift-store into main

pull from: HumanBean/Longshot:perf/lift-store

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 0 Commits 2 Files changed 8 +704 -22

HumanBean commented 2026-11-14 05:57:13 +00:00

Owner

• Copy link
• Copy Markdown

Why

A new work folder translates the game again (55-76 s here, minutes on smaller PCs) even when another folder on the same PC translated the same game with the same translator a minute before: the lift key holds the work and game folders' paths. This keeps finished translations under a key without those paths, so a new folder (a new dev_packs.py base, a base's first pack, a second checkout) copies one instead.

Builds on #371 (the object store): review that first; this PR's own commit is the top one.

What changed

• build: nx2/lift_store.py (new): the store, off unless LONGSHOT_LIFT_STORE=<dir>. save copies <work>/sa/gen (without .lift_key and build.py's lifted_objects.txt), <work>/sa/image, image/chunks.txt and image/hot_pieces.txt into a temporary folder, writes the list of files and sizes last and renames it in. restore checks every file against that list, copies the C and image, runs a check of the copied C, and only then puts the two boundary files over the folder's; any failure leaves the folder to lift as before. Copies are FICLONE clones where the file system has them (btrfs, XFS), plain copies elsewhere, never hard links (the later steps write in these folders). prune: the newest LONGSHOT_LIFT_STORE_KEEP (3) by last use; none used in the last hour.
• build: nx2/make_game.py: lift_key(..., paths=None) maps the environment's values through paths; lift_share_key is the lift key with the work and game folders' paths taken out (OOT_WORK, OOT_EXEFS, OOT_ROMFS, OOT_DUMP, A64_HOT_FUNCS). Every file is already hashed by its name inside its folder, and the lifted C holds no folder path (checked: no work or game path in an earlier folder's sa/gen or sa/image). In translate(), when the folder must lift: the share key is taken from the state the lift starts from (as the folder that saved it took it), a snapshot is restored if there is one (its C must not be cut short, lifted_cut_short, or it is forgotten), and the folder's own key is taken again afterwards from the restored boundaries and hot pieces, so the next build in that folder skips the lift as after a lift of its own. After a real lift whose inputs did not change while it ran, and whose C is whole, it is saved.
• What decides reuse: everything the lift key covers (the game's code, the translator's sources, the HLE list, the hot list, the chunk boundaries and hot pieces as the lift starts, the A64_/OOT_/RX_ settings other than the compile's, the Python), minus the two folders' paths. chunks.txt / hot_pieces.txt: the key holds their state before the lift, and the restore brings the state after it, so the new folder adopts the same boundaries and its objects match the saving folder's (with the object store, they come from there too).
• dev: scripts/dev_packs.py run: its builds use ~/.cache/longshot-lifts (%LOCALAPPDATA%\Longshot\lifts) unless LONGSHOT_LIFT_STORE is set (0: none).
• docs: docs/development.md (Building's switch table, Testing a change against a base).
• tests: tests/test_lift_store.py (new): save and restore, a snapshot not whole or cut short never used, prune, and the share key equal across folders where the lift key differs. tests/test_lift_cache.py's check of translate()'s text still holds: the lift's steps keep their order and wording, one level deeper.

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
LONGSHOT_LIFT_STORE=<dir>	Finished translations kept for every work folder	off (make_game.py); on in dev_packs.py	Off for players: one work folder for every version, and a version that changes the translator lifts anyway. On for development	developers with several checkouts or bases	one work folder (players); a change to the translator	Windows (plain copies)
LONGSHOT_LIFT_STORE_KEEP=<n>	Snapshots kept	3	keep: ~1.6 GB each where copies aren't clones

Measurements

main: 0caf0618 (2026-11-14)
this PR: perf/lift-store @ 7bd98427
Hardware: Ryzen 9 7900X (24 threads), 30 GB, NVMe SSD (btrfs), CachyOS Linux
Scene: no game run: dev_packs.py base of one commit into a new folder after an earlier folder saved its lift
Method: the build's "where the time went" table. Not measured yet: the measurement builds were stopped when the PC ran low on memory. Baseline to be: this PR @ 7bd98427, LONGSHOT_LIFT_STORE=0, same commit, so the binaries can be compared.
How to reproduce: two dev_packs.py base runs of one commit into two new folders with one LONGSHOT_LIFT_STORE; the second's lift line and its longshot / longshot_vk.so against a run at LONGSHOT_LIFT_STORE=0.

	this PR @ 7bd98427, LONGSHOT_LIFT_STORE=0 (baseline)	this PR @ 7bd98427, snapshot from an earlier folder
lift	not run yet (76 s cold on perf/object-store @ 34b7e787)	not run yet
total	not run yet	not run yet
binaries byte-identical		not run yet

not run yet

Testing

Tested: nothing in the game yet (above). No checks were run on the author's PC (tests/run_all.py, including the new tests/test_lift_store.py, is left to CI). The checks that read make_game.py's text (tests/test_lift_cache.py check_key_after_lift, tests/test_kept_translation.py, tests/test_dev_packs.py) were read against the change.

Not tested: any build with the store; Windows; two builds saving the same snapshot at once; a restore across drives.

Needs other testers:

• Windows: two dev_packs.py base runs of one commit into new folders: the second prints the game was translated with these tools in another work folder and its game starts (request to open)

Screenshots

Nothing on screen.

Risk and rollback

A snapshot used where the lift would have written other C would build a wrong game: reuse is gated by everything the existing lift key covers, minus only the two folders' paths, and a snapshot is checked whole before use. Off unless LONGSHOT_LIFT_STORE is set (dev_packs: LONGSHOT_LIFT_STORE=0). Reverting the commit is enough.

Reviewer notes

• The share key relies on the lift not reading the game folder (OOT_ROMFS, OOT_DUMP) beyond what the lift key already ignores today.
• LONGSHOT_* variables the lift reads (LONGSHOT_LIFT_JOBS, LONGSHOT_LIFT_NO_FORK) are not in either key, as today: they change how the lift runs, not what it writes.

Issues

See #371

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above). (no tests run on the author's PC; CI runs them)
• Tested in the game, or what wasn't is named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (not yet: the builds were stopped by low memory)
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• Flags noted, and the switch reference updated (docs/development.md, the README).
• A game-free test added for the fix, failing on main and passing here. (added, tests/test_lift_store.py; not run on the author's PC, CI runs it)
• Docs updated.

🤖 Generated with Claude Code

https://claude.ai/code/session_017vw8e34avQ5cTCWow16MXn

## Why A new work folder translates the game again (55-76 s here, minutes on smaller PCs) even when another folder on the same PC translated the same game with the same translator a minute before: the lift key holds the work and game folders' paths. This keeps finished translations under a key without those paths, so a new folder (a new `dev_packs.py` base, a base's first pack, a second checkout) copies one instead. Builds on https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/371 (the object store): review that first; this PR's own commit is the top one. ## What changed - build: `nx2/lift_store.py` (new): the store, off unless `LONGSHOT_LIFT_STORE=<dir>`. `save` copies `<work>/sa/gen` (without `.lift_key` and build.py's `lifted_objects.txt`), `<work>/sa/image`, `image/chunks.txt` and `image/hot_pieces.txt` into a temporary folder, writes the list of files and sizes last and renames it in. `restore` checks every file against that list, copies the C and image, runs a check of the copied C, and only then puts the two boundary files over the folder's; any failure leaves the folder to lift as before. Copies are `FICLONE` clones where the file system has them (btrfs, XFS), plain copies elsewhere, never hard links (the later steps write in these folders). `prune`: the newest `LONGSHOT_LIFT_STORE_KEEP` (3) by last use; none used in the last hour. - build: `nx2/make_game.py`: `lift_key(..., paths=None)` maps the environment's values through `paths`; `lift_share_key` is the lift key with the work and game folders' paths taken out (`OOT_WORK`, `OOT_EXEFS`, `OOT_ROMFS`, `OOT_DUMP`, `A64_HOT_FUNCS`). Every file is already hashed by its name inside its folder, and the lifted C holds no folder path (checked: no work or game path in an earlier folder's `sa/gen` or `sa/image`). In `translate()`, when the folder must lift: the share key is taken from the state the lift starts from (as the folder that saved it took it), a snapshot is restored if there is one (its C must not be cut short, `lifted_cut_short`, or it is forgotten), and the folder's own key is taken again afterwards from the restored boundaries and hot pieces, so the next build in that folder skips the lift as after a lift of its own. After a real lift whose inputs did not change while it ran, and whose C is whole, it is saved. - What decides reuse: everything the lift key covers (the game's code, the translator's sources, the HLE list, the hot list, the chunk boundaries and hot pieces as the lift starts, the `A64_`/`OOT_`/`RX_` settings other than the compile's, the Python), minus the two folders' paths. chunks.txt / hot_pieces.txt: the key holds their state before the lift, and the restore brings the state after it, so the new folder adopts the same boundaries and its objects match the saving folder's (with the object store, they come from there too). - dev: `scripts/dev_packs.py run`: its builds use `~/.cache/longshot-lifts` (`%LOCALAPPDATA%\Longshot\lifts`) unless `LONGSHOT_LIFT_STORE` is set (`0`: none). - docs: `docs/development.md` (Building's switch table, *Testing a change against a base*). - tests: `tests/test_lift_store.py` (new): save and restore, a snapshot not whole or cut short never used, prune, and the share key equal across folders where the lift key differs. `tests/test_lift_cache.py`'s check of `translate()`'s text still holds: the lift's steps keep their order and wording, one level deeper. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `LONGSHOT_LIFT_STORE=<dir>` | Finished translations kept for every work folder | off (`make_game.py`); on in `dev_packs.py` | Off for players: one work folder for every version, and a version that changes the translator lifts anyway. On for development | developers with several checkouts or bases | one work folder (players); a change to the translator | Windows (plain copies) | | `LONGSHOT_LIFT_STORE_KEEP=<n>` | Snapshots kept | 3 | keep: ~1.6 GB each where copies aren't clones | | | | ## Measurements main: 0caf0618 (2026-11-14) this PR: perf/lift-store @ 7bd98427 Hardware: Ryzen 9 7900X (24 threads), 30 GB, NVMe SSD (btrfs), CachyOS Linux Scene: no game run: `dev_packs.py base` of one commit into a new folder after an earlier folder saved its lift Method: the build's "where the time went" table. Not measured yet: the measurement builds were stopped when the PC ran low on memory. Baseline to be: `this PR @ 7bd98427, LONGSHOT_LIFT_STORE=0`, same commit, so the binaries can be compared. How to reproduce: two `dev_packs.py base` runs of one commit into two new folders with one `LONGSHOT_LIFT_STORE`; the second's lift line and its `longshot` / `longshot_vk.so` against a run at `LONGSHOT_LIFT_STORE=0`. | | this PR @ 7bd98427, LONGSHOT_LIFT_STORE=0 (baseline) | this PR @ 7bd98427, snapshot from an earlier folder | |---|---|---| | lift | not run yet (76 s cold on perf/object-store @ 34b7e787) | not run yet | | total | not run yet | not run yet | | binaries byte-identical | | not run yet | ``` not run yet ``` ## Testing **Tested:** nothing in the game yet (above). No checks were run on the author's PC (`tests/run_all.py`, including the new `tests/test_lift_store.py`, is left to CI). The checks that read `make_game.py`'s text (`tests/test_lift_cache.py` `check_key_after_lift`, `tests/test_kept_translation.py`, `tests/test_dev_packs.py`) were read against the change. **Not tested:** any build with the store; Windows; two builds saving the same snapshot at once; a restore across drives. **Needs other testers:** - Windows: two `dev_packs.py base` runs of one commit into new folders: the second prints `the game was translated with these tools in another work folder` and its game starts (request to open) ## Screenshots Nothing on screen. ## Risk and rollback A snapshot used where the lift would have written other C would build a wrong game: reuse is gated by everything the existing lift key covers, minus only the two folders' paths, and a snapshot is checked whole before use. Off unless `LONGSHOT_LIFT_STORE` is set (dev_packs: `LONGSHOT_LIFT_STORE=0`). Reverting the commit is enough. ## Reviewer notes - The share key relies on the lift not reading the game folder (`OOT_ROMFS`, `OOT_DUMP`) beyond what the lift key already ignores today. - `LONGSHOT_*` variables the lift reads (`LONGSHOT_LIFT_JOBS`, `LONGSHOT_LIFT_NO_FORK`) are not in either key, as today: they change how the lift runs, not what it writes. ## Issues See https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/371 ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). (no tests run on the author's PC; CI runs them) - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (not yet: the builds were stopped by low memory) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [ ] A game-free test added for the fix, failing on `main` and passing here. (added, tests/test_lift_store.py; not run on the author's PC, CI runs it) - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017vw8e34avQ5cTCWow16MXn

HumanBean added 2 commits 2026-11-14 05:57:13 +00:00

perf(build): one object store for every work folder ... 8a7bf09d45

LONGSHOT_OBJ_STORE=<dir> (nx2/object_store.py): nx2/build.py takes the
objects a build needs and its work folder lacks from the store, and gives
the store the folder's own, by hard links (a copy renamed in across
drives). The store has a folder per compiler and system headers (the work
folder's system_headers.txt). The folder's clean-up only removes its own
links; the store has its own (make_game.py, after each build): objects no
folder holds, unused for 14 days or oldest first over 8 GB. A link that
fails takes its objects out of the store. scripts/dev_packs.py turns it on
for its bases and packs. The object keys are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017vw8e34avQ5cTCWow16MXn

perf(build): a lift reused across work folders ... 7bd984273c

LONGSHOT_LIFT_STORE=<dir> (nx2/lift_store.py): after a lift, make_game.py
saves its lifted C, image, chunk boundaries and hot pieces under
lift_share_key, the lift key with the work and game folders' paths taken
out of the lift's environment, taken as the lift starts. A new work folder
whose share key matches copies them (copy-on-write clones where the file
system has them; never hard links) instead of translating the game, and
takes its own lift key after. A snapshot is written whole under a
temporary name and checked file by file before use; one with C cut short
is let go. The newest 3 are kept. scripts/dev_packs.py turns it on.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017vw8e34avQ5cTCWow16MXn

mooki added the

kind

performance

priority

high

labels 2026-11-14 06:29:04 +00:00

mooki requested changes 2026-11-14 06:40:38 +00:00

mooki left a comment

• Copy link
• Copy Markdown

Read head 7bd98427 against main 0caf0618: the complete two-commit diff, description, empty comments and reviews, lift inputs, snapshot readers and build consumers; ran eight game-free test files on Linux. Seven passed; test_lift_cleanup_error.py has five errors. No game, real lift, full build or timing run was made.

What's good

• The repeated work exists on main: a new work folder cannot match the old folder's lift key. The proposed store reuses translation output, not compiled objects or player data.
• The normal fingerprint is content-based. It includes the extracted game-code files, imported translator sources, nx2/oot_patches.py, the other imported patch modules, HLE names, hot_funcs.txt, chunks.txt, hot_pieces.txt, lift settings and Python. The three recent fresh-lift cases, at c0442078, 76699aad and 1ed75e61, changed oot_patches.py; this code would reject those old snapshots when the inputs remain stable during the operation. That conclusion is traced from the key and the retained build records, not a replay of those builds.
• Lift files are copies, with copy-on-write clones where supported, never hard links. Editing or deleting one restored folder does not change the snapshot or another restored folder. A save publishes its directory only after all copies and the manifest finish; an interrupted save's temporary directory is not a cache hit.

Must change

1. Keep the shared key tied to the inputs throughout restore and save. In nx2/make_game.py, lines 2333-2338, a hit is selected using the old inputs, but its copied C is stamped unconditionally with a newly computed key. If a translator file, patch table or hot list changes during the copy, old C is certified as belonging to the new inputs, and the following build skips the required lift. The miss branch has the converse window: an edit after share is computed but before inputs at line 2340 is accepted by the later comparison, so lines 2352-2356 save the new translation under the old shared key. Main's real-lift path already protects against changed inputs; the new store paths need that same protection starting before shared-key capture. Reject or retry when non-planning inputs change, rather than publishing a key for other inputs. Add tests that change a tracked input during restore and between shared-key capture and a miss, and check the next build cannot reuse mismatched C. Source-traced, not reproduced.

2. Hash path-backed lift inputs before removing their folder names. In nx2/make_game.py, lines 1711-1722, normalization is applied to every environment value, not just the five documented folder settings. A64_DEVIRT=<work>/devirt.txt consequently becomes the same value in two folders even when their tables differ. lift_key never reads either table: its work-folder inputs are only hot_funcs.txt and the two planning files. The consumer is real: third_party/a64-kit/lift/a64_lift.py, lines 908-917 and 982, reads that table, and lines 1693-1703 use it to choose emitted direct-call fast paths. An alternate A64_HOT_FUNCS=<work>/custom-hot.txt has the same problem when the normal hot_funcs.txt is absent. This newly permits cross-folder reuse for different requested translation inputs; it is separate from main's pre-existing same-path edit limitation. Hash the files actually selected by these settings, or keep their paths distinct until their contents are covered. Add two-folder tests with different tables and different custom hot lists. Source-traced, not reproduced.

3. Update the fresh-lift regression fixture for the new dependency. The unconditional _lift_store.root() at nx2/make_game.py, line 2332, breaks tests/test_lift_cleanup_error.py, which executes the actual fresh-lift branch with an explicit namespace. That namespace has no _lift_store. Running python3 -I -B tests/test_lift_cleanup_error.py on this head gives Ran 9 tests and FAILED (errors=5), each failing at line 2332 with NameError: name '_lift_store' is not defined. The affected cases cover a first lift, successful cleanup, missing-file races, a failed lift and an input changed during the lift. Keep those checks, supply the new dependencies with temporary store paths, and cover the hit/miss cases rather than weakening the assertions. Run on this head.

Measurements and smaller points

• There is no completed store-enabled build or before/after measurement in the description. The earlier cold-lift duration does not establish a saving here. Keep WIP until the named commit is compared with its store disabled and enabled on the same PC, with at least two alternating runs each, every result, median and spread. Compare the generated files and final binaries privately, and retain the interrupted attempts. No speed-up is established by this review.
• The new test checks copying, a size mismatch, cut-short C, pruning and one boundary-key change. It does not exercise simultaneous publishers, interruption, plain-copy fallback or the real translate() cache-hit branch. A normal concurrent save uses separate process-named temporary directories and only publishes a complete directory; that is a source trace, not a concurrency test.
• A late restore failure is not a complete rollback: nx2/lift_store.py, lines 151-164, can replace chunks.txt, fail on hot_pieces.txt, and remove the copied C while retaining the changed first planning file. The fallback then lifts from that remaining plan. Exercise this failure point explicitly; the current test fails before either planning file is replaced.
• The description leaves the checks to CI, but this head's AGENTS.md says Forgejo does not run source checks yet. Record actual local results instead. The first test_lift_cache.py attempt here stopped on missing NumPy in the system Python; it passed with the existing dependency Python. That prerequisite failure is not a PR defect.

Overlaps

• #371 is the direct parent, 8a7bf09d. Its object store is included in this diff; the lift-only change is the top commit. #30 removes the profile pathname from object keys; without that change, identical restored hot C can still recompile in each new folder. It merges pairwise with this head.
• #37 conflicts in nx2/build.py after successful compilation: its rekey_depfiles(vk_jobs) meets the object-store publication. Preserve both, finalizing dependency keys before sharing the objects.
• #31 changes chunk generation, and #38 changes the lifter. Both merge pairwise; their changed translator bytes invalidate this store's key. #370 changes shader fingerprints and preparation decisions, not this lift's identity, and also merges pairwise.
• #357 and #358 are already in this base. #360 changes the earlier game-file stage; its Python merges, but its documentation conflicts with the merged Windows lift-worker row. This head itself merges cleanly with main 0caf0618. No combined-stack build was run.

Not checked

No Windows execution, cross-drive restore, simultaneous-process or interruption test, whole suite, game, real lift or build-output comparison. All exercised checks were Python-only, with no GCC or Clang fixture to compile. The passing files were test_lift_store, test_object_store, test_lift_cache, test_kept_translation, test_lift_cut_short, test_dev_packs and test_failed_compile_keeps_objects. The runtime and shader-preparation messages in the pack test come from stand-ins, not a game run.

Read head `7bd98427` against main `0caf0618`: the complete two-commit diff, description, empty comments and reviews, lift inputs, snapshot readers and build consumers; ran eight game-free test files on Linux. Seven passed; `test_lift_cleanup_error.py` has five errors. No game, real lift, full build or timing run was made. ### What's good - The repeated work exists on main: a new work folder cannot match the old folder's lift key. The proposed store reuses translation output, not compiled objects or player data. - The normal fingerprint is content-based. It includes the extracted game-code files, imported translator sources, `nx2/oot_patches.py`, the other imported patch modules, HLE names, `hot_funcs.txt`, `chunks.txt`, `hot_pieces.txt`, lift settings and Python. The three recent fresh-lift cases, at `c0442078`, `76699aad` and `1ed75e61`, changed `oot_patches.py`; this code would reject those old snapshots when the inputs remain stable during the operation. That conclusion is traced from the key and the retained build records, not a replay of those builds. - Lift files are copies, with copy-on-write clones where supported, never hard links. Editing or deleting one restored folder does not change the snapshot or another restored folder. A save publishes its directory only after all copies and the manifest finish; an interrupted save's temporary directory is not a cache hit. ### Must change 1. **Keep the shared key tied to the inputs throughout restore and save.** In `nx2/make_game.py`, lines 2333-2338, a hit is selected using the old inputs, but its copied C is stamped unconditionally with a newly computed key. If a translator file, patch table or hot list changes during the copy, old C is certified as belonging to the new inputs, and the following build skips the required lift. The miss branch has the converse window: an edit after `share` is computed but before `inputs` at line 2340 is accepted by the later comparison, so lines 2352-2356 save the new translation under the old shared key. Main's real-lift path already protects against changed inputs; the new store paths need that same protection starting before shared-key capture. Reject or retry when non-planning inputs change, rather than publishing a key for other inputs. Add tests that change a tracked input during restore and between shared-key capture and a miss, and check the next build cannot reuse mismatched C. **Source-traced, not reproduced.** 2. **Hash path-backed lift inputs before removing their folder names.** In `nx2/make_game.py`, lines 1711-1722, normalization is applied to every environment value, not just the five documented folder settings. `A64_DEVIRT=<work>/devirt.txt` consequently becomes the same value in two folders even when their tables differ. `lift_key` never reads either table: its work-folder inputs are only `hot_funcs.txt` and the two planning files. The consumer is real: `third_party/a64-kit/lift/a64_lift.py`, lines 908-917 and 982, reads that table, and lines 1693-1703 use it to choose emitted direct-call fast paths. An alternate `A64_HOT_FUNCS=<work>/custom-hot.txt` has the same problem when the normal `hot_funcs.txt` is absent. This newly permits cross-folder reuse for different requested translation inputs; it is separate from main's pre-existing same-path edit limitation. Hash the files actually selected by these settings, or keep their paths distinct until their contents are covered. Add two-folder tests with different tables and different custom hot lists. **Source-traced, not reproduced.** 3. **Update the fresh-lift regression fixture for the new dependency.** The unconditional `_lift_store.root()` at `nx2/make_game.py`, line 2332, breaks `tests/test_lift_cleanup_error.py`, which executes the actual fresh-lift branch with an explicit namespace. That namespace has no `_lift_store`. Running `python3 -I -B tests/test_lift_cleanup_error.py` on this head gives `Ran 9 tests` and `FAILED (errors=5)`, each failing at line 2332 with `NameError: name '_lift_store' is not defined`. The affected cases cover a first lift, successful cleanup, missing-file races, a failed lift and an input changed during the lift. Keep those checks, supply the new dependencies with temporary store paths, and cover the hit/miss cases rather than weakening the assertions. **Run on this head.** ### Measurements and smaller points - There is no completed store-enabled build or before/after measurement in the description. The earlier cold-lift duration does not establish a saving here. Keep WIP until the named commit is compared with its store disabled and enabled on the same PC, with at least two alternating runs each, every result, median and spread. Compare the generated files and final binaries privately, and retain the interrupted attempts. No speed-up is established by this review. - The new test checks copying, a size mismatch, cut-short C, pruning and one boundary-key change. It does not exercise simultaneous publishers, interruption, plain-copy fallback or the real `translate()` cache-hit branch. A normal concurrent save uses separate process-named temporary directories and only publishes a complete directory; that is a source trace, not a concurrency test. - A late restore failure is not a complete rollback: `nx2/lift_store.py`, lines 151-164, can replace `chunks.txt`, fail on `hot_pieces.txt`, and remove the copied C while retaining the changed first planning file. The fallback then lifts from that remaining plan. Exercise this failure point explicitly; the current test fails before either planning file is replaced. - The description leaves the checks to CI, but this head's `AGENTS.md` says Forgejo does not run source checks yet. Record actual local results instead. The first `test_lift_cache.py` attempt here stopped on missing NumPy in the system Python; it passed with the existing dependency Python. That prerequisite failure is not a PR defect. ### Overlaps - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/371 is the direct parent, `8a7bf09d`. Its object store is included in this diff; the lift-only change is the top commit. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/30 removes the profile pathname from object keys; without that change, identical restored hot C can still recompile in each new folder. It merges pairwise with this head. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/37 conflicts in `nx2/build.py` after successful compilation: its `rekey_depfiles(vk_jobs)` meets the object-store publication. Preserve both, finalizing dependency keys before sharing the objects. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/31 changes chunk generation, and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/38 changes the lifter. Both merge pairwise; their changed translator bytes invalidate this store's key. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/370 changes shader fingerprints and preparation decisions, not this lift's identity, and also merges pairwise. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/357 and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/358 are already in this base. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/360 changes the earlier game-file stage; its Python merges, but its documentation conflicts with the merged Windows lift-worker row. This head itself merges cleanly with main `0caf0618`. No combined-stack build was run. ### Not checked No Windows execution, cross-drive restore, simultaneous-process or interruption test, whole suite, game, real lift or build-output comparison. All exercised checks were Python-only, with no GCC or Clang fixture to compile. The passing files were `test_lift_store`, `test_object_store`, `test_lift_cache`, `test_kept_translation`, `test_lift_cut_short`, `test_dev_packs` and `test_failed_compile_keeps_objects`. The runtime and shader-preparation messages in the pack test come from stand-ins, not a game run.

HumanBean changed title from WIP: perf(build): a lift reused across work folders, so a new one skips translating the same game to perf(build): a lift reused across work folders, so a new one skips translating the same game 2026-11-14 15:15:08 +00:00

mooki added the

area/install

label 2026-11-14 16:27:45 +00:00

Some checks failed

Hide checks

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Third-party parts / As their notes say (pull_request) Successful in 7s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 4s

Details

PR description / Description follows the template (pull_request_target) Failing after 12s

Details

This pull request has changes conflicting with the target branch.

• docs/development.md
• nx2/make_game.py
• scripts/dev_packs.py

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title perf(build): a lift reused across work folders, so a new one skips translating the same game (#372)

Merge commit body Reviewed-on: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/372

perf(build): a lift reused across work folders, so a new one skips translating the same game (#372) Reviewed-on: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/372 Copy merge commit message

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'perf/lift-store:HumanBean-perf/lift-store'

git switch 'HumanBean-perf/lift-store'

Sign in to join this conversation.

Reviewers

No reviewers

mooki

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Linux only

  

platform/steam-deck

Steam Deck

  

platform/windows

Windows only

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!372

No description provided.