MagicBeanDev/Longshot

Watch

Fork 98

perf(build): object keys name a play profile by its contents, not its path in the work folder #660

Open

dyst wants to merge 1 commit from dyst/Longshot:perf/profile-path-key into main

dyst commented

2026-11-15 02:40:05 +00:00

Member

Type: perf · Risk: low · Tested on: Windows 11, game-free checks · Needs: a timed two-folder build with the object store (Windows or Linux, any GPU)

Why

The object store (#371) and scripts/dev_packs.py let a new work folder take the objects another folder already compiled. Two kinds can't be shared today: the eight hot pieces and the renderer's own objects. make_game.py hands build.py both play profiles from inside the work folder (RX_PGO=<work>/longshot_pieces.prof, and since #569 RX_VK_PGO=<work>/longshot_vk.shipped.profdata), and the object key holds that path, so every new folder compiles them again. The hot pieces are the longest single compiles of a build.

Split out of #30 at HumanBean's request in its review.

What changed

Object keys now name a play profile by its contents only. build.py already puts a digest of the profile's bytes beside its path (-DLONGSHOT_PROFILE_SHA1, -DLONGSHOT_VK_PROFILE_SHA1), so the path is replaced by <profile> in the text the key is made from, as --sysroot= already is.

build: nx2/build.py flag_text: -fprofile-instr-use=<path> becomes -fprofile-instr-use=<profile>. The compile itself still gets the real path.
tests: tests/test_profile_path_key.py (new): two folders' profile paths give one key, another profile's contents another (for both defines), and every "-fprofile-instr-use=" + in build.py carries its digest in the same list, so the key still changes when a profile does. It fails on main.

Measurements

Verdict: game-free: the eight hot pieces' and the renderer objects' keys are now the same in a second work folder with the same profile, where main gives them another key, so the object store can share them; another profile still changes the key. The compile time this saves in a second folder is not timed yet: it will be added here when the two-folder builds under Must be tested are run, by me or by anyone who runs them first.

Hardware: Ryzen 9 3900X, 64 GB, Windows 11 (the key needs no GPU)
Scene: none (a build change, game-free, no dump)
Method: flag_text() from each tree's own nx2/build.py, for a hot piece's flags and a renderer object's, with the profile in two work folders and with another profile's digest; the key is the same or not, so there is nothing to take a median of
How to reproduce: the script below, python -I -B profile_key_compare.py <tree>, with a checkout of main @ 6c9edffe and of this PR @ 42a4e689

The object's key	Before: main @ `6c9edffe`	After: this PR @ `42a4e689`
a hot piece, the same profile in a second work folder	another key: compiled again	the same key: shared
a renderer object, the same profile in a second work folder	another key: compiled again	the same key: shared
either, another profile's contents	another key	another key

== longshot-main @ 6c9edffe
a hot piece (RX_PGO)             second folder, same profile: another key     another profile: another key
a renderer object (RX_VK_PGO)    second folder, same profile: another key     another profile: another key
== longshot-pgokey @ 42a4e689
a hot piece (RX_PGO)             second folder, same profile: same key        another profile: another key
a renderer object (RX_VK_PGO)    second folder, same profile: same key        another profile: another key

profile_key_compare.py

import importlib.util, sys
from pathlib import Path
root = Path(sys.argv[1]).resolve()
sys.path[:0] = [str(root / "nx2"), str(root / "third_party" / "a64-kit" / "tools")]
spec = importlib.util.spec_from_file_location("build_cmp", root / "nx2" / "build.py")
b = importlib.util.module_from_spec(spec); spec.loader.exec_module(b)
for what, define, name in (("a hot piece (RX_PGO)", "-DLONGSHOT_PROFILE_SHA1=", "longshot_pieces.prof"),
                           ("a renderer object (RX_VK_PGO)", "-DLONGSHOT_VK_PROFILE_SHA1=", "longshot_vk.shipped.profdata")):
    one = ["-O2", "-fprofile-instr-use=C:/a/work/" + name, define + "0123456789abcdef"]
    two = ["-O2", "-fprofile-instr-use=C:/b/work/" + name, define + "0123456789abcdef"]
    other = ["-O2", "-fprofile-instr-use=C:/a/work/" + name, define + "fedcba9876543210"]
    print("%-32s second folder, same profile: %-15s another profile: %s" % (
        what, "same key" if b.flag_text(one) == b.flag_text(two) else "another key",
        "same key" if b.flag_text(one) == b.flag_text(other) else "another key"))

Flags and settings

None: no switch, menu row, ini key or build flag changes.

Testing

Tested: tests/test_profile_path_key.py on this branch (passes) and against main's flag_text (the keys differ across folders); python -I -B tests/run_all.py on this branch and on main 6c9edffe, Windows 11 (with the Builder's llvm-mingw clang (12 at once), this branch: 530 passed, 16 failed, 35 skipped, no failure that main doesn't have; main @ 6c9edffe: 529 passed, 16 failed, 35 skipped (seven checks that timed out or were blocked by Windows Defender while other compiles ran passed when run again alone). The 16 that fail on main too: test_appimage_player_build, test_cpm_discovery, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_oot_state_tap, test_readback_late, test_seed_reader, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_vram_budget_row, test_walk_key).

Not tested: a real build: that the second folder takes the hot pieces and the renderer's objects from the store; Linux.

Must be tested:

Windows or Linux, any GPU, by hand: anyone with a build tree and their own game files (I will run it too and add the times here): two dev_packs.py base folders of one commit over one LONGSHOT_OBJ_STORE, on main @ 6c9edffe and on this PR, expecting the second folder's @@file lines to show the code_hot_* pieces and the vk_* objects as cached on this PR, where on main they are new; with both builds' "where the time went" tables.

Risks

What could break: a different profile at the same path still changes the key (the digest beside the path), so nothing stale is reused. A profile flag added later without its digest beside it would lose that; the new check fails on one.

Turning it off: revert the commit.

Compatibility: the eight hot pieces' and the renderer objects' keys change once. A player's next build compiles them again, and one whose last build kept its translation (--kept-lift) translates the game once more: kept_part is made from the same flag text, so the kept list no longer fits (KEPT_STALE). Saves, the options file, the shader caches and the vk ABI are unchanged.

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main.
python -I -B tests/run_all.py passes (any failure that also fails on main is named).
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
A game-free test added for the fix, failing on main and passing here.

🤖 Generated with Claude Code

**Type:** perf · **Risk:** low · **Tested on:** Windows 11, game-free checks · **Needs:** a timed two-folder build with the object store (Windows or Linux, any GPU) ## Why The object store (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/371) and `scripts/dev_packs.py` let a new work folder take the objects another folder already compiled. Two kinds can't be shared today: the eight hot pieces and the renderer's own objects. `make_game.py` hands `build.py` both play profiles from inside the work folder (`RX_PGO=<work>/longshot_pieces.prof`, and since https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/569 `RX_VK_PGO=<work>/longshot_vk.shipped.profdata`), and the object key holds that path, so every new folder compiles them again. The hot pieces are the longest single compiles of a build. Split out of https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/30 at HumanBean's request in its review. ## What changed Object keys now name a play profile by its contents only. `build.py` already puts a digest of the profile's bytes beside its path (`-DLONGSHOT_PROFILE_SHA1`, `-DLONGSHOT_VK_PROFILE_SHA1`), so the path is replaced by `<profile>` in the text the key is made from, as `--sysroot=` already is. - build: `nx2/build.py` `flag_text`: `-fprofile-instr-use=<path>` becomes `-fprofile-instr-use=<profile>`. The compile itself still gets the real path. - tests: `tests/test_profile_path_key.py` (new): two folders' profile paths give one key, another profile's contents another (for both defines), and every `"-fprofile-instr-use=" +` in `build.py` carries its digest in the same list, so the key still changes when a profile does. It fails on main. ## Measurements **Verdict:** game-free: the eight hot pieces' and the renderer objects' keys are now the same in a second work folder with the same profile, where main gives them another key, so the object store can share them; another profile still changes the key. The compile time this saves in a second folder is not timed yet: it will be added here when the two-folder builds under Must be tested are run, by me or by anyone who runs them first. Hardware: Ryzen 9 3900X, 64 GB, Windows 11 (the key needs no GPU) Scene: none (a build change, game-free, no dump) Method: `flag_text()` from each tree's own `nx2/build.py`, for a hot piece's flags and a renderer object's, with the profile in two work folders and with another profile's digest; the key is the same or not, so there is nothing to take a median of How to reproduce: the script below, `python -I -B profile_key_compare.py <tree>`, with a checkout of main @ 6c9edffe and of this PR @ 42a4e689 | The object's key | Before: main @ 6c9edffe | After: this PR @ 42a4e689 | |---|---|---| | a hot piece, the same profile in a second work folder | another key: compiled again | the same key: shared | | a renderer object, the same profile in a second work folder | another key: compiled again | the same key: shared | | either, another profile's contents | another key | another key | ``` == longshot-main @ 6c9edffe a hot piece (RX_PGO) second folder, same profile: another key another profile: another key a renderer object (RX_VK_PGO) second folder, same profile: another key another profile: another key == longshot-pgokey @ 42a4e689 a hot piece (RX_PGO) second folder, same profile: same key another profile: another key a renderer object (RX_VK_PGO) second folder, same profile: same key another profile: another key ``` <details><summary>profile_key_compare.py</summary> ```python import importlib.util, sys from pathlib import Path root = Path(sys.argv[1]).resolve() sys.path[:0] = [str(root / "nx2"), str(root / "third_party" / "a64-kit" / "tools")] spec = importlib.util.spec_from_file_location("build_cmp", root / "nx2" / "build.py") b = importlib.util.module_from_spec(spec); spec.loader.exec_module(b) for what, define, name in (("a hot piece (RX_PGO)", "-DLONGSHOT_PROFILE_SHA1=", "longshot_pieces.prof"), ("a renderer object (RX_VK_PGO)", "-DLONGSHOT_VK_PROFILE_SHA1=", "longshot_vk.shipped.profdata")): one = ["-O2", "-fprofile-instr-use=C:/a/work/" + name, define + "0123456789abcdef"] two = ["-O2", "-fprofile-instr-use=C:/b/work/" + name, define + "0123456789abcdef"] other = ["-O2", "-fprofile-instr-use=C:/a/work/" + name, define + "fedcba9876543210"] print("%-32s second folder, same profile: %-15s another profile: %s" % ( what, "same key" if b.flag_text(one) == b.flag_text(two) else "another key", "same key" if b.flag_text(one) == b.flag_text(other) else "another key")) ``` </details> ### Flags and settings None: no switch, menu row, ini key or build flag changes. ## Testing **Tested:** `tests/test_profile_path_key.py` on this branch (passes) and against main's `flag_text` (the keys differ across folders); `python -I -B tests/run_all.py` on this branch and on main 6c9edffe, Windows 11 (with the Builder's llvm-mingw clang (12 at once), this branch: 530 passed, 16 failed, 35 skipped, no failure that main doesn't have; main @ 6c9edffe: 529 passed, 16 failed, 35 skipped (seven checks that timed out or were blocked by Windows Defender while other compiles ran passed when run again alone). The 16 that fail on main too: test_appimage_player_build, test_cpm_discovery, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_oot_state_tap, test_readback_late, test_seed_reader, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_vram_budget_row, test_walk_key). **Not tested:** a real build: that the second folder takes the hot pieces and the renderer's objects from the store; Linux. **Must be tested:** - Windows or Linux, any GPU, by hand: anyone with a build tree and their own game files (I will run it too and add the times here): two `dev_packs.py base` folders of one commit over one `LONGSHOT_OBJ_STORE`, on main @ 6c9edffe and on this PR, expecting the second folder's `@@file` lines to show the `code_hot_*` pieces and the `vk_*` objects as cached on this PR, where on main they are new; with both builds' "where the time went" tables. ## Risks **What could break:** a different profile at the same path still changes the key (the digest beside the path), so nothing stale is reused. A profile flag added later without its digest beside it would lose that; the new check fails on one. **Turning it off:** revert the commit. **Compatibility:** the eight hot pieces' and the renderer objects' keys change once. A player's next build compiles them again, and one whose last build kept its translation (`--kept-lift`) translates the game once more: `kept_part` is made from the same flag text, so the kept list no longer fits (`KEPT_STALE`). Saves, the options file, the shader caches and the vk ABI are unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit

2026-11-15 02:40:05 +00:00

perf(build): object keys name a play profile by its contents, not its path in the work folder

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

42a4e68975

make_game.py gives build.py both play profiles from the work folder
(RX_PGO=<work>/longshot_pieces.prof for the eight hot pieces since the
lift kept them, RX_VK_PGO=<work>/longshot_vk.shipped.profdata for the
renderer's own objects since #569), and flag_text kept that path in the
object keys. So the object store (#371) and dev_packs.py could share
neither the hot pieces nor the renderer's objects with another work
folder: each new folder compiled them again. The -D..._PROFILE_SHA1 of
the profile's bytes beside the path is already in the key, so the path
becomes <profile>.

Split out of #30 at review (HumanBean).

tests/test_profile_path_key.py: two folders' paths give one key, another
profile's contents another; every -fprofile-instr-use= in build.py has
its contents' digest beside it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst referenced this pull request

2026-11-15 02:40:13 +00:00

perf(build): skip a link whose inputs and outputs are as the last one left them, and write compile_commands.json #30

dyst commented

2026-11-15 02:53:05 +00:00

Author

Member

Measurements now has a game-free comparison of the hot pieces' and the renderer objects' keys across two work folders, on main and here. Out of WIP so review and independent testing can start now: the measurement here is game-free (Measurements, with the script to rerun it), and the timed builds under Must be tested are not run yet. I will run them and edit this description with the numbers when they are done; anyone with a build tree and their own game files is welcome to run them first and post the results here.

dyst changed title from ~~WIP: perf(build): object keys name a play profile by its contents, not its path in the work folder~~ to perf(build): object keys name a play profile by its contents, not its path in the work folder

2026-11-15 02:53:09 +00:00

dyst referenced this pull request

2026-11-15 03:53:11 +00:00

perf(build): skip a link whose inputs and outputs are as the last one left them, and write compile_commands.json #30

dyst referenced this pull request

2026-11-15 03:53:14 +00:00

perf(build): skip a link whose inputs and outputs are as the last one left them, and write compile_commands.json #30