perf(build): object keys name a play profile by its contents, not its path in the work folder

dyst commented

2026-11-15 02:40:05 +00:00

Member

Type: perf · Risk: low · Tested on: Windows 11, game-free checks · Needs: a timed two-folder build with the object store (Windows or Linux, any GPU)

Why

The object store (#371) and scripts/dev_packs.py let a new work folder take the objects another folder already compiled. Two kinds can't be shared today: the eight hot pieces and the renderer's own objects. make_game.py hands build.py both play profiles from inside the work folder (RX_PGO=<work>/longshot_pieces.prof, and since #569 RX_VK_PGO=<work>/longshot_vk.shipped.profdata), and the object key holds that path, so every new folder compiles them again. The hot pieces are the longest single compiles of a build.

Split out of #30 at HumanBean's request in its review.

What changed

Object keys now name a play profile by its contents only. build.py already puts a digest of the profile's bytes beside its path (-DLONGSHOT_PROFILE_SHA1, -DLONGSHOT_VK_PROFILE_SHA1), so the path is replaced by <profile> in the text the key is made from, as --sysroot= already is.

build: nx2/build.py flag_text: -fprofile-instr-use=<path> becomes -fprofile-instr-use=<profile>. The compile itself still gets the real path.
tests: tests/test_profile_path_key.py (new): two folders' profile paths give one key, another profile's contents another (for both defines), and every "-fprofile-instr-use=" + in build.py carries its digest in the same list, so the key still changes when a profile does. It fails on main.

Measurements

Verdict: game-free: the eight hot pieces' and the renderer objects' keys are now the same in a second work folder with the same profile, where main gives them another key, so the object store can share them; another profile still changes the key. The compile time this saves in a second folder is not timed yet: it will be added here when the two-folder builds under Must be tested are run, by me or by anyone who runs them first.

Hardware: Ryzen 9 3900X, 64 GB, Windows 11 (the key needs no GPU)
Scene: none (a build change, game-free, no dump)
Method: flag_text() from each tree's own nx2/build.py, for a hot piece's flags and a renderer object's, with the profile in two work folders and with another profile's digest; the key is the same or not, so there is nothing to take a median of
How to reproduce: the script below, python -I -B profile_key_compare.py <tree>, with a checkout of main @ 1310ecc6 and of this PR @ 14e623ba

The object's key	Before: main @ `1310ecc6`	After: this PR @ `14e623ba`
a hot piece, the same profile in a second work folder	another key: compiled again	the same key: shared
a renderer object, the same profile in a second work folder	another key: compiled again	the same key: shared
either, another profile's contents	another key	another key

== longshot-main @ 1310ecc6
a hot piece (RX_PGO)             second folder, same profile: another key     another profile: another key
a renderer object (RX_VK_PGO)    second folder, same profile: another key     another profile: another key
== longshot-rb-660 @ 14e623ba
a hot piece (RX_PGO)             second folder, same profile: same key        another profile: another key
a renderer object (RX_VK_PGO)    second folder, same profile: same key        another profile: another key

profile_key_compare.py

import importlib.util, sys
from pathlib import Path
root = Path(sys.argv[1]).resolve()
sys.path[:0] = [str(root / "nx2"), str(root / "third_party" / "a64-kit" / "tools")]
spec = importlib.util.spec_from_file_location("build_cmp", root / "nx2" / "build.py")
b = importlib.util.module_from_spec(spec); spec.loader.exec_module(b)
for what, define, name in (("a hot piece (RX_PGO)", "-DLONGSHOT_PROFILE_SHA1=", "longshot_pieces.prof"),
                           ("a renderer object (RX_VK_PGO)", "-DLONGSHOT_VK_PROFILE_SHA1=", "longshot_vk.shipped.profdata")):
    one = ["-O2", "-fprofile-instr-use=C:/a/work/" + name, define + "0123456789abcdef"]
    two = ["-O2", "-fprofile-instr-use=C:/b/work/" + name, define + "0123456789abcdef"]
    other = ["-O2", "-fprofile-instr-use=C:/a/work/" + name, define + "fedcba9876543210"]
    print("%-32s second folder, same profile: %-15s another profile: %s" % (
        what, "same key" if b.flag_text(one) == b.flag_text(two) else "another key",
        "same key" if b.flag_text(one) == b.flag_text(other) else "another key"))

Flags and settings

None: no switch, menu row, ini key or build flag changes.

Testing

Tested: tests/test_profile_path_key.py on this branch @ 14e623ba (passes) and against main's flag_text (the keys differ across folders); the game-free key comparison under Measurements, run again on main @ 1310ecc6 and this PR @ 14e623ba (the same results as before the rebase). The 41 checks that read this diff (python -I -B tests/run_all.py --jobs 6 --exact with the checks under tests/ and nx2/test_*.py that name a changed file, and the ones it adds or changes), not the whole suite, on this branch @ 14e623ba (rebased onto main @ 1310ecc6), Windows 11 Pro, with the Builder's llvm-mingw clang (the v2.0.14 tools): 34 passed, 6 failed, 1 skipped (a tool this PC lacks). The 3 that fail also fail on main @ 1310ecc6 (the whole suite there: 500 passed, 17 failed): test_appimage_player_build, test_linux_preflight, test_vk_pgo. test_dll, test_driver_cache_folder, test_pipeline_tour are skipped on Windows in a whole run (Linux-only or a full game build); named, as here, they fail on main @ 1310ecc6 too. The checks: test_dll, test_appimage_licenses, test_appimage_player_build, test_bake_ds, test_bake_skeleton, test_build_cleanup_scope, test_build_hash_cache, test_build_times, test_compile_retry_jobs, test_compiler_failure, test_cpm_discovery, test_cpu_target, test_dev_packs, test_dlss_sdk_cache, test_driver_cache_folder, test_exclusive_marks, test_failed_compile_keeps_objects, test_fpx_sticky, test_kept_translation, test_late_fix_build, test_lift_cache, test_link_path_comma, test_linux_preflight, test_make_game_ngx, test_mem_gate, test_mem_gate_children, test_mem_gate_leases, test_mem_limit, test_pipeline_tour, test_prepare_at_build, test_profile_path_key, test_program_cache_format, test_recorder_sync, test_shader_stamp_cache, test_shader_stamp_inputs, test_split_giants, test_stale_hot_pieces, test_switches_removed, test_third_party_objects, test_utf8_paths, test_vk_pgo.

Not tested: a real build: that the second folder takes the hot pieces and the renderer's objects from the store; Linux.

Must be tested:

Windows or Linux, any GPU, by hand: anyone with a build tree and their own game files (I will run it too and add the times here): two dev_packs.py base folders of one commit over one LONGSHOT_OBJ_STORE, on main @ 1310ecc6 and on this PR @ 14e623ba, expecting the second folder's @@file lines to show the code_hot_* pieces and the vk_* objects as cached on this PR, where on main they are new; with both builds' "where the time went" tables.

Risks

What could break: a different profile at the same path still changes the key (the digest beside the path), so nothing stale is reused. A profile flag added later without its digest beside it would lose that; the new check fails on one.

Turning it off: revert the commit.

Compatibility: the eight hot pieces' and the renderer objects' keys change once. A player's next build compiles them again, and one whose last build kept its translation (--kept-lift) translates the game once more: kept_part is made from the same flag text, so the kept list no longer fits (KEPT_STALE). Saves, the options file, the shader caches and the vk ABI are unchanged.

Checklist

Everyone

One topic.
The title is a Conventional Commit (type(scope): description) of the status line's type.
Why gives the reason plainly and links the issues that fit (the tracker searched).
Measurements has a verdict, or one line saying why there is nothing to measure.
Testing names what was tested, what wasn't, and what must still be tested, where and how.
Rebased on current main.
python -I -B tests/run_all.py passes (any failure that also fails on main is named).
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
A game-free test added for the fix, failing on main and passing here.

🤖 Generated with Claude Code

**Type:** perf · **Risk:** low · **Tested on:** Windows 11, game-free checks · **Needs:** a timed two-folder build with the object store (Windows or Linux, any GPU) ## Why The object store (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/371) and `scripts/dev_packs.py` let a new work folder take the objects another folder already compiled. Two kinds can't be shared today: the eight hot pieces and the renderer's own objects. `make_game.py` hands `build.py` both play profiles from inside the work folder (`RX_PGO=<work>/longshot_pieces.prof`, and since https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/569 `RX_VK_PGO=<work>/longshot_vk.shipped.profdata`), and the object key holds that path, so every new folder compiles them again. The hot pieces are the longest single compiles of a build. Split out of https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/30 at HumanBean's request in its review. ## What changed Object keys now name a play profile by its contents only. `build.py` already puts a digest of the profile's bytes beside its path (`-DLONGSHOT_PROFILE_SHA1`, `-DLONGSHOT_VK_PROFILE_SHA1`), so the path is replaced by `<profile>` in the text the key is made from, as `--sysroot=` already is. - build: `nx2/build.py` `flag_text`: `-fprofile-instr-use=<path>` becomes `-fprofile-instr-use=<profile>`. The compile itself still gets the real path. - tests: `tests/test_profile_path_key.py` (new): two folders' profile paths give one key, another profile's contents another (for both defines), and every `"-fprofile-instr-use=" +` in `build.py` carries its digest in the same list, so the key still changes when a profile does. It fails on main. ## Measurements **Verdict:** game-free: the eight hot pieces' and the renderer objects' keys are now the same in a second work folder with the same profile, where main gives them another key, so the object store can share them; another profile still changes the key. The compile time this saves in a second folder is not timed yet: it will be added here when the two-folder builds under Must be tested are run, by me or by anyone who runs them first. Hardware: Ryzen 9 3900X, 64 GB, Windows 11 (the key needs no GPU) Scene: none (a build change, game-free, no dump) Method: `flag_text()` from each tree's own `nx2/build.py`, for a hot piece's flags and a renderer object's, with the profile in two work folders and with another profile's digest; the key is the same or not, so there is nothing to take a median of How to reproduce: the script below, `python -I -B profile_key_compare.py <tree>`, with a checkout of main @ 1310ecc6 and of this PR @ 14e623ba | The object's key | Before: main @ 1310ecc6 | After: this PR @ 14e623ba | |---|---|---| | a hot piece, the same profile in a second work folder | another key: compiled again | the same key: shared | | a renderer object, the same profile in a second work folder | another key: compiled again | the same key: shared | | either, another profile's contents | another key | another key | ``` == longshot-main @ 1310ecc6 a hot piece (RX_PGO) second folder, same profile: another key another profile: another key a renderer object (RX_VK_PGO) second folder, same profile: another key another profile: another key == longshot-rb-660 @ 14e623ba a hot piece (RX_PGO) second folder, same profile: same key another profile: another key a renderer object (RX_VK_PGO) second folder, same profile: same key another profile: another key ``` <details><summary>profile_key_compare.py</summary> ```python import importlib.util, sys from pathlib import Path root = Path(sys.argv[1]).resolve() sys.path[:0] = [str(root / "nx2"), str(root / "third_party" / "a64-kit" / "tools")] spec = importlib.util.spec_from_file_location("build_cmp", root / "nx2" / "build.py") b = importlib.util.module_from_spec(spec); spec.loader.exec_module(b) for what, define, name in (("a hot piece (RX_PGO)", "-DLONGSHOT_PROFILE_SHA1=", "longshot_pieces.prof"), ("a renderer object (RX_VK_PGO)", "-DLONGSHOT_VK_PROFILE_SHA1=", "longshot_vk.shipped.profdata")): one = ["-O2", "-fprofile-instr-use=C:/a/work/" + name, define + "0123456789abcdef"] two = ["-O2", "-fprofile-instr-use=C:/b/work/" + name, define + "0123456789abcdef"] other = ["-O2", "-fprofile-instr-use=C:/a/work/" + name, define + "fedcba9876543210"] print("%-32s second folder, same profile: %-15s another profile: %s" % ( what, "same key" if b.flag_text(one) == b.flag_text(two) else "another key", "same key" if b.flag_text(one) == b.flag_text(other) else "another key")) ``` </details> ### Flags and settings None: no switch, menu row, ini key or build flag changes. ## Testing **Tested:** `tests/test_profile_path_key.py` on this branch @ 14e623ba (passes) and against main's `flag_text` (the keys differ across folders); the game-free key comparison under Measurements, run again on main @ 1310ecc6 and this PR @ 14e623ba (the same results as before the rebase). The 41 checks that read this diff (`python -I -B tests/run_all.py --jobs 6 --exact` with the checks under `tests/` and `nx2/test_*.py` that name a changed file, and the ones it adds or changes), not the whole suite, on this branch @ 14e623ba (rebased onto main @ 1310ecc6), Windows 11 Pro, with the Builder's llvm-mingw clang (the v2.0.14 tools): 34 passed, 6 failed, 1 skipped (a tool this PC lacks). The 3 that fail also fail on main @ 1310ecc6 (the whole suite there: 500 passed, 17 failed): test_appimage_player_build, test_linux_preflight, test_vk_pgo. test_dll, test_driver_cache_folder, test_pipeline_tour are skipped on Windows in a whole run (Linux-only or a full game build); named, as here, they fail on main @ 1310ecc6 too. The checks: test_dll, test_appimage_licenses, test_appimage_player_build, test_bake_ds, test_bake_skeleton, test_build_cleanup_scope, test_build_hash_cache, test_build_times, test_compile_retry_jobs, test_compiler_failure, test_cpm_discovery, test_cpu_target, test_dev_packs, test_dlss_sdk_cache, test_driver_cache_folder, test_exclusive_marks, test_failed_compile_keeps_objects, test_fpx_sticky, test_kept_translation, test_late_fix_build, test_lift_cache, test_link_path_comma, test_linux_preflight, test_make_game_ngx, test_mem_gate, test_mem_gate_children, test_mem_gate_leases, test_mem_limit, test_pipeline_tour, test_prepare_at_build, test_profile_path_key, test_program_cache_format, test_recorder_sync, test_shader_stamp_cache, test_shader_stamp_inputs, test_split_giants, test_stale_hot_pieces, test_switches_removed, test_third_party_objects, test_utf8_paths, test_vk_pgo. **Not tested:** a real build: that the second folder takes the hot pieces and the renderer's objects from the store; Linux. **Must be tested:** - Windows or Linux, any GPU, by hand: anyone with a build tree and their own game files (I will run it too and add the times here): two `dev_packs.py base` folders of one commit over one `LONGSHOT_OBJ_STORE`, on main @ 1310ecc6 and on this PR @ 14e623ba, expecting the second folder's `@@file` lines to show the `code_hot_*` pieces and the `vk_*` objects as cached on this PR, where on main they are new; with both builds' "where the time went" tables. ## Risks **What could break:** a different profile at the same path still changes the key (the digest beside the path), so nothing stale is reused. A profile flag added later without its digest beside it would lose that; the new check fails on one. **Turning it off:** revert the commit. **Compatibility:** the eight hot pieces' and the renderer objects' keys change once. A player's next build compiles them again, and one whose last build kept its translation (`--kept-lift`) translates the game once more: `kept_part` is made from the same flag text, so the kept list no longer fits (`KEPT_STALE`). Saves, the options file, the shader caches and the vk ABI are unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit

2026-11-15 02:40:05 +00:00

perf(build): object keys name a play profile by its contents, not its path in the work folder

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

42a4e68975

make_game.py gives build.py both play profiles from the work folder
(RX_PGO=<work>/longshot_pieces.prof for the eight hot pieces since the
lift kept them, RX_VK_PGO=<work>/longshot_vk.shipped.profdata for the
renderer's own objects since #569), and flag_text kept that path in the
object keys. So the object store (#371) and dev_packs.py could share
neither the hot pieces nor the renderer's objects with another work
folder: each new folder compiled them again. The -D..._PROFILE_SHA1 of
the profile's bytes beside the path is already in the key, so the path
becomes <profile>.

Split out of #30 at review (HumanBean).

tests/test_profile_path_key.py: two folders' paths give one key, another
profile's contents another; every -fprofile-instr-use= in build.py has
its contents' digest beside it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst referenced this pull request

2026-11-15 02:40:13 +00:00

perf(build): skip a link whose inputs and outputs are as the last one left them, and write compile_commands.json #30

dyst commented

2026-11-15 02:53:05 +00:00

Author

Member

Measurements now has a game-free comparison of the hot pieces' and the renderer objects' keys across two work folders, on main and here. Out of WIP so review and independent testing can start now: the measurement here is game-free (Measurements, with the script to rerun it), and the timed builds under Must be tested are not run yet. I will run them and edit this description with the numbers when they are done; anyone with a build tree and their own game files is welcome to run them first and post the results here.

dyst changed title from ~~WIP: perf(build): object keys name a play profile by its contents, not its path in the work folder~~ to perf(build): object keys name a play profile by its contents, not its path in the work folder

2026-11-15 02:53:09 +00:00

dyst referenced this pull request

2026-11-15 03:53:11 +00:00

perf(build): skip a link whose inputs and outputs are as the last one left them, and write compile_commands.json #30

dyst referenced this pull request

2026-11-15 03:53:14 +00:00

perf(build): skip a link whose inputs and outputs are as the last one left them, and write compile_commands.json #30

siantm referenced this pull request

2026-11-15 10:00:25 +00:00

WIP: fix(build): honor selected external hot-function profiles #754

mooki added the

labels

2026-11-15 13:57:25 +00:00

dyst force-pushed perf/profile-path-key from 42a4e68975

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Third-party parts / As their notes say (pull_request) Successful in 8s

Details

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

to 14e623ba1a

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s

Details

Third-party parts / As their notes say (pull_request) Successful in 9s

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

2026-11-15 19:01:15 +00:00

Compare

dyst commented

2026-11-15 19:01:28 +00:00

Author

Member

Rebased onto main @ 1310ecc6 (head 14e623ba, no conflicts; it also merges cleanly with main @ 7bacf846). The 41 checks that read this diff, not the whole suite (the agreed process now): 34 passed; the 3 that fail also fail on main @ 1310ecc6, and 3 Linux-only or game-build checks, skipped on Windows in a whole run, fail when named on main too (listed under Testing). The game-free key comparison was run again on both trees, the same results. No code change.

Rebased onto main @ 1310ecc6 (head 14e623ba, no conflicts; it also merges cleanly with main @ 7bacf846). The 41 checks that read this diff, not the whole suite (the agreed process now): 34 passed; the 3 that fail also fail on main @ 1310ecc6, and 3 Linux-only or game-build checks, skipped on Windows in a whole run, fail when named on main too (listed under Testing). The game-free key comparison was run again on both trees, the same results. No code change.

dyst referenced this pull request

2026-11-15 19:03:44 +00:00

perf(build): the hot code compiles within memory with no profile, its giant functions first #791

dyst referenced this pull request

2026-11-15 19:03:44 +00:00

perf(build): the hot code compiles within memory with no profile, its giant functions first #791

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped

Details

Third-party parts / The same as upstream (pull_request) Has been skipped

Details

Website / Website on Cloudflare (pull_request) Has been skipped

Details

CI / Conventional Commit title (pull_request) Successful in 3s

Details

CI / Workflow lint (actionlint) (pull_request) Successful in 6s

Details

Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s

Details

Third-party parts / As their notes say (pull_request) Successful in 9s

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'perf/profile-path-key:dyst-perf/profile-path-key'

git switch 'dyst-perf/profile-path-key'

Rows
Columns

perf(build): object keys name a play profile by its contents, not its path in the work folder #660

Why

What changed

Measurements

Flags and settings

Testing

Risks

Checklist

Everyone

When it applies

Checkout