fix(build): an update keeps the headers its host and renderer objects were keyed by #2199
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2199
Loading…
Reference in a new issue
No description provided.
Delete branch "dazemc/Longshot:fix/object-reuse-cleanup"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
An update that changes any header under
host/orgpu/compiles every one of the Vulkan back end's own files and the host's again, even the ones that read nothing that changed, and the shared object store cannot stop it. The build in #1906's log says it in two lines:object store …: 0 objects from it, then561 files.How to see it on main: build a game (any Linux or Windows build: the launcher always sets
LONGSHOT_OBJ_STORE). Build it again from the same work folder with the same sources: the second build's log saysobject store …: 0 objects from itfor the host's and the back end's own objects, and its per-file lines show them compiled again. A developer's build does not show it:scripts/devtools/packs.py:509andscripts/devtools/build.py:457setLONGSHOT_KEEP_BUILD_FILES=1.Why: every back-end and host object is named by the headers its clang depfile lists (
renderer_deps, #37 and #673). A player's build ends inkeep_translation, which deletes the build folder and the depfiles with it:nx2/make_game.py:2483—shutil.rmtree(os.path.join(work, "sa", "build"), ignore_errors=True).nx2/build.py:555-556— with no depfile,renderer_depsreturns the whole-folders' hash instead:if not picked: return blob.nx2/build.py:1649— after each compile,rekey_depfilescaches the object under its depfile key:key = object_key(src, fl, renderer_deps(blob, roots, obj) + extras)thencache_put(obj, key).So the store and the folder's cache hold
<depfile key>.owhile the next update asks for<blob key>.o:object_store.fetchlinks nothing, the file is compiled again, andrekey_depfilesthen re-keys it to a key the store already had. It happens on every update that changes a header underhost/orgpu/, which most nightlies do. #1602 counted the same update for its own measurement: 105 objects (65 host.c, 40 renderer.cpp), 239 compiler-seconds, 37 s of wall time on 16 threads.What this changes: the headers a compile read now stay beside the object, in
obj.deps, as the root's place and the path below that root — path-free, so the same key comes back from another builder folder (the AppImage's mount is new at every start, a launcher's tools folder at every update). They are read as they are now, so an edited header still compiles the file, and a deleted one changes the key rather than being taken for the one that was.renderer_depsfalls back to the list when the depfile is gone or names nothing under the roots;rekey_depfileswrites it;keep_translationkeeps it (a few hundred small files) and nothing else of the build folder.Compatibility: the digest is made by one function for both routes, so the lists reproduce the depfile's digest exactly — checked against the 170 depfiles of a real work folder, 170 of 170 the same. No key of the objects those depfiles name moves, so the update that ships this does not compile them again for the new digest's sake (it compiles the 105 once for the missing lists, below).
Split out: the plan's flags line — the one that said
-O2in #1906's log because the object store's loop reboundfl— is #2217: review asked for the two bugs as two changes. This PR is the object-key fix only; no object key depends on that line, and the measurements below are unaffected by it.Read first:
nx2/build.py(renderer_deps,deps_digest,written_deps,rekey_depfiles),nx2/make_game.py(keep_translation), thentests/test_object_reuse_cleanup.py.See #1906, #1907, #2108.
Testing
Tested: game-free only. In the game: N/A — build tooling only, no game code path changes; the evidence is the suite below and the object-store measurement in Measurements.
tests/test_object_reuse_cleanup.py(new, 11 checks, no compiler, no game): the keyrekey_depfilesstores an object under is the one a later plan asks for; the clean-up does not change it; it is the same from another builder folder; an edited or deleted header changes it; a header of another root with the same name does not; an object taken from the store keeps its list; a work folder with neither a depfile nor a list keeps the blob; the store's loop does not rebindfl. On main the key check fails ('3d52f65a…' != '279d918c…', "the clean-up changed an object's key") and the plan-line check fails; on this branch all 11 pass.test_renderer_depfiles(11),test_host_depfiles(6),test_kept_translation(7),test_header_ext_deps(8),test_object_store,test_object_cache_prune,test_build_hash_cache(11),test_dev_update(25),test_make_game_cache(48),test_build_cleanup_scope(11),test_doc_anchors,test_lift_cache.mise run test(the whole suite) on the pre-split head613b775a, same command and environment as the baseline (prrefresh-logs/baseline-a2dc5385.log, a cleana2dc5385checkout, pinned Vulkan/SPIR-V headers,--strict --jobs 3, 8 GB scope): 912 passed, 23 failed, 18 skipped in 619 s. The failing names are exactly the baseline's 23 (diff empty):nx2/test_import_windows.pyand the 22tests/names of main's set,tests/test_sim_turf_pools.pyamong them. The one extra pass over the baseline's 911 is this PR's new check. The suite also caught two checks that cutbuild.py's Vulkan jobs out and exec them (test_shader_stamp_cache,test_dlss_sdk_cache,NameErroron the new helpers); they are updated here with their assertions unchanged.2b65ab36has its own full suite, same command, environment and scope as the baseline: 912 passed, 23 failed, 18 skipped in 442 s (--jobs 5, 12 GiB), the same 23 names as the baseline (diff empty, no new name). It ran from a worktree at that commit, not borrowed from the pre-split run.Still to test:
object store …: N objects from itto name the host's and the back end's own objects and their compile count to fall to the files that really changed (by hand; it needs a builder and a game, not a GPU test).Measurements
Verdict: with the same sources, flags and translation, a plan with the lists takes 92 objects from the object store; the same plan without them — a player's post-clean-up state, which main plans the same way — takes 0. Those are the host and back end objects that used to be keyed by the whole folders' hash. That cost is the one #1602 measured; this removes its cause, and the end-to-end saving on a real update is the first item of Still to test. Nothing in the plan gets slower.
Setup: Linux, Ryzen 9 5950X (16 cores, 32 threads, 16 GB), one work folder copied from the shared base at
a053f9e1(the same sources, toolchain and store), planned twice — once as a player's clean-up leaves it (sa/builddeleted: no depfile, no list) and once with theobj.depslists this PR writes (the same work folder with the build folder emptied of everything else). The two runs'object keys:fingerprints are identical: game-code headers5e2c6c3841, flags9244793c00, hot flagsa0a8ecd2c6, lifted codec1d9965f69. Nothing was compiled: the plans stop after the object store's line. Measured at868c7df7. The head2b65ab36differs from it by the split-out commit (removed, now #2217) and the two stale-check updates — none of it is in an object key; the suite on the head is 912/23/18, the baseline's 23 names.obj.deps(a player's post-clean-up state): this PR @868c7df7868c7df7Baseline: both columns are the same work folder, copied from the shared base at
a053f9e1(the same sources, toolchain and object store), planned by this PR'sbuild.pyat868c7df7; the only difference between them is the*.depslists, which the Before column does not have. That state is a player's post-clean-up work folder, and main plans it the same way: no depfile and no list, sorenderer_depsgives the whole folders' hash, the keys are the same, and the store answers 0 — which is the line #1906 reports. A plan on a main checkout itself was not run for this table.The 92 are what the store answered; the other 71 of the 163 were already in the folder's own cache, and the game's chunks are served from it in both runs (their key does not depend on depfiles). The 7 of the 170 depfiles that name no header under the watched folders were left out of the lists, as before.
Risks
split_giants.flag's hash ofsplit_giants.py, not a depfile, so an update that changes either still compiles all of them (#1727's key, by design). This PR is about the host's and the back end's own objects; a log that still says0 objects from itfor the chunks in a nightly that changed the translator is expected.sa/build/obj/*.deps, a few hundred files of a few KB (about 0.2 MB) in the work folder a player's build keeps.LONGSHOT_KEEP_BUILD_FILES=1and Free up space behave as before; a build folder that is deleted whole (a work folder nobody built in,sagone) keeps today's behaviour, the whole folders' hash.Checklist
Ready for review
main; the title is a Conventional Commit. (The second commit is the log line for this same reuse; see Summary.)mise run testpasses (any failure that also fails onmainis named): 912 passed, 23 failed, 18 skipped on this head (2b65ab36,--jobs 5, 12 GiB, pinned headers); the 23 names are exactlya2dc5385's baseline set, no new name.main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Tested in game: N/A — build tooling only; no game code path changes. Evidence: the suite and the object-store measurement in the body.
Body: the suite's numbers on the head (912 passed, 23 failed, 18 skipped, the baseline's 23 names exactly, no new name), and the measurement's commit named (
868c7df7; the head's third commit is two checks only). No code change since the suite.Body only, no code change: the Measurements table's Before column relabelled. It is not a main plan — it is the same work folder (from the shared base
a053f9e1) planned by this PR's build.py at868c7df7with the *.deps lists absent, i.e. a player's post-clean-up state, which main plans to the same keys (no depfile, no list, the folders' hash — the 0 line #1906 reports). Added the Baseline line the template asks for when the before side is not main; a main plan itself was not run. Suite box untouched (full suite on the head: 912 passed, 23 failed, 18 skipped, the baseline's 23 names exactly).613b775a1a2b65ab3667Split, as review asked. The plan's flags line (the object-store loop rebinding fl) is now #2217; this PR is the object-key fix only. The branch was rewritten: head
2b65ab36= the pre-split head613b775aminus that commit only — git diff613b775a2b65ab36is two hunks, nx2/build.py (a loop variable's name) and tests/test_object_reuse_cleanup.py (-9, the moved check). The suite evidence (912 passed, 23 failed, 18 skipped, the baseline's 23 names exactly) was taken on613b775aand is NOT claimed for2b65ab36; the diff above is why the code it exercised is the same. Description updated to the new head.fix(build): an update keeps the headers its host and renderer objects were keyed byto WIP: fix(build): an update keeps the headers its host and renderer objects were keyed byWIP again until its own head has a full suite, as #2176/#2200 were. Untick: the suite ran on
613b775a; the head2b65ab36differs from it by the split-out commit only (two hunks), and the re-run is queued for after the compile arms (--jobs 5, 12 GiB, pinned headers). Expected: 23, the baseline's names. Body keeps the disclosure, the tick is gone.Suite on this head now: 912 passed, 23 failed, 18 skipped in 442 s (--jobs 5, 12 GiB, pinned headers, worktree at
2b65ab36). The 23 names are exactly thea2dc5385baseline set, no new name. Box ticked with those numbers.WIP: fix(build): an update keeps the headers its host and renderer objects were keyed byto fix(build): an update keeps the headers its host and renderer objects were keyed byView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.