perf(build): keep unchanged button-prompt assets in source builds #2698
Closed
github-sync
wants to merge 1 commit from
github/pr/2698 into main
pull from: github/pr/2698
merge into: MagicBeanDev:main
MagicBeanDev:main
MagicBeanDev:github/pr/2893
MagicBeanDev:github/pr/2890
MagicBeanDev:github/pr/2887
MagicBeanDev:github/pr/2878
MagicBeanDev:github/pr/2875
MagicBeanDev:github/pr/2874
MagicBeanDev:github/pr/2873
MagicBeanDev:github/pr/2871
MagicBeanDev:github/pr/2756
MagicBeanDev:github/pr/2897
MagicBeanDev:github/pr/2896
MagicBeanDev:github/pr/2718
MagicBeanDev:github/pr/2842
MagicBeanDev:github/pr/2691
MagicBeanDev:github/pr/2895
MagicBeanDev:github/pr/2743
MagicBeanDev:github/pr/2765
MagicBeanDev:github/pr/2708
MagicBeanDev:github/pr/2732
MagicBeanDev:github/pr/2755
MagicBeanDev:github/pr/2817
MagicBeanDev:github/pr/2876
MagicBeanDev:release-please--branches--main
MagicBeanDev:release-please--branches--main--release-notes
MagicBeanDev:github/pr/2778
MagicBeanDev:github/pr/2888
MagicBeanDev:github/pr/2866
MagicBeanDev:github/pr/2877
MagicBeanDev:github/pr/2800
MagicBeanDev:github/pr/2829
MagicBeanDev:github/pr/2850
MagicBeanDev:github/pr/2889
MagicBeanDev:github/pr/2882
MagicBeanDev:github/pr/2860
MagicBeanDev:github/pr/2849
MagicBeanDev:github/pr/2886
MagicBeanDev:github/pr/2794
MagicBeanDev:github/pr/2840
MagicBeanDev:github/pr/2830
MagicBeanDev:github/pr/2834
MagicBeanDev:github/pr/2719
MagicBeanDev:github/pr/2742
MagicBeanDev:github/pr/2741
MagicBeanDev:github/pr/2820
MagicBeanDev:github/pr/2771
MagicBeanDev:github/pr/2859
MagicBeanDev:github/pr/2858
MagicBeanDev:github/pr/2823
MagicBeanDev:github/pr/2757
MagicBeanDev:github/pr/2769
MagicBeanDev:github/pr/2844
MagicBeanDev:github/pr/2843
MagicBeanDev:github/pr/2673
MagicBeanDev:github/pr/2855
MagicBeanDev:github/pr/2739
MagicBeanDev:github/pr/2807
MagicBeanDev:github/pr/2851
MagicBeanDev:github/pr/2789
MagicBeanDev:github/pr/2814
MagicBeanDev:github/pr/2818
MagicBeanDev:github/pr/2774
MagicBeanDev:github/pr/2772
MagicBeanDev:github/pr/2750
MagicBeanDev:github/pr/2786
MagicBeanDev:github/pr/2785
MagicBeanDev:github/pr/2782
MagicBeanDev:github/pr/2779
MagicBeanDev:github/pr/2775
MagicBeanDev:github/pr/2706
MagicBeanDev:github/pr/2766
MagicBeanDev:github/pr/2747
MagicBeanDev:github/pr/2759
MagicBeanDev:github/pr/2745
MagicBeanDev:github/pr/2740
MagicBeanDev:github/pr/2733
MagicBeanDev:github/pr/2725
MagicBeanDev:github/pr/2727
MagicBeanDev:github/pr/2728
MagicBeanDev:github/pr/2729
MagicBeanDev:github/pr/2721
MagicBeanDev:github/pr/2722
MagicBeanDev:github/pr/2715
MagicBeanDev:github/pr/2714
MagicBeanDev:github/pr/2707
MagicBeanDev:github/pr/2703
MagicBeanDev:github/pr/2705
MagicBeanDev:github/pr/2697
MagicBeanDev:github/pr/2679
MagicBeanDev:github/pr/2676
MagicBeanDev:github/pr/2669
MagicBeanDev:docs/what-hookshot-connects-to
MagicBeanDev:fix/hdr-encode-test
MagicBeanDev:fix/main-tests
MagicBeanDev:fix/play-cache-files
MagicBeanDev:perf/game-barrier-order
MagicBeanDev:feat/heap-lazy
MagicBeanDev:feat/texture-detail
MagicBeanDev:perf/astc-opaque-bc1
MagicBeanDev:fix/fs-dir-lock
MagicBeanDev:fix/framegen-native-kept
MagicBeanDev:fix/update-no-downgrade
MagicBeanDev:port/k10-gcn-f16-mirrors
MagicBeanDev:fix/framegen-mult-cap
MagicBeanDev:chore/release-as-2.1.1
MagicBeanDev:perf/pcache-load-switch
MagicBeanDev:fix/prepare-progress-log
MagicBeanDev:fix/update-closes-stale-game
MagicBeanDev:fix/hdr-scaled-sampled
MagicBeanDev:perf/first-start-preset
MagicBeanDev:perf/zero-upload
MagicBeanDev:fix/vcore-escape-2s
MagicBeanDev:test/fake-gh-atomic
MagicBeanDev:perf/batch-idle-flush
MagicBeanDev:test/finish-stub-a
MagicBeanDev:chore/record-2.1.0
MagicBeanDev:fix/main-red-tests
MagicBeanDev:fix/test-display-startup-driver-cache
MagicBeanDev:feat/mesh-shader-mode
MagicBeanDev:fix/menu-probes-at-startup
MagicBeanDev:fix/fps-counter-classic
MagicBeanDev:fix/test-log-batch-flake
MagicBeanDev:fix/narrow-screen-hud
MagicBeanDev:ci/prereleases
MagicBeanDev:release/v2.0.15
MagicBeanDev:hotfix/v2.0.15
MagicBeanDev:fix/present-ahead-replay
MagicBeanDev:release/v2.0.13
MagicBeanDev:fix/bracket-paths-2013
MagicBeanDev:fix/hdr-overlays-2013
MagicBeanDev:fix/fps-counter-scale-2013
MagicBeanDev:fix/stick-seen-at-rest
MagicBeanDev:hotfix/menu-fits-screen
MagicBeanDev:fix/f1-always-opens-menu
MagicBeanDev:fix/ini-keep-lines
MagicBeanDev:claude/cranky-pike-7cf247
MagicBeanDev:perf/texture-answer-flat
MagicBeanDev:fix/stamp-test-modules
MagicBeanDev:fix/appimage-fonts
MagicBeanDev:fix/display-awake
MagicBeanDev:release/v2.0.12
MagicBeanDev:fix/discord-windows
MagicBeanDev:rc/v2.0.12-tests
MagicBeanDev:fix/dlss-keep-feature
MagicBeanDev:rc/v2.0.12
MagicBeanDev:button-icons
No reviewers
Labels
Clear labels
accessibility
Barrier affecting people with disabilities
autorelease: pending
autorelease: tagged
bug
Something isn't working
dependencies
Pull requests that update a dependency file
documentation
Improvements or additions to documentation
duplicate
This issue or pull request already exists
enhancement
New feature or request
github_actions
Pull requests that update GitHub Actions code
good first issue
Good for newcomers
help wanted
Extra attention is needed
invalid
This doesn't seem right
question
Further information is requested
testing wanted
wontfix
This will not be worked on
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot!2698
Loading…
Reference in a new issue
No description provided.
Delete branch "github/pr/2698"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is for: contributors rebuilding from source should not spend another minute drawing the same button prompts. What is measured: on an Intel Core i9-13900K running Arch Linux, the finish stage at main
c544828ftook 67.707 and 66.898 seconds; two unchanged runs took 0.603 and 0.651 seconds, and the final head took 0.616 seconds, clearing the run-time files as well. Other work was running throughout. Not measured: Windows, a packaged Builder update, a complete compile/lift, or gameplay performance.Refs #2636.
Problem
nx2/make_game.pyremoves the game folder'spromptsdirectory and runsprompt_assets.pyon every build when fontTools is available. Most source edits do not change anything those assets depend on.This avoids drawing unchanged assets; it does not make the drawing itself faster. #2636's A1 plans to measure the finish stage, but its A1–A10 items do not already propose this prompt cache. It complements #2674 and #2675, which address unnecessary retranslation rather than prompt generation.
What changed
nx2/make_game.py: source-checkout builds request incremental prompt generation. Packaged builders still regenerate every time, using the finish stage's existing launcher distinction.RX_RELEASEcannot distinguish them: make_game sets it for source builds too. Missing fontTools still removes the replacement prompts, keeping the existing fallback to the game's icons.nx2/prompt_assets.py: fingerprint the inputs before generating, draw into a fresh sibling directory, and record the successful outputs only if a second fingerprint still matches. Reuse requires both matching inputs and every recorded output's size and SHA-256. A failed generation still removes the prompts, as before, so the game shows its own icons. An interrupted builder can leave the previous complete set without a stamp; the next build regenerates it. The drawing, encoding and asset formats are unchanged.tests/test_prompt_skip.py: game-free fixtures run the actual finish-stage prompt call and production cache/publication functions, with small authored files and a stand-in for expensive drawing.docs/development.md: explain reuse, its limits, andHOOKSHOT_FORCE_PROMPTS=1. The force switch followsenv_on: unset, empty,0,false,offandnoare off, ignoring case and surrounding spaces.What prevents a wrong skip
The input record covers:
SOURCE_DATE_EPOCH,FREETYPE_PROPERTIESand the encoder's custom CPU definitions.Languages and saved bindings are not build inputs here: the generator uses its source tables, and the runtime applies the selected controls.
The output record includes files and directories: the keyboard directory itself is a runtime marker. Legitimately absent optional outputs stay absent; a missing, truncated or same-size-corrupted recorded file forces regeneration. The runtime's own compressed copies and control-specific keys are excluded before hashing, not mistaken for generator output. The files the game derives at run time are still cleared by every build, including an unchanged hit: their own keys do not cover changes to
host/prompts.c. Only the known runtime filenames are removed; if one cannot be cleared, the build regenerates instead of keeping on doubt.An unreadable input cannot authorize reuse. The log gives the reason, even if a forced regeneration subsequently fails. A source edited during generation leaves no stamp, addressing the stale-input certification problem discussed in #2674's review.
How it was tested
2026-11-12; Arch Linux, kernel
7.2.15-1-omarchy, x86-64, Intel Core i9-13900K, 32 logical CPUs, Python 3.12.16. Basec544828ff; final head2ec89de8. No game or GPU program was launched for these checks.Finish-stage measurements
These execute the real finish-stage statements through completion, including binary/image copies, the unchanged shader-list check, pipeline-list work and prompt generation, in a disposable copy of an existing built game folder. Compilation, translation and shader preparation are not included. They are not whole rebuild times.
All optional prompt packages were present: fontTools 4.68.0, Pillow 13.0.0, astc-encoder-py 0.1.12, texture2ddecoder 1.0.6, archspec 0.2.7, NumPy 2.5.4 and zstandard 0.25.0.
c544828fc544828f63d5c96563d5c96563d5c965681208dd2ec89de82ec89de8Other work was running; none was a quiet measurement. CPU percentages are lifetime averages from snapshots, not interval measurements. The first generation is included to show that this is a skip, not a faster drawing algorithm.
The unchanged runs print:
Adding a harmless comment to the button-table source forces a fresh generation:
The comment was removed afterwards. A timestamp-only touch does not regenerate: the key checks content. An additional run on the final head verifies an unchanged hit with all six runtime-derived files present: all six are removed, while the 16 generated originals and the stamp retain their sizes and SHA-256 values. The focused fixture also compares the retained originals byte for byte. The final timing rows above include that cleanup.
Asset parity: all 16 regenerated files, 274,704,680 bytes, compare byte for byte with a full generation by main
c544828f. Only the new stamp is excluded. Both sides used the sameSOURCE_DATE_EPOCHbecause fontTools otherwise writes the current time into the OpenType header. This comparison setting is not introduced into normal builds. A development cache hit retains the earlier font timestamp; packaged builds retain their old full-generation behavior.Game-free checks
23 focused tests pass on the final change. They cover unchanged reuse, every input kind, package versions/presence, switch/parameter changes, missing/truncated/same-size-corrupted output, empty directories, clearing runtime derivatives while retaining byte-identical originals, regeneration when runtime cleanup fails, interruption, mid-generation edits, unreadable input, force-switch parsing, incomplete output, publication failure, and the packaged/no-fontTools paths. A non-zero generator result removes prompts for both source and packaged builds; interruption before that result leaves the old set unstamped.
The same unchanged-build test against main's finish stage fails behaviorally:
2 != 1: unchanged finish must not draw the prompts twice.Removing each of 18 guards/key components in memory makes its corresponding behavior test fail on the final change. Three later corrections also fail against the earlier version of this patch: returned failure retained old prompts in both build paths, runtime files survived a hit, and an uncleared runtime file did not force regeneration. The unreadable-input diagnostic has a separate before-fails/after-passes proof.
The whole suite one commit earlier, at
f20ea58b(before a failed generation was made to remove the prompts again and the run-time files to be cleared),python -I -B tests/run_all.py --jobs 8:Load was 7.37 → 19.48. The first full run, at 32 checks at once, was 471 passed, 1 failed, 20 skipped: a clang++ child compiling
texture_pass.cppfornx2/test_depth_sample_types.pydied with SIGKILL. That fixture and its input trees are unchanged from main. No kernel OOM line was found, so the cause is not established. The complete run with eight workers passes, including that check. The failed run is retained, not counted as a pass. On the final head2ec89de8the focused checks, the baseline failure and all guard mutations were rerun, and the whole suite once more with eight workers: 471 passed, 1 failed, 20 skipped in 136 s (load 11.14 → 22.62). The one failure istests/test_release_scripts.py,test_nightly_two_runs_at_once_end_at_the_newer_commit, the intermittent race reported in #2626; this change touches none of its files.No core dumps were recorded. Nothing generated from the game is in this change.
Not tested
No screenshots: the regenerated asset bytes match, and no visual change is intended.
Risks
The input list must grow when the generator gains new file dependencies or outputs outside the prompt directory. A regression flags new local module imports, but it does not replace reviewing new file reads. Generation temporarily needs room for both old and new assets. Publication uses completed directories and rollback on a rename error; interruption cannot create a stamped half-set, though a process killed between the two directory renames may require regeneration on the next build.
Overlaps
The open-PR scan found 47 PRs touching one of these files. The actual conflict hunks were read:
--wheel-icons, another input folder and output outsideprompts. Keep that work outside this skip or extend the input/output proof and staging; accepting both hunks mechanically is not enough.Assisted-by: GPT-6 Astra
Co-Authored-By: GPT-6 Astra [email protected]
Nit: if the builder is killed hard here (SIGKILL, a closed console on Windows, power loss), this
.prompts-*directory stays in the game folder with up to two full sets in it (about 550 MB), and nothing removes it later. A quick sweep of stale.prompts-*siblings at the start ofmake_promptswould clean that up.Nit: the first build after this lands, and any build after a failed one, has no stamp at all, so the log will say
stamp or outputs unreadable, which sounds like something broke. CatchingFileNotFoundErroron the stamp separately and sayingno stamp yetwould make it read better.Thanks, this is a nice win for anyone rebuilding from source: a minute saved on every finish stage, and the skip is careful about what it trusts.
I read the key against everything
generate()opens and imports (the four prompt modules, the romfs font, label font, dictionary, textures, layouts and size table, the Kenney fonts, the PNG overrides, the package versions), and the runtime list against whathost/prompts.cwrites intoprompts/(the.zscopies, the two.keyfiles,rsizetable.mod.zs). Both line up. The publish path keeps the old behaviour where it matters: no fontTools or a failed generator still removes the folder so the game falls back to its own icons, the stamp is removed before drawing so an interrupted run can't certify a stale set, and the second key check covers edits made mid-generation.launcheris the same packaged/source split the finish stage already uses, andHOOKSHOT_FORCE_PROMPTSgoes throughenv_onlike the other switches. CI is green on Linux and Windows, and the branch is up to date with main.Keeping packaged builders on full regeneration is fine for now. They rebuild everything anyway, and we can widen it later once this has had some use from source builds.
Two small nits inline, neither blocking. Also, as you note, #2261 adds an input folder and output outside
prompts/, so whichever of the two lands second has to extendprompt_key/prompt_outputsrather than just merging the hunks. I'll keep that in mind when merging.github-sync referenced this pull request2026-11-12 08:31:01 +00:00
Both taken, thank you: #2710.
prompts: regenerated (no stamp yet); a stamp that exists and cannot be read keeps its own line..prompts-*directories are removed at the start of the next build. Each generation now holds a lock on an owner file in its staging directory, so a leftover is told from a running build by the lock (free: removed at once, also right after a kill; held: left alone), not by age. Directories from this pull request's version carry no owner file and go after 24 hours. Tried with a realkill -9on Linux; the Windows locking call is exercised with a stand-in only.