fix(build): a .def or .inl edit compiles the files that include it again, so updates and packs no longer keep an old switch table #1602
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1602
Loading…
Reference in a new issue
No description provided.
Delete branch "klimovich008/Longshot:fix/header-ext-deps"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Windows 11, RX 9070 XT (AMD 26.11.1), Ryzen 7 5700X3D · Needs: Linux, any GPU (a clean and an incremental build after a
switches.defchange)TL;DR
The build decides what to compile again from the header files each source reads, but it only counted
.h,.hppand.inc. The switch table,gpu/vk/switches.def, is none of those, so a new switch row compiled nothing again: an incremental build, a developer's pack and a player's update kept the old table, and only a clean build had the new row. This PR counts.defand.inltoo. In a real build after a row change, main compiled 0 objects and its renderer lacked the row; with this PR 8 objects compiled and the row reached the renderer. A new game-free test fails on main and passes here.Breakdown
.def/.inlinclude) changes..defand.inl, so an edit togpu/vk/switches.defchanged no object's key and the old objects came back from the cache..defor.inledit under the watched folders now compiles again the files that read it, like a.hedit..def/.inlkeep exactly their keys; the lift key (no re-lift), the shader-cache stamp (no re-prepare), eden, sirit and the prebuilt third-party objects; no switch, setting or default.nx2/bake_ds/bake_all.py), a separate follow-up.Why
gpu/vk/switches.hincludesgpu/vk/switches.deftwice, andvk_main.cppandvk_shader.cppread the switch table through it.nx2/build.pykeys each host and renderer object by the headers it read (from the compiler's depfile, or, without one, by a hash of every header under the watched folders), and both of those only took.h,.hppand.inc. So adding a row toswitches.defleft every key as it was: the build reused the oldvk_main.oandvk_shader.o, and the game said "<NAME>: not a switch of this build" for a switch that was in the source. We hit it with a dev pack that added a switch row. A clean build was right; incremental builds, packs (scripts/dev_packs.py) and players' updates were not.No issue filed for it (searched the tracker and open PRs for switches.def, HEADER_EXTS, depfile, tree_headers, stale switch). Related: #62 (open) adds a
host/switches.defand counts.deffor it, inversion.py'sSOURCE_EXT(the same line as here) and in the host's header list as it was before the depfile keys (#673); with this PR merged, itshost/switches.defwould be counted by both key routes already, and itsversion.pyline drops out. #1527 (open) only edits rows ofgpu/vk/switches.def; with this PR its row changes reach incremental builds and updates.How to reproduce
Seen on: Windows 11, llvm-mingw 20261104 (any OS: the lists are plain Python)
Settings and switches: defaults
Steps: build the game; change a row of
gpu/vk/switches.def(for example a switch's summary text); build again in the same folder (or make a dev pack over a base).Expected:
vk_main.cppandvk_shader.cppcompile again and the new row is in the renderer.Actual: nothing compiles; the renderer keeps the old table.
What changed
The two extension lists that say what an
#includecan name gain.inland.def, and the build uses one list for both of its key routes so they can't drift again.nx2/build.py:HEADER_EXTS = (".h", ".hpp", ".inc", ".inl", ".def")(the depfile route,renderer_deps), andtree_headers(the folders' hash, the key without a depfile) usesHEADER_EXTSinstead of its own tuple.nx2/version.py:SOURCE_EXTgains.inland.def, so the source fingerprint in the log's first line andlongshot_version.txtchanges with a switch row too.version.pyandbuild.pyare not lift inputs (make_game.lift_sources()), so no re-lift.tests/test_header_ext_deps.py(new, standard library only, no compiler): a fake compiler that follows includes and writes a clang-style depfile, driven throughtree_headers→renderer_deps→object_key→compile_one→rekey_depfilesas the build runs them.Measurements
Verdict: a build-correctness fix, no speed claim. After a switch row change, main compiles nothing and ships the old table; this PR compiles the 2 includers (plus the usual stamped few) and the row reaches the renderer. The one cost is a one-time recompile on the update that ships it.
Hardware: Windows 11, Ryzen 7 5700X3D (16 threads), RX 9070 XT, llvm-mingw 20261104, no DLSS
Scene: none (builds)
Method: dev packs (
scripts/dev_packs.py) over a base built at mainef6038ab, in a throwaway worktree; the row change puts a marker into one switch's summary text inswitches.def(never committed); the marker is counted in the builtlongshot_vk.dll. Objects compiled from the build's log. The checker's runs; the implementer's earlier runs gave the same counts.How to reproduce: as How to reproduce above, then search the DLL for the new row's text.
Baseline: the builds ran on main
ef6038ab, the base our packs were built over that day. Since then main (876dc360) changednx2/build.pyonly in how lifted chunks find their function definitions (lifted_c.definitions), not in the object keys, the depfile route or the folders' hash, and the rebased commits are the same diff (git range-diff), so the counts hold. The existing checks that readbuild.py/version.py(test_renderer_depfiles.py,test_host_depfiles.py,test_version*.py) pass unchanged.ef6038ab(build.py as on main)vk_main.cpp,vk_shader.cpp(the includers),hos_main.c,vk_menu.cpp(the version stamp),windows_stop.c,vk_hangdump.cpp,msvc_oldnames.c,mingw_msvc_shims.c(keyed by the folders' hash: their depfiles name nothing under the watched folders); 23 compiler-secondslongshot_vk.dllkeep_translationdeletessa/buildwith the depfiles).c, 40 renderer.cpp; eden, sirit, lifted 0; lift 0 s; 239 compiler-seconds, 37 s walllongshot_vk.dll, that buildswitches.defas on main (what ships), after the aboveWhat a player's update pays once: the update that ships this compiles the 105 host and renderer objects again, about 240 compiler-seconds (37 s of wall time on 16 threads here; other core counts not measured), because the folders' hash now includes
switches.def. No re-lift, no shader re-prepare, no eden/sirit or prebuilt third-party rebuild. Those same 105 objects already compile again on any release that changes any header underhost/orgpu/(rc.6 → rc.7 changed 11), so on a normal release it costs nothing extra.Flags and settings
None.
Testing
Tested:
ef6038ab, started, loaded a save and played Hyrule Field for the standard 20 s window: 72.9 fps, game clock 1.000, no new error-like line kind against three same-day runs of main, no "not a switch" for any switch of the build. (A start, not a measurement: the fix changes no game code.)longshot_vk.dllonly with the fix.tests/test_header_ext_deps.py(8 checks, about 1 s): a new switch row recompiles its includer and leaves a file that doesn't read it alone; an.inledit recompiles its includer (through the depfile route, not only the hash); the folders' hash changes with a.def/.inledit; objects reading only.hfiles, and a folder with no.def/.inl, keep exactly their old keys; the fingerprint counts.def/.inl; every#include "..."underhost/andgpu/ends with a counted extension. On main 5 of 8 fail; here 8 of 8 pass. Broken variants (the old list) bring the stale object back and must fail; six mutations of the fix (each list reverted,.inlor.defdropped,.DEFin the wrong case, the oldSOURCE_EXT) each fail the test.python -I -B tests/run_all.py --jobs 4(Windows, llvm-mingw clang 20261104) on this branch (rebased, at e3749b3b: the head but for one condition in the new test's fake compiler, which passes on the head) and on main @876dc360: branch 684 passed, 46 failed, 51 skipped; main 683 passed, 46 failed, 51 skipped. The 46 failures are the same list on both (they fail on this Windows checkout on main too, among themtest_auto_handheld.pyandtest_bug_report_defaults.py); the extra pass is the new test. Before the rebase, the same against main430eef4f.test_version.py,test_version_long_paths.py,test_renderer_depfiles.pyandtest_host_depfiles.pypass.mise run lint: clean apart from the SC2016 note at.forgejo/workflows/build-windows.yml:82that main has too.Not tested:
RX_TARGET=windowson Linux): the lists are plain strings with no OS branch, and the test is standard-library only, but no Linux build was run.Must be tested:
gpu/vk/switches.def(for example add a word to one switch's summary) and build again in the same folder, expecting the second build to compilevk_main.cppandvk_shader.cppagain and the game to start with no error lines; on main the second build compiles neither (Test Farm: @Beanstalk, the comment below)Risks
What could break: a key that changes when it shouldn't (an object compiled again for nothing) or one that doesn't when it should. Objects that read no
.def/.inlkeep their exact keys (tested); the folders' hash for the host and renderer gains one file. So the few objects keyed by that hash even with a depfile (their depfile names no header under the watched folders:msvc_oldnames.c,mingw_msvc_shims.c,windows_stop.c,vk_hangdump.cpp) also compile again on aswitches.defedit, as they already do on any header edit there. A.defused only by the linker (a module definition file) under a watched folder would also count and cost a recompile when edited; none exists today.Turning it off: revert the commit.
Compatibility: on the update that ships this, the host and renderer objects compile once more (about 240 compiler-seconds here, see Measurements), as on any release that changes a header under
host/orgpu/. No re-lift, no shader or pipeline cache change, no settings change, noMKWVK_API_VERSIONchange.Left out, follow-ups:
nx2/bake_ds/bake_all.py:78: the offline shader baker reuses an existingbake_dsexe whatever changed in its sources (it includesswitches.h), so after a switch change its table can be stale. A developer tool, not reached bymake_game.py,build.pyor either builder; not a key, so a separate fix.scripts/dev_packs.py: itsHEADERSlist has no.def(aswitches.defchange counts as a link input, so a pack keeps both modules: conservative, correct), and its scan for host files that includegpu/headers reads only.c/.h/.cpp(a host.inlincludingswitches.defwould route a pack to the renderer alone; no such file exists).Checklist
Everyone
type(scope): description) of the status line's type.can't be run by hand is in Testing's Not testable by hand, with why.
in Risks.
main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.🤖 Generated with Claude Code
Tested in game: Windows 11, RX 9070 XT (AMD 26.11.1), Ryzen 7 5700X3D: the fix alone built as a dev pack over main
ef6038ab(switches.def as on main, so what ships), started, loaded a save and played Hyrule Field for 20 s: 72.9 fps, game clock 1.000, no new error-like line against three same-day runs of main, no 'not a switch' line. Separately, packs with a switch row changed: the new row is in longshot_vk.dll only with the fix (main: 0 objects compiled, old table).WIP: fix(build): a .def or .inl edit compiles the files that include it again, so updates and packs no longer keep an old switch tableto fix(build): a .def or .inl edit compiles the files that include it again, so updates and packs no longer keep an old switch table@Beanstalk linux, any GPU: build this branch clean, start the game to the title screen, then add a word to one switch's summary text in gpu/vk/switches.def and build again in the same folder. Expect the second build to compile vk_main.cpp and vk_shader.cpp again (on main it compiles neither) and the game to start with no error lines. Fail on a build error, a second build that compiles neither file, a crash or error lines in the log.
@klimovich008 only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.