WIP: feat(build): the build says how well the play profile fits the hot pieces #1976
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1976
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:feat/build-profile-fit"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
A build with a play profile of the hot pieces (
RX_PGOset to a profile file, orwork/longshot_pieces.prof, whichmake_game.pypicks up by itself) compiles the eight hot pieces with that profile's branch counts. A lifted function whose shape
changed since the profile was recorded (a translator change, another game version) is built without its counts, and
since the build compiles with
-Wno-everything, it said nothing: a profile left in a work folder from anothertranslation looked like one that works, and the build was slower code with no word of why.
The hot pieces' compile now asks clang for its own account, with its three warning groups given after the job's own
command and kept warnings under a
-Werror(-Wprofile-instr-out-of-date,-Wprofile-instr-missing,-Wprofile-instr-unprofiled, each with its-Wno-error=). clang says one of three things a file: "no profile dataavailable for file" when none of its functions has counts in the profile; else "of N functions, M have mismatched data"
and "of N functions, M have no data", each when M is not 0; else nothing, which is every function's counts matching
(clang does not say how many functions that is). After a compile in which every file compiled, the build says what clang
established, piece by piece: the pieces with functions built without their counts (how many of their functions no
longer match the profile, how many have no counts in it), the pieces with no counts in it at all, the pieces in which
clang names no such function, and the pieces not known. It says the profile is out of date for this translation only
when clang gave every piece's number of functions and most of the functions it has counts for no longer match;
otherwise, where some do not match, it says that this is not said.
The flags are left out of the object key, so nothing is compiled again for them; they only ask clang for warnings, and the
test compiles a file with and without them and compares the objects but a COFF header's time stamp, which a Windows
compile writes anew each time. The records are kept by object key in
pgo_fit.jsonin the work folder, with a formatnumber and the compiler's and system headers' stamp (the third-party objects'), so a build that takes the pieces from its
cache says them again. The file is read only by a build of profiled hot pieces, at most 1 MB; a record of the wrong shape,
type or range, a file of another stamp or format, a bigger or deeply nested one is no record, so the piece is not known
(compiled before this, taken from another work folder's objects, or made with another compiler), and nothing is compiled
again for it. The records are saved also when another file fails to compile, before the build stops, through a temporary
file of its own and a rename, over the file's records as they are then. Nothing changes for a build without a pieces
profile, or for any other file.
docs/dev/reference/build.md'sRX_PGOrow says so.Read first:
PGO_FIT_FLAGS,pgo_fit,pgo_fit_valid,pgo_fit_read,pgo_fit_save,pgo_fit_lines,pgo_fit_reportand the lines added to
compile_oneandmaininnx2/build.py(compile_onewrites its test out rather than callingpgo_fit_wanted: five checks cut it out alone and run it);tests/test_pgo_fit.py.Testing
Tested: Windows 11, i7-9700, the Builder's llvm-mingw clang 22.1.8 and Python 3.14.7; game-free checks only. No
build of the game with a pieces profile has run with this change yet, and no game start of such a build (Still to
test). This round's fixes for the review are two commits on the change rebased onto
main@f1f2dc5b(git range-diff"=" for the change's own commit):
2364426b, the tests, andc47f11e9, the fix. Each check ran in a hidden console of itsown, through the PC's job queue:
2364426b(the change as it was):python -I -B tests/test_pgo_fit.py --cc clang -vfails, 16failures and 22 errors (subtests counted) over its 12 tests: among them a kept record
[10, null, 0]stoppedbuild.main with a TypeError and a deeply nested kept file with a RecursionError, after a file that failed twice the
next build called every hot piece not known, and the mixed build was said as one sum over the pieces that warned.
nx2/build.pyis c47f11e9's), the same test three times and once without--cc: 12 tests pass each time,RealClangincluded (clang's three lines under-Werror, the same object bytes withand without the flags but the COFF time stamp). With main's one reporting call (
pgo_fit_report) removed in a copy,four of the coordinator tests fail.
tests/test_compile_retry_jobs.pyfailed: it runs main's compile and retry statements cut out alone, andthe report call among them had no definition there;
c47f11e9gives it a stand-in.c47f11e9:tests/test_pgo_fit.py --cc clang -v, 12 tests, all pass; the checks that cutcompile_oneor main's compile out alone, or run build.main:
tests/test_compile_retry_jobs.py(5 tests),test_compile_stops_on_exception.py(2),test_compiler_failure.py(1),test_third_party_objects.py(15),test_mem_gate_leases.py(4),test_a64_helper_build_flags.py(3),test_hot_pieces_split.py: all pass.c47f11e9,python -I -B tests/run_all.py --strict --changed origin/main(8 at once): 52 passed, 12 failed, 5skipped. The 12 (
test_appimage_player_build.py,test_dev_build.py,test_dev_packs.py,test_failed_compile_keeps_objects.py,test_job_queue.py,test_lift_cache.py,test_linux_preflight.py,test_make_game_ngx.py,test_mise_build.py,test_mods_list.py,test_recorder_sync.py,test_renhdrum.py)fail on
main@f1f2dc5btoo, run by name in the same job; none fails on this branch alone. One of the 12,test_recorder_sync.py, had found no Vulkan headers in that job (RX_CPMunset) and compiled nothing; run again with them atc47f11e9and onmain@f1f2dc5b, it fails onmaintoo with the same lines.RealClangthere is not yet seen with its ccache shim).Still to test:
main(anRX_PGO=genpackover main's base, Castle Town's market for 90 s with
HOS_PGO), then this branch built as a pack withRX_PGOset to that profile, expecting abuild: the play profileline that says it fits; the same with a profilerecorded on another translation (the
exp/x86-asm-syncbranch's own), expecting the out-of-date line; a profile ofanother game version none of whose functions is in a piece, expecting that piece named as having no counts at all; each
build repeated warm, expecting the same lines and no hot piece compiled again; clean build time and size against the
merge base;
mainbuiltwith the first profile after it, expecting every hot piece taken from the object store (the keys unchanged); and one
game start of this branch's build (by hand)
RX_PGOset to a pieces profile (orwork/longshot_pieces.prof), expecting abuild: the play profileline after the compile and the same object keys as without this change (by hand)Measurements
Verdict: a build tool's output; not measured yet. Its game-free test passes, and no build with a pieces profile
has run with it (Still to test).
Setup: Windows 11, i7-9700, llvm-mingw clang 22.1.8, Python 3.14.7;
python -I -B tests/test_pgo_fit.py --cc clang -vthroughthe PC's job queue
f1f2dc5btests/test_pgo_fit.pymainc47f11e9)Risks
Only builds with a pieces profile change, and only in what they print and in the work folder's
pgo_fit.json. The threewarning groups reach the hot pieces' compile after
-Wno-everything, each kept a warning under a-Werror, so clang printsthem for those pieces (the Builder's llvm-mingw clang 22.1.8 knows them: the test compiles with them, and with
-Werror). No object keychanges, so no player's build compiles anything again after updating. A first build after updating, or one whose hot
objects come from another work folder, says those pieces are not known. A revert is enough; no switch.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Possibly related, for context:
Recommendation: Request changes. The report can tell players to replace a mostly valid profile and can report a perfect fit when Clang has no data for an entire hot piece. A malformed diagnostics cache can also stop an otherwise successful build before linking.
Adds a build-log summary for hot-piece PGO and a work-local cache so warm builds can repeat it without rebuilding objects. Keeping diagnostics out of the object keys is useful. The report needs to distinguish complete, partial and unavailable evidence before players can act on it.
@fixwill do (14)nx2/build.py:1664: A piece that emits neither warning contributes zero functions to the sum. (more on the line)nx2/build.py:1618: Clang can report an entirely unprofiled translation unit through -Wprofile-instr-unprofiled. (more on the line)nx2/build.py:1641: The cache accepts any three-element list. (more on the line)nx2/build.py:1753: The appended warnings can become compile errors under a Clang configuration with -Werror. (more on the line)nx2/build.py:2312: Successful pieces lose their diagnostics when an unrelated compile fails twice. (more on the line)nx2/build.py:1648: Direct standalone and DLL builds can share one work folder without make_game.py's lock. (more on the line)nx2/build.py:2269: Every build reads the sidecar, including renderer-only and unprofiled builds. (more on the line)nx2/build.py:1638: The fit cache uses the bare object key, which excludes compiler and system-header identity. (more on the line)tests/test_pgo_fit.py:188: Linux CI puts a ccache clang shim first on PATH. (more on the line)tests/test_pgo_fit.py:29: Importing build runs environment-dependent setup before the mocks. (more on the line)tests/test_pgo_fit.py:40: The stand-in writes the argument log with the machine's default encoding and the parent reads it as UTF-8. (more on the line)tests/test_pgo_fit.py:140: The cache test calls the formatter directly. (more on the line)docs/dev/reference/build.md:66: The switch row promises counts after a profiled build. (more on the line)For the author (1) (a run, a measurement or a decision)
Checks: Still running:
PR description / Description follows the template (pull_request_target),CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven in the game build. The author reports six game-free tests passing in three runs on Windows at dd77940b, including a synthetic COFF object comparison. There is no profiled game build, Linux result or clean build time and size comparison against the base. The change claims no gameplay speedup.
Other systems: The Builder forwards the summary to its log without changing its progress markers. The diagnostic path applies to Windows and Linux hot-piece compiles, including AppImage builds with a pieces profile and updates that reuse a kept lift. Renderer PGO, host and third-party compile flags, and runtime paths for graphics, saves, audio and input are unchanged by inspection. The only reported object comparison is the synthetic Windows fixture.
Switches: Adds two internal Clang warning flags, enabled only when compiling a hot piece with an existing RX_PGO profile. That scope is appropriate once the warnings are guaranteed to stay informational. No user switch or default changes. RX_PGO is unset by default, and make_game.py can select a local pieces profile. The build reference updates the RX_PGO row. Build-only flags do not belong in gpu/vk/switches.def.
Reviewed:
03063b0786against6e1521a70a: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, docs/dev/standards.md, docs/dev/reference/build.md and tests/test_pgo_fit.py in full. Read every changed block in nx2/build.py and traced job selection, keying, cache hits, retries, linking and progress reporting. Read the relevant make_game.py, object_store.py, split_giants.py, toolchain, test-runner and CI paths. Unrelated build helpers and existing recorded-run notes were skimmed. No review-checklist.md or local Tooling checkout was present. The pre-rebase test commit was unavailable locally, so its results and range-diff are as reported in the PR.; nothing run (a code review).Checked and fine:
Questions:
Testing: Read-only review. The PR reports Windows game-free runs on the pre-rebase commit and twelve changed-suite failures reproduced on its then-base. Full mise run test and lint results are absent. The listed CI checks are still running.
Overlaps and merge order: None found in the checkout or supplied review history.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -64,3 +64,3 @@| `RX_GAME_ONLY=1` | The game alone, the renderer neither compiled nor linked (`dev.py pack` sets it when a change can't reach the renderer; compile only: not in the lift key) || `RX_RELEASE=1` | A release build: no console window, shaders prepared before the title (`make_game.py` sets it) || `RX_PGO=gen` or `<file>`, `RX_VK_PGO=gen` or `<file>` | Count branches for a profile, or build with one: the hot pieces, the renderer || `RX_PGO=gen` or `<file>`, `RX_VK_PGO=gen` or `<file>` | Count branches for a profile, or build with one: the hot pieces, the renderer. After the compile, a build with a pieces profile says how well it fits (`build: the play profile ...`): how many of the pieces' functions no longer match it and are built without their counts, how many have none, and that it is out of date when most of the functions it has counts for no longer match (the counts kept in `<work>/pgo_fit.json`; `tests/test_pgo_fit.py`) |L13 (minor): The switch row promises counts after a profiled build. A first build after updating can reuse every hot object and have no local metadata, as can a build served by another work folder's object store. Document the partial and unknown cases and explain that reusing those objects does not trigger recompilation for the report.
@ -2233,6 +2311,11 @@ def main():print("compiled in %.0fs, %d failed" % (time.time() - t0, still), flush=True)if still:L5 (minor): Successful pieces lose their diagnostics when an unrelated compile fails twice. For example, a failed host compile leaves the hot objects cached, but the next build reuses them and can never recover the fit. Persist successful records before returning on a partial failure, or save each record atomically when its object is cached.
@ -1609,0 +1615,4 @@# when there is a clang). The counts are kept by object key in# <work>/pgo_fit.json, so a build that takes the pieces from its cache says them again; a piece compiled before this, or by# another work folder (the object store), is said to be not known.PGO_FIT_FLAGS = ["-Wprofile-instr-out-of-date", "-Wprofile-instr-missing"]L2: Clang can report an entirely unprofiled translation unit through -Wprofile-instr-unprofiled. With that group suppressed by -Wno-everything, a valid profile containing no functions from a hot piece can turn into a cached perfect fit. Enable and handle the whole-file diagnostic, preserve an unknown state when a count is unavailable, and cover this with a real Clang fixture whose functions are entirely disjoint from the recording.
@ -1609,0 +1635,4 @@return fitdef pgo_fit_load(path):L8 (minor): The fit cache uses the bare object key, which excludes compiler and system-header identity. make_game.py invalidates local objects when that identity changes, but leaves the fit cache. If the new toolchain's store already has an object with the same key, the build can report the old compiler's result for it. Namespace this metadata by the compiler and header stamp and version its format, without changing object keys.
@ -1609,0 +1638,4 @@def pgo_fit_load(path):try:with open(path, encoding="utf-8") as f:PGO_FIT.update({k: v for k, v in json.load(f).items() if isinstance(v, list) and len(v) == 3})L3: The cache accepts any three-element list. An active entry containing [10, null, 0] raises TypeError while summing, after the objects have compiled and before they link. Negative or inconsistent counts can print impossible totals. Validate the integer fields and their ranges on load, and treat invalid entries as unknown.
@ -1609,0 +1645,4 @@def pgo_fit_save(path):try:with open(path + ".tmp", "w", encoding="utf-8") as f:L6 (minor): Direct standalone and DLL builds can share one work folder without make_game.py's lock. Both can truncate the same temporary file and replace the cache with independent snapshots, losing one build's records or publishing a partially written file. Use per-key atomic metadata or serialize the read-merge-write, with unique temporary files and cleanup on failure.
@ -1609,0 +1661,4 @@return ["build: the play profile's fit to the %d hot pieces is not known: they were compiled before it was noted" % len(keys)]before = "; %d of the %d pieces were compiled before this was noted" % (len(keys) - len(known), len(keys)) \if len(known) < len(keys) else ""total, mismatched, none = (sum(f[i] for f in known) for i in range(3))L1: A piece that emits neither warning contributes zero functions to the sum. With 100 fully matching functions in a silent piece and 6 mismatches out of 10 in another, the report claims most counts are stale when only 6 of 110 are. A cached piece with missing metadata can invalidate the same conclusion. Use complete per-piece totals or scope the counts to the measured subset and withhold a whole-profile verdict when coverage is incomplete. Test both mixtures.
@ -1678,3 +1751,3 @@t = time.time()try:with subprocess.Popen([cc] + fl + ["-c", c_src, "-o", obj], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) as p:with subprocess.Popen([cc] + fl + ["-c", c_src, "-o", obj] + fit, stdout=subprocess.PIPE, stderr=subprocess.PIPE,L4: The appended warnings can become compile errors under a Clang configuration with -Werror. A stale profile that previously built successfully can now fail both attempts. Make the enabled PGO groups explicitly nonfatal and verify a stale-profile compile with -Werror.
@ -2189,6 +2266,7 @@ def main():if CHUNK_OPT_O2 is not None: # (the chunks are at LONGSHOT_CHUNK_LEVEL; the line names the levels in use)shown = shown.replace("-O1", "-" + chunk_level())print("%d files, %d jobs, %s" % (len(jobs), a.jobs, shown), flush=True)pgo_fit_load(os.path.join(kit.WORK, "pgo_fit.json"))L7 (minor): Every build reads the sidecar, including renderer-only and unprofiled builds. An oversized file is read in full, and deeply nested JSON can raise RecursionError before any compile starts. Read it only when the selected jobs include profiled hot pieces and bound the cache read.
@ -0,0 +26,4 @@ROOT = Path(__file__).resolve().parents[1]sys.path[:0] = [str(ROOT / "third_party" / "a64-kit" / d) for d in ("lift", "tools")] + [str(ROOT / "nx2")]import build # noqa: E402L10: Importing build runs environment-dependent setup before the mocks. For example, RX_MARCH=native with no usable Clang exits before any test. On Linux the import also invokes pkg-config, and on suitable CPUs it can invoke a real Clang probe even for the stand-in tests. These extra commands are absent from the PR description. Cut the tested definitions with controlled stand-ins, as the repository's other compile checks do, and keep the real tool invocations explicit.
@ -0,0 +37,4 @@STAND_IN = """import sysa = sys.argv[1:]open(a[a.index("-o") + 1], "wb").write(b"object")open(a[a.index("-o") + 1] + ".args", "w").write("\\n".join(a))L11 (minor): The stand-in writes the argument log with the machine's default encoding and the parent reads it as UTF-8. On a Windows CP1252 installation with an accented name in the temp path, the test can fail on decoding. Write the log as UTF-8 and exercise a non-ASCII temp path.
@ -0,0 +137,4 @@build.pgo_fit_load(path)self.assertEqual(build.compile_one(hot), (hot[1], 0.0, None)) # (the object as it is: nothing compiled)self.assertFalse(os.path.exists(hot[2] + ".args"))lines = build.pgo_fit_lines([build.PLANNED[(hot[1], hot[2])]])L12 (minor): The cache test calls the formatter directly. Removing the reporting block from main leaves all six tests passing. Exercise the build coordinator with authored jobs and assert its output on a fresh compile and a cache hit, including a build without a pieces profile.
@ -0,0 +185,4 @@def test_the_lines_and_the_same_object(self):cc = toolchain.find("clang")profdata = os.path.join(os.path.dirname(cc), "llvm-profdata" + (".exe" if os.name == "nt" else ""))L9: Linux CI puts a ccache clang shim first on PATH. Resolving llvm-profdata beside that shim misses the real tool, and the test silently skips the only object-equivalence check. The runner counts the zero exit status as a pass even under --strict. Honor an explicit Clang selection and CC, resolve the matching llvm-profdata independently of wrappers, and require the profile runtime and comparison in the Windows and Linux jobs that provide those tools.
Recommendation: Request changes. The new diagnostic can give a misleading profile verdict and can prevent a build when its optional cache is malformed.
I confirmed the existing review's denominator and missing-translation-unit blockers from the source. A warning-free piece contributes zero functions in the aggregate, while an entirely unprofiled piece can take another warning path. Report what the compiler actually established and retain an explicit unknown for any piece without usable coverage. Validate the three cached fields as nonnegative integers before using them. In addition to testing a mixed hot build, exercise types from an otherwise syntactically valid cache and an unrelated compile failure followed by a successful retry. A successful hot object can be reused after the retry while the diagnostic for that object was never persisted.
Does it help: Unproven end to end. The author reports six focused Windows checks passing at dd77940b and an unchanged rebase. The current PR has no Windows or Linux hot-piece profile build with a matching, an entirely unprofiled, and a stale case, and CI is pending. The ordinary farm plan cannot install and vary the input profile. Verify with the author’s by-hand plan and identical input profiles, showing the object's cache behavior and emitted diagnostics.
Reviewed:
03063b0786against6e1521a70a. Read full diff, compile/retry and object-key paths and the existing review. No code executed.03063b0786c47f11e943Pushed
c47f11e9for futondestroyer's review (62717) and BeanBot's (62442), rebased onto mainf1f2dc5b:2364426badds the tests (build.main driven with authored jobs, a real clang's whole-file line under -Werror, the kept file's records) andc47f11e9the fix (what clang established, piece by piece, with an explicit unknown; the verdict only with every piece's count; validated, stamped, bounded records saved also when another file fails). Summary, Testing (what ran at the test commit and at the head, through the PC's queue), Still to test and Risks are updated.Re #1976 (comment): thank you, all four points are in
c47f11e9(fix), with the tests first in2364426b, which fail on the change as it was and pass at the fix (results in the description's Testing). The build now says per kind of piece what clang established (pieces with counts and their numbers, pieces with no data at all through-Wprofile-instr-unprofiled, pieces clang named nothing in, pieces not known), and calls the profile out of date only when clang gave every piece's number of functions; the kept file's records are checked as nonnegative integers in range, anything else being no record. tests/test_pgo_fit.py now runs build.main itself with authored jobs: a mixed hot build, kept files that are valid JSON with records of wrong types, and a file that fails twice followed by a build that compiles, where the hot objects are taken from the cache and their saved records are said. The by-hand Windows and Linux profile builds (matching, unprofiled, stale, a warm repeat) have not run yet and are still pending in Still to test.Re #1976 (comment): fixed in
c47f11e9. A piece clang says nothing of no longer adds zero functions to a sum: each piece is reported by what clang established, and the out-of-date verdict is given only when clang named every piece's number of functions; otherwise the build says the verdict is not said. Your mixture (6 of 10 in one piece, one silent piece) and a cached piece with no record are both in tests/test_pgo_fit.py (test_what_the_build_says, andCoordinator.test_a_mixed_build_gives_no_verdict_and_a_cache_hit_says_it_againthrough build.main), failing at2364426band passing atc47f11e9.Re #1976 (comment): fixed in
c47f11e9:-Wprofile-instr-unprofiledis asked for too and its line makes the piece's record "no counts in it at all", never a fit; a piece with no record stays not known.RealClangcompiles a source none of whose functions is in the recorded profile and checks clang'sno profile data available for fileline and the record, and the coordinator test has a whole-file piece in a mixed build; both fail at2364426b.Re #1976 (comment): fixed in
c47f11e9: the kept file is read throughpgo_fit_valid, which takes only ["fit"], ["unprofiled"] or ["counts", N, mismatched, missing] with nonnegative integers (not bools) and mismatched + missing within N; anything else is no record, so the piece is not known.KeptFile.test_what_is_not_a_recordcovers [10, null, 0], negatives, floats, bools, strings and impossible sums, andCoordinator.test_a_malformed_kept_file_does_not_stop_the_buildputs such records under the real hot pieces' keys and runs build.main (a TypeError at2364426b, the link reached atc47f11e9).Re #1976 (comment): fixed in
c47f11e9: the flags now include-Wno-error=profile-instr-out-of-date,-Wno-error=profile-instr-missingand-Wno-error=profile-instr-unprofiledafter the job's own.RealClangcompiles the matching, stale and unprofiled sources with-Werrorin the job's flags and requires exit 0 with clang's lines printed (it passes atc47f11e9with llvm-mingw clang 23).Re #1976 (comment): fixed in
c47f11e9(futondestroyer asked for it too):pgo_fit_reportsaves the hot pieces' records before the build stops on a failed file, and says the fit only when every file compiled.Coordinator.test_an_unrelated_failure_then_a_build_that_compilesfails a64_table twice, then builds again: the hot object is taken from the cache, not compiled again, and its saved record is said ("not known" at2364426b).Re #1976 (comment): partly fixed in
c47f11e9: the save writes through a temporary file of its own (mkstemp in the work folder, removed on failure) and a rename, so a reader never sees a partial file, and it merges this build's records over the file as it is at that moment. Two builds saving at the same instant can still lose the other's newest records; such a piece then reads "not known" in the next build, never a wrong fit.KeptFile.test_a_save_keeps_another_builds_recordscovers the merge; a lock across processes is left for a later round.Re #1976 (comment): fixed in
c47f11e9: main reads pgo_fit.json only when the jobs include a hot piece built with a profile, reads at most 1 MB (a bigger file is no file), and a deeply nested one (RecursionError) is no file.Coordinator.test_no_pieces_profile_no_line_and_no_kept_fileruns build.main without a profile with the reader made to fail if called, and the file left as it was.Re #1976 (comment): fixed in
c47f11e9without touching the object keys: the file has a format number and the stamp the third-party objects use (the compiler's version and driver defaults, the system headers as they preprocess, the Vulkan headers), and a file made under another stamp or format is read as no records.KeptFile.test_saved_and_read_backreads the same file under another stamp as empty.Re #1976 (comment): fixed in
2364426b: the real-clang check takes --cc, else CC, else the build's own clang, and asks clang itself for llvm-profdata (-print-prog-name: a ccache shim runs the real clang, which names the one beside it), then PATH with and without clang's major version. The test now declaresRUN_ALL_ARGS = ["--cc", "clang"], and given --cc (as run_all gives it) a clang without its profile runtime or llvm-profdata fails the check instead of skipping. It ran on Windows with llvm-mingw here; the Linux CI job has not run it yet.Re #1976 (comment): fixed in
2364426b: nx2/build.py is imported with the RX_, OOT_, LONGSHOT_, A64_ and HOS_ variables cleared, RX_MARCH=x86-64-v3 (no CPU probe with clang) and subprocess.run and Popen made to fail, so an import that starts pkg-config or clang fails the test. The coordinator tests run build.main with its tools stubbed as tests/test_a64_helper_build_flags.py does, and the only real tool invocations are inRealClang, named in the test's docstring.Re #1976 (comment): fixed in
2364426b: the stand-in compiler writes its notes as UTF-8, and every temporary folder of the test has a non-ASCII letter in its name (longshot-pgo-fit-é-). It ran on this PC's Windows 11 code page; a CP1252 system was not tried.Re #1976 (comment): fixed in
2364426b: theCoordinatortests run build.main itself with authored jobs and a stand-in compiler that prints clang's lines only for the warning groups it is given, through the compile, the retry alone and up to the link, and assert what main prints on a fresh compile, a cache hit, a stale profile, a failed file then a good build, malformed kept files and a build without a pieces profile. The reporting is one call in main,pgo_fit_report: with that line removed from a copy of the fix, four of the coordinator tests fail (run through the queue, in Testing).Re #1976 (comment): fixed in
c47f11e9: the RX_PGO row in docs/dev/reference/build.md now names the four kinds of piece, says the out-of-date verdict needs every piece's count, and that a piece with no record here (compiled before this, taken from another work folder's objects through the object store, or made with another compiler) is not known and is not compiled again for the report.Re #1976 (comment) (L14): the game-free coordinator check is in
2364426b(build.main with authored jobs: fresh, warm, stale, mixed, failed-then-good, malformed file, no profile). The by-hand Windows and Linux builds with fresh, stale and unprofiled pieces profiles, a warm repeat, the object keys, a game start and clean build time and size against the merge base have not run; they are in Still to test and pending with the person who runs the game builds here. The diagnostic counts and tuples in the test are authored, none comes from a game build.Tested corrected: an earlier run on this PC had no Vulkan headers for the renderer checks (
RX_CPMunset), so the ones it listed as also failing onmainhad compiled nothing on either side. They ran again with the headers atc47f11e9and onmain(queue job fb2-vk-1); the description now gives those results. No check fails at this head alone; nothing in the branch changed.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.