WIP: feat(build): nx2/dev.py says why the next build translates again, shows the builds' times, and builds again #28
No reviewers
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
needs measurements
question
testing wanted
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!28
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:feat/dev-tool"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
When a build translates the game again (about 5 minutes on Windows) there was no way to see why beyond "the key changed", and a build started any other way than
make_game.py's own had another environment, so other object and lift keys. This adds one entry point for developers,nx2/dev.py, and puts the build's environment in one place.What changed
This PR is stacked on #22: its diff against
maincarries #22's lift-key change (lift_reads_env(),LIFT_ENV_*innx2/make_game.py,check_lift_env()intests/test_lift_cache.py). The bullets below are this PR's own.build:
nx2/profiles.py(new): the environment the lift andbuild.pyrun with, the dictionarymake_game.pybuilt in place, asbuild_env(), in two profiles chosen byprofile():release(the builders') anddev(LONGSHOT_PROFILE=dev: noRX_RELEASE, so a console and no shader preparation at start; the lift is the same, since #22's key no longer hashesRX_RELEASE).TOOLCHAIN_ENVnames the toolchain's variables.build:
nx2/make_game.py:maintakes its environment fromprofiles.build_env(), andrecord_build()records how the build was started in<work>/last_build.json: its arguments, its Python, the profile, the toolchain's variables;lift_keykeeps what each key was made of inKEY_PARTS(the Python, the HLE functions, each file's hash by name, the variables), andwrite_lift_keywrites the stored key's parts to<work>/sa/lift_key_parts.json(LIFT_PARTS), outsidegen/.tools:
nx2/dev.py(new, standard library only):status: whether the next build skips the lift, and if not, which file, HLE function, variable or Python changed (lift_status(),differences());times [-n N]: the last builds' phases side by side (times_table(), from NotRexed/Hookshot#2693'sbuild_times.json);build [--dev]: the last build again, with its arguments, Python and tools, its--logleft out (build(),replayed());compdb: the last build'scompile_commands.json(NotRexed/Hookshot#2893) at the checkout's top, for clangd (compdb()).Nothing it prints comes from the game's files: names, times and hashes of the tools' own inputs.
tests:
tests/test_dev_tool.py(new): the differencesstatusnames (a file, a variable, the Python, an HLE function), the work folder it picks,times' table,build's replayed command (without--log) and its environment for each profile,compdb.tests/test_lift_cleanup_error.pytakeswrite_lift_key's new globals (KEY_PARTS,LIFT_PARTS).Flags and settings
LONGSHOT_PROFILEis the one new switch. The other names are existing variables the diff writes in quotes: the onesTOOLCHAIN_ENVrecords anddev.py buildpasses on, the two moved intoprofiles.build_env(), and the ones #22's lift-key change, carried in this diff, names.LONGSHOT_PROFILErelease(the builders'),dev(noRX_RELEASE: a console, no shader preparation at start);dev.py build --devsets itrelease)RX_RELEASEprofiles.build_env()in the release profile only (make_game.pyset it always); out of the lift key (#22)1in the release profile, as beforeRX_PGOprofiles.build_env()LONGSHOT_MINGWTOOLCHAIN_ENV(recorded, passed on bydev.py build); named inLIFT_ENV_NOT_OUTPUT(#22)LONGSHOT_MSVCTOOLCHAIN_ENVdev.py buildLONGSHOT_NGX_SDKTOOLCHAIN_ENVdev.py buildLONGSHOT_FSRTOOLCHAIN_ENVdev.py buildLONGSHOT_JOBSTOOLCHAIN_ENVdev.py buildRX_CPMTOOLCHAIN_ENVdev.py buildRX_MARCHTOOLCHAIN_ENV; out of the lift key (#22, named in its test)dev.py buildRX_TARGETTOOLCHAIN_ENV; stays in the lift key (LIFT_ENV_NAMES, #22)LONGSHOT_LIFT_JOBSLIFT_ENV_NOT_OUTPUT(#22)LONGSHOT_LIFT_NO_FORKLIFT_ENV_NOT_OUTPUT(#22)RX_JOBSLIFT_ENV_NOT_OUTPUT, #22)RX_VK_NAN_GUARDLIFT_ENV_NOT_OUTPUT, #22)nx2/bake_ds's own runRX_VK_NO_DRAW_PARAMSLIFT_ENV_NOT_OUTPUT, #22)nx2/bake_ds's own runRX_VK_NO_F16LIFT_ENV_NOT_OUTPUT, #22)nx2/bake_ds's own runRX_VK_SHARED_48KLIFT_ENV_NOT_OUTPUT, #22)nx2/bake_ds's own runRX_VK_NO_PROGRAM_CACHERX_VK_switch leaves the lift key unchangedMeasurements
Nothing to measure in the game: the change is to the build's tools and environment; the game from these builds plays (below).
What was run, on the GitHub version (NotRexed/Hookshot#2909), not on Longshot. Windows 11, in a work folder with a translation:
dev.py statusafterA64_NO_VFPX=1in the shell:lift: the next build translates the game again:thenenvironment A64_NO_VFPX: (unset) -> 1.dev.py buildreplayed the last build (the same arguments and Python, shown quoted).dev.py times -n 3showed the last builds' phases side by side.run_all.py, rebased on that day's GitHubmain, Windows: 54 pass;test_linux_updatefails the same way onmainthere.Testing
Tested: the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured:
dev.py status,buildandtimes, and the game from these builds (above).Not tested: Linux.
Needs other testers:
python -I nx2/dev.py statusafter a build, expecting it to name what changed since (request to open)Screenshots
Nothing on screen.
Risk and rollback
make_game.pynow takes its environment fromprofiles.build_env(); a difference from the dictionary it built in place would change object or lift keys and recompile or translate again (tests/test_dev_tool.pychecks the release profile is whatmake_game.pybuilt).record_buildandwrite_lift_keywrite two more files in the work folder (last_build.json,sa/lift_key_parts.json);record_buildignores a failed write.dev.pyis a new tool nothing else calls. Reverting the commits is enough.Reviewer notes
Carried over from NotRexed/Hookshot#2909 with the move to Longshot: the same change on Longshot's
main, with its rename applied. Its review and discussion are there.Stacked on #22 (carried from NotRexed/Hookshot#2874): review the commits after it; this rebases to them alone once #22 is in.
The dev-loop proposal's A9 (NotRexed/Hookshot#2636); its later pieces (scenario runs,
where, the bench lock, warnings) build on it.The checks and the in-game runs above were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks, and they were not run again here. Only the checks that read the changed files were run, not the whole
run_all.py.Issues
See NotRexed/Hookshot#2636 (A9), NotRexed/Hookshot#2874.
Checklist
For everyone
type(scope): description).main.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above).third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.docs/tested-configurations.mdfor each measured run.MKWVK_API_VERSIONbumped (gpu/vk/vk_api.h'sMkwVkHostorMkwVkRendererchanged).mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
feat(build): nx2/dev.py says why the next build translates again, shows the builds' times, and builds againto WIP: feat(build): nx2/dev.py says why the next build translates again, shows the builds' times, and builds againBack to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.
Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready.
Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.