WIP: feat(build): the lift key hashes only the environment that changes the lifted C

dyst commented

2026-11-12 23:38:22 +00:00

First-time contributor

Why

A switch left set in the shell made the next build translate the whole game again (about 5 minutes on Windows), even an RX_VK_* renderer switch or RX_RELEASE, which the lift never reads; so did the same game in another work or game folder, since the folders' paths were in the lift key.

What changed

build: lift_key in nx2/make_game.py hashes only the variables lift_reads_env() names, in place of every A64_, OOT_ and RX_ variable:
- in the key: LIFT_ENV_PREFIXES (A64_* and SUYU_RECOMP_*, the emitter's own switches, which weren't in the key at all before) and LIFT_ENV_NAMES (RX_TARGET);
- not in the key: the paths (LIFT_PATH_ENV: OOT_WORK, OOT_EXEFS, OOT_ROMFS, OOT_DUMP, A64_HOT_FUNCS), since what the lift reads through them is hashed by its contents already; the compile's own variables (COMPILE_ONLY_ENV); and LIFT_ENV_NOT_OUTPUT, the variables the lift's sources read that change nothing it writes (how many processes, where the compiler is), each with why.
build: the key's form changes, so the first build after this translates the game once.
tests: check_lift_env() in tests/test_lift_cache.py: the same game and inputs in other folders give the same key; RX_RELEASE, an RX_VK_ switch and RX_MARCH leave it unchanged, while SUYU_RECOMP_STRICT, RX_TARGET and an A64_ switch change it; every variable the lift's sources read is in the key, a path, or in LIFT_ENV_NOT_OUTPUT with a reason, so a new one can't slip through.

Flags and settings

No new switch. The names below are ones the diff writes in quotes: the existing variables the lift key now leaves out or keeps, and the ones the test sets.

Flag	New or changed	What it does	Default now	Goal: on by default?	Helps on	Doesn't help on	Untested on	Untested, but should help on
`LONGSHOT_LIFT_JOBS`	existing; named in `LIFT_ENV_NOT_OUTPUT` (it was not in the key before either)	how many processes the lift runs	unset	no (a developer switch)	-	-	Linux	-
`LONGSHOT_LIFT_NO_FORK`	existing; named in `LIFT_ENV_NOT_OUTPUT` (not in the key before either)	how the lift's processes are started	unset	no (a developer switch)	-	-	Linux	-
`LONGSHOT_MINGW`	existing; named in `LIFT_ENV_NOT_OUTPUT` (not in the key before either)	where the compiler is	unset	no (a developer switch)	-	-	Linux	-
`RX_JOBS`	changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`)	how many processes	unset	no (a developer switch)	-	-	Linux	-
`RX_VK_NAN_GUARD`	changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`)	a renderer switch; the lift's sources read it only for `nx2/bake_ds`'s own run	unset	no (a developer switch)	-	-	Linux	-
`RX_VK_NO_DRAW_PARAMS`	changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`)	a renderer switch; read by the lift's sources only for `nx2/bake_ds`'s own run	unset	no (a developer switch)	-	-	Linux	-
`RX_VK_NO_F16`	changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`)	a renderer switch; read by the lift's sources only for `nx2/bake_ds`'s own run	unset	no (a developer switch)	-	-	Linux	-
`RX_VK_SHARED_48K`	changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`)	a renderer switch; read by the lift's sources only for `nx2/bake_ds`'s own run	unset	no (a developer switch)	-	-	Linux	-
`RX_TARGET`	unchanged: stays in the lift key (`LIFT_ENV_NAMES`); also set in the test	the target the emitter and its tables are made for	unset	no (a developer switch)	-	-	Linux	-
`RX_MARCH`	changed: out of the lift key; named in the test only (test only)	test only: the check that it leaves the key unchanged	unset	no (a developer switch)	-	-	Linux	-
`RX_RELEASE`	changed: out of the lift key; named in the test only (test only)	test only: the check that it leaves the key unchanged	unset	no (a developer switch)	-	-	Linux	-
`RX_VK_NO_PROGRAM_CACHE`	changed: out of the lift key; named in the test only (test only)	test only: the check that an `RX_VK_` switch leaves the key unchanged	unset	no (a developer switch)	-	-	Linux	-

Measurements

Measured on the GitHub version (NotRexed/Hookshot#2874), not yet on Longshot: this PR stays WIP until it is.

Hardware: RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11
Method: after the first lift with the new key, a build with RX_VK_NO_PROGRAM_CACHE=1 in the shell.

	GitHub main	this PR (GitHub)
A build with `RX_VK_NO_PROGRAM_CACHE=1` in the shell	translated the game again	skipped the lift (15 s)

In the game: every build run in the game that day carried this make_game.py. The latest, headless through the control bridge from a save, in play in Kokiri Forest: 28-31 fps, nothing in the log about a lost device or a crash. The change decides only whether the lift runs; the C it writes is the same.

Checks: every check that reads the changed files (54), through run_all.py, rebased on that day's GitHub main, on Windows: 53 pass; test_linux_update fails the same way on main there.

Testing

Tested: the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured: the in-game run above (no main run beside it; the lifted C is the same).

Not tested: Linux. Nothing here is platform-specific.

Needs other testers:

Linux: a build after changing only a variable outside the lift's own (an A64_, OOT_ or RX_ one changes the C) skips the lift, expecting "lift: skipped" in the build's lines (request to open)

Screenshots

Nothing on screen.

Risk and rollback

A variable the lift does read but the key now leaves out would let a build reuse a translation made without it; check_lift_env() fails when the lift's sources read a variable that is neither in the key, a path, nor in LIFT_ENV_NOT_OUTPUT. The first build after this translates the game once (the key's form changes). Reverting the commit is enough (one more translation after it).

Reviewer notes

Carried over from NotRexed/Hookshot#2874 with the move to Longshot: the same change on Longshot's main, with its rename applied. Its review and discussion are there.

This is the dev-loop proposal's A2 (NotRexed/Hookshot#2636), with hckrman101's review point that the paths kept two checkouts of one tree from sharing a lift.

The checks and the in-game runs above were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks, and they were not run again here. Only the checks that read the changed files were run, not the whole run_all.py.

#28 is stacked on this one.

Issues

See NotRexed/Hookshot#2636 (A2).

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags table filled in; README / docs/development.md switch reference updated.
Screenshots for a change that shows on screen.
A row in docs/tested-configurations.md for each measured run.
MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed).
The SPIR-V headers regenerated with glslang (a shader changed).
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
Docs updated.

🤖 Generated with Claude Code

## Why A switch left set in the shell made the next build translate the whole game again (about 5 minutes on Windows), even an `RX_VK_*` renderer switch or `RX_RELEASE`, which the lift never reads; so did the same game in another work or game folder, since the folders' paths were in the lift key. ## What changed - build: `lift_key` in `nx2/make_game.py` hashes only the variables `lift_reads_env()` names, in place of every `A64_`, `OOT_` and `RX_` variable: - in the key: `LIFT_ENV_PREFIXES` (`A64_*` and `SUYU_RECOMP_*`, the emitter's own switches, which weren't in the key at all before) and `LIFT_ENV_NAMES` (`RX_TARGET`); - not in the key: the paths (`LIFT_PATH_ENV`: `OOT_WORK`, `OOT_EXEFS`, `OOT_ROMFS`, `OOT_DUMP`, `A64_HOT_FUNCS`), since what the lift reads through them is hashed by its contents already; the compile's own variables (`COMPILE_ONLY_ENV`); and `LIFT_ENV_NOT_OUTPUT`, the variables the lift's sources read that change nothing it writes (how many processes, where the compiler is), each with why. - build: the key's form changes, so the first build after this translates the game once. - tests: `check_lift_env()` in `tests/test_lift_cache.py`: the same game and inputs in other folders give the same key; `RX_RELEASE`, an `RX_VK_` switch and `RX_MARCH` leave it unchanged, while `SUYU_RECOMP_STRICT`, `RX_TARGET` and an `A64_` switch change it; every variable the lift's sources read is in the key, a path, or in `LIFT_ENV_NOT_OUTPUT` with a reason, so a new one can't slip through. ## Flags and settings No new switch. The names below are ones the diff writes in quotes: the existing variables the lift key now leaves out or keeps, and the ones the test sets. | Flag | New or changed | What it does | Default now | Goal: on by default? | Helps on | Doesn't help on | Untested on | Untested, but should help on | |---|---|---|---|---|---|---|---|---| | `LONGSHOT_LIFT_JOBS` | existing; named in `LIFT_ENV_NOT_OUTPUT` (it was not in the key before either) | how many processes the lift runs | unset | no (a developer switch) | - | - | Linux | - | | `LONGSHOT_LIFT_NO_FORK` | existing; named in `LIFT_ENV_NOT_OUTPUT` (not in the key before either) | how the lift's processes are started | unset | no (a developer switch) | - | - | Linux | - | | `LONGSHOT_MINGW` | existing; named in `LIFT_ENV_NOT_OUTPUT` (not in the key before either) | where the compiler is | unset | no (a developer switch) | - | - | Linux | - | | `RX_JOBS` | changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`) | how many processes | unset | no (a developer switch) | - | - | Linux | - | | `RX_VK_NAN_GUARD` | changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`) | a renderer switch; the lift's sources read it only for `nx2/bake_ds`'s own run | unset | no (a developer switch) | - | - | Linux | - | | `RX_VK_NO_DRAW_PARAMS` | changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`) | a renderer switch; read by the lift's sources only for `nx2/bake_ds`'s own run | unset | no (a developer switch) | - | - | Linux | - | | `RX_VK_NO_F16` | changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`) | a renderer switch; read by the lift's sources only for `nx2/bake_ds`'s own run | unset | no (a developer switch) | - | - | Linux | - | | `RX_VK_SHARED_48K` | changed: out of the lift key (`LIFT_ENV_NOT_OUTPUT`) | a renderer switch; read by the lift's sources only for `nx2/bake_ds`'s own run | unset | no (a developer switch) | - | - | Linux | - | | `RX_TARGET` | unchanged: stays in the lift key (`LIFT_ENV_NAMES`); also set in the test | the target the emitter and its tables are made for | unset | no (a developer switch) | - | - | Linux | - | | `RX_MARCH` | changed: out of the lift key; named in the test only (test only) | test only: the check that it leaves the key unchanged | unset | no (a developer switch) | - | - | Linux | - | | `RX_RELEASE` | changed: out of the lift key; named in the test only (test only) | test only: the check that it leaves the key unchanged | unset | no (a developer switch) | - | - | Linux | - | | `RX_VK_NO_PROGRAM_CACHE` | changed: out of the lift key; named in the test only (test only) | test only: the check that an `RX_VK_` switch leaves the key unchanged | unset | no (a developer switch) | - | - | Linux | - | ## Measurements Measured on the GitHub version (NotRexed/Hookshot#2874), not yet on Longshot: this PR stays WIP until it is. Hardware: RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11 Method: after the first lift with the new key, a build with `RX_VK_NO_PROGRAM_CACHE=1` in the shell. | | GitHub main | this PR (GitHub) | |---|---|---| | A build with `RX_VK_NO_PROGRAM_CACHE=1` in the shell | translated the game again | skipped the lift (15 s) | In the game: every build run in the game that day carried this `make_game.py`. The latest, headless through the control bridge from a save, in play in Kokiri Forest: 28-31 fps, nothing in the log about a lost device or a crash. The change decides only whether the lift runs; the C it writes is the same. Checks: every check that reads the changed files (54), through `run_all.py`, rebased on that day's GitHub `main`, on Windows: 53 pass; `test_linux_update` fails the same way on `main` there. ## Testing **Tested:** the runs under Measurements (on the GitHub version) and the game-free checks this description names. Checked but not measured: the in-game run above (no `main` run beside it; the lifted C is the same). **Not tested:** Linux. Nothing here is platform-specific. **Needs other testers:** - Linux: a build after changing only a variable outside the lift's own (an `A64_`, `OOT_` or `RX_` one changes the C) skips the lift, expecting "lift: skipped" in the build's lines (request to open) ## Screenshots Nothing on screen. ## Risk and rollback A variable the lift does read but the key now leaves out would let a build reuse a translation made without it; `check_lift_env()` fails when the lift's sources read a variable that is neither in the key, a path, nor in `LIFT_ENV_NOT_OUTPUT`. The first build after this translates the game once (the key's form changes). Reverting the commit is enough (one more translation after it). ## Reviewer notes Carried over from [NotRexed/Hookshot#2874](https://github.com/NotRexed/Hookshot/pull/2874) with the move to Longshot: the same change on Longshot's `main`, with its rename applied. Its review and discussion are there. This is the dev-loop proposal's A2 ([NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636)), with hckrman101's review point that the paths kept two checkouts of one tree from sharing a lift. The checks and the in-game runs above were done on the GitHub version; this branch is the same change with the rename applied (the same files and line counts). Forgejo runs no checks, and they were not run again here. Only the checks that read the changed files were run, not the whole `run_all.py`. #28 is stacked on this one. ## Issues See [NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636) (A2). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] Flags table filled in; README / docs/development.md switch reference updated. - [ ] Screenshots for a change that shows on screen. - [ ] A row in `docs/tested-configurations.md` for each measured run. - [ ] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). - [ ] The SPIR-V headers regenerated with glslang (a shader changed). - [ ] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [ ] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit

2026-11-12 23:38:22 +00:00

feat(build): the lift key hashes only the environment that changes the lifted C eea9128f8f

The key took every A64_, OOT_ and RX_ variable, so an RX_VK_ switch left set in the shell, or RX_RELEASE, which the lift
does not read, translated the game again (~5 minutes on Windows), and so did the same game in another work or game
folder: OOT_WORK, OOT_EXEFS, OOT_ROMFS, OOT_DUMP and A64_HOT_FUNCS are paths, and what the lift reads through them is
hashed by its contents already. The emitter's own SUYU_RECOMP_ switches were not in the key at all. This is the dev-loop
proposal's A2 (#2636), with hckrman101's point that the paths keep two checkouts of one tree from sharing a lift.

lift_key now hashes the variables lift_reads_env() names: A64_ and SUYU_RECOMP_, and RX_TARGET, but not the paths
(LIFT_PATH_ENV) nor the compile's own (COMPILE_ONLY_ENV). tests/test_lift_cache.py checks that the same game and inputs in
other folders give the same key, that RX_RELEASE, an RX_VK_ switch and RX_MARCH leave it and SUYU_RECOMP_STRICT, RX_TARGET
and an A64_ switch change it, and that every variable the lift's sources read is in the key, a path, or in
LIFT_ENV_NOT_OUTPUT with why it changes nothing. The key's form changes, so the first build after this translates once.

Measured on Windows: after that first lift, a build with RX_VK_NO_PROGRAM_CACHE=1 in the shell skipped the lift (15 s).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst referenced this pull request

2026-11-12 23:38:38 +00:00

WIP: feat(build): nx2/dev.py says why the next build translates again, shows the builds' times, and builds again #28

dyst changed title from ~~feat(build): the lift key hashes only the environment that changes the lifted C~~ to WIP: feat(build): the lift key hashes only the environment that changes the lifted C

2026-11-13 01:13:44 +00:00

dyst commented

2026-11-13 01:13:45 +00:00

Author

First-time contributor

Back to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.

dyst commented

2026-11-13 01:58:11 +00:00

Author

First-time contributor

Description brought to Longshot's PR template (#48). Still WIP: its numbers are from the GitHub version, and it isn't rebased on current main; it gets measured on Longshot (main against this PR, at named commits) before it is marked ready.

dyst commented

2026-11-13 02:00:27 +00:00

Author

First-time contributor

Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'feat/lift-key-env:dyst-feat/lift-key-env'

git switch 'dyst-feat/lift-key-env'

Rows
Columns

WIP: feat(build): the lift key hashes only the environment that changes the lifted C #22

Why

What changed

Flags and settings

Measurements

Testing

Screenshots

Risk and rollback

Reviewer notes

Issues

Checklist

For everyone

If applicable

Checkout