WIP: fix(builder): a refused extracted dump leaves the saved copy of the game code alone #1175

Draft
TrashPanda wants to merge 1 commit from TrashPanda/Longshot:fix/refused-dump-keeps-code into main
Contributor

Type: fix · Risk: low · Tested on: Linux, RTX 4070 Ti (game-free checks) · Needs: none

Why

Building a game folder again from an extracted dump of an unsupported version was refused, as it should be, but only after the dump's modules had replaced the folder's saved game code. Choosing the earlier, supported .nup again then reused the refused code and was refused too, until the folder was deleted.

Restored from Hookshot 2968, by jarynclouatre, and split out of #286 so it can be reviewed and merged on its own.

How to reproduce

Seen on: any PC
Settings and switches: defaults
Steps: build from a supported .nup, then from an extracted dump of another version (refused), then from the .nup again
Expected: the last build works
Actual: the last build is refused as well

What changed

nx2/make_game.py checks the build right after the dump's modules are unpacked, before .longshot_exefs is touched, and an accepted dump removes the .nup's stamp so the .nup is read again next time.

Commits, each with its original author and message:

75241396 fix(builder): a refused extracted dump leaves the saved copy of the game code alone (jarynclouatre)

In Hookshot Builder, a game folder that was built from a supported
.nup can be rebuilt from an extracted dump. When that dump is of
another game version, the build is refused ("This is build X. Hookshot
supports build Y"), which is right, but nx2/make_game.py checked the
version only after the dump's modules had replaced the folder's saved
copy (.hookshot_exefs). The stamp of the earlier .nup stayed.
Choosing that .nup again then said "reading it again is skipped",
built from the refused modules and was refused too, until the folder was
deleted.

In the extracted-dump branch of main():

  • check_supported_build runs right after the dump's modules are
    unpacked, before the romfs is linked in and before .hookshot_exefs is
    touched. A refused dump leaves the folder as it was.
  • An accepted dump removes the .nup stamp (.hookshot_romfs), since
    the saved copies are no longer that .nup's. The next build from the
    .nup reads it again.

The .nup branch already checks the build before it replaces anything;
it is unchanged.

  • tests/test_make_game_cache.py, two new cases (the cache test's fake
    .nup and modules, no game):
  • A refused extracted dump: the folder's files are byte for byte as
    before, and the earlier .nup then reuses its copy and reaches the lift
    with the supported build.
  • An accepted extracted dump: its modules are saved, the .nup stamp is
    gone, and the earlier .nup is read again.
    • Both fail on main and pass here.
  • Full suite on Linux: 473 passed, 53 skipped, 2 failed, both the same
    on main on this machine (test_bake_portability: zstandard is not
    installed; nx2/test_texture_init_barrier.py: clang++ is not
    installed).
  • No real game build was run.

Restored-from: https://github.com/NotRexed/Hookshot/pull/2968

Measurements

Verdict: nothing to measure: a correctness fix off the frame path, covered by game-free checks.

Flags and settings

None.

Testing

Tested: On Linux (kernel 7.0), RTX 4070 Ti, main @ 71b1edd8, this PR @ 75241396: tests/test_make_game_cache.py pass. tests/test_make_game_cache.py fails here on main too (19 failures: the harness mocks subprocess.Popen, which also catches the real clang profile probe on this PC). With that probe stubbed, all 41 cases pass, the two new ones included. The full suite (tests/run_all.py) has not been run on this branch yet; a comment will follow when it has.

Not tested: A real game build from a refused dump. Opened as work in progress until it runs in the game.

Must be tested:

  • Any OS, a game dump of another version (by hand): build from the .nup, then the other dump, then the .nup again, expecting the last build to work

Risks

What could break: the builder's extracted-dump path.

Turning it off: revert the commit.

Compatibility: Nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).

When it applies

  • A game-free test added for the fix, failing on main and passing here.
**Type:** fix · **Risk:** low · **Tested on:** Linux, RTX 4070 Ti (game-free checks) · **Needs:** none ## Why Building a game folder again from an extracted dump of an unsupported version was refused, as it should be, but only after the dump's modules had replaced the folder's saved game code. Choosing the earlier, supported `.nup` again then reused the refused code and was refused too, until the folder was deleted. Restored from [Hookshot 2968](https://github.com/NotRexed/Hookshot/pull/2968), by jarynclouatre, and split out of [#286](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/286) so it can be reviewed and merged on its own. ## How to reproduce **Seen on:** any PC **Settings and switches:** defaults **Steps:** build from a supported `.nup`, then from an extracted dump of another version (refused), then from the `.nup` again **Expected:** the last build works **Actual:** the last build is refused as well ## What changed `nx2/make_game.py` checks the build right after the dump's modules are unpacked, before `.longshot_exefs` is touched, and an accepted dump removes the `.nup`'s stamp so the `.nup` is read again next time. Commits, each with its original author and message: <details><summary><code>75241396</code> fix(builder): a refused extracted dump leaves the saved copy of the game code alone (jarynclouatre)</summary> In Hookshot Builder, a game folder that was built from a supported `.nup` can be rebuilt from an extracted dump. When that dump is of another game version, the build is refused ("This is build X. Hookshot supports build Y"), which is right, but `nx2/make_game.py` checked the version only after the dump's modules had replaced the folder's saved copy (`.hookshot_exefs`). The stamp of the earlier `.nup` stayed. Choosing that `.nup` again then said "reading it again is skipped", built from the refused modules and was refused too, until the folder was deleted. In the extracted-dump branch of `main()`: - `check_supported_build` runs right after the dump's modules are unpacked, before the romfs is linked in and before `.hookshot_exefs` is touched. A refused dump leaves the folder as it was. - An accepted dump removes the `.nup` stamp (`.hookshot_romfs`), since the saved copies are no longer that `.nup`'s. The next build from the `.nup` reads it again. The `.nup` branch already checks the build before it replaces anything; it is unchanged. - `tests/test_make_game_cache.py`, two new cases (the cache test's fake `.nup` and modules, no game): - A refused extracted dump: the folder's files are byte for byte as before, and the earlier `.nup` then reuses its copy and reaches the lift with the supported build. - An accepted extracted dump: its modules are saved, the `.nup` stamp is gone, and the earlier `.nup` is read again. - Both fail on main and pass here. - Full suite on Linux: 473 passed, 53 skipped, 2 failed, both the same on main on this machine (`test_bake_portability`: `zstandard` is not installed; `nx2/test_texture_init_barrier.py`: `clang++` is not installed). - No real game build was run. Restored-from: https://github.com/NotRexed/Hookshot/pull/2968 </details> ## Measurements **Verdict:** nothing to measure: a correctness fix off the frame path, covered by game-free checks. ### Flags and settings None. ## Testing **Tested:** On Linux (kernel 7.0), RTX 4070 Ti, `main` @ 71b1edd8, this PR @ 75241396: `tests/test_make_game_cache.py` pass. `tests/test_make_game_cache.py` fails here on `main` too (19 failures: the harness mocks `subprocess.Popen`, which also catches the real clang profile probe on this PC). With that probe stubbed, all 41 cases pass, the two new ones included. The full suite (`tests/run_all.py`) has not been run on this branch yet; a comment will follow when it has. **Not tested:** A real game build from a refused dump. Opened as work in progress until it runs in the game. **Must be tested:** - Any OS, a game dump of another version (by hand): build from the `.nup`, then the other dump, then the `.nup` again, expecting the last build to work ## Risks **What could break:** the builder's extracted-dump path. **Turning it off:** revert the commit. **Compatibility:** Nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). ### When it applies - [x] A game-free test added for the fix, failing on `main` and passing here.
fix(builder): a refused extracted dump leaves the saved copy of the game code alone
Some checks are pending
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
752413964d
In Hookshot Builder, a game folder that was built from a supported
`.nup` can be rebuilt from an extracted dump. When that dump is of
another game version, the build is refused ("This is build X. Hookshot
supports build Y"), which is right, but `nx2/make_game.py` checked the
version only after the dump's modules had replaced the folder's saved
copy (`.hookshot_exefs`). The stamp of the earlier `.nup` stayed.
Choosing that `.nup` again then said "reading it again is skipped",
built from the refused modules and was refused too, until the folder was
deleted.

In the extracted-dump branch of `main()`:

- `check_supported_build` runs right after the dump's modules are
unpacked, before the romfs is linked in and before `.hookshot_exefs` is
touched. A refused dump leaves the folder as it was.
- An accepted dump removes the `.nup` stamp (`.hookshot_romfs`), since
the saved copies are no longer that `.nup`'s. The next build from the
`.nup` reads it again.

The `.nup` branch already checks the build before it replaces anything;
it is unchanged.

- `tests/test_make_game_cache.py`, two new cases (the cache test's fake
`.nup` and modules, no game):
- A refused extracted dump: the folder's files are byte for byte as
before, and the earlier `.nup` then reuses its copy and reaches the lift
with the supported build.
- An accepted extracted dump: its modules are saved, the `.nup` stamp is
gone, and the earlier `.nup` is read again.
  - Both fail on main and pass here.
- Full suite on Linux: 473 passed, 53 skipped, 2 failed, both the same
on main on this machine (`test_bake_portability`: `zstandard` is not
installed; `nx2/test_texture_init_barrier.py`: `clang++` is not
installed).
- No real game build was run.

Restored-from: https://github.com/NotRexed/Hookshot/pull/2968
Author
Contributor

Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @ 71b1edd8 (make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. The build was an update of a folder with its kept game code; no dump was refused, so the changed path did not run.

Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @ 71b1edd8 (make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. The build was an update of a folder with its kept game code; no dump was refused, so the changed path did not run.
Collaborator

Possibly related, for context:

  • PR #546 fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does — Open builder PR also changes make_game.py and cache tests; check for interaction.
  • PR #360 feat(build): a new version's game files hard-linked from another game folder of the same .nup — Merged builder change validates game code before replacing saved files; related safeguard.
Possibly related, for context: - PR #546 fix(build): a game built from a romfs folder gets its damaged textures replaced, as one from a .nup does — Open builder PR also changes make_game.py and cache tests; check for interaction. - PR #360 feat(build): a new version's game files hard-linked from another game folder of the same .nup — Merged builder change validates game code before replacing saved files; related safeguard.
Owner

Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).

  • Tested on my own PC: ticked: Author's comment reports a scripted in-game run on their Linux RTX 4070 Ti, though the changed dump-refusal path did not run.
  • Nothing gets worse that I know of: ticked: Measurements gives a verdict of nothing to measure, and Risks names no slower, costlier or breaking change.
Brought up to the [ready-for-review standard](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/CONTRIBUTING.md#ready-for-review-out-of-wip) (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know). - *Tested on my own PC*: ticked: Author's comment reports a scripted in-game run on their Linux RTX 4070 Ti, though the changed dump-refusal path did not run. - *Nothing gets worse that I know of*: ticked: Measurements gives a verdict of nothing to measure, and Risks names no slower, costlier or breaking change.
Some checks are pending
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/refused-dump-keeps-code:TrashPanda-fix/refused-dump-keeps-code'
git switch 'TrashPanda-fix/refused-dump-keeps-code'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1175
No description provided.