WIP: fix(build): a direct lift removes an earlier lift's split parts before it writes its own #1176

Draft
TrashPanda wants to merge 1 commit from TrashPanda/Longshot:fix/relift-split-cleanup into main
Contributor

Type: fix · Risk: low · Tested on: Linux, RTX 4070 Ti (game-free checks) · Needs: none

Why

For developers: running nx2/lift.py --write again over a gen folder, after a saved chunk boundary moved, kept the earlier lift's split parts beside the new chunk with the same functions, and the link failed on duplicate definitions. Players' builds clear the folder first and are not affected.

Restored from Hookshot 2953, by jarynclouatre, and split out of #286 so it can be reviewed and merged on its own.

What changed

chunks.write removes an earlier lift's split parts together with its hot pieces, before split_big, so only this lift's parts are kept.

Commits, each with its original author and message:

8d034aab fix(build): a direct lift removes an earlier lift's split parts before it writes its own (jarynclouatre)

Developers only. nx2/lift.py --write run again over a gen folder,
after a saved boundary in chunks.txt moved to where the earlier lift had
cut a big chunk, kept that lift's split parts (code_NNN_M.c) beside
the new numbered chunk holding the same functions, and the link failed
on duplicate definitions.

chunks.write removed an earlier lift's stale numbered chunks and hot
pieces, but left its split parts for split_big. After the boundary
move the new chunk is the old first part byte for byte, so
_cut_already (meant for a worker that finished a cut whose result was
lost) found the old parts consistent with it and kept them as this
lift's. This needs the old first part to call none of the functions that
moved out: with such a call its declarations differ from the new
chunk's, and the old parts are rejected and removed, as before. The
packaged builder clears code_* before a translation and is not
affected.

Change

write() removes an earlier lift's split parts together with its hot
pieces, before split_big, so _cut_already only sees parts made in
this lift. split_big is called only from write(), and nothing
between the removal and split_big reads the parts.

Testing

  • tests/test_chunk_split.py, new case: two lifts through
    chunks.write with stand-in functions, the first with one chunk that
    split_big cuts, the second with a saved boundary at the first part's
    end. Each function is defined once and none of the first lift's parts is
    left. On main it fails with 333 of the 400 functions defined twice.
  • run_all.py chunk hot spawn lift: 11 passed.
  • Full suite on Linux: 2 failures, each for a tool this machine lacks
    (test_bake_portability: zstandard;
    nx2/test_texture_init_barrier.py: clang++); both fail the same on
    main.

Restored-from: https://github.com/NotRexed/Hookshot/pull/2953

Measurements

Verdict: nothing to measure: a correctness fix off the frame path, covered by game-free checks.

Flags and settings

None.

Testing

Tested: On Linux (kernel 7.0), RTX 4070 Ti, main @ 71b1edd8, this PR @ 8d034aab: tests/test_chunk_split.py pass. The full suite (tests/run_all.py) has not been run on this branch yet; a comment will follow when it has.

Not tested: A full lift of the game twice over one gen folder. Opened as work in progress until it runs in the game.

Must be tested:

  • None

Risks

What could break: a developer's direct lift.

Turning it off: revert the commit.

Compatibility: Nothing changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what
    can't be run by hand is in Testing's Not testable by hand, with why.
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and
    in Risks.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).

When it applies

  • A game-free test added for the fix, failing on main and passing here.
**Type:** fix · **Risk:** low · **Tested on:** Linux, RTX 4070 Ti (game-free checks) · **Needs:** none ## Why For developers: running `nx2/lift.py --write` again over a gen folder, after a saved chunk boundary moved, kept the earlier lift's split parts beside the new chunk with the same functions, and the link failed on duplicate definitions. Players' builds clear the folder first and are not affected. Restored from [Hookshot 2953](https://github.com/NotRexed/Hookshot/pull/2953), by jarynclouatre, and split out of [#286](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/286) so it can be reviewed and merged on its own. ## What changed `chunks.write` removes an earlier lift's split parts together with its hot pieces, before `split_big`, so only this lift's parts are kept. Commits, each with its original author and message: <details><summary><code>8d034aab</code> fix(build): a direct lift removes an earlier lift's split parts before it writes its own (jarynclouatre)</summary> Developers only. `nx2/lift.py --write` run again over a gen folder, after a saved boundary in chunks.txt moved to where the earlier lift had cut a big chunk, kept that lift's split parts (`code_NNN_M.c`) beside the new numbered chunk holding the same functions, and the link failed on duplicate definitions. `chunks.write` removed an earlier lift's stale numbered chunks and hot pieces, but left its split parts for `split_big`. After the boundary move the new chunk is the old first part byte for byte, so `_cut_already` (meant for a worker that finished a cut whose result was lost) found the old parts consistent with it and kept them as this lift's. This needs the old first part to call none of the functions that moved out: with such a call its declarations differ from the new chunk's, and the old parts are rejected and removed, as before. The packaged builder clears `code_*` before a translation and is not affected. #### Change `write()` removes an earlier lift's split parts together with its hot pieces, before `split_big`, so `_cut_already` only sees parts made in this lift. `split_big` is called only from `write()`, and nothing between the removal and `split_big` reads the parts. #### Testing - `tests/test_chunk_split.py`, new case: two lifts through `chunks.write` with stand-in functions, the first with one chunk that `split_big` cuts, the second with a saved boundary at the first part's end. Each function is defined once and none of the first lift's parts is left. On main it fails with 333 of the 400 functions defined twice. - `run_all.py chunk hot spawn lift`: 11 passed. - Full suite on Linux: 2 failures, each for a tool this machine lacks (`test_bake_portability`: `zstandard`; `nx2/test_texture_init_barrier.py`: `clang++`); both fail the same on main. Restored-from: https://github.com/NotRexed/Hookshot/pull/2953 </details> ## Measurements **Verdict:** nothing to measure: a correctness fix off the frame path, covered by game-free checks. ### Flags and settings None. ## Testing **Tested:** On Linux (kernel 7.0), RTX 4070 Ti, `main` @ 71b1edd8, this PR @ 8d034aab: `tests/test_chunk_split.py` pass. The full suite (`tests/run_all.py`) has not been run on this branch yet; a comment will follow when it has. **Not tested:** A full lift of the game twice over one gen folder. Opened as work in progress until it runs in the game. **Must be tested:** - None ## Risks **What could break:** a developer's direct lift. **Turning it off:** revert the commit. **Compatibility:** Nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Tested on my own PC where it runs (in the game, for a change the game runs), as Testing's Tested says; what can't be run by hand is in Testing's Not testable by hand, with why. - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Nothing gets worse that I know of: any cost (speed, latency, memory, a setup that worked before) is measured and in Risks. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). ### When it applies - [x] A game-free test added for the fix, failing on `main` and passing here.
fix(build): a direct lift removes an earlier lift's split parts before it writes its own
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
8d034aabaa
Developers only. `nx2/lift.py --write` run again over a gen folder,
after a saved boundary in chunks.txt moved to where the earlier lift had
cut a big chunk, kept that lift's split parts (`code_NNN_M.c`) beside
the new numbered chunk holding the same functions, and the link failed
on duplicate definitions.

`chunks.write` removed an earlier lift's stale numbered chunks and hot
pieces, but left its split parts for `split_big`. After the boundary
move the new chunk is the old first part byte for byte, so
`_cut_already` (meant for a worker that finished a cut whose result was
lost) found the old parts consistent with it and kept them as this
lift's. This needs the old first part to call none of the functions that
moved out: with such a call its declarations differ from the new
chunk's, and the old parts are rejected and removed, as before. The
packaged builder clears `code_*` before a translation and is not
affected.

## Change

`write()` removes an earlier lift's split parts together with its hot
pieces, before `split_big`, so `_cut_already` only sees parts made in
this lift. `split_big` is called only from `write()`, and nothing
between the removal and `split_big` reads the parts.

## Testing

- `tests/test_chunk_split.py`, new case: two lifts through
`chunks.write` with stand-in functions, the first with one chunk that
`split_big` cuts, the second with a saved boundary at the first part's
end. Each function is defined once and none of the first lift's parts is
left. On main it fails with 333 of the 400 functions defined twice.
- `run_all.py chunk hot spawn lift`: 11 passed.
- Full suite on Linux: 2 failures, each for a tool this machine lacks
(`test_bake_portability`: `zstandard`;
`nx2/test_texture_init_barrier.py`: `clang++`); both fail the same on
main.

Restored-from: https://github.com/NotRexed/Hookshot/pull/2953
Author
Contributor

Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @ 71b1edd8 (make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. The build's lift split its big chunks ("lift: big chunks split") and the game ran from the result.

Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @ 71b1edd8 (make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. The build's lift split its big chunks ("lift: big chunks split") and the game ran from the result.
Owner

Brought up to the ready-for-review standard (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know).

  • Tested on my own PC: ticked: Author's comment reports an in-game run on their Linux RTX 4070 Ti, with the lift's split chunks used by the game.
  • Nothing gets worse that I know of: ticked: Measurements gives a verdict of nothing to measure, and Risks names no cost.
Brought up to the [ready-for-review standard](https://git.magicbean.dev/MagicBeanDev/Longshot/src/branch/main/CONTRIBUTING.md#ready-for-review-out-of-wip) (#1454): the checklist's Everyone list has two new boxes, added here from what this PR's description and comments show (please check they match what you know). - *Tested on my own PC*: ticked: Author's comment reports an in-game run on their Linux RTX 4070 Ti, with the lift's split chunks used by the game. - *Nothing gets worse that I know of*: ticked: Measurements gives a verdict of nothing to measure, and Risks names no cost.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/relift-split-cleanup:TrashPanda-fix/relift-split-cleanup'
git switch 'TrashPanda-fix/relift-split-cleanup'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1176
No description provided.