WIP: fix(vk): the shipped state list is made behind the preparing screen, and a half-written pipeline list is cut back #1161

Draft
TrashPanda wants to merge 3 commits from TrashPanda/Longshot:fix/pipeline-list-fixes into main
Contributor

Type: fix · Risk: low · Tested on: Linux, RTX 4070 Ti (game-free checks) · Needs: none

Why

Two ways pipelines that should have been ready were built in play instead, as a stutter: an update prepared behind the preparing screen skipped the shipped state list, and a pipeline list or notes file left half-written by a crash or a full disk had every later run's pipelines appended after the broken end, where they were never read back.

Restored from Hookshot 2982, Hookshot 2983, by Joey Magnus, jarynclouatre, and split out of #287 so it can be reviewed and merged on its own.

What changed

prepare_with_screen calls state_list_run() for an update before precompile_run, as the console start already does. cut_part_record cuts a pipeline list or notes file back to its last whole record before it is appended to.

Commits, each with its original author and message:

d217bfe5 fix(vk): a background-prepared update makes the state list's pipelines behind the preparing screen too (jarynclouatre)

With RX_VK_PREPARE_BACKGROUND=1, an update builds what the notes name
before the game and the rest behind it. A start with console progress
(started_up in gpu/vk/vk_main.cpp) makes the shipped state list's
pipelines first: if (update) state_list_run(); before
precompile_run. The preparing screen's worker (prepare_with_screen)
has no such call, so a start that shows the screen left those pipelines
out, and they were built when first drawn, in play.

Change

The screen's worker calls state_list_run() for an update at the same
point, before precompile_run, inside its !prepare_stopped() check. A
stopped preparation still skips it. Nothing is added to the screen:
state_list_run takes no progress callback.

Testing

  • New tests/test_prepare_screen_sequence.py: both step sequences are
    cut from the source and run against stand-ins that record each call. For
    an update, both make the state list once, before precompile_run, then
    start the background pass. When it is not an update, neither makes the
    list. A stopped screen worker makes neither the list nor the precompile.
    On main the screen's update sequence has no state list; here it passes.
  • Full suite on Linux: 474 passed, 53 skipped, 2 failed, both the same
    on main on this machine (test_bake_portability: zstandard is not
    installed; nx2/test_texture_init_barrier.py: clang++ is not
    installed).
  • No game ran.

Restored-from: https://github.com/NotRexed/Hookshot/pull/2982

47ce4192 fix(vk): a half-written last record is cut off before the pipeline lists are added to (jarynclouatre)

A crash or a full disk while the game writes
hookshot_pipeline_states.bin can leave it ending in part of a record.
state_open_locked in gpu/vk/vk_draw.cpp reads the whole records,
sets keep, and opens the file with "ab", so every later run added
its new pipelines after the broken end. They were never read back: those
pipelines were built again in play at each start, and "Save my pipeline
list" sent a list whose end was garbage. notes_open_locked does the
same with the pipeline notes, also on its known path, where the file
is not read at all.

Change

A small cut_part_record(path, header, record) cuts the file back to
its last whole record with std::filesystem::resize_file and logs it.
Both opens call it before appending: the state list when it keeps the
list, the notes when the stamp and record size match. The whole records
stay; a list that ends on a record's edge is left alone, and one shorter
than its header starts new as before.

Testing

  • New tests/test_state_list_part_record.py: state_open_locked,
    notes_open_locked and the helper are cut from the source and run on
    lists of their own: one whole record followed by 0 bytes, 1 byte, and a
    record's size less one of the next. A new record is added as play adds
    one, then the file is read back: both records there, the file a whole
    number of records. The notes the same, with their records read before
    the open and without. On main the part-record cases fail; here they
    pass.
  • tests/test_pipeline_state_computes.py compiles state_open_locked
    too; its fixture now takes the helper along.
  • Linux renderer built (hookshot_vk.so).
  • Full suite on Linux: 474 passed, 53 skipped, 2 failed, both the same
    on main on this machine (test_bake_portability: zstandard is not
    installed; nx2/test_texture_init_barrier.py: clang++ is not
    installed).
  • No game ran; not built on Windows.

Restored-from: https://github.com/NotRexed/Hookshot/pull/2983

2f69945f test: the prepare sequence's stand-ins have main's compile_all (Compile shaders on All) (Joey Magnus)

Measurements

Verdict: nothing to measure: a correctness fix off the frame path, covered by game-free checks.

Flags and settings

None.

Testing

Tested: On Linux (kernel 7.0), RTX 4070 Ti, main @ 71b1edd8, this PR @ 2f69945f: the checks that read the changed files (tests/run_all.py --changed origin/main) pass. The full suite (tests/run_all.py) has not been run on this branch yet; a comment will follow when it has.

Not tested: An update with the preparing screen; a cut list in the game. Opened as work in progress until it runs in the game.

Must be tested:

  • None: the checks run both sequences and a cut file

Risks

What could break: the preparing screen's order, and the pipeline lists' files.

Turning it off: revert the commit.

Compatibility: A list with a broken end is cut back once, logged.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).

When it applies

  • A game-free test added for the fix, failing on main and passing here.
**Type:** fix · **Risk:** low · **Tested on:** Linux, RTX 4070 Ti (game-free checks) · **Needs:** none ## Why Two ways pipelines that should have been ready were built in play instead, as a stutter: an update prepared behind the preparing screen skipped the shipped state list, and a pipeline list or notes file left half-written by a crash or a full disk had every later run's pipelines appended after the broken end, where they were never read back. Restored from [Hookshot 2982](https://github.com/NotRexed/Hookshot/pull/2982), [Hookshot 2983](https://github.com/NotRexed/Hookshot/pull/2983), by Joey Magnus, jarynclouatre, and split out of [#287](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/287) so it can be reviewed and merged on its own. ## What changed `prepare_with_screen` calls `state_list_run()` for an update before `precompile_run`, as the console start already does. `cut_part_record` cuts a pipeline list or notes file back to its last whole record before it is appended to. Commits, each with its original author and message: <details><summary><code>d217bfe5</code> fix(vk): a background-prepared update makes the state list's pipelines behind the preparing screen too (jarynclouatre)</summary> With `RX_VK_PREPARE_BACKGROUND=1`, an update builds what the notes name before the game and the rest behind it. A start with console progress (`started_up` in `gpu/vk/vk_main.cpp`) makes the shipped state list's pipelines first: `if (update) state_list_run();` before `precompile_run`. The preparing screen's worker (`prepare_with_screen`) has no such call, so a start that shows the screen left those pipelines out, and they were built when first drawn, in play. #### Change The screen's worker calls `state_list_run()` for an update at the same point, before `precompile_run`, inside its `!prepare_stopped()` check. A stopped preparation still skips it. Nothing is added to the screen: `state_list_run` takes no progress callback. #### Testing - New `tests/test_prepare_screen_sequence.py`: both step sequences are cut from the source and run against stand-ins that record each call. For an update, both make the state list once, before `precompile_run`, then start the background pass. When it is not an update, neither makes the list. A stopped screen worker makes neither the list nor the precompile. On main the screen's update sequence has no state list; here it passes. - Full suite on Linux: 474 passed, 53 skipped, 2 failed, both the same on main on this machine (`test_bake_portability`: `zstandard` is not installed; `nx2/test_texture_init_barrier.py`: `clang++` is not installed). - No game ran. Restored-from: https://github.com/NotRexed/Hookshot/pull/2982 </details> <details><summary><code>47ce4192</code> fix(vk): a half-written last record is cut off before the pipeline lists are added to (jarynclouatre)</summary> A crash or a full disk while the game writes `hookshot_pipeline_states.bin` can leave it ending in part of a record. `state_open_locked` in `gpu/vk/vk_draw.cpp` reads the whole records, sets `keep`, and opens the file with `"ab"`, so every later run added its new pipelines after the broken end. They were never read back: those pipelines were built again in play at each start, and "Save my pipeline list" sent a list whose end was garbage. `notes_open_locked` does the same with the pipeline notes, also on its `known` path, where the file is not read at all. #### Change A small `cut_part_record(path, header, record)` cuts the file back to its last whole record with `std::filesystem::resize_file` and logs it. Both opens call it before appending: the state list when it keeps the list, the notes when the stamp and record size match. The whole records stay; a list that ends on a record's edge is left alone, and one shorter than its header starts new as before. #### Testing - New `tests/test_state_list_part_record.py`: `state_open_locked`, `notes_open_locked` and the helper are cut from the source and run on lists of their own: one whole record followed by 0 bytes, 1 byte, and a record's size less one of the next. A new record is added as play adds one, then the file is read back: both records there, the file a whole number of records. The notes the same, with their records read before the open and without. On main the part-record cases fail; here they pass. - `tests/test_pipeline_state_computes.py` compiles `state_open_locked` too; its fixture now takes the helper along. - Linux renderer built (`hookshot_vk.so`). - Full suite on Linux: 474 passed, 53 skipped, 2 failed, both the same on main on this machine (`test_bake_portability`: `zstandard` is not installed; `nx2/test_texture_init_barrier.py`: `clang++` is not installed). - No game ran; not built on Windows. Restored-from: https://github.com/NotRexed/Hookshot/pull/2983 </details> <details><summary><code>2f69945f</code> test: the prepare sequence's stand-ins have main's compile_all (Compile shaders on All) (Joey Magnus)</summary> </details> ## Measurements **Verdict:** nothing to measure: a correctness fix off the frame path, covered by game-free checks. ### Flags and settings None. ## Testing **Tested:** On Linux (kernel 7.0), RTX 4070 Ti, `main` @ 71b1edd8, this PR @ 2f69945f: `the checks that read the changed files (`tests/run_all.py --changed origin/main`)` pass. The full suite (`tests/run_all.py`) has not been run on this branch yet; a comment will follow when it has. **Not tested:** An update with the preparing screen; a cut list in the game. Opened as work in progress until it runs in the game. **Must be tested:** - None: the checks run both sequences and a cut file ## Risks **What could break:** the preparing screen's order, and the pipeline lists' files. **Turning it off:** revert the commit. **Compatibility:** A list with a broken end is cut back once, logged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). ### When it applies - [x] A game-free test added for the fix, failing on `main` and passing here.
With `RX_VK_PREPARE_BACKGROUND=1`, an update builds what the notes name
before the game and the rest behind it. A start with console progress
(`started_up` in `gpu/vk/vk_main.cpp`) makes the shipped state list's
pipelines first: `if (update) state_list_run();` before
`precompile_run`. The preparing screen's worker (`prepare_with_screen`)
has no such call, so a start that shows the screen left those pipelines
out, and they were built when first drawn, in play.

## Change

The screen's worker calls `state_list_run()` for an update at the same
point, before `precompile_run`, inside its `!prepare_stopped()` check. A
stopped preparation still skips it. Nothing is added to the screen:
`state_list_run` takes no progress callback.

## Testing

- New `tests/test_prepare_screen_sequence.py`: both step sequences are
cut from the source and run against stand-ins that record each call. For
an update, both make the state list once, before `precompile_run`, then
start the background pass. When it is not an update, neither makes the
list. A stopped screen worker makes neither the list nor the precompile.
On main the screen's update sequence has no state list; here it passes.
- Full suite on Linux: 474 passed, 53 skipped, 2 failed, both the same
on main on this machine (`test_bake_portability`: `zstandard` is not
installed; `nx2/test_texture_init_barrier.py`: `clang++` is not
installed).
- No game ran.

Restored-from: https://github.com/NotRexed/Hookshot/pull/2982
A crash or a full disk while the game writes
`hookshot_pipeline_states.bin` can leave it ending in part of a record.
`state_open_locked` in `gpu/vk/vk_draw.cpp` reads the whole records,
sets `keep`, and opens the file with `"ab"`, so every later run added
its new pipelines after the broken end. They were never read back: those
pipelines were built again in play at each start, and "Save my pipeline
list" sent a list whose end was garbage. `notes_open_locked` does the
same with the pipeline notes, also on its `known` path, where the file
is not read at all.

## Change

A small `cut_part_record(path, header, record)` cuts the file back to
its last whole record with `std::filesystem::resize_file` and logs it.
Both opens call it before appending: the state list when it keeps the
list, the notes when the stamp and record size match. The whole records
stay; a list that ends on a record's edge is left alone, and one shorter
than its header starts new as before.

## Testing

- New `tests/test_state_list_part_record.py`: `state_open_locked`,
`notes_open_locked` and the helper are cut from the source and run on
lists of their own: one whole record followed by 0 bytes, 1 byte, and a
record's size less one of the next. A new record is added as play adds
one, then the file is read back: both records there, the file a whole
number of records. The notes the same, with their records read before
the open and without. On main the part-record cases fail; here they
pass.
- `tests/test_pipeline_state_computes.py` compiles `state_open_locked`
too; its fixture now takes the helper along.
- Linux renderer built (`hookshot_vk.so`).
- Full suite on Linux: 474 passed, 53 skipped, 2 failed, both the same
on main on this machine (`test_bake_portability`: `zstandard` is not
installed; `nx2/test_texture_init_barrier.py`: `clang++` is not
installed).
- No game ran; not built on Windows.

Restored-from: https://github.com/NotRexed/Hookshot/pull/2983
test: the prepare sequence's stand-ins have main's compile_all (Compile shaders on All)
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 6s
Website / Website on Cloudflare (pull_request_target) Has been skipped
2f69945fde
Author
Contributor

Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @ 71b1edd8 (make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. The pipeline notes and state list loaded at start (32,886 noted pipelines found). The background-prepared update and a half-written last record were not reproduced in the game; the checks cover them.

Tested in game: Linux (Ubuntu 26.04.1, kernel 7.0.0-43), RTX 4070 Ti 12 GB, NVIDIA 595.99.02, Ryzen 7 9800X3D: this branch built together with the other 18 split from #218 and #284-#288, merged onto main @ 71b1edd8 (make_game.py --update, build v3.0.0-rc.5-82-g5952bb3d), then played by a scripted tour through the control bridge (scripts/memtour.py: a save loaded, Zora's Domain, the Market, Hyrule Field and Death Mountain Trail, 40 s each with walking and camera swings). All 4 stops reached, a clean quit, and no device loss or Vulkan error in the log. The pipeline notes and state list loaded at start (32,886 noted pipelines found). The background-prepared update and a half-written last record were not reproduced in the game; the checks cover them.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 6s
Website / Website on Cloudflare (pull_request_target) Has been skipped
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/pipeline-list-fixes:TrashPanda-fix/pipeline-list-fixes'
git switch 'TrashPanda-fix/pipeline-list-fixes'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1161
No description provided.