WIP: fix(vk): an update that changes no shader no longer rebuilds every pipeline #1418

Draft
TrashPanda wants to merge 2 commits from TrashPanda/Longshot:fix/no-pipeline-rebuild into main
Contributor

Type: fix · Risk: low · Tested on: Linux, game-free only · Needs: any GPU, a start after an update, in the game

Why

state_list_prepare deleted longshot_pipelines.bin.built, the marker that every noted pipeline is already in the pipeline cache, on every call. The preparation after every build calls it. So each update, even one that changed no shader, left the next start to build every noted pipeline again behind "Loading pipelines", after the preparation had already built them in its own process. Upstream that is 44,000 pipelines, minutes on a slower PC.

Ported from Hookshot 0824d761 (NotRexed): git.notrexed.dev/Hookshot/Hookshot@0824d76165

The tracker was searched for precompile_covers_notes and "Loading pipelines": no pull request or issue is this fix. #903, #900, #721 and #714 are other loading bugs.

How to reproduce

Seen on: Hookshot's report. Not yet run on Longshot.
Settings and switches: defaults.
Steps: start the game once so the pipeline cache is built; update to a build that changes no shader (the preparation runs); start again.
Expected: the second start loads the pipelines at create, with no full "Loading pipelines".
Actual: on main, the second start builds every noted pipeline again.

What changed

The marker goes only when the preparation notes a pipeline that was not noted before. Those new notes are added without their pipelines, and precompile_run builds them on the next start. A list that notes nothing new leaves the marker. The translator stamp still rebuilds everything when the shaders change (pcache_drop_stale).

  • renderer: state_list_prepare in gpu/vk/vk_draw.cpp counts what pipeline_note added and removes the marker only then, with a log line vk: state list: N pipelines not noted before: ....
  • test: tests/test_load_default.py checks that the marker's removal in state_list_prepare follows if (added) { and that the count comes from pipeline_note.

No switch: an extra rebuild was the only thing the old removal bought, and RX_VK_NO_STATE_LIST=1 still turns the preparation off.

Measurements

Verdict: the game-free check fails before the fix and passes here. Nothing in the game has been measured.

Hardware: Linux, no GPU run.
Scene: none.
Method: python -I -B tests/test_load_default.py on the test commit and on this head.
How to reproduce: check out db4c16d4 (the test alone, on main 544229a2) and run the command; then this head.

Before: main @ 544229a2 plus the test @ db4c16d4 After: this PR @ 9a234bef
tests/test_load_default.py fails: state_list_prepare removes the pipelines-built marker though it noted nothing new load_default: ok

python -I -B tests/run_all.py on this head (Linux, 16 at once): 602 passed, 38 failed, 81 skipped (no Vulkan or SPIR-V headers here). Each of the 38 also fails on main 544229a2 run alone, except tests/test_appimage_package_retry.py, which timed out under the full run's load (90 s) and passes alone on this head, twice.

Flags and settings

None: no switch is added or changed.

Testing

Tested: Linux, game-free. tests/test_load_default.py fails on the test commit and passes on 9a234bef.

Not tested: the game, any GPU, Windows. This PC is not running the game. Opened with --force because Joey asked.

Must be tested:

  • Any GPU, Windows or Linux: start once and let the pipelines build; run the preparation again with no shader change (a rebuild of the same commit through the launcher); start again. Expected: no full "Loading pipelines" on the second start, and no vk: state list: ... not noted before line. Then add a state the list did not have (or delete longshot_pipelines.bin): the line appears and the next start builds the new ones.

Risks

What could break: if pipeline_note reported a note as old when its pipeline is not in the cache, that pipeline would not be prebuilt and would compile in play, a hitch the first time it draws. The marker is also still removed when the translator stamp changes and when the pipeline cache is dropped.

Turning it off: none as a switch; reverting the commit restores the old removal.

Compatibility: nothing in saves or options. MKWVK_API_VERSION is unchanged.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). The 38 failures also fail on main here (this PC lacks the Vulkan headers and some tools); the one that did not, test_appimage_package_retry, is a timing failure under load and passes alone.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. The renderer was not built.
**Type:** fix · **Risk:** low · **Tested on:** Linux, game-free only · **Needs:** any GPU, a start after an update, in the game ## Why `state_list_prepare` deleted `longshot_pipelines.bin.built`, the marker that every noted pipeline is already in the pipeline cache, on every call. The preparation after every build calls it. So each update, even one that changed no shader, left the next start to build every noted pipeline again behind "Loading pipelines", after the preparation had already built them in its own process. Upstream that is 44,000 pipelines, minutes on a slower PC. Ported from Hookshot `0824d761` (NotRexed): https://git.notrexed.dev/Hookshot/Hookshot/commit/0824d76165fe3144ac7201f1d1d4d91647c7aadc The tracker was searched for `precompile_covers_notes` and "Loading pipelines": no pull request or issue is this fix. #903, #900, #721 and #714 are other loading bugs. ## How to reproduce **Seen on:** Hookshot's report. Not yet run on Longshot. **Settings and switches:** defaults. **Steps:** start the game once so the pipeline cache is built; update to a build that changes no shader (the preparation runs); start again. **Expected:** the second start loads the pipelines at create, with no full "Loading pipelines". **Actual:** on main, the second start builds every noted pipeline again. ## What changed The marker goes only when the preparation notes a pipeline that was not noted before. Those new notes are added without their pipelines, and `precompile_run` builds them on the next start. A list that notes nothing new leaves the marker. The translator stamp still rebuilds everything when the shaders change (`pcache_drop_stale`). - renderer: `state_list_prepare` in `gpu/vk/vk_draw.cpp` counts what `pipeline_note` added and removes the marker only then, with a log line `vk: state list: N pipelines not noted before: ...`. - test: `tests/test_load_default.py` checks that the marker's removal in `state_list_prepare` follows `if (added) {` and that the count comes from `pipeline_note`. No switch: an extra rebuild was the only thing the old removal bought, and `RX_VK_NO_STATE_LIST=1` still turns the preparation off. ## Measurements **Verdict:** the game-free check fails before the fix and passes here. Nothing in the game has been measured. Hardware: Linux, no GPU run. Scene: none. Method: `python -I -B tests/test_load_default.py` on the test commit and on this head. How to reproduce: check out `db4c16d4` (the test alone, on main `544229a2`) and run the command; then this head. | | Before: main @ 544229a2 plus the test @ db4c16d4 | After: this PR @ 9a234bef | |---|---|---| | `tests/test_load_default.py` | fails: state_list_prepare removes the pipelines-built marker though it noted nothing new | `load_default: ok` | `python -I -B tests/run_all.py` on this head (Linux, 16 at once): 602 passed, 38 failed, 81 skipped (no Vulkan or SPIR-V headers here). Each of the 38 also fails on `main` `544229a2` run alone, except `tests/test_appimage_package_retry.py`, which timed out under the full run's load (90 s) and passes alone on this head, twice. ### Flags and settings None: no switch is added or changed. ## Testing **Tested:** Linux, game-free. `tests/test_load_default.py` fails on the test commit and passes on `9a234bef`. **Not tested:** the game, any GPU, Windows. This PC is not running the game. Opened with `--force` because Joey asked. **Must be tested:** - Any GPU, Windows or Linux: start once and let the pipelines build; run the preparation again with no shader change (a rebuild of the same commit through the launcher); start again. Expected: no full "Loading pipelines" on the second start, and no `vk: state list: ... not noted before` line. Then add a state the list did not have (or delete `longshot_pipelines.bin`): the line appears and the next start builds the new ones. ## Risks **What could break:** if `pipeline_note` reported a note as old when its pipeline is not in the cache, that pipeline would not be prebuilt and would compile in play, a hitch the first time it draws. The marker is also still removed when the translator stamp changes and when the pipeline cache is dropped. **Turning it off:** none as a switch; reverting the commit restores the old removal. **Compatibility:** nothing in saves or options. `MKWVK_API_VERSION` is unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). The 38 failures also fail on `main` here (this PC lacks the Vulkan headers and some tools); the one that did not, `test_appimage_package_retry`, is a timing failure under load and passes alone. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. The renderer was not built.
state_list_prepare deletes the marker that every noted pipeline is already
in the pipeline cache, on every call. An update that changes no shader then
makes the next start build them all again. This check fails until the marker
goes only when a note is new.

Ported-from: 0824d76165
Co-authored-by: NotRexed <[email protected]>
Assisted-by: Grok 4.7
fix(vk): an update that changes no shader no longer rebuilds every pipeline
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
9a234bef10
state_list_prepare deleted the marker that every noted pipeline is already
in the pipeline cache, and the preparation after every build calls it. An
update that left the shaders alone then made the next start build them all
again. The marker now goes only when a note is new. The translator stamp
still rebuilds everything when the shaders change.

Ported-from: 0824d76165
Co-authored-by: NotRexed <[email protected]>
Assisted-by: Grok 4.7
Author
Contributor

@Beanstalk any GPU, two game starts in a row on this PR's build, from a new save, with the pipeline cache kept between them. Expected: the first start may build pipelines ("Loading pipelines"); the second start loads them at create with no full "Loading pipelines" and no log line vk: state list: ... pipelines not noted before. Compare the second start's time to the menu with main's second start. Not run in the game yet; Joey asked for this pull request so the farm can take that run.

@Beanstalk any GPU, two game starts in a row on this PR's build, from a new save, with the pipeline cache kept between them. Expected: the first start may build pipelines ("Loading pipelines"); the second start loads them at create with no full "Loading pipelines" and no log line `vk: state list: ... pipelines not noted before`. Compare the second start's time to the menu with main's second start. Not run in the game yet; Joey asked for this pull request so the farm can take that run.
Member

@TrashPanda only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@TrashPanda only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Collaborator

Possibly related, for context:

  • PR #737 feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing — Overlapping shader-cache changes also govern the precompile marker
  • PR #1161 WIP: fix(vk): the shipped state list is made behind the preparing screen, and a half-written pipeline list is cut back — Touches state-list preparation and pipeline precompilation sequence
Possibly related, for context: - PR #737 feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing — Overlapping shader-cache changes also govern the precompile marker - PR #1161 WIP: fix(vk): the shipped state list is made behind the preparing screen, and a half-written pipeline list is cut back — Touches state-list preparation and pipeline precompilation sequence
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 4s
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval waiting for a BigBean's 👍 on the request
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'fix/no-pipeline-rebuild:TrashPanda-fix/no-pipeline-rebuild'
git switch 'TrashPanda-fix/no-pipeline-rebuild'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1418
No description provided.