WIP: fix(vk): free a stalled shader preparation's pipeline notes #517

Draft

Amateurwizrd wants to merge 2 commits from Amateurwizrd/Longshot:fix/prepare-stall-notes into main

pull from: Amateurwizrd/Longshot:fix/prepare-stall-notes

merge into: MagicBeanDev:main

MagicBeanDev:main

Conversation 4 Commits 2 Files changed 3 +464 -28

Amateurwizrd commented 2026-11-14 17:36:24 +00:00

Contributor

• Copy link
• Copy Markdown

Type: fix · Risk: low · Tested on: Windows 11, RTX 5070 Ti 16 GB (game-free) · Needs: Linux

Why

When the driver stalls a shader preparation (no progress for 30 s), the preparation gives up on its workers and never
frees the run's pipeline notes, each holding a whole NvnState: gigabytes of host memory stay allocated for the rest of
the session after a stalled prediction. Searched the tracker: no issue covers it.

How to reproduce

Seen on: the game-free check below, Windows 11, RTX 5070 Ti, driver 621.01; in the game it needs a driver that stalls a preparation, which could not be produced on demand here
Settings and switches: defaults
Steps: run tests/test_prepare_abandon.py (this PR's check) against main's gpu/vk/vk_draw.cpp and gpu/vk/vk_progcache.cpp: its stand-in driver never returns from one worker's pipeline creation, and its callback throws in a borrow
Expected: after the give-up every note no worker holds is freed, and a throw in program_cache_borrow's callback leaves no module or layout alive
Actual: the notes stay allocated, and the borrowed modules and layouts are left alive

What changed

The preparation's workers now share its state, so whichever finishes last frees it, and the thread that gives up frees
at once every note no stalled worker still holds. A borrow from the program cache keeps what it loaded in a guard, so a
throw leaves nothing behind.

• vk: precompile_run and predict_run in gpu/vk/vk_draw.cpp share their state with the workers, so the last worker
  to return frees it; the thread that gives up frees at once, under the state's lock, every note no worker holds (a
  worker the driver never returns from keeps only its own note). The stall's log line says how much host memory the
  notes held and how much was freed.
• vk: program_cache_borrow in gpu/vk/vk_progcache.cpp keeps each variant's modules and layouts in a guard once the
  loader has handed it over, so a throw from the callback or a later variant's load leaves none of them behind.
• tests: tests/test_prepare_abandon.py (new) runs the real worker and borrow code with a stalled driver stand-in.

The idea of sharing the state with the workers was first seen in Hookshot-old's 9553c039; freeing the notes at the
stall and the borrow guard are new, and the code is written for Longshot.

Measurements

Verdict: it passes: the new check fails on main @ 13e8b6da in both runs and passes on this PR @ dcd64421 in both.

Hardware: Windows 11 Pro 26200, RTX 5070 Ti 16 GB, driver 621.01, Ryzen 7 7800X3D, 32 GB
Scene: none (game-free check)
Method: the new test, run 2 times on each side, interleaved (main, this PR, main, this PR); main gets this PR's test file only
How to reproduce: python -I -B tests/run_all.py prepare_abandon on this branch, and on main with tests/test_prepare_abandon.py copied in

	Before: main @ 13e8b6da	After: this PR @ dcd64421
tests/test_prepare_abandon.py (run 1 / run 2)	FAIL / FAIL	pass / pass

Flags and settings

None

Testing

Tested: on Windows 11 with the pinned llvm-mingw, the 124 checks that read this diff, one at a time, on this PR @ dcd64421 (main @ 13e8b6da): 121 pass; the 3 that fail (test_linux_driver_fault_guard, test_pipeline_tour, test_vk_occlusion_queries) also fail on main @ 13e8b6da; none fails here that passes on main. The renderer was not built at this head.

Not tested: a stalled preparation in the game: no driver stall could be produced on demand here; the new test
drives the same worker code with a stand-in driver that never returns.

Must be tested:

• Linux (Test Farm: @Beanstalk linux, build the renderer): the renderer builds with this PR, expecting no error in gpu/vk/vk_draw.cpp or gpu/vk/vk_progcache.cpp

Risks

What could break: the preparation's teardown changes: a worker returning after the give-up now frees its own note,
so a mistake there would be a use-after-free in the preparation workers (the test covers a worker returning late and
one never returning).

Turning it off: revert the commit.

Compatibility: Nothing changes.

Checklist

Everyone

• One topic.
• The title is a Conventional Commit (type(scope): description) of the status line's type.
• Why gives the reason plainly and links the issues that fit (the tracker searched).
• Measurements has a verdict, or one line saying why there is nothing to measure.
• Testing names what was tested, what wasn't, and what must still be tested, where and how.
• Rebased on current main.
• python -I -B tests/run_all.py passes (any failure that also fails on main is named).
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• A game-free test added for the fix, failing on main and passing here.
• Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows here; Linux is in Must be tested)

**Type:** fix · **Risk:** low · **Tested on:** Windows 11, RTX 5070 Ti 16 GB (game-free) · **Needs:** Linux ## Why When the driver stalls a shader preparation (no progress for 30 s), the preparation gives up on its workers and never frees the run's pipeline notes, each holding a whole `NvnState`: gigabytes of host memory stay allocated for the rest of the session after a stalled prediction. Searched the tracker: no issue covers it. ## How to reproduce **Seen on:** the game-free check below, Windows 11, RTX 5070 Ti, driver 621.01; in the game it needs a driver that stalls a preparation, which could not be produced on demand here **Settings and switches:** defaults **Steps:** run `tests/test_prepare_abandon.py` (this PR's check) against main's `gpu/vk/vk_draw.cpp` and `gpu/vk/vk_progcache.cpp`: its stand-in driver never returns from one worker's pipeline creation, and its callback throws in a borrow **Expected:** after the give-up every note no worker holds is freed, and a throw in `program_cache_borrow`'s callback leaves no module or layout alive **Actual:** the notes stay allocated, and the borrowed modules and layouts are left alive ## What changed The preparation's workers now share its state, so whichever finishes last frees it, and the thread that gives up frees at once every note no stalled worker still holds. A borrow from the program cache keeps what it loaded in a guard, so a throw leaves nothing behind. - vk: `precompile_run` and `predict_run` in `gpu/vk/vk_draw.cpp` share their state with the workers, so the last worker to return frees it; the thread that gives up frees at once, under the state's lock, every note no worker holds (a worker the driver never returns from keeps only its own note). The stall's log line says how much host memory the notes held and how much was freed. - vk: `program_cache_borrow` in `gpu/vk/vk_progcache.cpp` keeps each variant's modules and layouts in a guard once the loader has handed it over, so a throw from the callback or a later variant's load leaves none of them behind. - tests: `tests/test_prepare_abandon.py` (new) runs the real worker and borrow code with a stalled driver stand-in. The idea of sharing the state with the workers was first seen in Hookshot-old's 9553c039; freeing the notes at the stall and the borrow guard are new, and the code is written for Longshot. ## Measurements **Verdict:** it passes: the new check fails on main @ 13e8b6da in both runs and passes on this PR @ dcd64421 in both. Hardware: Windows 11 Pro 26200, RTX 5070 Ti 16 GB, driver 621.01, Ryzen 7 7800X3D, 32 GB Scene: none (game-free check) Method: the new test, run 2 times on each side, interleaved (main, this PR, main, this PR); main gets this PR's test file only How to reproduce: `python -I -B tests/run_all.py prepare_abandon` on this branch, and on main with `tests/test_prepare_abandon.py` copied in | | Before: main @ 13e8b6da | After: this PR @ dcd64421 | |---|---|---| | `tests/test_prepare_abandon.py` (run 1 / run 2) | FAIL / FAIL | pass / pass | ### Flags and settings None ## Testing **Tested:** on Windows 11 with the pinned llvm-mingw, the 124 checks that read this diff, one at a time, on this PR @ dcd64421 (main @ 13e8b6da): 121 pass; the 3 that fail (`test_linux_driver_fault_guard`, `test_pipeline_tour`, `test_vk_occlusion_queries`) also fail on main @ 13e8b6da; none fails here that passes on main. The renderer was not built at this head. **Not tested:** a stalled preparation in the game: no driver stall could be produced on demand here; the new test drives the same worker code with a stand-in driver that never returns. **Must be tested:** - Linux (Test Farm: @Beanstalk linux, build the renderer): the renderer builds with this PR, expecting no error in `gpu/vk/vk_draw.cpp` or `gpu/vk/vk_progcache.cpp` ## Risks **What could break:** the preparation's teardown changes: a worker returning after the give-up now frees its own note, so a mistake there would be a use-after-free in the preparation workers (the test covers a worker returning late and one never returning). **Turning it off:** revert the commit. **Compatibility:** Nothing changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A game-free test added for the fix, failing on `main` and passing here. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows here; Linux is in Must be tested)

Amateurwizrd added 1 commit 2026-11-14 17:36:25 +00:00

fix(vk): a preparation the driver stalled frees its pipeline notes, and a borrowed program's handles go if its callback throws ... 11da7d46fe

When a shader preparation made no progress for 30 s, `precompile_run` and
`predict_run` detached their workers and never freed the run's state: every
pipeline note of the run, each holding a whole `NvnState`, stayed allocated for
the session (gigabytes on a stalled prediction). Since 1f922f7c and 7759a6f3.

The state is now shared by the workers, so the last one to return frees it, and
the thread that gives up frees at once, under the state's lock, every note no
worker holds: a worker the driver never returns from keeps its own note, not
the run's. The stall line says how much host memory the notes held and how much
was freed. `program_cache_borrow` owns each variant's modules and layouts in a
guard once the loader has handed it over, so a throw from the callback or a
later variant's load leaves none of them behind.

Hookshot-old fixed the first half in 9553c039 (PR 2910) by sharing the state;
freeing the notes at the stall is new here. Written anew for Longshot.

Tested: tests/test_prepare_abandon.py (new: 121 notes alive after a stalled
prediction on main, 3 here), tests/test_prepare_stall.py and the other checks
that cut these functions, the renderer builds.
Not tested: a stalled preparation in the game.

Assisted-by: Claude

mooki added the

area/renderer

kind

bug

priority

high

gpu/nvidia

platform/linux

platform/windows

labels 2026-11-14 18:15:14 +00:00

HumanBean force-pushed fix/prepare-stall-notes from 11da7d46fe to 899490c5f4 2026-11-15 18:43:47 +00:00 Compare

HumanBean commented 2026-11-15 18:46:11 +00:00

Owner

• Copy link
• Copy Markdown

Rebased onto main @ bbe6fdf2. No tests were run here; the test boxes stay yours. No conflicts.

Rebased onto `main` @ bbe6fdf2. No tests were run here; the test boxes stay yours. No conflicts.

BeanIssues added the

risk

medium

label 2026-11-15 20:30:42 +00:00

Amateurwizrd force-pushed fix/prepare-stall-notes from 899490c5f4 to dcd644215f 2026-11-15 21:11:39 +00:00 Compare

Amateurwizrd commented 2026-11-15 21:12:36 +00:00

Author

Contributor

• Copy link
• Copy Markdown

Rebased onto main @ 13e8b6da (dcd64421). One new commit: main's quick compute pipelines (ae96bad3) made the preparation call precompile_compute_quick and mark Program::optimize_owed, so test_prepare_abandon's stubs gained both (the stub keeps the optimizer, as before). Re-measured on this main and the description moved to the current template: the new check fails on main twice and passes here twice; the 124 checks that read this diff: 121 pass, the 3 failing also fail on main.

Rebased onto main @ 13e8b6da (dcd64421). One new commit: main's quick compute pipelines (ae96bad3) made the preparation call `precompile_compute_quick` and mark `Program::optimize_owed`, so `test_prepare_abandon`'s stubs gained both (the stub keeps the optimizer, as before). Re-measured on this main and the description moved to the current template: the new check fails on main twice and passes here twice; the 124 checks that read this diff: 121 pass, the 3 failing also fail on main.

Amateurwizrd commented 2026-11-15 21:44:10 +00:00

Author

Contributor

• Copy link
• Copy Markdown

@Beanstalk linux, build the renderer at this head: it should build with no error in the files this PR changes; no change in play expected from the build alone.

@Beanstalk linux, build the renderer at this head: it should build with no error in the files this PR changes; no change in play expected from the build alone.

Beanstalk commented 2026-11-15 21:44:10 +00:00

Member

• Copy link
• Copy Markdown

@Amateurwizrd only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it.

@Amateurwizrd only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it.

Beanstalk added the

beanstalk: needs approval

label 2026-11-15 21:44:10 +00:00

Some checks are pending

Hide checks

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Website / Website on Cloudflare (pull_request) Blocked by required conditions

Details

PR description / Description follows the template (pull_request_target) Successful in 4s

Details

beanstalk/approval waiting for a BigBean's 👍 on the request

Details

Website / Website on Cloudflare (pull_request_target) Has been skipped

Details

This pull request is marked as a work in progress.

Some workflows are waiting to be reviewed.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'fix/prepare-stall-notes:Amateurwizrd-fix/prepare-stall-notes'

git switch 'Amateurwizrd-fix/prepare-stall-notes'

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/tooling

Developer tools: scripts, CI, build and release tooling

  

area/upscaling

DLSS, FSR, frame generation

  

beanstalk: approved

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: needs approval

Set by Beanstalk for show; the farm's database is what counts

  

beanstalk: running

Set by Beanstalk for show; the farm's database is what counts

  

ci: approved

A maintainer approved CircleCI for this fork PR (sticky; removed after 14 quiet days)

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/igpu

Integrated GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Affects Linux (a change for every system carries both Windows and Linux)

  

platform/macos

macOS (Apple silicon or Intel Macs)

  

platform/steam-deck

Steam Deck

  

platform/windows

Affects Windows (a change for every system carries both Windows and Linux)

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

regression

Worked in an earlier build, broken in a newer one: highest priority, above other bugs

  

risk

high

Broad or core change: the renderer, defaults on many systems, saves, installs, security

  

risk

low

Small, contained change: little can break

  

risk

medium

Changes behaviour in one area, or a default on some systems

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

  

ci: approved

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/tooling

area/upscaling

beanstalk: approved

beanstalk: needs approval

beanstalk: running

ci: approved

good first issue

gpu/amd

gpu/igpu

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/macos

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

regression

risk

high

risk

low

risk

medium

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

ci: approved

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!517

No description provided.