feat(prepare): Compile shaders All / Most / None, Most skipping the predicted pipelines #158

Merged

HumanBean merged 3 commits from HumanBean/Longshot:feat/compile-shaders into main 2026-11-13 08:14:25 +00:00

Conversation 3 Commits 3 Files changed 22 +388 -34

HumanBean commented 2026-11-13 07:48:30 +00:00

Owner

• Copy link
• Copy Markdown

Why

The shader preparation before a first start (after a build, an update or a new driver) takes about 7 minutes on an RTX 3090 with a 12-core CPU, and longer on smaller PCs. A third of it builds predicted pipelines for program variants no list names, and play almost never asks for them.

What changed

Work in progress. Planned, in this order:

• vk: the log counts the predicted pipelines play meets (committed: gpu/vk/vk_draw.cpp), so a tour can tell how often a prediction is used.
• Compile shaders: All (today: every listed and every predicted pipeline), Most (recommended, the default: everything except the predicted pipelines), None (the Builder skips the preparation; the game's first launch does Most's work before the title). A tooltip on the choice says what each costs.
• A flag that sets whether a preparation at launch (after None, a build, an update or a new driver) does Most or All; default Most. A Builder choice of All carries over unless the flag says otherwise.
• Nothing is built behind the game in play, unlike the earlier draft this takes the choice from.
• A preparation inside the game's own process currently leaves about 15,000 programs out of the index the game's create calls read, which shows as pop-in. None depends on that path, so this PR either fixes it or says so here.

Flags and settings

Flag	What it does	Default now	Recommendation	Helps on	Doesn't help on	Untested on
Compile shaders (launcher, Builder, make_game)	All / Most / None, as above	(to fill)	Most	every PC: 30% shorter preparation		AMD, Intel, Windows
Launch preparation flag (name to fill)	Most or All for a preparation at launch	Most	Most

Measurements

main: d40506c5 (2026-11-13)
this PR: feat/compile-shaders @ (to fill)
Hardware: RTX 3090, NVIDIA driver on Linux, 12-core / 24-thread CPU, 30 GB RAM
Method: full cold preparation on a fresh folder with an empty driver cache, the log's phase lines; tours to follow.

Preparation so far (one PC, cold, Builder-style):

	with the predicted pipelines (All, today)	without them (Most)
whole preparation	415 s, 413 s	292 s
predicted pipelines (50,547)	about 136 s	none

All takes about 1.4 times as long as Most.

Also measured and dropped: one pipeline cache per build thread on all 24 CPUs builds the 25,337 listed pipelines in 136 s against 137-140 s on today's 16 threads with one shared cache. The compile is bound by the 12 physical cores.

Still to measure: a tour of 10 minutes or more over several areas, on a folder prepared with the predicted pipelines and on one prepared without them, comparing pipelines compiled in play, waits, draws left out, frames over 33 ms, and how many predicted pipelines play met. This decides the default.

Testing

Tested: the preparation timings above.

Not tested: everything else so far (work in progress).

Needs other testers:

• AMD (RADV) and Windows: preparation time on Most and All, and a short play session on Most with no new compiles in play (request to open)

Screenshots

To come: the choice and its tooltip in the launcher and the Builder.

Risk and rollback

On Most, a draw that would have used a predicted pipeline compiles it in play: a hitch. The tours measure how often that happens; All brings back today's behaviour.

Issues

None

Checklist

For everyone

• One topic.
• The title is a Conventional Commit (type(scope): description).
• Rebased on current main.
• python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
• Tested in the game, or what wasn't is named under Testing.
• Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
• Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
• No game files, dumps, build output or generated files.
• third_party/ untouched (a new folder with a PROVENANCE.md instead).
• The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

• AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
• Flags noted, and the switch reference updated (docs/development.md, the README).
• Screenshots for a change that shows on screen.
• Docs updated.

## Why The shader preparation before a first start (after a build, an update or a new driver) takes about 7 minutes on an RTX 3090 with a 12-core CPU, and longer on smaller PCs. A third of it builds predicted pipelines for program variants no list names, and play almost never asks for them. ## What changed Work in progress. Planned, in this order: - vk: the log counts the predicted pipelines play meets (committed: `gpu/vk/vk_draw.cpp`), so a tour can tell how often a prediction is used. - Compile shaders: **All** (today: every listed and every predicted pipeline), **Most** (recommended, the default: everything except the predicted pipelines), **None** (the Builder skips the preparation; the game's first launch does Most's work before the title). A tooltip on the choice says what each costs. - A flag that sets whether a preparation at launch (after None, a build, an update or a new driver) does Most or All; default Most. A Builder choice of All carries over unless the flag says otherwise. - Nothing is built behind the game in play, unlike the earlier draft this takes the choice from. - A preparation inside the game's own process currently leaves about 15,000 programs out of the index the game's create calls read, which shows as pop-in. None depends on that path, so this PR either fixes it or says so here. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | Compile shaders (launcher, Builder, make_game) | All / Most / None, as above | (to fill) | Most | every PC: 30% shorter preparation | | AMD, Intel, Windows | | Launch preparation flag (name to fill) | Most or All for a preparation at launch | Most | Most | | | | ## Measurements main: d40506c5 (2026-11-13) this PR: feat/compile-shaders @ (to fill) Hardware: RTX 3090, NVIDIA driver on Linux, 12-core / 24-thread CPU, 30 GB RAM Method: full cold preparation on a fresh folder with an empty driver cache, the log's phase lines; tours to follow. Preparation so far (one PC, cold, Builder-style): | | with the predicted pipelines (All, today) | without them (Most) | |---|---|---| | whole preparation | 415 s, 413 s | 292 s | | predicted pipelines (50,547) | about 136 s | none | All takes about 1.4 times as long as Most. Also measured and dropped: one pipeline cache per build thread on all 24 CPUs builds the 25,337 listed pipelines in 136 s against 137-140 s on today's 16 threads with one shared cache. The compile is bound by the 12 physical cores. Still to measure: a tour of 10 minutes or more over several areas, on a folder prepared with the predicted pipelines and on one prepared without them, comparing pipelines compiled in play, waits, draws left out, frames over 33 ms, and how many predicted pipelines play met. This decides the default. ## Testing **Tested:** the preparation timings above. **Not tested:** everything else so far (work in progress). **Needs other testers:** - AMD (RADV) and Windows: preparation time on Most and All, and a short play session on Most with no new compiles in play (request to open) ## Screenshots To come: the choice and its tooltip in the launcher and the Builder. ## Risk and rollback On Most, a draw that would have used a predicted pipeline compiles it in play: a hitch. The tours measure how often that happens; All brings back today's behaviour. ## Issues None ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [ ] Tested in the game, or what wasn't is named under Testing. - [ ] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] Flags noted, and the switch reference updated (docs/development.md, the README). - [ ] Screenshots for a change that shows on screen. - [ ] Docs updated.

HumanBean added 1 commit 2026-11-13 07:48:30 +00:00

feat(vk): the log counts the predicted pipelines play meets ... 463d65c05e

Co-Authored-By: Claude Fable 5.2 <[email protected]>

HumanBean added the

testing wanted

label 2026-11-13 08:00:54 +00:00

HumanBean commented 2026-11-13 08:06:10 +00:00

Author

Owner

• Copy link
• Copy Markdown

Early result from the first tour, on a folder prepared with Most (no predicted pipelines). RTX 3090, Linux, first start after the preparation; about 8 minutes of play walking Kokiri Forest, Kakariko and the Market. The log counts against the 50,547 pipelines an All preparation predicts.

	Most folder
predicted pipelines play met	0 of 50,547
pipelines compiled in play	0 (the play files hold 80, all answered by the driver's cache)
draws or dispatches left out	0
waits for a pipeline	0
new graphics pipelines in the census	0

Limits: the Hyrule Field walk was cut short (the PC ran low on memory and the run was stopped), and the PC swapped heavily during the run, so its frame times are not valid and are not reported. The counts above don't depend on that.

Next: the same tour on a folder prepared with All to compare, a full Field walk, and then the choice itself (coded, not built or tested yet).

Early result from the first tour, on a folder prepared with **Most** (no predicted pipelines). RTX 3090, Linux, first start after the preparation; about 8 minutes of play walking Kokiri Forest, Kakariko and the Market. The log counts against the 50,547 pipelines an All preparation predicts. | | Most folder | |---|---| | predicted pipelines play met | 0 of 50,547 | | pipelines compiled in play | 0 (the play files hold 80, all answered by the driver's cache) | | draws or dispatches left out | 0 | | waits for a pipeline | 0 | | new graphics pipelines in the census | 0 | Limits: the Hyrule Field walk was cut short (the PC ran low on memory and the run was stopped), and the PC swapped heavily during the run, so its frame times are not valid and are not reported. The counts above don't depend on that. Next: the same tour on a folder prepared with **All** to compare, a full Field walk, and then the choice itself (coded, not built or tested yet).

HumanBean changed title from WIP: feat(prepare): Compile shaders All / Most / None, Most skipping the predicted pipelines to WIP: feat(prepare): Compile shaders All / Most, Most (the default) skipping the predicted pipelines 2026-11-13 08:13:49 +00:00

HumanBean commented 2026-11-13 08:13:50 +00:00

Author

Owner

• Copy link
• Copy Markdown

Pushed b9c17d3a (renderer: a preparation builds the predicted pipelines only on All; --compile-shaders all|most and RX_VK_COMPILE_SHADERS, default Most) and 4da99bc8 (an All / Most choice beside "Prepare shaders now" in the Builders, make_game.py --compile-shaders kept in the options file, the launcher setting, README and docs). None and the fix for pop-in after a preparation in the game's own process are split out to a follow-up PR.

Not yet checked on these commits: the build, tests/run_all.py, a cold default preparation (predictions skipped), an All preparation through make_game, the flag and options-file checks. Running now; results will follow here. The WPF Builder could not be compiled here (no .NET on this PC).

Second tour, on a folder prepared with All (today's behaviour), RTX 3090: predicted pipelines play met 0 of 50,547, compiled in play 0 (71 from the driver's cache), draws left out 0, waits 0. The driver kept 32 of 32 sampled predictions, so they were present and never asked for. Frame times void (the PC swapped).

Pushed b9c17d3a (renderer: a preparation builds the predicted pipelines only on All; --compile-shaders all|most and RX_VK_COMPILE_SHADERS, default Most) and 4da99bc8 (an All / Most choice beside "Prepare shaders now" in the Builders, make_game.py --compile-shaders kept in the options file, the launcher setting, README and docs). None and the fix for pop-in after a preparation in the game's own process are split out to a follow-up PR. Not yet checked on these commits: the build, tests/run_all.py, a cold default preparation (predictions skipped), an All preparation through make_game, the flag and options-file checks. Running now; results will follow here. The WPF Builder could not be compiled here (no .NET on this PC). Second tour, on a folder prepared with All (today's behaviour), RTX 3090: predicted pipelines play met 0 of 50,547, compiled in play 0 (71 from the driver's cache), draws left out 0, waits 0. The driver kept 32 of 32 sampled predictions, so they were present and never asked for. Frame times void (the PC swapped).

HumanBean added 2 commits 2026-11-13 08:13:50 +00:00

feat(vk): Compile shaders: a preparation builds the predicted pipelines only on All ... b9c17d3a36

A preparation (the Builder's --prepare-shaders run, or the game's own at a launch after an update or a new driver) built
about 50,000 predicted pipelines for program variants no list names: over 40 % more time (415 and 413 s against 292 s for
the whole preparation, RTX 3090, 24 threads, cold driver cache), for pipelines play almost never draws. Now it builds them
only when Compile shaders is All: compile_shaders=1 in longshot_options.ini, or RX_VK_COMPILE_SHADERS=all, which the game's
--compile-shaders all sets for one run. Most (0), the default, leaves them out; the log's "vk: Compile shaders:" line says
which and why. On Most the marker, sample and keys an earlier preparation on All left are removed.

A longshot_options.ini with only compile_shaders= in it (make_game.py writes it into a new folder) still makes the game's
first start a first start, so Auto (this PC) chooses as it does with no file.

Co-Authored-By: Claude Fable 5.2 <[email protected]>

feat(builder): Compile shaders (All / Most) in the Builders, make_game.py and the launcher's settings ... 4da99bc891

What the shaders' preparation builds, beside the "Prepare shaders now" tick in the three Builders, in the launcher's
settings (compile_shaders in longshot_options.ini) and as make_game.py --compile-shaders:
- All: the predicted pipelines too, about 40 % longer, for pipelines play almost never uses;
- Most (recommended, the default): every shader and the pipelines the game is known to use.

make_game.py keeps a choice given in the folder's longshot_options.ini, the line the launcher's setting writes, and reads it
when none is given, so a launcher's build and an update follow it, and so do the game's own preparations at a launch; the
preparation it runs is told the level. A Builder's build passes the choice shown; an update passes none, so the folder's
stands.

Co-Authored-By: Claude Fable 5.2 <[email protected]>

HumanBean changed title from WIP: feat(prepare): Compile shaders All / Most, Most (the default) skipping the predicted pipelines to feat(prepare): Compile shaders All / Most / None, Most skipping the predicted pipelines 2026-11-13 08:14:20 +00:00

HumanBean merged commit 1d6b832a13 into main 2026-11-13 08:14:25 +00:00

HumanBean deleted branch feat/compile-shaders 2026-11-13 08:14:26 +00:00

HumanBean referenced this pull request from a commit 2026-11-13 08:14:27 +00:00

feat(prepare): Compile shaders All / Most / None, Most skipping the predicted pipelines (#158)

HumanBean referenced this pull request 2026-11-13 08:23:19 +00:00

fix(vk): on Most the Builder's progress bar leaves the prediction's part out; its check runs All and Most #164

HumanBean referenced this pull request from a commit 2026-11-13 08:23:57 +00:00

fix(vk): on Most the Builder's progress bar leaves the prediction's part out; its check runs All and Most (#164)

gab9281 referenced this pull request 2026-11-13 08:28:13 +00:00

feat: a crash window that shows the crash report and sends it only on Send Help #141

HumanBean referenced this pull request 2026-11-13 08:35:11 +00:00

fix(builder): MainWindow.xaml is valid XML again, so the Windows builder compiles #167

HumanBean referenced this pull request from a commit 2026-11-13 08:35:30 +00:00

fix(builder): MainWindow.xaml is valid XML again, so the Windows builder compiles (#167)

HumanBean commented 2026-11-13 08:44:37 +00:00

Author

Owner

• Copy link
• Copy Markdown

Checks after the merge, on binaries built from 4da99bc8 (main 1d6b832a has no renderer or host change over it). RTX 3090, Linux, each folder a fresh copy:

• Cold Builder-style preparation (longshot --prepare-shaders), no choice set: "vk: Compile shaders: Most (the default): the predicted pipelines are left out"; no prediction ran (38,809 programs, 25,337 noted pipelines).
• make_game.py --update <folder> --compile-shaders all: wrote compile_shaders=1 into longshot_options.ini and ran the preparation with --compile-shaders all: "Compile shaders: All (RX_VK_COMPILE_SHADERS)", 50,547 predicted pipelines built.
• A launch-time preparation in that folder with no argument reads the file: "All (compile_shaders in longshot_options.ini)", 50,547 built.
• --compile-shaders most over that file: "Most (RX_VK_COMPILE_SHADERS)", no prediction, and the earlier preparation's prediction marker and key files are removed.

Wall times of these runs are not reported: the PC swapped heavily. The test that #158 left failing and a progress-bar jump on Most are fixed in #164 (merged).

Not checked: the WPF Builder (no .NET here), Windows, AMD.

Checks after the merge, on binaries built from 4da99bc8 (main 1d6b832a has no renderer or host change over it). RTX 3090, Linux, each folder a fresh copy: - Cold Builder-style preparation (`longshot --prepare-shaders`), no choice set: "vk: Compile shaders: Most (the default): the predicted pipelines are left out"; no prediction ran (38,809 programs, 25,337 noted pipelines). - `make_game.py --update <folder> --compile-shaders all`: wrote compile_shaders=1 into longshot_options.ini and ran the preparation with `--compile-shaders all`: "Compile shaders: All (RX_VK_COMPILE_SHADERS)", 50,547 predicted pipelines built. - A launch-time preparation in that folder with no argument reads the file: "All (compile_shaders in longshot_options.ini)", 50,547 built. - `--compile-shaders most` over that file: "Most (RX_VK_COMPILE_SHADERS)", no prediction, and the earlier preparation's prediction marker and key files are removed. Wall times of these runs are not reported: the PC swapped heavily. The test that #158 left failing and a progress-bar jump on Most are fixed in #164 (merged). Not checked: the WPF Builder (no .NET here), Windows, AMD.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/accessibility

Barriers for people with disabilities

  

area/audio

Sound and music

  

area/input

Controllers, keyboard, mouse

  

area/install

Installing, updating, building from your game

  

area/menu

In-game menu and settings

  

area/renderer

Vulkan renderer, shaders, effects

  

area/saves

Save files

  

area/upscaling

DLSS, FSR, frame generation

  

good first issue

A good start for a new contributor

  

gpu/amd

AMD GPUs

  

gpu/intel

Intel GPUs

  

gpu/nvidia

NVIDIA GPUs

  

help wanted

Extra hands or knowledge wanted

  

kind

bug

Something doesn't work as it should

  

kind

chore

Build, CI, dependencies, tidying

  

kind

docs

Documentation

  

kind

feature

A new feature or an improvement

  

kind

performance

Frame rate, frame time, loading, memory

  

kind

question

A question rather than a change

  

needs measurements

Waiting for before/after measurements

  

platform/linux

Linux only

  

platform/steam-deck

Steam Deck

  

platform/windows

Windows only

  

preview

Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  

priority

critical

Crashes, data loss, or blocks most players

  

priority

high

Important: next in line

  

priority

low

Nice to have

  

status

blocked

Waits on something else first

  

status

confirmed

Reproduced or agreed on

  

status

duplicate

Already reported or proposed

  

status

invalid

Not a problem in this project

  

status

needs-info

Waiting for more information from the reporter

  

status

wontfix

Will not be worked on

  

testing wanted

Wanted: a test on other hardware (posts a test request in #test-requests)

No labels

area/accessibility

area/audio

area/input

area/install

area/menu

area/renderer

area/saves

area/upscaling

good first issue

gpu/amd

gpu/intel

gpu/nvidia

help wanted

kind

bug

kind

chore

kind

docs

kind

feature

kind

performance

kind

question

needs measurements

platform/linux

platform/steam-deck

platform/windows

preview

priority

critical

priority

high

priority

low

status

blocked

status

confirmed

status

duplicate

status

invalid

status

needs-info

status

wontfix

testing wanted

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

MagicBeanDev/Longshot!158

No description provided.