MagicBeanDev/Longshot

Watch

Fork 98

perf(pipelines): 22,987 more pipelines prepared before the title, from a tour of every map, actor and cutscene #563

Merged

HumanBean merged 3 commits from GummiNaomi/Longshot:contrib/gumminaomi-pipeline-tour into main

2026-11-14 23:22:41 +00:00

GummiNaomi commented

2026-11-14 19:38:25 +00:00

Contributor

Why

New areas, enemies and cutscenes still stutter the first time they appear, because their pipelines aren't in the shipped list and compile in play. This adds 22,987 pipelines recorded by a scripted tour of every map, actor type and cutscene, so a new install builds them before the title.

What changed

pipelines: assets/pipeline_states/contrib/gumminaomi-tour-every-map-actor-cutscene.bin.xz + .md. A contributed list added with nx2/pipeline_states.py add: 22,987 pipelines no other contributed list holds (19,533 graphics, 1,488 mesh, 1,966 compute programs). 21,848 of them are in no shipped list at all, the tour's included (+52% on the merged 42,072). Recorded cold (RX_VK_PRECOMPILE=0) on an RTX 4070 SUPER at this PC's default settings, over these runs:

Part of the route	Not in the shipped list	Compute programs among them
Every map at each of its 370 start positions (`StartPos`), child and adult, a camera turn each, the overworld at noon and at midnight	9,390	782
Every actor type that has a model (2,086 of `ActorInfo`), made 16 at a time beside Link	2,798	195
The 480 cutscenes, requested by name	9,476	986
The menus (items, system, equipment, collection, map, world map, threads of time, key items)	94	3
A second pass of the maps	90	0

The script that drove it isn't in this PR: #111 is the pipeline tour, and the host change it needed is offered there (comment on #111). This list doesn't touch tour.bin.xz, so a new tour replaces nothing of it.

nx2: COMPUTE_MOST 1024 → 2560 (nx2/pipeline_states.py, and the README's number). The merged list now names 2,199 compute programs (233 before), half of the game's 4,127, so check stopped at 1024. The measurement below is the "measure again" its comment asks for, and the comment now records it.

Flags and settings

None.

Measurements

main: 7ff5256e (2026-11-14)
this PR: contrib/gumminaomi-pipeline-tour @ 9230a64e
Baseline: main @ 7ff5256e's shipped list, made by pipeline_states.py merge on main
Both columns run the same game build: test/play at af15c4a5 (main at f3fe1763 with opt-in branches that change no shaders), the only difference being longshot_pipeline_states_shipped.bin, merged on main and on this PR. This PR changes only that file's sources, so swapping it is the comparison. main's lists haven't changed since f3fe1763.
Hardware: RTX 4070 SUPER (driver 617.42), Ryzen 9 9950X3D (SMT off), 32 GB, Windows 11
Scene: the save "Hyrule Field, night", headless, the game's default options on this PC (Auto preset: DLSS, 2048x1152, 60 fps); RX_VK_LOAD=create, the default on GPUs with mesh shaders
Method: two rounds interleaved (main, PR, main, PR). Each round: no notes and no pipeline cache, then the build's step (longshot --prepare-shaders --compile-shaders most), then a first start and a later start to play, each measured 30 s into play. Times by wall clock; precompile lines from the logs; memory is the process's private bytes and working set (Get-Process) 30 s into play. The driver's own shader cache was warm in both arms (this PC had already compiled these pipelines in the tour), so the compile times are the warm case.
How to reproduce: pipeline_states.py merge --out on main and on this branch, each copied over a game folder's longshot_pipeline_states_shipped.bin, with longshot_pipelines.bin*, mkw_vk_pipelines*.bin, longshot_computes.bin and longshot_compute_variants.bin removed first.

	main @ `7ff5256e`	this PR @ `9230a64e`	Δ
Pipelines noted by the build's step	24,464 / 24,470	34,352 / 34,912	+40%
Build's step, wall (run 1 / run 2, median)	17.8 / 22.5 s, 20.2 s	20.0 / 20.5 s, 20.3 s	+0.1 s
Its precompile line	12.2 / 18.1 s	15.2 / 15.6 s
First start: precompile	2.7 / 2.4 s	3.4 / 3.6 s	+1.0 s
First start: launch to play	30.2 / 26.4 s, 28.3 s	27.7 / 27.5 s, 27.6 s	within noise
Later start: compute programs kept ready	234 (233 listed, 106 dispatched)	2,199 (all listed)	+1,965
Later start: loads at create, before the title	2.8 / 4.2 s, 3.5 s	4.5 / 4.3 s, 4.4 s	+0.9 s
Later start: launch to play	26.1 / 27.5 s, 26.8 s	27.8 / 27.5 s, 27.7 s	+0.9 s
Later start: private memory in play	22,877 / 22,927 MB, 22,902	23,638 / 24,770 MB, 24,204	+1.3 GB
Later start: working set in play	12,594 / 12,645 MB, 12,620	13,275 / 14,423 MB, 13,849	+1.2 GB

The cost is memory: the 1,965 extra compute programs are kept ready at every start, about 0.6 MB each, as the 12 Nov measurement in COMPUTE_MOST's comment predicted (0.6 to 1 MB). Start-up time grows by about a second. Pruning the list's compute programs to those play dispatches is the alternative, and this list would then lose the cutscenes' and actors' compute work.

main-1 build:  vk: precompile: 24464 noted pipelines of 15565 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (0 and 0 not made, 0 whose program variant is not kept), on 16 threads in 12.2 s (the compute ones 4.5 s)
pr-1 build:    vk: precompile: 34352 noted pipelines of 20550 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (0 and 0 not made, 0 whose program variant is not kept), on 16 threads in 15.2 s (the compute ones 4.7 s)
main-2 build:  vk: precompile: 24470 noted pipelines of 15565 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (0 and 0 not made, 0 whose program variant is not kept), on 16 threads in 18.1 s (the compute ones 8.9 s)
pr-2 build:    vk: precompile: 34912 noted pipelines of 20550 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (0 and 0 not made, 0 whose program variant is not kept), on 16 threads in 15.6 s (the compute ones 4.8 s)
main-1 first:  vk: precompile: 24464 noted pipelines of 15565 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (...), on 16 threads in 2.7 s (the compute ones 0.5 s)
pr-1 first:    vk: precompile: 34352 noted pipelines of 20550 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (...), on 16 threads in 3.4 s (the compute ones 0.4 s)
main-2 first:  vk: precompile: 24470 noted pipelines of 15565 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (...), on 16 threads in 2.4 s (the compute ones 0.4 s)
pr-2 first:    vk: precompile: 34912 noted pipelines of 20550 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (...), on 16 threads in 3.6 s (the compute ones 0.4 s)
main-1 later:  vk: precompile: load at create (RX_VK_LOAD=create): 24463 noted pipelines found, none built; 234 compute programs named (233 in the shipped list, 106 dispatched in play here); ... in 2801 ms
pr-1 later:    vk: precompile: load at create (RX_VK_LOAD=create): 32672 noted pipelines found, none built; 2199 compute programs named (2199 in the shipped list, 125 dispatched in play here); ... in 4511 ms
main-2 later:  vk: precompile: load at create (RX_VK_LOAD=create): 24463 noted pipelines found, none built; 234 compute programs named (233 in the shipped list, 106 dispatched in play here); ... in 4200 ms
pr-2 later:    vk: precompile: load at create (RX_VK_LOAD=create): 32672 noted pipelines found, none built; 2199 compute programs named (2199 in the shipped list, 106 dispatched in play here); ... in 4319 ms

Testing

Tested:

nx2/pipeline_states.py check: passes. Merged: 63,920 pipelines (compute 2,199, graphics 57,687, mesh 4,034) of 22,785 programs, version 2.
python -I -B tests/run_all.py on Windows 11, interleaved with main, once each:
- main @ 1be60eae: 191 passed, 59 failed, 321 skipped;
- this PR: 192 passed, 58 failed, 321 skipped.
- Every failure is the same on main: this PC has no g++/cxx setup for those checks. test_cpm_discovery failed on main only. tests/test_pipeline_state_sources.py passes.
The checks that read this PR's files, run with clang wrappers (52 checks, among them test_pipeline_state_computes, test_pipeline_state_sources and test_pipeline_states): all pass but 6 (nx2/test_dll.py, test_driver_cache_folder, test_oot_state_tap, test_tested_configurations, test_utf8_paths, test_walk_key), which fail the same way on main here.
In the game: the build's step, a first start and a later start with this list, twice (Measurements): no errors, every noted pipeline built ("0 and 0 not made").

Not tested:

A truly new PC: the driver's shader cache here already held these pipelines, so the build step's compile time with an empty driver cache is not measured. It costs more there, about 10,000 more pipelines to compile once.
Other GPUs. The records hold the game's NVN formats (version 2), so they apply to any GPU, as the README says.

Needs other testers:

A PC with 16 GB of RAM: the later start's +1.3 GB of private memory, expecting no paging in play (T-BPFC).

Screenshots

Nothing on screen.

Risk and rollback

More memory at every start (+1.3 GB here) and about a second more before the title. Reverting the commit is enough. If the memory matters on small PCs, the list's compute programs can be pruned without touching its graphics pipelines.

Reviewer notes

The list was recorded on a build of f3fe1763 with opt-in branches (late pages' fix on, the rest off), as its note says, not on current main. --prepare-shaders built every pipeline of it on this build ("0 not made"). A pipeline a newer main no longer asks for costs only its build time.
I also have a run at 2560x1440 DLSS with experimental switches (717 more pipelines). It's left out because those switches aren't main's defaults.

Issues

None. Related: #111 (the pipeline tour).

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

A file in docs/tested/ for each new measured run (format in docs/tested-configurations.md; no appended row or assembled view).
Docs updated.
AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

## Why New areas, enemies and cutscenes still stutter the first time they appear, because their pipelines aren't in the shipped list and compile in play. This adds 22,987 pipelines recorded by a scripted tour of every map, actor type and cutscene, so a new install builds them before the title. ## What changed - **pipelines: `assets/pipeline_states/contrib/gumminaomi-tour-every-map-actor-cutscene.bin.xz` + `.md`.** A contributed list added with `nx2/pipeline_states.py add`: 22,987 pipelines no other contributed list holds (19,533 graphics, 1,488 mesh, 1,966 compute programs). 21,848 of them are in no shipped list at all, the tour's included (+52% on the merged 42,072). Recorded cold (`RX_VK_PRECOMPILE=0`) on an RTX 4070 SUPER at this PC's default settings, over these runs: | Part of the route | Not in the shipped list | Compute programs among them | |---|---|---| | Every map at each of its 370 start positions (`StartPos`), child and adult, a camera turn each, the overworld at noon and at midnight | 9,390 | 782 | | Every actor type that has a model (2,086 of `ActorInfo`), made 16 at a time beside Link | 2,798 | 195 | | The 480 cutscenes, requested by name | 9,476 | 986 | | The menus (items, system, equipment, collection, map, world map, threads of time, key items) | 94 | 3 | | A second pass of the maps | 90 | 0 | The script that drove it isn't in this PR: #111 is the pipeline tour, and the host change it needed is offered there (comment on #111). This list doesn't touch `tour.bin.xz`, so a new tour replaces nothing of it. - **nx2: `COMPUTE_MOST` 1024 → 2560** (`nx2/pipeline_states.py`, and the README's number). The merged list now names 2,199 compute programs (233 before), half of the game's 4,127, so `check` stopped at 1024. The measurement below is the "measure again" its comment asks for, and the comment now records it. ## Flags and settings None. ## Measurements main: 7ff5256e (2026-11-14) this PR: contrib/gumminaomi-pipeline-tour @ 9230a64e Baseline: main @ 7ff5256e's shipped list, made by `pipeline_states.py merge` on main Both columns run the same game build: test/play at af15c4a5 (main at f3fe1763 with opt-in branches that change no shaders), the only difference being `longshot_pipeline_states_shipped.bin`, merged on main and on this PR. This PR changes only that file's sources, so swapping it is the comparison. main's lists haven't changed since f3fe1763. Hardware: RTX 4070 SUPER (driver 617.42), Ryzen 9 9950X3D (SMT off), 32 GB, Windows 11 Scene: the save "Hyrule Field, night", headless, the game's default options on this PC (Auto preset: DLSS, 2048x1152, 60 fps); `RX_VK_LOAD=create`, the default on GPUs with mesh shaders Method: two rounds interleaved (main, PR, main, PR). Each round: no notes and no pipeline cache, then the build's step (`longshot --prepare-shaders --compile-shaders most`), then a first start and a later start to play, each measured 30 s into play. Times by wall clock; precompile lines from the logs; memory is the process's private bytes and working set (Get-Process) 30 s into play. The driver's own shader cache was warm in both arms (this PC had already compiled these pipelines in the tour), so the compile times are the warm case. How to reproduce: `pipeline_states.py merge --out` on main and on this branch, each copied over a game folder's `longshot_pipeline_states_shipped.bin`, with `longshot_pipelines.bin*`, `mkw_vk_pipelines*.bin`, `longshot_computes.bin` and `longshot_compute_variants.bin` removed first. | | main @ 7ff5256e | this PR @ 9230a64e | Δ | |---|---|---|---| | Pipelines noted by the build's step | 24,464 / 24,470 | 34,352 / 34,912 | +40% | | Build's step, wall (run 1 / run 2, median) | 17.8 / 22.5 s, 20.2 s | 20.0 / 20.5 s, 20.3 s | +0.1 s | | Its precompile line | 12.2 / 18.1 s | 15.2 / 15.6 s | | | First start: precompile | 2.7 / 2.4 s | 3.4 / 3.6 s | +1.0 s | | First start: launch to play | 30.2 / 26.4 s, 28.3 s | 27.7 / 27.5 s, 27.6 s | within noise | | Later start: compute programs kept ready | 234 (233 listed, 106 dispatched) | 2,199 (all listed) | +1,965 | | Later start: loads at create, before the title | 2.8 / 4.2 s, 3.5 s | 4.5 / 4.3 s, 4.4 s | +0.9 s | | Later start: launch to play | 26.1 / 27.5 s, 26.8 s | 27.8 / 27.5 s, 27.7 s | +0.9 s | | Later start: private memory in play | 22,877 / 22,927 MB, 22,902 | 23,638 / 24,770 MB, 24,204 | +1.3 GB | | Later start: working set in play | 12,594 / 12,645 MB, 12,620 | 13,275 / 14,423 MB, 13,849 | +1.2 GB | The cost is memory: the 1,965 extra compute programs are kept ready at every start, about 0.6 MB each, as the 12 Nov measurement in `COMPUTE_MOST`'s comment predicted (0.6 to 1 MB). Start-up time grows by about a second. Pruning the list's compute programs to those play dispatches is the alternative, and this list would then lose the cutscenes' and actors' compute work. ``` main-1 build: vk: precompile: 24464 noted pipelines of 15565 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (0 and 0 not made, 0 whose program variant is not kept), on 16 threads in 12.2 s (the compute ones 4.5 s) pr-1 build: vk: precompile: 34352 noted pipelines of 20550 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (0 and 0 not made, 0 whose program variant is not kept), on 16 threads in 15.2 s (the compute ones 4.7 s) main-2 build: vk: precompile: 24470 noted pipelines of 15565 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (0 and 0 not made, 0 whose program variant is not kept), on 16 threads in 18.1 s (the compute ones 8.9 s) pr-2 build: vk: precompile: 34912 noted pipelines of 20550 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (0 and 0 not made, 0 whose program variant is not kept), on 16 threads in 15.6 s (the compute ones 4.8 s) main-1 first: vk: precompile: 24464 noted pipelines of 15565 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (...), on 16 threads in 2.7 s (the compute ones 0.5 s) pr-1 first: vk: precompile: 34352 noted pipelines of 20550 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (...), on 16 threads in 3.4 s (the compute ones 0.4 s) main-2 first: vk: precompile: 24470 noted pipelines of 15565 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (...), on 16 threads in 2.4 s (the compute ones 0.4 s) pr-2 first: vk: precompile: 34912 noted pipelines of 20550 programs and 9925 compute pipelines of 4127 compute programs built into the pipeline cache and let go (...), on 16 threads in 3.6 s (the compute ones 0.4 s) main-1 later: vk: precompile: load at create (RX_VK_LOAD=create): 24463 noted pipelines found, none built; 234 compute programs named (233 in the shipped list, 106 dispatched in play here); ... in 2801 ms pr-1 later: vk: precompile: load at create (RX_VK_LOAD=create): 32672 noted pipelines found, none built; 2199 compute programs named (2199 in the shipped list, 125 dispatched in play here); ... in 4511 ms main-2 later: vk: precompile: load at create (RX_VK_LOAD=create): 24463 noted pipelines found, none built; 234 compute programs named (233 in the shipped list, 106 dispatched in play here); ... in 4200 ms pr-2 later: vk: precompile: load at create (RX_VK_LOAD=create): 32672 noted pipelines found, none built; 2199 compute programs named (2199 in the shipped list, 106 dispatched in play here); ... in 4319 ms ``` ## Testing **Tested:** - `nx2/pipeline_states.py check`: passes. Merged: 63,920 pipelines (compute 2,199, graphics 57,687, mesh 4,034) of 22,785 programs, version 2. - `python -I -B tests/run_all.py` on Windows 11, interleaved with main, once each: - main @ 1be60eae: 191 passed, 59 failed, 321 skipped; - this PR: 192 passed, 58 failed, 321 skipped. - Every failure is the same on main: this PC has no `g++`/`cxx` setup for those checks. `test_cpm_discovery` failed on main only. `tests/test_pipeline_state_sources.py` passes. - The checks that read this PR's files, run with clang wrappers (52 checks, among them `test_pipeline_state_computes`, `test_pipeline_state_sources` and `test_pipeline_states`): all pass but 6 (`nx2/test_dll.py`, `test_driver_cache_folder`, `test_oot_state_tap`, `test_tested_configurations`, `test_utf8_paths`, `test_walk_key`), which fail the same way on main here. - In the game: the build's step, a first start and a later start with this list, twice (Measurements): no errors, every noted pipeline built ("0 and 0 not made"). **Not tested:** - A truly new PC: the driver's shader cache here already held these pipelines, so the build step's compile time with an empty driver cache is not measured. It costs more there, about 10,000 more pipelines to compile once. - Other GPUs. The records hold the game's NVN formats (version 2), so they apply to any GPU, as the README says. **Needs other testers:** - A PC with 16 GB of RAM: the later start's +1.3 GB of private memory, expecting no paging in play (T-BPFC). ## Screenshots Nothing on screen. ## Risk and rollback More memory at every start (+1.3 GB here) and about a second more before the title. Reverting the commit is enough. If the memory matters on small PCs, the list's compute programs can be pruned without touching its graphics pipelines. ## Reviewer notes - The list was recorded on a build of f3fe1763 with opt-in branches (late pages' fix on, the rest off), as its note says, not on current main. `--prepare-shaders` built every pipeline of it on this build ("0 not made"). A pipeline a newer main no longer asks for costs only its build time. - I also have a run at 2560x1440 DLSS with experimental switches (717 more pipelines). It's left out because those switches aren't main's defaults. ## Issues None. Related: #111 (the pipeline tour). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] A file in `docs/tested/` for each new measured run (format in `docs/tested-configurations.md`; no appended row or assembled view). - [x] Docs updated. - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers.

GummiNaomi added 3 commits

2026-11-14 19:38:25 +00:00

nx2: COMPUTE_MOST 1024 -> 2560 for the tour's list 011330e9f9

The pipeline tour (every map, actor and cutscene) meets 2,104 compute
programs together with the shipped list's 233, half of the game's
4,127. By the 2026-11-12 measurement that is 1.2 to 2.1 GB more heap
than none kept and some 6.5 s of precompile at each start; not
measured again yet.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

feat(pipelines): a contributed list from a scripted tour of every map, actor and cutscene f5e7c33f20

22,987 pipelines no other contributed list holds (19,533 graphics,
1,488 mesh, 1,966 compute programs), recorded cold on an RTX 4070 SUPER
at the game's defaults: every map at each of its 370 start positions as
child and adult, every actor type with a model, the 480 cutscenes and
the menus. 21,848 of them are in no shipped list at all.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs(tested): the tour list's start-up cost, main's shipped list against it

CI / Conventional Commit title (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions

Details

CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions

Details

CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions

Details

CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions

Details

Third-party parts / As their notes say (pull_request) Blocked by required conditions

Details

Third-party parts / The same as upstream (pull_request) Blocked by required conditions

Details

Website / Website on Cloudflare (pull_request) Blocked by required conditions

Details

beanstalk/approval awaiting maintainer approval

Details

PR description / Description follows the template (pull_request_target) Successful in 3s

Details

9230a64edf

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

GummiNaomi changed title from ~~WIP: perf(pipelines): 22,987 more pipelines prepared before the title, from a tour of every map, actor and cutscene~~ to perf(pipelines): 22,987 more pipelines prepared before the title, from a tour of every map, actor and cutscene

2026-11-14 19:42:45 +00:00