Warm the stage libraries from the program cache into the pipeline cache #1586

Merged

jarynclouatre merged 2 commits from feat/gpl-library-warm into main

2026-11-10 16:55:35 +00:00

jarynclouatre commented

2026-11-10 09:52:40 +00:00

(Migrated from github.com)

When it runs: hookshot_prepared.bin gets a second stamp, for the driver's part (the translator stamp, the device's pipeline cache UUID, vendor, device and driver version, and whether the libraries are wanted). With RX_VK_GPL set, the warm pass runs after a first prepare, and on its own, with nothing translated again, when only that stamp is stale: the libraries switched on later, a new driver or GPU. It walks every graphics and mesh program in the program cache, the ones the pipeline notes name first, and counts as done only once the pipeline cache is saved. A start that died inside the pass is not tried again for that driver, as before. Without RX_VK_GPL nothing is warmed, but the same stamp makes one predicted-pipelines pass run again after a driver update, and once on the first start with this change (the old file has no second stamp).

The precompile now builds its first graphics pipeline alone and the rest in parallel. After the warm, NVIDIA's driver faulted inside vkCreateGraphicsPipelines (nvoglv64+0xde2515, a null read) when all the precompile's workers began their graphics pipelines at once. guarded_graphics caught it, but a driver lock stayed held and the start hung at its first submit. Main was never seen to fault in the same test (0 in 100 starts of its own folder, 0 in 50 with a warmed folder's driver cache). If the first build never returns, the other graphics notes wait behind it, so the 30 s rule goes on to the game with none of them built.

RX_VK_GP in the program cache stamp also matched RX_VK_GPL, so switching the libraries on threw the program cache away; it is RX_VK_GP_ now. vk_shader.cpp is in the translator stamp, so this costs one program cache rebuild.

Testing, all on one PC: RTX 4070 Ti Super, driver 621.14, Windows 11, Ryzen 7 9800X3D. RX_VK_GPL=1 except where it says libraries off.

Build: on 779bdcf the Windows DLL builds and the two new tests (tests/test_prepared_stamps.py, tests/test_gpl_library_cache.py) pass, built with g++ on Linux. The game tests below ran on 7d1562d, before the rebase; nothing that moved since touches the prepare or the precompile.

Starts: a folder prepared and warmed once, then restored to that state before each start and stopped once the precompile was done. Without the first-alone change, starts faulted or hung: 14 in 100 on an earlier base, 1 in 50 on 7d1562d. With it: 0 in 100 on the earlier base and 0 in 100 on 7d1562d.

Route: the scripted 10-minute route from #1429 (boot, Hyrule Field, Kakariko, Market, Lost Woods, Gerudo Valley), first visit, one run each on 7d1562d, each on a fresh folder after its first start:

	main, libraries off	this, RX_VK_GPL=1
frames presented in 595 s	31,200	33,360
time lost against 60 fps, s	75.8	38.1
waits for pipelines (their time, s)	509 (63.6)	44 (19.2)
draws left out	362,088	3,720
Market: time lost, s	12.9	3.4

Most of what is left here comes from programs translated in play (92 in the run), whose libraries are cold.

Costs: the first start warmed 85,038 stage libraries (42,519 programs) in 418 s and reached the title after 437 s; main's first start took 23 s. Later starts spend about 1.4 s in the precompile. The pipeline cache file is about 2.1 GB.

Switches (set to anything): RX_VK_GPL_NO_CACHE keeps the libraries out of the pipeline cache, as before; RX_VK_NO_WARM skips the warm pass (it existed already); RX_VK_GPL_SYNC as in #1429.

Only NVIDIA was measured. Turning the libraries on by default for NVIDIA is the maintainer's call.

The prepare built each program's stage libraries with no pipeline cache of ours, for the driver's own disk cache, and only for programs it translated in that same prepare. Asked for under #1429's flag, the driver did not answer from its own cache: of 23 cold libraries probed right after such a warm, 21 had just been built. The libraries are now built into our pipeline cache, which the driver does answer from (title screen: 6 had and 0 cold, before 3 and 23). When it runs: hookshot_prepared.bin gets a second stamp, for the driver's part (the translator stamp, the device's pipeline cache UUID, vendor, device and driver version, and whether the libraries are wanted). With RX_VK_GPL set, the warm pass runs after a first prepare, and on its own, with nothing translated again, when only that stamp is stale: the libraries switched on later, a new driver or GPU. It walks every graphics and mesh program in the program cache, the ones the pipeline notes name first, and counts as done only once the pipeline cache is saved. A start that died inside the pass is not tried again for that driver, as before. Without RX_VK_GPL nothing is warmed, but the same stamp makes one predicted-pipelines pass run again after a driver update, and once on the first start with this change (the old file has no second stamp). The precompile now builds its first graphics pipeline alone and the rest in parallel. After the warm, NVIDIA's driver faulted inside vkCreateGraphicsPipelines (nvoglv64+0xde2515, a null read) when all the precompile's workers began their graphics pipelines at once. guarded_graphics caught it, but a driver lock stayed held and the start hung at its first submit. Main was never seen to fault in the same test (0 in 100 starts of its own folder, 0 in 50 with a warmed folder's driver cache). If the first build never returns, the other graphics notes wait behind it, so the 30 s rule goes on to the game with none of them built. RX_VK_GP in the program cache stamp also matched RX_VK_GPL, so switching the libraries on threw the program cache away; it is RX_VK_GP_ now. vk_shader.cpp is in the translator stamp, so this costs one program cache rebuild. Testing, all on one PC: RTX 4070 Ti Super, driver 621.14, Windows 11, Ryzen 7 9800X3D. RX_VK_GPL=1 except where it says libraries off. Build: on 779bdcf the Windows DLL builds and the two new tests (tests/test_prepared_stamps.py, tests/test_gpl_library_cache.py) pass, built with g++ on Linux. The game tests below ran on 7d1562d, before the rebase; nothing that moved since touches the prepare or the precompile. Starts: a folder prepared and warmed once, then restored to that state before each start and stopped once the precompile was done. Without the first-alone change, starts faulted or hung: 14 in 100 on an earlier base, 1 in 50 on 7d1562d. With it: 0 in 100 on the earlier base and 0 in 100 on 7d1562d. Route: the scripted 10-minute route from #1429 (boot, Hyrule Field, Kakariko, Market, Lost Woods, Gerudo Valley), first visit, one run each on 7d1562d, each on a fresh folder after its first start: | | main, libraries off | this, RX_VK_GPL=1 | |---|---|---| | frames presented in 595 s | 31,200 | 33,360 | | time lost against 60 fps, s | 75.8 | 38.1 | | waits for pipelines (their time, s) | 509 (63.6) | 44 (19.2) | | draws left out | 362,088 | 3,720 | | Market: time lost, s | 12.9 | 3.4 | Most of what is left here comes from programs translated in play (92 in the run), whose libraries are cold. Costs: the first start warmed 85,038 stage libraries (42,519 programs) in 418 s and reached the title after 437 s; main's first start took 23 s. Later starts spend about 1.4 s in the precompile. The pipeline cache file is about 2.1 GB. Switches (set to anything): RX_VK_GPL_NO_CACHE keeps the libraries out of the pipeline cache, as before; RX_VK_NO_WARM skips the warm pass (it existed already); RX_VK_GPL_SYNC as in #1429. Only NVIDIA was measured. Turning the libraries on by default for NVIDIA is the maintainer's call.

hckrman101 (Migrated from github.com) approved these changes

2026-11-10 16:39:56 +00:00

hckrman101 (Migrated from github.com) left a comment

Read through the diff; I didn't build or run it. CI is green and the logic holds up.

Checked: the sixth record word is read and written consistently (old 5-word records read as warmed = 0), prepare_check keeps warmed only for the same translator stamp, and both prepare_run paths write the new stamp only after warm_libraries returns true. The .warming guard still covers a crash inside the pass. In build_graphics, !cold on the pointer is the "not a cold probe" check, and pcache_built_since_save is atomic, so the save counting is fine. The first-graphics-alone lock in precompile_run is sound: later workers wait, then see graphics_begun and drop the lock. The RX_VK_GP_ prefix change is right (it was swallowing RX_VK_GPL). test_gpl_library_cache.py has no entry in run_all.py but is picked up by the tests/ listing and needs no args, so that's fine.

Nits, no action needed:

With the libraries cached, a ~2 GB pipeline cache gets loaded and saved on every start. You measured and disclosed it, but it is worth keeping an eye on for people with slow disks.
lib_cached ignores driver_cache_only, so predict_run's library builds now also land in our cache. That looks intended; a short comment would make it explicit.

Good to go from my side.

Read through the diff; I didn't build or run it. CI is green and the logic holds up. Checked: the sixth record word is read and written consistently (old 5-word records read as warmed = 0), prepare_check keeps `warmed` only for the same translator stamp, and both prepare_run paths write the new stamp only after warm_libraries returns true. The `.warming` guard still covers a crash inside the pass. In build_graphics, `!cold` on the pointer is the "not a cold probe" check, and pcache_built_since_save is atomic, so the save counting is fine. The first-graphics-alone lock in precompile_run is sound: later workers wait, then see `graphics_begun` and drop the lock. The `RX_VK_GP_` prefix change is right (it was swallowing RX_VK_GPL). test_gpl_library_cache.py has no entry in run_all.py but is picked up by the tests/ listing and needs no args, so that's fine. Nits, no action needed: - With the libraries cached, a ~2 GB pipeline cache gets loaded and saved on every start. You measured and disclosed it, but it is worth keeping an eye on for people with slow disks. - `lib_cached` ignores `driver_cache_only`, so predict_run's library builds now also land in our cache. That looks intended; a short comment would make it explicit. Good to go from my side.

Commenting is not possible because the repository is archived.

No reviewers

hckrman101