The Builder's headless shader prep is stopped by its 15-minute silence watchdog on slow (AMD) drivers, pushing a multi-hour first start onto the player #2242

Closed

opened 2026-11-11 10:38:30 +00:00 by kafasam · 2 comments

kafasam commented

2026-11-11 10:38:30 +00:00

(Migrated from github.com)

Machine: i7-12700K, RX 6700 XT (gfx1031), Windows, AMD proprietary driver. Two source builds of the v2.1.0 tree (python -I nx2/make_game.py) on this machine ended their headless preparation with:

the shaders were not prepared now (the game printed nothing for 15 minutes and was stopped): the game prepares them at its first start

In one of the two, the headless game had already printed prepared 38809 shader programs (46597 new, 70 failed, 9687 compute pipelines...) in 1437.8 s and went silent after — so it stopped mid-way through a run that was demonstrably still doing work. Running the same preparation interactively (game window shown) it completed:

precompile: 17927 programs, 10796 graphics and 9691 compute pipelines, 1204 mesh fallback programs in 3135609 ms (52 minutes; the shaders took 3136.7 s)

The first-start fallback this leaves is expensive on this driver class: the predicted-pipelines stage counts 65,815 pipelines here and ground at roughly 5-85 a minute without a budget (hours); RX_VK_PREDICT_S bounded it and the precompile's own 30-second no-progress escape fired at 21,677 of 21,691, which is what made the first session playable at all — but the builder-time preparation is what this machine never gets to keep.

Question: is the 15-minute silence shorter than an AMD-slow preparation's quiet stretches? The interactive run prints a progress line every 10 s, so the silent headless stretch looks like a different printing path rather than a stalled one. Could the watchdog scale with the driver, or the headless run print the same progress the start-up screen does? Happy to run an instrumented build here — the machine reproduces it on every source build.

Machine: i7-12700K, RX 6700 XT (gfx1031), Windows, AMD proprietary driver. Two source builds of the v2.1.0 tree (python -I nx2/make_game.py) on this machine ended their headless preparation with: `the shaders were not prepared now (the game printed nothing for 15 minutes and was stopped): the game prepares them at its first start` In one of the two, the headless game had already printed `prepared 38809 shader programs (46597 new, 70 failed, 9687 compute pipelines...) in 1437.8 s` and went silent after — so it stopped mid-way through a run that was demonstrably still doing work. Running the same preparation interactively (game window shown) it completed: `precompile: 17927 programs, 10796 graphics and 9691 compute pipelines, 1204 mesh fallback programs in 3135609 ms` (52 minutes; `the shaders took 3136.7 s`) The first-start fallback this leaves is expensive on this driver class: the predicted-pipelines stage counts 65,815 pipelines here and ground at roughly 5-85 a minute without a budget (hours); `RX_VK_PREDICT_S` bounded it and the precompile's own 30-second no-progress escape fired at 21,677 of 21,691, which is what made the first session playable at all — but the builder-time preparation is what this machine never gets to keep. Question: is the 15-minute silence shorter than an AMD-slow preparation's quiet stretches? The interactive run prints a progress line every 10 s, so the silent headless stretch looks like a different printing path rather than a stalled one. Could the watchdog scale with the driver, or the headless run print the same progress the start-up screen does? Happy to run an instrumented build here — the machine reproduces it on every source build.

addisonbair commented

2026-11-11 16:22:37 +00:00

(Migrated from github.com)

A related case that the silence watchdog can't catch: here the prep never goes silent. Main d7a7306, make_game.py --update on Linux, NVIDIA RTX PRO 2000 Blackwell Laptop, driver 580.178.04. The pipeline prediction reached 4 short of the total within its first 10 s and then stayed there, printing every 10 s for 1 h 47 min until I stopped it:

vk: predicted pipelines: 64793 of 64797 pipelines in 10 s, about 0 s left
...
vk: predicted pipelines: 64793 of 64797 pipelines in 6435 s, about 0 s left

The progress line keeps the watchdog fed, so it waits forever. The log doesn't say which 4 pipelines they are. The game's own start-up compile on the same machine once sat at start-up: 67095 of 67099 too, and got past it in a later run. HOOKSHOT_NO_PREPARE=1 for the build and RX_VK_PRECOMPILE=0 for the runs got me around it. Maybe the prep could log the pipelines still outstanding after N minutes, and give up on them.

A related case that the silence watchdog can't catch: here the prep never goes silent. Main d7a7306, `make_game.py --update` on Linux, NVIDIA RTX PRO 2000 Blackwell Laptop, driver 580.178.04. The pipeline prediction reached 4 short of the total within its first 10 s and then stayed there, printing every 10 s for **1 h 47 min** until I stopped it: ``` vk: predicted pipelines: 64793 of 64797 pipelines in 10 s, about 0 s left ... vk: predicted pipelines: 64793 of 64797 pipelines in 6435 s, about 0 s left ``` The progress line keeps the watchdog fed, so it waits forever. The log doesn't say which 4 pipelines they are. The game's own start-up compile on the same machine once sat at `start-up: 67095 of 67099` too, and got past it in a later run. `HOOKSHOT_NO_PREPARE=1` for the build and `RX_VK_PRECOMPILE=0` for the runs got me around it. Maybe the prep could log the pipelines still outstanding after N minutes, and give up on them.

NotRexed commented

2026-11-11 21:27:41 +00:00

(Migrated from github.com)

Thanks for the report. Reports of the same problem are now collected in #2505, which lists this one with your details and links back here, so one fix closes them together and the open list stays readable. Follow #2505 for the fix, and add new details there (or here). If this turns out to be a separate problem it will be reopened.