Game: up-front shader preparation does not finish on Intel Arc (Windows) with compiles one at a time #407
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot#407
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Longshot version
Native Windows development build of main
cf605c85bbaeaaa78176b933b3bc1a3366e13a78. Its log reportscf605c85 (source 8672daee).The long failing run used the co-op development build
1ed75e61b5bc083247950a262b1a48230f44909a, based on that main commit. Its log reportsunknown (source c058db8c).These are source builds, not a released or nightly package.
Channel
Built from source
What kind of problem
Something else
What happened
The shader preparation before play can sit without visible progress for minutes. In the long run below, it ended itself with “the GPU driver stopped responding” after five minutes without a progress change. It left a marker telling later starts to build the remaining shaders during play instead.
Main build, source 8672daee: one preparation-only observation, stopped by a quit request after 391.876 seconds, exit 0. It did not finish preparing shaders, but the watchdog did not fire during this bounded run. Before starting, there was no
.warmingor skipped-preparation marker. Existing caches held 555 program keys, about 69.1 MB of translated programs and 59.7 MB of driver pipeline data. No marker was manually changed.The longest reported pause on main was 57 seconds. During a 59.97-second CPU sample, the process used 60.11 CPU-seconds, roughly one logical CPU, while reporting 22 preparation workers.
Co-op build, source c058db8c: the uninterrupted run ended itself after 97.51 minutes, exit 4, at 262/38,809. The final five minutes had no displayed-counter advance. From the approximately 30-minute observation until exit, it advanced at about 7.2 completed work slots per minute. The displayed counter is completed slots divided by three and includes skipped variants; it is not a count of successful distinct driver compiles. An earlier one-minute sample used 59.47 CPU-seconds in 60 seconds.
A subsequent bounded co-op retry retained its cache and markers. The surviving
.warmingmarker automatically omitted seed-pass driver warming. The seed phase finished in 57.7 seconds, reporting 0 compute pipelines and 70 failed results, then continued into noted pipelines. A quit request ended it after 390.160 seconds, exit 0, still incomplete. This changed retry path is not a matched speed comparison or proof that the original stall was fixed.Neither build logged
VK_ERROR_DEVICE_LOSTor a caught driver fault. Both logged a compute translation failure for79116b290d46542fand intentional network exclusions. The watchdog's message alone does not establish an actual device loss. The exact program and thread inside the driver at the long run's stop remain unknown.The Intel setting has only two values, read at startup:
intel_shader_compiles=0: One at a time.intel_shader_compiles=1: Side by side.At main
cf605c85, these are defined ingpu/vk/vk_menu.h:86-87andgpu/vk/vk_menu.cpp:1414-1415; the INI clamp is at line 4153. The gate caches the startup choice ingpu/vk/vk_internal.h:132-135.int intel_compiles = 0; // (longshot) Intel's shader compiler on Windows: 0 one thread at a time, 1 side by side (vk_internal.h// IntelDriverGate; RX_VK_NO_INTEL_SERIAL=1 over it). Read once, at the start{TAB_DEBUG, "Intel shader compiles", &opts.intel_compiles, 0, 1, 1, tx_intel_compiles,"Intel graphics: how many shaders Intel's compiler is given at once. Applies after a restart.", hn_intel_compiles, true, true},*r.value = r.value == &opts.output_res ? parse_output(eq + 1) : std::min(std::max(v, r.lo), r.hi);inline bool intel_gate_on() {#ifdef _WIN32static const bool on = !intel_compiles_side_by_side() && dev.props.vendorID == 0x8086;return on;Not tried: Side by side, another driver or a different process priority. Side by side was left untested because a driver hang on this unattended PC could require a restart. Both sampled processes ran at BelowNormal priority. The builds and cache states differ, so their rates are not a controlled comparison.
What would help decide: a per-program timer around compute warming, recording the program key, variant, thread ID, start/end times and time waiting for the Intel gate. Existing compute diagnostics do not provide that paired record, and the last saved cache entry does not identify the blocked call.
Related discussions:
#252
#15
#190 (comment)
Where in the game
Before the title screen, during up-front shader preparation with
--prepare-shaders. No gameplay was involved.How often
Once so far
Graphics card and driver version
Intel Arc integrated graphics in a Core Ultra 9 185H, PCI device 0x7D55. Intel driver 32.0.101.9175, Vulkan 1.4.
CPU, RAM and operating system
Intel Core Ultra 9 185H, 16 cores and 22 logical processors, 64 GB RAM, Windows 11 Pro.
Changes from the defaults
intel_shader_compiles=0).RX_VK_NATIVE_FADD=1, matching this hardware and driver's default.RX_VK_PREPARE_STATUS_S=1on main and the short co-op retry; the long run used the normal status cadence. The 300-second watchdog was unchanged.RX_VK_IR=logandRX_VK_PTR_DIAG=1for diagnostic text. No shader dumps were produced.longshot_log.txt
Main development build:
Long co-op-build run:
These are text excerpts only. No caches, game files, executable or saves are attached.
Screenshots or video
None. These observations ran without a game window.
Before you send it
The first box is not asserted: the observations above are on the named development source builds, not a freshly tested release or nightly. The main observation did not reproduce the watchdog exit within its time limit.
Follow-up to "Not tried: Side by side" above. Tried now, with someone at the PC in case the driver hung, on the same native main
cf605c85(source 8672daee), Core Ultra 9 185H / Intel Arc, Windows 11, driver 32.0.101.9175.Both preparation runs started from byte-identical copies of the existing 591,471,179-byte cache. I moved only the interrupted-warming marker aside in both starting copies, so neither skipped the seed's driver-warm pass. Both loaded 864 program keys and the same driver pipeline cache. Compile shaders was Most; priority stayed BelowNormal.
The one-minute CPU samples used 63.70 CPU-seconds in 59.96 seconds and 1,135.53 in 59.95 seconds, about 1.06 versus 18.94 busy logical CPUs. These are the game's displayed preparation counters, not successful distinct driver compiles.
Both answered quit and exited 0, at 241.082 and 242.172 seconds. Neither completed preparation or logged device loss, a caught driver fault or a watchdog stop. Both retained the known
79116b290d46542ftranslation failure. Side by side was selected only for the process withRX_VK_NO_INTEL_SERIAL=1; no saved option was changed. This binary logs the serial gate positively but has no corresponding positive Side-by-side log line.This is one ordered pair on one PC and driver, not a full-preparation result, a general speed claim or a guarantee of stability.
Keeping P's cache, I then loaded the test save and stood where it starts, with Side by side on. The native option reply confirmed value1, “Side by side”. That run exited0 after 385.511 seconds, with 325.7 seconds between the first and last observed play states and no device-loss or driver-fault line. The inspected final picture shows Link, Navi and the HUD inside Link's house.
Corrected 14 November: this paragraph first said the run stood in the forest. The test save starts inside Link's house (the game reports the map as Kokiri Forest, but the scene and its music are the house's), and the picture shows that room. So this play run is a light indoor scene and says nothing about the open forest. Two nonfatal write-back wait timeouts occurred while loading. This was warm-cache play, not a repeated cold compile stress test.
The original cache, including its warming marker, is now restored. The saved options file is byte-identical to before the test.
Numbers from an Arc B580 for this, in play rather than in the preparation: with One at a time a first visit to an area freezes the picture for minutes; with Side by side it does not freeze at all, and nothing faulted in ten areas.
Arc B580 12 GB, driver 32.0.101.9154, Core i5-13600K (20 threads, 6 background workers), 64 GB, Windows 11. A source build of main
cf605c85with the B580 fixes of #443, #444 and #446 in (main alone loses the device on this card), a game folder never prepared (RX_VK_PRECOMPILE=0), each area entered by the debug teleport for the first time on this PC. An area can be new only once, so the two settings were run in different areas; the programs counted are the log'svk: in play:lines.RX_VK_TRANSLATE_WAIT_MS=8RX_VK_NO_INTEL_SERIAL=1)RX_VK_TRANSLATE_WAIT_MS=8translation ... ms in allreached 158,713 ms after Hyrule Field and 541,531 ms after the Market; side by side it was 1,636 ms after Kakariko. Both timers start before the gate, so most of that is queueing at the lock, not translating.VK_ERROR_DEVICE_LOST, no fault line, no exit, and the areas looked at are drawn right.RX_VK_COMPUTE_SERIALwas left at its default.So on Battlemage with this driver the gate costs minutes a new area and its reason (igvk64 faulting with two threads in it) did not show. I am preparing a pull request that makes Side by side the default on the Arc B-series from 101.9154, the family and driver it was run on, and leaves every other Intel GPU as it is; the up-front preparation on this card is not measured yet.
The pull request for the Arc B-series is #500 : the row gets an Auto value, its default, which is side by side on Battlemage from driver 101.9154 and one at a time on every other Intel GPU. One build, the default against RX_VK_NO_INTEL_SERIAL=0: 373 new programs with no frame under 5 fps against 372 with 181 s of it.