WIP: fix(vk): Intel's Windows driver no longer loses the device on float adds to storage buffers: they are the GPU's own atomic #191
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!191
Loading…
Reference in a new issue
No description provided.
Delete branch "Guep/Longshot:perf-intel-native-fadd"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
On Intel's Meteor Lake iGPU under Windows the game loses the device (
VK_ERROR_DEVICE_LOST) in most runs, in compute programs that add floats to storage buffers: 2 of 3 runs of main in this series, both in Kakariko Village, and 12 of 18 runs with the float-add loop over two series; 0 of 6 with this change. On Intel those adds are still a compare-and-swap loop (NotRexed/Hookshot#2306 made the GPU's own float add the default on NVIDIA's driver only, "until someone posts an Arc run in Kakariko and on Death Mountain Trail withRX_VK_NATIVE_FADD=1"): this is that run.The loop is bounded (it gives up after
CAS_LOOP_MAX_TRIES, 1024 attempts), so it cannot spin forever; the likely mechanism is that heavy contention on a few words (the sky's light sums many invocations into one) makes a dispatch long enough to pass Windows' GPU timeout (TDR, 2 s by default) on an iGPU at 5 to 8 fps, and Windows resets the device. One native atomic per add takes that contention out. Of the programs that lost the device, 499dbfd0 is a float add to a storage buffer and ab7699ee likely one; aba5e6af, 435a0138 and 45ab4d73 are not classified, so whether all of them are covered is not shown, only that none lost the device with the native add.What changed
native_fadd_wanted(gpu/vk/vk_main.cpp) asks forVK_EXT_shader_atomic_float'sshaderBufferFloat32AtomicAddby default, besides NVIDIA's driver, with Intel's Windows driver (VK_DRIVER_ID_INTEL_PROPRIETARY_WINDOWS) on what it was run on only: the Xe-LPG iGPUs of Meteor Lake and Arrow Lake (PCI device0x7Dxx; run on a0x7D55), from driver 101.9175 (Intel's WindowsdriverVersion,101 << 14 | 9175). Intel's other GPUs on Windows (Iris Xe and UHD, Arc discrete, Lunar Lake), older drivers, Mesa's Intel driver, AMD and NVK keep the loop. The feature check after it still applies: a GPU that does not reportshaderBufferFloat32AtomicAddkeeps the loop. The log line names the driver.tests/test_native_float_atomic_add.py's table: unset, Intel's Windows driver is native on0x7Dxxfrom 101.9175 only (not on 101.9174, nor on a Lunar Lake, Iris Xe, Arc A770 or Battlemage ID);RX_VK_NATIVE_FADD=1still asks for it on any of them and=0keeps the loop.docs/development.md'sRX_VK_NATIVE_FADDrow.Flags and settings
RX_VK_NATIVE_FADD(default changed)1: float adds to storage buffers are the GPU's own atomic where it has one;0: the compare-and-swap loop1with NVIDIA's driver, and Intel's Windows driver from 101.9175 on Meteor Lake and Arrow Lake iGPUs (0x7Dxx);0elsewhereMeasurements
main:
9a186f71(2026-11-13)this PR: ls/perf-intel-native-fadd @
c4df9364Hardware: Lenovo 83DJ; Intel Arc Graphics (Core Ultra 7 155H iGPU, Meteor Lake), driver 32.0.101.9175; Core Ultra 7 155H; 32 GB; Windows 11 Pro (26300)
Scene: a save in Kokiri Forest, then the control bridge's teleport to Kakariko Village and to Death Mountain Trail, about 130 s in each; output 1920x1080, Render resolution 4, handheld profile
Method: 3 runs per side, interleaved with the other Intel PRs' builds (main, split, pcache, fadd, then again),
RX_VK_NO_PREDICT=1on every side; this PR's side in a game folder of its own, since the program cache's stamp holds the float add's choice and its cache differs from main's. Per run and area: the median of thefps over the last 5 swindows (the first 4 after the teleport left out: the load) and of theGPU time ... ms a framelines inlongshot_log.txt; an area counts only with 5 windows or more. The table gives the median over the runs, n the runs that count, and the runs that ended in a device loss, with whereHow to reproduce: a save outside Link's house, the options above,
RX_VK_NO_PREDICT=1; teleport with the control bridge (cheats: teleport) to Kakariko Village, then Death Mountain Trail, about 130 s each; read the fps and GPU time lines oflongshot_log.txtper area9a186f71c4df9364Not a performance case: Kokiri Forest is +2 % GPU time, and main's Kakariko (n=2) and Death Mountain Trail (n=1) medians count only the runs that survived, so the fps differences there are not a result. The result is the device losses.
Testing
Tested: Windows 11, Intel Arc iGPU (Meteor Lake): the route above, 3 runs per side in this series, and 3 more in an earlier one on
77f605a2(same route, same settings). The native add lost the device in 0 of 6 runs; the loop (main and the two other Intel PRs' builds) in 12 of 18. The picture is the same in Kakariko Village and on Death Mountain Trail, captures compared; the flat grey mountainside on Death Mountain Trail shows on main as well, so it is not from this change.Not tested: Arrow Lake (default on, as the same Xe-LPG family), Iris Xe and UHD on Windows (default off), Arc discrete, Lunar Lake, older Intel drivers, Mesa's Intel driver (its UHD 770 has no buffer float add, per NotRexed/Hookshot#2306), adult Link's places, night.
Needs other testers:
RX_VK_NATIVE_FADD=1: Kakariko Village and Death Mountain Trail, 150 s each, expecting the same picture, equal or less GPU time and nodevice lost, before the default covers them (request to open)RX_VK_NATIVE_FADD=1and=0: the same, and whether the log says the driver reports the float add at all (request to open)Screenshots
To add: main and this PR at the same spot, from this series' captures.
Risk and rollback
A wrong picture where the light, haze and sky come from these compute programs, if Intel's float add were wrong; none was seen.
The first start after the update, on a Meteor Lake or Arrow Lake iGPU under Windows, is a long one:
support_float32_atomic_addis one word oftranslator_stamp(), soprogram_cache_stamp()changes for the whole file, and that start drops all oflongshot_programs.bin(and its.idx), the prepare's record and, throughpcache_drop_stale,mkw_vk_pipelines.bin. Everything is translated and prepared again. That is needed (a program made withOpAtomicFAddEXTmust not survive into a loop build, nor the reverse), and the same full reset happens for anyone who togglesRX_VK_NATIVE_FADD.RX_VK_NATIVE_FADD=0restores the loop; reverting the commit is enough.Reviewer notes
The device losses were in compute programs 499dbfd0721d2858 (the sky's light, a float add to a storage buffer, from
global_memory_to_storage_buffer_pass.cpp), ab7699ee7f536755 (likely a float add too), and aba5e6af8d4683eb, 435a0138b9013040 and 45ab4d73f7099d62, not classified yet; none was lost with the native add.Issues
See NotRexed/Hookshot#999, NotRexed/Hookshot#2306
Checklist
For everyone
type(scope): description).main. (onfbd42024; measured on9a186f71before the rebase)python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above). (not run onebaae3e6yet: stopped for a measurement series; its own check, tests/test_native_float_atomic_add.py, needs spirv-val (Linux), and its source and switch halves pass here)third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.docs/tested-configurations.mdfor each measured run. (none yet in this branch)tests/that reads changed source updated in this PR, not weakened.The change is small and sits on the existing safe path.
native_fadd_wantedonly decides whether the native add is asked for. The translator still getsOpAtomicFAddEXTonly whenVK_EXT_shader_atomic_floatis present andshaderBufferFloat32AtomicAddis reported:dev.float_atomic_add = atomic_float && afloat.shaderBufferFloat32AtomicAdd, and that one feature is enabled atvkCreateDevice.make_profilepasses it on assupport_float32_atomic_add, andDefineStorageBuffersalso requires descriptor aliasing and a shader that uses the add. An Intel Windows GPU that doesn't report the feature keeps the loop. NVIDIA, AMD (proprietary and RADV), Mesa's Intel driver (ANV) and NVK behave exactly as before. The switch check gets its table updated, includingRX_VK_NATIVE_FADD=0on Intel Windows, and nothing in it is weakened. The 0-of-6 against 12-of-18 device-loss evidence is persuasive. The unticked rebase box is honest: main has moved 16 commits (to1ef93e79) and changedvk_main.cppanddocs/development.md, but not in these hunks.Must change
fix, notperf. The PR exists because Intel loses the device in Kokiri Forest. The evidence is a bug reproduced on main and gone with the change (2/3 and 12/18 runs lost against 0/3 and 0/6), which is CONTRIBUTING's proof forfix. The version bump is the same (both are a patch), but git-cliff filesperfunder Performance. Intel players looking for the crash in the Fixes section of the v3.0.0-rc notes would miss it. The perf numbers also don't hold up as a perf case. Kokiri Forest is +2% GPU time (worse). Main's Kakariko (n=2) and Death Mountain Trail (n=1) medians count only the runs that survived, because the losses dropped the others.perfalso asks for the 1% low and p99, which aren't given. Suggested title:fix(vk): Intel's Windows driver no longer loses the device on float adds to storage buffers: they are the GPU's own atomic. Rename the commit to match, and make "Why" lead with the crash.Should change
No driver-version or architecture gate, and the largest Intel group is untested.
VK_DRIVER_ID_INTEL_PROPRIETARY_WINDOWScovers every Intel GPU on Windows: Arc discrete, Lunar Lake, and also Iris Xe and UHD (Tiger Lake/Alder Lake, plus the Gen9–Gen11 legacy driver branch). Wherever those drivers reportshaderBufferFloat32AtomicAdd, the PR turns the native add on with no runs behind it. Only one Meteor Lake iGPU on driver 32.0.101.9175 was tried. The "Untested on" column leaves out Iris Xe and UHD entirely. Either name them and open a test request for at least one Xe-LP laptop on the current driver, or gate the default to what was measured (a minimumdriverVersion, or Xe-LPG and later). Say which choice you made, as CONTRIBUTING asks of a new default. I couldn't check which of these drivers report the feature.The cache cost is described wrongly. The description says "the first start after the update translates the programs that add floats again". In fact
support_float32_atomic_addis one word oftranslator_stamp()(vk_shader.cpp), soprogram_cache_stamp()changes for the whole file. On Intel Windows the first start drops the entirelongshot_programs.bin(and its.idx), the prepare record and, throughpcache_drop_stale,mkw_vk_pipelines.bin. Everything is translated and prepared again. That invalidation is correct and needed, since a program carryingOpAtomicFAddEXTmust not survive into a loop build, and the reverse. But on the slowest GPUs Longshot supports it means one long first start. Say so in "Risk and rollback". The same full reset happens again for anyone who togglesRX_VK_NATIVE_FADD.Explain why a bounded loop loses the device. The CAS loop gives up after
CAS_LOOP_MAX_TRIES(1024) attempts, so it can't spin forever. The likely mechanism is that heavy contention pushes a dispatch past Windows' TDR timeout on an iGPU running at about 6 fps. One sentence on that in "Why" would tell reviewers the fix targets the cause. It would also show whether the three unclassified programs (aba5e6af…, 435a0138…, 45ab4d73…) are expected to be covered. Small related point: the description says Kokiri Forest, but main's two losses in this series were both in Kakariko Village.Nit: the new
native_fadd_wantedcomment adds "an RTX 5080" to the NVIDIA runs. No row indocs/tested-configurations.mdor in this PR backs it. Link the run, or drop it.Still open on the PR's own list: rebase,
tests/run_all.py, the test request, screenshots, the tested-configurations rows, and the Linux build (this touches only Windows-relevant behaviour, butvk_main.cppbuilds on both).@ -253,3 +253,1 @@/// value on); unset, only with NVIDIA's driver, the one the native add was tried on in the game (an RTX 4090 in Kakariko/// and on Death Mountain Trail). AMD's RDNA 3 and later and Intel's Arc report the feature too but were not tried: they/// keep the loop unless the switch asks, until runs on them say otherwise/// value on); unset, with NVIDIA's driver (an RTX 4090 and an RTX 5080 in Kakariko and on Death Mountain Trail) andNit: "an RTX 5080" is new here, and no row in docs/tested-configurations.md backs it. Link the run, or drop it.
@ -256,3 +256,3 @@static bool native_fadd_wanted(const char* env, VkDriverId driver) {if (env) return !env_value_off(env);return driver == VK_DRIVER_ID_NVIDIA_PROPRIETARY;return driver == VK_DRIVER_ID_NVIDIA_PROPRIETARY || driver == VK_DRIVER_ID_INTEL_PROPRIETARY_WINDOWS;This turns the native add on for every Intel GPU on Windows that reports
shaderBufferFloat32AtomicAdd, at any driver version: Iris Xe and UHD (Tiger Lake/Alder Lake, the Gen9-11 legacy branch), Arc discrete and Lunar Lake as well as the one Meteor Lake iGPU measured (32.0.101.9175). The feature check after this keeps it safe where the feature is missing, but none of the GPUs that report it were run besides that one. Either gate it (a minimumdriverVersion, or Xe-LPG and later) or name Iris Xe/UHD under Untested and get one test-request run on an Xe-LP laptop.@ -508,2 +509,3 @@else logf("vk: float atomic add on storage buffers: %s, %s", how,fadd_wanted ? "NVIDIA's driver" : "by default only with NVIDIA's driver (RX_VK_NATIVE_FADD=1: native where the GPU has it)");!fadd_wanted ? "by default only with NVIDIA's and Intel's Windows drivers (RX_VK_NATIVE_FADD=1: native where the GPU has it)": fadd_driver.driverID == VK_DRIVER_ID_NVIDIA_PROPRIETARY ? "NVIDIA's driver" : "Intel's Windows driver");Fine: with RX_VK_NATIVE_FADD unset,
fadd_wantedcan only be NVIDIA or Intel Windows, so the else-branch naming Intel is exact. Worth knowing: switching the default flipstranslator_stamp()'s float_add word, so on Intel Windows the first start after the update drops the whole program cache and mkw_vk_pipelines.bin (pcache_drop_stale), not only "the programs that add floats" as the description says. That's correct, but on these iGPUs it is one long first start: say so under Risk.c4df93642bebaae3e659WIP: perf(vk): on Intel's Windows driver float adds to storage buffers are the GPU's own atomicto WIP: fix(vk): Intel's Windows driver no longer loses the device on float adds to storage buffers: they are the GPU's own atomicThank you. Rebased on
fbd42024; the changes are in9cb92cdeandebaae3e6.fixnow, and "Why" leads with the device losses, with the right place: Kakariko Village, where main's two losses of this series were.0x7Dxx; the run was on a0x7D55), from driver 101.9175 (101 << 14 | 9175, the encoding the log'sdriver 0x1963d7confirms). Iris Xe and UHD, Arc discrete, Lunar Lake and older drivers keep the loop and are named under Untested; a test request for an Xe-LP laptop on the current driver is listed. The table covers 101.9174, Lunar Lake, Iris Xe, A770 and Battlemage IDs. Arrow Lake is on by family, not by a run: say if you would rather leave it out.mkw_vk_pipelines.bin, and that toggling the switch does the same.Still to do before
pr ready: the test requests, screenshots, the tested-configurations rows, a Linux build.tests/run_all.pynot run onebaae3e6yet (stopped for a measurement series); the check's source and switch halves pass here, the SPIR-V half needs spirv-val.Re-review at
ebaae3e6. Code only: I didn't build it, run the tests or run the game.Earlier points:
perf→fix: addressed. The title and both commits arefix, and "Why" now leads with the device losses, in Kakariko Village.0x7Dxxfrom 101.9175 only. I checked both halves:(device & 0xFF00u) == 0x7D00uis exactly0x7D00–0x7DFF. That range is Meteor Lake (0x7D40,0x7D45,0x7D55,0x7D60,0x7DD5) and Arrow Lake (0x7D41,0x7D51,0x7D67,0x7DD1), and as far as I know nothing else. Lunar Lake (0x64xx), Iris Xe (0x9Axx/0x46xx), Arc A-series (0x56xx) and Battlemage (0xE2xx) all fall outside it. The vendor isn't checked, butVK_DRIVER_ID_INTEL_PROPRIETARY_WINDOWSalready implies Intel.driverVersionis 18.14 bits (build in the low 14).101 << 14 | 9175= 1663959 =0x1963d7, which matches the run's log. The comparison is parenthesised, so>=doesn't bind before|. 9175 fits in 14 bits. Older branches (100.x, 30.0.100.x) compare lower, and a later major compares higher.dev.propsis filled inpick_devicebeforecreate_devicereads it.=1on Iris Xe and=0on the measured GPU. Iris Xe/UHD are now named under Untested, with a test request planned.docs/development.mdrow both say the first start, and any toggle of the switch, empty the whole program cache andmkw_vk_pipelines.bin.New since then:
fadd_wantedis true only for NVIDIA's driver or the gated Intel case.fbd42024. Of the files this touches, onlydocs/development.mdchanged there, in a hunk away from theRX_VK_NATIVE_FADDrow. No shaders or*_spv.hare touched.create_deviceis now 155 columns, followed by a two-word line. Re-wrap it. The test's docstring has the same kind of break ("NVK); set," / "RX_VK_NATIVE_FADD").Verdict: comment, not changes. The code is right and every earlier point is settled. Before
pr ready:tests/run_all.pyonebaae3e6(or CI's run on it).ebaae3e6(notc4df9364). The measured runs predate the gate, so paste thevk: float atomic add on storage buffers: native ..., Intel's Windows driver, Meteor Lake or Arrow Lakeline, to show the gate fires on the realdeviceID/driverVersion.docs/tested-configurations.mdrows.@ -268,2 +271,3 @@if (env) return !env_value_off(env);return driver == VK_DRIVER_ID_NVIDIA_PROPRIETARY;if (driver == VK_DRIVER_ID_NVIDIA_PROPRIETARY) return true;return driver == VK_DRIVER_ID_INTEL_PROPRIETARY_WINDOWS && (device & 0xFF00u) == 0x7D00u && version >= (101u << 14 | 9175u);Checked: the mask is exactly 0x7D00-0x7DFF, which is Meteor Lake and Arrow Lake only.
101u << 14 | 9175uis 0x1963d7 in Intel's 18.14 encoding and matches the run's log. The parentheses keep>=from binding before|. Good.@ -357,2 +361,2 @@// spirv_emit_context.cpp DefineStorageBuffers). By default with NVIDIA's driver only (native_fadd_wanted);// RX_VK_NATIVE_FADD=1 asks for it on any GPU, =0 keeps the loop on every one, the extension not even enabled// spirv_emit_context.cpp DefineStorageBuffers). By default with NVIDIA's driver, and Intel's Windows driver on the// Meteor Lake and Arrow Lake iGPUs (native_fadd_wanted); RX_VK_NATIVE_FADD=1 asks for it on any GPU, =0 keeps the loop on every one, the extension notNit: this line is 155 columns, and the next one is just "even enabled". Re-wrap the comment.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.