MagicBeanDev/Hookshot
Archived
Watch
2
0
Fork
You've already forked Hookshot
 0
Code Issues 125 Pull requests 205 Activity

perf(vk): add floats to storage buffers with the GPU's own atomic, not a compare-and-swap loop #2306

Closed
mooki000 wants to merge 3 commits from mooki000/atomic-float-add into main
pull from: mooki000/atomic-float-add
merge into: MagicBeanDev:main
Conversation 10 Commits 3 Files changed 13 +444 -10
mooki000 commented 2026-11-11 14:10:42 +00:00 (Migrated from github.com)

Compute programs that run every frame in Kakariko and on Death Mountain Trail add floats to storage buffers, and the translator makes each of those adds a compare-and-swap loop: on an RTX 4090 the busiest one takes 77-80 attempts per add on average, and with the GPU's own float add the frame's GPU time is 0.4-0.6 ms (about 7%) lower there. On NVIDIA's driver the add is now one instruction. On AMD and Intel it stays the loop by default, because the native add was tried in the game only on that RTX 4090. That default changes once someone posts a run on an RDNA 3/4 (or Arc) card in Kakariko and on Death Mountain Trail with RX_VK_NATIVE_FADD=1.

Refs #2336 (closed by the maintainer: tracked in this pull request).

This takes up the idea of yassine-milal's #886 (closed to keep the queue clear), written again on current main with three faults of that diff fixed; no code was copied from it.

What changed

  • vk_main.cpp: on NVIDIA's driver (VkPhysicalDeviceDriverProperties' driverID, asked before the extensions are chosen; NVK does not count), the device asks for VK_EXT_shader_atomic_float and turns on only shaderBufferFloat32AtomicAdd, the one feature used. RX_VK_NATIVE_FADD=1 asks for it on any GPU that reports the feature, and =0 keeps the loop everywhere without enabling the extension. The switch follows #2042's rule, so on, yes or true mean on and 0, false, off, no or empty mean off (native_fadd_wanted). The log names the choice and why: vk: float atomic add on storage buffers: native (VK_EXT_shader_atomic_float), NVIDIA's driver, or compare-exchange loop, by default only with NVIDIA's driver (RX_VK_NATIVE_FADD=1: native where the GPU has it), or compare-exchange loop, RX_VK_NATIVE_FADD=0, or the feature that is missing.
  • vk_shader.cpp: the profile tells the translator; the program cache's stamp includes the answer, so a hookshot_programs.bin made with the native add is never read on a GPU without it (#886 keyed only the environment switch).
  • Eden's back end: with the feature, RED/ATOM.ADD.F32 on a storage buffer is one OpAtomicFAddEXT on an f32 view of the buffer, and no loop is made. The global-address fallback (#598) adds natively too, through the f32 views and the page table; the f32 views are searched with their bound length, as the u32 ones are (#886 skipped that check for them). Sirit gains OpAtomicFAddEXT.
  • Without the native add nothing changes: the compare-and-swap loop with its 1,024-attempt ceiling (#548) stays. That covers every GPU that is not on NVIDIA's driver unless the switch asks for it.
  • docs/development.md: the RX_VK_NATIVE_FADD=0 or 1 row, with the default.

Which GPUs report the feature, from the drivers' sources: NVIDIA (the RTX 4090 here reports shaderBufferFloat32AtomicAdd = true); AMD RDNA 3 and later on Mesa RADV (gfx_level >= GFX11) and RDNA 3 on AMD's own driver (PAL's Gfx11); Intel Arc and later on Mesa ANV (has_lsc). RDNA 1 and 2 (RX 5000, RX 6000, Steam Deck) and Intel's older integrated graphics have the extension without this feature and keep the loop either way. Of these, only NVIDIA's driver gets the native add by default. The AMD and Intel ones need RX_VK_NATIVE_FADD=1 until they have a run here (Joey-Magnus's review).

How it was tested

  • tests/test_native_float_atomic_add.py: cuts the storage-buffer setup and the float-add emitter out of the back end into a Sirit fixture, translates a compute shader with the feature, without it and without descriptor aliasing, and checks one OpAtomicFAddEXT and no compare-and-swap in the first, the loop in the others, spirv-val --target-env vulkan1.3 on all three; and that the device, the profile, the cache stamp and the fallback follow. It also compiles native_fadd_wanted alone against 28 cases: unset, NVIDIA's driver alone gets the native add (not RADV, AMD's, Intel's or NVK); set, the switch's value by #2042's rule on any driver. It fails on main. The decision fails it when native_fadd_wanted reads the switch with atoi as before (RX_VK_NATIVE_FADD=[on]: the loop) or wants the native add on every driver ([(unset)], driver 1: native).

  • On the head (rebased onto main a5b955fb), through tests/run_all.py --strict with CXX=g++ and with CXX=clang++: test_native_float_atomic_add, test_amd_subgroup_profiles, test_program_stamp_knobs, test_env_on, test_global_atomic_add_f32 and test_cas_loop_ceiling pass (6 of 6 each). Against main's source, test_native_float_atomic_add fails with both compilers.

  • python -I -B tests/run_all.py on the head (Arch Linux, 32 at once, at a load of 60 to 140 from other work on this PC): 435 passed, 3 failed, 20 skipped; main a5b955fb the same way: 436 passed, 1 failed, 20 skipped. The four checks that failed in one of the two runs (test_appimage_package_retry, test_bake_mimo, and test_recorder_sync and test_control by timeout) were run again alone on both trees: all four pass on the branch, and test_appimage_package_retry fails on main again. So they are timing under load, not this change.

  • clang++ -Wall -Wextra on the changed files against main: no new warning besides four missing field 'pNext' initializer on new structs (two for the driver query, two for afloat), the form vk_main.cpp already has 37 of.

  • A stress program with the translator's exact loop: invocations each add 1.0 to one float; lost adds are the count minus the sum (median of 5 runs).

    adds to one float RTX 4090: loop, adds lost RTX 4090: OpAtomicFAddEXT, adds lost RTX 4090: loop / native time Intel UHD 770: loop, adds lost
    1,024 0 0 0.200 / 0.003 ms 0
    4,096 2,068 0 1.37 / 0.007 ms 2,442
    16,384 13,450 0 6.61 / 0.022 ms 10,704
    65,536 62,884 0 28.1 / 0.084 ms 46,654

    (RTX 4090: NVIDIA 615.78.08; Intel UHD 770: Mesa 26.2.5, which has no buffer float add, so only the loop.) Up to 1,024 adds on one float the loop loses nothing; beyond that it does.

In the game

RTX 4090, NVIDIA 615.78.08, Arch Linux, no window (scripted input), the game's default output 2048x1152, Resolution 100%, Game speed per frame; this branch's build, with and without RX_VK_NATIVE_FADD=0, each prepared first; a save loaded and Link left standing; 150 s each, numbers from present 1800 on (the 60 fps cap holds both ways, so this is GPU time, not frame rate). These were measured on the first version of this branch, on main fab8cb3. On NVIDIA's driver the review's changes leave the same native path as the default, so they were not repeated. The two runs after the rebase are below the table.

per frame Kakariko, child, day: native / loop Death Mountain Trail, adult, night: native / loop
GPU time (the log's vk: GPU time, every 120 presents: median, range) 7.2 (7.1-7.5) / 7.8 (7.5-8.0) ms 5.4 (4.4-5.7) / 5.8 (4.6-6.0) ms
all timed GPU work (RX_VK_PASS_TIMING=2, mean of 12 reports of 600 presents) 9.05 / 9.57 ms 6.38 / 6.78 ms
ab7699ee7f536755's dispatch, once a frame under 0.065 ms (not among the 25 costliest spans in any report) / 0.153 ms (in all 12) under 0.051 ms / 0.147 ms
435a0138b9013040's dispatch, once a frame under 0.065 ms / 0.197 ms (in all 12) under 0.051 ms / 0.186 ms (in all 12)
game frames 59.3 / 59.3 fps 59.4 / 59.6 fps

The other four float-add dispatches (below) are cheaper: not among the 25 costliest spans either way, except 592079442d7129da once with the loop on Death Mountain Trail (0.059 ms).

  • Which programs add floats, and how hard their loops work (a local counting build, not part of this change, run with the loop for about five minutes in each place): six compute programs, all adding to storage buffers: 499dbfd0721d2858 (#816's sky light), ab7699ee7f536755 (#886's Kakariko pass), 435a0138b9013040, 592079442d7129da, aba5e6af8d4683eb (Kakariko only) and 6214a97bec4cc0bf. Of 111.7 million adds in the two places, none reached the 1,024-attempt ceiling; the busiest, ab7699ee7f536755 (about 3,000 adds a frame), took 77-80 attempts an add on average and 704 at most. So on this card no add is lost, and the gain is time. #886's fog program 7984161db3e053b5 runs every frame in both places but has no 32-bit float add; this change does not touch it.

  • Pictures of the same present both ways (native, loop, difference x4) show the same light, haze and sky; the differences are animation (grass, Navi, Link's idle pose), as a present number is not exactly the same game frame in two runs.

    Kakariko, present 4500: native, loop, difference x4
    Death Mountain Trail, present 3300: native, loop, difference x4

  • After the review and the rebase: two headless runs of the rebased build (head 7219b3fd on main a5b955fb, the same save in Kakariko, 360 s each, RX_VK_PRECOMPILE=0 so programs were made as met). Both reached Kakariko and ran to the end with no fatal error, no device loss and no kernel driver message. Their pictures show the same light and sky both ways. Other work kept this PC at a load of 60 to 165, so they say nothing about speed. The log line of each:

    1: Kakariko with the default on NVIDIA's driver, the native float add, at present 6000. 2: the same with RX_VK_NATIVE_FADD=0, the loop, at present 6000

  • Rebased once more on 2026-11-11, onto main 9df055f2, as 50579bab: the only conflict was this change's row in docs/development.md beside a new row of main's; the three commits' code and tests are unchanged (compared hunk by hunk). Rerun on that head: test_native_float_atomic_add, test_amd_subgroup_profiles, test_global_atomic_add_f32, test_cas_loop_ceiling and the program cache's four checks pass through tests/run_all.py --strict, and test_native_float_atomic_add.py fails on main's tree. The whole suite and the two game runs above are from 7219b3fd on a5b955fb.

    vk: float atomic add on storage buffers: native (VK_EXT_shader_atomic_float), NVIDIA's driver
vk: float atomic add on storage buffers: compare-exchange loop, RX_VK_NATIVE_FADD=0

Not tested

  • AMD and Intel: the native add is off there by default until someone posts an RDNA 3/4 (or Arc) run in Kakariko and on Death Mountain Trail; with RX_VK_NATIVE_FADD=1 it has not been run on either (which of them report the feature is from the drivers' sources above). Windows with NVIDIA is now tested by @nullifyshot (RTX 5080, NVIDIA 621.14, Windows 11, headless): child, Kakariko and Death Mountain Trail and adult, Death Mountain Trail; the default picks the native add there too, about 4 % less GPU time. The team's test request T-54W3 asks for an AMD RX 7000 or newer or an Intel Arc. Still not tested: adult at night on Windows; NVK; the Steam Deck (keeps the loop either way). Frame rates: the RTX 4090 holds the 60 fps cap both ways here, so only GPU time was compared. #886's thin fog and 20-53 ms passes (Quadro P4000, Windows, v2.0.2) did not reproduce on this card: no add was dropped and the pictures match.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 [email protected]

Compute programs that run every frame in Kakariko and on Death Mountain Trail add floats to storage buffers, and the translator makes each of those adds a compare-and-swap loop: on an RTX 4090 the busiest one takes 77-80 attempts per add on average, and with the GPU's own float add the frame's GPU time is 0.4-0.6 ms (about 7%) lower there. On NVIDIA's driver the add is now one instruction. On AMD and Intel it stays the loop by default, because the native add was tried in the game only on that RTX 4090. That default changes once someone posts a run on an RDNA 3/4 (or Arc) card in Kakariko and on Death Mountain Trail with `RX_VK_NATIVE_FADD=1`. Refs #2336 (closed by the maintainer: tracked in this pull request). This takes up the idea of yassine-milal's #886 (closed to keep the queue clear), written again on current `main` with three faults of that diff fixed; no code was copied from it. ## What changed - `vk_main.cpp`: on NVIDIA's driver (`VkPhysicalDeviceDriverProperties`' `driverID`, asked before the extensions are chosen; NVK does not count), the device asks for `VK_EXT_shader_atomic_float` and turns on only `shaderBufferFloat32AtomicAdd`, the one feature used. `RX_VK_NATIVE_FADD=1` asks for it on any GPU that reports the feature, and `=0` keeps the loop everywhere without enabling the extension. The switch follows #2042's rule, so `on`, `yes` or `true` mean on and `0`, `false`, `off`, `no` or empty mean off (`native_fadd_wanted`). The log names the choice and why: `vk: float atomic add on storage buffers: native (VK_EXT_shader_atomic_float), NVIDIA's driver`, or `compare-exchange loop, by default only with NVIDIA's driver (RX_VK_NATIVE_FADD=1: native where the GPU has it)`, or `compare-exchange loop, RX_VK_NATIVE_FADD=0`, or the feature that is missing. - `vk_shader.cpp`: the profile tells the translator; the program cache's stamp includes the answer, so a `hookshot_programs.bin` made with the native add is never read on a GPU without it (#886 keyed only the environment switch). - Eden's back end: with the feature, `RED/ATOM.ADD.F32` on a storage buffer is one `OpAtomicFAddEXT` on an f32 view of the buffer, and no loop is made. The global-address fallback (#598) adds natively too, through the f32 views and the page table; the f32 views are searched with their bound length, as the u32 ones are (#886 skipped that check for them). Sirit gains `OpAtomicFAddEXT`. - Without the native add nothing changes: the compare-and-swap loop with its 1,024-attempt ceiling (#548) stays. That covers every GPU that is not on NVIDIA's driver unless the switch asks for it. - `docs/development.md`: the `RX_VK_NATIVE_FADD=0` or `1` row, with the default. Which GPUs report the feature, from the drivers' sources: NVIDIA (the RTX 4090 here reports `shaderBufferFloat32AtomicAdd = true`); AMD RDNA 3 and later on Mesa RADV (`gfx_level >= GFX11`) and RDNA 3 on AMD's own driver (PAL's Gfx11); Intel Arc and later on Mesa ANV (`has_lsc`). RDNA 1 and 2 (RX 5000, RX 6000, Steam Deck) and Intel's older integrated graphics have the extension without this feature and keep the loop either way. Of these, only NVIDIA's driver gets the native add by default. The AMD and Intel ones need `RX_VK_NATIVE_FADD=1` until they have a run here (Joey-Magnus's review). ## How it was tested - `tests/test_native_float_atomic_add.py`: cuts the storage-buffer setup and the float-add emitter out of the back end into a Sirit fixture, translates a compute shader with the feature, without it and without descriptor aliasing, and checks one `OpAtomicFAddEXT` and no compare-and-swap in the first, the loop in the others, `spirv-val --target-env vulkan1.3` on all three; and that the device, the profile, the cache stamp and the fallback follow. It also compiles `native_fadd_wanted` alone against 28 cases: unset, NVIDIA's driver alone gets the native add (not RADV, AMD's, Intel's or NVK); set, the switch's value by #2042's rule on any driver. It fails on `main`. The decision fails it when `native_fadd_wanted` reads the switch with `atoi` as before (`RX_VK_NATIVE_FADD=[on]: the loop`) or wants the native add on every driver (`[(unset)], driver 1: native`). - On the head (rebased onto `main` a5b955fb), through `tests/run_all.py --strict` with `CXX=g++` and with `CXX=clang++`: `test_native_float_atomic_add`, `test_amd_subgroup_profiles`, `test_program_stamp_knobs`, `test_env_on`, `test_global_atomic_add_f32` and `test_cas_loop_ceiling` pass (6 of 6 each). Against `main`'s source, `test_native_float_atomic_add` fails with both compilers. - `python -I -B tests/run_all.py` on the head (Arch Linux, 32 at once, at a load of 60 to 140 from other work on this PC): 435 passed, 3 failed, 20 skipped; `main` a5b955fb the same way: 436 passed, 1 failed, 20 skipped. The four checks that failed in one of the two runs (`test_appimage_package_retry`, `test_bake_mimo`, and `test_recorder_sync` and `test_control` by timeout) were run again alone on both trees: all four pass on the branch, and `test_appimage_package_retry` fails on `main` again. So they are timing under load, not this change. - `clang++ -Wall -Wextra` on the changed files against `main`: no new warning besides four `missing field 'pNext' initializer` on new structs (two for the driver query, two for `afloat`), the form `vk_main.cpp` already has 37 of. - A stress program with the translator's exact loop: invocations each add 1.0 to one float; lost adds are the count minus the sum (median of 5 runs). | adds to one float | RTX 4090: loop, adds lost | RTX 4090: `OpAtomicFAddEXT`, adds lost | RTX 4090: loop / native time | Intel UHD 770: loop, adds lost | |---|---|---|---|---| | 1,024 | 0 | 0 | 0.200 / 0.003 ms | 0 | | 4,096 | 2,068 | 0 | 1.37 / 0.007 ms | 2,442 | | 16,384 | 13,450 | 0 | 6.61 / 0.022 ms | 10,704 | | 65,536 | 62,884 | 0 | 28.1 / 0.084 ms | 46,654 | (RTX 4090: NVIDIA 615.78.08; Intel UHD 770: Mesa 26.2.5, which has no buffer float add, so only the loop.) Up to 1,024 adds on one float the loop loses nothing; beyond that it does. ## In the game RTX 4090, NVIDIA 615.78.08, Arch Linux, no window (scripted input), the game's default output 2048x1152, Resolution 100%, Game speed per frame; this branch's build, with and without `RX_VK_NATIVE_FADD=0`, each prepared first; a save loaded and Link left standing; 150 s each, numbers from present 1800 on (the 60 fps cap holds both ways, so this is GPU time, not frame rate). These were measured on the first version of this branch, on `main` fab8cb3. On NVIDIA's driver the review's changes leave the same native path as the default, so they were not repeated. The two runs after the rebase are below the table. | per frame | Kakariko, child, day: native / loop | Death Mountain Trail, adult, night: native / loop | |---|---|---| | GPU time (the log's `vk: GPU time`, every 120 presents: median, range) | **7.2** (7.1-7.5) / **7.8** (7.5-8.0) ms | **5.4** (4.4-5.7) / **5.8** (4.6-6.0) ms | | all timed GPU work (`RX_VK_PASS_TIMING=2`, mean of 12 reports of 600 presents) | 9.05 / 9.57 ms | 6.38 / 6.78 ms | | `ab7699ee7f536755`'s dispatch, once a frame | under 0.065 ms (not among the 25 costliest spans in any report) / 0.153 ms (in all 12) | under 0.051 ms / 0.147 ms | | `435a0138b9013040`'s dispatch, once a frame | under 0.065 ms / 0.197 ms (in all 12) | under 0.051 ms / 0.186 ms (in all 12) | | game frames | 59.3 / 59.3 fps | 59.4 / 59.6 fps | The other four float-add dispatches (below) are cheaper: not among the 25 costliest spans either way, except `592079442d7129da` once with the loop on Death Mountain Trail (0.059 ms). - Which programs add floats, and how hard their loops work (a local counting build, not part of this change, run with the loop for about five minutes in each place): six compute programs, all adding to storage buffers: `499dbfd0721d2858` (#816's sky light), `ab7699ee7f536755` (#886's Kakariko pass), `435a0138b9013040`, `592079442d7129da`, `aba5e6af8d4683eb` (Kakariko only) and `6214a97bec4cc0bf`. Of 111.7 million adds in the two places, none reached the 1,024-attempt ceiling; the busiest, `ab7699ee7f536755` (about 3,000 adds a frame), took 77-80 attempts an add on average and 704 at most. So on this card no add is lost, and the gain is time. #886's fog program `7984161db3e053b5` runs every frame in both places but has no 32-bit float add; this change does not touch it. - Pictures of the same present both ways (native, loop, difference x4) show the same light, haze and sky; the differences are animation (grass, Navi, Link's idle pose), as a present number is not exactly the same game frame in two runs. ![Kakariko, present 4500: native, loop, difference x4](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/70b6daa1af37b68393b13a47de41f0e4ff896ae1/atomic-add-kakariko-native-loop-diff.jpg) ![Death Mountain Trail, present 3300: native, loop, difference x4](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/70b6daa1af37b68393b13a47de41f0e4ff896ae1/atomic-add-death-mountain-trail-native-loop-diff.jpg) - After the review and the rebase: two headless runs of the rebased build (head 7219b3fd on `main` a5b955fb, the same save in Kakariko, 360 s each, `RX_VK_PRECOMPILE=0` so programs were made as met). Both reached Kakariko and ran to the end with no fatal error, no device loss and no kernel driver message. Their pictures show the same light and sky both ways. Other work kept this PC at a load of 60 to 165, so they say nothing about speed. The log line of each: ![1: Kakariko with the default on NVIDIA's driver, the native float add, at present 6000. 2: the same with RX_VK_NATIVE_FADD=0, the loop, at present 6000](https://gist.githubusercontent.com/mooki000/b86bd3f35969803fb032f71f035844f8/raw/38a2692029b2d8287b2fcd23a1817feda6c72362/hookshot-atomic-add-kakariko-nvidia-default-loop.jpg) - Rebased once more on 2026-11-11, onto `main` `9df055f2`, as `50579bab`: the only conflict was this change's row in `docs/development.md` beside a new row of main's; the three commits' code and tests are unchanged (compared hunk by hunk). Rerun on that head: `test_native_float_atomic_add`, `test_amd_subgroup_profiles`, `test_global_atomic_add_f32`, `test_cas_loop_ceiling` and the program cache's four checks pass through `tests/run_all.py --strict`, and `test_native_float_atomic_add.py` fails on `main`'s tree. The whole suite and the two game runs above are from `7219b3fd` on `a5b955fb`. ``` vk: float atomic add on storage buffers: native (VK_EXT_shader_atomic_float), NVIDIA's driver vk: float atomic add on storage buffers: compare-exchange loop, RX_VK_NATIVE_FADD=0 ``` ## Not tested - **AMD and Intel**: the native add is off there by default until someone posts an RDNA 3/4 (or Arc) run in Kakariko and on Death Mountain Trail; with `RX_VK_NATIVE_FADD=1` it has not been run on either (which of them report the feature is from the drivers' sources above). Windows with NVIDIA is now tested by @nullifyshot (RTX 5080, NVIDIA 621.14, Windows 11, headless): [child, Kakariko and Death Mountain Trail](https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/2306) and [adult, Death Mountain Trail](https://git.magicbean.dev/MagicBeanDev/Hookshot/pulls/2306); the default picks the native add there too, about 4 % less GPU time. The team's test request T-54W3 asks for an AMD RX 7000 or newer or an Intel Arc. Still not tested: adult at night on Windows; NVK; the Steam Deck (keeps the loop either way). Frame rates: the RTX 4090 holds the 60 fps cap both ways here, so only GPU time was compared. #886's thin fog and 20-53 ms passes (Quadro P4000, Windows, v2.0.2) did not reproduce on this card: no add was dropped and the pictures match. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
Joey-Magnus (Migrated from github.com) reviewed 2026-11-11 20:07:59 +00:00
Joey-Magnus (Migrated from github.com) left a comment

The change itself looks right to me, and the description is unusually thorough. What I checked in the source:

  • Device setup. Only shaderBufferFloat32AtomicAdd is enabled after the query, and the features struct is reset with its pNext kept (gpu/vk/vk_main.cpp:383-394). RX_VK_NATIVE_FADD=0 keeps the extension off entirely.
  • When the native path is chosen. native_f32_add is decided in DefineStorageBuffers, before its early return (spirv_emit_context.cpp:1827-1834). The constructor calls DefineStorageBuffers (line 585) before DefineGlobalMemoryFunctions (line 592), so the global fallback sees the same answer.
  • No dangling function. f32_add_cas is only skipped when native, and its only user is EmitStorageAtomicAddF32, which takes the native branch first.
  • The global fallback walks the F32 views with their bound length, as the U32 ones are walked. The PSB path takes the native add only at gp_level >= 3, as before.
  • The program-cache stamp includes the answer, so a native-add cache is never read on a GPU without the feature.

I ran test_native_float_atomic_add, test_global_atomic_add_f32, test_cas_loop_ceiling, test_amd_subgroup_profiles and the program_cache checks at the PR's head with --strict (spirv-val present), and all pass. On GitHub only the title check has run so far; the SPIR-V fixtures job will run the new test once a maintainer approves the workflows.

should fix (testing): by your own reading of the drivers, the native path turns on by default for AMD RDNA 3 and later, on RADV and on AMD's Windows driver. Those are RX 7000 and RX 9000, which docs/workflow.md puts in Tier 1 ("test here before merge"). It also turns on for Intel Arc (Tier 2). The only in-game runs are on an RTX 4090. The gain is about 0.5 ms of GPU time at a 60 fps cap, a polish-level win. A driver bug in float atomics on AMD, by contrast, would be a Tier 1 regression in the very compute passes (sky light, Kakariko) that are hard to tell apart visually. Please get one run on an RDNA 3/4 card (Linux RADV, ideally Windows too) in Kakariko and on Death Mountain Trail, with the log line and a picture, before this merges. #2202 shows there are RX 7900 XT testers around. If that can't happen soon, a smaller step is to enable it by default only on NVIDIA's driver for now (dev.props.vendorID/driverID) and widen it once AMD has a row in tested configurations.

minor: the branch conflicts with main in docs/tested-configurations.md, where both changes add rows at the top of the table. A rebase is enough. Keep your rows under any newer ones, since the table is newest first.

The change itself looks right to me, and the description is unusually thorough. What I checked in the source: - Device setup. Only `shaderBufferFloat32AtomicAdd` is enabled after the query, and the features struct is reset with its `pNext` kept (`gpu/vk/vk_main.cpp:383-394`). `RX_VK_NATIVE_FADD=0` keeps the extension off entirely. - When the native path is chosen. `native_f32_add` is decided in `DefineStorageBuffers`, before its early return (`spirv_emit_context.cpp:1827-1834`). The constructor calls `DefineStorageBuffers` (line 585) before `DefineGlobalMemoryFunctions` (line 592), so the global fallback sees the same answer. - No dangling function. `f32_add_cas` is only skipped when native, and its only user is `EmitStorageAtomicAddF32`, which takes the native branch first. - The global fallback walks the F32 views with their bound length, as the U32 ones are walked. The PSB path takes the native add only at `gp_level >= 3`, as before. - The program-cache stamp includes the answer, so a native-add cache is never read on a GPU without the feature. I ran `test_native_float_atomic_add`, `test_global_atomic_add_f32`, `test_cas_loop_ceiling`, `test_amd_subgroup_profiles` and the `program_cache` checks at the PR's head with `--strict` (spirv-val present), and all pass. On GitHub only the title check has run so far; the SPIR-V fixtures job will run the new test once a maintainer approves the workflows. **should fix (testing)**: by your own reading of the drivers, the native path turns on by default for AMD RDNA 3 and later, on RADV and on AMD's Windows driver. Those are RX 7000 and RX 9000, which `docs/workflow.md` puts in Tier 1 ("test here before merge"). It also turns on for Intel Arc (Tier 2). The only in-game runs are on an RTX 4090. The gain is about 0.5 ms of GPU time at a 60 fps cap, a polish-level win. A driver bug in float atomics on AMD, by contrast, would be a Tier 1 regression in the very compute passes (sky light, Kakariko) that are hard to tell apart visually. Please get one run on an RDNA 3/4 card (Linux RADV, ideally Windows too) in Kakariko and on Death Mountain Trail, with the log line and a picture, before this merges. #2202 shows there are RX 7900 XT testers around. If that can't happen soon, a smaller step is to enable it by default only on NVIDIA's driver for now (`dev.props.vendorID`/`driverID`) and widen it once AMD has a row in tested configurations. **minor**: the branch conflicts with main in `docs/tested-configurations.md`, where both changes add rows at the top of the table. A rebase is enough. Keep your rows under any newer ones, since the table is newest first.
Joey-Magnus (Migrated from github.com) reviewed 2026-11-11 21:26:22 +00:00
Joey-Magnus (Migrated from github.com) left a comment

The code is right: the feature is enabled only when queried and is in the stamp, and test_amd_subgroup_profiles, test_global_atomic_add_f32 and test_cas_loop_ceiling pass. Remaining:

  • The conflict is only docs/tested-configurations.md.
  • The native path is on by default for RDNA 3/4 and Arc, but only tested on an RTX 4090. Either get an RDNA 3/4 run or limit it to NVIDIA for now.
  • RX_VK_NATIVE_FADD uses atoi, so =on turns it off. Use env_value_off.
  • tests/test_amd_subgroup_profiles.py conflicts with #2343, where both add a dev field. Keep both.
The code is right: the feature is enabled only when queried and is in the stamp, and `test_amd_subgroup_profiles`, `test_global_atomic_add_f32` and `test_cas_loop_ceiling` pass. Remaining: - The conflict is only `docs/tested-configurations.md`. - The native path is on by default for RDNA 3/4 and Arc, but only tested on an RTX 4090. Either get an RDNA 3/4 run or limit it to NVIDIA for now. - `RX_VK_NATIVE_FADD` uses `atoi`, so `=on` turns it off. Use `env_value_off`. - `tests/test_amd_subgroup_profiles.py` conflicts with #2343, where both add a `dev` field. Keep both.
hckrman101 commented 2026-11-11 22:55:47 +00:00 (Migrated from github.com)

This now conflicts with main in docs/development.md, docs/tested-configurations.md, tests/test_amd_subgroup_profiles.py. Could you merge main into the branch (or rebase) when you get a chance? I'll take another look once it's up to date.

This now conflicts with main in `docs/development.md`, `docs/tested-configurations.md`, `tests/test_amd_subgroup_profiles.py`. Could you merge main into the branch (or rebase) when you get a chance? I'll take another look once it's up to date.
mooki000 commented 2026-11-11 23:50:41 +00:00 (Migrated from github.com)

Thanks for both reviews, and for running the fixtures with --strict. I took the smaller step, since I have no AMD card here:

  • NVIDIA only by default (50579bab): with RX_VK_NATIVE_FADD unset, the native add is asked for only on NVIDIA's driver (VkPhysicalDeviceDriverProperties' driverID, read before the extensions are chosen, so on AMD, Intel and NVK the extension is not even enabled and nothing changes from main). RX_VK_NATIVE_FADD=1 asks for it on any GPU that reports shaderBufferFloat32AtomicAdd, =0 keeps the loop everywhere. The log line names the choice and why, and the program cache's stamp holds the answer as before. I'll widen the default once someone posts an RDNA 3/4 (or Arc) run in Kakariko and on Death Mountain Trail with RX_VK_NATIVE_FADD=1.
  • atoi (48654c61): the switch is read with env_value_off now, so =on asks for the native add and 0, false, off, no or empty keep the loop. test_native_float_atomic_add.py compiles the decision alone against 28 cases (values by drivers).
  • Conflicts (a63d57e9, the first commit rebased): rebased onto main, first a5b955fb and then 9df055f2 (the second time only this change's row in docs/development.md conflicted). Our rows in docs/tested-configurations.md are under the newer ones, and test_amd_subgroup_profiles.py keeps both dev fields (#2343's vk12 and float_atomic_add). #2323 renamed disk_stamp to translator_stamp, so the test's stamp check follows that name and checks that program_cache_stamp still uses it.

Rerun on the new head: the branch's tests pass with g++ and clang++ under run_all.py --strict, and test_native_float_atomic_add.py still fails against main's source; in the whole of tests/run_all.py, the branch fails only checks that also fail on main under this PC's load (run again alone, they all pass on the branch). Two headless runs of the rebased build in Kakariko on the RTX 4090 both reached the village without a fault. The default one logged native (VK_EXT_shader_atomic_float), NVIDIA's driver and the one with RX_VK_NATIVE_FADD=0 logged compare-exchange loop, RX_VK_NATIVE_FADD=0. The GPU-time numbers in the description are still from the first version; they were not measured again.

The whole suite and the two game runs were made on the first rebase (7219b3fd on a5b955fb); on the head that is pushed, 50579bab, I reran the branch's tests and the program cache's checks, and the three commits' code is unchanged from that rebase.

Thanks for both reviews, and for running the fixtures with `--strict`. I took the smaller step, since I have no AMD card here: - **NVIDIA only by default** (50579bab): with `RX_VK_NATIVE_FADD` unset, the native add is asked for only on NVIDIA's driver (`VkPhysicalDeviceDriverProperties`' `driverID`, read before the extensions are chosen, so on AMD, Intel and NVK the extension is not even enabled and nothing changes from `main`). `RX_VK_NATIVE_FADD=1` asks for it on any GPU that reports `shaderBufferFloat32AtomicAdd`, `=0` keeps the loop everywhere. The log line names the choice and why, and the program cache's stamp holds the answer as before. I'll widen the default once someone posts an RDNA 3/4 (or Arc) run in Kakariko and on Death Mountain Trail with `RX_VK_NATIVE_FADD=1`. - **`atoi`** (48654c61): the switch is read with `env_value_off` now, so `=on` asks for the native add and `0`, `false`, `off`, `no` or empty keep the loop. `test_native_float_atomic_add.py` compiles the decision alone against 28 cases (values by drivers). - **Conflicts** (a63d57e9, the first commit rebased): rebased onto `main`, first a5b955fb and then 9df055f2 (the second time only this change's row in `docs/development.md` conflicted). Our rows in `docs/tested-configurations.md` are under the newer ones, and `test_amd_subgroup_profiles.py` keeps both `dev` fields (#2343's `vk12` and `float_atomic_add`). #2323 renamed `disk_stamp` to `translator_stamp`, so the test's stamp check follows that name and checks that `program_cache_stamp` still uses it. Rerun on the new head: the branch's tests pass with g++ and clang++ under `run_all.py --strict`, and `test_native_float_atomic_add.py` still fails against `main`'s source; in the whole of `tests/run_all.py`, the branch fails only checks that also fail on `main` under this PC's load (run again alone, they all pass on the branch). Two headless runs of the rebased build in Kakariko on the RTX 4090 both reached the village without a fault. The default one logged `native (VK_EXT_shader_atomic_float), NVIDIA's driver` and the one with `RX_VK_NATIVE_FADD=0` logged `compare-exchange loop, RX_VK_NATIVE_FADD=0`. The GPU-time numbers in the description are still from the first version; they were not measured again. The whole suite and the two game runs were made on the first rebase (7219b3fd on a5b955fb); on the head that is pushed, 50579bab, I reran the branch's tests and the program cache's checks, and the three commits' code is unchanged from that rebase.
github-sync commented 2026-11-12 06:27:14 +00:00
Owner

@nullifyshot commented on GitHub · 2026-11-12 06:23 UTC · original

Tested on Windows, which the description lists as not tested: RTX 5080 16 GB, NVIDIA 621.14, Windows 11 (build 26300), Ryzen 9 9900X3D. A build of this branch (50579bab) rebased locally onto main 87991bc (no conflicts), headless, the game's default output 2048x1152, Resolution 100%, the 60 fps cap with normal pacing; a save loaded, then teleported and Link left standing, 150 s in each place, vk: GPU time every 120 presents (median, range):

GPU time a frame Kakariko Village, child Death Mountain Trail, child
native add (the default here) 6.70 (6.5-7.5) ms, 75 reports 4.60 (4.4-5.2) ms, 75 reports
loop, RX_VK_NATIVE_FADD=0 7.00 (6.7-8.0) ms, 76 reports 4.80 (4.7-5.7) ms, 75 reports
  • The log says vk: float atomic add on storage buffers: native (VK_EXT_shader_atomic_float), NVIDIA's driver with the switch unset, and compare-exchange loop, RX_VK_NATIVE_FADD=0 with it, so the NVIDIA-only default picks the native add on Windows' driver too.
  • About 4 % less GPU time in both places, in line with your RTX 4090 numbers (7.2 / 7.8 and 5.4 / 5.8 ms).
  • Pictures of the same spot on Death Mountain Trail both ways show the same light, haze and sky; what differs is Navi and the idle animation.
  • tests/test_amd_subgroup_profiles.py passes here (llvm-mingw); tests/test_native_float_atomic_add.py skips on this PC (no spirv-val).
  • Not covered: adult Link and Death Mountain Trail at night (no adult save here), per-dispatch numbers with RX_VK_PASS_TIMING=2; one pair of runs each.
> [@nullifyshot](https://github.com/nullifyshot) commented on GitHub · 2026-11-12 06:23 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2306#issuecomment-6816405120) Tested on Windows, which the description lists as not tested: RTX 5080 16 GB, NVIDIA 621.14, Windows 11 (build 26300), Ryzen 9 9900X3D. A build of this branch (50579bab) rebased locally onto main 87991bc (no conflicts), headless, the game's default output 2048x1152, Resolution 100%, the 60 fps cap with normal pacing; a save loaded, then teleported and Link left standing, 150 s in each place, `vk: GPU time` every 120 presents (median, range): | GPU time a frame | Kakariko Village, child | Death Mountain Trail, child | |---|---|---| | native add (the default here) | **6.70** (6.5-7.5) ms, 75 reports | **4.60** (4.4-5.2) ms, 75 reports | | loop, `RX_VK_NATIVE_FADD=0` | **7.00** (6.7-8.0) ms, 76 reports | **4.80** (4.7-5.7) ms, 75 reports | - The log says `vk: float atomic add on storage buffers: native (VK_EXT_shader_atomic_float), NVIDIA's driver` with the switch unset, and `compare-exchange loop, RX_VK_NATIVE_FADD=0` with it, so the NVIDIA-only default picks the native add on Windows' driver too. - About 4 % less GPU time in both places, in line with your RTX 4090 numbers (7.2 / 7.8 and 5.4 / 5.8 ms). - Pictures of the same spot on Death Mountain Trail both ways show the same light, haze and sky; what differs is Navi and the idle animation. - `tests/test_amd_subgroup_profiles.py` passes here (llvm-mingw); `tests/test_native_float_atomic_add.py` skips on this PC (no spirv-val). - Not covered: adult Link and Death Mountain Trail at night (no adult save here), per-dispatch numbers with `RX_VK_PASS_TIMING=2`; one pair of runs each. <!-- github-sync:c:6816405120 -->
github-sync commented 2026-11-12 07:27:50 +00:00
Owner

@hckrman101 commented on GitHub · 2026-11-12 07:15 UTC · original

Let's get someone to test on AMD please

> [@hckrman101](https://github.com/hckrman101) commented on GitHub · 2026-11-12 07:15 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2306#issuecomment-6817306026) Let's get someone to test on AMD please <!-- github-sync:c:6817306026 -->
github-sync commented 2026-11-12 07:27:50 +00:00
Owner

@mooki000 commented on GitHub · 2026-11-12 07:16 UTC · original

@hckrman101 yes. This PC cannot do it (an RTX 4090, and an Intel UHD 770 whose Mesa has no buffer float add), so here is what an AMD run needs, for whoever has the card.

What this pull request does on AMD today: nothing by default. The native add is chosen only with NVIDIA's driver; on any other GPU the loop stays and the log says vk: float atomic add on storage buffers: compare-exchange loop, by default only with NVIDIA's driver (RX_VK_NATIVE_FADD=1: native where the GPU has it). So an AMD run answers one question: can the default include AMD.

Which cards: RDNA 3 or RDNA 4 (RX 7000 / 9000). RDNA 1 and 2 do not report shaderBufferFloat32AtomicAdd (from the drivers' sources, in the description), so there RX_VK_NATIVE_FADD=1 stays on the loop and says so.

The run (a build of this branch, or of main with it merged):

  1. Kakariko Village and Death Mountain Trail, Link standing still, about 150 s in each, default settings.
  2. Twice: once as it is (the loop on AMD), once with RX_VK_NATIVE_FADD=1.
  3. From each hookshot_log.txt: the vk: float atomic add on storage buffers: … line (with the switch it should read native (VK_EXT_shader_atomic_float), RX_VK_NATIVE_FADD=1), and the vk: GPU time … ms a frame lines after the first 30 s (one every 120 presents; their median is enough).
  4. A picture of the same spot both ways: the light, the haze and the sky are what these compute programs feed.

What decides it: the same picture and equal or less GPU time with the switch, and no device lost in the log, on Windows' driver and on Mesa RADV if both can be had; then the default can take AMD in a follow-up commit here. A wrong picture or a lost device, and it stays NVIDIA-only, which is what this pull request ships either way.

> [@mooki000](https://github.com/mooki000) commented on GitHub · 2026-11-12 07:16 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2306#issuecomment-6817324542) @hckrman101 yes. This PC cannot do it (an RTX 4090, and an Intel UHD 770 whose Mesa has no buffer float add), so here is what an AMD run needs, for whoever has the card. **What this pull request does on AMD today:** nothing by default. The native add is chosen only with NVIDIA's driver; on any other GPU the loop stays and the log says `vk: float atomic add on storage buffers: compare-exchange loop, by default only with NVIDIA's driver (RX_VK_NATIVE_FADD=1: native where the GPU has it)`. So an AMD run answers one question: can the default include AMD. **Which cards:** RDNA 3 or RDNA 4 (RX 7000 / 9000). RDNA 1 and 2 do not report `shaderBufferFloat32AtomicAdd` (from the drivers' sources, in the description), so there `RX_VK_NATIVE_FADD=1` stays on the loop and says so. **The run** (a build of this branch, or of main with it merged): 1. Kakariko Village and Death Mountain Trail, Link standing still, about 150 s in each, default settings. 2. Twice: once as it is (the loop on AMD), once with `RX_VK_NATIVE_FADD=1`. 3. From each `hookshot_log.txt`: the `vk: float atomic add on storage buffers: …` line (with the switch it should read `native (VK_EXT_shader_atomic_float), RX_VK_NATIVE_FADD=1`), and the `vk: GPU time … ms a frame` lines after the first 30 s (one every 120 presents; their median is enough). 4. A picture of the same spot both ways: the light, the haze and the sky are what these compute programs feed. **What decides it:** the same picture and equal or less GPU time with the switch, and no `device lost` in the log, on Windows' driver and on Mesa RADV if both can be had; then the default can take AMD in a follow-up commit here. A wrong picture or a lost device, and it stays NVIDIA-only, which is what this pull request ships either way. <!-- github-sync:c:6817324542 -->
github-sync commented 2026-11-12 08:31:10 +00:00
Owner

@nullifyshot commented on GitHub · 2026-11-12 07:30 UTC · original

Adult numbers, as promised: RTX 5080, Windows 11. Same build as before but rebased locally onto main c544828 (no conflicts), headless, the 60 fps cap with normal pacing. Loaded the save on Death Mountain Trail, switched to adult Link with the bridge's new age adult command (#2694), let it settle 10 s, then left Link standing 150 s; vk: GPU time every 120 presents:

GPU time a frame Death Mountain Trail, adult
native add (the default here) 5.40 (4.5-6.3) ms, 75 reports
loop, RX_VK_NATIVE_FADD=0 5.60 (4.8-6.6) ms, 75 reports
  • About 4 % less GPU time with the native add, the same gain as the child on this trail (4.60 vs 4.80 ms in my comment above); the adult scene costs about 0.8 ms more either way.
  • The log confirms the RTX 5080 and the path each run took (native with the switch unset, the compare-exchange loop with it set to 0).
  • One run each; night not covered.
> [@nullifyshot](https://github.com/nullifyshot) commented on GitHub · 2026-11-12 07:30 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2306#issuecomment-6817565938) Adult numbers, as promised: RTX 5080, Windows 11. Same build as before but rebased locally onto main c544828 (no conflicts), headless, the 60 fps cap with normal pacing. Loaded the save on Death Mountain Trail, switched to adult Link with the bridge's new `age adult` command (#2694), let it settle 10 s, then left Link standing 150 s; `vk: GPU time` every 120 presents: | GPU time a frame | Death Mountain Trail, adult | |---|---| | native add (the default here) | **5.40** (4.5-6.3) ms, 75 reports | | loop, `RX_VK_NATIVE_FADD=0` | **5.60** (4.8-6.6) ms, 75 reports | - About 4 % less GPU time with the native add, the same gain as the child on this trail (4.60 vs 4.80 ms in my comment above); the adult scene costs about 0.8 ms more either way. - The log confirms the RTX 5080 and the path each run took (native with the switch unset, the compare-exchange loop with it set to 0). - One run each; night not covered. <!-- github-sync:c:6817565938 -->
github-sync commented 2026-11-12 13:39:18 +00:00
Owner

@mooki000 commented on GitHub · 2026-11-12 12:45 UTC · original

Thanks @nullifyshot for both Windows RTX 5080 checks: the child runs in Kakariko and on Death Mountain Trail, then the adult follow-up on the trail. Your logs confirm the NVIDIA default selects the native add on Windows too, with lower GPU time in those comparisons; adult night and the AMD/Intel native paths remain untested.

> [@mooki000](https://github.com/mooki000) commented on GitHub · 2026-11-12 12:45 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2306#issuecomment-6823979497) Thanks @nullifyshot for both Windows RTX 5080 checks: the child runs in Kakariko and on Death Mountain Trail, then the adult follow-up on the trail. Your logs confirm the NVIDIA default selects the native add on Windows too, with lower GPU time in those comparisons; adult night and the AMD/Intel native paths remain untested. <!-- github-sync:c:6823979497 -->
github-sync commented 2026-11-12 14:42:18 +00:00
Owner

@Joey-Magnus approved on GitHub · 2026-11-12 14:32 UTC · original

Everything from my last review is fixed. With RX_VK_NATIVE_FADD unset, only NVIDIA's driver asks for the extension (native_fadd_wanted, vk_main.cpp:249). AMD, Intel and NVK don't even enable it, so nothing changes for them. The switch is read with env_value_off. The conflicts are resolved, keeping #2343's vk12 field in test_amd_subgroup_profiles.py. nullifyshot's Windows RTX 5080 runs, child and adult in Kakariko and on Death Mountain Trail, show the same picture and about 4% less GPU time. That matches your RTX 4090 numbers.

I merged it into current main and ran test_native_float_atomic_add, test_global_atomic_add_f32, test_cas_loop_ceiling, test_amd_subgroup_profiles, the program cache and stamp checks, test_env_on and the SPIR-V checks with spirv-val present. All 15 pass. It also merges cleanly together with #1975.

One thing for the maintainers: translator_stamp now always adds the flag, so every player re-translates once, not only NVIDIA players. #1975 and #2309 also force a re-translation, so shipping them in the same release means one re-prepare. Widening the default to RDNA 3/4 can wait for an AMD run.

> [@Joey-Magnus](https://github.com/Joey-Magnus) approved on GitHub · 2026-11-12 14:32 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2306#pullrequestreview-5876208347) Everything from my last review is fixed. With `RX_VK_NATIVE_FADD` unset, only NVIDIA's driver asks for the extension (`native_fadd_wanted`, `vk_main.cpp:249`). AMD, Intel and NVK don't even enable it, so nothing changes for them. The switch is read with `env_value_off`. The conflicts are resolved, keeping #2343's `vk12` field in `test_amd_subgroup_profiles.py`. nullifyshot's Windows RTX 5080 runs, child and adult in Kakariko and on Death Mountain Trail, show the same picture and about 4% less GPU time. That matches your RTX 4090 numbers. I merged it into current main and ran `test_native_float_atomic_add`, `test_global_atomic_add_f32`, `test_cas_loop_ceiling`, `test_amd_subgroup_profiles`, the program cache and stamp checks, `test_env_on` and the SPIR-V checks with spirv-val present. All 15 pass. It also merges cleanly together with #1975. One thing for the maintainers: `translator_stamp` now always adds the flag, so every player re-translates once, not only NVIDIA players. #1975 and #2309 also force a re-translation, so shipping them in the same release means one re-prepare. Widening the default to RDNA 3/4 can wait for an AMD run. <!-- github-sync:r:5876208347 -->
github-sync closed this pull request 2026-11-12 16:54:59 +00:00
Commenting is not possible because the repository is archived.
Reviewers
No reviewers
Labels
Clear labels   
accessibility
Barrier affecting people with disabilities

  
autorelease: pending

   
autorelease: tagged

   
bug
Something isn't working

  
dependencies
Pull requests that update a dependency file

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
github_actions
Pull requests that update GitHub Actions code

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
testing wanted

   
wontfix
This will not be worked on

No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Hookshot!2306
No description provided.