fix(shaders): F2I.S32 and I2F.U32.RZ conversions, LDSM address copy (ported from Fritchii) #2443

Merged

hckrman101 merged 2 commits from port/fritchii-shaders into main

2026-11-11 20:52:16 +00:00

hckrman101 commented

2026-11-11 19:23:16 +00:00

(Migrated from github.com)

What changes

F2I.S32 from F32 (in about 30,000 of the game's 42,856 shader stages). The clamp bound was INT32_MAX as a float, which rounds up to 2^31. An input of 2^31 or more therefore reached OpConvertFToS out of range, and SPIR-V leaves that undefined. The clamp now stops at the largest float below 2^31, and INT32_MAX is selected for inputs at or above 2^31. NaN still gives 0. Ported from Fritchii eba45b151 by Frizti, which ports zortos293/Lichii#1051 by myshadowww.
I2FP.F32.U32 with RZ rounding (359 stages). The IR carried RZ, but OpConvertUToF rounds to nearest. Values of 2^24 or more could round up, for example 0xffffffff to 2^32 instead of 0x4f7fffff, so a hash times 2^-32 could reach 1.0. The conversion is now built from the top 24 significant bits, and is exact below 2^24. Ported from Fritchii f6a8f5132 by zortos293.
LDSM .x2/.x4 whose address register is one of its destinations (1 stage, an HMMA compute program). The lowering copied the address to R253, and the non-transposed path then wrote each matrix's shuffled row address to that same R253. Matrices 1-3 therefore read rows from matrix 0's address. The copy now goes to R254, which neither path uses. From Fritchii f130f0c50 by Duck912.
clip_distance_written was a static in a header, so each file had its own copy. The prologue's copy was never set, and translation threads wrote the other one at the same time. The prologue now zeroes the clip distances on purpose. The SPIR-V is unchanged. Ported from Fritchii ff0f6aa30 by Frizti (zortos293/Lichii#902).

The translator changes, so the program cache's stamp changes. Every player prepares shaders once again, before play, as after any translator update. Nothing is compiled during play.

How the fixes were chosen

I reviewed every commit on Fritchii's active branch since 2026-10-01 that touches the shader recompiler or the host shaders: 240 commits, Fritchii's own and the Lichii and Eglantine commits it carries. Each was matched against the corresponding code path in Hookshot's translator.

I then ran the Ampere-to-Maxwell lowering offline over all 42,856 of the game's stages, using my own dump and keeping everything local. Every stage lowers except the HMMA networks, which take the native matrix path at run time. That makes the roughly 120 commits that only add instruction forms for other games irrelevant here. The rest are either already in Hookshot or are latent: the game's shaders never reach them (for example IsGeneric stopping at Generic31X, or a fragment PrimitiveId needing the Geometry capability). These four are the ones the game's shaders do reach.

Verified

tests/f2i_immediate gains value checks. Literal inputs go through the real lowering, the Maxwell translation and the production IR passes. The remaining IR is then evaluated as SPIR-V defines each instruction, and any conversion that receives an out-of-range value is reported.
- With this PR, all 43 cases pass and every module passes spirv-val. F2I.S32 turns 2^31, 2^32, the largest float and +inf into 0x7fffffff, -2^31 and -inf into 0x80000000, NaN into 0, and -13.75 into -13. I2F.RZ turns 0xffffffff and 0xffffff80 into 0x4f7fffff, and 0x01000001 into 0x4b800000.
- On main, the same fixture fails 8 cases: the four F2I saturations hand OpConvertFToS an out-of-range value, and the four large I2F inputs round to nearest.
New gpu/vk/tests/ldsm_address_test.cpp, run by tests/run_all.py, lowers LDSM for every count, both layouts, and address registers inside and outside the destinations. It checks that every shuffle reads the address copy and that nothing overwrites the copy before the last shuffle. On main it fails every overlapping .x2/.x4 non-transposed case; with this PR it passes.
I compared the program cache before and after, for the same game and GPU (RTX 3090, Linux): 42,578 of 53,412 program records change their SPIR-V, almost all of them from the F2I bound. There are no new or missing records, and the prepare stays before play.
tests/run_all.py: 357 passed, 13 skipped. The 4 failures (test_wayland_libdecor_hint, test_nup_read_name, test_bake_portability, test_builder_log_read_only) fail the same way on main on this machine.
In game (RTX 3090, Linux, headless): Kokiri Forest and Lake Hylia, compared with main: no visible difference. Luma mean and contrast stay within the run-to-run spread (Kokiri contrast 42.5-44.0 against 41.0-42.3 on main, which is time of day and exposure).

Not tested

AMD and Intel GPUs. NVIDIA's conversions already saturate, so a visible difference is most likely on drivers whose compiler takes the undefined conversion literally.
The LDSM stage's network output on its own: I don't know which of the game's passes uses that HMMA program.

Some of the game's shaders hand the driver a conversion whose result SPIR-V leaves undefined, or round the wrong way, and one LDSM reads the wrong shared-memory rows. These are fixes Lichii and Fritchii found in the same Eden shader translator; I checked each one against the game's own shaders before porting it. ### What changes - **F2I.S32 from F32** (in about 30,000 of the game's 42,856 shader stages). The clamp bound was `INT32_MAX` as a float, which rounds up to 2^31. An input of 2^31 or more therefore reached `OpConvertFToS` out of range, and SPIR-V leaves that undefined. The clamp now stops at the largest float below 2^31, and `INT32_MAX` is selected for inputs at or above 2^31. NaN still gives 0. Ported from Fritchii [eba45b151](https://github.com/Frizti/Fritchii/commit/eba45b151a2672e47ae472282e54e25eba534d8c) by Frizti, which ports zortos293/Lichii#1051 by myshadowww. - **I2FP.F32.U32 with RZ rounding** (359 stages). The IR carried RZ, but `OpConvertUToF` rounds to nearest. Values of 2^24 or more could round up, for example 0xffffffff to 2^32 instead of 0x4f7fffff, so a hash times 2^-32 could reach 1.0. The conversion is now built from the top 24 significant bits, and is exact below 2^24. Ported from Fritchii [f6a8f5132](https://github.com/Frizti/Fritchii/commit/f6a8f513249754e2dd69a7f04efe5f8bb34ab756) by zortos293. - **LDSM .x2/.x4 whose address register is one of its destinations** (1 stage, an HMMA compute program). The lowering copied the address to R253, and the non-transposed path then wrote each matrix's shuffled row address to that same R253. Matrices 1-3 therefore read rows from matrix 0's address. The copy now goes to R254, which neither path uses. From Fritchii [f130f0c50](https://github.com/Frizti/Fritchii/commit/f130f0c50a383f577f3f965d42417e68433fc9be) by Duck912. - **`clip_distance_written`** was a `static` in a header, so each file had its own copy. The prologue's copy was never set, and translation threads wrote the other one at the same time. The prologue now zeroes the clip distances on purpose. The SPIR-V is unchanged. Ported from Fritchii [ff0f6aa30](https://github.com/Frizti/Fritchii/commit/ff0f6aa306c1f757e9022339270e51c0fb287968) by Frizti (zortos293/Lichii#902). The translator changes, so the program cache's stamp changes. Every player prepares shaders once again, before play, as after any translator update. Nothing is compiled during play. ### How the fixes were chosen I reviewed every commit on Fritchii's active branch since 2026-10-01 that touches the shader recompiler or the host shaders: 240 commits, Fritchii's own and the Lichii and Eglantine commits it carries. Each was matched against the corresponding code path in Hookshot's translator. I then ran the Ampere-to-Maxwell lowering offline over all 42,856 of the game's stages, using my own dump and keeping everything local. Every stage lowers except the HMMA networks, which take the native matrix path at run time. That makes the roughly 120 commits that only add instruction forms for other games irrelevant here. The rest are either already in Hookshot or are latent: the game's shaders never reach them (for example `IsGeneric` stopping at Generic31X, or a fragment PrimitiveId needing the Geometry capability). These four are the ones the game's shaders do reach. ### Verified - `tests/f2i_immediate` gains value checks. Literal inputs go through the real lowering, the Maxwell translation and the production IR passes. The remaining IR is then evaluated as SPIR-V defines each instruction, and any conversion that receives an out-of-range value is reported. - With this PR, all 43 cases pass and every module passes `spirv-val`. F2I.S32 turns 2^31, 2^32, the largest float and +inf into 0x7fffffff, -2^31 and -inf into 0x80000000, NaN into 0, and -13.75 into -13. I2F.RZ turns 0xffffffff and 0xffffff80 into 0x4f7fffff, and 0x01000001 into 0x4b800000. - On main, the same fixture fails 8 cases: the four F2I saturations hand `OpConvertFToS` an out-of-range value, and the four large I2F inputs round to nearest. - New `gpu/vk/tests/ldsm_address_test.cpp`, run by `tests/run_all.py`, lowers LDSM for every count, both layouts, and address registers inside and outside the destinations. It checks that every shuffle reads the address copy and that nothing overwrites the copy before the last shuffle. On main it fails every overlapping .x2/.x4 non-transposed case; with this PR it passes. - I compared the program cache before and after, for the same game and GPU (RTX 3090, Linux): 42,578 of 53,412 program records change their SPIR-V, almost all of them from the F2I bound. There are no new or missing records, and the prepare stays before play. - `tests/run_all.py`: 357 passed, 13 skipped. The 4 failures (`test_wayland_libdecor_hint`, `test_nup_read_name`, `test_bake_portability`, `test_builder_log_read_only`) fail the same way on main on this machine. - In game (RTX 3090, Linux, headless): Kokiri Forest and Lake Hylia, compared with main: no visible difference. Luma mean and contrast stay within the run-to-run spread (Kokiri contrast 42.5-44.0 against 41.0-42.3 on main, which is time of day and exposure). ![kokiri](https://raw.githubusercontent.com/hckrman101/Hookshot/3d4e96e58c2aa91c6a8536596e9a6cf8855d291e/fritchii/shaders-kokiri-main-vs-pr.png) ![lake](https://raw.githubusercontent.com/hckrman101/Hookshot/3d4e96e58c2aa91c6a8536596e9a6cf8855d291e/fritchii/shaders-lake-hylia-main-vs-pr.png) ### Not tested - AMD and Intel GPUs. NVIDIA's conversions already saturate, so a visible difference is most likely on drivers whose compiler takes the undefined conversion literally. - The LDSM stage's network output on its own: I don't know which of the game's passes uses that HMMA program.

Joey-Magnus (Migrated from github.com) approved these changes

2026-11-11 21:22:17 +00:00

Joey-Magnus (Migrated from github.com) left a comment

Checked each change against the source:

F2I.S32: 0x1.fffffep30f is 2^31-128, the largest float below 2^31. The Select compares op_a after abs/neg, which is the value converted; floats in [2^30, 2^31) are already integers, so rounding can't carry across the bound. The compare is ordered, so NaN skips it and still reaches the NaN-to-0 select.
I2F RZ: FindUMsb(0) is guarded, msb <= 23 converts the masked source exactly, and neg is applied after, which is right since RZ is symmetric.
LDSM: the non-transposed path uses R250-R253 and the transposed path R246-R252, so R254 is free, and dest + count > 246 keeps the address below the scratch range.
Clip distances: the prologue runs before the body (structured_control_flow.cpp:684), so the shader's own stores win. The SPIR-V really is unchanged and the cross-thread race goes away.

Follow-ups, not regressions: F2I.U32 from F32 has the same problem (static_cast<f32>(UINT32_MAX) is 2^32, out of range for OpConvertFToU, floating_point_conversion_integer.cpp:159), as does F32 to S64. RM/RP rounding and signed RZ still round to nearest, because the SPIR-V convert never reads fp_control.rounding.

Checked each change against the source: - F2I.S32: `0x1.fffffep30f` is 2^31-128, the largest float below 2^31. The Select compares `op_a` after abs/neg, which is the value converted; floats in [2^30, 2^31) are already integers, so rounding can't carry across the bound. The compare is ordered, so NaN skips it and still reaches the NaN-to-0 select. - I2F RZ: `FindUMsb(0)` is guarded, msb <= 23 converts the masked source exactly, and neg is applied after, which is right since RZ is symmetric. - LDSM: the non-transposed path uses R250-R253 and the transposed path R246-R252, so R254 is free, and `dest + count > 246` keeps the address below the scratch range. - Clip distances: the prologue runs before the body (`structured_control_flow.cpp:684`), so the shader's own stores win. The SPIR-V really is unchanged and the cross-thread race goes away. Follow-ups, not regressions: F2I.U32 from F32 has the same problem (`static_cast<f32>(UINT32_MAX)` is 2^32, out of range for `OpConvertFToU`, `floating_point_conversion_integer.cpp:159`), as does F32 to S64. RM/RP rounding and signed RZ still round to nearest, because the SPIR-V convert never reads `fp_control.rounding`.