perf(fpx): fuse single-precision FMA, FRECPS and FRSQRTS on x86-64-v3 #2383

Merged

corentinbonjour merged 1 commit from perf/fpx-fused-fma32 into main

2026-11-11 19:30:12 +00:00

corentinbonjour commented

2026-11-11 16:48:40 +00:00

(Migrated from github.com)

What changed

suyu's runtime header uses __builtin_fmaf in recomp_fpx_fma32, recomp_fpx_recps32 and recomp_fpx_rsqrts32 only when RECOMP_FPX_HOST == 1 (an AArch64 host). On x86-64-v3, where RECOMP_FPX_FMA64 is already set from __FMA__, they took the binary64 form with a midpoint test, and fell back to the exact body on every binary32 midpoint.

New nx2/suyu_rt_fixes.py wraps a64_rtgen.runtime_h, as lift_fixes does for the lifter, and changes those three guards to RECOMP_FPX_HOST == 1 || RECOMP_FPX_FMA64. Nothing under third_party/ is edited. Each substitution must match exactly once, so a refreshed a64-kit that moves these lines fails loudly. nx2/lift.py and nx2/test_leaf.py install it next to lift_fixes.

Builds at x86-64-v2, baseline x86-64, MSVC and wasm are unchanged: RECOMP_FPX_FMA64 is 0 there. An existing game picks the change up on its next lift (a new file under nx2/ changes the lift key).

Measurements

llvm-mingw clang 23 from the v2.0.14 builder, at the build's lifted-code flags (-march=x86-64-v3 -ffp-contract=off -fno-strict-aliasing -O2), headers from the real generators. Synthetic kernels shaped like the lifted call forms; thread CPU cycles per element (QueryThreadCycleTime), median of 31 samples, 7 interleaved before/after rounds:

Kernel	FZ off	FZ on
FMA throughput	4.05 -> 2.79 (-31%)	-31%
FMA dependent chain	19.43 -> 11.07 (-43%)	-46%
4x4 mat*vec FMLA	30.78 -> 18.00 (-42%)	-12% (noisy)
FRECPS + FMUL Newton x2	14.33 -> 10.16 (-29%)	-25%
FRSQRTS Newton x2	21.90 -> 16.88 (-23%)	-21%

The new header emits vfmadd/vfnmadd and no fmaf call.

The game-wide gain depends on how much of a frame runs these instructions, which needs an in-game frame-time comparison. I have not run one.

How it was tested

tests/fpx_sticky_test.c gets a fast mode: the inline fast paths against recomp_fpe_*, value and FPSR, under random FPCR/FPSR with a*b near 1, 2 or 3 as in the Newton steps, plus three binary32 midpoints that must be kept when the build has FMA. tests/test_fpx_sticky.py runs it with 8M cases in both the v3 and v2 builds: 0 mismatches. Without suyu_rt_fixes installed the v3 run fails (the midpoints fall back).
A separate fuzz of 200M cases per op (specials, subnormals, FZ on and off, cancellations, midpoints, Newton-like pairs): 0 mismatches. It catches sabotage: dropping the FZ check on z gave 85k mismatches in 2M.
python -I -B tests/test_fpx_sticky.py with HOOKSHOT_MINGW at the builder's llvm-mingw: passes.
python -I -B tests/run_all.py on Windows: the same failing set as untouched main at 32693f0, none new (the failures here come from tools missing on this PC). nx2/test_leaf.py needs the dump and was not run.

Notes

The only behaviour change is that binary32 midpoints now stay on the fast path, with the same result. #2324 moves the a64_rtgen.write call in nx2/lift.py; this change patches at import time, so it still applies, with at most a small conflict in the import block.

On x86-64-v3 builds the single-precision FMADD/FMLA, FRECPS and FRSQRTS fast paths still compute in binary64 and round twice, although the CPU has a hardware FMA. One fused instruction gives the same results and makes FMA-bound guest code 22 to 45% faster. ## What changed suyu's runtime header uses `__builtin_fmaf` in `recomp_fpx_fma32`, `recomp_fpx_recps32` and `recomp_fpx_rsqrts32` only when `RECOMP_FPX_HOST == 1` (an AArch64 host). On x86-64-v3, where `RECOMP_FPX_FMA64` is already set from `__FMA__`, they took the binary64 form with a midpoint test, and fell back to the exact body on every binary32 midpoint. New `nx2/suyu_rt_fixes.py` wraps `a64_rtgen.runtime_h`, as `lift_fixes` does for the lifter, and changes those three guards to `RECOMP_FPX_HOST == 1 || RECOMP_FPX_FMA64`. Nothing under `third_party/` is edited. Each substitution must match exactly once, so a refreshed a64-kit that moves these lines fails loudly. `nx2/lift.py` and `nx2/test_leaf.py` install it next to `lift_fixes`. Builds at x86-64-v2, baseline x86-64, MSVC and wasm are unchanged: `RECOMP_FPX_FMA64` is 0 there. An existing game picks the change up on its next lift (a new file under `nx2/` changes the lift key). ## Measurements llvm-mingw clang 23 from the v2.0.14 builder, at the build's lifted-code flags (`-march=x86-64-v3 -ffp-contract=off -fno-strict-aliasing -O2`), headers from the real generators. Synthetic kernels shaped like the lifted call forms; thread CPU cycles per element (`QueryThreadCycleTime`), median of 31 samples, 7 interleaved before/after rounds: | Kernel | FZ off | FZ on | |---|---|---| | FMA throughput | 4.05 -> 2.79 (-31%) | -31% | | FMA dependent chain | 19.43 -> 11.07 (-43%) | -46% | | 4x4 mat*vec FMLA | 30.78 -> 18.00 (-42%) | -12% (noisy) | | FRECPS + FMUL Newton x2 | 14.33 -> 10.16 (-29%) | -25% | | FRSQRTS Newton x2 | 21.90 -> 16.88 (-23%) | -21% | The new header emits `vfmadd`/`vfnmadd` and no `fmaf` call. The game-wide gain depends on how much of a frame runs these instructions, which needs an in-game frame-time comparison. I have not run one. ## How it was tested - `tests/fpx_sticky_test.c` gets a `fast` mode: the inline fast paths against `recomp_fpe_*`, value and FPSR, under random FPCR/FPSR with `a*b` near 1, 2 or 3 as in the Newton steps, plus three binary32 midpoints that must be kept when the build has FMA. `tests/test_fpx_sticky.py` runs it with 8M cases in both the v3 and v2 builds: 0 mismatches. Without `suyu_rt_fixes` installed the v3 run fails (the midpoints fall back). - A separate fuzz of 200M cases per op (specials, subnormals, FZ on and off, cancellations, midpoints, Newton-like pairs): 0 mismatches. It catches sabotage: dropping the FZ check on `z` gave 85k mismatches in 2M. - `python -I -B tests/test_fpx_sticky.py` with `HOOKSHOT_MINGW` at the builder's llvm-mingw: passes. - `python -I -B tests/run_all.py` on Windows: the same failing set as untouched `main` at 32693f0, none new (the failures here come from tools missing on this PC). `nx2/test_leaf.py` needs the dump and was not run. ## Notes The only behaviour change is that binary32 midpoints now stay on the fast path, with the same result. #2324 moves the `a64_rtgen.write` call in `nx2/lift.py`; this change patches at import time, so it still applies, with at most a small conflict in the import block.

hckrman101 (Migrated from github.com) approved these changes

2026-11-11 17:36:30 +00:00

hckrman101 (Migrated from github.com) left a comment

Looks right to me. I went through it against the Arm semantics:

The three #if guards now take the __builtin_fmaf branch that an AArch64 host already uses, so x86-64-v3 runs exactly the code path FPX1 was designed and proven for there. The keep conditions after it are unchanged: only a normal result (or a finite zero-product for FMADD) under an open gate (IXC already set, FPCR clean of RMode/DN/AHP/FZ16 and the inhibit bit) is kept, and FZ inputs and z are still screened. NaNs, infinities, tiny results and the FRECPS/FRSQRTS inf*0 case (NaN from the fma, never kept, exact body gives 2.0/1.5) all still fall back, so rounding mode, default-NaN, NaN propagation and flush-to-zero are all still the exact body's job.
FRSQRTS: fmaf(-a, b, 3) * 0.5f matches Arm's single rounding of (3 - a*b)/2 whenever the halved value is normal, which is the only case kept; overflow before halving gives inf, so it falls back.
Gating: RECOMP_FPX_FMA64 is only 1 for HOST 1, or HOST 2 with __FMA__ on GCC/clang, so x86-64-v2/baseline (host_march() on a CPU without FMA), MSVC and wasm keep the binary64 form. Since the game is built with the march of the PC it runs on, no FMA instruction can reach a CPU without one.
Same results on both platforms: CI ran test_fpx_sticky.py (with the new fast mode) and it passed on ubuntu-24.04 and windows-2022 (llvm-mingw). The midpoint cases I checked by hand (1+2^-12 squared, the FRECPS and FRSQRTS ones) really are binary32 midpoints, so the test does show the fused path is the one in use.
Cache: make_game.lift_key hashes every .py under nx2/, so the new file and the changed lift.py trigger a relift, and build.py's object key hashes the generated headers in gen/, so a64_suyu.h changing recompiles the chunks. No stale lifted code.
Patching a64_rtgen.runtime_h works because write() looks up runtime_h as a module global at call time, and the exactly-once substitution check is a good guard for a64-kit refreshes.

CI: the failing checks are test_vcore_fs_open / test_vcore_video_acquire (main fails them too right now) and the AppImage job timing out. None of that is from this PR.

What's missing is evidence from the game: one relift and run, to check it boots and plays normally, plus a before/after frame time from somewhere FMA-heavy, would back up the perf claim. That's not blocking given how much the test covers, but it would be good to have before merge. Nit: expect a small import-block conflict with #2324, as you noted.

Looks right to me. I went through it against the Arm semantics: - The three `#if` guards now take the `__builtin_fmaf` branch that an AArch64 host already uses, so x86-64-v3 runs exactly the code path FPX1 was designed and proven for there. The keep conditions after it are unchanged: only a normal result (or a finite zero-product for FMADD) under an open gate (IXC already set, FPCR clean of RMode/DN/AHP/FZ16 and the inhibit bit) is kept, and FZ inputs and `z` are still screened. NaNs, infinities, tiny results and the FRECPS/FRSQRTS inf*0 case (NaN from the fma, never kept, exact body gives 2.0/1.5) all still fall back, so rounding mode, default-NaN, NaN propagation and flush-to-zero are all still the exact body's job. - FRSQRTS: `fmaf(-a, b, 3) * 0.5f` matches Arm's single rounding of `(3 - a*b)/2` whenever the halved value is normal, which is the only case kept; overflow before halving gives inf, so it falls back. - Gating: `RECOMP_FPX_FMA64` is only 1 for HOST 1, or HOST 2 with `__FMA__` on GCC/clang, so x86-64-v2/baseline (`host_march()` on a CPU without FMA), MSVC and wasm keep the binary64 form. Since the game is built with the march of the PC it runs on, no FMA instruction can reach a CPU without one. - Same results on both platforms: CI ran `test_fpx_sticky.py` (with the new `fast` mode) and it passed on ubuntu-24.04 and windows-2022 (llvm-mingw). The midpoint cases I checked by hand (1+2^-12 squared, the FRECPS and FRSQRTS ones) really are binary32 midpoints, so the test does show the fused path is the one in use. - Cache: `make_game.lift_key` hashes every .py under nx2/, so the new file and the changed lift.py trigger a relift, and build.py's object key hashes the generated headers in gen/, so `a64_suyu.h` changing recompiles the chunks. No stale lifted code. - Patching `a64_rtgen.runtime_h` works because `write()` looks up `runtime_h` as a module global at call time, and the exactly-once substitution check is a good guard for a64-kit refreshes. CI: the failing checks are `test_vcore_fs_open` / `test_vcore_video_acquire` (main fails them too right now) and the AppImage job timing out. None of that is from this PR. What's missing is evidence from the game: one relift and run, to check it boots and plays normally, plus a before/after frame time from somewhere FMA-heavy, would back up the perf claim. That's not blocking given how much the test covers, but it would be good to have before merge. Nit: expect a small import-block conflict with #2324, as you noted.