perf(video): extend lazy pool RGBA conversion to planar YUV420P and add x86_64 SSE2 NV12 interleave #2114

github-sync · 2026-11-11T03:36:34Z

Cloudiny commented

2026-11-11 03:36:34 +00:00

(Migrated from github.com)

Summary

Commit 61cbb67 introduced lazy NV12-to-RGBA conversion (pool_rgba() / o->rgba_stale), but only skipped eager sws_scale() when f->format == AV_PIX_FMT_NV12 (Media Foundation on Windows). On Linux (and any CPU FFmpeg decode outputting AV_PIX_FMT_YUV420P / AV_PIX_FMT_YUVA420P), queue_frame() already constructs p.nv12, yet still ran full 1080p bilinear sws_scale(..., AV_PIX_FMT_RGBA) and allocated an unused 8 MiB p.rgba buffer on every frame.

This PR extends the lazy-RGBA optimization to planar 4:2:0 software decodes and accelerates UV packing:

Extend lazy RGBA to planar 4:2:0 (YUV420P / YUVA420P): Leaves p.rgba = NULL and avoids sws_scale() in queue_frame() whenever p.nv12 is populated, relying on upstream's pool_rgba(o) (o->rgba_stale) to convert on demand if ReadDecoderOutputImageRgba or a non-API RGBA memory pool needs RGBA. HOS_VIDEO_RGBA=1 forces eager conversion if ever needed for debugging.
x86_64 SSE2 YUV420P/YUVA420P -> NV12 UV interleave (#if defined(__x86_64__) || defined(_M_X64)): Replaces scalar per-pixel UV packing with _mm_unpacklo_epi8 / _mm_unpackhi_epi8 (16 UV pairs per iteration), keeping scalar fallback for tail pixels and non-x86_64 builds.
Avoid redundant calloc zero-fill: Allocates p.nv12 with malloc(nv12_size) (only zeroing on odd dimensions or unrecognized pixel formats).
Thread count: Bumps default software decode thread_count (AV1: 4 -> 8, others: 2 -> 4) for Linux/software 1080p60 AV1 cutscenes. Preserves HOS_VIDEO_NO_CATCHUP default behavior (0, catch-up enabled).

NVIDIA & Cross-Platform Compatibility

Works on both Windows (MSVC/MinGW x64) and Linux (<emmintrin.h> gated cleanly by __x86_64__ || _M_X64).
Reuses upstream pool_rgba() / rgba_stale cleanly for full RGBA pool compatibility on non-NV12 games.

Files Changed

host/nn_video.c

Test Plan / Verification

Verified clean git apply --check against main (ca082bf).
Syntax-checked host/nn_video.c (clang -fsyntax-only -std=gnu11 -Wall -Wextra -Wno-macro-redefined -Wno-unused-parameter): 0 errors, 0 warnings.

## Summary Commit `61cbb67` introduced lazy NV12-to-RGBA conversion (`pool_rgba()` / `o->rgba_stale`), but only skipped eager `sws_scale()` when `f->format == AV_PIX_FMT_NV12` (Media Foundation on Windows). On Linux (and any CPU FFmpeg decode outputting `AV_PIX_FMT_YUV420P` / `AV_PIX_FMT_YUVA420P`), `queue_frame()` already constructs `p.nv12`, yet still ran full 1080p bilinear `sws_scale(..., AV_PIX_FMT_RGBA)` and allocated an unused 8 MiB `p.rgba` buffer on every frame. This PR extends the lazy-RGBA optimization to planar 4:2:0 software decodes and accelerates UV packing: 1. **Extend lazy RGBA to planar 4:2:0 (`YUV420P` / `YUVA420P`):** Leaves `p.rgba = NULL` and avoids `sws_scale()` in `queue_frame()` whenever `p.nv12` is populated, relying on upstream's `pool_rgba(o)` (`o->rgba_stale`) to convert on demand if `ReadDecoderOutputImageRgba` or a non-API RGBA memory pool needs RGBA. `HOS_VIDEO_RGBA=1` forces eager conversion if ever needed for debugging. 2. **x86_64 SSE2 `YUV420P`/`YUVA420P` -> `NV12` UV interleave (`#if defined(__x86_64__) || defined(_M_X64)`):** Replaces scalar per-pixel UV packing with `_mm_unpacklo_epi8` / `_mm_unpackhi_epi8` (16 UV pairs per iteration), keeping scalar fallback for tail pixels and non-x86_64 builds. 3. **Avoid redundant `calloc` zero-fill:** Allocates `p.nv12` with `malloc(nv12_size)` (only zeroing on odd dimensions or unrecognized pixel formats). 4. **Thread count:** Bumps default software decode `thread_count` (`AV1`: 4 -> 8, others: 2 -> 4) for Linux/software 1080p60 AV1 cutscenes. Preserves `HOS_VIDEO_NO_CATCHUP` default behavior (`0`, catch-up enabled). ## NVIDIA & Cross-Platform Compatibility - Works on both Windows (MSVC/MinGW x64) and Linux (`<emmintrin.h>` gated cleanly by `__x86_64__ || _M_X64`). - Reuses upstream `pool_rgba()` / `rgba_stale` cleanly for full RGBA pool compatibility on non-NV12 games. ## Files Changed - `host/nn_video.c` ## Test Plan / Verification - [x] Verified clean `git apply --check` against `main` (`ca082bf`). - [x] Syntax-checked `host/nn_video.c` (`clang -fsyntax-only -std=gnu11 -Wall -Wextra -Wno-macro-redefined -Wno-unused-parameter`): 0 errors, 0 warnings.

hckrman101 (Migrated from github.com) reviewed

2026-11-11 18:25:01 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks, the core idea checks out. On main, a dav1d/FFmpeg decode on Linux comes out as YUV420P, so queue_frame() still runs the full 1080p sws_scale to RGBA and allocates 8 MiB per frame, even though OoT only reads the NV12. Leaving p.rgba NULL lets acquire set rgba_stale, and pool_rgba() converts later only if RGBA is asked for. It goes through the same SWS_BILINEAR sws with the same default matrix as the old eager path, so colours should not change. The SSE2 interleave is fine too: SSE2 is in the x86-64 baseline, so no runtime CPU check is needed. The loads stay inside the row (x + 16 <= w / 2), strides come from linesize, the scalar loop handles the tail, and AArch64 and other hosts just take the scalar loop. Since it's a plain byte interleave it can't drift from the scalar output. A small check comparing the two would still be cheap to add, but I'm not asking for it.

What I'd like before this goes in:

The thread_count bump (inline) is a separate change from the lazy RGBA. It has no numbers behind it and affects every machine that decodes movies in software, the low-core tier included. Could you split it out, or post before/after timings (HOS_VIDEO_LOG=1 shows how far the picture falls behind) on a 4-core machine as well as your own? Note that #1929 on main already stops the movie thread holding a virtual core while dav1d decodes, which was most of the Linux stall.
A run in the game. The test plan is git apply --check and -fsyntax-only. This changes what Linux movies go through, so please report one cutscene on Linux: it plays, the colours look the same, and it keeps up with the sound.
CI: only the title check has run on this head. Build and tests haven't run (they probably need a maintainer to approve the workflow for a first-time contributor).

Smaller points are inline (env_on for the new switch, and odd sizes). The branch is behind main (env_on, #1929's hos_vcore_away in nn_video.c), but the hunks don't overlap, so it should merge cleanly. One wording note: the description says MSVC, but the Windows build is llvm-mingw. The _M_X64 guard does no harm.

Thanks, the core idea checks out. On main, a dav1d/FFmpeg decode on Linux comes out as YUV420P, so `queue_frame()` still runs the full 1080p `sws_scale` to RGBA and allocates 8 MiB per frame, even though OoT only reads the NV12. Leaving `p.rgba` NULL lets acquire set `rgba_stale`, and `pool_rgba()` converts later only if RGBA is asked for. It goes through the same `SWS_BILINEAR` sws with the same default matrix as the old eager path, so colours should not change. The SSE2 interleave is fine too: SSE2 is in the x86-64 baseline, so no runtime CPU check is needed. The loads stay inside the row (`x + 16 <= w / 2`), strides come from `linesize`, the scalar loop handles the tail, and AArch64 and other hosts just take the scalar loop. Since it's a plain byte interleave it can't drift from the scalar output. A small check comparing the two would still be cheap to add, but I'm not asking for it. What I'd like before this goes in: - **The thread_count bump** (inline) is a separate change from the lazy RGBA. It has no numbers behind it and affects every machine that decodes movies in software, the low-core tier included. Could you split it out, or post before/after timings (`HOS_VIDEO_LOG=1` shows how far the picture falls behind) on a 4-core machine as well as your own? Note that #1929 on main already stops the movie thread holding a virtual core while dav1d decodes, which was most of the Linux stall. - **A run in the game.** The test plan is `git apply --check` and `-fsyntax-only`. This changes what Linux movies go through, so please report one cutscene on Linux: it plays, the colours look the same, and it keeps up with the sound. - **CI**: only the title check has run on this head. Build and tests haven't run (they probably need a maintainer to approve the workflow for a first-time contributor). Smaller points are inline (`env_on` for the new switch, and odd sizes). The branch is behind main (`env_on`, #1929's `hos_vcore_away` in `nn_video.c`), but the hunks don't overlap, so it should merge cleanly. One wording note: the description says MSVC, but the Windows build is llvm-mingw. The `_M_X64` guard does no harm.

host/nn_video.c

This doubles the decoder threads (AV1 4→8, others 2→4) and isn't related to the lazy RGBA. dav1d's frame delay grows with the thread count, so this also means more memory and more pictures decoded ahead. That may be fine, but it needs a measurement, ideally on a 4-core machine too, or a separate PR. Otherwise I'd keep 4/2 here.

host/nn_video.c

					
				@ -359,3 +367,4 @@

				    }

				    if ((w | h) & 1) memset(p.nv12, 0, nv12_size);

				    uint8_t *dst[4] = {p.rgba, NULL, NULL, NULL};

				    int dst_stride[4] = {w * 4, 0, 0, 0};

Main now has env_on() from hookshot_env.h (already included in this file on main), which is the one rule for these switches: 0/false/off/no mean off. Its header also says to read a switch once into a static, while this calls getenv + atoi on every frame. Something like static int force = -1; if (force < 0) force = env_on("HOS_VIDEO_RGBA");, as video_log() does, would match. Nit, but cheap.

Main now has `env_on()` from `hookshot_env.h` (already included in this file on main), which is the one rule for these switches: `0/false/off/no` mean off. Its header also says to read a switch once into a static, while this calls `getenv` + `atoi` on every frame. Something like `static int force = -1; if (force < 0) force = env_on("HOS_VIDEO_RGBA");`, as `video_log()` does, would match. Nit, but cheap.

host/nn_video.c

					
				@ -374,1 +390,4 @@

				                _mm_storeu_si128((__m128i *)(uv_row + 2 * x), _mm_unpacklo_epi8(u16, v16));

				                _mm_storeu_si128((__m128i *)(uv_row + 2 * x + 16), _mm_unpackhi_epi8(u16, v16));

				            }

				#endif

Nit, low reach (OoT's movies are 1080p): with lazy RGBA, an odd-sized YUV420P picture now gets its RGBA from this NV12, not from the source planes. The last chroma column (odd w) is dropped, and the last chroma row (odd h) isn't stored, while pool_rgba reads (h+1)/2 chroma rows. Zero chroma also shows as green, not grey. Either keep the eager sws path for odd sizes (lazy_rgba = ... && !((w | h) & 1)), or fill the chroma with 128 and round the size up. MF's NV12 path already behaves this way on main, so this isn't a blocker.

Nit, low reach (OoT's movies are 1080p): with lazy RGBA, an odd-sized YUV420P picture now gets its RGBA from this NV12, not from the source planes. The last chroma column (odd w) is dropped, and the last chroma row (odd h) isn't stored, while `pool_rgba` reads `(h+1)/2` chroma rows. Zero chroma also shows as green, not grey. Either keep the eager sws path for odd sizes (`lazy_rgba = ... && !((w | h) & 1)`), or fill the chroma with 128 and round the size up. MF's NV12 path already behaves this way on main, so this isn't a blocker.

Joey-Magnus (Migrated from github.com) reviewed

2026-11-11 21:24:25 +00:00

Joey-Magnus (Migrated from github.com) left a comment

The main idea is sound: software decodes produce YUV420P, and every reader on main handles a picture with no RGBA (pool_rgba(), nn_video.c:204-216 and the other call sites). The SSE2 loop is correct for unaligned data, odd widths and non-x86. Three things:

YUVA420P loses its alpha through the NV12 path. Keep it on the eager path, or drop it from lazy_rgba.
The decoder thread bump (AV1 4 to 8, others 2 to 4, ~nn_video.c:291) is unmeasured and can compete with the game on a 4-core Deck. Please drop it or measure it.
HOS_VIDEO_RGBA is undocumented and read with getenv/atoi every frame, against #2042's env_on convention.

The main idea is sound: software decodes produce YUV420P, and every reader on main handles a picture with no RGBA (`pool_rgba()`, `nn_video.c:204-216` and the other call sites). The SSE2 loop is correct for unaligned data, odd widths and non-x86. Three things: - `YUVA420P` loses its alpha through the NV12 path. Keep it on the eager path, or drop it from `lazy_rgba`. - The decoder thread bump (AV1 4 to 8, others 2 to 4, ~`nn_video.c:291`) is unmeasured and can compete with the game on a 4-core Deck. Please drop it or measure it. - `HOS_VIDEO_RGBA` is undocumented and read with `getenv`/`atoi` every frame, against #2042's `env_on` convention.

~~github-sync referenced this pull request 2026-11-12 06:22:54 +00:00~~

chore(main): release 2.2.0-rc.1 #2071

Commenting is not possible because the repository is archived.

No reviewers