feat(video): decode the AV1 movies with dav1d on Windows (#2316) #2515

github-sync · 2026-11-11T21:31:15Z

Joey-Magnus commented

2026-11-11 21:31:15 +00:00

(Migrated from github.com)

Movies stutter and fall behind their sound on Windows (#2316, now tracked in #2503; on an RTX 4060; a GTX 1050 in Discord). The Windows FFmpeg has no AV1 decoder, so the 1080p AV1 movies go to the AV1 Video Extension's software decoder on every GPU. This PR decodes them with dav1d instead, the same decoder Linux uses.

Why it was Media Foundation's software decoder everywhere

The FFmpeg in the Builder's cpm part (9.0.2) is configured with --disable-everything plus h264, vp8, vp9 and opus. It has no AV1 decoder of any kind, and not even the AV1 parser.
host/nn_video.c therefore hands AV1 to host/mf_av1.c. That file asks Media Foundation for its synchronous decoder first, which is the extension's software one; the hardware decoder is used only when no synchronous one exists. So an RTX 4060 decodes the movies in software too, and a PC without the extension gets the Store dialog (#380).
The catch-up skips pictures that are late, but it still has to decode them. A decoder slower than real time leaves the picture behind the sound whatever the catch-up does.

What changed

host/dav1d_av1.c wraps dav1d behind the same open, decode, flush and close interface as mf_av1.c, handing back NV12, which nn_video.c keeps as it is. It fetches one picture per submission: a second dav1d_get_picture with no data sent in between makes dav1d drain, which waits for every frame in flight and leaves its frame threads decoding one picture at a time. At the end of the stream it drains everything. HOS_DAV1D_THREADS=<n> sets the thread count (default: dav1d's, one per logical core). HOS_NO_DAV1D=1 goes back to Media Foundation.
host/nn_video.c turns "our own AV1 decoder" into a small table of functions: dav1d first when it is built in, then Media Foundation. Nothing changes when FFmpeg has its own AV1 decoder (Linux).
One log line per movie (always on): the decoder, the mean and worst decode time per submission, the time between pictures, and how many pictures the catch-up skipped. Its format: video: a movie of <n> pictures on <decoder>: <mean> ms to decode each (worst <ms>), one shown every <ms> ms; <n> skipped to keep up with its sound. A slower-than-real-time decoder is flagged "(slower than it plays)". The next #2316 log will show whether decoding is what falls behind.
scripts/windows/dav1d.sh builds dav1d with llvm-mingw from the same release and source-tree hash that scripts/linux/deps.sh pins (1.5.4; it reads them from there). It installs into the CPM layout <cpm>/dav1d/1.5.4/{include,lib/libdav1d.a,COPYING}. The output is reproducible: two builds give the same libdav1d.a (SHA-256 e6f1adec…7e98 with llvm-mingw 20261104). That took clang's -mno-incremental-linker-compatible and nasm's --reproducible; without them the COFF objects carry timestamps.
nx2/build.py compiles dav1d_av1.c and links libdav1d.a when the CPM cache has dav1d, on the llvm-mingw route only (it is a MinGW library). Without it, or on the Visual Studio route, nothing changes and Media Foundation decodes the movies as before.
CI: a new Linux job in windows-builder.yml builds dav1d with the pinned Linux-hosted llvm-mingw and runs the wrapper's test on the Windows build under Wine. It uploads the CPM folder as the dav1d-windows artifact. The Linux test job installs libdav1d-dev for the new test.

How dav1d reaches players

Players build the game on their own PC, so dav1d has to reach them in the Builder's cpm part. The cpm part is normally reused from the latest release. This PR makes the release workflow add dav1d to it: the package job takes the dav1d-windows artifact and runs release_parts.py package --cpm-add. If the latest release's cpm part lacks dav1d/1.5.4, the job unpacks that part into tools/cpm with dav1d added. make_setup_parts.py then makes a new cpm part, and the release job uploads it to builder-parts like any other new part. Once a release carries it, later runs reuse it. No manual step is needed.

I checked this against the real v2.1.0 setup list and the dav1d built here. The new part is HookshotBuilder-cpm-73d3c9f5a31d.zip (19.6 MB, up from 18.6 MB). It holds the 451 files of HookshotBuilder-cpm-c30c77e4cf00.zip byte-for-byte, plus dav1d's 8 files. Two runs gave the same SHA-256, because the zip and libdav1d.a are both reproducible.

Testing

tests/test_dav1d_av1.py (new) uses a 3.8 KB synthetic clip in tests/dav1d_av1/: 12 pictures of a formula pattern, encoded losslessly with libaom, so no game data. Every picture must come out in order, with its pts, as the exact NV12 bytes of the pattern. That includes the pictures dav1d holds back until the end, and a second pass after a flush. It runs at 1 and 4 threads, and checks that HOS_NO_DAV1D=1 leaves the decoder closed. It passes on Linux with clang, clang++, gcc and g++ against the system dav1d. It also passes for the Windows build (llvm-mingw, the script's libdav1d.a) under Wine at 1 thread, 4 threads and the default. Breaking the end-of-stream drain makes it fail.
tests/test_cpm_discovery.py: dav1d is used only on the llvm-mingw route, never on the Visual Studio route or Linux, and only with both its header and library present.
tests/test_release_cpm_add.py: --cpm-add adds dav1d to a cpm part that lacks it and keeps every file the part had. It leaves a part that already has dav1d to be reused as it is, and adds a new version beside an old one.
tests/test_vcore_video_acquire.py and tests/test_run_all_declarations.py were updated for the new type and tool.
python -I -B tests/run_all.py: 420 passed, 0 failed on Linux, rebased on current main.
host/nn_video.c compiles for Windows with llvm-mingw and the cpm part's own FFmpeg headers, with and without HOOKSHOT_DAV1D, and for Linux.
Speed, on a synthetic 1080p AV1 clip (SVT-AV1, 300 pictures). The Windows build under Wine, on a Ryzen 7 9800X3D: 2.8 ms per picture on 1 thread, 1.9 ms on 4, 1.4 ms on all 16. A 30 fps movie allows 33 ms per picture. The real movies are harder to decode than this clip.

Not tested

The game on Windows. I'm on Linux with no Windows install, so I need a Windows tester to build with dav1d in the CPM cache and play the intro movie. Ideally on a PC that stutters today (#2316's RTX 4060, or a GTX 10/16), with the new log line before and after.
No timing comparison against Media Foundation, since Wine has no AV1 Video Extension.
The new CI job runs for the first time on this PR.
10-bit and 4:4:4 conversion paths (OoT's movies are 8-bit 4:2:0).

No screenshots: nothing new shows on screen. The movies should just play in sync, and the extension's Store dialog no longer appears once dav1d is in the Builder.

Movies stutter and fall behind their sound on Windows ([#2316](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2316), now tracked in [#2503](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2503); on an RTX 4060; a GTX 1050 in Discord). The Windows FFmpeg has no AV1 decoder, so the 1080p AV1 movies go to the AV1 Video Extension's software decoder on every GPU. This PR decodes them with dav1d instead, the same decoder Linux uses. ### Why it was Media Foundation's software decoder everywhere - The FFmpeg in the Builder's cpm part (9.0.2) is configured with `--disable-everything` plus h264, vp8, vp9 and opus. It has no AV1 decoder of any kind, and not even the AV1 parser. - `host/nn_video.c` therefore hands AV1 to `host/mf_av1.c`. That file asks Media Foundation for its **synchronous** decoder first, which is the extension's software one; the hardware decoder is used only when no synchronous one exists. So an RTX 4060 decodes the movies in software too, and a PC without the extension gets the Store dialog ([#380](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/380)). - The catch-up skips pictures that are late, but it still has to decode them. A decoder slower than real time leaves the picture behind the sound whatever the catch-up does. ### What changed - **`host/dav1d_av1.c`** wraps dav1d behind the same open, decode, flush and close interface as `mf_av1.c`, handing back NV12, which `nn_video.c` keeps as it is. It fetches one picture per submission: a second `dav1d_get_picture` with no data sent in between makes dav1d drain, which waits for every frame in flight and leaves its frame threads decoding one picture at a time. At the end of the stream it drains everything. `HOS_DAV1D_THREADS=<n>` sets the thread count (default: dav1d's, one per logical core). `HOS_NO_DAV1D=1` goes back to Media Foundation. - **`host/nn_video.c`** turns "our own AV1 decoder" into a small table of functions: dav1d first when it is built in, then Media Foundation. Nothing changes when FFmpeg has its own AV1 decoder (Linux). - **One log line per movie** (always on): the decoder, the mean and worst decode time per submission, the time between pictures, and how many pictures the catch-up skipped. Its format: `video: a movie of <n> pictures on <decoder>: <mean> ms to decode each (worst <ms>), one shown every <ms> ms; <n> skipped to keep up with its sound`. A slower-than-real-time decoder is flagged "(slower than it plays)". The next #2316 log will show whether decoding is what falls behind. - **`scripts/windows/dav1d.sh`** builds dav1d with llvm-mingw from the same release and source-tree hash that `scripts/linux/deps.sh` pins (1.5.4; it reads them from there). It installs into the CPM layout `<cpm>/dav1d/1.5.4/{include,lib/libdav1d.a,COPYING}`. The output is reproducible: two builds give the same `libdav1d.a` (SHA-256 `e6f1adec…7e98` with llvm-mingw 20261104). That took clang's `-mno-incremental-linker-compatible` and nasm's `--reproducible`; without them the COFF objects carry timestamps. - **`nx2/build.py`** compiles `dav1d_av1.c` and links `libdav1d.a` when the CPM cache has dav1d, on the llvm-mingw route only (it is a MinGW library). Without it, or on the Visual Studio route, nothing changes and Media Foundation decodes the movies as before. - **CI:** a new Linux job in `windows-builder.yml` builds dav1d with the pinned Linux-hosted llvm-mingw and runs the wrapper's test on the Windows build under Wine. It uploads the CPM folder as the `dav1d-windows` artifact. The Linux test job installs `libdav1d-dev` for the new test. ### How dav1d reaches players Players build the game on their own PC, so dav1d has to reach them in the Builder's **cpm part**. The cpm part is normally reused from the latest release. This PR makes the release workflow add dav1d to it: the package job takes the `dav1d-windows` artifact and runs `release_parts.py package --cpm-add`. If the latest release's cpm part lacks `dav1d/1.5.4`, the job unpacks that part into `tools/cpm` with dav1d added. `make_setup_parts.py` then makes a new cpm part, and the release job uploads it to `builder-parts` like any other new part. Once a release carries it, later runs reuse it. No manual step is needed. I checked this against the real v2.1.0 setup list and the dav1d built here. The new part is `HookshotBuilder-cpm-73d3c9f5a31d.zip` (19.6 MB, up from 18.6 MB). It holds the 451 files of `HookshotBuilder-cpm-c30c77e4cf00.zip` byte-for-byte, plus dav1d's 8 files. Two runs gave the same SHA-256, because the zip and `libdav1d.a` are both reproducible. ### Testing - `tests/test_dav1d_av1.py` (new) uses a 3.8 KB synthetic clip in `tests/dav1d_av1/`: 12 pictures of a formula pattern, encoded losslessly with libaom, so no game data. Every picture must come out in order, with its pts, as the exact NV12 bytes of the pattern. That includes the pictures dav1d holds back until the end, and a second pass after a flush. It runs at 1 and 4 threads, and checks that `HOS_NO_DAV1D=1` leaves the decoder closed. It passes on Linux with clang, clang++, gcc and g++ against the system dav1d. It also passes for **the Windows build** (llvm-mingw, the script's `libdav1d.a`) under Wine at 1 thread, 4 threads and the default. Breaking the end-of-stream drain makes it fail. - `tests/test_cpm_discovery.py`: dav1d is used only on the llvm-mingw route, never on the Visual Studio route or Linux, and only with both its header and library present. - `tests/test_release_cpm_add.py`: `--cpm-add` adds dav1d to a cpm part that lacks it and keeps every file the part had. It leaves a part that already has dav1d to be reused as it is, and adds a new version beside an old one. - `tests/test_vcore_video_acquire.py` and `tests/test_run_all_declarations.py` were updated for the new type and tool. - `python -I -B tests/run_all.py`: 420 passed, 0 failed on Linux, rebased on current `main`. - `host/nn_video.c` compiles for Windows with llvm-mingw and the cpm part's own FFmpeg headers, with and without `HOOKSHOT_DAV1D`, and for Linux. - Speed, on a synthetic 1080p AV1 clip (SVT-AV1, 300 pictures). The Windows build under Wine, on a Ryzen 7 9800X3D: 2.8 ms per picture on 1 thread, 1.9 ms on 4, 1.4 ms on all 16. A 30 fps movie allows 33 ms per picture. The real movies are harder to decode than this clip. ### Not tested - **The game on Windows.** I'm on Linux with no Windows install, so I need a Windows tester to build with dav1d in the CPM cache and play the intro movie. Ideally on a PC that stutters today (#2316's RTX 4060, or a GTX 10/16), with the new log line before and after. - No timing comparison against Media Foundation, since Wine has no AV1 Video Extension. - The new CI job runs for the first time on this PR. - 10-bit and 4:4:4 conversion paths (OoT's movies are 8-bit 4:2:0). No screenshots: nothing new shows on screen. The movies should just play in sync, and the extension's Store dialog no longer appears once dav1d is in the Builder.

hckrman101 (Migrated from github.com) reviewed

2026-11-11 21:38:03 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks, this is a careful piece of work and the diagnosis in the description matches the code: the CPM FFmpeg has no AV1 decoder, so nn_video.c falls through to mf_av1.c and its synchronous (software) MFT on every GPU. Linking dav1d directly is the right fix, and the per-movie log line will make the #2503 / #2316 reports much easier to read.

What I checked and found fine:

Decoder lifecycle in dav1d_av1.c: every dav1d_get_picture is paired with dav1d_picture_unref, the Dav1dData is unref'd on a refused send (and the 64-try cap can't spin forever), flush is dav1d_flush with the NV12 scratch kept, close frees everything and takes NULL. The one-picture-per-submission rule is right about dav1d's drain-on-second-call behaviour, and the EAGAIN path's second call is exactly that intended blocking drain.
Bounds in deliver(): odd sizes are truncated to even before the chroma loops, so for 4:2:0, 4:2:2 and 4:4:4 the source row and column indexes stay inside the plane; stride[1] is shared by U and V as dav1d defines it; high bit depth reads uint16_t rows with byte strides; 4:0:0 doesn't touch data[1].
nn_video.c: the decoder table is only reached when FFmpeg has no AV1 decoder, d->av1 is only dereferenced behind d->mf (dec_flush is guarded by d->ctx || d->mf), and without HOOKSHOT_DAV1D the table is just Media Foundation. So Linux and a Windows build without dav1d in the CPM cache behave as before (dav1d_windows() returns None off llvm-mingw or with a partial install, and the new test_cpm_discovery case covers that).
Build script: same release and the same unpacked-tree hash method as scripts/linux/deps.sh's tree_is, read from that file so they can't drift; --reproducible nasm plus -mno-incremental-linker-compatible is the right pair for stable COFF objects; the BSD-2 COPYING travels with the library. Actions are SHA-pinned, permissions: contents: read at the top, llvm-mingw checked by SHA-256.
Fixture: the test compares every decoded byte to the formula, which a lossless encode of that formula is the only way to pass, so it's self-made, not game-derived.
run_all: the new dav1d placeholder is in find_tools, the declarations test's TOOLS is updated, it's skipped on Windows and skipped (not failed) on Linux without libdav1d-dev.

Open points:

CI hasn't run. Only the title check ran on this head: the PR conflicts with main, and pull_request workflows don't start until it merges cleanly, so the new dav1d job (which you say runs for the first time here) and the Linux test job are both still unproven. The conflicts are nx2/build.py (main's linker() rework of the link lines from the same area) and docs/worklist.md (a row added next to yours); both look mechanical. A rebase gets CI going.
Thread count (inline): suggestion to default to 4 like the FFmpeg path on Linux.
No Windows run yet, as you say. Since nothing changes for players until the cpm part has dav1d, that can happen after merge, but a log line from a stuttering Windows PC with dav1d built in, before and after, would be good to see before the cpm part is rebuilt.

Nit: #2316 was closed into #2503; the worklist row could link #2503 too.

Overlap: #2114 also edits host/nn_video.c (pool RGBA conversion), with hunks next to yours in dec_init_sized and after dec_finalize, so whichever lands second gets a small textual rebase; the changes themselves don't interact.

Happy to approve once it's rebased and CI (including the new job) is green.

Thanks, this is a careful piece of work and the diagnosis in the description matches the code: the CPM FFmpeg has no AV1 decoder, so `nn_video.c` falls through to `mf_av1.c` and its synchronous (software) MFT on every GPU. Linking dav1d directly is the right fix, and the per-movie log line will make the #2503 / #2316 reports much easier to read. What I checked and found fine: - **Decoder lifecycle** in `dav1d_av1.c`: every `dav1d_get_picture` is paired with `dav1d_picture_unref`, the `Dav1dData` is unref'd on a refused send (and the 64-try cap can't spin forever), flush is `dav1d_flush` with the NV12 scratch kept, close frees everything and takes NULL. The one-picture-per-submission rule is right about dav1d's drain-on-second-call behaviour, and the EAGAIN path's second call is exactly that intended blocking drain. - **Bounds in `deliver()`**: odd sizes are truncated to even before the chroma loops, so for 4:2:0, 4:2:2 and 4:4:4 the source row and column indexes stay inside the plane; `stride[1]` is shared by U and V as dav1d defines it; high bit depth reads `uint16_t` rows with byte strides; 4:0:0 doesn't touch `data[1]`. - **`nn_video.c`**: the decoder table is only reached when FFmpeg has no AV1 decoder, `d->av1` is only dereferenced behind `d->mf` (`dec_flush` is guarded by `d->ctx || d->mf`), and without `HOOKSHOT_DAV1D` the table is just Media Foundation. So Linux and a Windows build without dav1d in the CPM cache behave as before (`dav1d_windows()` returns None off llvm-mingw or with a partial install, and the new `test_cpm_discovery` case covers that). - **Build script**: same release and the same unpacked-tree hash method as `scripts/linux/deps.sh`'s `tree_is`, read from that file so they can't drift; `--reproducible` nasm plus `-mno-incremental-linker-compatible` is the right pair for stable COFF objects; the BSD-2 `COPYING` travels with the library. Actions are SHA-pinned, `permissions: contents: read` at the top, llvm-mingw checked by SHA-256. - **Fixture**: the test compares every decoded byte to the formula, which a lossless encode of that formula is the only way to pass, so it's self-made, not game-derived. - **run_all**: the new `dav1d` placeholder is in `find_tools`, the declarations test's `TOOLS` is updated, it's skipped on Windows and skipped (not failed) on Linux without `libdav1d-dev`. Open points: 1. **CI hasn't run.** Only the title check ran on this head: the PR conflicts with `main`, and `pull_request` workflows don't start until it merges cleanly, so the new `dav1d` job (which you say runs for the first time here) and the Linux test job are both still unproven. The conflicts are `nx2/build.py` (main's `linker()` rework of the link lines from the same area) and `docs/worklist.md` (a row added next to yours); both look mechanical. A rebase gets CI going. 2. **Thread count** (inline): suggestion to default to 4 like the FFmpeg path on Linux. 3. **No Windows run yet**, as you say. Since nothing changes for players until the cpm part has dav1d, that can happen after merge, but a log line from a stuttering Windows PC with dav1d built in, before and after, would be good to see before the cpm part is rebuilt. Nit: #2316 was closed into #2503; the worklist row could link #2503 too. Overlap: #2114 also edits `host/nn_video.c` (pool RGBA conversion), with hunks next to yours in `dec_init_sized` and after `dec_finalize`, so whichever lands second gets a small textual rebase; the changes themselves don't interact. Happy to approve once it's rebased and CI (including the new job) is green.

host/dav1d_av1.c

					
				@ -0,0 +32,4 @@

				    if (env_on("HOS_NO_DAV1D")) return NULL;

				    Dav1dSettings s;

				    dav1d_default_settings(&s);

				    if (env_int("HOS_DAV1D_THREADS", 0) > 0) s.n_threads = env_int("HOS_DAV1D_THREADS", 0);

Suggestion: default to 4 threads, like the FFmpeg path on Linux (thread_count = 4 for AV1 in dec_init_sized), unless HOS_DAV1D_THREADS says otherwise. With n_threads = 0 dav1d takes one thread per logical core and, through its automatic max_frame_delay, holds more pictures back as the count grows. Since #2322 the decode no longer holds a virtual core, so on a 16 to 32 thread CPU those threads compete with the game's own while the movie plays. Your own numbers (1.9 ms on 4 threads against a 33 ms budget) suggest 4 is plenty, and it keeps Windows and Linux decoding the same way, which makes logs easier to compare.

Suggestion: default to 4 threads, like the FFmpeg path on Linux (`thread_count = 4` for AV1 in `dec_init_sized`), unless `HOS_DAV1D_THREADS` says otherwise. With `n_threads = 0` dav1d takes one thread per logical core and, through its automatic `max_frame_delay`, holds more pictures back as the count grows. Since #2322 the decode no longer holds a virtual core, so on a 16 to 32 thread CPU those threads compete with the game's own while the movie plays. Your own numbers (1.9 ms on 4 threads against a 33 ms budget) suggest 4 is plenty, and it keeps Windows and Linux decoding the same way, which makes logs easier to compare.