fix(video): a decoder refuses submissions while 16 wait, so a movie's first seconds are not skipped #110

Merged
HumanBean merged 2 commits from HumanBean/Longshot:fix/video-input-limit into main 2026-11-13 05:02:30 +00:00
Owner

Why

The first seconds of every movie stutter: the frames are on time (60 a second), but the movie shows about ten pictures a second, for about 5 s at the start of the first movie of a new game and about 2 s at the start of the second. A third of the first movie's pictures are skipped there.

What changed

  • video: dec_submit() in host/nn_video.c refuses a submission with RESULT_INPUT_FULL while 16 submissions wait in the decoder (input_full(), submit_refused()), as the console's decoder does when it has no room; the stream's last submission is always taken. Until now every submission was taken, so the game's movie player put a whole movie in at once, on the thread that also takes the pictures: while it was busy with that it took a picture only about every 105 ms, and the catch-up (the movie's clock) dropped the 4 to 6 pictures that had come due meanwhile.
  • video: each decoder has its own submission-ready event (ready_event(), input_room()), clear while the decoder is full and signalled again when an Acquire, a Flush or a Stop makes room. It was one event for all decoders, signalled for ever. A decoder that ends gives its event back for the next one (ready_event_give_back()), so none is left behind per movie.
  • video: the catch-up is unchanged and stays as the safety net for late frames.
  • log: a movie's line in the log also says how many submissions were put off, and a second line gives the longest wait between two of its pictures; with HOS_VIDEO_LOG=1 a third gives those waits by length (the first 5 s and after) and how far the last picture was from the movie's clock.
  • sync: hos_system_event_clear() in host/nn_os_sync.c, for the event above.
  • tests: tests/test_video_input_limit.py (new) plays a 600-picture movie through the decoder's own code with a stand-in for libavcodec; tests/test_vcore_video_acquire.py gets a stand-in for input_room().
  • docs: the switch in docs/development.md, a row in docs/tested-configurations.md.

Flags and settings

Flag New or changed What it does Default now Goal: on by default? Helps on Doesn't help on Untested on Untested, but should help on
HOS_VIDEO_INPUT_LIMIT=<n> New How many submissions a video decoder holds before it refuses the next; 0: no limit, as before 16 Yes: it is on. The switch is there to compare and to turn it off Every GPU and OS: it is host code, the same for all (measured on NVIDIA, Linux, dav1d) Movies whose frames are late (a PC short of memory or CPU): those pictures are still skipped, by the catch-up Windows (dav1d and Media Foundation), AMD, Intel, Steam Deck Windows, AMD, Intel, Steam Deck

Measurements

Baseline: this PR @ 61d4f4f8, HOS_VIDEO_INPUT_LIMIT=0 (three runs; median)
main: 4dd972a8 (2026-11-12)
this PR: fix/video-input-limit @ 61d4f4f8 (merged with main b9cdc432 as a81848e5)
Why main wasn't the comparison: with the limit at 0 the build takes every submission, as main does, so one build gives both sides in the same folder with the same caches. A build of main 4dd972a8 itself gave the same counts in two earlier runs (first movie left to play in full: 246 of 5,947 pictures skipped, about 229 of them in its first 4.8 s). The measured build is the commit's code; only comment text and the docs changed after it was built.
Hardware: RTX 3090 24 GB, NVIDIA 620.70.06, Ryzen 9 7900X, 30 GB, Linux (CachyOS, kernel 7.3.3)
Scene: a new game from an empty data folder: the first movie (an A press 11 s in ends it after about 820 pictures), then the second movie (9,254 pictures, 154 s) to its end. 2048x1152, default options, frame rate cap 60, no sound device (HOS_AUDIO=0), a private headless sway
Method: three runs each, interleaved (off, on, on, off, off, on), the whole session each time. Counts from the movie's lines in the log (HOS_VIDEO_LOG=1); frame times from RX_VK_FRAME_LOG, from 25 s after the first frame to 8 s before the last (the second movie without its first seconds and the scene's load at its end). The PC was short of memory and swapping in every run (110,000 to 590,000 pages swapped in a run): the picture counts of the clean runs do not depend on it, the frame times do. In the second run of each side it swapped hard enough to make frames late, and the pictures skipped there are the late frames' on both sides.
How to reproduce: an empty data folder (XDG_DATA_HOME), HOS_VIDEO_LOG=1, start the game and press nothing until the first movie has played 11 s, then A once a second until the second movie starts; let it play out; read the two video: a movie of ... lines. Again with HOS_VIDEO_INPUT_LIMIT=0.

this PR @ 61d4f4f8, HOS_VIDEO_INPUT_LIMIT=0 (three runs; median) this PR @ 61d4f4f8 (three runs; median)
First movie: pictures skipped (of about 820) 244, 247, 251; 247 8, 198, 1; 8
First movie, first 5 s: waits of over 90 ms between two pictures 44, 43, 43; 43 1, 0, 0; 0
First movie, first 5 s: pictures under 20 ms apart 29, 31, 25; 29 287, 293, 294; 293
Second movie: pictures skipped (of 9,254) 123, 700, 98; 123 0, 502, 0; 0
Second movie: skipped in its first 10 s 104, 165, 95; 104 0, 486, 0; 0
Second movie: skipped after its first 10 s, a minute 7.9, 222.6, 1.2; 7.9 0, 6.7, 0; 0
Second movie, first 5 s: waits of over 90 ms between two pictures 20, 21, 18; 20 0, 9, 0; 0
Second movie: longest wait between two pictures 303, 212, 132 ms; 212 45, 180, 41 ms; 45
Second movie's last picture against the movie's clock -0.012, -0.009, -0.008 s -0.008, -0.011, -0.007 s
Frames over 33 ms (of about 8,700) 2, 177, 1; 2 0, 146, 1; 1
Frame time, median / 99th percentile 16.68 / 19.99, 16.69 / 75.27, 16.68 / 18.41 ms 16.68 / 18.51, 16.68 / 46.67, 16.67 / 20.29 ms

The middle run of each side is the one with late frames (the PC swapping): there the limit cannot help, and does not hurt.

Limits of 4, 8 and 12 at the first movie's start (one run each, 14 s): 8, 9 and 3 pictures skipped, no Acquire left waiting. 16 is kept for the margin over what the AV1 decoder's frame threads can hold back.

# m-off-1
video: a movie of 809 pictures on libdav1d: 0.8 ms to decode each (worst 24.4), one shown every 16.7 ms; 244 skipped to keep up with its sound; 0 submissions put off while 0 waited
video: the longest wait between two of its pictures 117 ms
video: time between pictures, the first 5 s: under 20 ms 29, 20-50 0, 50-90 1, over 44; after: 453, 30, 0, 4; at the end the picture was -0.006 s behind the movie's clock
video: a movie of 9254 pictures on libdav1d: 0.6 ms to decode each (worst 24.9), one shown every 16.7 ms; 123 skipped to keep up with its sound; 0 submissions put off while 0 waited
video: the longest wait between two of its pictures 303 ms
video: time between pictures, the first 5 s: under 20 ms 165, 20-50 7, 50-90 3, over 20; after: 8534, 399, 1, 1; at the end the picture was -0.012 s behind the movie's clock
# m-on-1
video: a movie of 833 pictures on libdav1d: 0.6 ms to decode each (worst 24.5), one shown every 16.7 ms; 8 skipped to keep up with its sound; 619 submissions put off while 16 waited
video: the longest wait between two of its pictures 137 ms
video: time between pictures, the first 5 s: under 20 ms 287, 20-50 4, 50-90 0, over 1; after: 500, 29, 0, 0; at the end the picture was -0.008 s behind the movie's clock
video: a movie of 9254 pictures on libdav1d: 0.6 ms to decode each (worst 28.6), one shown every 16.7 ms; 0 skipped to keep up with its sound; 6935 submissions put off while 16 waited
video: the longest wait between two of its pictures 45 ms
video: time between pictures, the first 5 s: under 20 ms 289, 20-50 11, 50-90 0, over 0; after: 8564, 389, 0, 0; at the end the picture was -0.008 s behind the movie's clock
# m-off-2
video: a movie of 816 pictures on libdav1d: 0.8 ms to decode each (worst 23.1), one shown every 16.7 ms; 247 skipped to keep up with its sound; 0 submissions put off while 0 waited
video: the longest wait between two of its pictures 193 ms
video: time between pictures, the first 5 s: under 20 ms 31, 20-50 0, 50-90 0, over 43; after: 459, 27, 0, 5; at the end the picture was -0.007 s behind the movie's clock
video: a movie of 9254 pictures on libdav1d: 0.9 ms to decode each (worst 96.1), one shown every 16.7 ms; 700 skipped to keep up with its sound; 0 submissions put off while 0 waited
video: the longest wait between two of its pictures 212 ms
video: time between pictures, the first 5 s: under 20 ms 165, 20-50 6, 50-90 2, over 21; after: 7751, 474, 96, 38; at the end the picture was -0.009 s behind the movie's clock
# m-on-2
video: a movie of 881 pictures on libdav1d: 2.5 ms to decode each (worst 89.6), one shown every 16.7 ms; 198 skipped to keep up with its sound; 516 submissions put off while 16 waited
video: the longest wait between two of its pictures 199 ms
video: time between pictures, the first 5 s: under 20 ms 293, 20-50 3, 50-90 1, over 0; after: 315, 32, 14, 21; at the end the picture was 0.037 s behind the movie's clock
video: a movie of 9254 pictures on libdav1d: 0.8 ms to decode each (worst 79.5), one shown every 16.7 ms; 502 skipped to keep up with its sound; 6610 submissions put off while 16 waited
video: the longest wait between two of its pictures 180 ms
video: time between pictures, the first 5 s: under 20 ms 10, 20-50 40, 50-90 37, over 9; after: 8120, 473, 46, 16; at the end the picture was -0.011 s behind the movie's clock
# m-off-3
video: a movie of 847 pictures on libdav1d: 0.8 ms to decode each (worst 23.9), one shown every 16.7 ms; 251 skipped to keep up with its sound; 0 submissions put off while 0 waited
video: the longest wait between two of its pictures 194 ms
video: time between pictures, the first 5 s: under 20 ms 25, 20-50 0, 50-90 0, over 43; after: 490, 29, 0, 5; at the end the picture was -0.011 s behind the movie's clock
video: a movie of 9254 pictures on libdav1d: 0.6 ms to decode each (worst 46.4), one shown every 16.7 ms; 98 skipped to keep up with its sound; 0 submissions put off while 0 waited
video: the longest wait between two of its pictures 132 ms
video: time between pictures, the first 5 s: under 20 ms 179, 20-50 4, 50-90 4, over 18; after: 8504, 444, 2, 0; at the end the picture was -0.008 s behind the movie's clock
# m-on-3
video: a movie of 772 pictures on libdav1d: 0.7 ms to decode each (worst 22.0), one shown every 16.7 ms; 1 skipped to keep up with its sound; 578 submissions put off while 16 waited
video: the longest wait between two of its pictures 38 ms
video: time between pictures, the first 5 s: under 20 ms 294, 20-50 5, 50-90 0, over 0; after: 440, 28, 0, 0; at the end the picture was 0.005 s behind the movie's clock
video: a movie of 9254 pictures on libdav1d: 0.6 ms to decode each (worst 24.7), one shown every 16.7 ms; 0 skipped to keep up with its sound; 6935 submissions put off while 16 waited
video: the longest wait between two of its pictures 41 ms
video: time between pictures, the first 5 s: under 20 ms 286, 20-50 14, 50-90 0, over 0; after: 8545, 408, 0, 0; at the end the picture was -0.007 s behind the movie's clock

Testing

Tested: On the PC above (Linux, RTX 3090, NVIDIA 620.70.06, libdav1d), in the game: both movies of a new game, six full sessions as measured, plus the first movie left to play in full (5,947 pictures) with the limit on: 3 pictures skipped in its first 10 s (229 on main), the movie ends, the second one starts. Checked in those runs: a movie stopped early by a button (a Flush with submissions still waiting: the next movie starts and plays), the end of a stream (the last picture marked, the movie stops, the game goes on into the scene behind it), two movies in a row (the second decoder made while the first is torn down), and a refusal on the thread that also takes the pictures (the game goes on to its Acquire, which makes room: 6,935 refusals in the second movie, one a frame). python -I -B tests/run_all.py: 505 passed, 2 failed, 24 skipped on the merged tree; the two failures are tests/test_bake_portability.py (no zstandard module on this PC) and tests/test_builder_log_read_only.py (no Tk library on this PC), neither touches this change and both fail the same way without it. tests/test_video_input_limit.py fails on main and passes here.

Not tested: The merged tree (a81848e5) in the game: the game runs were made from 61d4f4f8, whose host/nn_video.c is the same apart from the rename's spelling. Windows, with dav1d or with Media Foundation's AV1 decoder. With a sound device (the runs had none; the movie's clock is the host's, not the sound's). Movies later in the game (cutscenes with a movie, the title's clips when they loop), a movie paused or sought. A frame rate cap above 60. Games other than OoT that use these decoders (VP8, VP9): the limit is on for every codec.

Needs other testers:

  • Windows, any GPU: a new game's two movies play smoothly from their first second and end normally; the log's video: a movie of ... lines say 0 to about 10 pictures skipped and submissions put off above 0 (test request T-58KT)
  • AMD and Intel GPUs, Steam Deck: the same (test request T-BW29)
  • A PC whose Windows has no AV1 Video Extension and no dav1d build: the movies stay black with sound as before, and the game goes on after them (test request T-JJF9)

Screenshots

Nothing on screen (a movie's pace; no picture changes).

Risk and rollback

A game whose movie player does not expect a refused submission would stop at its first movie; OoT's takes it and submits again. If a movie hangs or the game stops at a movie's start, HOS_VIDEO_INPUT_LIMIT=0 gives main's behaviour back; reverting the commit is enough too. The refusal itself waits for nothing, so it cannot block a thread; room is made by Acquire, Flush and Stop, which the game calls whether or not it was refused.

Reviewer notes

  • Start with the comment above RESULT_INPUT_FULL in host/nn_video.c, then submit_refused(), input_room() and ready_event().
  • Left for a follow-up on purpose: at a stream's end the decoder's held pictures go into the queue of 8 pictures; if one is still there and 8 come out at once the oldest would be dropped. Not seen in the game (9,254 of 9,254 pictures handed out), and not new.
  • This fixes the stutter at each movie's start. The long stretches of late frames in the session of #73 (9 to 16 fps for 38 s of the second movie) are a different thing: they came back here only in the runs where the PC swapped, and this change does not touch them.

Issues

See #73

Checklist

For everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description).
  • Rebased on current main. (main b9cdc432 merged in: a81848e5)
  • python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
  • Tested in the game, or what wasn't is named under Testing.
  • Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
  • Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Flags table filled in; README / docs/development.md switch reference updated.
  • A row in docs/tested-configurations.md for each measured run.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Linux built; Windows not built)
  • Docs updated.
## Why The first seconds of every movie stutter: the frames are on time (60 a second), but the movie shows about ten pictures a second, for about 5 s at the start of the first movie of a new game and about 2 s at the start of the second. A third of the first movie's pictures are skipped there. ## What changed - video: `dec_submit()` in `host/nn_video.c` refuses a submission with `RESULT_INPUT_FULL` while 16 submissions wait in the decoder (`input_full()`, `submit_refused()`), as the console's decoder does when it has no room; the stream's last submission is always taken. Until now every submission was taken, so the game's movie player put a whole movie in at once, on the thread that also takes the pictures: while it was busy with that it took a picture only about every 105 ms, and the catch-up (the movie's clock) dropped the 4 to 6 pictures that had come due meanwhile. - video: each decoder has its own submission-ready event (`ready_event()`, `input_room()`), clear while the decoder is full and signalled again when an Acquire, a Flush or a Stop makes room. It was one event for all decoders, signalled for ever. A decoder that ends gives its event back for the next one (`ready_event_give_back()`), so none is left behind per movie. - video: the catch-up is unchanged and stays as the safety net for late frames. - log: a movie's line in the log also says how many submissions were put off, and a second line gives the longest wait between two of its pictures; with `HOS_VIDEO_LOG=1` a third gives those waits by length (the first 5 s and after) and how far the last picture was from the movie's clock. - sync: `hos_system_event_clear()` in `host/nn_os_sync.c`, for the event above. - tests: `tests/test_video_input_limit.py` (new) plays a 600-picture movie through the decoder's own code with a stand-in for libavcodec; `tests/test_vcore_video_acquire.py` gets a stand-in for `input_room()`. - docs: the switch in `docs/development.md`, a row in `docs/tested-configurations.md`. ## Flags and settings | Flag | New or changed | What it does | Default now | Goal: on by default? | Helps on | Doesn't help on | Untested on | Untested, but should help on | |---|---|---|---|---|---|---|---|---| | `HOS_VIDEO_INPUT_LIMIT=<n>` | New | How many submissions a video decoder holds before it refuses the next; `0`: no limit, as before | 16 | Yes: it is on. The switch is there to compare and to turn it off | Every GPU and OS: it is host code, the same for all (measured on NVIDIA, Linux, dav1d) | Movies whose frames are late (a PC short of memory or CPU): those pictures are still skipped, by the catch-up | Windows (dav1d and Media Foundation), AMD, Intel, Steam Deck | Windows, AMD, Intel, Steam Deck | ## Measurements Baseline: this PR @ 61d4f4f8, `HOS_VIDEO_INPUT_LIMIT=0` (three runs; median) main: 4dd972a8 (2026-11-12) this PR: fix/video-input-limit @ 61d4f4f8 (merged with main b9cdc432 as a81848e5) Why main wasn't the comparison: with the limit at 0 the build takes every submission, as main does, so one build gives both sides in the same folder with the same caches. A build of main 4dd972a8 itself gave the same counts in two earlier runs (first movie left to play in full: 246 of 5,947 pictures skipped, about 229 of them in its first 4.8 s). The measured build is the commit's code; only comment text and the docs changed after it was built. Hardware: RTX 3090 24 GB, NVIDIA 620.70.06, Ryzen 9 7900X, 30 GB, Linux (CachyOS, kernel 7.3.3) Scene: a new game from an empty data folder: the first movie (an A press 11 s in ends it after about 820 pictures), then the second movie (9,254 pictures, 154 s) to its end. 2048x1152, default options, frame rate cap 60, no sound device (`HOS_AUDIO=0`), a private headless sway Method: three runs each, interleaved (off, on, on, off, off, on), the whole session each time. Counts from the movie's lines in the log (`HOS_VIDEO_LOG=1`); frame times from `RX_VK_FRAME_LOG`, from 25 s after the first frame to 8 s before the last (the second movie without its first seconds and the scene's load at its end). The PC was short of memory and swapping in every run (110,000 to 590,000 pages swapped in a run): the picture counts of the clean runs do not depend on it, the frame times do. In the second run of each side it swapped hard enough to make frames late, and the pictures skipped there are the late frames' on both sides. How to reproduce: an empty data folder (`XDG_DATA_HOME`), `HOS_VIDEO_LOG=1`, start the game and press nothing until the first movie has played 11 s, then A once a second until the second movie starts; let it play out; read the two `video: a movie of ...` lines. Again with `HOS_VIDEO_INPUT_LIMIT=0`. | | this PR @ 61d4f4f8, `HOS_VIDEO_INPUT_LIMIT=0` (three runs; median) | this PR @ 61d4f4f8 (three runs; median) | |---|---|---| | First movie: pictures skipped (of about 820) | 244, 247, 251; 247 | 8, 198, 1; 8 | | First movie, first 5 s: waits of over 90 ms between two pictures | 44, 43, 43; 43 | 1, 0, 0; 0 | | First movie, first 5 s: pictures under 20 ms apart | 29, 31, 25; 29 | 287, 293, 294; 293 | | Second movie: pictures skipped (of 9,254) | 123, 700, 98; 123 | 0, 502, 0; 0 | | Second movie: skipped in its first 10 s | 104, 165, 95; 104 | 0, 486, 0; 0 | | Second movie: skipped after its first 10 s, a minute | 7.9, 222.6, 1.2; 7.9 | 0, 6.7, 0; 0 | | Second movie, first 5 s: waits of over 90 ms between two pictures | 20, 21, 18; 20 | 0, 9, 0; 0 | | Second movie: longest wait between two pictures | 303, 212, 132 ms; 212 | 45, 180, 41 ms; 45 | | Second movie's last picture against the movie's clock | -0.012, -0.009, -0.008 s | -0.008, -0.011, -0.007 s | | Frames over 33 ms (of about 8,700) | 2, 177, 1; 2 | 0, 146, 1; 1 | | Frame time, median / 99th percentile | 16.68 / 19.99, 16.69 / 75.27, 16.68 / 18.41 ms | 16.68 / 18.51, 16.68 / 46.67, 16.67 / 20.29 ms | The middle run of each side is the one with late frames (the PC swapping): there the limit cannot help, and does not hurt. Limits of 4, 8 and 12 at the first movie's start (one run each, 14 s): 8, 9 and 3 pictures skipped, no Acquire left waiting. 16 is kept for the margin over what the AV1 decoder's frame threads can hold back. ``` # m-off-1 video: a movie of 809 pictures on libdav1d: 0.8 ms to decode each (worst 24.4), one shown every 16.7 ms; 244 skipped to keep up with its sound; 0 submissions put off while 0 waited video: the longest wait between two of its pictures 117 ms video: time between pictures, the first 5 s: under 20 ms 29, 20-50 0, 50-90 1, over 44; after: 453, 30, 0, 4; at the end the picture was -0.006 s behind the movie's clock video: a movie of 9254 pictures on libdav1d: 0.6 ms to decode each (worst 24.9), one shown every 16.7 ms; 123 skipped to keep up with its sound; 0 submissions put off while 0 waited video: the longest wait between two of its pictures 303 ms video: time between pictures, the first 5 s: under 20 ms 165, 20-50 7, 50-90 3, over 20; after: 8534, 399, 1, 1; at the end the picture was -0.012 s behind the movie's clock # m-on-1 video: a movie of 833 pictures on libdav1d: 0.6 ms to decode each (worst 24.5), one shown every 16.7 ms; 8 skipped to keep up with its sound; 619 submissions put off while 16 waited video: the longest wait between two of its pictures 137 ms video: time between pictures, the first 5 s: under 20 ms 287, 20-50 4, 50-90 0, over 1; after: 500, 29, 0, 0; at the end the picture was -0.008 s behind the movie's clock video: a movie of 9254 pictures on libdav1d: 0.6 ms to decode each (worst 28.6), one shown every 16.7 ms; 0 skipped to keep up with its sound; 6935 submissions put off while 16 waited video: the longest wait between two of its pictures 45 ms video: time between pictures, the first 5 s: under 20 ms 289, 20-50 11, 50-90 0, over 0; after: 8564, 389, 0, 0; at the end the picture was -0.008 s behind the movie's clock # m-off-2 video: a movie of 816 pictures on libdav1d: 0.8 ms to decode each (worst 23.1), one shown every 16.7 ms; 247 skipped to keep up with its sound; 0 submissions put off while 0 waited video: the longest wait between two of its pictures 193 ms video: time between pictures, the first 5 s: under 20 ms 31, 20-50 0, 50-90 0, over 43; after: 459, 27, 0, 5; at the end the picture was -0.007 s behind the movie's clock video: a movie of 9254 pictures on libdav1d: 0.9 ms to decode each (worst 96.1), one shown every 16.7 ms; 700 skipped to keep up with its sound; 0 submissions put off while 0 waited video: the longest wait between two of its pictures 212 ms video: time between pictures, the first 5 s: under 20 ms 165, 20-50 6, 50-90 2, over 21; after: 7751, 474, 96, 38; at the end the picture was -0.009 s behind the movie's clock # m-on-2 video: a movie of 881 pictures on libdav1d: 2.5 ms to decode each (worst 89.6), one shown every 16.7 ms; 198 skipped to keep up with its sound; 516 submissions put off while 16 waited video: the longest wait between two of its pictures 199 ms video: time between pictures, the first 5 s: under 20 ms 293, 20-50 3, 50-90 1, over 0; after: 315, 32, 14, 21; at the end the picture was 0.037 s behind the movie's clock video: a movie of 9254 pictures on libdav1d: 0.8 ms to decode each (worst 79.5), one shown every 16.7 ms; 502 skipped to keep up with its sound; 6610 submissions put off while 16 waited video: the longest wait between two of its pictures 180 ms video: time between pictures, the first 5 s: under 20 ms 10, 20-50 40, 50-90 37, over 9; after: 8120, 473, 46, 16; at the end the picture was -0.011 s behind the movie's clock # m-off-3 video: a movie of 847 pictures on libdav1d: 0.8 ms to decode each (worst 23.9), one shown every 16.7 ms; 251 skipped to keep up with its sound; 0 submissions put off while 0 waited video: the longest wait between two of its pictures 194 ms video: time between pictures, the first 5 s: under 20 ms 25, 20-50 0, 50-90 0, over 43; after: 490, 29, 0, 5; at the end the picture was -0.011 s behind the movie's clock video: a movie of 9254 pictures on libdav1d: 0.6 ms to decode each (worst 46.4), one shown every 16.7 ms; 98 skipped to keep up with its sound; 0 submissions put off while 0 waited video: the longest wait between two of its pictures 132 ms video: time between pictures, the first 5 s: under 20 ms 179, 20-50 4, 50-90 4, over 18; after: 8504, 444, 2, 0; at the end the picture was -0.008 s behind the movie's clock # m-on-3 video: a movie of 772 pictures on libdav1d: 0.7 ms to decode each (worst 22.0), one shown every 16.7 ms; 1 skipped to keep up with its sound; 578 submissions put off while 16 waited video: the longest wait between two of its pictures 38 ms video: time between pictures, the first 5 s: under 20 ms 294, 20-50 5, 50-90 0, over 0; after: 440, 28, 0, 0; at the end the picture was 0.005 s behind the movie's clock video: a movie of 9254 pictures on libdav1d: 0.6 ms to decode each (worst 24.7), one shown every 16.7 ms; 0 skipped to keep up with its sound; 6935 submissions put off while 16 waited video: the longest wait between two of its pictures 41 ms video: time between pictures, the first 5 s: under 20 ms 286, 20-50 14, 50-90 0, over 0; after: 8545, 408, 0, 0; at the end the picture was -0.007 s behind the movie's clock ``` ## Testing **Tested:** On the PC above (Linux, RTX 3090, NVIDIA 620.70.06, libdav1d), in the game: both movies of a new game, six full sessions as measured, plus the first movie left to play in full (5,947 pictures) with the limit on: 3 pictures skipped in its first 10 s (229 on main), the movie ends, the second one starts. Checked in those runs: a movie stopped early by a button (a Flush with submissions still waiting: the next movie starts and plays), the end of a stream (the last picture marked, the movie stops, the game goes on into the scene behind it), two movies in a row (the second decoder made while the first is torn down), and a refusal on the thread that also takes the pictures (the game goes on to its Acquire, which makes room: 6,935 refusals in the second movie, one a frame). `python -I -B tests/run_all.py`: 505 passed, 2 failed, 24 skipped on the merged tree; the two failures are `tests/test_bake_portability.py` (no `zstandard` module on this PC) and `tests/test_builder_log_read_only.py` (no Tk library on this PC), neither touches this change and both fail the same way without it. `tests/test_video_input_limit.py` fails on main and passes here. **Not tested:** The merged tree (a81848e5) in the game: the game runs were made from 61d4f4f8, whose `host/nn_video.c` is the same apart from the rename's spelling. Windows, with dav1d or with Media Foundation's AV1 decoder. With a sound device (the runs had none; the movie's clock is the host's, not the sound's). Movies later in the game (cutscenes with a movie, the title's clips when they loop), a movie paused or sought. A frame rate cap above 60. Games other than OoT that use these decoders (VP8, VP9): the limit is on for every codec. **Needs other testers:** - Windows, any GPU: a new game's two movies play smoothly from their first second and end normally; the log's `video: a movie of ...` lines say 0 to about 10 pictures skipped and `submissions put off` above 0 (test request T-58KT) - AMD and Intel GPUs, Steam Deck: the same (test request T-BW29) - A PC whose Windows has no AV1 Video Extension and no dav1d build: the movies stay black with sound as before, and the game goes on after them (test request T-JJF9) ## Screenshots Nothing on screen (a movie's pace; no picture changes). ## Risk and rollback A game whose movie player does not expect a refused submission would stop at its first movie; OoT's takes it and submits again. If a movie hangs or the game stops at a movie's start, `HOS_VIDEO_INPUT_LIMIT=0` gives main's behaviour back; reverting the commit is enough too. The refusal itself waits for nothing, so it cannot block a thread; room is made by Acquire, Flush and Stop, which the game calls whether or not it was refused. ## Reviewer notes - Start with the comment above `RESULT_INPUT_FULL` in `host/nn_video.c`, then `submit_refused()`, `input_room()` and `ready_event()`. - Left for a follow-up on purpose: at a stream's end the decoder's held pictures go into the queue of 8 pictures; if one is still there and 8 come out at once the oldest would be dropped. Not seen in the game (9,254 of 9,254 pictures handed out), and not new. - This fixes the stutter at each movie's start. The long stretches of late frames in the session of #73 (9 to 16 fps for 38 s of the second movie) are a different thing: they came back here only in the runs where the PC swapped, and this change does not touch them. ## Issues See #73 ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. (main b9cdc432 merged in: a81848e5) - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags table filled in; README / docs/development.md switch reference updated. - [x] A row in `docs/tested-configurations.md` for each measured run. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Linux built; Windows not built) - [x] Docs updated.
OoT's movie player submits a whole movie to the decoder at once, on the thread that
also takes the pictures. While it was busy with that (about 5 s for a 99 s movie) it
took a picture only every ~105 ms, and the catch-up dropped the 4-6 that had come due
meanwhile: about 245 pictures of the first movie and 100 of the second were skipped
at their starts, with every frame on time.

The console's decoder takes only so much input: a submission it has no room for fails
with a result the game takes as "not now", and its submission-ready event is clear
until there is room. dec_submit now refuses with that result while 16 submissions wait
(HOS_VIDEO_INPUT_LIMIT=<n>, 0: none, as before), never the stream's last one, and
each decoder has its own ready event that follows it. The catch-up stays as it was.

The movie's line in the log also says how many submissions were put off and the
longest wait between two of its pictures; HOS_VIDEO_LOG adds those waits by length.

Measured on an RTX 3090 (Linux), three runs a side: first movie 244, 247, 251 pictures
skipped without the limit and 8, 198, 1 with it; second movie (9,254 pictures) 123,
700, 98 and 0, 502, 0. The middle run of each side had late frames (the PC swapping).

Co-Authored-By: Claude Fable 5.2 <[email protected]>
merge: Longshot main (b9cdc432) into the video decoder's input limit
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 2s
CI / Workflow lint (actionlint) (pull_request) Failing after 4s
Third-party parts / As their notes say (pull_request) Successful in 5s
PR description / Description follows the template (pull_request_target) Failing after 3s
a81848e510
The change's files in the rename's spelling (host/nn_video.c, tests/test_video_input_limit.py);
docs/tested-configurations.md keeps main's rows with this change's row on top.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
Author
Owner

Self-review of the change, and three test requests opened (T-58KT Windows, T-BW29 AMD, Intel and Steam Deck, T-JJF9 Windows without an AV1 decoder).

Read again line by line; nothing found that blocks it, and these are the points a reviewer should weigh:

  • On for every game and codec. The refusal is on for VP8 and VP9 decoders too, and only this game's AV1 movies were run. A game whose player does not expect a refused submission could stop at its first movie. HOS_VIDEO_INPUT_LIMIT=0 is the way back; limiting it to AV1 would be a one-line change if a reviewer prefers that.
  • No wait in the refusal. submit_refused takes the decoder's lock, counts and returns; room is made only by Acquire, Flush and Stop. A game that submits and acquires on one thread therefore cannot block in it. A game that waits on the ready event without ever acquiring would wait for ever, as it would on the console.
  • What counts as full. Submissions not yet handed to the decoder plus pictures decoded and not taken. Pictures the decoder holds inside (frame threads) count as neither, so a full decoder always has something for the next Acquire.
  • The stream's last submission is never refused, so the end of a movie cannot be lost to the limit.
  • Ready events. One per decoder now, cleared while full. An ended decoder's event goes back to a pool of 8, signalled; a game still holding it sees ready, as before. The pool's static initialisation and input_limit()'s are unguarded, like the file's other lazy statics.
  • Reporting only, minor: the time between pictures is not reset by a Flush, so the first picture after a movie is stopped early can show as one long wait in the new log line; and the first-5-seconds bucket uses the movie clock, which is unset until the first picture. Neither affects playback.
  • Not covered by the runs: a sound device (the picture was checked against the movie's clock, which is the host's), seek, pause, looping clips, Windows, AMD, Intel. The numbers are from the commit before the rename; the merged tree's file differs only by the rename's spelling and was not run in the game.
Self-review of the change, and three test requests opened (T-58KT Windows, T-BW29 AMD, Intel and Steam Deck, T-JJF9 Windows without an AV1 decoder). Read again line by line; nothing found that blocks it, and these are the points a reviewer should weigh: - **On for every game and codec.** The refusal is on for VP8 and VP9 decoders too, and only this game's AV1 movies were run. A game whose player does not expect a refused submission could stop at its first movie. `HOS_VIDEO_INPUT_LIMIT=0` is the way back; limiting it to AV1 would be a one-line change if a reviewer prefers that. - **No wait in the refusal.** `submit_refused` takes the decoder's lock, counts and returns; room is made only by Acquire, Flush and Stop. A game that submits and acquires on one thread therefore cannot block in it. A game that waits on the ready event without ever acquiring would wait for ever, as it would on the console. - **What counts as full.** Submissions not yet handed to the decoder plus pictures decoded and not taken. Pictures the decoder holds inside (frame threads) count as neither, so a full decoder always has something for the next Acquire. - **The stream's last submission is never refused**, so the end of a movie cannot be lost to the limit. - **Ready events.** One per decoder now, cleared while full. An ended decoder's event goes back to a pool of 8, signalled; a game still holding it sees ready, as before. The pool's static initialisation and `input_limit()`'s are unguarded, like the file's other lazy statics. - **Reporting only, minor:** the time between pictures is not reset by a Flush, so the first picture after a movie is stopped early can show as one long wait in the new log line; and the first-5-seconds bucket uses the movie clock, which is unset until the first picture. Neither affects playback. - **Not covered by the runs:** a sound device (the picture was checked against the movie's clock, which is the host's), seek, pause, looping clips, Windows, AMD, Intel. The numbers are from the commit before the rename; the merged tree's file differs only by the rename's spelling and was not run in the game.
Author
Owner

License box ticked: the change is our own code under GPL-3.0-or-later, with nothing copied in and no game code or generated output in the diff.

License box ticked: the change is our own code under GPL-3.0-or-later, with nothing copied in and no game code or generated output in the diff.
HumanBean changed title from WIP: fix(video): a decoder refuses submissions while 16 wait, so a movie's first seconds are not skipped to fix(video): a decoder refuses submissions while 16 wait, so a movie's first seconds are not skipped 2026-11-13 04:55:10 +00:00
Author
Owner

Checklist tidied: the three boxes that do not apply (screenshots, API version, SPIR-V headers) are deleted instead of marked n/a. The Windows build box stays, unticked: host code changed and Windows has not been built here.

Checklist tidied: the three boxes that do not apply (screenshots, API version, SPIR-V headers) are deleted instead of marked n/a. The Windows build box stays, unticked: host code changed and Windows has not been built here.
HumanBean deleted branch fix/video-input-limit 2026-11-13 05:02:30 +00:00
First-time contributor

Measured on a Steam Deck LCD (VanGogh: AMD Custom GPU 0405, RADV VANGOGH driver 0x6801063, 4 CPU cores, 15 GB RAM), Linux, Longshot fabcd2c0 (v3.0.0-rc.2-3-gfabcd2c0, source a6589a6e), which includes this PR (ae7b1165 is an ancestor), default options with HOS_VIDEO_LOG=1, game audio confirmed open:

  • Default (INPUT_LIMIT=16): the first endcard movie, 5947 pictures: 4702-4766 skipped to keep up with its sound, 917-960 submissions put off while 16 waited. Two further 1800-picture movies: 230/271/199 skipped, 1158-1330 put off.
  • HOS_VIDEO_INPUT_LIMIT=0 (this PR's off state): same movie: 1082-1227 skipped, decode 4.8-5.3 ms/picture.
  • The pre-rename Hookshot build without this PR (c356bef4 era) showed ~1134 skipped on the same endcard movie, i.e. the =0 state matches the old behaviour; the default limit roughly 4x the skips on this hardware.

The 1800-picture movies stay mild on both arms (51-259 vs 65-82 skips); the effect is concentrated on the long endcard movie. dav1d decode alone is fast (2.3-9.3 ms against the 16.7 ms budget, 8 worker threads ~72% of one core in aggregate), so this looks like submit/decode interleaving rather than decoder speed: on 4 cores, refusing submissions while 16 wait appears to idle the decoder between the acquire thread's drains. The PR notes Steam Deck as "untested"; on this Deck the limit is a clear regression, while HOS_VIDEO_INPUT_LIMIT=0 behaves as before the PR. Numbers are per-run ranges over several interleaved runs; happy to run a matched A/B with more repetitions or a RX_VK_FRAME_LOG capture if useful.

Measured on a Steam Deck LCD (VanGogh: AMD Custom GPU 0405, RADV VANGOGH driver 0x6801063, 4 CPU cores, 15 GB RAM), Linux, Longshot `fabcd2c0` (v3.0.0-rc.2-3-gfabcd2c0, source a6589a6e), which includes this PR (ae7b1165 is an ancestor), default options with `HOS_VIDEO_LOG=1`, game audio confirmed open: - Default (`INPUT_LIMIT=16`): the first endcard movie, 5947 pictures: **4702-4766 skipped to keep up with its sound**, 917-960 submissions put off while 16 waited. Two further 1800-picture movies: 230/271/199 skipped, 1158-1330 put off. - `HOS_VIDEO_INPUT_LIMIT=0` (this PR's off state): same movie: **1082-1227 skipped**, decode 4.8-5.3 ms/picture. - The pre-rename Hookshot build without this PR (c356bef4 era) showed ~1134 skipped on the same endcard movie, i.e. the `=0` state matches the old behaviour; the default limit roughly **4x** the skips on this hardware. The 1800-picture movies stay mild on both arms (51-259 vs 65-82 skips); the effect is concentrated on the long endcard movie. dav1d decode alone is fast (2.3-9.3 ms against the 16.7 ms budget, 8 worker threads ~72% of one core in aggregate), so this looks like submit/decode interleaving rather than decoder speed: on 4 cores, refusing submissions while 16 wait appears to idle the decoder between the acquire thread's drains. The PR notes Steam Deck as "untested"; on this Deck the limit is a clear regression, while `HOS_VIDEO_INPUT_LIMIT=0` behaves as before the PR. Numbers are per-run ranges over several interleaved runs; happy to run a matched A/B with more repetitions or a `RX_VK_FRAME_LOG` capture if useful.
First-time contributor

T-BW29 (Steam Deck): fails with the limit on. On a Steam Deck the opening movie freezes about 14 s in and ends 7.5 s behind its sound. With HOS_VIDEO_INPUT_LIMIT=0 it stays in time.

Setup: Steam Deck OLED, SteamOS 3.9.5 (kernel 7.2.7-valve2), RADV Mesa 26.2.0-devel, libdav1d. Build: Longshot e7ed2f7a (with #110) plus Deck patches that don't touch host/nn_video.c. A copy of the user's install, started to the title, so the first movie plays (~3,200-3,700 pictures, 1920x1080, one every 16.7 ms). HOS_VIDEO_LOG=1. Two runs each, interleaved: on, off, on, off.

run limit pictures decoded skipped submissions put off decode each longest wait most behind the clock at the end
1 on (16) 3221 2619 575 9.6 ms 503 ms 7.83 s 7.518 s behind
2 off (0) 3646 2177 0 6.7 ms 478 ms 0.06 s 0.019 s behind
3 on (16) 3270 2668 557 9.7 ms 494 ms 5.81 s 7.541 s behind
4 off (0) 3685 2194 0 6.8 ms 423 ms 0.05 s 0.014 s behind

Run 1 timeline with the limit (acquire N: picture at ..., wall clock ...): on time until acquire 420 (14.30 s / 14.31 s). The next logged one is acquire 600 at 34.73 s against 40.35 s on the wall clock (5.62 s behind), then 42.65 s / 49.56 s and 50.52 s / 58.35 s. Between 14 s and 40 s only ~180 pictures were taken, so the picture sticks. Time between pictures after the first 5 s: under 20 ms 62, 20-50 ms 209, 50-90 ms 35, over 90 ms 156.

Run 2 timeline without it: within 0.06 s of the clock at every logged acquire up to 1440 (59.52 s / 59.53 s). Time between pictures after the first 5 s: 319, 775, 173, 123.

The user reported the same: "movies play ~10 s, then stick on one frame", on his installed build. The Hookshot build before #110 played them through.

Not decode speed: dav1d needs 6.7-9.7 ms a picture against a 16.7 ms budget. The limit's refusals (550+ a movie) seem to leave the decoder starved once it falls behind. The ~2,200 pictures still skipped without the limit come from the game's own rate during the movie on the Deck: about 24 acquires a second for a 60-picture-a-second movie. That's the frame rate, not a freeze, so "0 to about 10 skipped" can't happen on a Deck.

Workaround on the Deck: HOS_VIDEO_INPUT_LIMIT=0 in longshot_env.txt (now set on the user's Deck). Not tested: later movies, Windows. #252 (a slow movie shows its pictures) is being tested next on the Deck with the limit on.

**T-BW29 (Steam Deck): fails with the limit on.** On a Steam Deck the opening movie freezes about 14 s in and ends 7.5 s behind its sound. With `HOS_VIDEO_INPUT_LIMIT=0` it stays in time. **Setup:** Steam Deck OLED, SteamOS 3.9.5 (kernel 7.2.7-valve2), RADV Mesa 26.2.0-devel, libdav1d. Build: Longshot e7ed2f7a (with #110) plus Deck patches that don't touch `host/nn_video.c`. A copy of the user's install, started to the title, so the first movie plays (~3,200-3,700 pictures, 1920x1080, one every 16.7 ms). `HOS_VIDEO_LOG=1`. Two runs each, interleaved: on, off, on, off. | run | limit | pictures decoded | skipped | submissions put off | decode each | longest wait | most behind the clock | at the end | |---|---|---|---|---|---|---|---|---| | 1 | on (16) | 3221 | 2619 | 575 | 9.6 ms | 503 ms | 7.83 s | 7.518 s behind | | 2 | off (0) | 3646 | 2177 | 0 | 6.7 ms | 478 ms | 0.06 s | 0.019 s behind | | 3 | on (16) | 3270 | 2668 | 557 | 9.7 ms | 494 ms | 5.81 s | 7.541 s behind | | 4 | off (0) | 3685 | 2194 | 0 | 6.8 ms | 423 ms | 0.05 s | 0.014 s behind | **Run 1 timeline with the limit** (`acquire N: picture at ..., wall clock ...`): on time until acquire 420 (14.30 s / 14.31 s). The next logged one is acquire 600 at 34.73 s against 40.35 s on the wall clock (5.62 s behind), then 42.65 s / 49.56 s and 50.52 s / 58.35 s. Between 14 s and 40 s only ~180 pictures were taken, so the picture sticks. Time between pictures after the first 5 s: under 20 ms 62, 20-50 ms 209, 50-90 ms 35, over 90 ms 156. **Run 2 timeline without it:** within 0.06 s of the clock at every logged acquire up to 1440 (59.52 s / 59.53 s). Time between pictures after the first 5 s: 319, 775, 173, 123. The user reported the same: "movies play ~10 s, then stick on one frame", on his installed build. The Hookshot build before #110 played them through. **Not decode speed:** dav1d needs 6.7-9.7 ms a picture against a 16.7 ms budget. The limit's refusals (550+ a movie) seem to leave the decoder starved once it falls behind. The ~2,200 pictures still skipped without the limit come from the game's own rate during the movie on the Deck: about 24 acquires a second for a 60-picture-a-second movie. That's the frame rate, not a freeze, so "0 to about 10 skipped" can't happen on a Deck. **Workaround on the Deck:** `HOS_VIDEO_INPUT_LIMIT=0` in `longshot_env.txt` (now set on the user's Deck). Not tested: later movies, Windows. #252 (a slow movie shows its pictures) is being tested next on the Deck with the limit on.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!110
No description provided.