Feed the microphone to the ocarina's input streams, with a picker in the Insert menu #937

2026-11-09T02:03:12Z

AR51AurA commented

2026-11-09 02:03:12 +00:00

(Migrated from github.com)

What this adds

The ocarina's microphone mode (play an instrument, or hum) didn't work: the game's microphone functions were placeholder stubs, so the game never received any sound. This PR implements them on SDL's recording devices (SDL3's, or SDL2's capture devices, as the Linux AppImage carries SDL2), and adds a Microphone row to the Insert menu's Gameplay tab:

Off: the game receives silence.
Default: follows Windows' default recording device.
Any named microphone: each recording device Windows knows about is listed by name.

The choice applies straight away and is saved in hookshot_options.ini by device name (microphone_device=), so it survives devices being reordered.

What the game does (from how it calls these functions)

At startup, the game calls nn::audio::ListVirtualInputDevices. With the stub returning 0 devices, it never opened an input stream.
- When it finds a device, it opens two streams: one on that device, one on GetTrueRawAudioInputDevice.
- Both use the same StreamFormat {PcmInt16, 1 channel, 48000 Hz} and a capacity of 4800 frames.
With the ocarina's microphone mode on:
- It starts the streams with StartInputStream when GetInputStreamState != 2, so 2 means started.
- Each frame it calls GetInputStreamFilledFrameCount, then ReadInputStream for up to 256 frames.
- It also calls GetInputStreamStatistics(stream, true). The result comes back through x8, and the game checks the second 64-bit word against 1.
The game works out the notes itself, so the host only has to supply raw samples.

Implementation

host/nn_audio_in.c (new):
- Device functions: ListVirtualInputDevices (always 1 device, as on the console), GetDefaultVirtualInputDevice, GetTrueRawAudioInputDevice and IsVirtualInputDeviceStreamFormatSupported.
- Stream functions: Open (the overload the game uses), Close, Start, Stop, Purge, GetState, Get{Filled,Free,Capacity}FrameCount, Read, GetStatistics (all zeros: nothing lost), and gain get/set.
- Each stream keeps a handle in the first 8 bytes of the game's InputStreamType, as nn_audio.c does for its own objects.
- A started stream reads an SDL recording stream, converted to the stream's format (SDL resamples and downmixes).
- If the game stops reading, anything beyond the stream's capacity is dropped, so audio doesn't build up seconds of delay.
- The started streams share one source: the microphone is opened once and each stream gets its own copy of what it records.
- A chosen microphone is always looked up by name in a list made just before opening it (SDL2's ids are places in its list, and the menu's list can be old). Opened again under a new id, the source keeps that id.
- A chosen microphone that is missing, does not open, stops sending, or is no longer listed: the default stands in (silence if there is none) and it is tried again after 3 s, then twice as long each time up to a minute, said once in the log.
- While open, a chosen microphone is looked for in a fresh list every 5 s, and at once when the menu shows it as not connected: one that is pulled out but goes on sending zeros (SDL3 on Windows) is treated as gone.
- With the microphone Off or no SDL, the stream produces silence in real time, so the game keeps reading as it would on the console.
host/hos_sdl.c / .h: hos_sdl_mic_*, loaded as optional: on SDL3 a recording stream per microphone; on SDL2 (the Linux AppImage's) a capture device whose callback fills a ring of about a second, read under the device's lock. Both convert to the stream's format.
gpu/vk/vk_api.h: MkwVkHost gains mic_count, mic_name and mic_select (the menu's choice: off, the default, or a microphone by its whole name and place in the list), and MKWVK_API_VERSION goes up by one.
host/rx_backend.c: connects hos_mic_count, hos_mic_name and hos_mic_select to the menu.
gpu/vk/vk_menu.cpp, vk_menu.h:
- The new row lists the devices again each time the menu opens, keeping the chosen one by name. At the start they're only listed when one was chosen by name (Off and Default don't start SDL audio for it).
- The name saved in the ini is cut to 60 bytes at a UTF-8 boundary (other readers of the file read 80-byte lines); the host is told the listed device's whole name, and takes a kept name as the start of the device's.
- A chosen microphone that is unplugged stays chosen, shown as "(not connected)", and is used again when it's back.
- Changing it doesn't switch the quality preset to Custom.
readme.md: the row in the Insert menu's table, and a note under it.
HOS_MIC_LOG=1 (off by default, documented at the top of nn_audio_in.c and in the readme) logs a line a second while a stream runs: frames read and peak level.

Testing

Game-free check: tests/test_audio_in.py (run by tests/run_all.py, with --cc gcc as the other host C checks). It compiles host/nn_audio_in.c with stand-ins for SDL, the menu and the clock, and drives the streams as the game does. It covers the device list, open, start (state 2), silence at the stream's rate while Off, the microphone's samples once a device is picked, dropping past the capacity, purge, stop and close, a shared source, two devices with one name, names past 60 bytes, a device pulled out that keeps sending zeros, one that stalls, and the retries (46 cases). It also builds the menu's microphone code from vk_menu.cpp and checks duplicates, unplug and replug, Off/Default, and the UTF-8 cut.
In game: v2.0.5 plus this change, Windows, AMD Radeon RX 9070 XT (Vulkan driver 2.0.406), microphone: a Focusrite USB audio interface (input "Analogue 1 + 2").
Unplugging and replugging, the stand-in and the zero-sending device are covered by the test, not yet tried in game.
Linux: the SDL2 path compiles against the POSIX shims and its ring buffer was checked on its own, but it has not been played on a Linux machine yet.
Humming and whistling: the ocarina plays back the song hummed or whistled, in place of playing it on the instrument.
The game opens both streams at its start (and again later, after closing them), but the microphone itself only opens when the game starts listening, and closes when it stops: several start/stop cycles in one session, and nothing recorded in between:

audio in: stream 0 opened: format 2, 1 channel, 48000 Hz, 4800 frames
audio in: stream 1 opened (raw): format 2, 1 channel, 48000 Hz, 4800 frames
...
audio in: stream 0: Analogue 1 + 2 (Focusrite USB Audio) open (48000 Hz, 1 channel)
audio in: stream 1 (raw): Analogue 1 + 2 (Focusrite USB Audio) open (48000 Hz, 1 channel)
audio in: stream 0 stopped (641726 frames read)
audio in: stream 1 stopped (638430 frames read)

Notes

The menu's default is Default rather than Off, because the game itself only opens the microphone while its ocarina microphone mode is in use: nothing is recorded until the player turns that on in the game, which is what the console does. Off is one step to the left.
Windows must allow desktop apps to use the microphone (Settings → Privacy & security → Microphone). Otherwise SDL gets silence.
The streams' state values (1 stopped, 2 started) and the statistics layout come from how the game uses them, not from SDK documentation.

Fixes #189

### What this adds The ocarina's microphone mode (play an instrument, or hum) didn't work: the game's microphone functions were placeholder stubs, so the game never received any sound. This PR implements them on SDL's recording devices (SDL3's, or SDL2's capture devices, as the Linux AppImage carries SDL2), and adds a **Microphone** row to the Insert menu's Gameplay tab: - **Off:** the game receives silence. - **Default:** follows Windows' default recording device. - **Any named microphone:** each recording device Windows knows about is listed by name. The choice applies straight away and is saved in `hookshot_options.ini` by device name (`microphone_device=`), so it survives devices being reordered. ### What the game does (from how it calls these functions) - At startup, the game calls `nn::audio::ListVirtualInputDevices`. With the stub returning 0 devices, it never opened an input stream. - When it finds a device, it opens two streams: one on that device, one on `GetTrueRawAudioInputDevice`. - Both use the same `StreamFormat` `{PcmInt16, 1 channel, 48000 Hz}` and a capacity of 4800 frames. - With the ocarina's microphone mode on: - It starts the streams with `StartInputStream` when `GetInputStreamState != 2`, so 2 means started. - Each frame it calls `GetInputStreamFilledFrameCount`, then `ReadInputStream` for up to 256 frames. - It also calls `GetInputStreamStatistics(stream, true)`. The result comes back through x8, and the game checks the second 64-bit word against 1. - The game works out the notes itself, so the host only has to supply raw samples. ### Implementation - **`host/nn_audio_in.c` (new):** - Device functions: `ListVirtualInputDevices` (always 1 device, as on the console), `GetDefaultVirtualInputDevice`, `GetTrueRawAudioInputDevice` and `IsVirtualInputDeviceStreamFormatSupported`. - Stream functions: `Open` (the overload the game uses), `Close`, `Start`, `Stop`, `Purge`, `GetState`, `Get{Filled,Free,Capacity}FrameCount`, `Read`, `GetStatistics` (all zeros: nothing lost), and gain get/set. - Each stream keeps a handle in the first 8 bytes of the game's `InputStreamType`, as `nn_audio.c` does for its own objects. - A started stream reads an SDL recording stream, converted to the stream's format (SDL resamples and downmixes). - If the game stops reading, anything beyond the stream's capacity is dropped, so audio doesn't build up seconds of delay. - The started streams share one source: the microphone is opened once and each stream gets its own copy of what it records. - A chosen microphone is always looked up by name in a list made just before opening it (SDL2's ids are places in its list, and the menu's list can be old). Opened again under a new id, the source keeps that id. - A chosen microphone that is missing, does not open, stops sending, or is no longer listed: the default stands in (silence if there is none) and it is tried again after 3 s, then twice as long each time up to a minute, said once in the log. - While open, a chosen microphone is looked for in a fresh list every 5 s, and at once when the menu shows it as not connected: one that is pulled out but goes on sending zeros (SDL3 on Windows) is treated as gone. - With the microphone Off or no SDL, the stream produces silence in real time, so the game keeps reading as it would on the console. - **`host/hos_sdl.c` / `.h`:** `hos_sdl_mic_*`, loaded as optional: on SDL3 a recording stream per microphone; on SDL2 (the Linux AppImage's) a capture device whose callback fills a ring of about a second, read under the device's lock. Both convert to the stream's format. - **`gpu/vk/vk_api.h`:** `MkwVkHost` gains `mic_count`, `mic_name` and `mic_select` (the menu's choice: off, the default, or a microphone by its whole name and place in the list), and `MKWVK_API_VERSION` goes up by one. - **`host/rx_backend.c`:** connects `hos_mic_count`, `hos_mic_name` and `hos_mic_select` to the menu. - **`gpu/vk/vk_menu.cpp`, `vk_menu.h`:** - The new row lists the devices again each time the menu opens, keeping the chosen one by name. At the start they're only listed when one was chosen by name (Off and Default don't start SDL audio for it). - The name saved in the ini is cut to 60 bytes at a UTF-8 boundary (other readers of the file read 80-byte lines); the host is told the listed device's whole name, and takes a kept name as the start of the device's. - A chosen microphone that is unplugged stays chosen, shown as "(not connected)", and is used again when it's back. - Changing it doesn't switch the quality preset to Custom. - `readme.md`: the row in the Insert menu's table, and a note under it. - `HOS_MIC_LOG=1` (off by default, documented at the top of `nn_audio_in.c` and in the readme) logs a line a second while a stream runs: frames read and peak level. ### Testing - Game-free check: `tests/test_audio_in.py` (run by `tests/run_all.py`, with `--cc gcc` as the other host C checks). It compiles `host/nn_audio_in.c` with stand-ins for SDL, the menu and the clock, and drives the streams as the game does. It covers the device list, open, start (state 2), silence at the stream's rate while Off, the microphone's samples once a device is picked, dropping past the capacity, purge, stop and close, a shared source, two devices with one name, names past 60 bytes, a device pulled out that keeps sending zeros, one that stalls, and the retries (46 cases). It also builds the menu's microphone code from `vk_menu.cpp` and checks duplicates, unplug and replug, Off/Default, and the UTF-8 cut. - In game: v2.0.5 plus this change, Windows, AMD Radeon RX 9070 XT (Vulkan driver 2.0.406), microphone: a Focusrite USB audio interface (input "Analogue 1 + 2"). - Unplugging and replugging, the stand-in and the zero-sending device are covered by the test, not yet tried in game. - Linux: the SDL2 path compiles against the POSIX shims and its ring buffer was checked on its own, but it has not been played on a Linux machine yet. - Humming and whistling: the ocarina plays back the song hummed or whistled, in place of playing it on the instrument. - The game opens both streams at its start (and again later, after closing them), but the microphone itself only opens when the game starts listening, and closes when it stops: several start/stop cycles in one session, and nothing recorded in between: ``` audio in: stream 0 opened: format 2, 1 channel, 48000 Hz, 4800 frames audio in: stream 1 opened (raw): format 2, 1 channel, 48000 Hz, 4800 frames ... audio in: stream 0: Analogue 1 + 2 (Focusrite USB Audio) open (48000 Hz, 1 channel) audio in: stream 1 (raw): Analogue 1 + 2 (Focusrite USB Audio) open (48000 Hz, 1 channel) audio in: stream 0 stopped (641726 frames read) audio in: stream 1 stopped (638430 frames read) ``` ### Notes - The menu's default is **Default** rather than Off, because the game itself only opens the microphone while its ocarina microphone mode is in use: nothing is recorded until the player turns that on in the game, which is what the console does. Off is one step to the left. - Windows must allow desktop apps to use the microphone (Settings → Privacy & security → Microphone). Otherwise SDL gets silence. - The streams' state values (1 stopped, 2 started) and the statistics layout come from how the game uses them, not from SDK documentation. Fixes #189

hckrman101 (Migrated from github.com) reviewed

2026-11-09 07:40:34 +00:00

hckrman101 (Migrated from github.com) left a comment

Fun feature! A few issues, the first three being real bugs in the device handling: same-name devices, a mic unplugged mid-game, and the saved device being lost when it's missing at start. The rest are smaller.

gpu/vk/vk_menu.cpp

The chosen mic is matched by name and the first match wins, so of two devices with the same name (two USB Audio Devices is common) the second can never stay selected: pick it (opts.mic=3), and the next menu open (or the start's refresh_mics(&mic_saved_name)) moves it back to index 0, quietly switching the ocarina to the other mic. Preferring the current index when its name still matches, and only searching otherwise, would fix it.

The chosen mic is matched by name and the first match wins, so of two devices with the same name (two `USB Audio Device`s is common) the second can never stay selected: pick it (`opts.mic=3`), and the next menu open (or the start's `refresh_mics(&mic_saved_name)`) moves it back to index 0, quietly switching the ocarina to the other mic. Preferring the current index when its name still matches, and only searching otherwise, would fix it.

gpu/vk/vk_menu.cpp

					
				@ -426,6 +461,52 @@ int detect_pad_layout() {

				    if (a == HOS_PAD_SOUTH && b == HOS_PAD_EAST && x == HOS_PAD_NORTH && y == HOS_PAD_WEST) return 2;

				    return 3;

				}

If the saved mic is absent at start, refresh_mics falls back to Default (opts.mic = 1), and the next save() (any option change) writes no microphone_device line, so the choice is lost for good: plug the headset back in and the game stays on Default. Keeping mic_saved_name and writing it back while the device is missing would preserve it.

If the saved mic is absent at start, `refresh_mics` falls back to Default (`opts.mic = 1`), and the next `save()` (any option change) writes no `microphone_device` line, so the choice is lost for good: plug the headset back in and the game stays on Default. Keeping `mic_saved_name` and writing it back while the device is missing would preserve it.

gpu/vk/vk_menu.cpp

					
				@ -429,0 +490,4 @@

				            if (mic_key(host->mic_name(i)) == mic_want) at = i;

				        opts.mic = at >= 0 ? at + 2 : 2 + mic_listed;

				    } else if (opts.mic >= 2) {

				        opts.mic = 1;

nit: this chain of "not the preset's" pointer comparisons grows by one per non-graphics option, and forgetting one silently resets the preset to Custom. A preset_exempt flag on Row, set in the rows[] table, would scale better.

nit: this chain of "not the preset's" pointer comparisons grows by one per non-graphics option, and forgetting one silently resets the preset to Custom. A `preset_exempt` flag on `Row`, set in the `rows[]` table, would scale better.

gpu/vk/vk_menu.cpp

This reader got a bigger buffer for microphone_device, but the ini's other readers still use char line[80] (host/nn_hid.c binds_load, host/nn_applet.c, host/hos_main.c's ultrawide read). A long device name (PipeWire/Windows names easily pass 80 chars) gets split by fgets and its tail read as a separate line, and nn_applet.c's key reader has no whole-line guard. Bumping those buffers too (or skipping over-long lines) would avoid misparses.

This reader got a bigger buffer for `microphone_device`, but the ini's other readers still use `char line[80]` (`host/nn_hid.c` `binds_load`, `host/nn_applet.c`, `host/hos_main.c`'s ultrawide read). A long device name (PipeWire/Windows names easily pass 80 chars) gets split by `fgets` and its tail read as a separate line, and `nn_applet.c`'s key reader has no whole-line guard. Bumping those buffers too (or skipping over-long lines) would avoid misparses.

gpu/vk/vk_replay.cpp

nit: option 18 is a bare number here, in nn_audio_in.c (rx_option(18, 1)), the vk_api.h comment and the test. Adding it to the HOS_OPT_* enum like its neighbours would keep them from drifting apart.

nit: option `18` is a bare number here, in `nn_audio_in.c` (`rx_option(18, 1)`), the `vk_api.h` comment and the test. Adding it to the `HOS_OPT_*` enum like its neighbours would keep them from drifting apart.

				`@ -0,0 +1,501 @@`
				`/* (hookshot) nn::audio's input streams: the microphone, as the ocarina's "play with your voice / an instrument" option uses it.`

 @ -335,0 +426,4 @@
         /* (no changes allowed: SDL converts the device's sound to this. SDL2's id is a place in its list, so the caller takes
            it from a list made just before (host/nn_audio_in.c find_device). SDL2 opens a device by its name: of two with one
            name it takes the first, whichever was chosen; telling them apart is SDL3's) */
         const char *name = id ? ((const char *(*)(int, int))rec.DeviceName2)((int)id - 1, 1) : NULL;

 @ -0,0 +227,4 @@
     }
     src->mode = mode, src->want_id = want_id;
     snprintf(src->want, sizeof src->want, "%s", want);
     src->failures = 0, src->retry_wait = RETRY_NS;

 @ -0,0 +267,4 @@
                 hos_log("audio in: %s is back", wanted(src));
                 src->failures = 0, src->retry_wait = RETRY_NS;
             }
         } else if (now - src->heard_ns > QUIET_NS) {   /* (pulled out, or its driver stopped) */

Rows
Columns