feat(control): drive a running game over a local socket, with a CLI #926

Merged

hckrman101 merged 21 commits from control-bridge into main

2026-11-11 16:18:39 +00:00

hckrman101 commented

2026-11-09 01:38:21 +00:00

(Migrated from github.com)

Tests and tools that drive a running game today write presses into a file the game polls and scrape hookshot_log.txt for its state, timed in wall-clock seconds. This adds a local, opt-in control socket so a script can press buttons, wait for the game, take screenshots, teleport and change settings over one connection, with replies timed in frames.

What it is

HOS_CONTROL=<socket path> starts a small bridge inside the game process; without it nothing listens. One JSON request a line, one reply a line (with the request's id and the current frame); subscription events on the same connection. scripts/hookshot_ctl.py is a standard-library client with a launch helper. The protocol, the security model, examples and the known limits are in docs/control.md (linked from docs/development.md).

command	what
`status`	frame, fps, draws a frame, a state guessed from the draws (boot/intro/menu/loading/play), the renderer's presents, the window, and for OoT the current map, whether an event is playing, whether a map is loading, Link's age
`press` / `stick` / `sequence`	input timed in frames from the next controller read; `HOS_INPUT_SCRIPT`'s syntax accepted; `exclusive` keeps the player's devices out; the reply when the input is over (or at once with `"wait": false`)
`release`	lets go of the caller's inputs and those left by clients that have gone; `"all": true` for every client's
`wait`	`present` (N frames), `frame`, `in_play`, `state`, `loading_start`, `loading_end`, `change`, `map`
`shot`	a PNG of the next present (or the first after a frame), written whole then renamed
`option_get` / `option_set`	the Insert menu's settings rows by their `hookshot_options.ini` key, through the menu's own setter
`teleport` (OoT)	the Debug tab's teleport by destination index or map name. The reply says whether the game sent it (and to which map) or refused it (and why); no argument lists the destinations
`saves`	the save folder the game uses, its slots newest first
`subscribe`	presents (every N frames), state changes, map changes, log lines matching a filter
`mem_read` / `mem_read_many` / `mem_write`	guest memory, only with a second opt-in: `HOS_CONTROL_MEM=ro` or `rw`
`quit`	reply, then the same exit as closing the window

Hooks outside the bridge are small: host/nn_hid.c (the bridge's input in the controller read), host/hos_main.c (start; the current map, the state flags, and the teleport request and its outcome beside the existing debug teleport), host/hos_thread.c (log lines), host/hos_mem.c (bounds for guest memory), host/nn_fs.c (the save folder), host/rx_backend.c and gpu/vk/vk_api.h (four renderer functions: frame_info, shot, setting, setting_set; MKWVK_API_VERSION 14 -> 15), gpu/vk/vk_replay.cpp (shots at a present, with their own image and readback, never waited for), gpu/vk/vk_menu.cpp (settings by key, under a lock menu_tick also holds).

Design choices

Frames, not seconds. Inputs start at the first controller read after the request and last N reads; every reply carries the frame. Scripts no longer depend on how fast the PC is.
The game never waits for a client. The bridge has its own thread for all socket I/O; the controller read takes one short lock that is never held over I/O. A client that stops reading is dropped once 8 MB of replies wait for it.
Settings by key, not by position. Rows come and go with the GPU, HDR and the debug tools, so a row is named by its hookshot_options.ini key and changed through the menu's own setter (presets, restart notice and the options file behave as with the arrow keys).
Teleports answer. The game can refuse a teleport (an event playing, a map loading, one every 8 s, no map for Link's age); the reply says so with the reason, instead of a wait map running to its timeout. A teleport the game does not take in time (not in play) is withdrawn, so it does not happen later behind the script's back.
Inputs belong to a client. release lets go of the caller's own inputs (and of those left by clients that are gone), not another script's. An input sent without waiting stays after its client leaves, so a one-shot hookshot_ctl.py press --no-wait works, and is freed once over.
Guest memory is a separate opt-in and is read and written on the game thread between frames, so a read is one frame's memory and a write lands between frames.

Security model

Off unless HOS_CONTROL is set. Guest memory is a second opt-in (HOS_CONTROL_MEM=ro|rw); anything else leaves it off. Every write and setting change is logged.
Local only, this user only. Linux: a Unix socket (no TCP, no abstract name) created 0600 with the umask set around bind, in a folder that must be owned by the user and writable by no one else (a shared /tmp is refused), and SO_PEERCRED checked on each client. A file in the way that is not a socket is never replaced; a live socket is left alone. Windows: a named pipe with a DACL for the current user alone, FILE_FLAG_FIRST_PIPE_INSTANCE and PIPE_REJECT_REMOTE_CLIENTS.
Bounded: 64 KB a request line, 200-character ids, 8 clients, 64 waiting requests, 256 queued inputs, 1 MB a memory request, 8 MB of unread replies before a client is dropped. Guest memory outside the guest window and the image, the system region, the heap or an address region is refused, not faulted. Numbers past int64 are refused before any conversion; memory addresses given as strings must be decimal or 0x hex digits.
A memory request is handed to the game thread only once it is complete (its ranges and buffer filled in), under the bridge's lock.
The client checks what the game sends back before using it as a path (slots --thumbs), shows the game's control characters as ? in plain output, and launch --save refuses source saves holding links or special files.
With the Landlock sandbox proposal (#882): the bridge thread starts before the restriction, so it is covered only with Landlock ABI 8 (TSYNC); on older kernels it would be outside the sandbox, like the window's main thread there. If both land, the bridge should start after the restriction (its folder allowed) or stay off without ABI 8. This PR does not depend on #882.

Tested

Linux (CachyOS, NVIDIA).

Live, against the real game, built from this branch with main 364d1994 merged in (headless, HOS_NO_SDL=1, a copy of one save slot made by launch --save --slot 04, HOS_CONTROL_MEM=rw), 24 of 24 checks:
- The run folder was 0700 and the socket 0600. option_get before the renderer was up: unavailable; right after the first frame: answered at once.
- A presses until play (boot -> intro -> menu -> loading -> play, Kokiri Forest); status with map and game (event, loading, age).
- 300 one-frame presses sent without waiting (more than the 256 the queue holds), then input.queued back to 0.
- release from one client left another client's held input alone; release with all let it go.
- The teleport list (31 destinations). teleport 1: sent, to HyruleField_Child, accepted. A second teleport at once: refused, "another teleport is still in progress (one every 8 s)". wait map with the map's name in lower case answered on arrival. In an earlier run, a teleport sent while the arrival event played was answered refused, "an event is playing". teleport 0 --arrive 120 (the client): sent, and in Kokiri Forest 182 frames later.
- Guest memory, rw: the heap's mapped end found by reads (one page past it: out_of_range), 8 bytes written there, read back, then the old bytes written back and read back; an address string "-1" refused bad_request; mem read of the image as u32.
- option set fps_counter on and off; shots (named, and the default folder); a client that shut its sending side after a wait got its reply, then the close; watch with present events; quit: the game gone 1.5 s later, the socket removed. The source save was not changed.
- The live runs found one bug, fixed here: between the game's update taking a teleport and recording what came of it, the bridge answered cancelled for a teleport the game then sent. The fake game in the tests now takes and finishes in two steps, as the game does.
tests/test_control.py: the bridge built with a fake game and driven over a real socket. Covered: every command, malformed and oversized input, numbers past int64, too-long ids, memory address strings, disconnects mid-press, too many clients, a client that never reads, a half-closed client, the socket's mode, a file in the way, a shared folder refused, map/teleport (sent, refused, not taken, replaced)/state, presses without waiting (more than the queue holds), release per client and for all, memory at each level, quit. Also clean under AddressSanitizer and UndefinedBehaviorSanitizer (clang).
tests/test_hookshot_ctl.py: the client against a fake bridge. Covered: every command's request and output, exit codes, watch, launch, teleport (--arrive, --no-wait, refused), release --all, how long the client waits for a long input, --save with a planted link refused, a hostile slot listing refused.
tests/run_all.py on the final head (main 2e62a977 merged in): 358 passed, 4 failed, 13 skipped. The same four fail on clean main here, for reasons of this PC: test_nup_read_name and test_bake_portability (no zstandard module), test_builder_log_read_only (no Tk library), test_wayland_libdecor_hint (clang 23's -Wunused-but-set-global in its own harness). The bridge's two checks pass.
hookshot-build: a full game build (the one the live runs used), and hookshot_vk.so of the final head with the ABI check (no undefined mkwvk symbols). The final merge of main (5 commits: release, builder, HDR/FSR, vcores, tests) had no conflicts and was not run in the game again.
Windows: host/control.c and tests/control_test.c compile and link with llvm-mingw 20261104 at -Wall -Wextra -Werror.

Known limits

Windows: host/control.c and tests/control_test.c compile and link with llvm-mingw 20261104 (clang 23) at -Wall -Wextra -Werror, and the pipe tests run in the windows-2022 CI job, but they were not run here. Untested: the bridge inside a Windows game build, launch on Windows, and refusing remote pipe clients.
The state is a heuristic from draw counts; scenes with unusual counts could be misread. For OoT, status's game flags and map are the game's own values.
A teleport the game sent but that did not move Link: the reply's accepted is the game's event manager's answer to the request; false was seen once, when the previous teleport's arrival event was still playing and the map did not change. Whether false always means no move is not established, so it is reported, not turned into an error.
The first settings request after start can wait, on the bridge's thread, for the start-up check of the upscalers (NVIDIA's NGX start, about 3 s). The game's frames do not wait for it.
HOS_CONTROL_MEM=rw was checked against the game's memory (a write read back, then put back), not against what the GPU draws from a written value.
Named game values for mem (Link's position, health, ...) are not in this PR.

Tests and tools that drive a running game today write presses into a file the game polls and scrape `hookshot_log.txt` for its state, timed in wall-clock seconds. This adds a local, opt-in control socket so a script can press buttons, wait for the game, take screenshots, teleport and change settings over one connection, with replies timed in frames. ## What it is `HOS_CONTROL=<socket path>` starts a small bridge inside the game process; without it nothing listens. One JSON request a line, one reply a line (with the request's `id` and the current frame); subscription events on the same connection. `scripts/hookshot_ctl.py` is a standard-library client with a `launch` helper. The protocol, the security model, examples and the known limits are in `docs/control.md` (linked from `docs/development.md`). | command | what | |---|---| | `status` | frame, fps, draws a frame, a state guessed from the draws (boot/intro/menu/loading/play), the renderer's presents, the window, and for OoT the current map, whether an event is playing, whether a map is loading, Link's age | | `press` / `stick` / `sequence` | input timed in frames from the next controller read; `HOS_INPUT_SCRIPT`'s syntax accepted; `exclusive` keeps the player's devices out; the reply when the input is over (or at once with `"wait": false`) | | `release` | lets go of the caller's inputs and those left by clients that have gone; `"all": true` for every client's | | `wait` | `present` (N frames), `frame`, `in_play`, `state`, `loading_start`, `loading_end`, `change`, `map` | | `shot` | a PNG of the next present (or the first after a frame), written whole then renamed | | `option_get` / `option_set` | the Insert menu's settings rows by their `hookshot_options.ini` key, through the menu's own setter | | `teleport` (OoT) | the Debug tab's teleport by destination index or map name. The reply says whether the game sent it (and to which map) or refused it (and why); no argument lists the destinations | | `saves` | the save folder the game uses, its slots newest first | | `subscribe` | presents (every N frames), state changes, map changes, log lines matching a filter | | `mem_read` / `mem_read_many` / `mem_write` | guest memory, only with a second opt-in: `HOS_CONTROL_MEM=ro` or `rw` | | `quit` | reply, then the same exit as closing the window | Hooks outside the bridge are small: `host/nn_hid.c` (the bridge's input in the controller read), `host/hos_main.c` (start; the current map, the state flags, and the teleport request and its outcome beside the existing debug teleport), `host/hos_thread.c` (log lines), `host/hos_mem.c` (bounds for guest memory), `host/nn_fs.c` (the save folder), `host/rx_backend.c` and `gpu/vk/vk_api.h` (four renderer functions: `frame_info`, `shot`, `setting`, `setting_set`; `MKWVK_API_VERSION` 14 -> 15), `gpu/vk/vk_replay.cpp` (shots at a present, with their own image and readback, never waited for), `gpu/vk/vk_menu.cpp` (settings by key, under a lock `menu_tick` also holds). ## Design choices - **Frames, not seconds.** Inputs start at the first controller read after the request and last N reads; every reply carries the frame. Scripts no longer depend on how fast the PC is. - **The game never waits for a client.** The bridge has its own thread for all socket I/O; the controller read takes one short lock that is never held over I/O. A client that stops reading is dropped once 8 MB of replies wait for it. - **Settings by key, not by position.** Rows come and go with the GPU, HDR and the debug tools, so a row is named by its `hookshot_options.ini` key and changed through the menu's own setter (presets, restart notice and the options file behave as with the arrow keys). - **Teleports answer.** The game can refuse a teleport (an event playing, a map loading, one every 8 s, no map for Link's age); the reply says so with the reason, instead of a `wait map` running to its timeout. A teleport the game does not take in time (not in play) is withdrawn, so it does not happen later behind the script's back. - **Inputs belong to a client.** `release` lets go of the caller's own inputs (and of those left by clients that are gone), not another script's. An input sent without waiting stays after its client leaves, so a one-shot `hookshot_ctl.py press --no-wait` works, and is freed once over. - **Guest memory is a separate opt-in** and is read and written on the game thread between frames, so a read is one frame's memory and a write lands between frames. ## Security model - Off unless `HOS_CONTROL` is set. Guest memory is a second opt-in (`HOS_CONTROL_MEM=ro|rw`); anything else leaves it off. Every write and setting change is logged. - Local only, this user only. Linux: a Unix socket (no TCP, no abstract name) created 0600 with the umask set around `bind`, in a folder that must be owned by the user and writable by no one else (a shared `/tmp` is refused), and `SO_PEERCRED` checked on each client. A file in the way that is not a socket is never replaced; a live socket is left alone. Windows: a named pipe with a DACL for the current user alone, `FILE_FLAG_FIRST_PIPE_INSTANCE` and `PIPE_REJECT_REMOTE_CLIENTS`. - Bounded: 64 KB a request line, 200-character ids, 8 clients, 64 waiting requests, 256 queued inputs, 1 MB a memory request, 8 MB of unread replies before a client is dropped. Guest memory outside the guest window and the image, the system region, the heap or an address region is refused, not faulted. Numbers past int64 are refused before any conversion; memory addresses given as strings must be decimal or `0x` hex digits. - A memory request is handed to the game thread only once it is complete (its ranges and buffer filled in), under the bridge's lock. - The client checks what the game sends back before using it as a path (`slots --thumbs`), shows the game's control characters as `?` in plain output, and `launch --save` refuses source saves holding links or special files. - With the Landlock sandbox proposal (#882): the bridge thread starts before the restriction, so it is covered only with Landlock ABI 8 (`TSYNC`); on older kernels it would be outside the sandbox, like the window's main thread there. If both land, the bridge should start after the restriction (its folder allowed) or stay off without ABI 8. This PR does not depend on #882. ## Tested Linux (CachyOS, NVIDIA). - **Live, against the real game**, built from this branch with main 364d1994 merged in (headless, `HOS_NO_SDL=1`, a copy of one save slot made by `launch --save --slot 04`, `HOS_CONTROL_MEM=rw`), 24 of 24 checks: - The run folder was 0700 and the socket 0600. `option_get` before the renderer was up: `unavailable`; right after the first frame: answered at once. - A presses until play (boot -> intro -> menu -> loading -> play, Kokiri Forest); `status` with `map` and `game` (`event`, `loading`, `age`). - 300 one-frame presses sent without waiting (more than the 256 the queue holds), then `input.queued` back to 0. - `release` from one client left another client's held input alone; `release` with `all` let it go. - The teleport list (31 destinations). `teleport 1`: `sent`, to `HyruleField_Child`, `accepted`. A second teleport at once: `refused`, "another teleport is still in progress (one every 8 s)". `wait map` with the map's name in lower case answered on arrival. In an earlier run, a teleport sent while the arrival event played was answered `refused`, "an event is playing". `teleport 0 --arrive 120` (the client): sent, and in Kokiri Forest 182 frames later. - Guest memory, `rw`: the heap's mapped end found by reads (one page past it: `out_of_range`), 8 bytes written there, read back, then the old bytes written back and read back; an address string `"-1"` refused `bad_request`; `mem read` of the image as u32. - `option set fps_counter` on and off; shots (named, and the default folder); a client that shut its sending side after a `wait` got its reply, then the close; `watch` with present events; `quit`: the game gone 1.5 s later, the socket removed. The source save was not changed. - The live runs found one bug, fixed here: between the game's update taking a teleport and recording what came of it, the bridge answered `cancelled` for a teleport the game then sent. The fake game in the tests now takes and finishes in two steps, as the game does. - `tests/test_control.py`: the bridge built with a fake game and driven over a real socket. Covered: every command, malformed and oversized input, numbers past int64, too-long ids, memory address strings, disconnects mid-press, too many clients, a client that never reads, a half-closed client, the socket's mode, a file in the way, a shared folder refused, map/teleport (sent, refused, not taken, replaced)/state, presses without waiting (more than the queue holds), release per client and for all, memory at each level, quit. Also clean under AddressSanitizer and UndefinedBehaviorSanitizer (clang). - `tests/test_hookshot_ctl.py`: the client against a fake bridge. Covered: every command's request and output, exit codes, watch, launch, teleport (`--arrive`, `--no-wait`, refused), `release --all`, how long the client waits for a long input, `--save` with a planted link refused, a hostile slot listing refused. - `tests/run_all.py` on the final head (main 2e62a977 merged in): 358 passed, 4 failed, 13 skipped. The same four fail on clean main here, for reasons of this PC: `test_nup_read_name` and `test_bake_portability` (no `zstandard` module), `test_builder_log_read_only` (no Tk library), `test_wayland_libdecor_hint` (clang 23's `-Wunused-but-set-global` in its own harness). The bridge's two checks pass. - `hookshot-build`: a full game build (the one the live runs used), and `hookshot_vk.so` of the final head with the ABI check (no undefined `mkwvk` symbols). The final merge of main (5 commits: release, builder, HDR/FSR, vcores, tests) had no conflicts and was not run in the game again. - Windows: `host/control.c` and `tests/control_test.c` compile and link with llvm-mingw 20261104 at `-Wall -Wextra -Werror`. ## Known limits - **Windows**: `host/control.c` and `tests/control_test.c` compile and link with llvm-mingw 20261104 (clang 23) at `-Wall -Wextra -Werror`, and the pipe tests run in the windows-2022 CI job, but they were not run here. Untested: the bridge inside a Windows game build, `launch` on Windows, and refusing remote pipe clients. - **The state is a heuristic** from draw counts; scenes with unusual counts could be misread. For OoT, `status`'s `game` flags and `map` are the game's own values. - **A teleport the game sent but that did not move Link**: the reply's `accepted` is the game's event manager's answer to the request; `false` was seen once, when the previous teleport's arrival event was still playing and the map did not change. Whether `false` always means no move is not established, so it is reported, not turned into an error. - **The first settings request after start** can wait, on the bridge's thread, for the start-up check of the upscalers (NVIDIA's NGX start, about 3 s). The game's frames do not wait for it. - `HOS_CONTROL_MEM=rw` was checked against the game's memory (a write read back, then put back), not against what the GPU draws from a written value. - Named game values for `mem` (Link's position, health, ...) are not in this PR.

NotRexed commented

2026-11-09 05:31:04 +00:00

(Migrated from github.com)

Thank you for this! It changes how the game runs at a level that needs a maintainer design decision first, so I'm closing it to keep the queue clear: needs maintainer design decision, reopen after discussion. (A local control socket that can read and write guest memory is a new interface: needs a design discussion first.)