MagicBeanDev/Hookshot

Archived

Watch

Fork 0

feat(runtime): add an opt-in local scene testing API #1543

Open

mikaba-eu wants to merge 17 commits from mikaba-eu/pr/runtime-scene-api into main

mikaba-eu commented

2026-11-10 07:31:11 +00:00

(Migrated from github.com)

Reproducing scene-transition stalls needs repeatable inputs and scene changes without touching the player's saves. This opt-in local testing API was implemented with GPT-6 Astra medium & xhigh.

The Linux API authenticates Unix-socket peers and bounds requests; the client checks server process identity. Analysis sessions copy saves and settings into private directories. Typed scene, event and scalar operations require locally supplied profiles tied to the exact game image and run on a configured guest worker. Input ownership, readiness checks, bounded wait diagnostics and sampler ownership fixes make these sessions easier to reproduce. Other platforms retain disabled API stubs.

No game-specific bindings, game data, saves, generated fixtures or build outputs are included. Renderer experiment switches and external DLSS support are optional integrations; their collectors and external layer files are not supplied by this PR. The API itself has no dependency on the renderer-performance PR. The present-CSV parser compatibility fix is identical in both PRs to keep either merge order compatible.

Refreshed against upstream 0a28ff29, retaining its present-phase timestamps, driver-cache initialization and virtual-core wait escape policy. The analyzer accepts both three-column and extended present traces; the presentation fixture checks one debug notification per completed native present.

Validation: full Linux source suite on 963dbd8a: 386 passed, 0 failed, 14 skipped. After merging 0a28ff29 and updating fixture declarations, 26 affected checks passed, 0 failed, 0 skipped on final head 134d7f78. The skips in the full run cover platform/game/GPU requirements and unavailable optional tools. Authored fixtures exercise the actual socket server, guest-call adapter, input ownership, cancellation and sampler handoff. No game or GPU was started for this refresh. Historical copied-save scene/event checks used the combined local tree, not this exact PR head. A fresh end-to-end scene transition, physical input devices and Windows runtime behavior remain unverified.

Final refresh: merged upstream ddafc1f9; 12 affected builder/update/fixture checks passed afterward.

The final source includes upstream 2e62a977; its release-only changes passed both affected release/update checks.

Reproducing scene-transition stalls needs repeatable inputs and scene changes without touching the player's saves. This opt-in local testing API was implemented with GPT-6 Astra medium & xhigh. The Linux API authenticates Unix-socket peers and bounds requests; the client checks server process identity. Analysis sessions copy saves and settings into private directories. Typed scene, event and scalar operations require locally supplied profiles tied to the exact game image and run on a configured guest worker. Input ownership, readiness checks, bounded wait diagnostics and sampler ownership fixes make these sessions easier to reproduce. Other platforms retain disabled API stubs. No game-specific bindings, game data, saves, generated fixtures or build outputs are included. Renderer experiment switches and external DLSS support are optional integrations; their collectors and external layer files are not supplied by this PR. The API itself has no dependency on the renderer-performance PR. The present-CSV parser compatibility fix is identical in both PRs to keep either merge order compatible. Refreshed against upstream `0a28ff29`, retaining its present-phase timestamps, driver-cache initialization and virtual-core wait escape policy. The analyzer accepts both three-column and extended present traces; the presentation fixture checks one debug notification per completed native present. Validation: full Linux source suite on `963dbd8a`: **386 passed, 0 failed, 14 skipped**. After merging `0a28ff29` and updating fixture declarations, **26 affected checks passed, 0 failed, 0 skipped** on final head `134d7f78`. The skips in the full run cover platform/game/GPU requirements and unavailable optional tools. Authored fixtures exercise the actual socket server, guest-call adapter, input ownership, cancellation and sampler handoff. No game or GPU was started for this refresh. Historical copied-save scene/event checks used the combined local tree, not this exact PR head. A fresh end-to-end scene transition, physical input devices and Windows runtime behavior remain unverified. Final refresh: merged upstream `ddafc1f9`; 12 affected builder/update/fixture checks passed afterward. The final source includes upstream `2e62a977`; its release-only changes passed both affected release/update checks.

hckrman101 commented

2026-11-10 17:49:36 +00:00

(Migrated from github.com)

Heads-up: this conflicts with main now (a lot landed today). Could you merge main into the branch (or rebase) and resolve the conflicts? Thanks!

mikaba-eu commented

2026-11-10 19:14:32 +00:00

(Migrated from github.com)

Merged current main (1392ea3) and resolved the conflicts, preserving upstream replacements where they supersede our changes. Migrated local tests to the new per-test runner declarations. Independent source suites and the combined suite pass; fresh Linux gameplay results and limitations are recorded in the updated description. Hyrule Field and a warm Forest run completed the short movement tests, but camera hitches remain; earlier memory-guard stops and the incomplete Market event are not reported as passes. AI-assisted development: GPT-6-Astra (medium and xhigh).

Merged current main (`1392ea3`) and resolved the conflicts, preserving upstream replacements where they supersede our changes. Migrated local tests to the new per-test runner declarations. Independent source suites and the combined suite pass; fresh Linux gameplay results and limitations are recorded in the updated description. Hyrule Field and a warm Forest run completed the short movement tests, but camera hitches remain; earlier memory-guard stops and the incomplete Market event are not reported as passes. AI-assisted development: GPT-6-Astra (medium and xhigh).

mikaba-eu commented

2026-11-10 19:39:32 +00:00

(Migrated from github.com)

Refreshed again to main 117eae7 and resolved the new Anti-Lag source-fixture conflict. Complete Linux source suite re-run: 329 passed, 0 failed, 18 skipped. Existing gameplay evidence remains explicitly tied to its tested build.

Refreshed again to main `117eae7` and resolved the new Anti-Lag source-fixture conflict. Complete Linux source suite re-run: 329 passed, 0 failed, 18 skipped. Existing gameplay evidence remains explicitly tied to its tested build.

Joey-Magnus (Migrated from github.com) requested changes

2026-11-11 20:06:14 +00:00

Joey-Magnus (Migrated from github.com) left a comment

There's real care in here. The socket server binds an abstract name and checks SO_PEERCRED for the same UID. It takes one bounded SEQPACKET command per connection, with 250 ms timeouts, a 512-byte request, and a reject for packets that are truncated or contain a NUL. Its parsers refuse overflow and trailing junk. Memory reads are capped at 256 bytes, kept inside the guest window, and go through process_vm_readv, so a bad address can't fault the game. With HOS_DEBUG_SOCKET unset, the Windows build gets inline stubs and the Linux build returns early.

Given the size, I focused on the socket server (host/hos_debug.c), the guest-call path (host/hos_debug_guest.c), and the edits to shared runtime files that every player runs (nn_hid.c, nn_os_sync.c, hos_thread.c, nn_video.c, posix/prof.c, hos_setjmp.c, display_sdl.c, rx_backend.c, mkw_glue.c). I didn't review the 14 Python scripts in depth.

blocking — one topic per PR (CONTRIBUTING, "Commits and pull requests" 2): besides the scene-testing API, this PR carries:

a fix for a race in the Linux signal sampler's buffer handoff (host/posix/prof.c, sample_one/on_sample);
sampler thread refresh between batches (posix/prof.c);
a .cfi_undefined %rip unwind fix for recomp_sj_longjmp (host/hos_setjmp.c:128);
HOS_START_MINIMIZED (hos_sdl.c, display_sdl.c);
HOS_REQUIRE_WINDOW (rx_backend.c:266);
virtual-core wait tracing (hos_thread.c, HOS_VCORE_TRACE_MS);
event-wait tracing (nn_os_sync.c, HOS_SYNC_WAIT_TRACE_MS);
movie acquire tracing (nn_video.c, HOS_VIDEO_ACQUIRE_TRACE);
clock markers in the all-thread sample file (mkw_glue.c);
present-trace and wait analyzers.

Several of these are good fixes in their own right; the sampler race and the longjmp unwind especially deserve to land quickly and on their own. Bundled here, they wait on review of a 7.6k-line feature, and a regression in any one would revert all of them. Please split them out: one PR per fix, one per trace switch (or one "wait diagnostics" PR), then the API on top.

blocking — conflicts: host/hos_main.c, host/nn_hid.c and host/nn_video.c conflict with current main. Note also that CI hasn't run on this head (title check only).

should fix — a test that never runs (tests/test_analysis_dlss.py, 328 lines): it loads tests/test_dlss5_api.py and scripts/dlss5_api.py and skipUnlesses on both, but neither file exists on main or in this PR. It always skips, including under CI's --strict, because a unittest skip isn't a missing declared tool. Drop it here and bring it with the DLSS adapter PR it belongs to.

should fix — a cost while off on every Linux run (host/hos_debug_guest.c:338): hos_debug_scene_poll is called from npad_states on every pad poll, and starts with getenv("HOS_DEBUG_SOCKET"), a scan of the environment several times a frame for every Linux player. CONTRIBUTING asks that switches cost nothing while off. Check the flag hos_debug_init already sets (expose enabled, or cache the getenv in a static) before anything else.

should fix — undocumented switches: HOS_START_MINIMIZED, HOS_REQUIRE_WINDOW, HOS_SYNC_WAIT_TRACE_MS, HOS_VIDEO_ACQUIRE_TRACE and HOS_DEBUG_CONTROL appear in no doc (HOS_START_MINIMIZED only in a source comment). They belong in docs/development.md's switch reference, next to the HOS_ switches already there.

question — scripted tests under the API: with HOS_DEBUG_SOCKET set, hos_debug_input returns HOS_DEBUG_PLAYER rather than HOS_DEBUG_LEGACY in player mode (host/nn_hid.c:796-798), so HOS_INPUT_SCRIPT/HOS_INPUT_LIVE presses are silently ignored. Is that intended? If so, log it once at start-up so a combined run doesn't look broken.

Once it's split, I'm happy to review the API part on its own. At that size the guest-call adapter can get the close look it needs, since it writes guest memory and calls configured guest functions on a game thread.

There's real care in here. The socket server binds an abstract name and checks `SO_PEERCRED` for the same UID. It takes one bounded SEQPACKET command per connection, with 250 ms timeouts, a 512-byte request, and a reject for packets that are truncated or contain a NUL. Its parsers refuse overflow and trailing junk. Memory reads are capped at 256 bytes, kept inside the guest window, and go through `process_vm_readv`, so a bad address can't fault the game. With `HOS_DEBUG_SOCKET` unset, the Windows build gets inline stubs and the Linux build returns early. Given the size, I focused on the socket server (`host/hos_debug.c`), the guest-call path (`host/hos_debug_guest.c`), and the edits to shared runtime files that every player runs (`nn_hid.c`, `nn_os_sync.c`, `hos_thread.c`, `nn_video.c`, `posix/prof.c`, `hos_setjmp.c`, `display_sdl.c`, `rx_backend.c`, `mkw_glue.c`). I didn't review the 14 Python scripts in depth. **blocking — one topic per PR (CONTRIBUTING, "Commits and pull requests" 2):** besides the scene-testing API, this PR carries: - a fix for a race in the Linux signal sampler's buffer handoff (`host/posix/prof.c`, `sample_one`/`on_sample`); - sampler thread refresh between batches (`posix/prof.c`); - a `.cfi_undefined %rip` unwind fix for `recomp_sj_longjmp` (`host/hos_setjmp.c:128`); - `HOS_START_MINIMIZED` (`hos_sdl.c`, `display_sdl.c`); - `HOS_REQUIRE_WINDOW` (`rx_backend.c:266`); - virtual-core wait tracing (`hos_thread.c`, `HOS_VCORE_TRACE_MS`); - event-wait tracing (`nn_os_sync.c`, `HOS_SYNC_WAIT_TRACE_MS`); - movie acquire tracing (`nn_video.c`, `HOS_VIDEO_ACQUIRE_TRACE`); - clock markers in the all-thread sample file (`mkw_glue.c`); - present-trace and wait analyzers. Several of these are good fixes in their own right; the sampler race and the longjmp unwind especially deserve to land quickly and on their own. Bundled here, they wait on review of a 7.6k-line feature, and a regression in any one would revert all of them. Please split them out: one PR per fix, one per trace switch (or one "wait diagnostics" PR), then the API on top. **blocking — conflicts:** `host/hos_main.c`, `host/nn_hid.c` and `host/nn_video.c` conflict with current main. Note also that CI hasn't run on this head (title check only). **should fix — a test that never runs** (`tests/test_analysis_dlss.py`, 328 lines): it loads `tests/test_dlss5_api.py` and `scripts/dlss5_api.py` and `skipUnless`es on both, but neither file exists on main or in this PR. It always skips, including under CI's `--strict`, because a `unittest` skip isn't a missing declared tool. Drop it here and bring it with the DLSS adapter PR it belongs to. **should fix — a cost while off on every Linux run** (`host/hos_debug_guest.c:338`): `hos_debug_scene_poll` is called from `npad_states` on every pad poll, and starts with `getenv("HOS_DEBUG_SOCKET")`, a scan of the environment several times a frame for every Linux player. CONTRIBUTING asks that switches cost nothing while off. Check the flag `hos_debug_init` already sets (expose `enabled`, or cache the getenv in a static) before anything else. **should fix — undocumented switches:** `HOS_START_MINIMIZED`, `HOS_REQUIRE_WINDOW`, `HOS_SYNC_WAIT_TRACE_MS`, `HOS_VIDEO_ACQUIRE_TRACE` and `HOS_DEBUG_CONTROL` appear in no doc (`HOS_START_MINIMIZED` only in a source comment). They belong in `docs/development.md`'s switch reference, next to the `HOS_` switches already there. **question — scripted tests under the API:** with `HOS_DEBUG_SOCKET` set, `hos_debug_input` returns `HOS_DEBUG_PLAYER` rather than `HOS_DEBUG_LEGACY` in player mode (`host/nn_hid.c:796-798`), so `HOS_INPUT_SCRIPT`/`HOS_INPUT_LIVE` presses are silently ignored. Is that intended? If so, log it once at start-up so a combined run doesn't look broken. Once it's split, I'm happy to review the API part on its own. At that size the guest-call adapter can get the close look it needs, since it writes guest memory and calls configured guest functions on a game thread.

Joey-Magnus (Migrated from github.com) reviewed

2026-11-11 21:45:58 +00:00

Joey-Magnus (Migrated from github.com) left a comment

No new commits since my review at 2ef23f2, and it still conflicts with main in host/hos_main.c, host/nn_hid.c and host/nn_video.c. None of the bundled fixes has reached main separately: no .cfi_undefined in hos_setjmp.c, and no HOS_START_MINIMIZED, HOS_REQUIRE_WINDOW or HOS_VCORE_TRACE_MS.

The open points are unchanged:

split out the sampler race in posix/prof.c and the longjmp unwind fix first; both can land quickly on their own;
hos_debug_scene_poll calls getenv on every pad poll for every Linux player while the API is off;
tests/test_analysis_dlss.py can only ever skip without #1542;
five new HOS_ switches are missing from docs/development.md;
with HOS_DEBUG_SOCKET set, HOS_INPUT_SCRIPT presses are silently ignored.

Once the API is on its own, I'm happy to look at the guest-call adapter closely.

No new commits since my review at 2ef23f2, and it still conflicts with main in `host/hos_main.c`, `host/nn_hid.c` and `host/nn_video.c`. None of the bundled fixes has reached main separately: no `.cfi_undefined` in `hos_setjmp.c`, and no `HOS_START_MINIMIZED`, `HOS_REQUIRE_WINDOW` or `HOS_VCORE_TRACE_MS`. The open points are unchanged: - split out the sampler race in `posix/prof.c` and the longjmp unwind fix first; both can land quickly on their own; - `hos_debug_scene_poll` calls `getenv` on every pad poll for every Linux player while the API is off; - `tests/test_analysis_dlss.py` can only ever skip without #1542; - five new `HOS_` switches are missing from `docs/development.md`; - with `HOS_DEBUG_SOCKET` set, `HOS_INPUT_SCRIPT` presses are silently ignored. Once the API is on its own, I'm happy to look at the guest-call adapter closely.

hckrman101 commented

2026-11-11 22:55:33 +00:00

(Migrated from github.com)

This now conflicts with main in host/hos_main.c, host/mkw_glue.c, host/nn_hid.c, host/nn_os_sync.c, host/nn_video.c, host/posix/prof.c. Could you merge main into the branch (or rebase) when you get a chance? I'll take another look once it's up to date.

This now conflicts with main in `host/hos_main.c`, `host/mkw_glue.c`, `host/nn_hid.c`, `host/nn_os_sync.c`, `host/nn_video.c`, `host/posix/prof.c`. Could you merge main into the branch (or rebase) when you get a chance? I'll take another look once it's up to date.