MagicBeanDev/Hookshot
Archived
Watch
6
0
Fork
You've already forked Hookshot
 0
Code Issues 125 Pull requests 205 Activity

perf(host): the game's zstd stream decode on the host's libzstd (HOS_NATIVE_ZSTD, opt-in) #2738

Closed
github-sync wants to merge 8 commits from github/pr/2738 into main
pull from: github/pr/2738
merge into: MagicBeanDev:main
Conversation 5 Commits 8 Files changed 5 +820 -4
github-sync commented 2026-11-12 10:34:16 +00:00
Owner

@klimovich008 opened this pull request on GitHub · 2026-11-12 10:11 UTC · original
Merged on GitHub in a2766f9c19

The game decodes every streamed file with its own lifted copy of zstd (ParallelDecompThread, about 3.5 ms of CPU a frame in the standard test). This adds an opt-in switch, HOS_NATIVE_ZSTD, that hands that decode to the host's libzstd 1.5.7, already vendored and linked for host/prompts.c. In a stub, libzstd takes about a third of the lifted decoder's time per MiB. In the game the difference is too small for 4 pairs of the standard test to resolve, so it stays default off.

This is a performance change, not a bug fix.

What changed

  • host/hos_zstd.c (new): hos_zstd_stream, a replacement for the game's per-chunk wrapper f_80115c80 around zstd's buffer-less API (frame header from the first chunk, begin_usingDDict, then the nextSrcSizeToDecompress / decompressContinue loop with the game's staging buffer for a piece split between chunks). Every field of the game's stream object (state, dst, room, bytes still to come, staged fill) and every result (bytes to come, 0, -1, -2 for dstSize_tooSmall, -3 for input past the frame's end) is kept as the game's code leaves it. libzstd keeps its own ZSTD_DCtx per stream; the game's context is never written. The game's dictionary is used by reference: a ZSTD_DDict made from the game's content and checked against a copy of the game's entropy tables, so a dictionary remade at the same address is never served stale. Output pages are marked with HOS_WRITTEN after the writes. States 3 and 4 (failed, done) and anything unexpected (a misaligned workspace, an allocation failure) go back to the lifted code. Fixed tables of 64 streams and 64 dictionaries, under a spin lock; the peak in use is logged.
    • HOS_NATIVE_ZSTD=1: decodes on the host and gives the virtual core up while it decodes (hos_vcore_away / hos_vcore_back), as a sleep does.
    • HOS_NATIVE_ZSTD=2: decodes on the host and keeps the core.
    • unset or 0: off, the lifted code runs as before.
  • nx2/oot_patches.py: patch_zstd_stream puts if (hos_zstd_stream(c)) return; first in f_80115c80, for the supported build only, counted by report() like the other hooks (warns unless found exactly once).
  • nx2/build.py: ZSTD_HOST, the host files that get the zstd include path and headers in their dependency key (prompts.c, hos_zstd.c).
  • tests/test_native_zstd.py (new, game-free; zstandard from RUN_ALL_MODULES makes the frames): compiles hos_zstd.c as it is, with the vendored zstd decoder, over a small fake guest window. It feeds frames with and without checksums, with a dictionary (entropy tables and raw content), empty and one byte long, RLE and raw blocks, whole and in chunks from 7 bytes to 128 KB + 3 and at random, so headers and blocks are split through the staging buffer. Each call's result is checked against the stream object's remaining-byte field and state byte, and the final state against what the game's wrapper leaves. Failures must answer as the game's do: wrong checksum or dictionary, bad magic, unknown content size -1; too little room -2; bytes after the frame -3. Also covered: a dictionary remade in place with the same id; the fallbacks (switch off, misaligned workspace, all 64 entries used); page marks after the last write; the core given up once a call with =1 and kept with =2. A broken copy (the staging fill never stored) runs inside the test and must fail.
  • docs/tested-configurations.md: the three rows below.

How it was tested

  • The game's own decoder, locally (no game data here or in the test): the game's f_80115c80 and its callees, run in Unicorn from a local image, against hos_zstd.c on the same inputs. 110 cases (the test's kinds plus larger and multi-frame sets), about 88,000 calls. Results, every stream field, the staging buffer, the dictionary region and the decoded output were identical at every call. Differences were only in the up-to-32 bytes just past the decoded output, which both decoders' fast copies over-write: the game's writes 3-11 bytes past it, libzstd up to 27. For a corrupt block, the partial output's last bytes differ; both report -1. A broken variant fails.
  • python -I -B tests/run_all.py on the merged tree (Windows, llvm-mingw): 457 passed, 36 skipped, 1 failed. The failure is nx2/test_nup_paths.py, which cannot create its file-link fixture on this PC (no symlink right); this PR does not touch that test or what it covers.
  • Stub timing (quiet machine, under the GPU lock; A11's 1,528 KiB synthetic set, levels 3-19, with and without dictionary and checksum): libzstd through hos_zstd.c at the build's flags (-O2 -march=x86-64-v3, ZSTD_DISABLE_ASM) takes 0.99 ms per MiB, against 3.09 ms for the lifted decoder at -O2 (#909) and 3.70 ms at -O1. That is 0.27-0.36 of the lifted -O2 time per frame.
  • The standard test (Hyrule Field, camera turning at full speed, HOS_FPS_MAX=240, 20 s window): exe cacdc864b4239f07 (1bde1868, before the merge with main), hookshot_vk.dll bfdac38fbf7d7c23. RX 9070 XT, Vulkan driver 0x8001a0, Ryzen 7 5700X3D, 64 GB, Windows 11, 5120x1440 (32:9). 2 warm-ups, then 4 alternating pairs per arm, same binaries:
arm fps CPU ms a frame p99 ms worst ms frames > 100 / > 250 ms game clock save load s (median) core escapes (with ParallelDecompThread holding)
off 44.05 115.05 29.1 220.6 2 / 0 0.98 29.0 4 (0)
=1 (core given up) 44.15 114.01 31.0 333.1 4 / 2 0.98 28.7 4 (0)
=2 (core kept) 44.70 112.71 29.5 56.7 0 / 0 1.00 28.5 3 (0)

The CPU-time differences (-1.0 and -2.3 ms a frame) are inside the run-to-run range (111.5-117.9 ms). The stub predicts about 2.4 ms a frame (3.54 ms x 0.68), which 4 pairs cannot resolve. Save load: -0.3 and -0.5 s, also inside the range. All of =1's frames over 100 ms came in its first run, which also had 5 virtual-core waits over 200 ms, against 1-2 in every other run. The control's first run (220.6 ms) and a warm-up (267.2 ms) spiked too, so I don't attribute them. The escapes were measured at main's 2 s escape default (#2127). None had ParallelDecompThread holding a core, so giving the core up buys nothing measurable here. =2 is the mode to try.

These runs were on a main from before #2602, which fixed a regression on main (this machine: about 41 → 53 fps and 125 → 78 ms of CPU a frame on the standard test), so every arm ran on the slower main. I can rerun the A/B on a main with #2602 if wanted.

Not covered

  • Linux (the code has no OS branch, but it was not built or run there), NVIDIA, Intel, the Steam Deck, other CPUs.
  • More than 4 pairs, other scenes, first visits with cold caches, sessions longer than a minute.
  • The second zstd path in the game, f_8010b380 (whole-buffer decode): not hooked. A follow-up if this one proves out.
  • libzstd's assembly Huffman decoder (ZSTD_DISABLE_ASM stays, as for the rest of the build): a follow-up.
  • Corrupt frames: the partial output's last bytes before the error differ from the game's decoder (both return -1).

Related

  • #909 (the lifted decoder at -O2, merged).
  • #2322 / #2301: the same pattern of a host routine in place of a lifted one, behind a switch.
  • #1929 and CIB's rule on virtual-core escapes: the core give-up here is measured at the 2 s default only, and is a by-product, not the point.
  • Prior art: zortos293/Lichii#869 puts this game's zstd decode on the host's libzstd in an emulator fork, one level lower (the block decoder, with the game's history buffer). This PR replaces the game's own per-chunk wrapper instead, so libzstd keeps its own state. I did not read its code; it's credited as related work.

🤖 Generated with Claude Code

> [@klimovich008](https://github.com/klimovich008) opened this pull request on GitHub · 2026-11-12 10:11 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2738) > **Merged on GitHub** in https://git.magicbean.dev/MagicBeanDev/Hookshot/commit/a2766f9c194f06e0836e3b1c4c563e4675702386 The game decodes every streamed file with its own lifted copy of zstd (ParallelDecompThread, about 3.5 ms of CPU a frame in the standard test). This adds an opt-in switch, `HOS_NATIVE_ZSTD`, that hands that decode to the host's libzstd 1.5.7, already vendored and linked for `host/prompts.c`. In a stub, libzstd takes about a third of the lifted decoder's time per MiB. In the game the difference is too small for 4 pairs of the standard test to resolve, so it stays **default off**. This is a performance change, not a bug fix. ## What changed - **`host/hos_zstd.c`** (new): `hos_zstd_stream`, a replacement for the game's per-chunk wrapper `f_80115c80` around zstd's buffer-less API (frame header from the first chunk, `begin_usingDDict`, then the `nextSrcSizeToDecompress` / `decompressContinue` loop with the game's staging buffer for a piece split between chunks). Every field of the game's stream object (state, dst, room, bytes still to come, staged fill) and every result (bytes to come, 0, -1, -2 for `dstSize_tooSmall`, -3 for input past the frame's end) is kept as the game's code leaves it. libzstd keeps its own `ZSTD_DCtx` per stream; the game's context is never written. The game's dictionary is used by reference: a `ZSTD_DDict` made from the game's content and checked against a copy of the game's entropy tables, so a dictionary remade at the same address is never served stale. Output pages are marked with `HOS_WRITTEN` after the writes. States 3 and 4 (failed, done) and anything unexpected (a misaligned workspace, an allocation failure) go back to the lifted code. Fixed tables of 64 streams and 64 dictionaries, under a spin lock; the peak in use is logged. - `HOS_NATIVE_ZSTD=1`: decodes on the host and gives the virtual core up while it decodes (`hos_vcore_away` / `hos_vcore_back`), as a sleep does. - `HOS_NATIVE_ZSTD=2`: decodes on the host and keeps the core. - unset or `0`: off, the lifted code runs as before. - **`nx2/oot_patches.py`**: `patch_zstd_stream` puts `if (hos_zstd_stream(c)) return;` first in `f_80115c80`, for the supported build only, counted by `report()` like the other hooks (warns unless found exactly once). - **`nx2/build.py`**: `ZSTD_HOST`, the host files that get the zstd include path and headers in their dependency key (`prompts.c`, `hos_zstd.c`). - **`tests/test_native_zstd.py`** (new, game-free; `zstandard` from `RUN_ALL_MODULES` makes the frames): compiles `hos_zstd.c` as it is, with the vendored zstd decoder, over a small fake guest window. It feeds frames with and without checksums, with a dictionary (entropy tables and raw content), empty and one byte long, RLE and raw blocks, whole and in chunks from 7 bytes to 128 KB + 3 and at random, so headers and blocks are split through the staging buffer. Each call's result is checked against the stream object's remaining-byte field and state byte, and the final state against what the game's wrapper leaves. Failures must answer as the game's do: wrong checksum or dictionary, bad magic, unknown content size -1; too little room -2; bytes after the frame -3. Also covered: a dictionary remade in place with the same id; the fallbacks (switch off, misaligned workspace, all 64 entries used); page marks after the last write; the core given up once a call with `=1` and kept with `=2`. A broken copy (the staging fill never stored) runs inside the test and must fail. - **`docs/tested-configurations.md`**: the three rows below. ## How it was tested - **The game's own decoder, locally (no game data here or in the test):** the game's `f_80115c80` and its callees, run in Unicorn from a local image, against `hos_zstd.c` on the same inputs. 110 cases (the test's kinds plus larger and multi-frame sets), about 88,000 calls. Results, every stream field, the staging buffer, the dictionary region and the decoded output were identical at every call. Differences were only in the up-to-32 bytes just past the decoded output, which both decoders' fast copies over-write: the game's writes 3-11 bytes past it, libzstd up to 27. For a corrupt block, the partial output's last bytes differ; both report -1. A broken variant fails. - `python -I -B tests/run_all.py` on the merged tree (Windows, llvm-mingw): 457 passed, 36 skipped, 1 failed. The failure is `nx2/test_nup_paths.py`, which cannot create its file-link fixture on this PC (no symlink right); this PR does not touch that test or what it covers. - **Stub timing** (quiet machine, under the GPU lock; A11's 1,528 KiB synthetic set, levels 3-19, with and without dictionary and checksum): libzstd through `hos_zstd.c` at the build's flags (`-O2 -march=x86-64-v3`, `ZSTD_DISABLE_ASM`) takes **0.99 ms per MiB**, against 3.09 ms for the lifted decoder at -O2 (#909) and 3.70 ms at -O1. That is 0.27-0.36 of the lifted -O2 time per frame. - **The standard test** (Hyrule Field, camera turning at full speed, `HOS_FPS_MAX=240`, 20 s window): exe `cacdc864b4239f07` (1bde1868, before the merge with main), `hookshot_vk.dll` `bfdac38fbf7d7c23`. RX 9070 XT, Vulkan driver 0x8001a0, Ryzen 7 5700X3D, 64 GB, Windows 11, 5120x1440 (32:9). 2 warm-ups, then 4 alternating pairs per arm, same binaries: | arm | fps | CPU ms a frame | p99 ms | worst ms | frames > 100 / > 250 ms | game clock | save load s (median) | core escapes (with ParallelDecompThread holding) | |---|---|---|---|---|---|---|---|---| | off | 44.05 | 115.05 | 29.1 | 220.6 | 2 / 0 | 0.98 | 29.0 | 4 (0) | | `=1` (core given up) | 44.15 | 114.01 | 31.0 | 333.1 | 4 / 2 | 0.98 | 28.7 | 4 (0) | | `=2` (core kept) | 44.70 | 112.71 | 29.5 | 56.7 | 0 / 0 | 1.00 | 28.5 | 3 (0) | The CPU-time differences (-1.0 and -2.3 ms a frame) are inside the run-to-run range (111.5-117.9 ms). The stub predicts about 2.4 ms a frame (3.54 ms x 0.68), which 4 pairs cannot resolve. Save load: -0.3 and -0.5 s, also inside the range. All of `=1`'s frames over 100 ms came in its first run, which also had 5 virtual-core waits over 200 ms, against 1-2 in every other run. The control's first run (220.6 ms) and a warm-up (267.2 ms) spiked too, so I don't attribute them. The escapes were measured at main's 2 s escape default (#2127). None had ParallelDecompThread holding a core, so giving the core up buys nothing measurable here. `=2` is the mode to try. These runs were on a main from before #2602, which fixed a regression on main (this machine: about 41 → 53 fps and 125 → 78 ms of CPU a frame on the standard test), so every arm ran on the slower main. I can rerun the A/B on a main with #2602 if wanted. ## Not covered - Linux (the code has no OS branch, but it was not built or run there), NVIDIA, Intel, the Steam Deck, other CPUs. - More than 4 pairs, other scenes, first visits with cold caches, sessions longer than a minute. - The second zstd path in the game, `f_8010b380` (whole-buffer decode): not hooked. A follow-up if this one proves out. - libzstd's assembly Huffman decoder (`ZSTD_DISABLE_ASM` stays, as for the rest of the build): a follow-up. - Corrupt frames: the partial output's last bytes before the error differ from the game's decoder (both return -1). ## Related - #909 (the lifted decoder at -O2, merged). - #2322 / #2301: the same pattern of a host routine in place of a lifted one, behind a switch. - #1929 and CIB's rule on virtual-core escapes: the core give-up here is measured at the 2 s default only, and is a by-product, not the point. - Prior art: [zortos293/Lichii#869](https://github.com/zortos293/Lichii/pull/869) puts this game's zstd decode on the host's libzstd in an emulator fork, one level lower (the block decoder, with the game's history buffer). This PR replaces the game's own per-chunk wrapper instead, so libzstd keeps its own state. I did not read its code; it's credited as related work. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- github-sync:item:2738 -->
The game decodes its .zs files on ParallelDecompThread through f_80115c80, its own wrapper around zstd's
buffer-less streaming API. Lifted, it costs 11-17% of a core while the game streams, and during loads it runs
for long stretches without an OS call while holding a virtual core.

nx2/oot_patches.py puts hos_zstd_stream(c) first in f_80115c80. With HOS_NATIVE_ZSTD=1 the host does the
same calls into the vendored libzstd 1.5.7 (decompressBegin_usingDDict, nextSrcSizeToDecompress,
decompressContinue), keeps every field of the game's stream object and every result the game's wrapper
gives (still to come, 0, -1, -2, -3), uses the game's DDict content by reference, and gives the virtual
core up while it decodes (=2 keeps it). Off by default: the lifted function runs as before.

tests/test_native_zstd.py builds host/hos_zstd.c as it is with the vendored decoder and checks it on frames
made with python-zstandard: outputs, results and fields over many chunk splits, dictionaries, checksums,
the failure codes, fallbacks to the lifted code, page marks for the GPU mirror, and a broken copy that
must fail.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Its memory may have been freed (FreeMemoryPages decommits pages). An entry now goes only when its stream fails
or ends, or is reset when its object starts the next frame; with all 64 in use a new stream decodes lifted.
Also logs each new peak of streams decoding on the host.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The mark was taken before zs_chunk updated the output pointer, room, still-to-come, staging fill and state;
a GPU mirror scan between the two could clear it and miss those writes (found by the gpt-6.1-sol review).
The test's fixture now keeps each page as it was at its last mark, so a page written after its mark fails;
the old order fails it.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The cache took the same address, content, size, dictID, entropy flag and repeat offsets for the same
dictionary; a DDict made again in place for another dictionary with the same ID and size kept the old
entropy tables (found by the gpt-6.1-sol review). It now keeps a copy of the game's own DDict tables and
compares them. The test covers that case with small frames (they use the dictionary's tables), checks every
result against the stream object's still-to-come and output pointer, and checks HOS_NATIVE_ZSTD=2 decodes.

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2 while the frame decodes, 4 after its last call; a host decode that left another state mid-frame passed
before (found by the gpt-6.1-sol review, round 3).

Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
nx2/oot_patches.py: main moved the per-file patches into patch_file(); the zstd stream hook is there now, counted by
report() like the other supported-build hooks. nx2/build.py: ZSTD_HOST beside main's dav1d flags.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
github-sync commented 2026-11-12 12:37:39 +00:00
Author
Owner

@CIB commented on GitHub · 2026-11-12 12:22 UTC · original

Linux test

I built and ran this on Linux (the part the PR says is not covered).

  • Arch Linux, kernel 6.19.8, Ryzen 7 5700X3D, RTX 4090, NVIDIA 590.48.01 (open kernel module), Vulkan 1.4.
  • Game-free test: python3 -I -B tests/test_native_zstd.py -> both cases PASS (zstandard 0.25.0).
  • Build: passes on current main fce53453 with the host file and the build.py change. My shared lift predates the PR, so I applied patch_zstd_stream to the generated zstd stream wrapper manually; a fresh lift will do it. (Using the PR's own older base in that lift hit an unrelated a64_stubs/nn_audio duplicate, so the runs below are current main + the PR's host side.)
  • In-game: fresh profile, anonymous window, 45 s from the same Start, 2 runs per arm:
HOS_NATIVE_ZSTD crashes fps peak host streams
off 0/2 60.0 -
1 0/2 60.0 3-4
2 0/2 59.9-60.0 2-4

The hook is active (zstd: the game's stream decode on the host's libzstd 1.5.7 ...) and no arm crashed or regressed.

Caveat: the intro is vsync-capped at 60, so this is not a CPU-time measurement. I did not run the Hyrule Field standard test or measure CPU ms/frame; the 2.4 ms/frame prediction remains untested on Linux. If a Linux perf row is wanted, I can run an uncapped Hyrule Field A/B with process CPU time, but it needs the save/camera automation.

No game files or saves used.

🤖 Generated with opencode using DeepSeek V4.1 Flash.

> [@CIB](https://github.com/CIB) commented on GitHub · 2026-11-12 12:22 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2738#issuecomment-6823484440) ### Linux test I built and ran this on Linux (the part the PR says is not covered). - Arch Linux, kernel 6.19.8, Ryzen 7 5700X3D, RTX 4090, NVIDIA 590.48.01 (open kernel module), Vulkan 1.4. - **Game-free test:** `python3 -I -B tests/test_native_zstd.py` -> both cases PASS (zstandard 0.25.0). - **Build:** passes on current main `fce53453` with the host file and the `build.py` change. My shared lift predates the PR, so I applied `patch_zstd_stream` to the generated zstd stream wrapper manually; a fresh lift will do it. (Using the PR's own older base in that lift hit an unrelated `a64_stubs`/`nn_audio` duplicate, so the runs below are current main + the PR's host side.) - **In-game:** fresh profile, anonymous window, 45 s from the same Start, 2 runs per arm: | `HOS_NATIVE_ZSTD` | crashes | fps | peak host streams | |---|---:|---:|---:| | off | 0/2 | 60.0 | - | | 1 | 0/2 | 60.0 | 3-4 | | 2 | 0/2 | 59.9-60.0 | 2-4 | The hook is active (`zstd: the game's stream decode on the host's libzstd 1.5.7 ...`) and no arm crashed or regressed. **Caveat:** the intro is vsync-capped at 60, so this is not a CPU-time measurement. I did not run the Hyrule Field standard test or measure CPU ms/frame; the 2.4 ms/frame prediction remains untested on Linux. If a Linux perf row is wanted, I can run an uncapped Hyrule Field A/B with process CPU time, but it needs the save/camera automation. No game files or saves used. --- 🤖 Generated with [opencode](https://opencode.ai) using DeepSeek V4.1 Flash. <!-- github-sync:c:6823484440 -->
github-sync commented 2026-11-12 14:44:55 +00:00
Author
Owner

@Joey-Magnus reviewed on GitHub · 2026-11-12 14:34 UTC · original

I merged this into current main and ran tests/test_native_zstd.py (zstandard 0.25.0) and test_patch_hooks_defined.py: both pass, and so does the broken-copy control. I read host/hos_zstd.c. Off, hos_zstd_stream returns after a cached env_on (line 295), so the default path costs one call per chunk. The stream table only frees an entry on failure, on done, or when a new frame starts. An object the game abandons mid-stream keeps its entry, which is documented, and after 64 of those new streams fall back to the lifted decoder. The DDict is checked against a copy of the game's entropy tables, so one remade in place isn't served stale. CIB's Linux run covers the "not covered: Linux" line.

I see no correctness problem. What's left is whether to carry 336 lines of host code and a hook in the lifted f_80115c80 for a gain 4 pairs couldn't resolve. Those runs were also on main before #2602. With #2602 CPU time a frame is about 78 ms instead of 115, so the predicted ~2.4 ms is a bigger share and might show. I'd rerun =2 against off on current main before deciding.

> [@Joey-Magnus](https://github.com/Joey-Magnus) reviewed on GitHub · 2026-11-12 14:34 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2738#pullrequestreview-5876242896) I merged this into current main and ran tests/test_native_zstd.py (zstandard 0.25.0) and test_patch_hooks_defined.py: both pass, and so does the broken-copy control. I read host/hos_zstd.c. Off, `hos_zstd_stream` returns after a cached `env_on` (line 295), so the default path costs one call per chunk. The stream table only frees an entry on failure, on done, or when a new frame starts. An object the game abandons mid-stream keeps its entry, which is documented, and after 64 of those new streams fall back to the lifted decoder. The DDict is checked against a copy of the game's entropy tables, so one remade in place isn't served stale. CIB's Linux run covers the "not covered: Linux" line. I see no correctness problem. What's left is whether to carry 336 lines of host code and a hook in the lifted f_80115c80 for a gain 4 pairs couldn't resolve. Those runs were also on main before #2602. With #2602 CPU time a frame is about 78 ms instead of 115, so the predicted ~2.4 ms is a bigger share and might show. I'd rerun `=2` against off on current main before deciding. <!-- github-sync:r:5876242896 -->
github-sync commented 2026-11-12 15:52:30 +00:00
Author
Owner

@jordanella commented on GitHub · 2026-11-12 15:05 UTC · original

Test request T-M4T5: pass

Hardware: NVIDIA GeForce RTX 3080 (10 GB), driver 621.14, AMD Ryzen 9 3900X (12 cores, 24 threads), Windows 11. Built f0f7dfda (the PR head; v2.1.0-rc.3-260-gf0f7dfda), with the Builder's llvm-mingw; the build log names the hook: oot_patches: the zstd stream decode's host hook in code_000.c.

What I checked: the same build, switch off against HOS_NATIVE_ZSTD=2 and =1, alternating (off, =2, =1, off, =2, =1). Each run starts headless from a save through the control bridge with HOS_FPS_MAX=240, teleports to Hyrule Field (Lost Woods bridge), settles 10 s, then holds the right stick fully right (the camera turning at full speed) and samples the status fps once a second for 20 s.

Result: no difference beyond run-to-run spread, and nothing broken.

arm run 1 median fps (min-max) run 2 median fps (min-max)
off 27.2 (13.6-37.0) 25.2 (14.7-32.1)
=2 (core kept) 25.2 (11.7-31.2) 26.6 (14.7-31.1)
=1 (core given up) not counted: the teleport was refused (an event was playing), so it sampled Kokiri Forest 26.6 (12.7-36.9)

  • No device lost, crash or exception line in any run.

  • The host decoder was in use in every =1 and =2 run, and in none of the off runs:

    zstd: the game's stream decode on the host's libzstd 1.5.7 (HOS_NATIVE_ZSTD=2)
zstd: 1 stream(s) decoding on the host at once
zstd: 2 stream(s) decoding on the host at once

    =1 reached 3 streams at once; nothing came near the 64-entry tables.

  • This scene is CPU-bound here (about 26 fps against the 44 in your RX 9070 XT / 5700X3D table), so a ~2 ms/frame saving would show as about 1.3 fps, inside the 2 fps spread between the two off runs. Two runs per arm can't resolve it; it confirms no regression on NVIDIA and a Zen 2 CPU, not a gain.

> [@jordanella](https://github.com/jordanella) commented on GitHub · 2026-11-12 15:05 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2738#issuecomment-6827299363) Test request T-M4T5: pass Hardware: NVIDIA GeForce RTX 3080 (10 GB), driver 621.14, AMD Ryzen 9 3900X (12 cores, 24 threads), Windows 11. Built f0f7dfda (the PR head; `v2.1.0-rc.3-260-gf0f7dfda`), with the Builder's llvm-mingw; the build log names the hook: `oot_patches: the zstd stream decode's host hook in code_000.c`. What I checked: the same build, switch off against `HOS_NATIVE_ZSTD=2` and `=1`, alternating (off, =2, =1, off, =2, =1). Each run starts headless from a save through the control bridge with `HOS_FPS_MAX=240`, teleports to Hyrule Field (Lost Woods bridge), settles 10 s, then holds the right stick fully right (the camera turning at full speed) and samples the status fps once a second for 20 s. Result: no difference beyond run-to-run spread, and nothing broken. | arm | run 1 median fps (min-max) | run 2 median fps (min-max) | |---|---|---| | off | 27.2 (13.6-37.0) | 25.2 (14.7-32.1) | | `=2` (core kept) | 25.2 (11.7-31.2) | 26.6 (14.7-31.1) | | `=1` (core given up) | not counted: the teleport was refused (an event was playing), so it sampled Kokiri Forest | 26.6 (12.7-36.9) | - No `device lost`, crash or exception line in any run. - The host decoder was in use in every `=1` and `=2` run, and in none of the off runs: ``` zstd: the game's stream decode on the host's libzstd 1.5.7 (HOS_NATIVE_ZSTD=2) zstd: 1 stream(s) decoding on the host at once zstd: 2 stream(s) decoding on the host at once ``` `=1` reached 3 streams at once; nothing came near the 64-entry tables. - This scene is CPU-bound here (about 26 fps against the 44 in your RX 9070 XT / 5700X3D table), so a ~2 ms/frame saving would show as about 1.3 fps, inside the 2 fps spread between the two off runs. Two runs per arm can't resolve it; it confirms no regression on NVIDIA and a Zen 2 CPU, not a gain. <!-- github-sync:c:6827299363 -->
github-sync commented 2026-11-12 16:55:57 +00:00
Author
Owner

@hckrman101 commented on GitHub · 2026-11-12 16:34 UTC · original

Test request T-M4T5: pass

Hardware: AMD Radeon RX 580 8 GB (RADV Polaris10), Mesa 26.2.6, CachyOS Linux, 8 threads, 15 GB RAM. Built f0f7dfda (the PR head). One build; HOS_NATIVE_ZSTD unset, =1 and =2 alternated (3 runs each, the order rotated each round, after one warm-up run).

What I checked: start to play from the test save, then Debug teleports Kokiri Forest → Hyrule Field → Market, with a 34 s window in each scene, the camera turning (HOS_FPS_MAX=240 RX_VK_FRAMEGEN=0 RX_VK_PRECOMPILE=0, headless). CPU time of the process and of the four ParallelDecompThread threads, sampled from /proc once a second. Loads are timed from the log: game start to first present and to the first frame of play; each teleport from its request to the first frame of 300+ draws after the loading screen. Frame times from RX_VK_FRAME_TRACE.

  • libzstd: the build uses the vendored copy. The log says zstd: the game's stream decode on the host's libzstd 1.5.7. The binary has no runtime libzstd dependency (ldd shows none; the only dynamic zstd symbols are libzstd's two weak trace hooks). The system libzstd here is also 1.5.7, but it isn't used, so a missing or different system library can't affect it.
  • tests/test_native_zstd.py passes on Linux (gcc, the PR's own fixture, including the fallback cases and the broken-copy control): "the host's zstd stream decode answers, writes and marks as the game's wrapper does"; "the broken copy fails".

Results, per run → median:

off =1 =2
decode-thread CPU, start → play 12.4 / 12.0 / 7.2 → 12.0 s 4.4 / 4.0 / 3.7 → 4.0 s 5.6 / 5.3 / 4.6 → 5.3 s
decode-thread CPU, Field load 1.03 / 1.56 / 1.24 → 1.24 s 0.41 / 0.55 / 0.65 → 0.55 s 0.60 / 0.66 / 0.63 → 0.63 s
decode-thread CPU, Market load 5.31 / 2.89 / 4.28 → 4.28 s 2.23 / 1.40 / 2.03 → 2.03 s 1.11 / 1.39 / 1.51 → 1.39 s
start → first present 24.5 / 14.5 / 19.7 → 19.7 s 27.3 / 13.6 / 13.0 → 13.6 s 25.2 / 16.4 / 13.9 → 16.4 s
start → in play (save loaded) 78.1 / 63.7 / 63.8 → 63.8 s 74.6 / 55.4 / 49.0 → 55.4 s 72.3 / 64.9 / 49.4 → 64.9 s
Field teleport → play 12.5 / 15.0 / 12.2 → 12.5 s 12.0 / 10.3 / 12.8 → 12.0 s 12.8 / 10.7 / 10.0 → 10.7 s
Market teleport → play 29.9 / 25.2 / 30.1 → 29.9 s 32.3 / 21.7 / 22.6 → 22.6 s 25.6 / 28.1 / 19.3 → 25.6 s
Kokiri fps median; 1% low 21.9; 8.1 21.6; 9.3 21.6; 9.6
Field fps median; 1% low; frames > 100 ms 32.2; 3.8; 45 31.1; 4.0; 45 30.5; 3.8; 46
Market fps median; 1% low; frames > 100 ms 39.2; 3.6; 72 36.9; 3.5; 65 38.9; 3.8; 73
  • The decode threads use 2-3x less CPU with it on, in every run. They do nothing in the play windows (0.000 cores in all arms), so the saving is all in loads.
  • The load times move the same way but stay inside the run-to-run range. The first round was slower for every arm (a colder page cache), and the harness presses A once a second to reach play, so start → play is coarse. The Field and Market hitches (frames > 100 ms) are pipeline compiles with background precompile off, the same in every arm.
  • fps in play is unchanged within noise. =1's Market median came out a little lower in all three runs (37.8 / 34.3 / 36.9 vs 40.2 / 39.2 / 38.9 off), and =2 matched off. The decode threads are idle in those windows, so I don't think it's the decode; with 3 runs I can't tell.

Correctness:

  • Every on run logs the switch line and then 1, 2, 3 stream(s) decoding on the host at once. No stream fell back to the lifted code (a stream decodes lifted: 0 in all runs).
  • Screenshots once a second, =1 vs off, no camera turn, matched by time after each scene drew: mean absolute difference 3-8 of 255 and 0.4-3% of pixels more than 32 off. That's the same as two runs of one build (wind and animation), with the same mean brightness per scene. No missing or corrupted textures or models in Kokiri, the Field or the Market.
  • No crash, hang or error in 12 runs (=1 / =2 / off plus the warm-up and shot runs).
  • Not covered: audio. These runs are headless with audio off (HOS_AUDIO=0), so audio streams weren't checked.
> [@hckrman101](https://github.com/hckrman101) commented on GitHub · 2026-11-12 16:34 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2738#issuecomment-6829578815) Test request T-M4T5: pass Hardware: AMD Radeon RX 580 8 GB (RADV Polaris10), Mesa 26.2.6, CachyOS Linux, 8 threads, 15 GB RAM. Built f0f7dfda (the PR head). One build; `HOS_NATIVE_ZSTD` unset, `=1` and `=2` alternated (3 runs each, the order rotated each round, after one warm-up run). What I checked: start to play from the test save, then Debug teleports Kokiri Forest → Hyrule Field → Market, with a 34 s window in each scene, the camera turning (`HOS_FPS_MAX=240 RX_VK_FRAMEGEN=0 RX_VK_PRECOMPILE=0`, headless). CPU time of the process and of the four `ParallelDecompThread` threads, sampled from `/proc` once a second. Loads are timed from the log: game start to first present and to the first frame of play; each teleport from its request to the first frame of 300+ draws after the loading screen. Frame times from `RX_VK_FRAME_TRACE`. - libzstd: the build uses the vendored copy. The log says `zstd: the game's stream decode on the host's libzstd 1.5.7`. The binary has no runtime libzstd dependency (`ldd` shows none; the only dynamic zstd symbols are libzstd's two weak trace hooks). The system libzstd here is also 1.5.7, but it isn't used, so a missing or different system library can't affect it. - `tests/test_native_zstd.py` passes on Linux (gcc, the PR's own fixture, including the fallback cases and the broken-copy control): "the host's zstd stream decode answers, writes and marks as the game's wrapper does"; "the broken copy fails". Results, per run → median: | | off | `=1` | `=2` | |---|---|---|---| | decode-thread CPU, start → play | 12.4 / 12.0 / 7.2 → **12.0 s** | 4.4 / 4.0 / 3.7 → **4.0 s** | 5.6 / 5.3 / 4.6 → **5.3 s** | | decode-thread CPU, Field load | 1.03 / 1.56 / 1.24 → **1.24 s** | 0.41 / 0.55 / 0.65 → **0.55 s** | 0.60 / 0.66 / 0.63 → **0.63 s** | | decode-thread CPU, Market load | 5.31 / 2.89 / 4.28 → **4.28 s** | 2.23 / 1.40 / 2.03 → **2.03 s** | 1.11 / 1.39 / 1.51 → **1.39 s** | | start → first present | 24.5 / 14.5 / 19.7 → 19.7 s | 27.3 / 13.6 / 13.0 → 13.6 s | 25.2 / 16.4 / 13.9 → 16.4 s | | start → in play (save loaded) | 78.1 / 63.7 / 63.8 → 63.8 s | 74.6 / 55.4 / 49.0 → 55.4 s | 72.3 / 64.9 / 49.4 → 64.9 s | | Field teleport → play | 12.5 / 15.0 / 12.2 → 12.5 s | 12.0 / 10.3 / 12.8 → 12.0 s | 12.8 / 10.7 / 10.0 → 10.7 s | | Market teleport → play | 29.9 / 25.2 / 30.1 → 29.9 s | 32.3 / 21.7 / 22.6 → 22.6 s | 25.6 / 28.1 / 19.3 → 25.6 s | | Kokiri fps median; 1% low | 21.9; 8.1 | 21.6; 9.3 | 21.6; 9.6 | | Field fps median; 1% low; frames > 100 ms | 32.2; 3.8; 45 | 31.1; 4.0; 45 | 30.5; 3.8; 46 | | Market fps median; 1% low; frames > 100 ms | 39.2; 3.6; 72 | 36.9; 3.5; 65 | 38.9; 3.8; 73 | - The decode threads use 2-3x less CPU with it on, in every run. They do nothing in the play windows (0.000 cores in all arms), so the saving is all in loads. - The load times move the same way but stay inside the run-to-run range. The first round was slower for every arm (a colder page cache), and the harness presses A once a second to reach play, so start → play is coarse. The Field and Market hitches (frames > 100 ms) are pipeline compiles with background precompile off, the same in every arm. - fps in play is unchanged within noise. `=1`'s Market median came out a little lower in all three runs (37.8 / 34.3 / 36.9 vs 40.2 / 39.2 / 38.9 off), and `=2` matched off. The decode threads are idle in those windows, so I don't think it's the decode; with 3 runs I can't tell. Correctness: - Every on run logs the switch line and then `1`, `2`, `3 stream(s) decoding on the host at once`. No stream fell back to the lifted code (`a stream decodes lifted`: 0 in all runs). - Screenshots once a second, `=1` vs off, no camera turn, matched by time after each scene drew: mean absolute difference 3-8 of 255 and 0.4-3% of pixels more than 32 off. That's the same as two runs of one build (wind and animation), with the same mean brightness per scene. No missing or corrupted textures or models in Kokiri, the Field or the Market. - No crash, hang or error in 12 runs (`=1` / `=2` / off plus the warm-up and shot runs). - Not covered: audio. These runs are headless with audio off (`HOS_AUDIO=0`), so audio streams weren't checked. <!-- github-sync:c:6829578815 -->
github-sync closed this pull request 2026-11-12 17:58:50 +00:00
github-sync commented 2026-11-12 17:58:51 +00:00
Author
Owner

@hckrman101 approved on GitHub · 2026-11-12 17:03 UTC · original

I merged main (bd9b07e6) into the branch: 6d0a9a5f. The only conflict was the rows in docs/tested-configurations.md, all kept. Main's newer oot_patches hooks don't touch this one's function, and test_native_zstd, test_patch_hooks_defined and the oot_patches/build tests pass.

Merging this as opt-in on the four test results: our RX 580 run (decode-thread CPU 2-3x lower during loads, no regression in play), jordanella's RTX 3080, CIB's RTX 4090 on Linux, and Joey's read, which found no correctness problem. Turning it on by default would need an audio check (none of the runs had audio on) and a rerun on current main, as Joey suggested.

Thanks for this, and for the careful fallbacks!

> [@hckrman101](https://github.com/hckrman101) approved on GitHub · 2026-11-12 17:03 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2738#pullrequestreview-5878663856) I merged main (bd9b07e6) into the branch: 6d0a9a5f. The only conflict was the rows in docs/tested-configurations.md, all kept. Main's newer oot_patches hooks don't touch this one's function, and test_native_zstd, test_patch_hooks_defined and the oot_patches/build tests pass. Merging this as opt-in on the four test results: our RX 580 run (decode-thread CPU 2-3x lower during loads, no regression in play), jordanella's RTX 3080, CIB's RTX 4090 on Linux, and Joey's read, which found no correctness problem. Turning it on by default would need an audio check (none of the runs had audio on) and a rerun on current main, as Joey suggested. Thanks for this, and for the careful fallbacks! <!-- github-sync:r:5878663856 -->
Commenting is not possible because the repository is archived.
Reviewers
No reviewers
Labels
Clear labels   
accessibility
Barrier affecting people with disabilities

  
autorelease: pending

   
autorelease: tagged

   
bug
Something isn't working

  
dependencies
Pull requests that update a dependency file

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
github_actions
Pull requests that update GitHub Actions code

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
testing wanted

   
wontfix
This will not be worked on

No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Hookshot!2738
No description provided.