perf(host): the game's whole-buffer zstd decode on the host's libzstd (HOS_NATIVE_ZSTD_FRAMES, opt-in) #155

Merged
HumanBean merged 9 commits from klimovich008/Longshot:perf/native-zstd-frames into main 2026-11-15 19:06:02 +00:00
Contributor

Why

Area loads and streaming spend CPU in the game's own copy of zstd. NotRexed/Hookshot#2738 moved its stream decode (ParallelDecompThread) to the host's libzstd. The game's other zstd path, the resource loaders' whole-buffer decode f_8010b380 on the ARes_Loading threads, is about 22% of a core in Kokiri Forest and 6% in the standard test (CPU report). This PR moves that decode to libzstd too, behind a switch that stays off by default: in game no gain was shown: on Longshot's standard test (3 pairs, diagnostic) the switch was behind in every pair, within noise, and the earlier Hookshot runs resolved no difference.

This is a performance change, not a bug fix. It also fixes a first-call data race (C memory model; no wrong output seen) in #2738's HOS_NATIVE_ZSTD switch, found in review.

What changed

  • host: hos_zstd_frames in host/hos_zstd.c, run first in f_8010b380 (zstd's ZSTD_decompressMultiFrame, inlined: x0 the game's context, x1/x2 the output and its room, x3/x4 the frames, x5/x6 a raw dictionary, x7 the game's DDict or 0). It returns what the game's function returns: the decoded size, or a zstd error code the three callers map through ZSTD_getErrorCode, as before.
    • Decodes with ZSTD_decompress_usingDDict (the game's dictionary by reference, through the stream decode's DDict cache, checked against a copy of the game's tables) or ZSTD_decompress_usingDict(NULL, 0); libzstd contexts come from the stream decode's spare list.
    • The whole output room is marked with HOS_WRITTEN after the decode (libzstd may keep literals in the room's far end while it decodes).
    • Runs lifted, as before, for a raw dictionary (none of the callers passes one), a magicless frame format in the game's context, an allocation failure, or a dictionary libzstd does not take as the game's. Each kind is logged once.
  • host, the race fix (also in #2738's code): zs_mode (HOS_NATIVE_ZSTD) and the new zf_mode cached their switch in a plain static that several decode threads read and wrote, and the once-logged fallback table was unsynchronized. zs_publish now publishes each switch once with a compare-and-swap, and only the thread that published it logs it. zs_note claims its table entry under the existing zs_lock and logs after releasing it. Only first calls and logging change; the decode path does not.
  • nx2: patch_zstd_frames in nx2/oot_patches.py puts if (hos_zstd_frames(c)) return; first in f_8010b380, counted by report() like the stream hook (it warns unless found exactly once).
  • tests: tests/test_native_zstd.py, game-free:
    • whole-buffer cases (with and without dictionary and checksum, raw and RLE blocks, empty, one byte, unknown content size, two frames with a skippable one between, more room than needed), each compared with a switch-off run outside the output room, with guard bytes around it;
    • the failure codes (checksum, dictionary, bad magic, too little room, truncated input), a DDict made again in place, and the fallbacks;
    • first calls from two threads at once, two ways: a stand-in env_on holds each caller that reads a switch unset until two have, so both go on to publish it; and a stand-in log holds the first thread in its first log line while the second runs start to end;
    • broken copies that must fail: the DDict not passed, a switch published after its log line, a publication that always stores and reports success, a fallback logged under the lock. The code before the race fix fails the concurrent checks (each switch logged twice). The fallback table's lock itself is not shown by a test: nothing holds a thread between its read and its write, and there was no thread-sanitizer run.
  • docs: both zstd switches in docs/development.md's switch reference (NotRexed/Hookshot#2738 had not added HOS_NATIVE_ZSTD); one file per measured run under docs/tested/ (the table in docs/tested-configurations.md is frozen).

Flags and settings

Flag What it does Default now Recommendation Helps on Doesn't help on Untested on
HOS_NATIVE_ZSTD_FRAMES (set) The resource loaders' whole-buffer zstd decode on the host's libzstd off off until a measurement resolves a gain not shown yet (no resolved difference) n/a Linux, NVIDIA, Intel, the Steam Deck, hybrid (P/E-core) CPUs

HOS_NATIVE_ZSTD (from #2738) is unchanged apart from the race fix.

Measurements

main: 839e85ff (2026-11-13), the base measured on
this PR: perf/native-zstd-frames @ 97915751 (839e85ff with this PR's commits; reachable as the tag pr155-measured-97915751 on my fork). The head now is rebased onto main 5e072fc8; host/hos_zstd.c is byte for byte the measured one and the zstd hook's text in nx2/oot_patches.py is the same (checked with cmp and a diff of the hook's lines). The test file changed after the measurement (Windows threads); it is not part of the game build.
Hardware: RX 9070 XT, AMD 26.10.2 (Vulkan driver 0x8001a0), Ryzen 7 5700X3D (no efficiency cores), 64 GB, Windows 11
Scene: the standard test: Hyrule Field from the test save, the camera turning at full speed; a 1280x720 window (16:9), the test profile's options (output 2048x1152), no upscaler, no frame generation, frame rate unlocked (HOS_FPS_MAX=240)
Method: one binary (longshot v3.0.0-rc.2-10-g97915751, exe d64c70b3c72deeb8, longshot_vk.dll 8eb6d56dbc601d8f), the switch unset and =1, warm-ups, then 3 alternating pairs, 20 s windows from the load (run_profile.py aee7483e, std_test.py adc2e9f0); fps and frame times from the present trace, CPU from the process. A run counts when its game clock is within 1.000 ± 0.01 and its window has no frame over 250 ms. With 3 pairs, a difference under 5% is diagnostic. One file per run: docs/tested/2026-11-13-zstd-frames-*-klimovich008-pair*.md.
How to reproduce: build this branch, then for each run set HOS_FPS_MAX=240 and HOS_FPS_LOG=1, load the save and hold the right stick fully right; alternate HOS_NATIVE_ZSTD_FRAMES unset and =1.

Baseline: this PR @ 97915751, HOS_NATIVE_ZSTD_FRAMES unset, pair 1 (ctl-0), with the same in pairs 2 (ctl-1) and 3 (ctl-2). The control is this PR's own binary with the switch off, not a build of main. It isolates the switch, but it does not measure this PR's default-off cost or the shared first-call change against main.

Each run, in the order they ran (columns left to right):

this PR @ 97915751, HOS_NATIVE_ZSTD_FRAMES unset, pair 1 (ctl-0) this PR @ 97915751, HOS_NATIVE_ZSTD_FRAMES =1, pair 1 (frames-0) this PR @ 97915751, HOS_NATIVE_ZSTD_FRAMES =1, pair 2 (frames-1) this PR @ 97915751, HOS_NATIVE_ZSTD_FRAMES unset, pair 2 (ctl-1) this PR @ 97915751, HOS_NATIVE_ZSTD_FRAMES unset, pair 3 (ctl-2) this PR @ 97915751, HOS_NATIVE_ZSTD_FRAMES =1, pair 3 (frames-2)
fps 63.39 62.94 65.09 65.99 66.04 65.45
CPU ms a frame 63.34 65.84 63.41 60.56 61.84 61.88
p99 ms 24.0 24.5 21.0 21.5 20.0 21.6
worst frame ms 59.2 605.1 72.3 46.8 53.6 72.4
frames over 100 ms 0 3 0 0 0 0
counts yes no: a frame over 250 ms yes yes yes yes
this PR @ 97915751, =1 minus unset: the 2 accepted pairs (2, 3) this PR @ 97915751, =1 minus unset: all 3 pairs
fps, per pair -0.90, -0.59 -0.45, -0.90, -0.59
fps: mean / median -0.75 / -0.75 -0.65 / -0.59
CPU ms a frame, per pair +2.85, +0.04 +2.50, +2.85, +0.04
CPU: mean / median +1.45 / +1.45 +1.80 / +2.50
  • No gain shown; diagnostic (3 pairs, differences under 5%). The switch was behind in every pair. This does not show a slowdown either: 2 or 3 pairs cannot separate it from noise.
  • Pair 1's run with the switch had 605, 128 and 114 ms frames just after its window opened. Its load was seen at 26.3 s, against 28.4-30.3 s in the other five runs, so its window opened about 2.4 s earlier relative to the game. Every run has a frame of 540-604 ms and one of 110-130 ms about 2.1 s after its load, normally before the window. In this run they fell inside it, so the run does not count.
  • The warm-ups, all with the switch unset, had frames of up to 1130 ms inside their windows.

Earlier, on Hookshot (32:9, before the race fix)

Build: d0b8141f, Hookshot v2.1.0-rc.3-250-g46780503 with #2738 and this change before the race fix (exe a46bf3a229310166, hookshot_vk.dll 418aa37c49bc96ce).

Kokiri Forest, 3 warm-ups, then 6 runs an arm: 5 of 6 pairs count.

this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES unset this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES=1 this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES=1, HOS_NATIVE_ZSTD=2
fps 82.2 (78.7-82.8) 81.7 (80.0-83.5) 82.0 (80.2-84.3)
CPU ms a frame 59.16 (58.0-62.6) 58.13 (57.3-61.7), paired -0.80 58.11 (57.1-59.9), paired -1.40
p99 ms 18.5 (17.5-20.1) 18.7 (16.2-19.9) 19.2 (15.5-19.8)
worst frame ms 40.6 26.2 94.1
frames over 100 ms 0 0 0
game clock (window) 0.999-1.004 0.998-1.002 0.998-1.002
save load s 29.28 (28.7-29.6) 28.95 (28.5-29.5) 29.06 (27.9-29.4)
core escapes (with ParallelDecompThread holding) 0 (0) 3 (0) 0 (0)
  • Pair 1 is left out: its =1 run had frames of 276 and 253 ms in the window and a clock reading of 0.933 over that stretch. The same pair's control run had a 131.6 ms frame, and the warm-ups, which run with the switch off, had frames of 295 and 345 ms. With pair 1 included, the paired CPU differences are -1.66 and -2.09 ms.

The standard test, 2 warm-ups, then 6 pairs: all 6 count.

this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES unset this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES=1
fps 53.25 (52.5-53.5) 52.6 (51.6-54.0)
CPU ms a frame 76.69 (75.9-77.3) 75.86 (74.0-78.8), paired -0.47
p99 ms 25.7 (24.7-29.7) 25.4 (23.5-37.8)
worst frame ms 60.9 96.0
frames over 100 ms 0 0
game clock (window) 0.998-1.004 0.997-1.003
save load s 29.20 (28.3-29.7) 28.74 (28.4-29.8)
core escapes (with ParallelDecompThread holding) 5 (0) 4 (0)
  • A stricter clock view, the median of each run's last four clock lines, voids two runs with the switch (pairs 1 and 4: 0.948 and 0.955). Those four lines include the clock's dip at the end of the save load, which in these two runs fell across two lines (0.661 then 0.895; 0.695 then 0.912) instead of one, as in the other ten (0.645-0.805). Inside the window both read 1.000 and 1.000, and 1.000 and 0.998, with worst frames of 62.4 and 50.5 ms and save loads of 29.7 and 28.9 s. Two of six runs with the switch against none of six without it is not enough to say whether that split follows the switch. On the 4 pairs that view keeps: fps 53.25 / 52.95, CPU 76.28 / 75.64 ms (paired -0.72), p99 25.7 / 25.4, save load 29.20 / 28.52 s.
  • The CPU differences are inside the control's run-to-run range (58.0-62.6 ms in Kokiri, 75.9-77.3 ms in Hyrule). Save load is 0.2-0.5 s shorter in every comparison, inside a range of about 1.4 s.
  • No run logged a fallback: every whole-buffer decode ran on the host.
  • Another agent held the GPU lock for about 5 minutes between two Hyrule runs (not during one); the run after it is not an outlier.

HOS_NATIVE_ZSTD=1 while moving (context for #2738)

A tester saw HOS_NATIVE_ZSTD=1 slower than unset in Kokiri Forest while moving (RTX 5090, i7-13700K: 72.2 against 66.7 fps). I checked it on this PC with #2738 as merged (before the race fix) and this change switched off (exe e9be69d36122b25e): Kokiri Forest from the default save, running with the camera turning, 2 warm-ups, then 6 triples. Two triples are left out by the rules above: an unset run with frames of 576, 284 and 135 ms, and an =2 run with a window clock reading of 0.990. On the other 4:

this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD unset this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD=1 this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD=2
fps 51.4 53.55 (paired +2.0) 52.2 (paired +0.75)
CPU ms a frame 85.71 83.60 (paired -2.46) 85.16 (paired -0.89)
p99 ms 30.3 30.4 31.0
worst frame ms 39.7 49.9 47.7
save load s 29.57 28.93 28.87
  • =1 was faster in all 4 of those pairs; in one more pair (valid for unset against =1 only) it was slower, 51.0 against 53.2 fps.
  • The main thread (35% of a core), the Presentation Thread (73-74%) and the resource loaders cost the same in every arm, and =1 makes no more virtual-core hand-offs than unset. ParallelDecompThread went from about 2% of a core to about 0.
  • This does not rule the report out: this PC's CPU has no efficiency cores, and the stream decode was only 0-5% of a core in this window against 30% in the report's.
  • This PR's whole-buffer decode never gives its core up, so it does not share what =1 changes.

Testing

Tested:

  • The game's own function, locally (no game data here or in the test): f_8010b380 called through one of its callers in Unicorn from a local image, against hos_zstd_frames on the same inputs: 46 cases, 49 calls (the test's kinds plus larger sets up to 2 MB, errors, a DDict made again). Results, including error codes, and the decoded output were identical. In 2 of the cases with 1000 bytes more room than the content, the bytes in the room past the content differ (libzstd keeps literals there while it decodes). A broken copy fails 8 cases.
  • python -I -B tests/run_all.py on this branch at f81d2f93, on main 5e072fc8 (Windows, llvm-mingw, LONGSHOT_MINGW set): 519 passed, 37 skipped, 5 failed: tests/test_module_ids.py, tests/test_pad_layout.py, tests/test_framegen_resources.py, tests/test_tested_configurations.py and tests/test_oot_state_tap.py. All five fail the same way on unmodified main 5e072fc8 on this PC; this PR touches none of what they cover.
  • tests/test_native_zstd.py passes with Clang targeting Microsoft's runtime (23.1.5, x86_64-pc-windows-msvc), llvm-mingw and MinGW gcc 16.2, and every broken copy in it fails with each. On Windows the concurrent fixture uses Windows threads; its POSIX branch was also built and run here with MinGW gcc and winpthreads (correct code passes, the three concurrent broken copies fail). host/hos_zstd.c compiles as a whole (clang -fsyntax-only against a 839e85ff build's generated headers).
  • The build log of the measured build shows both zstd hooks applied once, with no warnings.
  • In game: the runs above, on Windows 11 with an RX 9070 XT.

Not tested:

  • Other scenes on Longshot: A/Bs are the standard test only.
  • The race fix in game: it changes only how the switches are read once and logged, and was tested in the fixture only.
  • Linux (no OS branch in the code, but not built or run there).
  • First visits with cold caches, other scenes, sessions longer than a minute, more than 6 pairs.
  • Raw dictionaries and the magicless format: no caller passes them, so they stay lifted.

Needs other testers:

  • NVIDIA, Intel: the standard test (Hyrule Field, camera turning, unlocked frame rate), HOS_NATIVE_ZSTD_FRAMES unset against =1, 3 alternating pairs, expecting no slowdown and no new long frames (T-S3LD)
  • Hybrid (P/E-core) CPU: the same, plus HOS_NATIVE_ZSTD=2 and =1 against unset, expecting no slowdown (T-J7B5)
  • Linux: the same on any GPU, expecting no slowdown and the log line naming the switch (T-6JMF)

Screenshots

Nothing on screen.

Risk and rollback

A wrong decode would show as missing or broken models, textures or effects after an area load, or a load that fails. It is off unless HOS_NATIVE_ZSTD_FRAMES is set; unset, the hook returns at its first check and the game's own decode runs. The race fix changes only first-call setup and logging; reverting its commit restores the old first-call code.

Reviewer notes

Start with hos_zstd_frames and the race fix at the top of host/hos_zstd.c, then the whole-buffer and concurrent cases in tests/test_native_zstd.py. Prior art: zortos293/Lichii#869 puts this game's zstd decode on the host's libzstd in an emulator fork. I did not read its code; it's credited as related work. Also related: NotRexed/Hookshot#909 (the lifted decoder at -O2).

Issues

See NotRexed/Hookshot#2738 (its first-call race is fixed here).

Checklist

For everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description).
  • Rebased on current main.
  • python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
  • Tested in the game, or what wasn't is named under Testing.
  • Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
  • Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (By switch, on this PR's own binary: the control is the switch off, not a build of main, so the default-off cost and the first-call change against main are not measured.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Flags noted, and the switch reference updated (docs/development.md, the README).
  • A file under docs/tested/ for each measured run.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.

🤖 Generated with Claude Code

## Why Area loads and streaming spend CPU in the game's own copy of zstd. [NotRexed/Hookshot#2738](https://github.com/NotRexed/Hookshot/pull/2738) moved its stream decode (ParallelDecompThread) to the host's libzstd. The game's other zstd path, the resource loaders' whole-buffer decode `f_8010b380` on the ARes_Loading threads, is about 22% of a core in Kokiri Forest and 6% in the standard test (CPU report). This PR moves that decode to libzstd too, behind a switch that stays off by default: in game no gain was shown: on Longshot's standard test (3 pairs, diagnostic) the switch was behind in every pair, within noise, and the earlier Hookshot runs resolved no difference. This is a performance change, not a bug fix. It also fixes a first-call data race (C memory model; no wrong output seen) in #2738's `HOS_NATIVE_ZSTD` switch, found in review. ## What changed - host: `hos_zstd_frames` in `host/hos_zstd.c`, run first in `f_8010b380` (zstd's `ZSTD_decompressMultiFrame`, inlined: x0 the game's context, x1/x2 the output and its room, x3/x4 the frames, x5/x6 a raw dictionary, x7 the game's DDict or 0). It returns what the game's function returns: the decoded size, or a zstd error code the three callers map through `ZSTD_getErrorCode`, as before. - Decodes with `ZSTD_decompress_usingDDict` (the game's dictionary by reference, through the stream decode's DDict cache, checked against a copy of the game's tables) or `ZSTD_decompress_usingDict(NULL, 0)`; libzstd contexts come from the stream decode's spare list. - The whole output room is marked with `HOS_WRITTEN` after the decode (libzstd may keep literals in the room's far end while it decodes). - Runs lifted, as before, for a raw dictionary (none of the callers passes one), a magicless frame format in the game's context, an allocation failure, or a dictionary libzstd does not take as the game's. Each kind is logged once. - host, the race fix (also in #2738's code): `zs_mode` (`HOS_NATIVE_ZSTD`) and the new `zf_mode` cached their switch in a plain static that several decode threads read and wrote, and the once-logged fallback table was unsynchronized. `zs_publish` now publishes each switch once with a compare-and-swap, and only the thread that published it logs it. `zs_note` claims its table entry under the existing `zs_lock` and logs after releasing it. Only first calls and logging change; the decode path does not. - nx2: `patch_zstd_frames` in `nx2/oot_patches.py` puts `if (hos_zstd_frames(c)) return;` first in `f_8010b380`, counted by `report()` like the stream hook (it warns unless found exactly once). - tests: `tests/test_native_zstd.py`, game-free: - whole-buffer cases (with and without dictionary and checksum, raw and RLE blocks, empty, one byte, unknown content size, two frames with a skippable one between, more room than needed), each compared with a switch-off run outside the output room, with guard bytes around it; - the failure codes (checksum, dictionary, bad magic, too little room, truncated input), a DDict made again in place, and the fallbacks; - first calls from two threads at once, two ways: a stand-in `env_on` holds each caller that reads a switch unset until two have, so both go on to publish it; and a stand-in log holds the first thread in its first log line while the second runs start to end; - broken copies that must fail: the DDict not passed, a switch published after its log line, a publication that always stores and reports success, a fallback logged under the lock. The code before the race fix fails the concurrent checks (each switch logged twice). The fallback table's lock itself is not shown by a test: nothing holds a thread between its read and its write, and there was no thread-sanitizer run. - docs: both zstd switches in `docs/development.md`'s switch reference ([NotRexed/Hookshot#2738](https://github.com/NotRexed/Hookshot/pull/2738) had not added `HOS_NATIVE_ZSTD`); one file per measured run under `docs/tested/` (the table in `docs/tested-configurations.md` is frozen). ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `HOS_NATIVE_ZSTD_FRAMES` (set) | The resource loaders' whole-buffer zstd decode on the host's libzstd | off | off until a measurement resolves a gain | not shown yet (no resolved difference) | n/a | Linux, NVIDIA, Intel, the Steam Deck, hybrid (P/E-core) CPUs | `HOS_NATIVE_ZSTD` (from #2738) is unchanged apart from the race fix. ## Measurements main: 839e85ff (2026-11-13), the base measured on this PR: perf/native-zstd-frames @ 97915751 (839e85ff with this PR's commits; reachable as the tag [pr155-measured-97915751](https://git.magicbean.dev/klimovich008/Longshot/src/tag/pr155-measured-97915751) on my fork). The head now is rebased onto main 5e072fc8; `host/hos_zstd.c` is byte for byte the measured one and the zstd hook's text in `nx2/oot_patches.py` is the same (checked with `cmp` and a diff of the hook's lines). The test file changed after the measurement (Windows threads); it is not part of the game build. Hardware: RX 9070 XT, AMD 26.10.2 (Vulkan driver 0x8001a0), Ryzen 7 5700X3D (no efficiency cores), 64 GB, Windows 11 Scene: the standard test: Hyrule Field from the test save, the camera turning at full speed; a 1280x720 window (16:9), the test profile's options (output 2048x1152), no upscaler, no frame generation, frame rate unlocked (`HOS_FPS_MAX=240`) Method: one binary (`longshot v3.0.0-rc.2-10-g97915751`, exe `d64c70b3c72deeb8`, `longshot_vk.dll` `8eb6d56dbc601d8f`), the switch unset and `=1`, warm-ups, then 3 alternating pairs, 20 s windows from the load (`run_profile.py` aee7483e, `std_test.py` adc2e9f0); fps and frame times from the present trace, CPU from the process. A run counts when its game clock is within 1.000 ± 0.01 and its window has no frame over 250 ms. With 3 pairs, a difference under 5% is diagnostic. One file per run: `docs/tested/2026-11-13-zstd-frames-*-klimovich008-pair*.md`. How to reproduce: build this branch, then for each run set `HOS_FPS_MAX=240` and `HOS_FPS_LOG=1`, load the save and hold the right stick fully right; alternate `HOS_NATIVE_ZSTD_FRAMES` unset and `=1`. Baseline: this PR @ 97915751, `HOS_NATIVE_ZSTD_FRAMES` unset, pair 1 (ctl-0), with the same in pairs 2 (ctl-1) and 3 (ctl-2). **The control is this PR's own binary with the switch off, not a build of main.** It isolates the switch, but it does not measure this PR's default-off cost or the shared first-call change against main. Each run, in the order they ran (columns left to right): | | this PR @ 97915751, `HOS_NATIVE_ZSTD_FRAMES` unset, pair 1 (ctl-0) | this PR @ 97915751, `HOS_NATIVE_ZSTD_FRAMES` `=1`, pair 1 (frames-0) | this PR @ 97915751, `HOS_NATIVE_ZSTD_FRAMES` `=1`, pair 2 (frames-1) | this PR @ 97915751, `HOS_NATIVE_ZSTD_FRAMES` unset, pair 2 (ctl-1) | this PR @ 97915751, `HOS_NATIVE_ZSTD_FRAMES` unset, pair 3 (ctl-2) | this PR @ 97915751, `HOS_NATIVE_ZSTD_FRAMES` `=1`, pair 3 (frames-2) | |---|---|---|---|---|---|---| | fps | 63.39 | 62.94 | 65.09 | 65.99 | 66.04 | 65.45 | | CPU ms a frame | 63.34 | 65.84 | 63.41 | 60.56 | 61.84 | 61.88 | | p99 ms | 24.0 | 24.5 | 21.0 | 21.5 | 20.0 | 21.6 | | worst frame ms | 59.2 | 605.1 | 72.3 | 46.8 | 53.6 | 72.4 | | frames over 100 ms | 0 | 3 | 0 | 0 | 0 | 0 | | counts | yes | no: a frame over 250 ms | yes | yes | yes | yes | | | this PR @ 97915751, `=1` minus unset: the 2 accepted pairs (2, 3) | this PR @ 97915751, `=1` minus unset: all 3 pairs | |---|---|---| | fps, per pair | -0.90, -0.59 | -0.45, -0.90, -0.59 | | fps: mean / median | -0.75 / -0.75 | -0.65 / -0.59 | | CPU ms a frame, per pair | +2.85, +0.04 | +2.50, +2.85, +0.04 | | CPU: mean / median | +1.45 / +1.45 | +1.80 / +2.50 | - No gain shown; diagnostic (3 pairs, differences under 5%). The switch was behind in every pair. This does not show a slowdown either: 2 or 3 pairs cannot separate it from noise. - Pair 1's run with the switch had 605, 128 and 114 ms frames just after its window opened. Its load was seen at 26.3 s, against 28.4-30.3 s in the other five runs, so its window opened about 2.4 s earlier relative to the game. Every run has a frame of 540-604 ms and one of 110-130 ms about 2.1 s after its load, normally before the window. In this run they fell inside it, so the run does not count. - The warm-ups, all with the switch unset, had frames of up to 1130 ms inside their windows. ### Earlier, on Hookshot (32:9, before the race fix) Build: d0b8141f, Hookshot v2.1.0-rc.3-250-g46780503 with #2738 and this change before the race fix (exe `a46bf3a229310166`, `hookshot_vk.dll` `418aa37c49bc96ce`). Kokiri Forest, 3 warm-ups, then 6 runs an arm: 5 of 6 pairs count. | | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES` unset | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES=1` | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES=1`, `HOS_NATIVE_ZSTD=2` | |---|---|---|---| | fps | 82.2 (78.7-82.8) | 81.7 (80.0-83.5) | 82.0 (80.2-84.3) | | CPU ms a frame | 59.16 (58.0-62.6) | 58.13 (57.3-61.7), paired -0.80 | 58.11 (57.1-59.9), paired -1.40 | | p99 ms | 18.5 (17.5-20.1) | 18.7 (16.2-19.9) | 19.2 (15.5-19.8) | | worst frame ms | 40.6 | 26.2 | 94.1 | | frames over 100 ms | 0 | 0 | 0 | | game clock (window) | 0.999-1.004 | 0.998-1.002 | 0.998-1.002 | | save load s | 29.28 (28.7-29.6) | 28.95 (28.5-29.5) | 29.06 (27.9-29.4) | | core escapes (with ParallelDecompThread holding) | 0 (0) | 3 (0) | 0 (0) | - Pair 1 is left out: its `=1` run had frames of 276 and 253 ms in the window and a clock reading of 0.933 over that stretch. The same pair's control run had a 131.6 ms frame, and the warm-ups, which run with the switch off, had frames of 295 and 345 ms. With pair 1 included, the paired CPU differences are -1.66 and -2.09 ms. The standard test, 2 warm-ups, then 6 pairs: all 6 count. | | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES` unset | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES=1` | |---|---|---| | fps | 53.25 (52.5-53.5) | 52.6 (51.6-54.0) | | CPU ms a frame | 76.69 (75.9-77.3) | 75.86 (74.0-78.8), paired -0.47 | | p99 ms | 25.7 (24.7-29.7) | 25.4 (23.5-37.8) | | worst frame ms | 60.9 | 96.0 | | frames over 100 ms | 0 | 0 | | game clock (window) | 0.998-1.004 | 0.997-1.003 | | save load s | 29.20 (28.3-29.7) | 28.74 (28.4-29.8) | | core escapes (with ParallelDecompThread holding) | 5 (0) | 4 (0) | - A stricter clock view, the median of each run's last four clock lines, voids two runs with the switch (pairs 1 and 4: 0.948 and 0.955). Those four lines include the clock's dip at the end of the save load, which in these two runs fell across two lines (0.661 then 0.895; 0.695 then 0.912) instead of one, as in the other ten (0.645-0.805). Inside the window both read 1.000 and 1.000, and 1.000 and 0.998, with worst frames of 62.4 and 50.5 ms and save loads of 29.7 and 28.9 s. Two of six runs with the switch against none of six without it is not enough to say whether that split follows the switch. On the 4 pairs that view keeps: fps 53.25 / 52.95, CPU 76.28 / 75.64 ms (paired -0.72), p99 25.7 / 25.4, save load 29.20 / 28.52 s. - The CPU differences are inside the control's run-to-run range (58.0-62.6 ms in Kokiri, 75.9-77.3 ms in Hyrule). Save load is 0.2-0.5 s shorter in every comparison, inside a range of about 1.4 s. - No run logged a fallback: every whole-buffer decode ran on the host. - Another agent held the GPU lock for about 5 minutes between two Hyrule runs (not during one); the run after it is not an outlier. ### `HOS_NATIVE_ZSTD=1` while moving (context for #2738) A tester saw `HOS_NATIVE_ZSTD=1` slower than unset in Kokiri Forest while moving (RTX 5090, i7-13700K: 72.2 against 66.7 fps). I checked it on this PC with #2738 as merged (before the race fix) and this change switched off (exe `e9be69d36122b25e`): Kokiri Forest from the default save, running with the camera turning, 2 warm-ups, then 6 triples. Two triples are left out by the rules above: an unset run with frames of 576, 284 and 135 ms, and an `=2` run with a window clock reading of 0.990. On the other 4: | | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD` unset | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD=1` | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD=2` | |---|---|---|---| | fps | 51.4 | 53.55 (paired +2.0) | 52.2 (paired +0.75) | | CPU ms a frame | 85.71 | 83.60 (paired -2.46) | 85.16 (paired -0.89) | | p99 ms | 30.3 | 30.4 | 31.0 | | worst frame ms | 39.7 | 49.9 | 47.7 | | save load s | 29.57 | 28.93 | 28.87 | - `=1` was faster in all 4 of those pairs; in one more pair (valid for unset against `=1` only) it was slower, 51.0 against 53.2 fps. - The main thread (35% of a core), the Presentation Thread (73-74%) and the resource loaders cost the same in every arm, and `=1` makes no more virtual-core hand-offs than unset. ParallelDecompThread went from about 2% of a core to about 0. - This does not rule the report out: this PC's CPU has no efficiency cores, and the stream decode was only 0-5% of a core in this window against 30% in the report's. - This PR's whole-buffer decode never gives its core up, so it does not share what `=1` changes. ## Testing **Tested:** - The game's own function, locally (no game data here or in the test): `f_8010b380` called through one of its callers in Unicorn from a local image, against `hos_zstd_frames` on the same inputs: 46 cases, 49 calls (the test's kinds plus larger sets up to 2 MB, errors, a DDict made again). Results, including error codes, and the decoded output were identical. In 2 of the cases with 1000 bytes more room than the content, the bytes in the room past the content differ (libzstd keeps literals there while it decodes). A broken copy fails 8 cases. - `python -I -B tests/run_all.py` on this branch at f81d2f93, on main 5e072fc8 (Windows, llvm-mingw, `LONGSHOT_MINGW` set): 519 passed, 37 skipped, 5 failed: `tests/test_module_ids.py`, `tests/test_pad_layout.py`, `tests/test_framegen_resources.py`, `tests/test_tested_configurations.py` and `tests/test_oot_state_tap.py`. All five fail the same way on unmodified main 5e072fc8 on this PC; this PR touches none of what they cover. - `tests/test_native_zstd.py` passes with Clang targeting Microsoft's runtime (23.1.5, x86_64-pc-windows-msvc), llvm-mingw and MinGW gcc 16.2, and every broken copy in it fails with each. On Windows the concurrent fixture uses Windows threads; its POSIX branch was also built and run here with MinGW gcc and winpthreads (correct code passes, the three concurrent broken copies fail). `host/hos_zstd.c` compiles as a whole (`clang -fsyntax-only` against a 839e85ff build's generated headers). - The build log of the measured build shows both zstd hooks applied once, with no warnings. - In game: the runs above, on Windows 11 with an RX 9070 XT. **Not tested:** - Other scenes on Longshot: A/Bs are the standard test only. - The race fix in game: it changes only how the switches are read once and logged, and was tested in the fixture only. - Linux (no OS branch in the code, but not built or run there). - First visits with cold caches, other scenes, sessions longer than a minute, more than 6 pairs. - Raw dictionaries and the magicless format: no caller passes them, so they stay lifted. **Needs other testers:** - NVIDIA, Intel: the standard test (Hyrule Field, camera turning, unlocked frame rate), `HOS_NATIVE_ZSTD_FRAMES` unset against `=1`, 3 alternating pairs, expecting no slowdown and no new long frames (T-S3LD) - Hybrid (P/E-core) CPU: the same, plus `HOS_NATIVE_ZSTD=2` and `=1` against unset, expecting no slowdown (T-J7B5) - Linux: the same on any GPU, expecting no slowdown and the log line naming the switch (T-6JMF) ## Screenshots Nothing on screen. ## Risk and rollback A wrong decode would show as missing or broken models, textures or effects after an area load, or a load that fails. It is off unless `HOS_NATIVE_ZSTD_FRAMES` is set; unset, the hook returns at its first check and the game's own decode runs. The race fix changes only first-call setup and logging; reverting its commit restores the old first-call code. ## Reviewer notes Start with `hos_zstd_frames` and the race fix at the top of `host/hos_zstd.c`, then the whole-buffer and concurrent cases in `tests/test_native_zstd.py`. Prior art: [zortos293/Lichii#869](https://github.com/zortos293/Lichii/pull/869) puts this game's zstd decode on the host's libzstd in an emulator fork. I did not read its code; it's credited as related work. Also related: [NotRexed/Hookshot#909](https://github.com/NotRexed/Hookshot/pull/909) (the lifted decoder at -O2). ## Issues See [NotRexed/Hookshot#2738](https://github.com/NotRexed/Hookshot/pull/2738) (its first-call race is fixed here). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [x] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (By switch, on this PR's own binary: the control is the switch off, not a build of main, so the default-off cost and the first-call change against main are not measured.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [x] A file under `docs/tested/` for each measured run. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
klimovich008 force-pushed perf/native-zstd-frames from a0ff2d9491
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to 12f8cc8ed9
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 3s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-13 20:26:01 +00:00
Compare
Author
Contributor

Measured on Longshot and rebased:

  • The standard test, 839e85ff + this PR (97915751), 3 alternating pairs: HOS_NATIVE_ZSTD_FRAMES unset 65.99 fps / 61.84 ms CPU a frame, =1 65.09 fps / 63.41 ms. No gain, diagnostic. The switch was behind in all 3 pairs (fps -0.45 / -0.90 / -0.59).
  • The 605 ms frame: it came in the switch's first run. That run's load was seen 2.4 s earlier than in the others, so the frame every run has about 2.1 s after its load fell inside its window. That run fails the window rule and is reported as data. The Measurements section has the details.
  • Rebased onto main 514591a3 (12f8cc8e). host/hos_zstd.c, the test and the zstd hook are byte for byte the measured ones.
    • run_all: 494 passed, 4 failed. All four fail on main on this PC too; test_menu_i18n.py fails on main 514591a3 itself.
    • The zstd test passes with llvm-mingw and with MinGW gcc.
  • Docs: two rows in docs/tested-configurations.md for these runs.
  • Test requests opened: T-S3LD (NVIDIA, Intel), T-J7B5 (CPU with performance and efficiency cores), T-6JMF (Linux). They need the "testing wanted" label before anyone can claim them.

The switch stays off by default.

Measured on Longshot and rebased: - **The standard test, 839e85ff + this PR (97915751), 3 alternating pairs:** `HOS_NATIVE_ZSTD_FRAMES` unset 65.99 fps / 61.84 ms CPU a frame, `=1` 65.09 fps / 63.41 ms. No gain, diagnostic. The switch was behind in all 3 pairs (fps -0.45 / -0.90 / -0.59). - **The 605 ms frame:** it came in the switch's first run. That run's load was seen 2.4 s earlier than in the others, so the frame every run has about 2.1 s after its load fell inside its window. That run fails the window rule and is reported as data. The Measurements section has the details. - **Rebased onto main 514591a3 (12f8cc8e).** `host/hos_zstd.c`, the test and the zstd hook are byte for byte the measured ones. - `run_all`: 494 passed, 4 failed. All four fail on main on this PC too; `test_menu_i18n.py` fails on main 514591a3 itself. - The zstd test passes with llvm-mingw and with MinGW gcc. - **Docs:** two rows in `docs/tested-configurations.md` for these runs. - **Test requests opened:** T-S3LD (NVIDIA, Intel), T-J7B5 (CPU with performance and efficiency cores), T-6JMF (Linux). They need the "testing wanted" label before anyone can claim them. The switch stays off by default.
klimovich008 changed title from WIP: perf(host): the game's whole-buffer zstd decode on the host's libzstd (HOS_NATIVE_ZSTD_FRAMES, opt-in) to perf(host): the game's whole-buffer zstd decode on the host's libzstd (HOS_NATIVE_ZSTD_FRAMES, opt-in) 2026-11-13 20:27:54 +00:00
mooki left a comment

Read head 12f8cc8e, the complete discussion and all five changed files against main cf605c85, including the hook's callers and the shared decoder cache; ran test_native_zstd.py and test_env_on.py with GCC and Clang, plus test_patch_hooks_defined.py: all passed.

What's good

  • The first-call race is real in main. Publishing the scalar switch atomically, and claiming the fallback message under the lock but logging outside it, addresses that without holding the lock across decoding.
  • The whole-buffer hook reuses the existing context and dictionary ownership rules. Its return value reaches the callers' existing error handling, and the output room is marked after decoding, including error returns.
  • The tests check output, guard bytes, dictionary replacement and concurrent first calls. All five deliberately broken variants were rejected with both compilers. The description correctly says that the measurements show no gain, and the switch stays off by default.

Needed before it can merge

  1. Keep the concurrent test runnable with Windows' Microsoft-target Clang. tests/test_native_zstd.py lines 168-169 now include pthread.h and sched.h unconditionally, and line 641 adds -pthread. tests/run_all.py can select ordinary clang++ on Windows, not just llvm-mingw. The Microsoft CRT/SDK does not supply these POSIX headers or thread functions, so this newly mandatory part of the test cannot compile there. The reported llvm-mingw and MinGW GCC passes do not cover that compiler target. Use a Windows thread implementation alongside the POSIX one, keeping the same concurrent assertions and broken variants, and run the test with Clang targeting Microsoft's runtime. This is a source portability finding; I did not execute that Windows configuration.
  2. Resolve the measured-runs conflict using main's new format. docs/tested-configurations.md lines 17-18 conflict with main's newer rows at the start of the table. This is the only textual merge conflict at cf605c85; the other four changed files merge. Main now freezes that table, so move this PR's new measurements into individual files under docs/tested/, rather than appending the seven summary rows during the rebase. Keep main's existing rows. The format is at cf605c85bb/docs/tested/README.md.

Smaller points

  • The latest table's paired -0.65 fps and +1.80 CPU ms match the arithmetic means of the three listed pair differences, not their medians. Those medians are -0.59 fps and +2.50 ms. Label the paired summaries as means, or use medians, and show the accepted pairs separately from the rejected first pair. A row per run would make the calculation and exclusions checkable.
  • The baseline is explicitly the same PR binary with the switch off, not a separate build of main. That is a useful switch comparison, but it does not measure the PR's default-off cost or the shared first-call change against main. Keep that distinction in the measurement checklist too. The reported ranges support “no gain shown”, not a speed-up or proof of no slowdown.

Not checked

No full build, whole suite, game run, GPU measurement or Windows execution here. The original-decoder comparison and the gameplay results remain the author's evidence. No raw run records are linked in the discussion, so I could not independently recompute the run medians or assess background load. Measured commit 97915751 was unavailable in the fetched history and through the fork's public commit and raw-file URLs, so I have not verified the claimed byte-for-byte carry to this head.

Read head `12f8cc8e`, the complete discussion and all five changed files against main `cf605c85`, including the hook's callers and the shared decoder cache; ran `test_native_zstd.py` and `test_env_on.py` with GCC and Clang, plus `test_patch_hooks_defined.py`: all passed. ### What's good - The first-call race is real in main. Publishing the scalar switch atomically, and claiming the fallback message under the lock but logging outside it, addresses that without holding the lock across decoding. - The whole-buffer hook reuses the existing context and dictionary ownership rules. Its return value reaches the callers' existing error handling, and the output room is marked after decoding, including error returns. - The tests check output, guard bytes, dictionary replacement and concurrent first calls. All five deliberately broken variants were rejected with both compilers. The description correctly says that the measurements show no gain, and the switch stays off by default. ### Needed before it can merge 1. **Keep the concurrent test runnable with Windows' Microsoft-target Clang.** `tests/test_native_zstd.py` lines 168-169 now include `pthread.h` and `sched.h` unconditionally, and line 641 adds `-pthread`. `tests/run_all.py` can select ordinary `clang++` on Windows, not just llvm-mingw. The Microsoft CRT/SDK does not supply these POSIX headers or thread functions, so this newly mandatory part of the test cannot compile there. The reported llvm-mingw and MinGW GCC passes do not cover that compiler target. Use a Windows thread implementation alongside the POSIX one, keeping the same concurrent assertions and broken variants, and run the test with Clang targeting Microsoft's runtime. This is a source portability finding; I did not execute that Windows configuration. 2. **Resolve the measured-runs conflict using main's new format.** `docs/tested-configurations.md` lines 17-18 conflict with main's newer rows at the start of the table. This is the only textual merge conflict at `cf605c85`; the other four changed files merge. Main now freezes that table, so move this PR's new measurements into individual files under `docs/tested/`, rather than appending the seven summary rows during the rebase. Keep main's existing rows. The format is at https://git.magicbean.dev/MagicBeanDev/Longshot/src/commit/cf605c85bbaeaaa78176b933b3bc1a3366e13a78/docs/tested/README.md. ### Smaller points - The latest table's paired `-0.65` fps and `+1.80` CPU ms match the arithmetic means of the three listed pair differences, not their medians. Those medians are `-0.59` fps and `+2.50` ms. Label the paired summaries as means, or use medians, and show the accepted pairs separately from the rejected first pair. A row per run would make the calculation and exclusions checkable. - The baseline is explicitly the same PR binary with the switch off, not a separate build of main. That is a useful switch comparison, but it does not measure the PR's default-off cost or the shared first-call change against main. Keep that distinction in the measurement checklist too. The reported ranges support “no gain shown”, not a speed-up or proof of no slowdown. ### Not checked No full build, whole suite, game run, GPU measurement or Windows execution here. The original-decoder comparison and the gameplay results remain the author's evidence. No raw run records are linked in the discussion, so I could not independently recompute the run medians or assess background load. Measured commit `97915751` was unavailable in the fetched history and through the fork's public commit and raw-file URLs, so I have not verified the claimed byte-for-byte carry to this head.
klimovich008 force-pushed perf/native-zstd-frames from 12f8cc8ed9
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 3s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to f81d2f93b0
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
Website / Website on Cloudflare (pull_request_target) Has been skipped
2026-11-14 12:16:33 +00:00
Compare
Author
Contributor

@mooki thanks for review 1244. Every point, in order. New head f81d2f93, rebased onto main 5e072fc8.

Needed 1: the concurrent test with Microsoft-target Clang. Fixed in 1b179574.

  • The fixture now has a small thread layer:
    • under _WIN32: CreateThread, WaitForSingleObject + CloseHandle, SwitchToThread, with WIN32_LEAN_AND_MEAN and NOMINMAX;
    • elsewhere: pthreads and sched_yield.
  • -pthread is added only when os.name != "nt". The assertions and all broken copies are unchanged.
  • Run here with Clang 23.1.5 targeting x86_64-pc-windows-msvc, and with llvm-mingw and MinGW gcc 16.2. All pass, and every broken copy fails with each.
  • On Windows every compiler now takes the Windows branch. So I also built and ran the POSIX branch once with MinGW gcc and winpthreads, as a local check outside the repo: correct code passes, and the three concurrent broken copies fail.

Needed 2: the measured runs in main's new format. Fixed by the rebase and f81d2f93.

  • The branch was rebuilt on 5e072fc8 without the three commits that only added rows to the frozen table. docs/tested-configurations.md is now main's, unchanged.
  • The six Longshot runs are one file each: docs/tested/2026-11-13-zstd-frames-{off,on}-klimovich008-pair{1,2,3}.md. Each has its pair, its position in the pair and its own numbers. The rejected run says why.
  • tests/test_tested_configurations.py accepts them. Its one failure here, the CRLF pipe check, also fails on unmodified main on Windows.
  • The Hookshot-era runs stay only in the description.

Smaller 1: means against medians, the accepted pairs apart. In the description: a row per run in run order, with a "counts" column. A second table shows the per-pair differences, with means and medians labelled, for the 2 accepted pairs and for all 3. Accepted pairs: fps -0.75 (mean = median), CPU +1.45 ms (mean = median). All 3: fps mean -0.65, median -0.59; CPU mean +1.80, median +2.50.

Smaller 2: what the control measures.

  • The description now says plainly that the control is this PR's own binary with the switch off, not a build of main. So it does not measure the default-off cost or the shared first-call change against main.
  • The Measured checklist box says the same.
  • The wording is now "no gain shown", and it says the runs show no slowdown either way.

Not checked: 97915751. It's now on my fork as the tag pr155-measured-97915751: https://git.magicbean.dev/klimovich008/Longshot/src/tag/pr155-measured-97915751

  • The carry to this head was checked by cmp of host/hos_zstd.c, which is byte for byte the same, and a diff of the zstd hook's lines in nx2/oot_patches.py: same text, other line numbers.
  • The test file changed since then; it isn't part of the game build.

Also run:

  • run_all at f81d2f93: 519 passed, 5 failed. All five fail the same way on unmodified main 5e072fc8 on this PC: test_module_ids, test_pad_layout, test_framegen_resources, test_tested_configurations, test_oot_state_tap.
  • host/hos_zstd.c syntax-compiles whole.
  • A second review of the new head found no defects.
@mooki thanks for review 1244. Every point, in order. New head `f81d2f93`, rebased onto main `5e072fc8`. **Needed 1: the concurrent test with Microsoft-target Clang.** Fixed in `1b179574`. - The fixture now has a small thread layer: - under `_WIN32`: `CreateThread`, `WaitForSingleObject` + `CloseHandle`, `SwitchToThread`, with `WIN32_LEAN_AND_MEAN` and `NOMINMAX`; - elsewhere: pthreads and `sched_yield`. - `-pthread` is added only when `os.name != "nt"`. The assertions and all broken copies are unchanged. - Run here with Clang 23.1.5 targeting `x86_64-pc-windows-msvc`, and with llvm-mingw and MinGW gcc 16.2. All pass, and every broken copy fails with each. - On Windows every compiler now takes the Windows branch. So I also built and ran the POSIX branch once with MinGW gcc and winpthreads, as a local check outside the repo: correct code passes, and the three concurrent broken copies fail. **Needed 2: the measured runs in main's new format.** Fixed by the rebase and `f81d2f93`. - The branch was rebuilt on `5e072fc8` without the three commits that only added rows to the frozen table. `docs/tested-configurations.md` is now main's, unchanged. - The six Longshot runs are one file each: `docs/tested/2026-11-13-zstd-frames-{off,on}-klimovich008-pair{1,2,3}.md`. Each has its pair, its position in the pair and its own numbers. The rejected run says why. - `tests/test_tested_configurations.py` accepts them. Its one failure here, the CRLF pipe check, also fails on unmodified main on Windows. - The Hookshot-era runs stay only in the description. **Smaller 1: means against medians, the accepted pairs apart.** In the description: a row per run in run order, with a "counts" column. A second table shows the per-pair differences, with means and medians labelled, for the 2 accepted pairs and for all 3. Accepted pairs: fps -0.75 (mean = median), CPU +1.45 ms (mean = median). All 3: fps mean -0.65, median -0.59; CPU mean +1.80, median +2.50. **Smaller 2: what the control measures.** - The description now says plainly that the control is this PR's own binary with the switch off, not a build of main. So it does not measure the default-off cost or the shared first-call change against main. - The Measured checklist box says the same. - The wording is now "no gain shown", and it says the runs show no slowdown either way. **Not checked: 97915751.** It's now on my fork as the tag `pr155-measured-97915751`: https://git.magicbean.dev/klimovich008/Longshot/src/tag/pr155-measured-97915751 - The carry to this head was checked by `cmp` of `host/hos_zstd.c`, which is byte for byte the same, and a diff of the zstd hook's lines in `nx2/oot_patches.py`: same text, other line numbers. - The test file changed since then; it isn't part of the game build. **Also run:** - `run_all` at `f81d2f93`: 519 passed, 5 failed. All five fail the same way on unmodified main `5e072fc8` on this PC: `test_module_ids`, `test_pad_layout`, `test_framegen_resources`, `test_tested_configurations`, `test_oot_state_tap`. - `host/hos_zstd.c` syntax-compiles whole. - A second review of the new head found no defects.
mooki left a comment

Second look at head f81d2f93. Every point of the first review is settled, and nothing new was found. The first review was a comment, not a change request, so nothing has to be withdrawn.

Read f81d2f93, the reply and the revised series against 12f8cc8e; ran the native zstd test with GCC and Clang, the 25 catalogue checks, and Windows-target syntax checks of the concurrent fixture: all passed.

Previous points

  1. Windows threads: settled. tests/test_native_zstd.py lines 168-190 select the Win32 implementation on Windows and pthreads elsewhere. The callbacks have the right Windows signature, and each joined handle is closed. Line 662 omits -pthread on Windows. The assertions and mutation definitions are unchanged; both Linux compiler runs still reject all five broken variants. The Win32 concurrent fixture also passes Clang's x86_64-w64-mingw32 syntax check here. The reported Microsoft-target Windows execution is separate evidence from the author, not a run I repeated.
  2. Conflict and run files: settled. The head is based on main 5e072fc8 and merges without conflicts. docs/tested-configurations.md is byte-for-byte main's. The six docs/tested/2026-11-13-zstd-frames-{off,on}-klimovich008-pair{1,2,3}.md files each have one result at line 3, with their own context, run identifier and position in the pair. The rejected run is kept and labelled. The older Hookshot results remain historical context in the description, not new rows in the frozen table.
  3. Means, medians and rejected pairs: settled. The Measurements tables now distinguish the two accepted pairs from all three, and label both statistics. I checked the arithmetic against line 3 of each run file and the six columns in the description: the numbers agree, including the rounding of the accepted-pair summaries.
  4. What the control measures: settled. Both the Baseline paragraph and the Measured checklist item now say this is the same PR binary with the switch off, not a build of main. They explicitly leave the default-off cost and the shared first-call change against main unmeasured. “No gain shown” remains the appropriate conclusion, not a guarantee of no slowdown.
  5. Measured source: settled. I fetched pr155-measured-97915751 from the fork. host/hos_zstd.c, including lines 371-404, is identical between that measured commit, the previous reviewed head and this head. The whole-buffer hook text in nx2/oot_patches.py lines 350-366 is identical too.

Rebase and new changes

The series comparison shows the requested thread repair and replacement of the table additions with run files. The existing decoder changes and behavioral assertions were preserved. The unrelated damage and ocarina changes in oot_patches.py come from main, not additional work in this PR. I found no new defect or remaining change to request.

Limits

No game, full build, whole suite or Windows execution here. The Windows check used llvm-mingw headers and was syntax-only; it does not replace the author's Microsoft-runtime run. The raw frame traces and background-load records remain unavailable, so the gameplay measurements remain the author's evidence. Source equality does not turn those older runs into measurements of the newer main.

Second look at head `f81d2f93`. **Every point of the first review is settled, and nothing new was found.** The first review was a comment, not a change request, so nothing has to be withdrawn. Read `f81d2f93`, the reply and the revised series against `12f8cc8e`; ran the native zstd test with GCC and Clang, the 25 catalogue checks, and Windows-target syntax checks of the concurrent fixture: all passed. ### Previous points 1. **Windows threads: settled.** `tests/test_native_zstd.py` lines 168-190 select the Win32 implementation on Windows and pthreads elsewhere. The callbacks have the right Windows signature, and each joined handle is closed. Line 662 omits `-pthread` on Windows. The assertions and mutation definitions are unchanged; both Linux compiler runs still reject all five broken variants. The Win32 concurrent fixture also passes Clang's `x86_64-w64-mingw32` syntax check here. The reported Microsoft-target Windows execution is separate evidence from the author, not a run I repeated. 2. **Conflict and run files: settled.** The head is based on main `5e072fc8` and merges without conflicts. `docs/tested-configurations.md` is byte-for-byte main's. The six `docs/tested/2026-11-13-zstd-frames-{off,on}-klimovich008-pair{1,2,3}.md` files each have one result at line 3, with their own context, run identifier and position in the pair. The rejected run is kept and labelled. The older Hookshot results remain historical context in the description, not new rows in the frozen table. 3. **Means, medians and rejected pairs: settled.** The Measurements tables now distinguish the two accepted pairs from all three, and label both statistics. I checked the arithmetic against line 3 of each run file and the six columns in the description: the numbers agree, including the rounding of the accepted-pair summaries. 4. **What the control measures: settled.** Both the Baseline paragraph and the Measured checklist item now say this is the same PR binary with the switch off, not a build of main. They explicitly leave the default-off cost and the shared first-call change against main unmeasured. “No gain shown” remains the appropriate conclusion, not a guarantee of no slowdown. 5. **Measured source: settled.** I fetched `pr155-measured-97915751` from the fork. `host/hos_zstd.c`, including lines 371-404, is identical between that measured commit, the previous reviewed head and this head. The whole-buffer hook text in `nx2/oot_patches.py` lines 350-366 is identical too. ### Rebase and new changes The series comparison shows the requested thread repair and replacement of the table additions with run files. The existing decoder changes and behavioral assertions were preserved. The unrelated damage and ocarina changes in `oot_patches.py` come from main, not additional work in this PR. I found no new defect or remaining change to request. ### Limits No game, full build, whole suite or Windows execution here. The Windows check used llvm-mingw headers and was syntax-only; it does not replace the author's Microsoft-runtime run. The raw frame traces and background-load records remain unavailable, so the gameplay measurements remain the author's evidence. Source equality does not turn those older runs into measurements of the newer main.
mooki approved these changes 2026-11-14 12:45:19 +00:00
mooki left a comment

Approving at f81d2f93. Every point of the review is answered: the test's thread layer for Windows, the rebase onto current main with the results in per-run files, the labelled statistics, what the control measures, and the measured source being this source. The second look found nothing new: #155 (comment)

As said there, the gameplay numbers are the author's and show no gain yet; the switch stays off by default, which is the right state for it.

Approving at `f81d2f93`. Every point of the review is answered: the test's thread layer for Windows, the rebase onto current main with the results in per-run files, the labelled statistics, what the control measures, and the measured source being this source. The second look found nothing new: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/155#issuecomment-17920 As said there, the gameplay numbers are the author's and show no gain yet; the switch stays off by default, which is the right state for it.
Owner

Should we enable this by default? What do you think?

Should we enable this by default? What do you think?
HumanBean left a comment

Code review only (diff and surrounding code read; nothing built or run). I found no correctness bug in the hook.

What I checked:

  • hos_zstd_frames returns 0 (the lifted code runs) for a raw dictionary, a magicless context, no libzstd context and a DDict libzstd won't take; it is a single early return when the switch is off. The DCtx and DDict reference handling is paired on every path (spare list and zs_dict_put under zs_lock, zs_dict_put(NULL) is safe, ZSTD_freeDCtx(NULL) is a no-op).
  • The first-call race fix (zs_publish compare-exchange so only one caller logs; zs_note now takes zs_lock for its table). Its caller in the stream path is not under zs_lock, so no self-deadlock.
  • HOS_WRITTEN over the whole room is the safe choice, since libzstd can leave literals in the room's far end.
  • patch_zstd_frames is idempotent and only applies with protect_hearts, like the stream hook.

Points for the author and maintainers:

  1. The PR's own numbers show no gain (behind in 3 of 3 pairs, within noise), and the switch stays off. That makes this a dormant code path in the lifted game's loader function, kept only for testing. Please say why it should land before a measurement resolves a gain (for example the three open test requests), or hold it until then. The race fix is separate and worth having; it could land on its own.
  2. The header comment says the three callers "test only that it is one", while the PR text says they map the result through ZSTD_getErrorCode. The two disagree. If a caller maps the code, libzstd 1.5.7's error numbering must match the game's older copy. The local Unicorn comparison (identical codes in 46 cases) covers it, but the comment in host/hos_zstd.c should say which is true.
  3. The log text changed from "a stream decodes lifted" to "a decode runs lifted"; tests pass per the description, nothing else reads it as far as I can see.

Not blocking. Leaving this as a comment, not an approval, because of point 1 and because Linux and non-AMD hardware are unmeasured.

Code review only (diff and surrounding code read; nothing built or run). I found no correctness bug in the hook. What I checked: - `hos_zstd_frames` returns 0 (the lifted code runs) for a raw dictionary, a magicless context, no libzstd context and a DDict libzstd won't take; it is a single early return when the switch is off. The DCtx and DDict reference handling is paired on every path (spare list and `zs_dict_put` under `zs_lock`, `zs_dict_put(NULL)` is safe, `ZSTD_freeDCtx(NULL)` is a no-op). - The first-call race fix (`zs_publish` compare-exchange so only one caller logs; `zs_note` now takes `zs_lock` for its table). Its caller in the stream path is not under `zs_lock`, so no self-deadlock. - `HOS_WRITTEN` over the whole room is the safe choice, since libzstd can leave literals in the room's far end. - `patch_zstd_frames` is idempotent and only applies with `protect_hearts`, like the stream hook. Points for the author and maintainers: 1. The PR's own numbers show no gain (behind in 3 of 3 pairs, within noise), and the switch stays off. That makes this a dormant code path in the lifted game's loader function, kept only for testing. Please say why it should land before a measurement resolves a gain (for example the three open test requests), or hold it until then. The race fix is separate and worth having; it could land on its own. 2. The header comment says the three callers "test only that it is one", while the PR text says they map the result through `ZSTD_getErrorCode`. The two disagree. If a caller maps the code, libzstd 1.5.7's error numbering must match the game's older copy. The local Unicorn comparison (identical codes in 46 cases) covers it, but the comment in `host/hos_zstd.c` should say which is true. 3. The log text changed from "a stream decodes lifted" to "a decode runs lifted"; tests pass per the description, nothing else reads it as far as I can see. Not blocking. Leaving this as a comment, not an approval, because of point 1 and because Linux and non-AMD hardware are unmeasured.
HumanBean deleted branch perf/native-zstd-frames 2026-11-15 19:06:02 +00:00
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 3s
Website / Website on Cloudflare (pull_request_target) Has been skipped
Some workflows are waiting to be reviewed.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!155
No description provided.