MagicBeanDev/Longshot
MagicBeanDev/Longshot
Watch
8
2
Fork
You've already forked Longshot
 19
Code Issues 32 Pull requests 76 Projects 1 Releases 6 Packages Wiki Activity Actions 137

WIP: perf(host): the game's whole-buffer zstd decode on the host's libzstd (HOS_NATIVE_ZSTD_FRAMES, opt-in) #155

Draft
klimovich008 wants to merge 9 commits from klimovich008/Longshot:perf/native-zstd-frames into main
pull from: klimovich008/Longshot:perf/native-zstd-frames
merge into: MagicBeanDev:main
Conversation 0 Commits 9 Files changed 5 +363 -35
klimovich008 commented 2026-11-13 07:38:58 +00:00
First-time contributor

Why

Area loads and streaming spend CPU in the game's own copy of zstd. NotRexed/Hookshot#2738 moved its stream decode (ParallelDecompThread) to the host's libzstd. The game's other zstd path, the resource loaders' whole-buffer decode f_8010b380 on the ARes_Loading threads, is about 22% of a core in Kokiri Forest and 6% in the standard test (CPU report). This PR moves that decode to libzstd too, behind a switch that stays off by default: in game the difference is too small for 5 pairs (Kokiri Forest) or 6 (the standard test) to resolve.

This is a performance change, not a bug fix. It also fixes a first-call data race (C memory model; no wrong output seen) in #2738's HOS_NATIVE_ZSTD switch, found in review.

What changed

  • host: hos_zstd_frames in host/hos_zstd.c, run first in f_8010b380 (zstd's ZSTD_decompressMultiFrame, inlined: x0 the game's context, x1/x2 the output and its room, x3/x4 the frames, x5/x6 a raw dictionary, x7 the game's DDict or 0). It returns what the game's function returns: the decoded size, or a zstd error code the three callers map through ZSTD_getErrorCode, as before.
    • Decodes with ZSTD_decompress_usingDDict (the game's dictionary by reference, through the stream decode's DDict cache, checked against a copy of the game's tables) or ZSTD_decompress_usingDict(NULL, 0); libzstd contexts come from the stream decode's spare list.
    • The whole output room is marked with HOS_WRITTEN after the decode (libzstd may keep literals in the room's far end while it decodes).
    • Runs lifted, as before, for a raw dictionary (none of the callers passes one), a magicless frame format in the game's context, an allocation failure, or a dictionary libzstd does not take as the game's. Each kind is logged once.
  • host, the race fix (also in #2738's code): zs_mode (HOS_NATIVE_ZSTD) and the new zf_mode cached their switch in a plain static that several decode threads read and wrote, and the once-logged fallback table was unsynchronized. zs_publish now publishes each switch once with a compare-and-swap, and only the thread that published it logs it. zs_note claims its table entry under the existing zs_lock and logs after releasing it. Only first calls and logging change; the decode path does not.
  • nx2: patch_zstd_frames in nx2/oot_patches.py puts if (hos_zstd_frames(c)) return; first in f_8010b380, counted by report() like the stream hook (it warns unless found exactly once).
  • tests: tests/test_native_zstd.py, game-free:
    • whole-buffer cases (with and without dictionary and checksum, raw and RLE blocks, empty, one byte, unknown content size, two frames with a skippable one between, more room than needed), each compared with a switch-off run outside the output room, with guard bytes around it;
    • the failure codes (checksum, dictionary, bad magic, too little room, truncated input), a DDict made again in place, and the fallbacks;
    • first calls from two threads at once, two ways: a stand-in env_on holds each caller that reads a switch unset until two have, so both go on to publish it; and a stand-in log holds the first thread in its first log line while the second runs start to end;
    • broken copies that must fail: the DDict not passed, a switch published after its log line, a publication that always stores and reports success, a fallback logged under the lock. The code before the race fix fails the concurrent checks (each switch logged twice). The fallback table's lock itself is not shown by a test: nothing holds a thread between its read and its write, and there was no thread-sanitizer run.
  • docs: both zstd switches in docs/development.md's switch reference (#2738 had not added HOS_NATIVE_ZSTD); five rows in docs/tested-configurations.md.

Flags and settings

Flag What it does Default now Recommendation Helps on Doesn't help on Untested on
HOS_NATIVE_ZSTD_FRAMES (set) The resource loaders' whole-buffer zstd decode on the host's libzstd off off until a measurement resolves a gain not shown yet (no resolved difference) n/a Linux, NVIDIA, Intel, the Steam Deck, hybrid (P/E-core) CPUs

HOS_NATIVE_ZSTD (from #2738) is unchanged apart from the race fix.

Measurements

main: not measured on Longshot yet. This PR stays WIP until it is, on a Longshot base.
this PR: perf/native-zstd-frames @ a0ff2d94, not yet measured itself; the numbers below are its Hookshot form d0b8141f, before the rename and before the race fix (which changes only first-call setup and logging; the decode path is the same)
Hardware: RX 9070 XT, AMD 26.10.2 (Vulkan driver 0x8001a0), Ryzen 7 5700X3D (no efficiency cores), 64 GB, Windows 11
Scene: Kokiri Forest (the default save, standing) and the standard test (Hyrule Field, the camera turning at full speed); 5120x1440 (32:9, the standard test's aspect until 2026-11-13), no upscaler, no frame generation, frame rate unlocked (HOS_FPS_MAX=240)
Method: same binary, the switch off and on, alternating runs, 20 s windows after the save loads; fps and frame times from the present trace, CPU from the process. A run counts when every game-clock reading whose interval lies wholly inside the window is within 1.000 ± 0.01 and the window has no frame over 250 ms; a pair with a run that fails is left out whole. Medians, with the mean of paired CPU differences.
How to reproduce: build this branch, then for each run set HOS_FPS_MAX=240 and HOS_FPS_LOG=1, load the save and stand (Kokiri) or hold the right stick fully right (Hyrule); alternate HOS_NATIVE_ZSTD_FRAMES unset and =1.

Baseline: this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES unset: the build's own switch-off runs, which run main's code path. Main was not the comparison because the switch is off by default, so off and on one binary isolates the change; a Longshot main @ / this PR @ comparison comes with the measurement on a Longshot base. The A/B is by switch on one binary, so the two sides differ only in this decode.

Build: d0b8141f, Hookshot v2.1.0-rc.3-250-g46780503 with #2738 and this change before the race fix (exe a46bf3a229310166, hookshot_vk.dll 418aa37c49bc96ce).

Kokiri Forest, 3 warm-ups, then 6 runs an arm: 5 of 6 pairs count.

this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES unset this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES=1 this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES=1, HOS_NATIVE_ZSTD=2
fps 82.2 (78.7-82.8) 81.7 (80.0-83.5) 82.0 (80.2-84.3)
CPU ms a frame 59.16 (58.0-62.6) 58.13 (57.3-61.7), paired -0.80 58.11 (57.1-59.9), paired -1.40
p99 ms 18.5 (17.5-20.1) 18.7 (16.2-19.9) 19.2 (15.5-19.8)
worst frame ms 40.6 26.2 94.1
frames over 100 ms 0 0 0
game clock (window) 0.999-1.004 0.998-1.002 0.998-1.002
save load s 29.28 (28.7-29.6) 28.95 (28.5-29.5) 29.06 (27.9-29.4)
core escapes (with ParallelDecompThread holding) 0 (0) 3 (0) 0 (0)
  • Pair 1 is left out: its =1 run had frames of 276 and 253 ms in the window and a clock reading of 0.933 over that stretch. The same pair's control run had a 131.6 ms frame, and the warm-ups, which run with the switch off, had frames of 295 and 345 ms. With pair 1 included, the paired CPU differences are -1.66 and -2.09 ms.

The standard test, 2 warm-ups, then 6 pairs: all 6 count.

this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES unset this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD_FRAMES=1
fps 53.25 (52.5-53.5) 52.6 (51.6-54.0)
CPU ms a frame 76.69 (75.9-77.3) 75.86 (74.0-78.8), paired -0.47
p99 ms 25.7 (24.7-29.7) 25.4 (23.5-37.8)
worst frame ms 60.9 96.0
frames over 100 ms 0 0
game clock (window) 0.998-1.004 0.997-1.003
save load s 29.20 (28.3-29.7) 28.74 (28.4-29.8)
core escapes (with ParallelDecompThread holding) 5 (0) 4 (0)
  • A stricter clock view, the median of each run's last four clock lines, voids two runs with the switch (pairs 1 and 4: 0.948 and 0.955). Those four lines include the clock's dip at the end of the save load, which in these two runs fell across two lines (0.661 then 0.895; 0.695 then 0.912) instead of one, as in the other ten (0.645-0.805). Inside the window both read 1.000 and 1.000, and 1.000 and 0.998, with worst frames of 62.4 and 50.5 ms and save loads of 29.7 and 28.9 s. Two of six runs with the switch against none of six without it is not enough to say whether that split follows the switch. On the 4 pairs that view keeps: fps 53.25 / 52.95, CPU 76.28 / 75.64 ms (paired -0.72), p99 25.7 / 25.4, save load 29.20 / 28.52 s.
  • The CPU differences are inside the control's run-to-run range (58.0-62.6 ms in Kokiri, 75.9-77.3 ms in Hyrule). Save load is 0.2-0.5 s shorter in every comparison, inside a range of about 1.4 s.
  • No run logged a fallback: every whole-buffer decode ran on the host.
  • Another agent held the GPU lock for about 5 minutes between two Hyrule runs (not during one); the run after it is not an outlier.

HOS_NATIVE_ZSTD=1 while moving (context for #2738)

A tester saw HOS_NATIVE_ZSTD=1 slower than unset in Kokiri Forest while moving (RTX 5090, i7-13700K: 72.2 against 66.7 fps). I checked it on this PC with #2738 as merged (before the race fix) and this change switched off (exe e9be69d36122b25e): Kokiri Forest from the default save, running with the camera turning, 2 warm-ups, then 6 triples. Two triples are left out by the rules above: an unset run with frames of 576, 284 and 135 ms, and an =2 run with a window clock reading of 0.990. On the other 4:

this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD unset this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD=1 this PR @ d0b8141f (Hookshot), HOS_NATIVE_ZSTD=2
fps 51.4 53.55 (paired +2.0) 52.2 (paired +0.75)
CPU ms a frame 85.71 83.60 (paired -2.46) 85.16 (paired -0.89)
p99 ms 30.3 30.4 31.0
worst frame ms 39.7 49.9 47.7
save load s 29.57 28.93 28.87
  • =1 was faster in all 4 of those pairs; in one more pair (valid for unset against =1 only) it was slower, 51.0 against 53.2 fps.
  • The main thread (35% of a core), the Presentation Thread (73-74%) and the resource loaders cost the same in every arm, and =1 makes no more virtual-core hand-offs than unset. ParallelDecompThread went from about 2% of a core to about 0.
  • This does not rule the report out: this PC's CPU has no efficiency cores, and the stream decode was only 0-5% of a core in this window against 30% in the report's.
  • This PR's whole-buffer decode never gives its core up, so it does not share what =1 changes.

Testing

Tested:

  • The game's own function, locally (no game data here or in the test): f_8010b380 called through one of its callers in Unicorn from a local image, against hos_zstd_frames on the same inputs: 46 cases, 49 calls (the test's kinds plus larger sets up to 2 MB, errors, a DDict made again). Results, including error codes, and the decoded output were identical. In 2 of the cases with 1000 bytes more room than the content, the bytes in the room past the content differ (libzstd keeps literals there while it decodes). A broken copy fails 8 cases.
  • python -I -B tests/run_all.py on this branch at ca40bb33, on main 39647e1a (Windows, llvm-mingw, LONGSHOT_MINGW set; since then rebased onto 6b7fa356, whose #148 touches only the README, the release script and its test): 496 passed, 38 skipped, 4 failed. The failures are nx2/test_nup_paths.py, tests/test_check_third_party.py, tests/test_setup_runtime.py and tests/test_release.py, which fail on main on this PC too; this PR does not touch what they cover. tests/test_native_zstd.py passes with llvm-mingw and with MinGW gcc 16.2, and every broken copy in it fails. host/hos_zstd.c compiles as a whole (clang -fsyntax-only against a build's generated headers).
  • The build log of the measured build shows both zstd hooks applied once, with no warnings.
  • In game: the runs above, on Windows 11 with an RX 9070 XT.

Not tested:

  • This branch's own build (with the race fix and the rename) in game: the measured builds were Hookshot 46780503 builds. To be measured on a Longshot base before this leaves WIP.
  • The race fix in game: it changes only how the switches are read once and logged, and was tested in the fixture only.
  • Linux (no OS branch in the code, but not built or run there).
  • First visits with cold caches, other scenes, sessions longer than a minute, more than 6 pairs.
  • Raw dictionaries and the magicless format: no caller passes them, so they stay lifted.

Needs other testers:

  • NVIDIA, Intel: Kokiri Forest while moving, HOS_NATIVE_ZSTD_FRAMES unset against =1, expecting fps and frame times unchanged or better (request to open)
  • Hybrid (P/E-core) CPU: the same, plus HOS_NATIVE_ZSTD=2 against unset, expecting no slowdown (request to open)
  • Linux: the same on any GPU, expecting no slowdown (request to open)

Screenshots

Nothing on screen.

Risk and rollback

A wrong decode would show as missing or broken models, textures or effects after an area load, or a load that fails. It is off unless HOS_NATIVE_ZSTD_FRAMES is set; unset, the hook returns at its first check and the game's own decode runs. The race fix changes only first-call setup and logging; reverting its commit restores the old first-call code.

Reviewer notes

Start with hos_zstd_frames and the race fix at the top of host/hos_zstd.c, then the whole-buffer and concurrent cases in tests/test_native_zstd.py. Prior art: zortos293/Lichii#869 puts this game's zstd decode on the host's libzstd in an emulator fork. I did not read its code; it's credited as related work. Also related: NotRexed/Hookshot#909 (the lifted decoder at -O2).

Issues

See NotRexed/Hookshot#2738 (its first-call race is fixed here).

Checklist

For everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description).
  • Rebased on current main.
  • python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
  • Tested in the game, or what wasn't is named under Testing.
  • Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
  • Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (By switch on one Hookshot build; to be measured on a Longshot base.)
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Flags noted, and the switch reference updated (docs/development.md, the README).
  • A row in docs/tested-configurations.md for each measured run.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.

🤖 Generated with Claude Code

## Why Area loads and streaming spend CPU in the game's own copy of zstd. [NotRexed/Hookshot#2738](https://github.com/NotRexed/Hookshot/pull/2738) moved its stream decode (ParallelDecompThread) to the host's libzstd. The game's other zstd path, the resource loaders' whole-buffer decode `f_8010b380` on the ARes_Loading threads, is about 22% of a core in Kokiri Forest and 6% in the standard test (CPU report). This PR moves that decode to libzstd too, behind a switch that stays off by default: in game the difference is too small for 5 pairs (Kokiri Forest) or 6 (the standard test) to resolve. This is a performance change, not a bug fix. It also fixes a first-call data race (C memory model; no wrong output seen) in #2738's `HOS_NATIVE_ZSTD` switch, found in review. ## What changed - host: `hos_zstd_frames` in `host/hos_zstd.c`, run first in `f_8010b380` (zstd's `ZSTD_decompressMultiFrame`, inlined: x0 the game's context, x1/x2 the output and its room, x3/x4 the frames, x5/x6 a raw dictionary, x7 the game's DDict or 0). It returns what the game's function returns: the decoded size, or a zstd error code the three callers map through `ZSTD_getErrorCode`, as before. - Decodes with `ZSTD_decompress_usingDDict` (the game's dictionary by reference, through the stream decode's DDict cache, checked against a copy of the game's tables) or `ZSTD_decompress_usingDict(NULL, 0)`; libzstd contexts come from the stream decode's spare list. - The whole output room is marked with `HOS_WRITTEN` after the decode (libzstd may keep literals in the room's far end while it decodes). - Runs lifted, as before, for a raw dictionary (none of the callers passes one), a magicless frame format in the game's context, an allocation failure, or a dictionary libzstd does not take as the game's. Each kind is logged once. - host, the race fix (also in #2738's code): `zs_mode` (`HOS_NATIVE_ZSTD`) and the new `zf_mode` cached their switch in a plain static that several decode threads read and wrote, and the once-logged fallback table was unsynchronized. `zs_publish` now publishes each switch once with a compare-and-swap, and only the thread that published it logs it. `zs_note` claims its table entry under the existing `zs_lock` and logs after releasing it. Only first calls and logging change; the decode path does not. - nx2: `patch_zstd_frames` in `nx2/oot_patches.py` puts `if (hos_zstd_frames(c)) return;` first in `f_8010b380`, counted by `report()` like the stream hook (it warns unless found exactly once). - tests: `tests/test_native_zstd.py`, game-free: - whole-buffer cases (with and without dictionary and checksum, raw and RLE blocks, empty, one byte, unknown content size, two frames with a skippable one between, more room than needed), each compared with a switch-off run outside the output room, with guard bytes around it; - the failure codes (checksum, dictionary, bad magic, too little room, truncated input), a DDict made again in place, and the fallbacks; - first calls from two threads at once, two ways: a stand-in `env_on` holds each caller that reads a switch unset until two have, so both go on to publish it; and a stand-in log holds the first thread in its first log line while the second runs start to end; - broken copies that must fail: the DDict not passed, a switch published after its log line, a publication that always stores and reports success, a fallback logged under the lock. The code before the race fix fails the concurrent checks (each switch logged twice). The fallback table's lock itself is not shown by a test: nothing holds a thread between its read and its write, and there was no thread-sanitizer run. - docs: both zstd switches in `docs/development.md`'s switch reference (#2738 had not added `HOS_NATIVE_ZSTD`); five rows in `docs/tested-configurations.md`. ## Flags and settings | Flag | What it does | Default now | Recommendation | Helps on | Doesn't help on | Untested on | |---|---|---|---|---|---|---| | `HOS_NATIVE_ZSTD_FRAMES` (set) | The resource loaders' whole-buffer zstd decode on the host's libzstd | off | off until a measurement resolves a gain | not shown yet (no resolved difference) | n/a | Linux, NVIDIA, Intel, the Steam Deck, hybrid (P/E-core) CPUs | `HOS_NATIVE_ZSTD` (from #2738) is unchanged apart from the race fix. ## Measurements main: not measured on Longshot yet. This PR stays WIP until it is, on a Longshot base. this PR: perf/native-zstd-frames @ a0ff2d94, not yet measured itself; the numbers below are its Hookshot form d0b8141f, before the rename and before the race fix (which changes only first-call setup and logging; the decode path is the same) Hardware: RX 9070 XT, AMD 26.10.2 (Vulkan driver 0x8001a0), Ryzen 7 5700X3D (no efficiency cores), 64 GB, Windows 11 Scene: Kokiri Forest (the default save, standing) and the standard test (Hyrule Field, the camera turning at full speed); 5120x1440 (32:9, the standard test's aspect until 2026-11-13), no upscaler, no frame generation, frame rate unlocked (`HOS_FPS_MAX=240`) Method: same binary, the switch off and on, alternating runs, 20 s windows after the save loads; fps and frame times from the present trace, CPU from the process. A run counts when every game-clock reading whose interval lies wholly inside the window is within 1.000 ± 0.01 and the window has no frame over 250 ms; a pair with a run that fails is left out whole. Medians, with the mean of paired CPU differences. How to reproduce: build this branch, then for each run set `HOS_FPS_MAX=240` and `HOS_FPS_LOG=1`, load the save and stand (Kokiri) or hold the right stick fully right (Hyrule); alternate `HOS_NATIVE_ZSTD_FRAMES` unset and `=1`. Baseline: this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES` unset: the build's own switch-off runs, which run main's code path. Main was not the comparison because the switch is off by default, so off and on one binary isolates the change; a Longshot main @ / this PR @ comparison comes with the measurement on a Longshot base. The A/B is by switch on one binary, so the two sides differ only in this decode. Build: d0b8141f, Hookshot v2.1.0-rc.3-250-g46780503 with #2738 and this change before the race fix (exe `a46bf3a229310166`, `hookshot_vk.dll` `418aa37c49bc96ce`). Kokiri Forest, 3 warm-ups, then 6 runs an arm: 5 of 6 pairs count. | | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES` unset | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES=1` | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES=1`, `HOS_NATIVE_ZSTD=2` | |---|---|---|---| | fps | 82.2 (78.7-82.8) | 81.7 (80.0-83.5) | 82.0 (80.2-84.3) | | CPU ms a frame | 59.16 (58.0-62.6) | 58.13 (57.3-61.7), paired -0.80 | 58.11 (57.1-59.9), paired -1.40 | | p99 ms | 18.5 (17.5-20.1) | 18.7 (16.2-19.9) | 19.2 (15.5-19.8) | | worst frame ms | 40.6 | 26.2 | 94.1 | | frames over 100 ms | 0 | 0 | 0 | | game clock (window) | 0.999-1.004 | 0.998-1.002 | 0.998-1.002 | | save load s | 29.28 (28.7-29.6) | 28.95 (28.5-29.5) | 29.06 (27.9-29.4) | | core escapes (with ParallelDecompThread holding) | 0 (0) | 3 (0) | 0 (0) | - Pair 1 is left out: its `=1` run had frames of 276 and 253 ms in the window and a clock reading of 0.933 over that stretch. The same pair's control run had a 131.6 ms frame, and the warm-ups, which run with the switch off, had frames of 295 and 345 ms. With pair 1 included, the paired CPU differences are -1.66 and -2.09 ms. The standard test, 2 warm-ups, then 6 pairs: all 6 count. | | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES` unset | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD_FRAMES=1` | |---|---|---| | fps | 53.25 (52.5-53.5) | 52.6 (51.6-54.0) | | CPU ms a frame | 76.69 (75.9-77.3) | 75.86 (74.0-78.8), paired -0.47 | | p99 ms | 25.7 (24.7-29.7) | 25.4 (23.5-37.8) | | worst frame ms | 60.9 | 96.0 | | frames over 100 ms | 0 | 0 | | game clock (window) | 0.998-1.004 | 0.997-1.003 | | save load s | 29.20 (28.3-29.7) | 28.74 (28.4-29.8) | | core escapes (with ParallelDecompThread holding) | 5 (0) | 4 (0) | - A stricter clock view, the median of each run's last four clock lines, voids two runs with the switch (pairs 1 and 4: 0.948 and 0.955). Those four lines include the clock's dip at the end of the save load, which in these two runs fell across two lines (0.661 then 0.895; 0.695 then 0.912) instead of one, as in the other ten (0.645-0.805). Inside the window both read 1.000 and 1.000, and 1.000 and 0.998, with worst frames of 62.4 and 50.5 ms and save loads of 29.7 and 28.9 s. Two of six runs with the switch against none of six without it is not enough to say whether that split follows the switch. On the 4 pairs that view keeps: fps 53.25 / 52.95, CPU 76.28 / 75.64 ms (paired -0.72), p99 25.7 / 25.4, save load 29.20 / 28.52 s. - The CPU differences are inside the control's run-to-run range (58.0-62.6 ms in Kokiri, 75.9-77.3 ms in Hyrule). Save load is 0.2-0.5 s shorter in every comparison, inside a range of about 1.4 s. - No run logged a fallback: every whole-buffer decode ran on the host. - Another agent held the GPU lock for about 5 minutes between two Hyrule runs (not during one); the run after it is not an outlier. ### `HOS_NATIVE_ZSTD=1` while moving (context for #2738) A tester saw `HOS_NATIVE_ZSTD=1` slower than unset in Kokiri Forest while moving (RTX 5090, i7-13700K: 72.2 against 66.7 fps). I checked it on this PC with #2738 as merged (before the race fix) and this change switched off (exe `e9be69d36122b25e`): Kokiri Forest from the default save, running with the camera turning, 2 warm-ups, then 6 triples. Two triples are left out by the rules above: an unset run with frames of 576, 284 and 135 ms, and an `=2` run with a window clock reading of 0.990. On the other 4: | | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD` unset | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD=1` | this PR @ d0b8141f (Hookshot), `HOS_NATIVE_ZSTD=2` | |---|---|---|---| | fps | 51.4 | 53.55 (paired +2.0) | 52.2 (paired +0.75) | | CPU ms a frame | 85.71 | 83.60 (paired -2.46) | 85.16 (paired -0.89) | | p99 ms | 30.3 | 30.4 | 31.0 | | worst frame ms | 39.7 | 49.9 | 47.7 | | save load s | 29.57 | 28.93 | 28.87 | - `=1` was faster in all 4 of those pairs; in one more pair (valid for unset against `=1` only) it was slower, 51.0 against 53.2 fps. - The main thread (35% of a core), the Presentation Thread (73-74%) and the resource loaders cost the same in every arm, and `=1` makes no more virtual-core hand-offs than unset. ParallelDecompThread went from about 2% of a core to about 0. - This does not rule the report out: this PC's CPU has no efficiency cores, and the stream decode was only 0-5% of a core in this window against 30% in the report's. - This PR's whole-buffer decode never gives its core up, so it does not share what `=1` changes. ## Testing **Tested:** - The game's own function, locally (no game data here or in the test): `f_8010b380` called through one of its callers in Unicorn from a local image, against `hos_zstd_frames` on the same inputs: 46 cases, 49 calls (the test's kinds plus larger sets up to 2 MB, errors, a DDict made again). Results, including error codes, and the decoded output were identical. In 2 of the cases with 1000 bytes more room than the content, the bytes in the room past the content differ (libzstd keeps literals there while it decodes). A broken copy fails 8 cases. - `python -I -B tests/run_all.py` on this branch at ca40bb33, on main 39647e1a (Windows, llvm-mingw, `LONGSHOT_MINGW` set; since then rebased onto 6b7fa356, whose #148 touches only the README, the release script and its test): 496 passed, 38 skipped, 4 failed. The failures are `nx2/test_nup_paths.py`, `tests/test_check_third_party.py`, `tests/test_setup_runtime.py` and `tests/test_release.py`, which fail on `main` on this PC too; this PR does not touch what they cover. `tests/test_native_zstd.py` passes with llvm-mingw and with MinGW gcc 16.2, and every broken copy in it fails. `host/hos_zstd.c` compiles as a whole (`clang -fsyntax-only` against a build's generated headers). - The build log of the measured build shows both zstd hooks applied once, with no warnings. - In game: the runs above, on Windows 11 with an RX 9070 XT. **Not tested:** - This branch's own build (with the race fix and the rename) in game: the measured builds were Hookshot 46780503 builds. To be measured on a Longshot base before this leaves WIP. - The race fix in game: it changes only how the switches are read once and logged, and was tested in the fixture only. - Linux (no OS branch in the code, but not built or run there). - First visits with cold caches, other scenes, sessions longer than a minute, more than 6 pairs. - Raw dictionaries and the magicless format: no caller passes them, so they stay lifted. **Needs other testers:** - NVIDIA, Intel: Kokiri Forest while moving, `HOS_NATIVE_ZSTD_FRAMES` unset against `=1`, expecting fps and frame times unchanged or better (request to open) - Hybrid (P/E-core) CPU: the same, plus `HOS_NATIVE_ZSTD=2` against unset, expecting no slowdown (request to open) - Linux: the same on any GPU, expecting no slowdown (request to open) ## Screenshots Nothing on screen. ## Risk and rollback A wrong decode would show as missing or broken models, textures or effects after an area load, or a load that fails. It is off unless `HOS_NATIVE_ZSTD_FRAMES` is set; unset, the hook returns at its first check and the game's own decode runs. The race fix changes only first-call setup and logging; reverting its commit restores the old first-call code. ## Reviewer notes Start with `hos_zstd_frames` and the race fix at the top of `host/hos_zstd.c`, then the whole-buffer and concurrent cases in `tests/test_native_zstd.py`. Prior art: [zortos293/Lichii#869](https://github.com/zortos293/Lichii/pull/869) puts this game's zstd decode on the host's libzstd in an emulator fork. I did not read its code; it's credited as related work. Also related: [NotRexed/Hookshot#909](https://github.com/NotRexed/Hookshot/pull/909) (the lifted decoder at -O2). ## Issues See [NotRexed/Hookshot#2738](https://github.com/NotRexed/Hookshot/pull/2738) (its first-call race is fixed here). ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. (By switch on one Hookshot build; to be measured on a Longshot base.) - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Flags noted, and the switch reference updated (docs/development.md, the README). - [x] A row in `docs/tested-configurations.md` for each measured run. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) 
The resource loaders (the ARes_Loading threads) decode whole files with zstd's decompressMultiFrame, inlined in f_8010b380:
about a fifth of a core while an area loads in the CPU report's Kokiri and Temple runs, more than the stream decode #2738
moved. With HOS_NATIVE_ZSTD_FRAMES=1 a hook first in f_8010b380 does the call with libzstd's ZSTD_decompress_usingDDict
(the game's DDict by reference, as the stream decode takes it) and returns its result; off, the function runs as lifted.
A raw dictionary or another frame format stays lifted. tests/test_native_zstd.py covers the new path, with its own broken
copy.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
sol review r1: a frames call that failed to take its dictionary put a context back with no bound check, so 64 streams
ending meanwhile could overflow zs_spare. The test now compares all memory outside the output room with the switch-off run,
and adds cases with 64 KB more room than the content.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
sol review r2: memory around the room started as zero, so a write of a zero past it went unseen.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
sol review r3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Codex's review: zs_mode, zf_mode and zs_note read and wrote plain statics from several decode threads (a C data race;
no output error was seen). Each switch is now published once with a compare-and-swap, and only the thread that
publishes it logs it. zs_note claims its table entry under zs_lock and logs after releasing it. This covers the merged
stream switch (#2738) as well as the whole-buffer one.

tests/test_native_zstd.py runs first calls from two threads at once: a stand-in hos_log holds the first thread in its
first log line while the second runs start to end. The code before this fix logs each switch twice there; two broken
copies (a switch published after its log line, a fallback logged under zs_lock) must fail.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Codex's review of the concurrent check: it started the second thread from the first one's log line, after the switch
was published, so a publication that always stores and reports success passed it. A stand-in env_on now holds each
caller that reads a switch unset until two have, for the stream and the whole-buffer switch; only one may log. The
broken copy that always stores fails it, as does the code before the race fix. The fallback table's lock is still not
shown by a test (no probe holds a thread between its read and its write).

Also the tested-configurations rows: the game clock read inside the measured window, and the Kokiri pair with
in-window frames over 250 ms named.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs: the HOS_NATIVE_ZSTD_FRAMES rows name their builds as Hookshot ones
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
Details
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
Details
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Details
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Details
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Details
PR description / Description follows the template (pull_request_target) Successful in 3s
Details
a0ff2d9491 
Assisted-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
Details
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
Details
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
Details
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
Details
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Details
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Details
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Details
PR description / Description follows the template (pull_request_target) Successful in 3s
Details
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/native-zstd-frames:klimovich008-perf/native-zstd-frames'
git switch 'klimovich008-perf/native-zstd-frames'
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area/accessibility
Barriers for people with disabilities

  
area/audio
Sound and music

  
area/input
Controllers, keyboard, mouse

  
area/install
Installing, updating, building from your game

  
area/menu
In-game menu and settings

  
area/renderer
Vulkan renderer, shaders, effects

  
area/saves
Save files

  
area/upscaling
DLSS, FSR, frame generation

  
good first issue
A good start for a new contributor

  
gpu/amd
AMD GPUs

  
gpu/intel
Intel GPUs

  
gpu/nvidia
NVIDIA GPUs

  
help wanted
Extra hands or knowledge wanted

  
kind
bug
 Something doesn't work as it should

  
kind
chore
 Build, CI, dependencies, tidying

  
kind
docs
 Documentation

  
kind
feature
 A new feature or an improvement

  
kind
performance
 Frame rate, frame time, loading, memory

  
kind
question
 A question rather than a change

  
needs measurements
Waiting for before/after measurements

  
platform/linux
Linux only

  
platform/steam-deck
Steam Deck

  
platform/windows
Windows only

  
preview
Maintainers: deploy a preview of this PR's website (read its changes first). Removed after the run.

  
priority
critical
 Crashes, data loss, or blocks most players

  
priority
high
 Important: next in line

  
priority
low
 Nice to have

  
status
blocked
 Waits on something else first

  
status
confirmed
 Reproduced or agreed on

  
status
duplicate
 Already reported or proposed

  
status
invalid
 Not a problem in this project

  
status
needs-info
 Waiting for more information from the reporter

  
status
wontfix
 Will not be worked on

  
testing wanted
Wanted: a test on other hardware (posts a test request in #test-requests)

No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!155
No description provided.