WIP: perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_SEQ_NATIVE, off by default) #1670

Draft
Molier wants to merge 4 commits from Molier/Longshot:perf/zstd-native-seq into main
Contributor

Type: perf · Risk: low · Tested on: Linux (Ryzen 7 5800X, RX 9070 XT), game-free differential tests against libzstd 1.5.7, direct disassembly comparison against this build's own lifted code · Needs: a heavier streaming/loading scene than this PR's own route to show a measured gain; Windows; AMD/Intel/NVIDIA cross-check

Why

Area loads and streaming spend CPU in the game's own copy of zstd. f_801102e0 is the main module's zstd 1.4
sequence loop (decoding and executing a block's sequences): its loading threads spend most of their time in it
arriving in Hyrule Castle (89% of ParallelDecompThread's samples there, per the source this port is based on).
This is the port's third zstd path on this project: the stream decode (f_80115c80, HOS_NATIVE_ZSTD) and the
resource loaders' whole-buffer decode (f_8010b380, HOS_NATIVE_ZSTD_FRAMES) already run on the host's libzstd
upstream (#155); this one is a different function (the sequence loop inside a block's decode), not covered by
either.

Ported from Project ReLink commit bec02c50 (pixl27, GPL-3.0 per ReLink's own LICENSE file at bec02c50),
onto this project's own lift. ReLink's author has not opened a Longshot PR for this (checked the open/closed PR
list and PR authors for "pixl27"/"ReLink"/"relink": no match); this port is not a duplicate of anything already
proposed here.

What changed

  • host/zstd_native.c (new): hos_zstd_seq, run first in the lifted f_801102e0. It is a transcription of the
    real function's 352 instructions on locals instead of the lifted context (load for load, store for store, in
    the game's order), marking the pages it wrote once at the end instead of per instruction. HOS_ZSTD_SEQ_NATIVE
    is off by default (unset, or off/false/no/0): the lifted body runs exactly as before. =1 runs the
    native loop; =2 runs both and compares every call (zstd_compare), keeping the lifted result and logging a
    difference. Off by default until a Bosie A/B against main shows a gain (see Measurements: this PR's own route
    does not show one). Keeps ReLink's own transcription comments and measurement notes; the credit line and ReLink's
    commit are at the top of the file.
  • nx2/oot_patches.py: patch_zstd_seq puts the call at the head of the lifted f_801102e0, wired into
    patch_file and report's done counts like the project's other zstd hooks (patch_zstd_stream,
    patch_zstd_frames).
  • docs/development.md and gpu/vk/switches.def: HOS_ZSTD_SEQ_NATIVE and HOS_ZSTD_TIME in the switch
    reference (gpu/vk/switches.def had no row for either; tests/test_switch_table.py failed on both before this).
    Renamed from the first push's HOS_ZSTD_NATIVE, which sat one word-order away from the existing
    HOS_NATIVE_ZSTD/HOS_NATIVE_ZSTD_FRAMES switches (a different zstd version and a different function) and
    invited exactly the kind of typo this review caught.
  • tests/test_zstd_native_seq.py: the loop against libzstd 1.5.7's block decoder on 11 frames (text, mixed,
    random, zeros, levels 1/3/19, checksum, empty, 1000 bytes; 40 compressed blocks, 323,399 sequences, 14 runs
    through the end-of-output path), a canary past the output's end, a mutant of the loop that must fail, the kill
    switch's values, and the patch's placement.
  • tests/test_zstd_native_compare.py: HOS_ZSTD_SEQ_NATIVE=2 really runs both and compares (not a stub): a matching
    self-comparison logs 0 differences, a corrupted one logs 1 and keeps the lifted (correct) output; HOS_ZSTD_TIME
    runs.

Changes against ReLink's own hunk, for this lift: the hook is named ZSTD_SEQ_* (the stream hook already has
ZSTD_HEAD/ZSTD_CALL) and wired into patch_file/report's done counts like the other hooks; switches read
through gpu/vk/compat/longshot_env.h's one rule (env_value_off, env_int, env_on); _Thread_local for
zstd_bypass, as the rest of host/.

Measurements

Verdict: correctness is thoroughly verified (byte-for-byte against this build's own disassembly, 60,000 of
60,000 real in-game compare-mode calls agree, every game-free differential test passes); no fps or frame-time gain
is demonstrated on the route tested, which turns out not to be the scene the port targets. Still WIP: off by
default until a heavier scene shows a real gain against main.

Baseline: our nightly-338-based build game-p12a (o1-frame-lag, lifted with A64_LOCAL_REGS=6 A64_LOCAL_FLAGS=1 A64_LOCAL_FUNCS=<list>), HOS_ZSTD_SEQ_NATIVE=0 against =1 on the same build; not main against this PR, which
is still to run.
Hardware: Linux, Ryzen 7 5800X, RX 9070 XT
Scene: hf48run (a moving Hyrule Field route plus a Kakariko<->Hyrule Field teleport load), 2560x1440, render
resolution fixed 100%, no frame generation, uncapped
Method: (a) HOS_ZSTD_SEQ_NATIVE=2 (compare mode) for one full hf48run run, counting the log's own mismatch tally;
(b) HOS_ZSTD_SEQ_NATIVE=0 (lifted) against =1 (native), 3 interleaved runs each, median/mean/p99/lows on
the route's load segment, and ParallelDecompThread's CPU share from the same runs' thread-time log lines; (c)
screenshots at each segment's end, pixel-diffed against a matching run

compare mode (a), one run off (lifted), 3 runs on (native, default), 3 runs
sequence loops compared / mismatches 60,000 / 0 - -
load segment median frame time - 17.59 ms 17.25 ms (-1.9%)
load segment mean - 15.93 ms 15.19 ms
load segment p99 - 38.09 ms 40.15 ms
load segment 1% low / 0.1% low fps - 8.5 / 2.5 9.7 / 3.0
load segment frames >20/>33 ms - 972 / 55 1,079 / 67
ParallelDecompThread, mean % of a core - 6.97% (38 samples) 6.88% (26 samples)
zstd: the sequence loop (f_801102e0) native and lifted, compared on every call (HOS_ZSTD_SEQ_NATIVE=2)
zstd: 60000 sequence loops compared, 0 different

Reading, honestly: the load segment's median gain (17.59 -> 17.25 ms, 1.9%) is smaller than the off arm's
own run-to-run spread (17.66/17.00/18.11 ms, 6.3%) - not distinguishable from 3 runs' noise. ParallelDecompThread's
CPU share is essentially identical on and off (6.97% vs 6.88% of a core) and low in absolute terms either way; the
>33ms count moved the wrong way for "on" (55->67), with the two arms' own ranges overlapping, so this is not a
confident regression claim either. This route's decompression load (6-7% of a core) is much lighter than the
Hyrule-Castle-loading-screen scene the port's own source attributes 89% of ParallelDecompThread's samples to:
hf48run's teleport load is not that scene, so this A/B does not show the gain the port targets, for a route
reason, not a correctness reason. A heavier streaming/loading scene is needed to show (or rule out) a real fps
gain; not run this session.

Screenshots at each hf48run segment's end (game-p12a, native mode, against game-o2, a different build run in
lifted mode), pixel-diffed: mean absolute per-channel diff 8.9-21.9, max 240-254, none byte-identical but every
pair visually the same scene, same HUD state, same object placement on inspection - consistent with the sub-degree
camera-angle drift two independent 45 s scripted camera sweeps always have, not a correctness issue, and consistent
with the 0-mismatch compare-mode result above. Comparing two different builds (not on/off on the same build) is a
weaker check than the rest of this section; the 0-mismatch compare-mode run on one build is the stronger evidence.

Flags and settings

One new switch, off by default (lifted, unchanged from before this PR) pending a Bosie A/B.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
HOS_ZSTD_SEQ_NATIVE=1 Runs the native port of f_801102e0's sequence loop instead of the lifted body n/a (always lifted) opts into the native loop no fps gain shown on the route tested (see above); correctness: confirmed identical either way off (the default) until a heavier scene shows a gain against main; the transcription is verified byte-for-byte so =1 costs nothing to try
HOS_ZSTD_SEQ_NATIVE=2 Runs both and compares every call, logging any difference, keeping the lifted result n/a a diagnostic/verification mode, not for normal play (roughly doubles this thread's CPU during a decompression burst) finds a real disagreement, if any, without ever returning a wrong result confirmed working: 60,000/60,000 agree in one real in-game run off for normal play; worth one real session with it on after this ships, per our own internal review of this port (not a path in this repository)

Testing

Tested:

  • Game-free: tests/test_zstd_native_seq.py and tests/test_zstd_native_compare.py pass (differential against
    libzstd 1.5.7, a mutant that must fail, the kill switch's values, HOS_ZSTD_SEQ_NATIVE=2's real compare behaviour).
  • Host compile: host/zstd_native.c alone, -fsyntax-only against the real build flags and generated headers:
    exit 0, no warnings.
  • Static correctness: the native loop was checked instruction-for-instruction and register-for-register against
    this build's own disassembled f_801102e0 (352 instructions) and f_834ab4a0 (the slow-path callee), and the
    real caller's rep-state init/write-back and last-literal copy around the three real call sites - all confirmed
    to match the port, not merely ReLink's own notes (our own internal review, not a path in this repository).
  • In-game, Linux, Ryzen 7 5800X + RX 9070 XT: a full hf48run run under HOS_ZSTD_SEQ_NATIVE=2 (60,000 compared
    calls, 0 mismatches, no crash, no device-lost line); the 3+3 A/B and screenshot comparison above.

Not tested: a scene where the sequence loop is actually the hot path (Hyrule Castle's own loading screen, or
an equivalent heavy streaming load) - the one route run this session (hf48run's teleport load) is lighter than
that and does not show a measured fps change either way; Windows; AMD/Intel/NVIDIA GPUs (irrelevant to this host-
only change, but untested); the one caller-side detail not located by direct disassembly (dctx->fseEntropy = 1's
exact write site - independently implied by the differential tests passing on multi-block frames, not confirmed
instruction-by-instruction; noted as an open item in our own internal review, not confirmed here).

Must be tested:

  • Linux or Windows, any GPU: a heavy loading/streaming scene (Hyrule Castle's interior load, or a similarly
    decompression-heavy area), HOS_ZSTD_SEQ_NATIVE=0 against unset, 3+ interleaved runs, expecting a measurable median
    frame-time or load-time gain where ParallelDecompThread's CPU share is actually high, and no mismatch under
    HOS_ZSTD_SEQ_NATIVE=2 (Test Farm: @Beanstalk linux, a heavy area load with and without HOS_ZSTD_SEQ_NATIVE=0)
  • Any platform, one real session: HOS_ZSTD_SEQ_NATIVE=2 through a Hyrule Castle load, checking longshot_log.txt
    for any "zstd: native != lifted" line (none expected), before relying on the default in a long session

Risks

What could break: a wrong decode would show as missing or broken models, textures, effects or music after an
area load, or a load that fails/crashes. Mitigated by: the transcription being checked instruction-for-instruction
against this build's own disassembly (not just trusted from the source port); the differential test against
libzstd across 40 compressed blocks and 323k sequences, with a canary past the output's end and a mutant that must
fail; and a real in-game compare-mode run agreeing on 60,000 of 60,000 calls. The one caller-side contract point
not instruction-verified (dctx->fseEntropy) is a field the native loop itself never reads or writes, so it
cannot be a source of a wrong decode, only (in principle) of the caller's own unrelated state.

Turning it off: HOS_ZSTD_SEQ_NATIVE=0.

Compatibility: Nothing changes: no save, option, cache or ABI change. third_party/ untouched (this file is
new, outside it).

Issues

None filed. Prior art and credit: Project ReLink commit bec02c50
(pixl27, GPL-3.0 per ReLink's own LICENSE file at that commit) - credited in the commit message and at the top of
host/zstd_native.c.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly, names the prior art and credits its author.
  • Measurements has a verdict: correctness proven, no fps gain shown on the route tested, and says plainly why.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named). Not run in full this
    session; the switch's own tests and the host compile check were run (see Testing).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched.
  • The license is OK: not ticked, for a maintainer's ruling. host/zstd_native.c is a transcription of the
    game binary's own compiled zstd 1.4 sequence loop (f_801102e0), instruction for instruction, from ReLink
    (GPL-3.0 per ReLink's own LICENSE at bec02c50). zstd itself is BSD/GPLv2, but this is written from the game's
    machine code, which CONTRIBUTING's Licensing section may count as "decompiled game code or anything generated
    from the dump". If it does, the alternative is a host loop written from zstd's own source
    (ZSTD_decompressSequences_body) that keeps the same in/out contract, with zstd's own notice. Question for
    the maintainers:
    is an instruction-for-instruction transcription of the game's compiled code (not generated
    from the dump itself, and not zstd's own source) acceptable under CONTRIBUTING's Licensing rule, or does it need
    to be rewritten from zstd's upstream source instead? We would rather ask than ship this unresolved.

When it applies

  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each (3), the median reported.
  • Switches documented (docs/development.md).
  • A file in docs/tested/ for each new measured run. Not added this session (the numbers above are quoted
    from this project's own tracking reports, not paths in this repository, not yet re-captured as docs/tested/
    fragments); add before marking this PR ready, and before it should be considered Ready for review.
  • A game-free test added, failing on main (no such function exists there) and passing here.
  • A check under tests/ that reads the changed source (the patch-placement check in
    tests/test_zstd_native_seq.py), not weakened.
**Type:** perf · **Risk:** low · **Tested on:** Linux (Ryzen 7 5800X, RX 9070 XT), game-free differential tests against libzstd 1.5.7, direct disassembly comparison against this build's own lifted code · **Needs:** a heavier streaming/loading scene than this PR's own route to show a measured gain; Windows; AMD/Intel/NVIDIA cross-check ## Why Area loads and streaming spend CPU in the game's own copy of zstd. `f_801102e0` is the main module's zstd 1.4 sequence loop (decoding and executing a block's sequences): its loading threads spend most of their time in it arriving in Hyrule Castle (89% of `ParallelDecompThread`'s samples there, per the source this port is based on). This is the port's third zstd path on this project: the stream decode (`f_80115c80`, `HOS_NATIVE_ZSTD`) and the resource loaders' whole-buffer decode (`f_8010b380`, `HOS_NATIVE_ZSTD_FRAMES`) already run on the host's libzstd upstream (#155); this one is a different function (the sequence loop inside a block's decode), not covered by either. Ported from [Project ReLink](https://github.com/pixl27/ProjectReLink) commit `bec02c50` (pixl27, GPL-3.0 per ReLink's own LICENSE file at bec02c50), onto this project's own lift. ReLink's author has not opened a Longshot PR for this (checked the open/closed PR list and PR authors for "pixl27"/"ReLink"/"relink": no match); this port is not a duplicate of anything already proposed here. ## What changed - `host/zstd_native.c` (new): `hos_zstd_seq`, run first in the lifted `f_801102e0`. It is a transcription of the real function's 352 instructions on locals instead of the lifted context (load for load, store for store, in the game's order), marking the pages it wrote once at the end instead of per instruction. `HOS_ZSTD_SEQ_NATIVE` is **off by default** (unset, or `off`/`false`/`no`/`0`): the lifted body runs exactly as before. `=1` runs the native loop; `=2` runs both and compares every call (`zstd_compare`), keeping the lifted result and logging a difference. Off by default until a Bosie A/B against `main` shows a gain (see Measurements: this PR's own route does not show one). Keeps ReLink's own transcription comments and measurement notes; the credit line and ReLink's commit are at the top of the file. - `nx2/oot_patches.py`: `patch_zstd_seq` puts the call at the head of the lifted `f_801102e0`, wired into `patch_file` and `report`'s done counts like the project's other zstd hooks (`patch_zstd_stream`, `patch_zstd_frames`). - `docs/development.md` and `gpu/vk/switches.def`: `HOS_ZSTD_SEQ_NATIVE` and `HOS_ZSTD_TIME` in the switch reference (`gpu/vk/switches.def` had no row for either; `tests/test_switch_table.py` failed on both before this). Renamed from the first push's `HOS_ZSTD_NATIVE`, which sat one word-order away from the existing `HOS_NATIVE_ZSTD`/`HOS_NATIVE_ZSTD_FRAMES` switches (a different zstd version and a different function) and invited exactly the kind of typo this review caught. - `tests/test_zstd_native_seq.py`: the loop against libzstd 1.5.7's block decoder on 11 frames (text, mixed, random, zeros, levels 1/3/19, checksum, empty, 1000 bytes; 40 compressed blocks, 323,399 sequences, 14 runs through the end-of-output path), a canary past the output's end, a mutant of the loop that must fail, the kill switch's values, and the patch's placement. - `tests/test_zstd_native_compare.py`: `HOS_ZSTD_SEQ_NATIVE=2` really runs both and compares (not a stub): a matching self-comparison logs 0 differences, a corrupted one logs 1 and keeps the lifted (correct) output; `HOS_ZSTD_TIME` runs. Changes against ReLink's own hunk, for this lift: the hook is named `ZSTD_SEQ_*` (the stream hook already has `ZSTD_HEAD`/`ZSTD_CALL`) and wired into `patch_file`/`report`'s done counts like the other hooks; switches read through `gpu/vk/compat/longshot_env.h`'s one rule (`env_value_off`, `env_int`, `env_on`); `_Thread_local` for `zstd_bypass`, as the rest of `host/`. ## Measurements **Verdict: correctness is thoroughly verified (byte-for-byte against this build's own disassembly, 60,000 of 60,000 real in-game compare-mode calls agree, every game-free differential test passes); no fps or frame-time gain is demonstrated on the route tested, which turns out not to be the scene the port targets. Still WIP: off by default until a heavier scene shows a real gain against `main`.** Baseline: our nightly-338-based build game-p12a (o1-frame-lag, lifted with `A64_LOCAL_REGS=6 A64_LOCAL_FLAGS=1 A64_LOCAL_FUNCS=<list>`), `HOS_ZSTD_SEQ_NATIVE=0` against `=1` on the same build; not main against this PR, which is still to run. Hardware: Linux, Ryzen 7 5800X, RX 9070 XT Scene: `hf48run` (a moving Hyrule Field route plus a Kakariko<->Hyrule Field teleport load), 2560x1440, render resolution fixed 100%, no frame generation, uncapped Method: (a) `HOS_ZSTD_SEQ_NATIVE=2` (compare mode) for one full `hf48run` run, counting the log's own mismatch tally; (b) `HOS_ZSTD_SEQ_NATIVE=0` (lifted) against `=1` (native), 3 interleaved runs each, median/mean/p99/lows on the route's `load` segment, and `ParallelDecompThread`'s CPU share from the same runs' thread-time log lines; (c) screenshots at each segment's end, pixel-diffed against a matching run | | compare mode (a), one run | off (lifted), 3 runs | on (native, default), 3 runs | |---|---|---|---| | sequence loops compared / mismatches | 60,000 / **0** | - | - | | `load` segment median frame time | - | 17.59 ms | 17.25 ms (-1.9%) | | `load` segment mean | - | 15.93 ms | 15.19 ms | | `load` segment p99 | - | 38.09 ms | 40.15 ms | | `load` segment 1% low / 0.1% low fps | - | 8.5 / 2.5 | 9.7 / 3.0 | | `load` segment frames >20/>33 ms | - | 972 / 55 | 1,079 / 67 | | `ParallelDecompThread`, mean % of a core | - | 6.97% (38 samples) | 6.88% (26 samples) | ``` zstd: the sequence loop (f_801102e0) native and lifted, compared on every call (HOS_ZSTD_SEQ_NATIVE=2) zstd: 60000 sequence loops compared, 0 different ``` **Reading, honestly**: the `load` segment's median gain (17.59 -> 17.25 ms, 1.9%) is smaller than the off arm's own run-to-run spread (17.66/17.00/18.11 ms, 6.3%) - not distinguishable from 3 runs' noise. `ParallelDecompThread`'s CPU share is essentially identical on and off (6.97% vs 6.88% of a core) and low in absolute terms either way; the `>33ms` count moved the *wrong* way for "on" (55->67), with the two arms' own ranges overlapping, so this is not a confident regression claim either. This route's decompression load (6-7% of a core) is much lighter than the Hyrule-Castle-loading-screen scene the port's own source attributes 89% of `ParallelDecompThread`'s samples to: `hf48run`'s teleport load is not that scene, so this A/B does not show the gain the port targets, for a route reason, not a correctness reason. A heavier streaming/loading scene is needed to show (or rule out) a real fps gain; not run this session. Screenshots at each `hf48run` segment's end (game-p12a, native mode, against game-o2, a different build run in lifted mode), pixel-diffed: mean absolute per-channel diff 8.9-21.9, max 240-254, none byte-identical but every pair visually the same scene, same HUD state, same object placement on inspection - consistent with the sub-degree camera-angle drift two independent 45 s scripted camera sweeps always have, not a correctness issue, and consistent with the 0-mismatch compare-mode result above. Comparing two different builds (not on/off on the same build) is a weaker check than the rest of this section; the 0-mismatch compare-mode run on one build is the stronger evidence. ### Flags and settings One new switch, off by default (lifted, unchanged from before this PR) pending a Bosie A/B. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `HOS_ZSTD_SEQ_NATIVE=1` | Runs the native port of `f_801102e0`'s sequence loop instead of the lifted body | n/a (always lifted) | opts into the native loop | no fps gain shown on the route tested (see above); correctness: confirmed identical either way | off (the default) until a heavier scene shows a gain against `main`; the transcription is verified byte-for-byte so `=1` costs nothing to try | | `HOS_ZSTD_SEQ_NATIVE=2` | Runs both and compares every call, logging any difference, keeping the lifted result | n/a | a diagnostic/verification mode, not for normal play (roughly doubles this thread's CPU during a decompression burst) | finds a real disagreement, if any, without ever returning a wrong result | confirmed working: 60,000/60,000 agree in one real in-game run | off for normal play; worth one real session with it on after this ships, per our own internal review of this port (not a path in this repository) | ## Testing **Tested:** - Game-free: `tests/test_zstd_native_seq.py` and `tests/test_zstd_native_compare.py` pass (differential against libzstd 1.5.7, a mutant that must fail, the kill switch's values, `HOS_ZSTD_SEQ_NATIVE=2`'s real compare behaviour). - Host compile: `host/zstd_native.c` alone, `-fsyntax-only` against the real build flags and generated headers: exit 0, no warnings. - Static correctness: the native loop was checked instruction-for-instruction and register-for-register against this build's own disassembled `f_801102e0` (352 instructions) and `f_834ab4a0` (the slow-path callee), and the real caller's rep-state init/write-back and last-literal copy around the three real call sites - all confirmed to match the port, not merely ReLink's own notes (our own internal review, not a path in this repository). - In-game, Linux, Ryzen 7 5800X + RX 9070 XT: a full `hf48run` run under `HOS_ZSTD_SEQ_NATIVE=2` (60,000 compared calls, 0 mismatches, no crash, no device-lost line); the 3+3 A/B and screenshot comparison above. **Not tested:** a scene where the sequence loop is actually the hot path (Hyrule Castle's own loading screen, or an equivalent heavy streaming load) - the one route run this session (`hf48run`'s teleport load) is lighter than that and does not show a measured fps change either way; Windows; AMD/Intel/NVIDIA GPUs (irrelevant to this host- only change, but untested); the one caller-side detail not located by direct disassembly (`dctx->fseEntropy = 1`'s exact write site - independently implied by the differential tests passing on multi-block frames, not confirmed instruction-by-instruction; noted as an open item in our own internal review, not confirmed here). **Must be tested:** - Linux or Windows, any GPU: a heavy loading/streaming scene (Hyrule Castle's interior load, or a similarly decompression-heavy area), `HOS_ZSTD_SEQ_NATIVE=0` against unset, 3+ interleaved runs, expecting a measurable median frame-time or load-time gain where `ParallelDecompThread`'s CPU share is actually high, and no mismatch under `HOS_ZSTD_SEQ_NATIVE=2` (Test Farm: @Beanstalk linux, a heavy area load with and without `HOS_ZSTD_SEQ_NATIVE=0`) - Any platform, one real session: `HOS_ZSTD_SEQ_NATIVE=2` through a Hyrule Castle load, checking `longshot_log.txt` for any `"zstd: native != lifted"` line (none expected), before relying on the default in a long session ## Risks **What could break:** a wrong decode would show as missing or broken models, textures, effects or music after an area load, or a load that fails/crashes. Mitigated by: the transcription being checked instruction-for-instruction against this build's own disassembly (not just trusted from the source port); the differential test against libzstd across 40 compressed blocks and 323k sequences, with a canary past the output's end and a mutant that must fail; and a real in-game compare-mode run agreeing on 60,000 of 60,000 calls. The one caller-side contract point not instruction-verified (`dctx->fseEntropy`) is a field the native loop itself never reads or writes, so it cannot be a source of a wrong *decode*, only (in principle) of the caller's own unrelated state. **Turning it off:** `HOS_ZSTD_SEQ_NATIVE=0`. **Compatibility:** Nothing changes: no save, option, cache or ABI change. `third_party/` untouched (this file is new, outside it). ## Issues None filed. Prior art and credit: [Project ReLink](https://github.com/pixl27/ProjectReLink) commit `bec02c50` (pixl27, GPL-3.0 per ReLink's own LICENSE file at that commit) - credited in the commit message and at the top of `host/zstd_native.c`. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly, names the prior art and credits its author. - [x] Measurements has a verdict: correctness proven, no fps gain shown on the route tested, and says plainly why. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). Not run in full this session; the switch's own tests and the host compile check were run (see Testing). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched. - [ ] The license is OK: **not ticked, for a maintainer's ruling.** `host/zstd_native.c` is a transcription of the game binary's own compiled zstd 1.4 sequence loop (`f_801102e0`), instruction for instruction, from ReLink (GPL-3.0 per ReLink's own LICENSE at bec02c50). zstd itself is BSD/GPLv2, but this is written from the game's machine code, which CONTRIBUTING's Licensing section may count as "decompiled game code or anything generated from the dump". If it does, the alternative is a host loop written from zstd's own source (`ZSTD_decompressSequences_body`) that keeps the same in/out contract, with zstd's own notice. **Question for the maintainers:** is an instruction-for-instruction transcription of the game's compiled code (not generated from the dump itself, and not zstd's own source) acceptable under CONTRIBUTING's Licensing rule, or does it need to be rewritten from zstd's upstream source instead? We would rather ask than ship this unresolved. ### When it applies - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each (3), the median reported. - [x] Switches documented (`docs/development.md`). - [ ] A file in `docs/tested/` for each new measured run. Not added this session (the numbers above are quoted from this project's own tracking reports, not paths in this repository, not yet re-captured as `docs/tested/` fragments); add before marking this PR ready, and before it should be considered Ready for review. - [x] A game-free test added, failing on `main` (no such function exists there) and passing here. - [x] A check under `tests/` that reads the changed source (the patch-placement check in `tests/test_zstd_native_seq.py`), not weakened.
Molier changed title from perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_NATIVE) to WIP: perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_NATIVE) 2026-11-18 23:33:31 +00:00
Molier force-pushed perf/zstd-native-seq from 1d35b9c373
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 4s
to 867d58872e
All checks were successful
PR description / Description follows the template (pull_request_target) Successful in 6s
2026-11-18 23:47:00 +00:00
Compare
Molier changed title from WIP: perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_NATIVE) to WIP: perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_SEQ_NATIVE, off by default) 2026-11-18 23:53:55 +00:00
Molier force-pushed perf/zstd-native-seq from 867d58872e
All checks were successful
PR description / Description follows the template (pull_request_target) Successful in 6s
to 5eea179183
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 17s
2026-11-20 14:38:18 +00:00
Compare
Author
Contributor

Rebased onto main @ 89c3d21e. The only conflict was the docs row (#1802). tests/test_zstd_native_compare.py and tests/test_zstd_native_seq.py pass; host compile of host/zstd_native.c clean. Still off by default and WIP: no main-vs-PR load-time measurement on this base yet.

Rebased onto `main` @ `89c3d21e`. The only conflict was the docs row (#1802). `tests/test_zstd_native_compare.py` and `tests/test_zstd_native_seq.py` pass; host compile of `host/zstd_native.c` clean. Still off by default and WIP: no main-vs-PR load-time measurement on this base yet.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 17s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/zstd-native-seq:Molier-perf/zstd-native-seq'
git switch 'Molier-perf/zstd-native-seq'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1670
No description provided.