WIP: perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_SEQ_NATIVE, off by default) #1670
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1670
Loading…
Reference in a new issue
No description provided.
Delete branch "Molier/Longshot:perf/zstd-native-seq"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: low · Tested on: Linux (Ryzen 7 5800X, RX 9070 XT), game-free differential tests against libzstd 1.5.7, direct disassembly comparison against this build's own lifted code · Needs: a heavier streaming/loading scene than this PR's own route to show a measured gain; Windows; AMD/Intel/NVIDIA cross-check
Why
Area loads and streaming spend CPU in the game's own copy of zstd.
f_801102e0is the main module's zstd 1.4sequence loop (decoding and executing a block's sequences): its loading threads spend most of their time in it
arriving in Hyrule Castle (89% of
ParallelDecompThread's samples there, per the source this port is based on).This is the port's third zstd path on this project: the stream decode (
f_80115c80,HOS_NATIVE_ZSTD) and theresource loaders' whole-buffer decode (
f_8010b380,HOS_NATIVE_ZSTD_FRAMES) already run on the host's libzstdupstream (#155); this one is a different function (the sequence loop inside a block's decode), not covered by
either.
Ported from Project ReLink commit
bec02c50(pixl27, GPL-3.0 per ReLink's own LICENSE file at bec02c50),onto this project's own lift. ReLink's author has not opened a Longshot PR for this (checked the open/closed PR
list and PR authors for "pixl27"/"ReLink"/"relink": no match); this port is not a duplicate of anything already
proposed here.
What changed
host/zstd_native.c(new):hos_zstd_seq, run first in the liftedf_801102e0. It is a transcription of thereal function's 352 instructions on locals instead of the lifted context (load for load, store for store, in
the game's order), marking the pages it wrote once at the end instead of per instruction.
HOS_ZSTD_SEQ_NATIVEis off by default (unset, or
off/false/no/0): the lifted body runs exactly as before.=1runs thenative loop;
=2runs both and compares every call (zstd_compare), keeping the lifted result and logging adifference. Off by default until a Bosie A/B against
mainshows a gain (see Measurements: this PR's own routedoes not show one). Keeps ReLink's own transcription comments and measurement notes; the credit line and ReLink's
commit are at the top of the file.
nx2/oot_patches.py:patch_zstd_seqputs the call at the head of the liftedf_801102e0, wired intopatch_fileandreport's done counts like the project's other zstd hooks (patch_zstd_stream,patch_zstd_frames).docs/development.mdandgpu/vk/switches.def:HOS_ZSTD_SEQ_NATIVEandHOS_ZSTD_TIMEin the switchreference (
gpu/vk/switches.defhad no row for either;tests/test_switch_table.pyfailed on both before this).Renamed from the first push's
HOS_ZSTD_NATIVE, which sat one word-order away from the existingHOS_NATIVE_ZSTD/HOS_NATIVE_ZSTD_FRAMESswitches (a different zstd version and a different function) andinvited exactly the kind of typo this review caught.
tests/test_zstd_native_seq.py: the loop against libzstd 1.5.7's block decoder on 11 frames (text, mixed,random, zeros, levels 1/3/19, checksum, empty, 1000 bytes; 40 compressed blocks, 323,399 sequences, 14 runs
through the end-of-output path), a canary past the output's end, a mutant of the loop that must fail, the kill
switch's values, and the patch's placement.
tests/test_zstd_native_compare.py:HOS_ZSTD_SEQ_NATIVE=2really runs both and compares (not a stub): a matchingself-comparison logs 0 differences, a corrupted one logs 1 and keeps the lifted (correct) output;
HOS_ZSTD_TIMEruns.
Changes against ReLink's own hunk, for this lift: the hook is named
ZSTD_SEQ_*(the stream hook already hasZSTD_HEAD/ZSTD_CALL) and wired intopatch_file/report's done counts like the other hooks; switches readthrough
gpu/vk/compat/longshot_env.h's one rule (env_value_off,env_int,env_on);_Thread_localforzstd_bypass, as the rest ofhost/.Measurements
Verdict: correctness is thoroughly verified (byte-for-byte against this build's own disassembly, 60,000 of
60,000 real in-game compare-mode calls agree, every game-free differential test passes); no fps or frame-time gain
is demonstrated on the route tested, which turns out not to be the scene the port targets. Still WIP: off by
default until a heavier scene shows a real gain against
main.Baseline: our nightly-338-based build game-p12a (o1-frame-lag, lifted with
A64_LOCAL_REGS=6 A64_LOCAL_FLAGS=1 A64_LOCAL_FUNCS=<list>),HOS_ZSTD_SEQ_NATIVE=0against=1on the same build; not main against this PR, whichis still to run.
Hardware: Linux, Ryzen 7 5800X, RX 9070 XT
Scene:
hf48run(a moving Hyrule Field route plus a Kakariko<->Hyrule Field teleport load), 2560x1440, renderresolution fixed 100%, no frame generation, uncapped
Method: (a)
HOS_ZSTD_SEQ_NATIVE=2(compare mode) for one fullhf48runrun, counting the log's own mismatch tally;(b)
HOS_ZSTD_SEQ_NATIVE=0(lifted) against=1(native), 3 interleaved runs each, median/mean/p99/lows onthe route's
loadsegment, andParallelDecompThread's CPU share from the same runs' thread-time log lines; (c)screenshots at each segment's end, pixel-diffed against a matching run
loadsegment median frame timeloadsegment meanloadsegment p99loadsegment 1% low / 0.1% low fpsloadsegment frames >20/>33 msParallelDecompThread, mean % of a coreReading, honestly: the
loadsegment's median gain (17.59 -> 17.25 ms, 1.9%) is smaller than the off arm'sown run-to-run spread (17.66/17.00/18.11 ms, 6.3%) - not distinguishable from 3 runs' noise.
ParallelDecompThread'sCPU share is essentially identical on and off (6.97% vs 6.88% of a core) and low in absolute terms either way; the
>33mscount moved the wrong way for "on" (55->67), with the two arms' own ranges overlapping, so this is not aconfident regression claim either. This route's decompression load (6-7% of a core) is much lighter than the
Hyrule-Castle-loading-screen scene the port's own source attributes 89% of
ParallelDecompThread's samples to:hf48run's teleport load is not that scene, so this A/B does not show the gain the port targets, for a routereason, not a correctness reason. A heavier streaming/loading scene is needed to show (or rule out) a real fps
gain; not run this session.
Screenshots at each
hf48runsegment's end (game-p12a, native mode, against game-o2, a different build run inlifted mode), pixel-diffed: mean absolute per-channel diff 8.9-21.9, max 240-254, none byte-identical but every
pair visually the same scene, same HUD state, same object placement on inspection - consistent with the sub-degree
camera-angle drift two independent 45 s scripted camera sweeps always have, not a correctness issue, and consistent
with the 0-mismatch compare-mode result above. Comparing two different builds (not on/off on the same build) is a
weaker check than the rest of this section; the 0-mismatch compare-mode run on one build is the stronger evidence.
Flags and settings
One new switch, off by default (lifted, unchanged from before this PR) pending a Bosie A/B.
HOS_ZSTD_SEQ_NATIVE=1f_801102e0's sequence loop instead of the lifted bodymain; the transcription is verified byte-for-byte so=1costs nothing to tryHOS_ZSTD_SEQ_NATIVE=2Testing
Tested:
tests/test_zstd_native_seq.pyandtests/test_zstd_native_compare.pypass (differential againstlibzstd 1.5.7, a mutant that must fail, the kill switch's values,
HOS_ZSTD_SEQ_NATIVE=2's real compare behaviour).host/zstd_native.calone,-fsyntax-onlyagainst the real build flags and generated headers:exit 0, no warnings.
this build's own disassembled
f_801102e0(352 instructions) andf_834ab4a0(the slow-path callee), and thereal caller's rep-state init/write-back and last-literal copy around the three real call sites - all confirmed
to match the port, not merely ReLink's own notes (our own internal review, not a path in this repository).
hf48runrun underHOS_ZSTD_SEQ_NATIVE=2(60,000 comparedcalls, 0 mismatches, no crash, no device-lost line); the 3+3 A/B and screenshot comparison above.
Not tested: a scene where the sequence loop is actually the hot path (Hyrule Castle's own loading screen, or
an equivalent heavy streaming load) - the one route run this session (
hf48run's teleport load) is lighter thanthat and does not show a measured fps change either way; Windows; AMD/Intel/NVIDIA GPUs (irrelevant to this host-
only change, but untested); the one caller-side detail not located by direct disassembly (
dctx->fseEntropy = 1'sexact write site - independently implied by the differential tests passing on multi-block frames, not confirmed
instruction-by-instruction; noted as an open item in our own internal review, not confirmed here).
Must be tested:
decompression-heavy area),
HOS_ZSTD_SEQ_NATIVE=0against unset, 3+ interleaved runs, expecting a measurable medianframe-time or load-time gain where
ParallelDecompThread's CPU share is actually high, and no mismatch underHOS_ZSTD_SEQ_NATIVE=2(Test Farm: @Beanstalk linux, a heavy area load with and withoutHOS_ZSTD_SEQ_NATIVE=0)HOS_ZSTD_SEQ_NATIVE=2through a Hyrule Castle load, checkinglongshot_log.txtfor any
"zstd: native != lifted"line (none expected), before relying on the default in a long sessionRisks
What could break: a wrong decode would show as missing or broken models, textures, effects or music after an
area load, or a load that fails/crashes. Mitigated by: the transcription being checked instruction-for-instruction
against this build's own disassembly (not just trusted from the source port); the differential test against
libzstd across 40 compressed blocks and 323k sequences, with a canary past the output's end and a mutant that must
fail; and a real in-game compare-mode run agreeing on 60,000 of 60,000 calls. The one caller-side contract point
not instruction-verified (
dctx->fseEntropy) is a field the native loop itself never reads or writes, so itcannot be a source of a wrong decode, only (in principle) of the caller's own unrelated state.
Turning it off:
HOS_ZSTD_SEQ_NATIVE=0.Compatibility: Nothing changes: no save, option, cache or ABI change.
third_party/untouched (this file isnew, outside it).
Issues
None filed. Prior art and credit: Project ReLink commit
bec02c50(pixl27, GPL-3.0 per ReLink's own LICENSE file at that commit) - credited in the commit message and at the top of
host/zstd_native.c.Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). Not run in full thissession; the switch's own tests and the host compile check were run (see Testing).
third_party/untouched.host/zstd_native.cis a transcription of thegame binary's own compiled zstd 1.4 sequence loop (
f_801102e0), instruction for instruction, from ReLink(GPL-3.0 per ReLink's own LICENSE at bec02c50). zstd itself is BSD/GPLv2, but this is written from the game's
machine code, which CONTRIBUTING's Licensing section may count as "decompiled game code or anything generated
from the dump". If it does, the alternative is a host loop written from zstd's own source
(
ZSTD_decompressSequences_body) that keeps the same in/out contract, with zstd's own notice. Question forthe maintainers: is an instruction-for-instruction transcription of the game's compiled code (not generated
from the dump itself, and not zstd's own source) acceptable under CONTRIBUTING's Licensing rule, or does it need
to be rewritten from zstd's upstream source instead? We would rather ask than ship this unresolved.
When it applies
docs/development.md).docs/tested/for each new measured run. Not added this session (the numbers above are quotedfrom this project's own tracking reports, not paths in this repository, not yet re-captured as
docs/tested/fragments); add before marking this PR ready, and before it should be considered Ready for review.
main(no such function exists there) and passing here.tests/that reads the changed source (the patch-placement check intests/test_zstd_native_seq.py), not weakened.perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_NATIVE)to WIP: perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_NATIVE)1d35b9c373867d58872eWIP: perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_NATIVE)to WIP: perf(host): the game's zstd sequence loop runs natively (HOS_ZSTD_SEQ_NATIVE, off by default)867d58872e5eea179183Rebased onto
main@89c3d21e. The only conflict was the docs row (#1802).tests/test_zstd_native_compare.pyandtests/test_zstd_native_seq.pypass; host compile ofhost/zstd_native.cclean. Still off by default and WIP: no main-vs-PR load-time measurement on this base yet.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.