perf(host): the game's whole-buffer zstd decode on the host's libzstd (HOS_NATIVE_ZSTD_FRAMES, opt-in) #155
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!155
Loading…
Reference in a new issue
No description provided.
Delete branch "klimovich008/Longshot:perf/native-zstd-frames"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
Area loads and streaming spend CPU in the game's own copy of zstd. NotRexed/Hookshot#2738 moved its stream decode (ParallelDecompThread) to the host's libzstd. The game's other zstd path, the resource loaders' whole-buffer decode
f_8010b380on the ARes_Loading threads, is about 22% of a core in Kokiri Forest and 6% in the standard test (CPU report). This PR moves that decode to libzstd too, behind a switch that stays off by default: in game no gain was shown: on Longshot's standard test (3 pairs, diagnostic) the switch was behind in every pair, within noise, and the earlier Hookshot runs resolved no difference.This is a performance change, not a bug fix. It also fixes a first-call data race (C memory model; no wrong output seen) in #2738's
HOS_NATIVE_ZSTDswitch, found in review.What changed
hos_zstd_framesinhost/hos_zstd.c, run first inf_8010b380(zstd'sZSTD_decompressMultiFrame, inlined: x0 the game's context, x1/x2 the output and its room, x3/x4 the frames, x5/x6 a raw dictionary, x7 the game's DDict or 0). It returns what the game's function returns: the decoded size, or a zstd error code the three callers map throughZSTD_getErrorCode, as before.ZSTD_decompress_usingDDict(the game's dictionary by reference, through the stream decode's DDict cache, checked against a copy of the game's tables) orZSTD_decompress_usingDict(NULL, 0); libzstd contexts come from the stream decode's spare list.HOS_WRITTENafter the decode (libzstd may keep literals in the room's far end while it decodes).zs_mode(HOS_NATIVE_ZSTD) and the newzf_modecached their switch in a plain static that several decode threads read and wrote, and the once-logged fallback table was unsynchronized.zs_publishnow publishes each switch once with a compare-and-swap, and only the thread that published it logs it.zs_noteclaims its table entry under the existingzs_lockand logs after releasing it. Only first calls and logging change; the decode path does not.patch_zstd_framesinnx2/oot_patches.pyputsif (hos_zstd_frames(c)) return;first inf_8010b380, counted byreport()like the stream hook (it warns unless found exactly once).tests/test_native_zstd.py, game-free:env_onholds each caller that reads a switch unset until two have, so both go on to publish it; and a stand-in log holds the first thread in its first log line while the second runs start to end;docs/development.md's switch reference (NotRexed/Hookshot#2738 had not addedHOS_NATIVE_ZSTD); one file per measured run underdocs/tested/(the table indocs/tested-configurations.mdis frozen).Flags and settings
HOS_NATIVE_ZSTD_FRAMES(set)HOS_NATIVE_ZSTD(from #2738) is unchanged apart from the race fix.Measurements
main:
839e85ff(2026-11-13), the base measured onthis PR: perf/native-zstd-frames @ 97915751 (
839e85ffwith this PR's commits; reachable as the tag pr155-measured-97915751 on my fork). The head now is rebased onto main5e072fc8;host/hos_zstd.cis byte for byte the measured one and the zstd hook's text innx2/oot_patches.pyis the same (checked withcmpand a diff of the hook's lines). The test file changed after the measurement (Windows threads); it is not part of the game build.Hardware: RX 9070 XT, AMD 26.10.2 (Vulkan driver 0x8001a0), Ryzen 7 5700X3D (no efficiency cores), 64 GB, Windows 11
Scene: the standard test: Hyrule Field from the test save, the camera turning at full speed; a 1280x720 window (16:9), the test profile's options (output 2048x1152), no upscaler, no frame generation, frame rate unlocked (
HOS_FPS_MAX=240)Method: one binary (
longshot v3.0.0-rc.2-10-g97915751, exed64c70b3c72deeb8,longshot_vk.dll8eb6d56dbc601d8f), the switch unset and=1, warm-ups, then 3 alternating pairs, 20 s windows from the load (run_profile.pyaee7483e,std_test.pyadc2e9f0); fps and frame times from the present trace, CPU from the process. A run counts when its game clock is within 1.000 ± 0.01 and its window has no frame over 250 ms. With 3 pairs, a difference under 5% is diagnostic. One file per run:docs/tested/2026-11-13-zstd-frames-*-klimovich008-pair*.md.How to reproduce: build this branch, then for each run set
HOS_FPS_MAX=240andHOS_FPS_LOG=1, load the save and hold the right stick fully right; alternateHOS_NATIVE_ZSTD_FRAMESunset and=1.Baseline: this PR @ 97915751,
HOS_NATIVE_ZSTD_FRAMESunset, pair 1 (ctl-0), with the same in pairs 2 (ctl-1) and 3 (ctl-2). The control is this PR's own binary with the switch off, not a build of main. It isolates the switch, but it does not measure this PR's default-off cost or the shared first-call change against main.Each run, in the order they ran (columns left to right):
HOS_NATIVE_ZSTD_FRAMESunset, pair 1 (ctl-0)HOS_NATIVE_ZSTD_FRAMES=1, pair 1 (frames-0)HOS_NATIVE_ZSTD_FRAMES=1, pair 2 (frames-1)HOS_NATIVE_ZSTD_FRAMESunset, pair 2 (ctl-1)HOS_NATIVE_ZSTD_FRAMESunset, pair 3 (ctl-2)HOS_NATIVE_ZSTD_FRAMES=1, pair 3 (frames-2)=1minus unset: the 2 accepted pairs (2, 3)=1minus unset: all 3 pairsEarlier, on Hookshot (32:9, before the race fix)
Build: d0b8141f, Hookshot v2.1.0-rc.3-250-g46780503 with #2738 and this change before the race fix (exe
a46bf3a229310166,hookshot_vk.dll418aa37c49bc96ce).Kokiri Forest, 3 warm-ups, then 6 runs an arm: 5 of 6 pairs count.
HOS_NATIVE_ZSTD_FRAMESunsetHOS_NATIVE_ZSTD_FRAMES=1HOS_NATIVE_ZSTD_FRAMES=1,HOS_NATIVE_ZSTD=2=1run had frames of 276 and 253 ms in the window and a clock reading of 0.933 over that stretch. The same pair's control run had a 131.6 ms frame, and the warm-ups, which run with the switch off, had frames of 295 and 345 ms. With pair 1 included, the paired CPU differences are -1.66 and -2.09 ms.The standard test, 2 warm-ups, then 6 pairs: all 6 count.
HOS_NATIVE_ZSTD_FRAMESunsetHOS_NATIVE_ZSTD_FRAMES=1HOS_NATIVE_ZSTD=1while moving (context for #2738)A tester saw
HOS_NATIVE_ZSTD=1slower than unset in Kokiri Forest while moving (RTX 5090, i7-13700K: 72.2 against 66.7 fps). I checked it on this PC with #2738 as merged (before the race fix) and this change switched off (exee9be69d36122b25e): Kokiri Forest from the default save, running with the camera turning, 2 warm-ups, then 6 triples. Two triples are left out by the rules above: an unset run with frames of 576, 284 and 135 ms, and an=2run with a window clock reading of 0.990. On the other 4:HOS_NATIVE_ZSTDunsetHOS_NATIVE_ZSTD=1HOS_NATIVE_ZSTD=2=1was faster in all 4 of those pairs; in one more pair (valid for unset against=1only) it was slower, 51.0 against 53.2 fps.=1makes no more virtual-core hand-offs than unset. ParallelDecompThread went from about 2% of a core to about 0.=1changes.Testing
Tested:
f_8010b380called through one of its callers in Unicorn from a local image, againsthos_zstd_frameson the same inputs: 46 cases, 49 calls (the test's kinds plus larger sets up to 2 MB, errors, a DDict made again). Results, including error codes, and the decoded output were identical. In 2 of the cases with 1000 bytes more room than the content, the bytes in the room past the content differ (libzstd keeps literals there while it decodes). A broken copy fails 8 cases.python -I -B tests/run_all.pyon this branch atf81d2f93, on main5e072fc8(Windows, llvm-mingw,LONGSHOT_MINGWset): 519 passed, 37 skipped, 5 failed:tests/test_module_ids.py,tests/test_pad_layout.py,tests/test_framegen_resources.py,tests/test_tested_configurations.pyandtests/test_oot_state_tap.py. All five fail the same way on unmodified main5e072fc8on this PC; this PR touches none of what they cover.tests/test_native_zstd.pypasses with Clang targeting Microsoft's runtime (23.1.5, x86_64-pc-windows-msvc), llvm-mingw and MinGW gcc 16.2, and every broken copy in it fails with each. On Windows the concurrent fixture uses Windows threads; its POSIX branch was also built and run here with MinGW gcc and winpthreads (correct code passes, the three concurrent broken copies fail).host/hos_zstd.ccompiles as a whole (clang -fsyntax-onlyagainst a839e85ffbuild's generated headers).Not tested:
Needs other testers:
HOS_NATIVE_ZSTD_FRAMESunset against=1, 3 alternating pairs, expecting no slowdown and no new long frames (T-S3LD)HOS_NATIVE_ZSTD=2and=1against unset, expecting no slowdown (T-J7B5)Screenshots
Nothing on screen.
Risk and rollback
A wrong decode would show as missing or broken models, textures or effects after an area load, or a load that fails. It is off unless
HOS_NATIVE_ZSTD_FRAMESis set; unset, the hook returns at its first check and the game's own decode runs. The race fix changes only first-call setup and logging; reverting its commit restores the old first-call code.Reviewer notes
Start with
hos_zstd_framesand the race fix at the top ofhost/hos_zstd.c, then the whole-buffer and concurrent cases intests/test_native_zstd.py. Prior art: zortos293/Lichii#869 puts this game's zstd decode on the host's libzstd in an emulator fork. I did not read its code; it's credited as related work. Also related: NotRexed/Hookshot#909 (the lifted decoder at -O2).Issues
See NotRexed/Hookshot#2738 (its first-call race is fixed here).
Checklist
For everyone
type(scope): description).main.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above).third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each measured run.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
a0ff2d949112f8cc8ed9Measured on Longshot and rebased:
839e85ff+ this PR (97915751), 3 alternating pairs:HOS_NATIVE_ZSTD_FRAMESunset 65.99 fps / 61.84 ms CPU a frame,=165.09 fps / 63.41 ms. No gain, diagnostic. The switch was behind in all 3 pairs (fps -0.45 / -0.90 / -0.59).514591a3(12f8cc8e).host/hos_zstd.c, the test and the zstd hook are byte for byte the measured ones.run_all: 494 passed, 4 failed. All four fail on main on this PC too;test_menu_i18n.pyfails on main514591a3itself.docs/tested-configurations.mdfor these runs.The switch stays off by default.
WIP: perf(host): the game's whole-buffer zstd decode on the host's libzstd (HOS_NATIVE_ZSTD_FRAMES, opt-in)to perf(host): the game's whole-buffer zstd decode on the host's libzstd (HOS_NATIVE_ZSTD_FRAMES, opt-in)Read head
12f8cc8e, the complete discussion and all five changed files against maincf605c85, including the hook's callers and the shared decoder cache; rantest_native_zstd.pyandtest_env_on.pywith GCC and Clang, plustest_patch_hooks_defined.py: all passed.What's good
Needed before it can merge
tests/test_native_zstd.pylines 168-169 now includepthread.handsched.hunconditionally, and line 641 adds-pthread.tests/run_all.pycan select ordinaryclang++on Windows, not just llvm-mingw. The Microsoft CRT/SDK does not supply these POSIX headers or thread functions, so this newly mandatory part of the test cannot compile there. The reported llvm-mingw and MinGW GCC passes do not cover that compiler target. Use a Windows thread implementation alongside the POSIX one, keeping the same concurrent assertions and broken variants, and run the test with Clang targeting Microsoft's runtime. This is a source portability finding; I did not execute that Windows configuration.docs/tested-configurations.mdlines 17-18 conflict with main's newer rows at the start of the table. This is the only textual merge conflict atcf605c85; the other four changed files merge. Main now freezes that table, so move this PR's new measurements into individual files underdocs/tested/, rather than appending the seven summary rows during the rebase. Keep main's existing rows. The format is atcf605c85bb/docs/tested/README.md.Smaller points
-0.65fps and+1.80CPU ms match the arithmetic means of the three listed pair differences, not their medians. Those medians are-0.59fps and+2.50ms. Label the paired summaries as means, or use medians, and show the accepted pairs separately from the rejected first pair. A row per run would make the calculation and exclusions checkable.Not checked
No full build, whole suite, game run, GPU measurement or Windows execution here. The original-decoder comparison and the gameplay results remain the author's evidence. No raw run records are linked in the discussion, so I could not independently recompute the run medians or assess background load. Measured commit
97915751was unavailable in the fetched history and through the fork's public commit and raw-file URLs, so I have not verified the claimed byte-for-byte carry to this head.12f8cc8ed9f81d2f93b0@mooki thanks for review 1244. Every point, in order. New head
f81d2f93, rebased onto main5e072fc8.Needed 1: the concurrent test with Microsoft-target Clang. Fixed in
1b179574._WIN32:CreateThread,WaitForSingleObject+CloseHandle,SwitchToThread, withWIN32_LEAN_AND_MEANandNOMINMAX;sched_yield.-pthreadis added only whenos.name != "nt". The assertions and all broken copies are unchanged.x86_64-pc-windows-msvc, and with llvm-mingw and MinGW gcc 16.2. All pass, and every broken copy fails with each.Needed 2: the measured runs in main's new format. Fixed by the rebase and
f81d2f93.5e072fc8without the three commits that only added rows to the frozen table.docs/tested-configurations.mdis now main's, unchanged.docs/tested/2026-11-13-zstd-frames-{off,on}-klimovich008-pair{1,2,3}.md. Each has its pair, its position in the pair and its own numbers. The rejected run says why.tests/test_tested_configurations.pyaccepts them. Its one failure here, the CRLF pipe check, also fails on unmodified main on Windows.Smaller 1: means against medians, the accepted pairs apart. In the description: a row per run in run order, with a "counts" column. A second table shows the per-pair differences, with means and medians labelled, for the 2 accepted pairs and for all 3. Accepted pairs: fps -0.75 (mean = median), CPU +1.45 ms (mean = median). All 3: fps mean -0.65, median -0.59; CPU mean +1.80, median +2.50.
Smaller 2: what the control measures.
Not checked: 97915751. It's now on my fork as the tag
pr155-measured-97915751: https://git.magicbean.dev/klimovich008/Longshot/src/tag/pr155-measured-97915751cmpofhost/hos_zstd.c, which is byte for byte the same, and a diff of the zstd hook's lines innx2/oot_patches.py: same text, other line numbers.Also run:
run_allatf81d2f93: 519 passed, 5 failed. All five fail the same way on unmodified main5e072fc8on this PC:test_module_ids,test_pad_layout,test_framegen_resources,test_tested_configurations,test_oot_state_tap.host/hos_zstd.csyntax-compiles whole.Second look at head
f81d2f93. Every point of the first review is settled, and nothing new was found. The first review was a comment, not a change request, so nothing has to be withdrawn.Read
f81d2f93, the reply and the revised series against12f8cc8e; ran the native zstd test with GCC and Clang, the 25 catalogue checks, and Windows-target syntax checks of the concurrent fixture: all passed.Previous points
tests/test_native_zstd.pylines 168-190 select the Win32 implementation on Windows and pthreads elsewhere. The callbacks have the right Windows signature, and each joined handle is closed. Line 662 omits-pthreadon Windows. The assertions and mutation definitions are unchanged; both Linux compiler runs still reject all five broken variants. The Win32 concurrent fixture also passes Clang'sx86_64-w64-mingw32syntax check here. The reported Microsoft-target Windows execution is separate evidence from the author, not a run I repeated.5e072fc8and merges without conflicts.docs/tested-configurations.mdis byte-for-byte main's. The sixdocs/tested/2026-11-13-zstd-frames-{off,on}-klimovich008-pair{1,2,3}.mdfiles each have one result at line 3, with their own context, run identifier and position in the pair. The rejected run is kept and labelled. The older Hookshot results remain historical context in the description, not new rows in the frozen table.pr155-measured-97915751from the fork.host/hos_zstd.c, including lines 371-404, is identical between that measured commit, the previous reviewed head and this head. The whole-buffer hook text innx2/oot_patches.pylines 350-366 is identical too.Rebase and new changes
The series comparison shows the requested thread repair and replacement of the table additions with run files. The existing decoder changes and behavioral assertions were preserved. The unrelated damage and ocarina changes in
oot_patches.pycome from main, not additional work in this PR. I found no new defect or remaining change to request.Limits
No game, full build, whole suite or Windows execution here. The Windows check used llvm-mingw headers and was syntax-only; it does not replace the author's Microsoft-runtime run. The raw frame traces and background-load records remain unavailable, so the gameplay measurements remain the author's evidence. Source equality does not turn those older runs into measurements of the newer main.
Approving at
f81d2f93. Every point of the review is answered: the test's thread layer for Windows, the rebase onto current main with the results in per-run files, the labelled statistics, what the control measures, and the measured source being this source. The second look found nothing new: #155 (comment)As said there, the gameplay numbers are the author's and show no gain yet; the switch stays off by default, which is the right state for it.
Should we enable this by default? What do you think?
Code review only (diff and surrounding code read; nothing built or run). I found no correctness bug in the hook.
What I checked:
hos_zstd_framesreturns 0 (the lifted code runs) for a raw dictionary, a magicless context, no libzstd context and a DDict libzstd won't take; it is a single early return when the switch is off. The DCtx and DDict reference handling is paired on every path (spare list andzs_dict_putunderzs_lock,zs_dict_put(NULL)is safe,ZSTD_freeDCtx(NULL)is a no-op).zs_publishcompare-exchange so only one caller logs;zs_notenow takeszs_lockfor its table). Its caller in the stream path is not underzs_lock, so no self-deadlock.HOS_WRITTENover the whole room is the safe choice, since libzstd can leave literals in the room's far end.patch_zstd_framesis idempotent and only applies withprotect_hearts, like the stream hook.Points for the author and maintainers:
ZSTD_getErrorCode. The two disagree. If a caller maps the code, libzstd 1.5.7's error numbering must match the game's older copy. The local Unicorn comparison (identical codes in 46 cases) covers it, but the comment inhost/hos_zstd.cshould say which is true.Not blocking. Leaving this as a comment, not an approval, because of point 1 and because Linux and non-AMD hardware are unmeasured.