perf(host): the game's zstd decodes run on the host's libzstd by default (HOS_NATIVE_ZSTD=0: as lifted code) #1402
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1402
Loading…
Reference in a new issue
No description provided.
Delete branch "GummiNaomi/Longshot:perf/native-zstd-default"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: medium · Tested on: Windows 11, RTX 4070 SUPER (NVIDIA 617.42), Ryzen 9 9950X3D · Needs: Linux; a Steam Deck or another CPU of four cores; a played session
Why
Every area load unpacks the game's
.zsfiles with the game's own copy of zstd, which runs as lifted code. In a profile of the walk from Castle Town into Hyrule Field (HOS_PROFILE=1), in the frames around the area change the two kinds of loader thread (ParallelDecompThread, the ARes_Loading threads) spend 10 to 33 % of their samples in one lifted function they share,f_801102e0, which I take to be that decoder's inner loop (not checked further), and a stream decode holds one of the six virtual cores for as long as it runs.Both decodes have had a path through the host's own libzstd for a while:
HOS_NATIVE_ZSTDfor the streams andHOS_NATIVE_ZSTD_FRAMESfor the resource loaders' whole buffers (#155). They are opt-in. They were measured when they were written on a save's load in Hyrule Field, which did not get shorter (26.2-29.2 s against 25.9-28.9 s,docs/tested-configurations.md), so there was no reason to turn them on.Measured over many area loads instead of one, they do help: the loading screens are about a tenth shorter. So this turns them on by default.
What changed
Both switches are on unless they say off.
host/hos_zstd.c):zs_wanted(name)reads a switch: unset or empty is on,0,false,offornois off.zs_mode()andzf_mode()ask it.HOS_NATIVE_ZSTDunset is what=1was (the stream decoded by libzstd, the thread's virtual core given up meanwhile);=2keeps the core, as before;=0runs the game's own decoder as lifted code.HOS_NATIVE_ZSTD_FRAMES=0does the same for the whole-buffer decode. Nothing else in the file changes: the same calls into libzstd, the same fallbacks to the lifted code (a misaligned workspace, a dictionary libzstd does not take, a raw dictionary, another frame format), the same log lines.tests/test_native_zstd.pyruns the stream decode and the whole-buffer decode with the switches unset and empty as well (done on the host, the output the original, the core given up and taken back as with=1); its "switch off" cases already set0.docs/development.md; six files underdocs/tested/.Measurements
Verdict: the loading screens of an area change are about a tenth shorter: 5.21 and 5.63 s a teleport with this change's default against 6.38 and 5.82 s with both switches at 0 (the same build, in turn). Two earlier pairs on main with the switches set by hand agree: 5.56 and 5.51 s against 6.28 and 6.28 s. Over the eight tours no "off" tour is as short as any "on" tour, the closest by 0.19 s. Keeping the core (
HOS_NATIVE_ZSTD=2) is no different from giving it up here (5.54 and 5.27 s). The long frames around an area change do not move, and nothing else was seen to.Hardware: Windows 11 Pro 26200, RTX 4070 SUPER 12 GB (NVIDIA 617.42), Ryzen 9 9950X3D (16 cores, SMT off), 32 GB
Scene: the repository's
kokiri-startsave, then the debug teleport round the Market, Hyrule Field, Kakariko, Kokiri Forest, the Lost Woods and Zora's River, 20 s to arrive and 12 s more in each, for 10 minutes (36 to 38 teleports); headless,scripts/longshot_ctl.py launch --profile test, uncapped (HOS_FPS_MAX=1000)Method: one build, the three settings in turn, twice; from the frame trace (
RX_VK_FRAME_TRACE=1, every side) from 600 frames after the save's arrival to the end: the time on loading screens is the sum of the frames with under 2 ms of draws (draw_ms), divided by the tour's teleports; the long frames are the others of 50 ms or moreBaseline: this PR with
HOS_NATIVE_ZSTD=0andHOS_NATIVE_ZSTD_FRAMES=0, which is main's default (the game's decoder as lifted code). The build is main544229a2with this change (in the working tree when the tours ran, committed after asea7ed5b5unchanged) and the link fix our MSVC-target build needs (#1379)How to reproduce:
HOS_NATIVE_ZSTD=0andHOS_NATIVE_ZSTD_FRAMES=0against no switch, a tour of teleports with the control bridge (longshot_ctl.py teleport <map> --arrive 20); the log'szstd: the game's ... decode on the host's libzstdlines say what runs whereHOS_NATIVE_ZSTD=0HOS_NATIVE_ZSTD_FRAMES=0: this PR @ea7ed5b5(two tours)ea7ed5b5(two tours)HOS_NATIVE_ZSTD=2: this PR @ea7ed5b5(two tours)runs liftedlines)The two earlier pairs, on main
544229a2with #1359 and the switches set by hand (=1andFRAMES=1against unset, the same tour):544229a2with #1359 (two tours)HOS_NATIVE_ZSTD=1HOS_NATIVE_ZSTD_FRAMES=1: main @544229a2with #1359 (two tours)The picture, both ways: one full-size shot in each of the six areas with the switches unset and with both set, compared by eye by two people: everything is drawn on both, and what differs is what two runs differ by (the pose, the clouds, the camera still settling after the save's load).
Flags and settings
No new switch; two existing ones change their default.
HOS_NATIVE_ZSTD1on, the core given up;2on, the core kept1);2the core kept;0the lifted codeHOS_NATIVE_ZSTD_FRAMES0the lifted codeTesting
Tested: Windows 11, RTX 4070 SUPER, NVIDIA 617.42: the ten tours above (six on this change's build, four before it), 36 to 38 area loads each, no error or device loss in any log and no decode left to the lifted code with the switches on; the six areas' shots by eye.
tests/test_native_zstd.py --cxx clang++passes with this change, and withhost/hos_zstd.cput back to main's its new cases fail ("the switch unset: a call was not done on the host").python -I -B tests/run_all.pywith clang here, this PR @ea7ed5b5: 393 passed, 107 failed, 221 skipped; main @544229a2in the same session: 393 passed, 107 failed, 221 skipped, the same 107 (this PC's toolchain: no gcc, Windows line endings, fixtures needing tools not installed).Not tested: Linux; a CPU with few cores, where giving the core up or keeping it may matter (here it does not); a played session; the two switches apart; saves written and read back across the two decoders (they decode, they do not write; nothing was compared byte for byte beyond the unit test's outputs); a game file from a mod.
Must be tested:
HOS_NATIVE_ZSTD=0 HOS_NATIVE_ZSTD_FRAMES=0, expecting no crash, the same picture after each load, and loads no longer than with the switches at 0 (Test Farm: @Beanstalk linux, load times with and without the two switches at 0)Risks
What could break: the two decoders are not the same code.
host/hos_zstd.clists where they differ: the bytes left just past the decoded output, and frames the format forbids, which libzstd 1.5.7 refuses and the game's older copy decodes (a block larger than the frame's window; a zstd compressor makes neither). A game file or a mod's file of that kind would fail to load where it loaded before. With the default the decoding thread also gives its virtual core up while it decodes, which changes which of the game's threads run when during a load; #155's own measurement had four frames over 100 ms in the first of four runs with=1, which these tours do not show (53 and 52 very long frames against 60 and 56), but that was another scene.Turning it off:
HOS_NATIVE_ZSTD=0andHOS_NATIVE_ZSTD_FRAMES=0.Compatibility:
HOS_NATIVE_ZSTD=1,=2andHOS_NATIVE_ZSTD_FRAMES=1mean what they meant. Someone who had neither set gets the host's decoder where they had the lifted one.Issues
None found for this (searched
zstd,decompress,loading time,loading screen). #155 added the whole-buffer path and the measurements this builds on.Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each new measured run (format indocs/tested-configurations.md).tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
@Beanstalk linux and windows amd: a tour of teleports (the Market, Hyrule Field, Kakariko, Kokiri Forest), once with this PR's default and once with
HOS_NATIVE_ZSTD=0 HOS_NATIVE_ZSTD_FRAMES=0(the game's own decoder as lifted code, as on main). Expecting: no crash; the same picture after each load on both; load times with the default no longer than with the two switches at 0 (here, on a 9950X3D on Windows: about a tenth shorter); and the log'szstd: the game's stream decode on the host's libzstdandzstd: the game's whole-buffer decode on the host's libzstdlines with the default, with nozstd: a decode runs liftedline after them.🤖 Generated with Claude Code
@GummiNaomi the PR got a new push (
ea7ed5b) after your comment. Review it and approve again.WIP: perf(host): the game's zstd decodes run on the host's libzstd by default (HOS_NATIVE_ZSTD=0: as lifted code)to perf(host): the game's zstd decodes run on the host's libzstd by default (HOS_NATIVE_ZSTD=0: as lifted code)