WIP: fix(mods): raise the size table entries of the files inside the packs mods serve #942
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!942
Loading…
Reference in a new issue
No description provided.
Delete branch "snaildev99/Longshot:mods/pack-members"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: low · Tested on: Windows 11, RTX 3090 (in the game, and the game-free checks) · Needs: Linux, any GPU: one game-free test
Why
The game loads each file inside an actor pack (
Pack/Actor/*.pack.zs) as a resource of its own, with memory sizedby that file's own entry in the resource size table, and the game's entries leave no room to spare. #134's merge
sized loose files only. So a mod that changes a file inside a pack and ships no size table (or one made for an older
version of the mod) loads that file, runs out of room for what the game builds from it, and the game sits at a
loading screen for ever. Character Select hit exactly this once (the adult player's animation parameters, fixed by
hand in its own table since). Depends on #134.
How to reproduce
Seen on: Windows, an emulator and #134's merge alike (the hang is the game's, not the host's)
Settings and switches: defaults
Steps: a mod that adds a few fields to a member of
Pack/Actor/Player_Adult.pack.zs(e.g. the adult'sAnimationParam) and ships no size table; play as adult Link through a load
Expected: the area loads
Actual: the loading screen never ends
What changed
At each start with mods, every pack a mod serves is compared with the game's copy, and the entry of each file in it
that grew, or that the game's pack does not have, is raised before the merged table is written.
restbl_sarc_eachandrestbl_pack_raiseinhost/restbl.c(after valentderah/totk-pc-recompruntime/restbl.c, MIT, credited in the header): grown members byrestbl_grow, new ones byrestbl_estimate.pack_members()inhost/mods.c, called fromfit_sizes()for every served.pack.zs; packs are read withhost/zs.c, which already decodes frames made with the game's dictionaries. An unreadable pack is named in thelog; the summary line counts members and packs.
restbl_growgives a grown BYML parameter file (.bgyml,.byml) at least 8 entry bytes for each byte it grew, plus 4 KiB: their list elements cost far more in the heap than in the file (fits over the game's own 5,772 files of two such types are in the commit message). In proportion, as for other types, was not enough in the game (Testing). This holds for loose files too; it only ever raises.docs/mods.md.Measurements
Verdict: a few packs add 10 to 15 ms to the start (10 ms of 278 in the game); 200 packs about 40 ms. No entry is raised for packs as the
game's.
Hardware: Windows 11, the PC the checks ran on
Scene: none (a host-side step at the start)
Method: a local harness (not in this PR) running
restbl_pack_raisewithhost/zs.con real packs against the game'scopies and table, 3 runs after a warm-up, dictionaries loaded once before; the range
How to reproduce: build
host/restbl.c,host/zs.cand zstd's decoder with a small main that reads each mod pack andthe game's copy and calls
restbl_pack_raisefb2a419cc4399147Flags and settings
None
Testing
Tested:
phive__ControllerSetParamgot 125,664 where 365,136 is known to load. With the BYML rule the same start raises it to 365,152 (the cloth parameters to 851,072), the adult save loads, and animations and cloth are normal in the field and in the menu.tests/test_mods_packs.py(new):restbl_pack_raiseon made-up packs (grown,equal, smaller, streamed,
.zsand new members; a pack the game lacks; a larger entry kept; unreadable archives), andhost/mods.cover a made-up game folder whose packs use a dictionary like the game's (trained by the test). It failson the base and passes here.
tests/run_all.py: no new failures against the base.Not tested: the test's Linux half (host/posix, as
tests/test_mods_size_table.py).Must be tested:
python -I -B tests/test_mods_packs.py(with the zstandard package), expectingPASS: the members of the packs mods serve(Test Farm: @Beanstalk linux, run tests/test_mods_packs.py)Risks
What could break: a start with a mod that serves packs takes a little longer (above); a pack that cannot be read is
left as before and named in the log. A grown BYML file's heap is about twice what it needs (memory, a few hundred KiB for a large mod). A BYML file whose new elements are much smaller in the file than as objects could still need more than 8 a byte: such a mod's own table still wins when it is larger. A file new to a pack gets an estimate even when the game lists its path (its own copy is in another actor's pack, which is not read): larger than the game's entry, never smaller.
Turning it off: revert the commit.
Compatibility: Nothing changes for mods without packs;
modcache/rsizetable.zsis written again at the next start.Checklist
Everyone
type(scope): description) of the status line's type.main. (on #134's head)python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.🤖 Generated with Claude Code
Tested in game: Windows 11, RTX 3090 (driver 610.88), a build of #134's head
4f8b975dwith this branch merged in, a character mod with 5 actor packs whose own size table had nine pack members' entries put back to the game's: the start's log raises 21 entries for the members of 5 packs (the adult player's ControllerSetParam 34160 -> 365152), the adult save loads, animations and cloth are normal; the merge took 278 ms, 10 of it the packs. The first cut (growth in proportion) crashed at that load, which is why BYML files grow by 8 a byte now. This branch is stacked on #134 (it depends on it), so the diff shows #134's commits too until that merges.The measurements table's columns name their commits now (#134 @
5192b46f, this PR @8b3e555c; the numbers were taken on the heads before the rebase onto it, which changed nothing in these files).8b3e555cbe1b90cf203b1b90cf203b2e2f9fb5a6Moved onto #134's new head
fafc7ab2(it was rebased onto main @343a1dfa): head2e2f9fb5. No conflict. One line more in tests/test_mods_packs.py: main's host/mods.c asks texts.c for a path now (#604), so the test's fixture has a stand-in for hos_texts_romfs beside the one for the prompts. The test passes; not built or played at this head; the measurements were taken on the heads before the move, which changed nothing in these files.Moved onto #134's new head
fb2a419c(rebased onto main @e4d035e9; the Mods tab's functions are API version 25 there, after main's audio outputs): headc4399147. No conflict, no change in this branch's own files; its tests pass at this head. Not built or played at it.2e2f9fb5a6c4399147f6c4399147f66e5e80128aMoved onto #134's new head
3fbc327a(the list of mods ismods.jsonthere now: #134 (comment)). Head6e5e8012.No conflict; the commits are unchanged. One commit on top:
tests/test_mods_packs.pybuildshost/mods_list.cand has a stand-in forhos_options_file(and forSleepin the Linux fixture).tests/run_all.pyon Windows: nothing fails that #134's head passes. Not built or played at this head; the Linux halves of the changed fixtures were read, not run.6e5e80128a28067b8fa1Moved onto #134's new head
2b52cd1b(#134 was rebased onto main @8a4aa282: the Mods tab's functions are API version 26, and the patches' hash is named in the lift's key after main's #22: #134 (comment)). Head28067b8f.No conflict; the commits are unchanged.
tests/run_all.pyon Windows: nothing fails that #134's head passes. Not built or played at this head.28067b8fa17fb78c60e5Moved onto #134's new head
c5cead8d(#134 was rebased onto main @ac778751and brings main's new players' mods guide in line with it: #134 (comment)). Head7fb78c60.No conflict; the commits are unchanged.
Nothing fails in
tests/run_all.pyon Windows that #134's head passes. Not built or played at this head.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.