WIP: feat(mods): an archive several mods have is served as one, each file inside it from the mod that changed it #1089
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1089
Loading…
Reference in a new issue
No description provided.
Delete branch "snaildev99/Longshot:mods/archive-merge"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Windows 11, RTX 3090 (in the game, and the game-free checks) · Needs: Linux, any GPU: the mods checks
Why
A mod replaces whole files, and many of the game's files are archives of hundreds of small files: the actor packs,
the menu layouts, the texts. A mod that changes one of those small files has to carry the whole archive. So two mods
that changed DIFFERENT files inside the same archive clash: the lower mod's whole archive wins and the other mod's
change is lost, though the two never touched the same thing. A character mod and a button prompt mod that both carry
the menu layout archive are the usual case; players use outside merge tools for it today.
See #134 (code mod support, the Mods tab), #942 (the size table entries of the files inside packs).
What changed
At the start, when several mods that are on have the same archive, Longshot now makes ONE archive of them, each file
inside it from the mod that changed it, and serves that. Where two mods changed the same file inside it the mod
lower in the list wins that file, as for whole files, and the log and the Mods tab say so. The archive is kept in
modcache/archivesand made again only when one of the copies, their order or the game's own copy changes. The mods'own files are never written.
restbl_sarc_mergeinhost/restbl.c(declared inrestbl.h, beside the SARC reader #942 added). Each copyis compared with the game's, member by member; a member the lowest mod leaves out stays out, one an upper mod
leaves out stays in; a name an archive holds several times (the layout archives: four) is one member; a member
that is an archive is taken whole; each member keeps at least the alignment its own archive gave it; the result
is never over 256 MiB nor larger than the archives it is made of together.
merge_archives/merge_oneinhost/mods.c: the paths several mods have (putkeeps the upper mods'paths), the cache and its record, the log lines, the Mods tab's counts (
mod_text).fit_sizes,hos_mods_romfsandhos_mods_fileuse the merged archive, so the size table has its own entry and, for a pack,its members' (
restbl_pack_raise).hos_zs_headinhost/zs.c: the first bytes a.zsfile decompresses to, to tell an archive withoutdecompressing it all.
that cannot be read (named in the log), an archive that would hold nothing of the upper mods, one that would be too
large.
whole or not at all. A member's name reaches the log and the record without control characters.
tests/test_mods_archives.py(new);tests/test_mods_packs.py(two mods on one pack now expect the mergedpack's members).
docs/mods.md,docs/development.md.Measurements
Verdict: a start with two mods on the same eight archives takes about 130 ms longer the first time (566 against
434 ms in a harness) and 8 ms longer after that (442 against 434); a start with one mod, or with mods that share no
archive, does nothing new. In the game the times will be longer than a harness's: the log line has them.
Hardware: HARDWARE (the main session fills this in)
Scene: none: a harness that runs
hos_mods_initover the game's RomFS and two mod folders, no gameMethod:
host/mods.c,host/zs.c,host/restbl.cbuilt natively with amainthat timeshos_mods_init; acharacter mod (6,905 files, 8 of them archives the game has) and a second mod with the same 8 archives made from the
game's copies; 5 interleaved runs each after a warm-up, the median
How to reproduce: not reproducible without the game's files; the game-free test covers the behaviour
hos_mods_init, a character mod and a second mod with the same 8 archivesc4399147e4134107e4134107Size: a merged archive is a little larger than its copies, since each member keeps the alignment its own archive gave
it. These eight (large) archives: +0.3 to +3.3% over the larger copy. Over the game's archives in general (868 of them
merged from two changed copies each): a pack +2.5% (median; a quarter of them over 5%, the most +89%, small packs),
a text archive +0.1%, a layout archive +0.4%. They are kept uncompressed: 21.8 MB on disk for these eight where the
mods' compressed copies take 2.3 MB.
Flags and settings
None. (No switch: an archive that cannot be merged falls back to the lower mod's whole file by itself. Deleting
modcache/archivesonly makes the archives again.)Testing
Tested:
mods: archives several mods have: 7 served as one archive of the mods' copies (0 as made before), 1 as the lowest mod's own file, 213 ms, a line an archive (Pack/Actor/Player_Adult.pack.zs: 2 mods' copies served as one archive ... (12 of <the first mod>, 1 of <the second>); 512 files in it), and the game reading them frommodcache/archives. The Mods tab says for both mods that their archives are served as one. So the game loads an actor pack, a layout archive and a text archive written as a frame of raw blocks.tests/test_mods_archives.py(the merge alone on made-up archives: different members, thesame member, the same change, members added and left out, three copies, names held four times, a nested archive,
nothing to make, eleven unreadable archives, too many members, members that share bytes, the size limit, alignment
kept up to 0x2000, a small archive staying small; and
host/mods.cover folders: what is served, the log and the Modstab's lines, the size table, the cache used and made again (a changed mod file, a changed game file, an archive cut
or damaged in place), a mod switched off, the order changed, a member name with line breaks, an archive that cannot
be written, no mod or game file written). 47 mutations of the source in all: the test fails on 43; the four it
passes are a check done twice (two), a bound that later checks stand in for, and one that needs an archive over 256
MiB.
tests/run_all.py: 562 passed, 22 failed, 85 skipped;the base (#942's head): 561 passed, 22 failed, 85 skipped; the same 22 fail on both (test_auto_handheld, test_bug_report, test_bug_report_defaults,
test_crash_dialog, test_crt_options, test_fastfp_fma, test_fpx_sticky, test_guest_imported, test_hardware_report,
test_linux_preflight, test_make_game_cache, test_missing_nup, test_pads_after_menu, test_pending_guest_import,
test_portable_builder_home, test_portable_builder_startup, test_release, test_run_all_changed, test_spawn_emit,
test_tested_configurations, test_vk_occlusion_queries, test_walk_key). Over the real game's archives (harness): eight
merged archives read back and compared member by member, in both orders of the two mods; and 868 of the game's own
archives merged from two changed copies each, compared member by member. An independent review ran the merge against
a model of its rule (8,000 random cases) and under AddressSanitizer (1.2 million damaged archives); its findings are
fixed in the second commit.
Not tested: a second start in the game (the archives served from the cache). On Linux: the folder halves of
the mods checks (the fixtures were only read). A second mod made by someone else.
Must be tested:
python -I -B tests/test_mods_archives.py,tests/test_mods_packs.py,tests/test_mods_sizes.py,tests/test_mods_size_table.py,tests/test_fs_links.py, expecting PASS (Test Farm: @Beanstalk linux, the modschecks)
Risks
What could break: a game with two or more mods that carry the same archive now gets an archive Longshot built
(other padding, the data in hash order, a frame of raw blocks) where it got one mod's own file. If the game does not
take a pack in that form, those mods' actors fail to load. A game with one mod, or with mods that share no archive,
is served exactly what it was. A mod's copy made from ANOTHER game version's archive counts every member that
version differs in as its change (as it does when used alone), where before a lower mod's whole file hid them. Memory
at a first start: every copy of one archive is held decompressed at once (five mods on one 24 MiB archive: about 160
MiB for a moment); from the cache only the files' bytes are read.
Turning it off: revert the commit; or switch one of the two mods off on the Mods tab.
Compatibility: a new folder
modcache/archivesin the game folder (the merged archives, uncompressed: as largeas the archives themselves). Nothing else changes.
Checklist
Everyone
type(scope): description) of the status line's type.main. (stacked on #942)python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.Linux and the game's own build: not done)
🤖 Generated with Claude Code
Tested in game: Windows 11, RTX 3090 (driver 610.88), a build of this branch with a character mod and, below it in the list, a second mod carrying the same eight archives (four actor packs, the layout archive, two text archives) made from the game's own with one text reworded: the title screen shows the second mod's text and the first mod's menu page, layouts, texts and cloth settings all work; the log says 7 archives were served as one in 213 ms and the game reads them from modcache/archives; the Mods tab says so for both mods. Stacked on #942 and through it on #134, so the diff shows their commits too; the last two commits are this change.
e4134107cfea13f08005Moved onto #134's new head
3fbc327a(the list of mods ismods.jsonthere now: #134 (comment)). Headea13f080.No conflict; the commits are unchanged. It sits on #942's new head. One commit on top:
tests/test_mods_archives.pybuildshost/mods_list.c, has the stand-ins, and writesmods.jsonwhere it switched a mod off and changed the order between two starts (the old file is read once only); #134'stests/test_mods_list.pygets stand-ins for the two zstd functions the archive merge calls.tests/run_all.pyon Windows: nothing fails that #134's head passes. Not built or played at this head; the Linux halves of the changed fixtures were read, not run.ea13f08005ae8b4670daMoved onto #134's new head
2b52cd1b(#134 was rebased onto main @8a4aa282: the Mods tab's functions are API version 26, and the patches' hash is named in the lift's key after main's #22: #134 (comment)). Headae8b4670.No conflict; the commits are unchanged. It sits on #942's new head.
tests/run_all.pyon Windows: nothing fails that #134's head passes. Not built or played at this head.ae8b4670da73818d8e69Moved onto #134's new head
c5cead8d(#134 was rebased onto main @ac778751and brings main's new players' mods guide in line with it: #134 (comment)). Head73818d8e.No conflict. One commit on top: the players' guide (
docs/using-mods.md, main's #1210) gets a paragraph under "When two mods change the same file": an archive two mods both change is served as one, and only a file inside it that both changed goes to the lower mod.Nothing fails in
tests/run_all.pyon Windows that #134's head passes. Not built or played at this head.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.