WIP: feat(mods): an archive several mods have is served as one, each file inside it from the mod that changed it #1089

Draft
snaildev99 wants to merge 23 commits from snaildev99/Longshot:mods/archive-merge into main
Contributor

Type: feat · Risk: medium · Tested on: Windows 11, RTX 3090 (in the game, and the game-free checks) · Needs: Linux, any GPU: the mods checks

Why

A mod replaces whole files, and many of the game's files are archives of hundreds of small files: the actor packs,
the menu layouts, the texts. A mod that changes one of those small files has to carry the whole archive. So two mods
that changed DIFFERENT files inside the same archive clash: the lower mod's whole archive wins and the other mod's
change is lost, though the two never touched the same thing. A character mod and a button prompt mod that both carry
the menu layout archive are the usual case; players use outside merge tools for it today.

See #134 (code mod support, the Mods tab), #942 (the size table entries of the files inside packs).

What changed

At the start, when several mods that are on have the same archive, Longshot now makes ONE archive of them, each file
inside it from the mod that changed it, and serves that. Where two mods changed the same file inside it the mod
lower in the list wins that file, as for whole files, and the log and the Mods tab say so. The archive is kept in
modcache/archives and made again only when one of the copies, their order or the game's own copy changes. The mods'
own files are never written.

  • merge: restbl_sarc_merge in host/restbl.c (declared in restbl.h, beside the SARC reader #942 added). Each copy
    is compared with the game's, member by member; a member the lowest mod leaves out stays out, one an upper mod
    leaves out stays in; a name an archive holds several times (the layout archives: four) is one member; a member
    that is an archive is taken whole; each member keeps at least the alignment its own archive gave it; the result
    is never over 256 MiB nor larger than the archives it is made of together.
  • loader: merge_archives / merge_one in host/mods.c: the paths several mods have (put keeps the upper mods'
    paths), the cache and its record, the log lines, the Mods tab's counts (mod_text). fit_sizes,
    hos_mods_romfs and hos_mods_file use the merged archive, so the size table has its own entry and, for a pack,
    its members' (restbl_pack_raise).
  • zs: hos_zs_head in host/zs.c: the first bytes a .zs file decompresses to, to tell an archive without
    decompressing it all.
  • kept as before (the lower mod's whole file): a file that is no archive, an archive the game does not have, a copy
    that cannot be read (named in the log), an archive that would hold nothing of the upper mods, one that would be too
    large.
  • the cache: an archive made before is served only with the size and the hash its record gives; the record is written
    whole or not at all. A member's name reaches the log and the record without control characters.
  • tests: tests/test_mods_archives.py (new); tests/test_mods_packs.py (two mods on one pack now expect the merged
    pack's members).
  • docs: README (Mods), docs/mods.md, docs/development.md.

Measurements

Verdict: a start with two mods on the same eight archives takes about 130 ms longer the first time (566 against
434 ms in a harness) and 8 ms longer after that (442 against 434); a start with one mod, or with mods that share no
archive, does nothing new. In the game the times will be longer than a harness's: the log line has them.

Hardware: HARDWARE (the main session fills this in)
Scene: none: a harness that runs hos_mods_init over the game's RomFS and two mod folders, no game
Method: host/mods.c, host/zs.c, host/restbl.c built natively with a main that times hos_mods_init; a
character mod (6,905 files, 8 of them archives the game has) and a second mod with the same 8 archives made from the
game's copies; 5 interleaved runs each after a warm-up, the median
How to reproduce: not reproducible without the game's files; the game-free test covers the behaviour

hos_mods_init, a character mod and a second mod with the same 8 archives Before: #942 @ c4399147 After, the archives made before: this PR @ e4134107 After, first start: this PR @ e4134107
The whole of it, median of 5 434 ms 442 ms 566 ms
The archives' part (its log line) none 19 ms 137 ms
What the game is served for the 8 archives the lower mod's whole files: the upper mod's changes are lost 7 archives made of both mods' copies, 1 the lower mod's own (nothing to merge in it) the same
base:          hos_mods_init [418, 430, 434, 439, 451] ms, median 434
branch cached: hos_mods_init [442, 442, 442, 463, 462] ms, median 442; the archives' part [25, 18, 19, 19, 19]
branch first:  hos_mods_init [541, 568, 566, 556, 642] ms, median 566; the archives' part [127, 140, 130, 137, 160]
mods: archives several mods have: 7 served as one archive of the mods' copies (7 as made before), 1 as the lowest mod's own file (0 of them: a copy cannot be read), 19 ms

Size: a merged archive is a little larger than its copies, since each member keeps the alignment its own archive gave
it. These eight (large) archives: +0.3 to +3.3% over the larger copy. Over the game's archives in general (868 of them
merged from two changed copies each): a pack +2.5% (median; a quarter of them over 5%, the most +89%, small packs),
a text archive +0.1%, a layout archive +0.4%. They are kept uncompressed: 21.8 MB on disk for these eight where the
mods' compressed copies take 2.3 MB.

Flags and settings

None. (No switch: an archive that cannot be merged falls back to the lower mod's whole file by itself. Deleting
modcache/archives only makes the archives again.)

Testing

Tested:

  • In the game, Windows 11, RTX 3090 (driver 610.88), a build of this branch: a character mod (6,903 files; it changes files inside four actor packs, the layout archive and two text archives) and below it a second mod with the same eight archives made from the game's own copies, one text reworded. Without the merge the second mod's copies would be served whole and the first mod's changes to them lost. With it: the title screen has the second mod's text, and the first mod's menu page, its layouts, its texts and its cloth settings in the player's pack all work (a character picked, its dress moving as cloth). The log: mods: archives several mods have: 7 served as one archive of the mods' copies (0 as made before), 1 as the lowest mod's own file, 213 ms, a line an archive (Pack/Actor/Player_Adult.pack.zs: 2 mods' copies served as one archive ... (12 of <the first mod>, 1 of <the second>); 512 files in it), and the game reading them from modcache/archives. The Mods tab says for both mods that their archives are served as one. So the game loads an actor pack, a layout archive and a text archive written as a frame of raw blocks.
  • Windows 11: tests/test_mods_archives.py (the merge alone on made-up archives: different members, the
    same member, the same change, members added and left out, three copies, names held four times, a nested archive,
    nothing to make, eleven unreadable archives, too many members, members that share bytes, the size limit, alignment
    kept up to 0x2000, a small archive staying small; and host/mods.c over folders: what is served, the log and the Mods
    tab's lines, the size table, the cache used and made again (a changed mod file, a changed game file, an archive cut
    or damaged in place), a mod switched off, the order changed, a member name with line breaks, an archive that cannot
    be written, no mod or game file written). 47 mutations of the source in all: the test fails on 43; the four it
    passes are a check done twice (two), a bound that later checks stand in for, and one that needs an archive over 256
    MiB. tests/run_all.py: 562 passed, 22 failed, 85 skipped;
    the base (#942's head): 561 passed, 22 failed, 85 skipped; the same 22 fail on both (test_auto_handheld, test_bug_report, test_bug_report_defaults,
    test_crash_dialog, test_crt_options, test_fastfp_fma, test_fpx_sticky, test_guest_imported, test_hardware_report,
    test_linux_preflight, test_make_game_cache, test_missing_nup, test_pads_after_menu, test_pending_guest_import,
    test_portable_builder_home, test_portable_builder_startup, test_release, test_run_all_changed, test_spawn_emit,
    test_tested_configurations, test_vk_occlusion_queries, test_walk_key). Over the real game's archives (harness): eight
    merged archives read back and compared member by member, in both orders of the two mods; and 868 of the game's own
    archives merged from two changed copies each, compared member by member. An independent review ran the merge against
    a model of its rule (8,000 random cases) and under AddressSanitizer (1.2 million damaged archives); its findings are
    fixed in the second commit.

Not tested: a second start in the game (the archives served from the cache). On Linux: the folder halves of
the mods checks (the fixtures were only read). A second mod made by someone else.

Must be tested:

  • Linux, any GPU: python -I -B tests/test_mods_archives.py, tests/test_mods_packs.py, tests/test_mods_sizes.py,
    tests/test_mods_size_table.py, tests/test_fs_links.py, expecting PASS (Test Farm: @Beanstalk linux, the mods
    checks)

Risks

What could break: a game with two or more mods that carry the same archive now gets an archive Longshot built
(other padding, the data in hash order, a frame of raw blocks) where it got one mod's own file. If the game does not
take a pack in that form, those mods' actors fail to load. A game with one mod, or with mods that share no archive,
is served exactly what it was. A mod's copy made from ANOTHER game version's archive counts every member that
version differs in as its change (as it does when used alone), where before a lower mod's whole file hid them. Memory
at a first start: every copy of one archive is held decompressed at once (five mods on one 24 MiB archive: about 160
MiB for a moment); from the cache only the files' bytes are read.

Turning it off: revert the commit; or switch one of the two mods off on the Mods tab.

Compatibility: a new folder modcache/archives in the game folder (the merged archives, uncompressed: as large
as the archives themselves). Nothing else changes.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched). (the main session: search)
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main. (stacked on #942)
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
  • A game-free test added for the fix, failing on main and passing here.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows: the checks build it;
    Linux and the game's own build: not done)
  • Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** medium · **Tested on:** Windows 11, RTX 3090 (in the game, and the game-free checks) · **Needs:** Linux, any GPU: the mods checks ## Why A mod replaces whole files, and many of the game's files are archives of hundreds of small files: the actor packs, the menu layouts, the texts. A mod that changes one of those small files has to carry the whole archive. So two mods that changed DIFFERENT files inside the same archive clash: the lower mod's whole archive wins and the other mod's change is lost, though the two never touched the same thing. A character mod and a button prompt mod that both carry the menu layout archive are the usual case; players use outside merge tools for it today. See #134 (code mod support, the Mods tab), #942 (the size table entries of the files inside packs). ## What changed At the start, when several mods that are on have the same archive, Longshot now makes ONE archive of them, each file inside it from the mod that changed it, and serves that. Where two mods changed the same file inside it the mod lower in the list wins that file, as for whole files, and the log and the Mods tab say so. The archive is kept in `modcache/archives` and made again only when one of the copies, their order or the game's own copy changes. The mods' own files are never written. - merge: `restbl_sarc_merge` in `host/restbl.c` (declared in `restbl.h`, beside the SARC reader #942 added). Each copy is compared with the game's, member by member; a member the lowest mod leaves out stays out, one an upper mod leaves out stays in; a name an archive holds several times (the layout archives: four) is one member; a member that is an archive is taken whole; each member keeps at least the alignment its own archive gave it; the result is never over 256 MiB nor larger than the archives it is made of together. - loader: `merge_archives` / `merge_one` in `host/mods.c`: the paths several mods have (`put` keeps the upper mods' paths), the cache and its record, the log lines, the Mods tab's counts (`mod_text`). `fit_sizes`, `hos_mods_romfs` and `hos_mods_file` use the merged archive, so the size table has its own entry and, for a pack, its members' (`restbl_pack_raise`). - zs: `hos_zs_head` in `host/zs.c`: the first bytes a `.zs` file decompresses to, to tell an archive without decompressing it all. - kept as before (the lower mod's whole file): a file that is no archive, an archive the game does not have, a copy that cannot be read (named in the log), an archive that would hold nothing of the upper mods, one that would be too large. - the cache: an archive made before is served only with the size and the hash its record gives; the record is written whole or not at all. A member's name reaches the log and the record without control characters. - tests: `tests/test_mods_archives.py` (new); `tests/test_mods_packs.py` (two mods on one pack now expect the merged pack's members). - docs: README (Mods), `docs/mods.md`, `docs/development.md`. ## Measurements **Verdict:** a start with two mods on the same eight archives takes about 130 ms longer the first time (566 against 434 ms in a harness) and 8 ms longer after that (442 against 434); a start with one mod, or with mods that share no archive, does nothing new. In the game the times will be longer than a harness's: the log line has them. Hardware: HARDWARE (the main session fills this in) Scene: none: a harness that runs `hos_mods_init` over the game's RomFS and two mod folders, no game Method: `host/mods.c`, `host/zs.c`, `host/restbl.c` built natively with a `main` that times `hos_mods_init`; a character mod (6,905 files, 8 of them archives the game has) and a second mod with the same 8 archives made from the game's copies; 5 interleaved runs each after a warm-up, the median How to reproduce: not reproducible without the game's files; the game-free test covers the behaviour | `hos_mods_init`, a character mod and a second mod with the same 8 archives | Before: #942 @ c4399147 | After, the archives made before: this PR @ e4134107 | After, first start: this PR @ e4134107 | |---|---|---|---| | The whole of it, median of 5 | 434 ms | 442 ms | 566 ms | | The archives' part (its log line) | none | 19 ms | 137 ms | | What the game is served for the 8 archives | the lower mod's whole files: the upper mod's changes are lost | 7 archives made of both mods' copies, 1 the lower mod's own (nothing to merge in it) | the same | ``` base: hos_mods_init [418, 430, 434, 439, 451] ms, median 434 branch cached: hos_mods_init [442, 442, 442, 463, 462] ms, median 442; the archives' part [25, 18, 19, 19, 19] branch first: hos_mods_init [541, 568, 566, 556, 642] ms, median 566; the archives' part [127, 140, 130, 137, 160] mods: archives several mods have: 7 served as one archive of the mods' copies (7 as made before), 1 as the lowest mod's own file (0 of them: a copy cannot be read), 19 ms ``` Size: a merged archive is a little larger than its copies, since each member keeps the alignment its own archive gave it. These eight (large) archives: +0.3 to +3.3% over the larger copy. Over the game's archives in general (868 of them merged from two changed copies each): a pack +2.5% (median; a quarter of them over 5%, the most +89%, small packs), a text archive +0.1%, a layout archive +0.4%. They are kept uncompressed: 21.8 MB on disk for these eight where the mods' compressed copies take 2.3 MB. ### Flags and settings None. (No switch: an archive that cannot be merged falls back to the lower mod's whole file by itself. Deleting `modcache/archives` only makes the archives again.) ## Testing **Tested:** - In the game, Windows 11, RTX 3090 (driver 610.88), a build of this branch: a character mod (6,903 files; it changes files inside four actor packs, the layout archive and two text archives) and below it a second mod with the same eight archives made from the game's own copies, one text reworded. Without the merge the second mod's copies would be served whole and the first mod's changes to them lost. With it: the title screen has the second mod's text, and the first mod's menu page, its layouts, its texts and its cloth settings in the player's pack all work (a character picked, its dress moving as cloth). The log: `mods: archives several mods have: 7 served as one archive of the mods' copies (0 as made before), 1 as the lowest mod's own file, 213 ms`, a line an archive (`Pack/Actor/Player_Adult.pack.zs: 2 mods' copies served as one archive ... (12 of <the first mod>, 1 of <the second>); 512 files in it`), and the game reading them from `modcache/archives`. The Mods tab says for both mods that their archives are served as one. So the game loads an actor pack, a layout archive and a text archive written as a frame of raw blocks. - Windows 11: `tests/test_mods_archives.py` (the merge alone on made-up archives: different members, the same member, the same change, members added and left out, three copies, names held four times, a nested archive, nothing to make, eleven unreadable archives, too many members, members that share bytes, the size limit, alignment kept up to 0x2000, a small archive staying small; and `host/mods.c` over folders: what is served, the log and the Mods tab's lines, the size table, the cache used and made again (a changed mod file, a changed game file, an archive cut or damaged in place), a mod switched off, the order changed, a member name with line breaks, an archive that cannot be written, no mod or game file written). 47 mutations of the source in all: the test fails on 43; the four it passes are a check done twice (two), a bound that later checks stand in for, and one that needs an archive over 256 MiB. `tests/run_all.py`: 562 passed, 22 failed, 85 skipped; the base (#942's head): 561 passed, 22 failed, 85 skipped; the same 22 fail on both (test_auto_handheld, test_bug_report, test_bug_report_defaults, test_crash_dialog, test_crt_options, test_fastfp_fma, test_fpx_sticky, test_guest_imported, test_hardware_report, test_linux_preflight, test_make_game_cache, test_missing_nup, test_pads_after_menu, test_pending_guest_import, test_portable_builder_home, test_portable_builder_startup, test_release, test_run_all_changed, test_spawn_emit, test_tested_configurations, test_vk_occlusion_queries, test_walk_key). Over the real game's archives (harness): eight merged archives read back and compared member by member, in both orders of the two mods; and 868 of the game's own archives merged from two changed copies each, compared member by member. An independent review ran the merge against a model of its rule (8,000 random cases) and under AddressSanitizer (1.2 million damaged archives); its findings are fixed in the second commit. **Not tested:** a second start in the game (the archives served from the cache). On Linux: the folder halves of the mods checks (the fixtures were only read). A second mod made by someone else. **Must be tested:** - Linux, any GPU: `python -I -B tests/test_mods_archives.py`, `tests/test_mods_packs.py`, `tests/test_mods_sizes.py`, `tests/test_mods_size_table.py`, `tests/test_fs_links.py`, expecting PASS (Test Farm: @Beanstalk linux, the mods checks) ## Risks **What could break:** a game with two or more mods that carry the same archive now gets an archive Longshot built (other padding, the data in hash order, a frame of raw blocks) where it got one mod's own file. If the game does not take a pack in that form, those mods' actors fail to load. A game with one mod, or with mods that share no archive, is served exactly what it was. A mod's copy made from ANOTHER game version's archive counts every member that version differs in as its change (as it does when used alone), where before a lower mod's whole file hid them. Memory at a first start: every copy of one archive is held decompressed at once (five mods on one 24 MiB archive: about 160 MiB for a moment); from the cache only the files' bytes are read. **Turning it off:** revert the commit; or switch one of the two mods off on the Mods tab. **Compatibility:** a new folder `modcache/archives` in the game folder (the merged archives, uncompressed: as large as the archives themselves). Nothing else changes. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). (the main session: search) - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [ ] Rebased on current `main`. (stacked on #942) - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Numbers from the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] A game-free test added for the fix, failing on `main` and passing here. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows: the checks build it; Linux and the game's own build: not done) - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Mods that each grow files (HD texture packs) could not be combined: only one
resource size table was used. Hookshot now merges every enabled mod's table
into the game's at start (the button prompts' entries kept) and raises the
entry of every file a mod serves to fit its size (host/mods.c, host/restbl.c,
host/zs.c). #2270's merge of the prompts' entries into the winning mod's table
stays as the fallback when no merged table can be made.

The Insert menu gets a Mods tab (shown only when there are mods): each mod on
or off and its place in the list, kept in hookshot_mods.txt and applied at the
next start (the restart question names them), and the cheats of the enabled
mods, switched at once.

ExeFS patches (IPS, IPS32, IPSwitch .pchtxt) are applied to main's image when
the game is built (nx2/exefs_patches.py, nx2/image.py), so the code is
translated from the patched instructions; a write into the padding after the
code (a code cave) makes it code. exefs_patches.txt in the game folder lists
what a build applied; the Mods tab says when a patch waits for a build.

Atmosphere cheat-code files (separate from the Debug tab's built-in cheats)
run in a cheat VM after dmnt's (host/cheats.c), every presented frame, on
guest memory the game was given; cheats that write the game's code are
listed as not supported.

The list, order and switches, the size table's merge, the patch formats and
the cheat VM follow valentderah/totk-pc-recomp (MIT), whose notice is kept in
host/restbl.h and nx2/exefs_patches.py.

Co-authored-by: Valiantsin Dzerakh <[email protected]>
Co-authored-by: conongnang <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The control bridge's bridge_fill names each MenuTab and checks the count: with the Mods tab
added before Debug, its table needs "mods" there.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The compact bar (added to main after the Mods tab was written) draws the
selected row of the current tab. The Mods tab has no such rows: its lines come
from the host. With the bar asked for on the Mods tab the full page is drawn.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
- hos_mods_file gives prompts.c no table when the merged one is served.
- A mod that is off: its ExeFS files are not counted as waiting for a build
  (the Mods tab's text and the log's line).
- The Mods tab's texts and longshot_mods.txt's header say Longshot.
- A Restart or Quit waiting for its second press is cleared on the Mods tab.
- The merged size table: a mod file whose path is in another case than the
  game's takes the game's entry (looked up only when the mod's own case
  misses the table).
- exefs_patches.py: bytes into a relocated word are left out and the file's
  note says so; the comment on DT_RELR words says what happens.
- The compact bar on the Mods tab dims the game behind, as the full menu.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
Co-Authored-By: Claude Fable 5.2 <[email protected]>
The test cuts restart_values out of vk_menu.cpp; with the Mods tab that
function also fills the mods' slot, so the harness declares the slot (none
here), can_mods and the host's mods_changed beside its pad_raw.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
docs/mods.md (from #93) describes the loader as it was: ExeFS left out, one
mod's size table, no Mods tab. With this branch an exefs folder's patches
belong to the build, the size table is merged at a start, and the Mods tab's
longshot_mods.txt can change the order and the switches; without that file
the order is the documented one. The example's check for the old ExeFS log
line now looks for the line that lists the mod with its ExeFS folder.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
test(audio): the playback callbacks' check allows the Mods tab's functions after them
Some checks failed
PR description / Description follows the template (pull_request_target) Has been cancelled
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
fb2a419c54
main's check (#1001) asked for API version 24 exactly and for the six playback fields to be MkwVkHost's last. The Mods tab's
five functions follow them as version 25: the check asks for at least 24, for the six in their place after take_text, and
for nothing but the Mods tab's five after them.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
The game loads each file inside an actor pack (Pack/Actor/x.pack.zs, a SARC
archive) as a resource of its own, in a heap sized by that file's own entry
in the resource size table, and the stock entries fit to the byte. The merge
sized loose files only (restbl.c said the packs are sized as a whole, which
is not so for this game: nearly every member has its own entry). A mod that
grows a file inside a pack and ships no table, or a stale one, kept the
game's entry: the file loads, the heap runs out for what is parsed out of
it, a loading thread fails on a null object and the game waits at a loading
screen for ever.

restbl.c gets restbl_sarc_each and restbl_pack_raise, after
valentderah/totk-pc-recomp runtime/restbl.c (MIT, the notice is in
restbl.h): a member that grew against the game's copy of the pack is raised
by restbl_grow, a member the game's copy lacks gets restbl_estimate, the
others are left alone. host/mods.c's fit_sizes calls it (pack_members) for
every pack a mod serves, reading both packs with host/zs.c, which already
decodes frames made with the game's dictionaries. A pack that cannot be read
is named in the log and its members keep their entries; the summary line
counts the members and packs raised, and up to 16 lines name them.

Measured on this game's own files (a local harness, nothing committed): the
nine packs of a character mod that grows the player's parameter files take
15 ms and raise 21 entries, among them the two whose growth once hung a
loading screen; 200 stock packs served unchanged take about 40 ms and raise
none.

tests/test_mods_packs.py checks restbl_pack_raise on made-up packs (grown,
equal, smaller, streamed, .zs and new members, a new pack, an entry already
larger, unreadable archives) and host/mods.c over a made-up game folder whose
packs use a trained dictionary as the game's do. It fails on the base (no
restbl_pack_raise; with it, the grown member keeps the game's entry) and
passes here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
fix(mods): a grown BYML file's entry: at least 8 bytes a byte grown; the merge's time in parts
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 8s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
c4399147f6
A run in the game crashed with the previous commit: a character mod whose
own size table was set back to the game's entries for the files inside its
packs (the case this branch is for) loaded the adult player's pack and its
loading thread failed on a null object, the heap of the player's
ControllerSetParam run out. That file had grown from 15,256 to 56,116 bytes
by 555 cloth reactions and options; growth in proportion gave 125,664.

What a BYML parameter file adds is mostly list elements, which the game
builds into objects that cost far more than their bytes. Least squares over
the game's own files: ControllerSetParam (5,602 files) 6,619 + 1.04 a byte +
60 to 180 an element of each list, at most 319 off; ClothParam (170 files)
4,653 + 1.90 a byte + 289 a mesh + 31 a collision shape, at most 784 off. For
the crashed file the fit asks about 178,600 (3.5 a byte grown). restbl_grow
now gives a grown .bgyml or .byml file at least 8 entry bytes for each byte it
grew, plus 4 KiB: twice the fits' need for such growth, and the rule the
character mod's own builder uses, whose entries for these files load in the
game (365,136 for the crashed file; the new rule gives 365,152). Other types
(.ainb and the physics binaries, about 1 a byte in the game's table) keep
the proportion. This applies to loose BYML files that grow too; it only
ever raises an entry.

The merge's log line now gives its time in parts (the mods' tables, the
files, the packs among them, writing): a start with the 1 GB character mod
took 7 to 9 s in the game where a harness over the same folder takes 250 ms
(tables 53, files 195 of which packs 10), and the next run should say which
part the game's seconds are.

tests/test_mods_packs.py adds a member of the crashed file's shape (made-up
sizes) that must get at least 8 a byte grown plus 4 KiB and more than twice
the proportion, and an .ainb member that keeps the proportion; it fails with
the previous commit's restbl.c and passes here.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A mod replaces whole files, and many of the game's files are SARC archives
of hundreds of small files (the actor packs, the layout archives, the text
archives). A mod that changes one of those small files ships the whole
archive, so two mods that changed different files inside one archive
clashed: the lowest mod's whole file won and the other's change was lost.

At the start, for every path several enabled mods have that is a SARC
archive of the game's in a .zs file, host/mods.c merge_archives now makes
one archive of the copies and serves it for that path:

- restbl_sarc_merge (host/restbl.c, beside the SARC reader the pack members'
  entries use) compares each copy with the game's own, member by member. A
  member comes from the mod whose copy differs from the game's for it, or
  that adds it; the lowest such mod where several do, as for whole files.
  A member of the game's is left out only when the lowest mod leaves it
  out (what that mod's file alone would do); a mod above leaving one out
  changes nothing. Two mods with the same bytes for a member do not clash.
  A name an archive holds several times (the layout archives hold each
  layout four times) is taken as one member, a member that is an archive
  itself is taken whole.
- A SARC archive does not say how its members are aligned. Each member
  keeps the alignment its own archive gave it: the largest power of two its
  offset there is a multiple of, from the archive's start and from its
  data's start, between 8 and 8 KiB. That is never less than it had.
- The archive goes to modcache/archives in the game folder as a frame of
  raw blocks (hos_zs_write, as the merged size table and the button
  prompts' files), with a record of what it was made of: a hash of every
  copy's bytes in the list's order and of the game's. A start whose record
  matches reads the copies' compressed bytes only and serves the archive
  made before; a changed copy, another order, a mod switched off or a
  damaged archive make it again. Mod files are only read.
- The size table follows by itself: fit_sizes sizes what is served, so the
  merged archive's own entry and, for a pack, its members' entries
  (restbl_pack_raise) come from the merged archive.
- The log names each merged archive with what each mod gave, and each
  member two mods changed with the mod that won it. The Mods tab's line of
  a mod says how many of its archives are served as one with other mods'
  copies and how many files inside them a lower mod takes; such an archive
  no longer counts as a file taken whole.
- Kept as before (the lowest mod's whole file): a file that is no archive
  (found from the first bytes of the game's copy, hos_zs_head, without
  decompressing it all), an archive the game lacks, a copy or a game file
  that cannot be read as a little-endian SARC archive with named members
  and the game's hash key (the log names the copy), an archive that would
  hold nothing of the mods above, and one that cannot be written.

The button prompts and the PC texts meet a merged archive as they meet any
mod's file: a mod's layout or text archive wins over theirs.

tests/test_mods_archives.py: restbl_sarc_merge alone on made-up archives
(different members, the same member, the same change, members added and
left out, three copies, names held four times, a nested archive, nothing to
make, nine unreadable archives, too many members, alignment kept), and
host/mods.c over folders with dictionary-compressed archives (what is
served, the log and the Mods tab's lines, the size table, the cache used
and made again, a mod switched off, the order changed, no mod or game file
written). tests/test_mods_packs.py's two mods on one pack now expect the
merged pack: the member only the upper mod grew is served, so its entry is
raised.

Not run in the game. Measured over the real game's files with a harness
(not committed): eight archives of a character mod and of a made-up second
mod, 21.9 MB decompressed, are merged in 126 to 168 ms at a first start and
checked in 10 ms at the next ones; the merged archives are 0.3 to 4.5%
larger than the larger copy (padding).

Co-Authored-By: Claude Fable 5.2 <[email protected]>
fix(mods): the archive merge after review: a name with line breaks, a size bound, the cached archive checked by its hash, less padding
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 3s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
e4134107cf
An independent review of the previous commit found one crash, one way to
make Longshot write far more than it read, and a wrong number. Each fix has
a check in tests/test_mods_archives.py that fails on the previous commit.

- A member name is any bytes up to a NUL. The clashes were kept as text
  lines with the name in them and the log read the two copies' numbers
  back out of that text: a name with a line break made a line of its own,
  and the numbers on it indexed the copies' array (a read outside it; the
  game exited with an access violation at its first start with such a
  pack in two mods). The clashes are now an array of numbers and names,
  never parsed back; a name is kept without control characters and cut at
  160 bytes before it reaches the record or the log, so it cannot add a
  line to either; the record's numbers are range-checked where they are
  read and again where they are used.
- restbl_sarc_merge takes a limit and never returns an archive larger than
  it, nor larger than the game's archive and the copies together. A copy
  whose nodes share one member's bytes (300 nodes over one 64 KiB member:
  a file of a few KB) used to be written out node by node: 19 MiB here,
  up to 4 GiB a path by the old bounds. host/mods.c gives ARCHIVE_MAX (256
  MiB, which a typedef holds to PACK_MAX), so a merged pack is always one
  the size table's step can read back. Beyond the bound: the lowest mod's
  whole file, as before the merge, and a log line.
- An archive made before is served again only when the file has the size
  AND the hash its record gives (the record now holds both; its format is
  "longshot archive merge 2"). An archive damaged in place without a change
  of size was served again; now it is made again. So what is served is
  always an archive this code wrote whole. The record is written through
  <record>.new and a rename, with its writes checked.
- The data's first member no longer gets the largest alignment (8 KiB): its
  offset from the data's start is 0 and says nothing, so it keeps what its
  offset from the archive's start gave it, which is all its own archive
  gave it. Every merged archive's data used to start on a multiple of
  0x2000: about 8 KiB an archive, several times a small pack (median +256%
  over the game's packs laid out by the old rule). By the new rule: a pack
  +2.5% (median; +2.0% of all the packs' bytes; over 5% for 2,392 of 8,385,
  the most +129%), a layout archive +0.4%, a text archive +0.1%. The
  comment in restbl.c had "2 to 5%", which held for large archives only.
- Every copy whose change of a member is not what the archive gets has it
  counted as lost and named in the log (two mods with one change, a third
  with another: only the second was counted; a change the lowest mod makes
  too is not lost).
- A copy that cannot be decompressed (no size in its frame, a dictionary
  the game does not have) is said to be that, not "not an archive".
- docs/mods.md: the bound, that copies are compared with this game
  version's archive, and that the merged archive is a few percent larger
  and kept uncompressed.

tests/test_mods_archives.py: a member name with line breaks changed by two
mods (the game starts, one line in the log and the record, the record fits
at the next start); members that share bytes (alone and through
host/mods.c: nothing over 4 KB in modcache); the limit to the byte; a small
archive stays within 256 bytes of its members laid out afresh; the
alignment cap (0x2000, neither 0x1000 nor 0x4000); a name that does not end
inside the archive; a name above ASCII, merged and refused when hashed
another way than the game's; one name's nodes in their order with
different bytes; the game's own file changed (made again); an archive
damaged in place (made again); an archive that cannot be written (no
record that says it is not needed; made at the next start);
hos_mods_file. The folder checks that need no dictionary now run without
the zstandard package too.

Not run in the game, nor on Linux.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
Author
Contributor

Tested in game: Windows 11, RTX 3090 (driver 610.88), a build of this branch with a character mod and, below it in the list, a second mod carrying the same eight archives (four actor packs, the layout archive, two text archives) made from the game's own with one text reworded: the title screen shows the second mod's text and the first mod's menu page, layouts, texts and cloth settings all work; the log says 7 archives were served as one in 213 ms and the game reads them from modcache/archives; the Mods tab says so for both mods. Stacked on #942 and through it on #134, so the diff shows their commits too; the last two commits are this change.

Tested in game: Windows 11, RTX 3090 (driver 610.88), a build of this branch with a character mod and, below it in the list, a second mod carrying the same eight archives (four actor packs, the layout archive, two text archives) made from the game's own with one text reworded: the title screen shows the second mod's text and the first mod's menu page, layouts, texts and cloth settings all work; the log says 7 archives were served as one in 213 ms and the game reads them from modcache/archives; the Mods tab says so for both mods. Stacked on #942 and through it on #134, so the diff shows their commits too; the last two commits are this change.
snaildev99 force-pushed mods/archive-merge from e4134107cf
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 3s
Website / Website on Cloudflare (pull_request_target) Has been skipped
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to ea13f08005
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 12s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-16 10:12:58 +00:00
Compare
Author
Contributor

Moved onto #134's new head 3fbc327a (the list of mods is mods.json there now: #134 (comment)). Head ea13f080.

No conflict; the commits are unchanged. It sits on #942's new head. One commit on top: tests/test_mods_archives.py builds host/mods_list.c, has the stand-ins, and writes mods.json where it switched a mod off and changed the order between two starts (the old file is read once only); #134's tests/test_mods_list.py gets stand-ins for the two zstd functions the archive merge calls.

tests/run_all.py on Windows: nothing fails that #134's head passes. Not built or played at this head; the Linux halves of the changed fixtures were read, not run.

Moved onto #134's new head 3fbc327a (the list of mods is `mods.json` there now: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/134#issuecomment-34804). Head **ea13f080**. No conflict; the commits are unchanged. It sits on #942's new head. One commit on top: `tests/test_mods_archives.py` builds `host/mods_list.c`, has the stand-ins, and writes `mods.json` where it switched a mod off and changed the order between two starts (the old file is read once only); #134's `tests/test_mods_list.py` gets stand-ins for the two zstd functions the archive merge calls. `tests/run_all.py` on Windows: nothing fails that #134's head passes. Not built or played at this head; the Linux halves of the changed fixtures were read, not run.
snaildev99 force-pushed mods/archive-merge from ea13f08005
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 12s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to ae8b4670da
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 10s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
2026-11-16 11:19:37 +00:00
Compare
Author
Contributor

Moved onto #134's new head 2b52cd1b (#134 was rebased onto main @ 8a4aa282: the Mods tab's functions are API version 26, and the patches' hash is named in the lift's key after main's #22: #134 (comment)). Head ae8b4670.

No conflict; the commits are unchanged. It sits on #942's new head.

tests/run_all.py on Windows: nothing fails that #134's head passes. Not built or played at this head.

Moved onto #134's new head 2b52cd1b (#134 was rebased onto main @ 8a4aa282: the Mods tab's functions are API version 26, and the patches' hash is named in the lift's key after main's #22: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/134#issuecomment-35102). Head **ae8b4670**. No conflict; the commits are unchanged. It sits on #942's new head. `tests/run_all.py` on Windows: nothing fails that #134's head passes. Not built or played at this head.
snaildev99 force-pushed mods/archive-merge from ae8b4670da
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 10s
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
Website / Website on Cloudflare (pull_request) Has been cancelled
to 73818d8e69
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 6s
2026-11-16 19:47:33 +00:00
Compare
Author
Contributor

Moved onto #134's new head c5cead8d (#134 was rebased onto main @ ac778751 and brings main's new players' mods guide in line with it: #134 (comment)). Head 73818d8e.

No conflict. One commit on top: the players' guide (docs/using-mods.md, main's #1210) gets a paragraph under "When two mods change the same file": an archive two mods both change is served as one, and only a file inside it that both changed goes to the lower mod.

Nothing fails in tests/run_all.py on Windows that #134's head passes. Not built or played at this head.

Moved onto #134's new head c5cead8d (#134 was rebased onto main @ ac778751 and brings main's new players' mods guide in line with it: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/134#issuecomment-37590). Head **73818d8e**. No conflict. One commit on top: the players' guide (`docs/using-mods.md`, main's #1210) gets a paragraph under "When two mods change the same file": an archive two mods both change is served as one, and only a file inside it that both changed goes to the lower mod. Nothing fails in `tests/run_all.py` on Windows that #134's head passes. Not built or played at this head.
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
PR description / Description follows the template (pull_request_target) Successful in 6s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'mods/archive-merge:snaildev99-mods/archive-merge'
git switch 'snaildev99-mods/archive-merge'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1089
No description provided.