WIP: feat(mods): a mod's code module (exefs/subsdk*) is built in with the game #984
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!984
Loading…
Reference in a new issue
No description provided.
Delete branch "snaildev99/Longshot:mods/code-modules"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Windows 11, RTX 3090 (in the game, and the game-free checks) · Needs: Linux, any GPU: one build with a module mod
Why
Some mods ship compiled code of their own beside a patch file (
exefs/subsdk*). Longshot ignores that file, so such amod is listed and does nothing: its patch's stubs find their pointer empty and run the game's own code. Three public
HUD mods are of this kind (Always-On Rupee Counter, HUD of Time, Edge-to-Edge Ultrawide HUD). This builds such a
module in with the game, as the patch files of WIP #134 are, which this depends on.
What changed
A mod's
exefs/subsdk0..9is loaded after the game's own modules when the game is built, translated with it andstarted before the game. The running game answers
svcQueryMemorymodule by module, which is how such a modulefinds the game's main. Modules that install hooks by writing code while the game runs are left out with a note (a follow-up on top of this builds those in too; it is not part of this PR). A module the image builder cannot place (a relocation kind it does not know, an offset outside the module, headers that disagree about its size, a header asking for more than 64 MB) is left out with a note; it never stops the build.
nx2/exefs_modules.py(new):collect(the mods' order and switches, as the patches),classify(systemcalls against
host/hos_svc.c's, imports against the game's exports and host/),load,entries;nx2/image.pyloads the modules after sdk and adds the entries;nx2/make_game.pykeys the lift on them and liststhem in
exefs_patches.txt;nx2/chunks.pywith_mod_modules.host/hos_svc.cimage_region,query_memory;nx2/standalone.pyregions,region_lines.docs/mods.md,docs/development.md.Measurements
Verdict: no cost measured in the game (a module of 4 KB; the frame rate was not compared). The lift's discovery with three real module mods: +68 functions
(54 of the modules, 4 stubs, 10 entries), +10,574 of 17.6 million instructions, no game function lost, no game
address moved.
Flags and settings
HOS_QUERY_MEMORY_OLDsvcQueryMemoryTesting
Tested:
exefs/subsdk7and its.ips. The build's log saysexefs: module Rupee Counter/exefs/subsdk7 (4188 bytes)and that it is built in; the game's log has the module's own[rupee] module readyat the start and its lines all through play; the rupee counter stays on screen in the field, which is what the mod does. Three starts, a child save, no stop.tests/run_all.pyon Windows: 541 passed, 24 failed, 81 skipped; the same 24 fail on #134's head in the same run (539 passed there).tests/test_exefs_modules.pyandtests/test_query_memory.py(new).collectand through the lift's discovery on the real game.Not tested: Linux; HUD of Time and Edge-to-Edge in the game (they pass the game-free checks as the Rupee Counter does); this branch built without its follow-ups.
Must be tested:
exefsfolder as a mod), start the game, expecting[rupee] module readyin the log and the rupee counter staying on screen (by hand).Risks
What could break: SDK code that asks
svcQueryMemoryabout the image now gets a module's segment, not the wholeimage (the console's answer); a constant a module adds to an address that is a code address of main without being a
jump target cuts one game function in two at that place (the first part tail-calls the second: correct, a call
slower there).
Turning it off:
HOS_QUERY_MEMORY_OLD=1for the host's answer; without a module mod the build is byte for byteas before apart from the region table in
gen/hos_image.c.Compatibility: the first build after updating translates the game again (the translator's files changed), as
after any change to the lift; after that a new, changed or switched module mod translates it again, as a patch file does.
Checklist
Everyone
type(scope): description) of the status line's type.main. (on #134's head)python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Tested in game: Windows 11, RTX 3090 (driver 610.88), builds of #134's head
5192b46fwith this branch and its follow-ups merged in, the Always-On Rupee Counter's exefs folder (a subsdk7 module and its .ips) as a mod, a child save in the field: the build's log says the module is built in, the game's log has the module's own '[rupee] module ready' and its lines all through play, and the rupee counter stays on screen, which is what the mod does; three starts, no stop. This branch is stacked on #134 (it depends on it), so the diff shows #134's commits too until that merges.f929331c7d4886c09230Moved onto #134's new head
fafc7ab2(it was rebased onto main @343a1dfa): head4886c092. No conflict, no change in this branch's own files; its tests pass. Not built or played at this head.4886c092305c9361de93Moved onto #134's new head
fb2a419c(rebased onto main @e4d035e9; the Mods tab's functions are API version 25 there, after main's audio outputs): head5c9361de. No conflict, no change in this branch's own files; its tests pass at this head. Not built or played at it.The rest of the module work is up as branches on the fork, not as pull requests (they stack on this one):
mods/module-startup(head 8db1fa8d, on this PR's head): modules that install their hooks at start-up, exlaunch style. The build runs the module's start-up once in a processor emulator against the image, takes what it wrote as it takes a patch file's bytes, and leaves the start-up out of the game. With it Ragdoll Button plays (Windows 11, RTX 3090, on an earlier base): its seven hooks, its settings file on the SD card folder. A second reader went through it for what a broken or hostile module can do to the build; its eleven findings are answered there with tests (bounds on the start-up's time and memory, user mode in the emulator, a hook into a function the host replaces refused by name).mods/modules-stack(head 2d846a5b): the same, plus #940's note and a commit that makes the Mods tab and the log say what became of a module (built in, waiting for a build, left out and why).Both: the mod checks pass and
tests/run_all.pymatches #134's head here (568 passed, 21 failed, the same twenty-one). Not built or played at these heads.5c9361de9306a8fef992Moved onto #134's new head
3fbc327a(the list of mods ismods.jsonthere now: #134 (comment)). Head06a8fef9.No conflict; the commits are unchanged. One commit on top, and it is a fix, not only a test change:
nx2/exefs_modules.pystill handed the oldlongshot_mods.txttoexefs_patches.ordered, which takesmods.jsonsince #134's change, so the modules would have been read as all on, by name, while the patch files followed the list. It now finds the list exactly asexefs_patches.collectdoes (list_file, the old file once,HOS_OPTIONS_FILE,make_game.py --userpassed down).tests/test_exefs_modules.pyfails if the two ever giveordereddifferent files, with and without a user folder.tests/run_all.pyon Windows: nothing fails that #134's head passes. Not built or played at this head; the Linux halves of the changed fixtures were read, not run.06a8fef992c498d3b946Moved onto #134's new head
2b52cd1b(#134 was rebased onto main @8a4aa282: the Mods tab's functions are API version 26, and the patches' hash is named in the lift's key after main's #22: #134 (comment)). Headc498d3b9.No conflict. One commit on top, and it is a fix the new base forces: main's lift key (#22) hashes only named variables, and this branch gives the lift its modules through
OOT_EXEFS_MODULESwith their hash inOOT_EXEFS_MODULES_SHA1, which was in the key only because everyOOT_variable was. On the new base a build after a module was added, changed or removed would have kept the old translation, with no error. The hash is inLIFT_ENV_NAMESnow and the JSON's path inLIFT_PATH_ENV;tests/test_lift_cache.pyhas both cases, and the hash's case fails without the change (run that way first). Main's own check of the lift's variables does not see a hash that is only set, so nothing else would have caught it.The branches on top of this one (
mods/module-startup,mods/modules-stack,mods/module-gapson the fork) are moved too; a start-up recording reaches the key through the same hash, andtests/test_exefs_startup.pychecks that link now.tests/run_all.pyon Windows: nothing fails that #134's head passes. Not built or played at this head.c498d3b94602e563a2cdMoved onto #134's new head
c5cead8d(#134 was rebased onto main @ac778751and brings main's new players' mods guide in line with it: #134 (comment)). Head02e563a2.No conflict. One commit on top: the players' guide (
docs/using-mods.md, main's #1210) says under "What doesn't work" that a mod's code module is built in at a build like a code patch, and which kind is left out. (The branches on the fork above this one are moved too;mods/module-startuphad one conflict innx2/lift.py, where main's lift now writes its coverage report while the emit's processes work: the module check that reads the functions runs inside that report.)Nothing fails in
tests/run_all.pyon Windows that #134's head passes. Not built or played at this head.Played at this head with the branches above it (
mods/module-gapson the fork = #134 @c5cead8d, this pull request @02e563a2,mods/module-startup,mods/modules-stackand the import slot fix): Windows 11, RTX 3090, a full build in 8 min 22 s, 0 failed compiles, on main's new lift (the emit in 16 new processes: the modules' translator forms and jump tables reached the new processes). Five mods with a code module and two with patches only, in one game.subsdk7, 4 KB)subsdk9, 25 KB)subsdk7, exlaunch)subsdk9)mods/module-gaps): its file, save and controller calls all go through the mod, profiles get their own save folders, the old save is untouched. Without the fix it was built in and did nothingsubsdk6, exlaunch)FATAL: svc 0x74 not implemented, from the module, mappingmain's code to patch a function. Everything before that worksTwo modules share the file name
subsdk7and twosubsdk9; both pairs load.What the round found, for this stack:
svc 0x74/0x75) and so may stop the game; and the host could refuse that call with an error instead of stopping, so the mod's feature fails and the game goes on.texts/(the microphone hint), and the mod does not recognise it: the label shows as a missing message. WithHOS_TEXTS=0the label is right. So a mod that replaces the text file wins over the rewording, and a mod that patches it in memory loses to it. The row's value text is still missing withHOS_TEXTS=0; the mod's own log says it does not know that text's format, and whether that is the same on an emulator is not checked.nn::oe::RestartProgramreturns without restarting, as known: the mod says so in its log and the change applies at the next start by hand.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.