WIP: feat(mods): a code mod is switched off and on at a restart, without a build #1036
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1036
Loading…
Reference in a new issue
No description provided.
Delete branch "snaildev99/Longshot:mods/patch-switch-stack"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Windows 11, RTX 3090 (in the game, twice, on the base before the last rebase; the game-free checks at this head) · Needs: Linux, any GPU: a build and the round below; a frame-rate comparison
Why
With #134 a mod's code is built into the game, so switching such a mod Off or On only takes effect at the next build. Until then the game runs half a mod: Off leaves its code in while its files are gone (seen: a menu drawn without its layout), On serves its files while its code is not there (seen: a menu page that opens and does nothing). The Mods tab says so, but a switch that needs a ten-minute build is not much of a switch, and a tester on #134 asked for exactly this gap to be closed.
This makes the switch work the way it does for any other mod: Off, restart, the plain game; On, restart, the mod is back. No build.
The idea in one line: for a mod to be switched back on without a build, its code has to be in the game already. So the build keeps both versions of the few functions a mod changes, the game's own and the mod's, and the game picks one at its start. With the mod Off the game runs its own, unchanged code; the mod's version is there and never reached.
It is stacked on #134 and on the patch-build follow-ups (#960, #961 and three branches that have no pull request yet, see What changed), so only the last three commits are this change.
What changed
A mod with ExeFS patches that is in
modswhen the game is built, On or Off, is switched at the next start.nx2/exefs_switch.py, new; hooks inexefs_patches.py,lift.py,chunks.py,build.py): every function that holds a word a mod writes is translated once more for each other state of the mods that touch it. The function as written gets one line at its top that reads a byte the host sets (hos_patch_on[]) and goes to the right version; every other function is translated once, as now. It works a function at a time, so a real code cave (branches, calls,adrp) needs no special rule. The lift is told the calls of every state, so a function only the game's own code calls is a function with the mod Off too. Jump tables in data a mod writes are read again per state.host/mod_switch.c, new;host/mods.c,host/hos_main.c): at the start, before any game code runs, the switches are set from the Mods tab's list and the bytes of the mods that are On are written into the loaded image (first the game's own byte of each place, then each On mod's in the list's order).image.binitself stays the game's.exefs_patches.txtmarks each file with its switch (code+switch<n>), a mod that is Off included.mods/port-sites: a note when a patch lands where Longshot hooks the game's code;mods/atmosphere-patches: a console pack'satmosphere/exefs_patches/<name>/folder is a mod;mods/lift-key-bytes: the lift's key follows the patched bytes, so a renamed mod keeps the translation). The switch builds on the last one's key.Measurements
Verdict: the cost in the build is small and measured; the cost while playing is not measured yet (expected: one byte load and a predictable branch at the entry of the switched functions).
Hardware: Windows 11, RTX 3090, driver 610.88
Scene: none (the build's own numbers)
Method: the build's log and a dry run of the build's steps over the game's code
How to reproduce: build with the two mods below in
mods; theexefs: switched at a startline.pchtxt: functions translated twiceFlags and settings
LONGSHOT_PATCH_SWITCHmods, On or Off0: none, every function as before this PR;on: only the mods On at the build0checked to give the same translated files as the base;onuntested in a gameTesting
Tested:
5192b46f), before and after an independent review's nine findings were answered. Mods: Character Select v0.2.1 (6,903 files, a 7.7 KB patch with a code cave) On, and Auto Jump (a.pchtxt) Off at the build. Four starts each time, no build between them: (1) the mod works, Auto Jump is listed Off; (2) Character Select switched Off on the Mods tab, restart: the plain game, three tabs in the game's menu, nothing drawn wrong; (3) On, restart: the mod back whole; (4) Auto Jump switched On, restart: its change works and the other mod still does. The Mods tab only ever asked for a restart. The log:its ExeFS patches are off at this start (built in behind a switch),1 of 2 switched mods on; 15481 bytes of the game's code and data set to match.tests/test_exefs_switch.py(new): a made-up program through the build's own steps, compiled and run in every state against a small interpreter: a cave with a conditional branch, a call and anadr; a mod off at the build; two mods on one function (four states); a third past the cap; a function only the unpatched code calls; Longshot's own edits of the translated C with two mods on the same functions; jump tables; a file saved again.tests/test_mods_switch.py(new):host/mods.candhost/mod_switch.cover made-up mods and memory, several starts: the switches, every byte, the Mods tab's lines, the log.tests/run_all.pyon Windows at this head: 564 passed, 21 failed, 85 skipped; the same twenty-one fail on #134's head in the same session (557 passed there).Not tested: this head in a game (the two rounds were on the base before #134's last rebase; nothing in this branch's own files changed since); Linux; frame rate; the emit's spawned workers together with the picking line; more than two mods on one real function.
Must be tested:
mods, then on the Mods tab switch it Off and restart (the plain game), On and restart (the mod back), with no build between;longshot_log.txthas thebuilt in behind a switchlines (by hand).LONGSHOT_PATCH_SWITCH=0and without it: the frame rate, three runs a side.Risks
What could break: a state of the switches the tests do not cover reaching code translated for another state (the review looked for this and found the design sound: every way from one function to another goes through the function's own name, which is the picking line). A switched function that is hot runs its usual version from the hot file with the mod On and a plain copy with it Off. The game holds the code of mods that are Off: it is never reached, but it is in the executable, and a mod with an instruction the lift lacks shows up at the build even when Off.
Turning it off:
LONGSHOT_PATCH_SWITCH=0at the build: no switches, the translation as before this PR.Compatibility: the first build translates the game again. A game folder built before this has no switches; its mods need a build to switch, as before.
Reviewer notes
Two decisions here are the maintainer's, and the code follows either answer with one setting:
mods, On or Off, so any of them can be switched later. The other choice (LONGSHOT_PATCH_SWITCH=on): only the mods On at the build, so nothing a player switched Off is in the executable, and a mod that was Off needs a build to come on.Checklist
Everyone
type(scope): description) of the status line's type.main. (on #134's head and the patch-build follow-ups)python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
An independent review of the switch found nine things the game round did not meet. Each is answered here with a test that fails on the commit before and passes now. 1. The lift's key did not cover what a64_switch.c carried (HIGH). The table named each switched file and the hash of its bytes; the key is made of the bytes the files write. A file saved again, renamed or encoded another way kept the key and the old table, so its switch stayed off for good while the Mods tab said "built into the game". a64_switch.c now has numbers and bytes only (what became of each switch of the plan, and the writes); which file is of which switch is in exefs_patches.txt ("code+switch<n>"), which every build writes, a switched mod that is off included, and host/mods.c reads it there. 2. Longshot's own edits of the lifted C and the line at a switched function's top (HIGH). The line was put in when the function was written, so an edit that puts a host call first in a function put it above the line, and the call ran again in the version the line left for; an edit that matches the head with the first instruction found nothing; and of several other versions under one head only the first was edited. The line now goes in after the edits: chunks.edit_c takes the lines out before its first pass and puts them in after its last (exefs_switch.strip_lines, add_lines; a64_switch.c records them), and each other version is edited by itself (exefs_switch.edit). Not the picking function of its own the review proposed: that needs the function as written under another symbol, a #define above it, and the lift's cuts of a file (chunks._cut, split_hot.py) are made at a function's head line, which would leave the #define in the piece before. 3. A function only the unpatched code calls was no entry (MEDIUM): the image takes its entries from main with every switch on, so the version with a mod off called code with no function and was written as a trap. The calls of the game's own words and of each switched mod's own words at every place such a mod writes are entries now (exefs_switch.entries, nx2/image.py), and so are the code addresses the game's own words form there with ADR (plain_image, nx2/lift.py). Where a version would still stop at code that is no function and the function as written would not, the function's switches are built in for good, with a line in the log: such a mod takes a build to switch, as before, and no state traps. 4. A mod that is off decided how a mod that is on was built (MEDIUM): make_returns saw the off mod's bytes in the on mod's hook routine and left its B where a build without the off mod writes a RET. Two switched mods of which one changes what make_returns writes for the other are no switches (exefs_switch.entangled, settled before the whole-list steps); built in or left out as they are at the build, the step sees exactly what the game has. A note about bytes a switched mod takes says "while that mod is on". 5. The cap's re-walk dropped the table of a jump another function shares (LOW): the table is kept unless the walk without the mod reads it again, and read again from the image without the mod when another function still holds the jump. 6. Two mods that ship the same patch file (LOW): each file is its own mod's by the list's line, so one mod's file is not counted for the other's switch. 7. The log said a deleted switched mod's code stays in the game (LOW): a switched file's line is not counted there; its switch is off. 8. Jump tables (LOW): the table check runs for every switch, not only those that write data, once with every switch off and then a switch at a time, each both ways (two mods on one table); a version's jump that its own code does not resolve has no targets left from the lift. 9. A mod that is off could stop the supported build (LOW): an output-size place that only a version with a mod off has counts as found, with a warning that the place is lost while that mod is on. tests/test_exefs_switch.py is written again around the shapes the review named, each run in every state against the interpreter: a function only the unpatched code calls; Longshot's own head hooks on functions two mods switch, through chunks.write and edit_c as the supported build runs them, with the host calls counted; a hook's return with an off mod over its routine; a left-out mod whose routine shared a jump; the hot file cut by split_hot.py and the compile's cut of a giant; the kept translation for a file saved again. tests/test_mods_switch.py: the list of a later build, the same file in two mods, a deleted mod, a left-out mod's bytes, a write outside the game's memory. Two of patch-build's tests read a line of the list: its kind is "code+switch0" there now. Co-Authored-By: Claude Fable 5.2 <[email protected]>Tested in game: Windows 11, RTX 3090 (driver 610.88), two builds of this change on the base before #134's last rebase (
5192b46f), the F1 menu's Mods tab and the game's own menu: Character Select v0.2.1 (a 7.7 KB patch with a code cave) On and Auto Jump (.pchtxt) Off at the build. Off on the Mods tab and a restart: the plain game, three tabs, nothing drawn wrong; On and a restart: the mod back whole; Auto Jump On and a restart: its change works; no build between any of them, the tab only asked for restarts. The log: 'its ExeFS patches are off at this start (built in behind a switch)'. Not built at this head (nothing in this change's own files moved in the rebase). Stacked on #134, #960 and #961 and three more patch-build branches, so the diff shows those commits too; the last three commits are this change.36b9bd51c00ec18e14c7Moved onto #134's new head
fb2a419c(rebased onto main @e4d035e9; the Mods tab's functions are API version 25 there, after main's audio outputs): head0ec18e14. No conflict, no change in this branch's own files; its tests pass at this head. Not built or played at it.0ec18e14c704bd8dc670Moved onto #134's new head
3fbc327a(the list of mods ismods.jsonthere now: #134 (comment)). Head04bd8dc6.The same conflicts as #960 and #961 in
nx2/exefs_patches.py, and one innx2/make_game.py(this branch'slift_envcall with #134'scollect(..., user=...)). One commit on top:tests/test_mods_switch.pybuildshost/mods_list.c, has the stand-ins, writesmods.jsonfor its three changes between starts, and has a new check that the build reads the samemods.jsoninto the same order and switches the plan was made from (a mod that is off is listed too); the atmosphere, overlaps and switch build tests writemods.json;tests/test_mods_list.pystubs the three passes that need a game image.tests/test_longshot_ctl.pyfailed once in the parallel run on a locked temporary file and passes alone (it does not touch the mods).tests/run_all.pyon Windows: nothing fails that #134's head passes. Not built or played at this head; the Linux halves of the changed fixtures were read, not run.04bd8dc670934d41ebc6Moved onto #134's new head
2b52cd1b(#134 was rebased onto main @8a4aa282: the Mods tab's functions are API version 26, and the patches' hash is named in the lift's key after main's #22: #134 (comment)). Head934d41eb.No conflict. One commit on top, for the lift's key on the new base (tests and one line, no change in behaviour): the switch's plan (which file is of which switch, an off mod taken in or left out) reaches the key through the patches' list, whose hash #134 names;
LONGSHOT_PATCH_SWITCHis read only by the plan, so it is listed as changing nothing the lift writes by itself.tests/test_exefs_lift_key.pyshows it throughmake_game.lift_key:=0against unset, an off mod joining or leaving the plan and a changed byte change the key; a renamed mod, and the variable alone with the same list, do not.A reviewer note this brought up (as it was before the move, not new): once switches exist, the key also holds whether each switched mod is On, because a function more than two mods touch is built with the extra ones in their current state. So switching at a start needs no build, as played, but the next build after a switch translates the game again where it could keep the translation. Narrowing that to the mods that are built in for good is a small follow-up; say if you want it in this pull request.
tests/run_all.pyon Windows: nothing fails that #134's head passes. Not built or played at this head.934d41ebc6d25f2625ffMoved onto #134's new head
c5cead8d(#134 was rebased onto main @ac778751and brings main's new players' mods guide in line with it: #134 (comment)). Headd25f2625.One conflict, in
nx2/lift.py: main's lift now passes its coverage report tochunks.writeto run beside the emit's processes (#1128 and the commit before it); that argument and this branch's line that writesa64_switch.cafter the emit are both kept. Read against main's new emit in new processes (the Windows path): the other versions of a switched function are made in the lift's own process before the write, the picking line is added in that process as each function comes back, and the FPX shapes of the other versions still reacha64_fpx.hthrough the header function this branch wraps. That path is not covered by a game-free test here (it starts at 4,000 functions, andtests/test_spawn_emit.pyfails on main on this PC), so it wants a real Windows build before this is taken.One commit on top: the players' guide (
docs/using-mods.md, main's #1210) says that a mod whose patches a build took in is switched at a start without a build.Nothing fails in
tests/run_all.pyon Windows that #134's head passes. Not built or played at this head.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.