Crash when meeting Zelda #1112
Labels
No labels
accessibility
autorelease: pending
autorelease: tagged
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
testing wanted
wontfix
No milestone
No assignees
1 participant
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Hookshot#1112
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hookshot version
hookshot v2.0.7 (source 9b541c9c)
What kind of problem
Crash, freeze or black screen
What happened
The game freezes and then crashes.
Where in the game
First encounter with Zelda, when she tells Link to look inside
How often
Every time
Graphics card and driver version
RTX 4070 Ti, 621,14
CPU, RAM and operating system
Intel 13600KF, 32GB, Windows 11
Changes from the defaults
No response
Other software hooked into the game
No, nothing hooked in
hookshot_log.txt
hookshot_log.txt
Screenshots or video
No response
Before you send it
I looked into this from the log. Some findings that may help whoever can reproduce it.
Where it crashes. The read of guest
0x48is in the game's own render code, not in Hookshot. The guest backtrace'slr 0x80060308is the return fromnn::os::LockMutex(bl 0x83c62460at0x80060304). The fault is a few instructions later, in a cache search loop of the same function:The registers in the log match this exactly:
x23 = 0,x27 = 0(the outer loop's first pass), andx10 = -0x160withx9 = 0x160, from theneg x10, x9at0x80060360just before the loop.x23is only reloaded later in the loop (ldp x24, x23, [sp, #0x30]at0x800607bc). On the first pass it is null, and it is only dereferenced when an entry in the list atx21matches every key field compared before it (+0x140..+0x15c). The translated code is faithful to the original ARM. So on a console that list never has a matching entry at that point, while here it does, probably a timing or ordering difference betweenModuleSystemWorker4and whoever fills that list.What does not reproduce it. On Linux (RTX 4070 Ti, current
main8b2e1d3) I played the whole meeting through twice without a crash (and a third run partway): the window line ("Take a look through this window"), the legend topics and Zelda's Letter. That included the reporter's exact settings:HOS_ASPECT=2.3889(picture 3440x1440, scene 2584x1080),HOS_OUTPUT_RES=1(2560x1440) andlanguage.txt=pt-BR. So the remaining differences are Windows itself and the CPU (13600KF, 6P+8E cores), which points to thread timing rather than a setting.How I got there quickly, in case it helps reproduce it on Windows: teleport to Hyrule Castle (Cheats page), then send a second
oot_teleportrequest whose event name isHyruleCastle_Goto_HyruleCastle_MeetZeldainstead ofDm_SY_BancChangeGo. I used a local change that reads the name from a file at each request, becauseHOS_OOT_TELEPORT_EVENTwould also replace the first teleport's event. From the castle gate, that event plays the whole meeting.Two more crashes with this signature, both in the Market guard house (child, at the Castle Town entrance), not the Zelda meeting. Windows 11, RTX 3080 Ti, Hookshot v2.0.11 (source d6369f1e). The game had a RomFS-only randomizer mod loaded; it changes nothing in the guard house (the only differences in
Market_Child_Staticare shop shelves in the Bazaar and the Bombchu shop).sead::MainThread,read 0x100000000048, guest 0x48, guest backtracelr 0x80060308 0x8028b294 0x80662ee4 0x806602f8 0x80076d68 0x80a47f20 0x8006e72c 0x8006d57c 0x80081358 0x80070bc0 0x802ead44 ...ModuleSystemWorker5: same fault, same registers as in the analysis above (x23 = 0,x27 = 0,x9 = 0x160,x10 = -0x160), and the same first nine frames, then0x8007fe40 0x80df26c0 0x80df2608 0x834a2ec8.So it isn't tied to one scene or one thread. One thing both had in common: the CPU was at 100% from unrelated background processes (Python test runs), and the game was running at 11 to 25 fps just before each crash. If that loop races another thread, a starved CPU would make it more likely, which could be why it doesn't reproduce on an idle machine. I'll try the same room with the machine idle and add the result here.
The guard-house part of this (crashes breaking crates and pots in the Market guard house) may be covered by #1202 (guard-house render-target pool and texture handles), in the v2.0.12 branch. The Zelda-meeting crash is still on the fix list.
I read the log in the first post against the lifted code. The crash is the game's render-target pool running out: the same fault as the guard-house crash of #1202. Nothing below was run in the game; it is all from the attached log and the code.
From the log (v2.0.7)
lr 0x80060308is insidef_8005ed00(0x8005ed00 to 0x80062b70), the allocator of the render targets' pool: the functionnx2/oot_patches.pynames at the pool-size patch ("the next target came back null (crash in f_8005ed00 ...)") and #1202 names for the guard house.mov x23, xzrat 0x8006029c) and falls into the common path. The read of[x23+0x48]that @Joey-Magnus found is the first use of that block.x20 = 0x83dc7d62is the address of that list's format string, and the function loads it into x20 only on that path. Withx23 = 0this shows the allocation had failed before the crash; no race is needed to get there.0x13100000at[x0+0x48]is the pool's size, 305 MB. The picture was 3440x1440, 1.34 times the pixels the 305 MB are sized for, and v2.0.7 scaled the pool with the output resolution only.Why it is #1202's fault too
@hiKareeem's two guard-house crashes (v2.0.11) have the same
lr 0x80060308and the same registers, and #1202 hitf_8005ed00in the same room at 2560x1440 with the 305 MB pool. One onsead::MainThreadand one onModuleSystemWorker5fits: the crash is on whichever thread asks for the target that no longer fits.Since when it should be fixed
f91d23a6(issue 1031), in v2.0.8 and later: the pool scales with the widescreen picture. For 3440x1440 that is 410 MB by its formula.a8e8c2ec(#1202), in v2.0.12 and later: at least 512 MB.@Tgo1014: on v2.0.14 or the current release candidate, with the same settings, line 28 of your log should read
applet: render target pool 512 MB (the game's 305 MB). If the meeting still crashes there, that log would be new information. On any versionHOS_RT_POOL_MB=600in the environment (orhookshot_env.txt) sets the size by hand.Not involved, by the same log: the last
ran without a coreline (#1929) is line 4515 and the crash is line 5182; thevcores:statistics lines from line 4640 on, eight of them, all say0 escapes in 5 s. The lastunimplemented importis line 1200.What I did not do: reproduce it. @Joey-Magnus's two clean runs at these settings were on
8b2e1d3, which is beforef91d23a6, so presumably with 305 MB: how much of the pool the scene takes depends on more than the settings (I do not know on what yet). I have runs queued on Linux (RTX 4090, main49f75dc8) of the meeting at the reporter's settings with the pool forced back to 305 MB, and of the guard house, and will add the counts.Thanks for the report. Reports of the same problem are now collected in #2507, which lists this one with your details and links back here, so one fix closes them together and the open list stays readable. Follow #2507 for the fix, and add new details there (or here). If this turns out to be a separate problem it will be reopened.