WIP: perf(host): the physics' triangle set-up and tolerance test on host code, bit for bit (HOS_NATIVE_SOLVER, on by default) #1900
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1900
Loading…
Reference in a new issue
No description provided.
Delete branch "snaildev99/Longshot:perf/native-solver"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Two small routines of the game's rigid-body physics run on host code with the game's exact results, behind a new switch, on by default (
HOS_NATIVE_SOLVER=0gives the lifted code):f_80171a90(108 instructions, a leaf). It copies a mesh triangle's vertices into a triangle shape, computes its plane N = (B - A) x (C - A) and d = -(A . N), both scaled by 1 / sqrt(N . N), and writes the plane, its negative, its copies, the flags and the markers.f_803be11c(19 instructions, a leaf). It tests whether three vectors are each within a tolerance of three others (FABD, FMAX, FCMGE over four lanes).In a Market profile their own time is 0.038 % and 0.038 % of the game's code; on a Kakariko and Death Mountain Trail route, 0.156 % and 0.072 %.
Four more pieces are in
host/oot_solver.cwith their tests and are exact offline, but they are not wired: no address of theirs is innx2/oot_patches.py, so the lift writes those functions as on main. Each measured slower than main's lifted code, or did not save more than its hook costs with the switch off:f_80168130;f_81a043b0;f_8174bb00;f_81a012e0.The test keeps their splices and checks that the patch file names none of them.
The goal is less CPU work for the same frames. A call the host cannot prove exact runs the lifted routine, with nothing written. A call is answered when:
The host reads everything first, proves, then writes. It writes every register the routine leaves changed (set-up: x0, x8-x12, v0-v7, v16-v20, NZCV; tolerance test: x0, x8, v0-v3, NZCV) and marks what it writes (
HOS_WRITTEN).How:
nx2/oot_patches.py(SOLVER_NATIVE,patch_solver_native; the supported build only) puts the hook first in each function:hos_solver_liveis 1 until the first call has read the switch, then 0 unless it is on. The hooks are declared(Ctx *A64_NOESC c, A64Fn lifted), as the lifted code's own calls are. Each hooked function is also declared at file scope, becausenx2/split_giants.pymay put the hook in a piece written beforef_<addr>.Read first:
host/oot_solver.c:hos_solver_80171a90,hos_solver_803be11c, the operand rule, check mode.nx2/oot_patches.py:SOLVER_NATIVE,SOLVER_NATIVE_DECLS,patch_solver_native.tests/test_oot_solver.py.gpu/vk/switches.def.Testing
Tested: Windows 11, RTX 3090 (NVIDIA 610.88), Ryzen 9 5900X. The triangle set-up ran in the game, in test trees on main
068fb8ebthat merge nine rewrite branches, with this branch's code at 9540f366. At that point the quaternion matrix was still wired and the tolerance test was not yet written.HOS_NATIVE_SOLVER=check, the Market and Hyrule Field, about five minutes): every answered call was compared with the lifted routine, register for register and byte for byte.test/rewrites@ e614ef49, in the CPU measurement below, every other rewrite on; all 15 runs held 60 fps): in one run's lastoot: solverline, the triangle set-up answered 254,558 calls and left none to the lifted code; the quaternion matrix answered 1,523,615 and left 210,435 to the lifted code for their values.Exactness offline. Unicorn runs the game's own routine on the executable. A driver runs the real
host/oot_solver.cwith the spliced lifted text thatnx2/oot_patches.py's own patch makes. Per case it compares every x register, SP, every q register (128 bits), NZCV, FPSR and every byte of the regions the case names. A call left to the lifted code must equal the lifted run in everything. Two builds were run (x86-64-v3 and x86-64), then the same cases in check mode and with the switch unset.A64_NOESC: 30,000 cases each, equal.>for>=, a register, the tolerance unchecked, a difference's sign kept, the N flag), 44 to 1,807 of 6,000.Run again on this branch, with the upkeep commit's declarations (the harness builds the spliced lifted text from this branch's
nx2/oot_patches.py), seed 61, 30,000 cases each:HOS_NATIVE_SOLVER=0: equal, nothing answered.Game-free checks at this head:
tests/test_oot_solver.pypasses. It covers:tests/test_switch_table.py,tests/test_patch_hooks_defined.py,test_log_levels,test_utf8_paths,test_doc_anchors,test_check_third_partyandtest_env_onpass.nx2/oot_patches.pyornx2/entry_hooks.pypass. The other five fail here and on maina104182balike:test_dev_packs,test_high_fps_fixes,test_libc_hooks,test_lift_cache,test_song_note_cue.tests/run_all.pywas not run.Still to test:
HOS_NATIVE_SOLVER=checkin the Market and on Death Mountain Trail. Expectoot: solverlines with the triangle set-up's and the tolerance test's calls taken on the host andMISMATCHES 0. Then=1: bodies resting, rolling and colliding look as before (by hand).HOS_NATIVE_SOLVER=0against unset (on), and main against this branch with the switch unset. Measure the game's CPU seconds with the frame rate capped at 60, standing and turning, and the frame times uncapped. Expect less CPU time with the switch on, the same frame rate or better, no cost with it=0, and the picture as without it (@Beanstalk linux, amd).Measurements
Verdict: less CPU work, measured together with the other rewrite branches, not yet alone. In a test tree that merges nine rewrite branches (named in the baseline below; the cloth and particle effect rewrites and some later pieces are not among them), at a 60 fps cap in the Market:
How much of that is this branch is not separated. That tree had this branch's earlier wiring (the triangle set-up and the quaternion matrix; no tolerance test), so it is not this head's wiring either. The Test Farm request asks for this branch's own number.
Baseline: main
068fb8eb(#1802) against the test treetest/rewrites@ e614ef49. That tree is main068fb8ebwith nine rewrite branches merged (not every rewrite there is: the cloth and particle effect branches were not in it): the seven maths units, the maths library, animation, bounds, crowd, visibility, glue, the effect triggers with the reverb, and this branch at 9540f366. It was lifted withA64_FPXKEEP=0, so that the lift is main's apart from the hooks. It is not this PR alone: the run-time switches were all on together. Main has moved since068fb8eb; this branch was not measured on current main.Setup: Windows 11, Ryzen 9 5900X, RTX 3090, 3840x2160; the child Market, hands-free; frame rate capped at 60 (all 15 runs held 60). Each run: 30 s standing, then 30 s turning the camera. Five variants, interleaved over three rounds, the PC otherwise idle. The figure: CPU seconds per second of play (cores busy on average) of the whole process, from the processor's cycle counter; the median of three runs (the three in brackets).
068fb8ebThe off figures are inside the runs' spread. The process includes the renderer's threads (about 1.4 cores standing), which the rewrites do not touch. On the executors, where the physics runs: -4.1 % / -4.0 % with the nine on.
Offline: CPU time a call, not the game. Against main's real lift: main's lifter run alone (
nx2/lift.py --standalone) on068fb8eb, the functions cut from its output and compiled with main's flags. Ryzen 9 5900X, best rounds on a busy PC, FPCR 0 / FZ:A64_NOESC, -7 / -8 % with itProjected from the two routines' self shares (not measured in the game): about 0.05 % of game code in the Market and 0.13 % on the route.
The switch
HOS_NATIVE_SOLVER(new, run time,kill, on by default):0off (the lifted routines, main's results); unset or1on (host code where it is exact, the rest lifted).checkkeeps the host's answer aside, runs the lifted routine for real, compares every register the routine leaves changed and every byte it writes, and keeps the lifted result.Every 10 s an
oot: solver (HOS_NATIVE_SOLVER=...)line gives, for each piece, the calls taken on the host and the calls left to the lifted code by reason (FPCR/FPSR, addresses, overlap, values, other). In check mode it adds the calls compared andMISMATCHES. It ships on: docs/dev/standards.md has a switch ship at its value from the first push, never off because it is new, and the measurement with the rewrite branches together says on.HOS_NATIVE_SOLVER=0gives main's lifted code. This PR's own measurement (the farm request) can still turn it off before it leaves WIP.Risks
What could break: a wrong triangle plane would show as a contact on the wrong side or at the wrong depth against meshes: bodies sinking into or bouncing off the ground. A wrong tolerance answer would change a physics decision (inferred from the routine's shape; its caller was not read whole). In the game's check run every answered triangle set-up matched the lifted routine. The tolerance test has not been compared in the game.
Open: in the CPU measurement, the renderer's
mkw_vk capture copythread used 10 to 18 % more CPU (0.009 to 0.017 cores), in both variants with the rewrites on and in neither without. That thread copies guest memory the game marked as written, and the native kernels mark what they write (HOS_WRITTEN; here the triangle shape's arrays; the tolerance test writes no memory). Which rewrite causes it is not determined; this branch is one of several candidates.Turning it off:
HOS_NATIVE_SOLVER=0.After updating: the game is translated again once (
nx2/oot_patches.pyis in the lift's key). The routines run on the host by default, with the game's exact results; saves,longshot_options.iniand the caches do not change.Checklist
Ready for review
main; the title is a Conventional Commit.068fb8eb, in check mode and on; the tolerance test has not; this head is not built or played.)mise run testpasses (any failure that also fails onmainis named). (The tests that read the changed files pass, apart from five that fail on maina104182btoo; the whole suite not run.)main, or Measurements says why there is nothing to measure. (Measured with the other rewrite branches merged and this branch's earlier wiring; this branch alone not yet.)third_party/untouched; nothing GPL-incompatible.When it applies
gpu/vk/switches.def. (The row is there; on by default,=0gives main's lifted code; this PR's own measurement can still turn it off.)tests/that reads changed source is updated, not weakened. (tests/test_oot_solver.py, new; it finds functions by their head lines.)Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
The two wired hooks pass their own function to the host (`hos_solver_<addr>(c, f_<addr>)`) for the check mode. nx2/split_giants.py writes a giant lifted function's pieces before `f_<addr>` itself, so a hook in the first piece would name a function not yet declared ("use of undeclared identifier", as two other hooks did in a merged build). SOLVER_NATIVE_DECLS now also declares `void f_<addr>(Ctx *__restrict A64_NOESC c);` at file scope for f_80171a90 and f_803be11c. The hooks' declarations already take A64_NOESC. tests/test_oot_solver.py finds the functions around the walk by their head line `void f_<addr>(Ctx *__restrict A64_NOESC c) {`, not their first mention, which is now the declaration. Co-Authored-By: Claude Opus 5.5 <[email protected]>Tested in game: Ran in the game in merged test trees of the rewrite branches on main
068fb8eb(Windows, RTX 3090), at the branch's earlier wiring: in check mode in the Market and Hyrule Field the triangle set-up answered 531,947 calls (and the quaternion matrix, since unwired, 1,667,707) with 0 mismatches, then switched on in a 15-run CPU measurement at a 60 fps cap (every run held 60); the tolerance test has not run in the game, and this head itself is not built or played.@Beanstalk linux, and amd: this branch with HOS_NATIVE_SOLVER unset against HOS_NATIVE_SOLVER=1 (the same build), and main against this branch with the switch unset, each in the child Market for a few minutes and in Hyrule Field by day for a few minutes. Measure the game's CPU seconds with the frame rate capped at 60 (on a PC that holds 60 there), standing and turning the camera, and the frame times uncapped. Expect less CPU time with the switch at 1, the same frame rate or better, no extra CPU time with the switch unset against main, the picture and physics as without the switch, an "oot: solver (HOS_NATIVE_SOLVER=1)" line in the log with calls taken on the host, and no "MISMATCH" in the log.
@snaildev99 only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was fora664ba4, and the PR has changed since (eeb6b69), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.Recommendation: Request changes. The default-off hook has a data race, and the change retains four substantial implementations that cannot be reached by the game. Its performance benefit and its cost with the switch off remain unproven for this head.
The change can run two physics routines on the host when it can establish equivalent results, with a check mode for comparison. It adds a fallback and targeted tests, but also adds unused code and a hook paid by players even when the switch is off; the current evidence does not establish a net gain.
Does it help: Unproven. The reported 2.7% and 2.4% whole-process CPU reductions are for several rewrites together, including an earlier version of this branch. Offline measurements report 39–45% and 77–83% lower per-call time for the two routines, but do not establish the game-level effect or frame pacing at this head. The reported default-off measurements also use the combined tree and are within the run spread.
Other systems: The hooks affect the lifted game on Windows and Linux even with the switch unset. This head has not been built or played; the tolerance routine has not run in game. Linux, AMD, other Tier 1 hardware, uncapped frame times, build time, and the possible 11–14% increase in the renderer capture-copy thread remain unverified. The earlier Windows check-mode run matched 531,947 triangle calls, but used different wiring.
Switches: HOS_NATIVE_SOLVER is new: unset or 0 is off, 1 enables host answers, and check compares against the lift. Its table row is present. Off is the cautious default for an unverified implementation, but measurements have not established whether this default or the enabled path improves the game.
Reviewed:
a664ba4211againsta104182b40: Read the repository rules, pull request template, commit history, change summary, hook diff, and the changed host and test code in part, including the wired routines, gate and tests. Output limits prevented a full reading of host/oot_solver.c, tests/test_oot_solver.py and nx2/oot_patches.py; the review is therefore incomplete. docs/review-checklist.md and the adjacent Tooling checkout were absent. The supplied comments and test reports were read; there are no earlier reviews.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Testing: Read-only review; no code, scripts, builds or tests were run. The author reports targeted game-free checks passing, but not the full suite, and reports that this head has not been built or played.
Overlaps and merge order: Unknown; the network is closed and no overlapping pull requests were supplied.
Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.
@ -0,0 +7,4 @@flags and its material into the collide work buffer and calls the collide callback. The hook isfor the loop's head label and does one triangle's body, the material lookup f_818a0d60 included;the callback's call and the loop's control stay lifted. NOT SPLICED IN: the lifted code nevercalls it. Its two splices in the loop cost the lifted body 12 to 28 % with the switch off andThis and the other three unwired implementations add substantial code and tests without a game caller. Please remove them and the unused splice machinery until a version can be wired and shown to help.
@ -0,0 +92,4 @@hos_log("oot: solver: pieces of the rigid-body physics on the host (HOS_NATIVE_SOLVER%s)",m == SOLVER_CHECK ? "=check: each call compared with the lifted code, which is kept" : "=1");m = __atomic_load_n(&solver_mode_now, __ATOMIC_RELAXED);if (!m) hos_solver_live = 0;This write can race with a hook reading hos_solver_live on another game thread, including with the default switch setting. Please use atomic stores and atomic loads in the generated hooks; volatile does not synchronize them.
@ -790,0 +801,4 @@SOLVER_NATIVE_TAG = " /* (nx2/oot_patches.py: HOS_NATIVE_SOLVER, host/oot_solver.c) */\n"SOLVER_NATIVE = {# the triangle set-up, a leaf"80171a90": (" if (__builtin_expect(hos_solver_live, 0) && hos_solver_80171a90(c, f_80171a90)) return;", ()),This hook also runs with the switch off, but the reported game measurements use an earlier combined rewrite tree. Please measure this head against its base both off and on, including frame pacing, and investigate the reported capture-copy CPU increase.
@fix
I changed this pull request by bringing it up to date: the new head is
eeb6b69c83. I rebased it ontomain(a764653726).Conflicts resolved
Left for the author
Nothing was built or run: check the change. It will be reviewed again.
Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.
a664ba4211eeb6b69c83Recommendation: Request changes. The default-off gate still has a data race, and the patch carries four implementations the game cannot call. The measured per-call savings have yet to show a net gain in the game on this revision.
Adds opt-in host paths for triangle collision setup and a vector tolerance test, with lifted fallback and a diagnostic comparison mode. These paths could reduce CPU work in physics-heavy scenes. Both hooks change the generated code even with the switch unset, and the four unwired routines add build and maintenance work without helping players.
Does it help: Unproven. The author reports offline per-call savings of 39–45% for triangle setup and 77–83% for the tolerance test. The three interleaved Windows Market runs reported 2.7% and 2.4% lower whole-process CPU use with nine rewrites enabled and an earlier solver configuration. The combined-tree off deltas of 0.3% and 0.2% were within run spread. There is no isolated game measurement of
eeb6b69c83againsta764653726, or evidence for uncapped frame pacing or clean-build cost.Other systems: Both supported Windows and Linux builds get the generated hooks and lose the lifter's guest-local optimization for the two functions. The triangle path also changes dirty-page marking. The combined run reported 0.009–0.017 additional cores in capture copy, with the cause unresolved. Current-head physics and pacing evidence is missing on NVIDIA, AMD/RADV and Steam Deck. The diff directly changes neither renderer interfaces nor save, audio, input or settings formats. The patch file participates in the lift cache key, so players will relift after updating.
Switches: HOS_NATIVE_SOLVER is a new developer A/B switch documented in switches.def. Unset or 0 uses the lift, 1 enables eligible native calls, and check compares eligible calls and keeps the lifted result. Off avoids enabling unverified physics, but the project requires a measured shipping default and the off-path cost is still unknown.
Reviewed:
eeb6b69c83againsta764653726: Reviewedeeb6b69c83againsta764653726. Read the base AGENTS.md, CONTRIBUTING.md, Forgejo template and applicable standards, the commit history, the four-file diff, the active host routines and their fixtures, and the relevant lifter, split-function, build, FP and write-watch integration. Skimmed the unwired routines and fixtures and the supplied run reports. The surrounding unchanged parts of nx2/oot_patches.py were read selectively. docs/review-checklist.md and the adjacent Tooling checkout were absent. Generated game code and the private Unicorn harness were unavailable. The supplied comments and stale review were read. The bot's claimed atomic-gate and dead-code edits are absent from this checkout.; nothing run (a code review).Checked and fine:
Blocking (to change before merge)
Waiting on the author (a run, a measurement or a decision; no edit does it)
Testing: Read-only review. No code, scripts, builds or tests were run. The author reports targeted game-free passes and 30,000 offline cases per active hook with zero mismatches before the rebase. The earlier merged-tree Windows run matched 531,947 triangle calls. The full suite and current-head game runs are outstanding.
Overlaps and merge order: No merge dependency is established. The nine-rewrite integration tree supplies historical benchmark evidence only.
Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.
@ -105,6 +105,7 @@ SW(HOS_MOVIE_PACE, kill, bool, "on: a 60 FPS cap is raised for a movie's pictureSW(HOS_MUTEX_TRACE, diag, int, "0: off", "Logs mutex waits longer than <ms>, with the thread that held the mutex.")SW(HOS_MUTEX_WATCH, diag, string, "off", "Logs every operation on the mutex at that guest address (hex) and its condition variables' signals.")SW(HOS_MW_WAKE, kill, enum, "only changes to objects linked into a multi-wait wake WaitAny", "=all: every event, semaphore or queue change wakes every WaitAny, as before.")SW(HOS_NATIVE_SOLVER, attempt, enum, "off: the rigid-body physics runs as lifted", "1 runs pieces of the Havok rigid-body physics (the triangle set-up, a tolerance test of three vectors) on host code where it gives the game's bits exactly, the rest lifted; check runs both on every call and logs any difference (slow). An oot: solver line every 10 s counts the calls (host/oot_solver.c).")Check mode compares only eligible calls. Inputs that fail the guards run lifted without a comparison, so replace “both on every call” with wording that reflects that.
@ -0,0 +7,4 @@flags and its material into the collide work buffer and calls the collide callback. The hook isfor the loop's head label and does one triangle's body, the material lookup f_818a0d60 included;the callback's call and the loop's control stay lifted. NOT SPLICED IN: the lifted code nevercalls it. Its two splices in the loop cost the lifted body 12 to 28 % with the switch off andRemove the four unreachable implementations and their fixtures and splice support. The game cannot benefit from them in this change.
@ -0,0 +92,4 @@hos_log("oot: solver: pieces of the rigid-body physics on the host (HOS_NATIVE_SOLVER%s)",m == SOLVER_CHECK ? "=check: each call compared with the lifted code, which is kept" : "=1");m = __atomic_load_n(&solver_mode_now, __ATOMIC_RELAXED);if (!m) hos_solver_live = 0;Concurrent first calls race on this volatile store and the generated reads, even when the switch is unset. Make all gate accesses atomic.
@ -0,0 +157,4 @@}/* (the totals so far, the calling thread's included: tests, and the last line of a run) */void hos_solver_end(void) {Only the fixture calls hos_solver_end. The runtime emits periodic totals while calls arrive. Drop the claim about a final line or connect it to the runtime lifecycle if a final total is needed.
@ -0,0 +465,4 @@return -S_ALIAS;}const __m128 A = xyzz(b), B = xyzz(b + 12), C = xyzz(b + 24);const __m128 E1 = _mm_sub_ps(B, A), E2 = _mm_sub_ps(C, A);A signaling NaN reaches SUBPS before the class check and can leave host MXCSR changed on fallback. Validate inputs first and contain speculative FP status changes.
@ -810,0 +821,4 @@SOLVER_NATIVE_TAG = " /* (nx2/oot_patches.py: HOS_NATIVE_SOLVER, host/oot_solver.c) */\n"SOLVER_NATIVE = {# the triangle set-up, a leaf"80171a90": (" if (__builtin_expect(hos_solver_live, 0) && hos_solver_80171a90(c, f_80171a90)) return;", ()),Measure the finished patch against its base with the switch unset and enabled. Include frame pacing and isolate the reported capture-copy CPU increase.
@ -1759,6 +1831,15 @@ def patch_file(s, name, protect_hearts=False):if floor != out:lines.append("oot_patches: a moving floor's carry above 60 fps in " + name)out = floorif protect_hearts: # (HOS_NATIVE_SOLVER: the supported build's functions)Record clean-build time and output size against the base. This changes the lift players rebuild after updating.
WIP: perf(host): the physics' triangle set-up and tolerance test on host code, bit for bit (HOS_NATIVE_SOLVER, off by default)to WIP: perf(host): the physics' triangle set-up and tolerance test on host code, bit for bit (HOS_NATIVE_SOLVER, on by default)HOS_NATIVE_SOLVERis now on by default (f64e6146), as docs/dev/standards.md asks: a switch ships at its value from the first push, not off because it is new.HOS_NATIVE_SOLVER=0gives main's lifted code. The row's kind iskill, unset reads as on (env_on_or), and the test runs it unset and=1(on),=0and=off(off) andcheck; it fails when the old read is put back. The read of a check value now tests that the switch is set first, which unset-as-on needs. Title and description updated to match.Recommendation: Request changes. The host code is written from the game's machine code down to its register allocation, which CONTRIBUTING.md's Licensing rule does not allow in the repository. Beyond that, about half of what this PR ships is code the game never calls: four host routines, each measured slower than the lift or not worth its hook, plus their tests and splice code. And the two routines that are wired have not been measured alone. Their own projection is about 0.05 to 0.13 % of game code, well inside the noise of any game measurement.
For players: two small leaves of the Havok rigid-body physics (the triangle set-up
f_80171a90and a three-vector tolerance testf_803be11c) run on host SSE code with the lift's exact bits, and fall back to the lifted code for any call the host cannot prove exact. The exactness work is careful. What it buys in the game is unproven, and the patch carries dead code.Does it help: Unproven. Offline, per call: the triangle set-up takes 45.7 ns against 74.7/82.1 ns, and the tolerance test 13.9 ns against 58.6/85.2 ns. In the game, the only numbers are -2.7 % / -2.4 % whole-process CPU in a tree that merged nine rewrite branches, with the switches all on together. That tree had this branch's earlier wiring, with the quaternion matrix wired and no tolerance test, so none of that gain can be credited to this head's two pieces. The PR's own projection from self time is ~0.05 % (Market) to ~0.13 % (Kakariko/DMT route) of game code. No fps, frame-time or 1 % low number exists for this head.
Other systems: Every OS and GPU vendor takes the same host path. It is CPU-only, with no renderer or driver path, so AMD, Intel and the Steam Deck can only differ through CPU speed. The hooks are on the supported build only. With the switch at 0, each call pays one volatile load and a branch not taken.
nx2/oot_patches.pyis in the lift key, so every player relifts once after updating. The build.py change compiles every host file with-DA64_MARK_BITMAP. On main no host file usesA64_MARK(only comments mention it), so it changes only this file's marks, anda64_gpu_pagesis defined inhost/mark_bitmap.c. The open risk the description names (mkw_vk capture copy+10 to 18 % in the combined tree) is still unattributed.Switches:
HOS_NATIVE_SOLVER(new,kill, on by default;=0lifted;checkcompares). Shipping on is consistent with standards.md, and the way back is there. Exactness is proven offline for both pieces, and in the game for the triangle set-up only (check mode, 531,947 calls, 0 mismatches, at the older wiring). The gain alone is unmeasured, so the default is unproven rather than wrong: keep it on, and let this head's own measurement decide. If that measurement shows nothing past the noise, the right answer is to remove the switch and the code, not to ship it off.Reviewed:
f64e61463againsta764653726(the PR's base), then the same commits rebased onto mainb62932a96(see below). Read in full:host/oot_solver.c, thenx2/oot_patches.pyhunks (SOLVER_NATIVE*,_solver_label_line,patch_solver_native, thepatch_fileandreporthooks),nx2/build.py,gpu/vk/switches.def,tests/test_mark_bitmap_host.py, and the parts oftests/test_oot_solver.pythat run code (the splice checks, the compile and run). Also read:nx2/rtgen.py'sA64_MARK/A64_MARK_BITMAPandnx2/fastfp.py's MXCSR window. Skimmed: the test's C reference fixtures. Nothing run (a code review).Fixed in this review: prepared, not pushed (the push to snaildev99's head branch was refused by the permission system here, so these are suggestions). The commits rebase onto main
b62932a96one by one, with one basic conflict inreport()innx2/oot_patches.py. Keep both blocks: main'ssprint presswarning first, then this PR'sHOS_NATIVE_SOLVERwarning. After that the tree is identical to f64e61463's diff (interdiff empty). On top of it, one nit commit (see the inline comments): the file header said "Off by default.",hos_solver_end's comment claimed a "last line of a run" that only tests produce, and the switch row's "check runs both on every call" read as if every call were compared.Checked and fine:
cls_bad: lanes outside zero or [2^-63, 2^63) are caught (0x20000000 = 2^-63, 0x5F000000 = 2^63, with an unsigned compare via the sign flip). Differences of in-class values are zero or at least 2^-86 in magnitude, so they are never subnormal. Products of in-class values stay in [2^-126, 2^126).CMP w8, #0with w8 in {0, ~0} is N = (w8 != 0), Z = !N, C = 1, V = 0, which is right. It writes no guest memory.x1+0x1a..0x6cin one mark, the arrays in theirs).a64_ff_begin) clear MXCSR's flags on entry and restore the caller's value on exit, and nothing else reads MXCSR as guest state (onlyhos_setjmpsaves and restores it). So a stray sticky flag cannot reach FPSR. Rounding mode and FTZ/DAZ cannot change an in-class result either.hos_solver_liverace point, not carried. The gate is the repository's existing idiom for flags the lifted code reads (hos_fixed_resolution,hos_scene_rgba16f,hos_oc_wait:extern volatile int). It is an aligned int that only goes 1 to 0, and a late reader of 1 just calls in, wheresolver_mode(), read atomically, returns 0. No wrong answer is possible.solver_insidestops the lifted routine from re-entering its own hook, and the lifted result is the one kept.protect_hearts(the supported build), placed once, and declared at file scope forsplit_giants.py. A second patch run changes nothing.test_oot_solver.pycompiles and runs only inside a temp dir. It downloads nothing and touches nothing outside the repository.Blocking (to change before merge)
Licensing: the host code is a translation of the game's compiled routines, not written from their behaviour. CONTRIBUTING.md's Licensing section allows "no decompiled game code or anything generated from the dump". Only facts of behaviour are allowed (a function's arguments, its result, the order of calls), described in your own words. This file goes well past that. It is not a literal one-statement-per-instruction transcription with the disassembly beside it (as 1911 was), but it reproduces the compiled code's own register allocation and leftovers, which come only from reading its instructions:
host/oot_solver.c:57: "the hook writes the bytes the game's instructions write and leaves every register and flag as they leave them".host/oot_solver.c:206-211: "State at the head (read from the instructions): x19 the callback object, x20 ..., [sp+0x30..0x90) the quad's vertices".host/oot_solver.c:332: the binary search'sADDS, then LT: N != V, reproduced as flag arithmetic.host/oot_solver.c:363:w->x[30] = 0x81a01514, a return address inside the game's code.host/oot_solver.c:473-514, the wired triangle set-up: temporaries named after the game's registers (m16,m17,v19,sq18,sq20,v6), andsetv(t->v[0..12], ...)rebuilding what the routine leaves in v0-v7 and v16-v20, with x8-x12 and NZCV (:507-508,:533-535).host/oot_solver.c:611-618, the wired tolerance test: "FABD, FMAX ... FCMGE", and "It leaves x0, x8, v0 = k1 & k2 & (k0, k1, k2, k3), v1 = ..., v3 = [x0+0x20]".host/oot_solver.c:564: "the game's own association and lanes, the fourth lanes included".tests/test_oot_solver.py:14, 282, 435, 462: the scalar references are "written here in plain scalar C from the routines' instructions".The plane maths (a cross product scaled by 1/sqrt(N . N)) and "three vectors within a tolerance" are behaviour, and could be written from it. Matching the routine's register state and its exact operation order, though, is a transcription of Havok's compiled code as it sits in the game, and there is no public source to cite for it. The design depends on that: the hook returns in place of the lifted function and must leave the registers its callers might read. So this cannot be fixed by rewording comments. It needs either a maintainer's ruling that this kind of register-exact host replacement is allowed (and the Licensing paragraph updated to say so), or a different design that does not carry the game's code in the repository. Until then it does not merge. The same reading applies to the siblings that hook lifted routines this way.
host/oot_solver.c:204-428, 558-610, 648-703, 705-764, 772-803, 833-864, 897-960. The mesh walk, quaternion matrix, box support vertex and capsule bounds are unreachable: no address of theirs is in the patch file, by design. The description and the comments say each was measured slower than today's lift, or not worth its hook. That is roughly half of this 960-line file. On top of it comeWALK_UNWIRED/LEAVES_UNWIREDand their fixtures intests/test_oot_solver.py, and the label-splice support inpatch_solver_native(_solver_label_line, thelabelstuples, thegototarget check, thetable/declsparameters), which only the unwired walk uses. Players get nothing from it, and every build compiles it. It also has to be kept correct against a lift it no longer matches: the quaternion comment already says the lift changed under it. Remove those four pieces, their counters (K_WALK,K_QUAT,K_CORNER,K_CAPBOX) and their tests, and keep them on a branch or in notes. Bring each back in its own PR when it can be wired and measured to help, as the "one gate at the routine's entry" idea for the walk would be.Not mergeable with main
b62932a96. Rebase as above: one conflict inreport(), keep both blocks.Waiting on the author (a run or a measurement; no edit does it)
=0against unset, interleaved, in the Market and a physics-heavy route. Give CPU seconds at a 60 cap, plus uncapped average fps, 1 % lows and frame times. With a projection of ~0.1 % of game code, say plainly if it is inside the noise, and then whether two leaves are worth the file. The Beanstalk request targetsa664ba4and has to be asked again for the new head.HOS_NATIVE_SOLVER=checkon this head withMISMATCHES 0for both pieces.Suggestions
SolverTri/SolverWithinand their apply/compare could share one generic "answer, then apply or compare" entry, instead of a hand-written copy per piece.Testing: the author reports offline Unicorn/driver comparisons (160k + 300k cases, 0 mismatches, deliberate faults caught) and the targeted game-free tests passing. In the game, the triangle set-up ran in check mode and on, in the merged tree at the older wiring (Windows, RTX 3090). This head has not been built or played, and Linux has not been built. CI on
f64e6146is blocked (pending). All of this holds the merge.Overlaps and merge order: the siblings by the same author (1898 HOS_MATH, also the physics narrow phase; 1899, 1902, 1903, 1904, 1905, 1909, 1910, 1911, 2001, 2036) all add rows to
gpu/vk/switches.defand hooks tonx2/oot_patches.py. Several carry the same "fix(build): the host's native hooks test the watch bitmap too" commit (thenx2/build.pyhunk) and add the sametests/test_mark_bitmap_host.py. Whichever merges first takes that commit, and the rest drop it on rebase: otherwise they hit add/add conflicts. 1898 should be checked against this one so the two do not hook the same narrow-phase functions. No order between them otherwise.@ -105,6 +105,7 @@ SW(HOS_MOVIE_PACE, kill, bool, "on: a 60 FPS cap is raised for a movie's pictureSW(HOS_MUTEX_TRACE, diag, int, "0: off", "Logs mutex waits longer than <ms>, with the thread that held the mutex.")SW(HOS_MUTEX_WATCH, diag, string, "off", "Logs every operation on the mutex at that guest address (hex) and its condition variables' signals.")SW(HOS_MW_WAKE, kill, enum, "only changes to objects linked into a multi-wait wake WaitAny", "=all: every event, semaphore or queue change wakes every WaitAny, as before.")SW(HOS_NATIVE_SOLVER, kill, enum, "on: pieces of the rigid-body physics on host code where it gives the game's bits exactly, the rest lifted", "=0 (or false, off, no): the rigid-body physics runs as lifted, as on main; 1 (or unset) runs pieces of the Havok rigid-body physics (the triangle set-up, a tolerance test of three vectors) on host code where it gives the game's bits exactly, the rest lifted; check runs both on every call and logs any difference (slow). An oot: solver line every 10 s counts the calls (host/oot_solver.c).")Nit: check runs the lifted code on every call but compares only the calls the host answered. Suggest "check runs the lifted code on every call and compares it with the host's answer where the host has one".
@ -0,0 +1,960 @@/* (longshot) HOS_NATIVE_SOLVER: pieces of the game's rigid-body physics (Havok Physics) on host code, bit for bit.Off by default.Nit: stale since
f64e6146. It is on by default now (HOS_NATIVE_SOLVER=0runs every call lifted).@ -0,0 +159,4 @@}/* (the totals so far, the calling thread's included: tests, and the last line of a run) */void hos_solver_end(void) {Nit: only the tests call
hos_solver_end. The game logs only the lines every 10 s, so there is no "last line of a run".@ -0,0 +201,4 @@return _mm_andnot_si128(_mm_or_si128(in, _mm_cmpeq_epi32(a, _mm_setzero_si128())), _mm_set1_epi32(-1));}/* ---- walk: one triangle of the mesh triangle walk f_81a012e0, from its loop head 0x81a013a0 to the callback's callBlocking: the walk, and below it quat (558), corner (648), capbox (705) and their entries (772-803, 833-864, 897-960), are never spliced in. Each was measured slower or not worth its hook. Remove them, their counters and their tests, and bring each back in its own PR once it is wired and measured to help.
@ -0,0 +470,4 @@const __m128 A = xyzz(b), B = xyzz(b + 12), C = xyzz(b + 24);const __m128 E1 = _mm_sub_ps(B, A), E2 = _mm_sub_ps(C, A);__m128i bad = _mm_or_si128(_mm_or_si128(cls_bad(A), cls_bad(B)), _mm_or_si128(cls_bad(C), _mm_or_si128(cls_bad(E1), cls_bad(E2))));const __m128 m16 = _mm_mul_ps(E2, _mm_shuffle_ps(E1, E1, _MM_SHUFFLE(1, 0, 2, 1))); /* E2 * E1.yzxy */Blocking (licensing): these temporaries and the
setv(t->v[...])below rebuild the game routine's own registers (v16-v20, v0-v7, x8-x12, NZCV), which can only come from its instructions. CONTRIBUTING.md's Licensing allows behaviour, not decompiled or dump-made code. See the summary.@ -810,0 +849,4 @@"""The physics pieces' hooks (SOLVER_NATIVE): (s, the functions of SOLVER_NATIVE in s with their hooks, made now orbefore, and the functions of SOLVER_NATIVE in s whose places were not all found, left as they are). table and decls:another table of the same shape and its declarations (a piece that is not wired here: its test and its offlineproof make its splices with this same code)"""With the walk gone, the label-splice half of this (
_solver_label_line,labels, the goto-target check,table/decls) has no user. The leaves need only the head insert.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.