WIP: perf(host): the box-against-view test and the occlusion grid's build on host code, bit for bit (HOS_NATIVE_VISIBILITY, on by default) #1902

Draft
snaildev99 wants to merge 12 commits from snaildev99/Longshot:perf/native-visibility into main
Contributor

Summary

Two of the game's visibility routines run on host code with the game's exact results, behind a new switch, on by default (HOS_NATIVE_VISIBILITY=0 gives the lifted code): the box-against-view test the per-model visibility job calls for every model (f_80029240, 42 instructions, a leaf) and the build of the coarse occlusion depth grid that job tests models against, once a frame (f_808226e0, 1,466 instructions with its callee f_80823e00 and the logf it calls). In the Market profile they hold 0.46 % and 0.69 % of the game's code time (0.56 % and 0.77 % on a Kakariko and trail route), nearly all on the module workers.

The goal is less CPU work for the same frames. A call the host cannot prove exact runs the lifted routine, with nothing written: the box test answers when every compared lane, product and sum is in a class where the PC's IEEE arithmetic gives Arm's bits (no NaN, infinity or subnormal; products and distances zero or in [2^-62, 2^62); sums zero or at least 2^-126) and the lift's own float gate is open (FPCR 0 or FZ, FPSR.IXC set); the grid build checks every float operation as it is made (operands zero or normal, results finite and above the smallest normal or an exact zero, no division by zero, no conversion out of range), saves every region it writes first and puts them back on a decline, and needs a hardware FMA (x86-64-v3).

How: nx2/oot_patches.py (patch_visibility, the supported build only) puts a gate first in each lifted function: hos_vis_on_<addr> is 1 until the first call has read the switch, then 0 unless it is on, so with the switch off the gate is one load and an untaken branch. host/oot_visibility.c answers with host/visibility_kernels.c (the box test: SSE2 with the planes four at a time, one plane a lane, the same operations on the same operands as the game's; a plain C version for other hosts and plane counts above 6) and host/visibility_occlusion.c (the grid: the game's operations in the game's order, its writes marked for the GPU mirror with HOS_WRITTEN). The hook writes every register the routine leaves (box: x0, x8, x9, x10, v0..v3, NZCV; grid: the callee-saved registers, v8..v15's upper halves cleared as the routine's restore leaves them). Each gated function is also declared at file scope (VIS_FWD), because nx2/split_giants.py may put the gate in a piece written before f_<addr>.

Read first: host/oot_visibility.c, host/visibility_kernels.c, host/visibility_occlusion.c (vis_occlusion_808226e0), nx2/oot_patches.py (patch_visibility), tests/test_native_visibility.py, the two rows in gpu/vk/switches.def.

Testing

Tested: Windows 11, RTX 3090 (NVIDIA 610.88), Ryzen 9 5900X. In the game, with this branch's code (at 38f0ad29, before the switch rows and the declaration changes) merged with other rewrite branches into a test tree on main 068fb8eb, HOS_NATIVE_VISIBILITY=check, the Market and Hyrule Field for about five minutes: the box test answered 6,291,456 of 6,291,456 calls on the host and the grid build 4,095 of 4,096, with no DIFFERENT line (every answered call compared with the lifted routine, register for register and byte for byte). Then the CPU measurement below (test tree test/rewrites @ e614ef49, this code at 9e7311d1, HOS_NATIVE_VISIBILITY=1 with the other eight branches' switches on): all 15 runs held 60 fps. This head (the rebase onto current main and the upkeep commit) is not built or played.

Exactness offline (Unicorn runs the game's own routine on the executable; a driver runs the real host files): the box test on 1,000,000 calls (frusta with boxes inside, outside and across, 127,050 boxes exactly on a plane, the volume's own box touching the model's, plane counts 0 to 9, 100, negative and extreme, edge values, cancelling products, random bits, eight FPCR and many FPSR settings): 661,041 answered natively, 0 mismatches in every x register, SP, all 32 q registers (128 bits), NZCV, FPSR and memory, in the SSE build and the plain C build. Five deliberately broken versions are caught (11 to 3,514 mismatching cases of 10,000 each). The grid build on 19,625 calls that the game's routine finished in Unicorn (views with projections landing exactly on .5, on the grid's edges and on +0 / -0; grids 1x1 to 24x16 with sub-cells 1 to 4; tile levels 0 to 4; height fields up to 12x12; stale contents; edge values): 11,399 answered natively, 0 mismatches in every byte of every block, FPSR, x19 to x30, SP and q8 to q15, in the FMA build and the build without (which answers only the routine's "off" path); 375 cases the game's routine did not finish were all declined with memory as it was; five broken versions are caught (21 to 218 cases), the last one (the running minimum's tie rule for -0 / +0) after a generator of signed-zero ties was added (43 of 4,000). Run again on this branch's host files after the rebase (they are byte for byte the ones proven; the harnesses run the hooks against the stock lift's routines, so the gate's declaration is not in what they compile): the box test 250,000 calls, 164,889 answered natively, 0 mismatches in both builds; the grid build 19,644 finished calls, 11,484 answered natively, 0 mismatches in both builds, 356 unfinished all declined.

Game-free checks at this head: tests/test_native_visibility.py passes (the gates once each, first in their functions, declared at file scope, the supported build only; both box kernels against a model of the game's instructions on 60,000 cases; the declines; the hooks off, on, check and on another build; the grid on an empty scene against a result worked out by hand), and so do tests/test_switch_table.py (731 switches), tests/test_patch_hooks_defined.py and 36 of the other 41 tests that read nx2/oot_patches.py or nx2/entry_hooks.py. The other five fail here and on main a104182b alike: test_dev_packs, test_high_fps_fixes, test_libc_hooks, test_lift_cache, test_song_note_cue. tests/run_all.py was not run.

Still to test:

  • Linux, any GPU: the Market for a few minutes and Hyrule Field for a few minutes, this branch with HOS_NATIVE_VISIBILITY=0 against unset (on), and main against this branch with the switch unset; the game's CPU seconds with the frame rate capped at 60, and the frame times uncapped. Expecting less CPU time with the switch on, the same frame rate or better, no cost with it =0, and the picture as without it (@Beanstalk linux).
  • This head on Windows: HOS_NATIVE_VISIBILITY=check in the Market and Hyrule Field, expecting the visibility: count lines with calls taken and no DIFFERENT line; then =1, expecting no model missing or popping (by hand).

Measurements

Verdict: less CPU work, measured together with the other rewrite branches, not yet alone. In a test tree that merges nine rewrite branches (named in the baseline below; the cloth and particle effect rewrites are not among them), at a 60 fps cap in the Market, the nine on used 2.7 % less CPU standing and 2.4 % less turning (the whole process) than main, and the hooks in with every switch off cost nothing measurable (-0.3 % / -0.2 %). How much of that is this branch is not separated: the Test Farm request asks for this branch's own number.

Baseline: main 068fb8eb (#1802) against the test tree test/rewrites @ e614ef49 (main 068fb8eb with nine rewrite branches: the seven maths units, the maths library, animation, bounds, crowd, visibility (this branch at 9e7311d1), glue, the effect triggers with the six-channel reverb, and the solver; the cloth and particle effect rewrites, the two- and four-channel reverb and later pieces of other branches are not among them), lifted with A64_FPXKEEP=0 so that the lift is main's apart from the hooks. Not this PR alone: the run-time switches were all on together. Main has moved 28 commits since 068fb8eb; this branch was not measured on current main.
Setup: Windows 11, Ryzen 9 5900X, RTX 3090, 3840x2160; the child Market, hands-free; frame rate capped at 60 (all 15 runs held 60); each run 30 s standing, then 30 s turning the camera; five variants interleaved over three rounds, the PC otherwise idle. CPU seconds per second of play (cores busy on average) of the whole process, from the processor's cycle counter; the median of three runs (the three in brackets).

Before: main @ 068fb8eb After, every switch off: test/rewrites @ e614ef49 After, the nine branches on: test/rewrites @ e614ef49 Δ off Δ on
Market standing, cores 5.156 [5.144 5.156 5.161] 5.140 [5.303 5.140 5.125] 5.017 [5.007 5.017 5.049] -0.3 % -2.7 %
Market turning, cores 5.639 [5.676 5.627 5.639] 5.630 [5.722 5.630 5.628] 5.504 [5.504 5.498 5.577] -0.2 % -2.4 %

The off figures are inside the runs' spread. The process includes the renderer's threads (about 1.4 cores standing), which the nine do not touch; on the executors the nine on were -4.1 % / -4.0 %. The module workers, where these two routines run, went 0.847 to 0.844 cores standing and 0.884 to 0.875 turning with the nine on (they also run other branches' routines).

Offline (CPU time a call, not the game; Ryzen 9 5900X with other jobs running; best of 5; the hook timed through its gate). Main's form is the lift with registers as C locals that main writes for these functions, approximated offline from the stock lift with the translator kit's own localize_regs (not main's lifter); a gated function is written in the older context form, and "switch off" is that form plus the gate.

routine, calls main's form (approximated) this PR, switch on factor switch off against main's form
f_80029240, calls that test planes 79-199 ns 45-61 ns 1.3x to 3.4x -3 % to +10 %
f_80029240, calls that end at the volume's box or have no planes 9-22 ns 28-35 ns 0.3x to 0.64x
f_808226e0, five frames' states 88-152 us 44-76 us 1.7x to 2.1x +1 % to +7 %

The box test's fixed cost through the hook is about 20 ns, more than a whole lifted call that stops before the planes; the share of such calls in the game is not measured. The switch-off column was measured before the hook's declaration took A64_NOESC (the upkeep commit), and the game measurement above found no off cost at the process's scale.

The switch

HOS_NATIVE_VISIBILITY (new, run time, kill, on by default): 0 off (the lifted routines, main's results); unset or 1 on (host code where it is exact, the rest lifted); check runs the host code into a copy, then the lifted routine, compares registers and every region written, logs visibility: f_<addr> DIFFERENT from the lifted routine: ... on a difference and keeps the lifted result. HOS_NATIVE_VISIBILITY_LOG=1 (always in check mode) logs visibility: <routine> <n> calls: taken <k>, lifted for float mode / range / input / path ..., checked: different <d> every 2^20 calls (2^12 for the grid). Another game build leaves it off with a log line. It ships on: docs/dev/standards.md has a switch ship at its value from the first push, never off because it is new, and the measurement with the rewrite branches together says on. HOS_NATIVE_VISIBILITY=0 gives main's lifted code. This PR's own measurement (the farm request) can still turn it off before it leaves WIP.

Risks

What could break: a wrong answer from the box test would cull a model that should be drawn or draw one that should be culled (a model missing or popping); a wrong grid would do the same through the occlusion test. Every answered call in the in-game check run matched the lifted routine. The grid build saves and restores up to its whole output on a decline. Open: in the CPU measurement the renderer's mkw_vk capture copy thread used 10 to 18 % more CPU (0.009 to 0.017 cores) in both variants with the nine on and in neither without; that thread copies guest memory the game marked as written, and the native kernels mark what they write (HOS_WRITTEN; here the grid build, the box test writes no memory). Which rewrite causes it is not determined; this branch is one of several candidates.

Turning it off: HOS_NATIVE_VISIBILITY=0.

After updating: the routines run on the host by default, with the game's exact results; saves, longshot_options.ini and the caches do not change.

Checklist

Ready for review

  • One topic, on current main; the title is a Conventional Commit.
  • Tested on my own PC, in the game for a change the game runs, as Testing says. (The code ran in the game in a merged test tree on main 068fb8eb, in check mode and on; this head is not built or played.)
  • mise run test passes (any failure that also fails on main is named). (The 39 tests that read the changed files pass, the five failures are main's; the whole suite not run.)
  • Measured against main, or Measurements says why there is nothing to measure. (Measured with nine rewrite branches merged; this branch alone not yet.)
  • No game files, build output or hand-edited generated files; third_party/ untouched; nothing GPL-incompatible.

When it applies

  • A switch added or changed: it ships at the default the measurements chose, with its row in gpu/vk/switches.def. (The rows are there; on by default, =0 gives main's lifted code; this PR's own measurement can still turn it off.)
  • A check under tests/ that reads changed source is updated, not weakened. (tests/test_native_visibility.py, new; its "no other function was gated" slice finds the functions by their head lines.)
  • Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows: in the merged test tree. Linux: not built.)
  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.

Ready to merge

  • Everything in Still to test has run, its results in Tested.
  • Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers").
  • Approved in review, every comment answered.
  • CI passes on the head; on current main, measured again or a comment says why the numbers still hold.

🤖 Generated with Claude Code

## Summary Two of the game's visibility routines run on host code with the game's exact results, behind a new switch, on by default (`HOS_NATIVE_VISIBILITY=0` gives the lifted code): the box-against-view test the per-model visibility job calls for every model (f_80029240, 42 instructions, a leaf) and the build of the coarse occlusion depth grid that job tests models against, once a frame (f_808226e0, 1,466 instructions with its callee f_80823e00 and the logf it calls). In the Market profile they hold 0.46 % and 0.69 % of the game's code time (0.56 % and 0.77 % on a Kakariko and trail route), nearly all on the module workers. The goal is less CPU work for the same frames. A call the host cannot prove exact runs the lifted routine, with nothing written: the box test answers when every compared lane, product and sum is in a class where the PC's IEEE arithmetic gives Arm's bits (no NaN, infinity or subnormal; products and distances zero or in [2^-62, 2^62); sums zero or at least 2^-126) and the lift's own float gate is open (FPCR 0 or FZ, FPSR.IXC set); the grid build checks every float operation as it is made (operands zero or normal, results finite and above the smallest normal or an exact zero, no division by zero, no conversion out of range), saves every region it writes first and puts them back on a decline, and needs a hardware FMA (x86-64-v3). How: `nx2/oot_patches.py` (`patch_visibility`, the supported build only) puts a gate first in each lifted function: `hos_vis_on_<addr>` is 1 until the first call has read the switch, then 0 unless it is on, so with the switch off the gate is one load and an untaken branch. `host/oot_visibility.c` answers with `host/visibility_kernels.c` (the box test: SSE2 with the planes four at a time, one plane a lane, the same operations on the same operands as the game's; a plain C version for other hosts and plane counts above 6) and `host/visibility_occlusion.c` (the grid: the game's operations in the game's order, its writes marked for the GPU mirror with `HOS_WRITTEN`). The hook writes every register the routine leaves (box: x0, x8, x9, x10, v0..v3, NZCV; grid: the callee-saved registers, v8..v15's upper halves cleared as the routine's restore leaves them). Each gated function is also declared at file scope (`VIS_FWD`), because `nx2/split_giants.py` may put the gate in a piece written before `f_<addr>`. Read first: `host/oot_visibility.c`, `host/visibility_kernels.c`, `host/visibility_occlusion.c` (`vis_occlusion_808226e0`), `nx2/oot_patches.py` (`patch_visibility`), `tests/test_native_visibility.py`, the two rows in `gpu/vk/switches.def`. ## Testing **Tested:** Windows 11, RTX 3090 (NVIDIA 610.88), Ryzen 9 5900X. In the game, with this branch's code (at 38f0ad29, before the switch rows and the declaration changes) merged with other rewrite branches into a test tree on main 068fb8eb, `HOS_NATIVE_VISIBILITY=check`, the Market and Hyrule Field for about five minutes: the box test answered 6,291,456 of 6,291,456 calls on the host and the grid build 4,095 of 4,096, with no `DIFFERENT` line (every answered call compared with the lifted routine, register for register and byte for byte). Then the CPU measurement below (test tree `test/rewrites` @ e614ef49, this code at 9e7311d1, `HOS_NATIVE_VISIBILITY=1` with the other eight branches' switches on): all 15 runs held 60 fps. This head (the rebase onto current main and the upkeep commit) is not built or played. Exactness offline (Unicorn runs the game's own routine on the executable; a driver runs the real host files): the box test on 1,000,000 calls (frusta with boxes inside, outside and across, 127,050 boxes exactly on a plane, the volume's own box touching the model's, plane counts 0 to 9, 100, negative and extreme, edge values, cancelling products, random bits, eight FPCR and many FPSR settings): 661,041 answered natively, 0 mismatches in every x register, SP, all 32 q registers (128 bits), NZCV, FPSR and memory, in the SSE build and the plain C build. Five deliberately broken versions are caught (11 to 3,514 mismatching cases of 10,000 each). The grid build on 19,625 calls that the game's routine finished in Unicorn (views with projections landing exactly on .5, on the grid's edges and on +0 / -0; grids 1x1 to 24x16 with sub-cells 1 to 4; tile levels 0 to 4; height fields up to 12x12; stale contents; edge values): 11,399 answered natively, 0 mismatches in every byte of every block, FPSR, x19 to x30, SP and q8 to q15, in the FMA build and the build without (which answers only the routine's "off" path); 375 cases the game's routine did not finish were all declined with memory as it was; five broken versions are caught (21 to 218 cases), the last one (the running minimum's tie rule for -0 / +0) after a generator of signed-zero ties was added (43 of 4,000). Run again on this branch's host files after the rebase (they are byte for byte the ones proven; the harnesses run the hooks against the stock lift's routines, so the gate's declaration is not in what they compile): the box test 250,000 calls, 164,889 answered natively, 0 mismatches in both builds; the grid build 19,644 finished calls, 11,484 answered natively, 0 mismatches in both builds, 356 unfinished all declined. Game-free checks at this head: `tests/test_native_visibility.py` passes (the gates once each, first in their functions, declared at file scope, the supported build only; both box kernels against a model of the game's instructions on 60,000 cases; the declines; the hooks off, on, check and on another build; the grid on an empty scene against a result worked out by hand), and so do `tests/test_switch_table.py` (731 switches), `tests/test_patch_hooks_defined.py` and 36 of the other 41 tests that read `nx2/oot_patches.py` or `nx2/entry_hooks.py`. The other five fail here and on main a104182b alike: `test_dev_packs`, `test_high_fps_fixes`, `test_libc_hooks`, `test_lift_cache`, `test_song_note_cue`. `tests/run_all.py` was not run. **Still to test:** - Linux, any GPU: the Market for a few minutes and Hyrule Field for a few minutes, this branch with `HOS_NATIVE_VISIBILITY=0` against unset (on), and main against this branch with the switch unset; the game's CPU seconds with the frame rate capped at 60, and the frame times uncapped. Expecting less CPU time with the switch on, the same frame rate or better, no cost with it `=0`, and the picture as without it (@Beanstalk linux). - This head on Windows: `HOS_NATIVE_VISIBILITY=check` in the Market and Hyrule Field, expecting the `visibility:` count lines with calls taken and no `DIFFERENT` line; then `=1`, expecting no model missing or popping (by hand). ## Measurements **Verdict:** less CPU work, measured together with the other rewrite branches, not yet alone. In a test tree that merges nine rewrite branches (named in the baseline below; the cloth and particle effect rewrites are not among them), at a 60 fps cap in the Market, the nine on used 2.7 % less CPU standing and 2.4 % less turning (the whole process) than main, and the hooks in with every switch off cost nothing measurable (-0.3 % / -0.2 %). How much of that is this branch is not separated: the Test Farm request asks for this branch's own number. Baseline: main 068fb8eb (#1802) against the test tree `test/rewrites` @ e614ef49 (main 068fb8eb with nine rewrite branches: the seven maths units, the maths library, animation, bounds, crowd, visibility (this branch at 9e7311d1), glue, the effect triggers with the six-channel reverb, and the solver; the cloth and particle effect rewrites, the two- and four-channel reverb and later pieces of other branches are not among them), lifted with `A64_FPXKEEP=0` so that the lift is main's apart from the hooks. Not this PR alone: the run-time switches were all on together. Main has moved 28 commits since 068fb8eb; this branch was not measured on current main. Setup: Windows 11, Ryzen 9 5900X, RTX 3090, 3840x2160; the child Market, hands-free; frame rate capped at 60 (all 15 runs held 60); each run 30 s standing, then 30 s turning the camera; five variants interleaved over three rounds, the PC otherwise idle. CPU seconds per second of play (cores busy on average) of the whole process, from the processor's cycle counter; the median of three runs (the three in brackets). | | Before: main @ 068fb8eb | After, every switch off: test/rewrites @ e614ef49 | After, the nine branches on: test/rewrites @ e614ef49 | Δ off | Δ on | |---|---|---|---|---|---| | Market standing, cores | 5.156 [5.144 5.156 5.161] | 5.140 [5.303 5.140 5.125] | 5.017 [5.007 5.017 5.049] | -0.3 % | -2.7 % | | Market turning, cores | 5.639 [5.676 5.627 5.639] | 5.630 [5.722 5.630 5.628] | 5.504 [5.504 5.498 5.577] | -0.2 % | -2.4 % | The off figures are inside the runs' spread. The process includes the renderer's threads (about 1.4 cores standing), which the nine do not touch; on the executors the nine on were -4.1 % / -4.0 %. The module workers, where these two routines run, went 0.847 to 0.844 cores standing and 0.884 to 0.875 turning with the nine on (they also run other branches' routines). Offline (CPU time a call, not the game; Ryzen 9 5900X with other jobs running; best of 5; the hook timed through its gate). Main's form is the lift with registers as C locals that main writes for these functions, approximated offline from the stock lift with the translator kit's own `localize_regs` (not main's lifter); a gated function is written in the older context form, and "switch off" is that form plus the gate. | routine, calls | main's form (approximated) | this PR, switch on | factor | switch off against main's form | |---|---|---|---|---| | f_80029240, calls that test planes | 79-199 ns | 45-61 ns | 1.3x to 3.4x | -3 % to +10 % | | f_80029240, calls that end at the volume's box or have no planes | 9-22 ns | 28-35 ns | 0.3x to 0.64x | | | f_808226e0, five frames' states | 88-152 us | 44-76 us | 1.7x to 2.1x | +1 % to +7 % | The box test's fixed cost through the hook is about 20 ns, more than a whole lifted call that stops before the planes; the share of such calls in the game is not measured. The switch-off column was measured before the hook's declaration took `A64_NOESC` (the upkeep commit), and the game measurement above found no off cost at the process's scale. ### The switch `HOS_NATIVE_VISIBILITY` (new, run time, `kill`, on by default): `0` off (the lifted routines, main's results); unset or `1` on (host code where it is exact, the rest lifted); `check` runs the host code into a copy, then the lifted routine, compares registers and every region written, logs `visibility: f_<addr> DIFFERENT from the lifted routine: ...` on a difference and keeps the lifted result. `HOS_NATIVE_VISIBILITY_LOG=1` (always in check mode) logs `visibility: <routine> <n> calls: taken <k>, lifted for float mode / range / input / path ..., checked: different <d>` every 2^20 calls (2^12 for the grid). Another game build leaves it off with a log line. It ships on: docs/dev/standards.md has a switch ship at its value from the first push, never off because it is new, and the measurement with the rewrite branches together says on. `HOS_NATIVE_VISIBILITY=0` gives main's lifted code. This PR's own measurement (the farm request) can still turn it off before it leaves WIP. ## Risks **What could break:** a wrong answer from the box test would cull a model that should be drawn or draw one that should be culled (a model missing or popping); a wrong grid would do the same through the occlusion test. Every answered call in the in-game check run matched the lifted routine. The grid build saves and restores up to its whole output on a decline. Open: in the CPU measurement the renderer's `mkw_vk capture copy` thread used 10 to 18 % more CPU (0.009 to 0.017 cores) in both variants with the nine on and in neither without; that thread copies guest memory the game marked as written, and the native kernels mark what they write (`HOS_WRITTEN`; here the grid build, the box test writes no memory). Which rewrite causes it is not determined; this branch is one of several candidates. **Turning it off:** `HOS_NATIVE_VISIBILITY=0`. **After updating:** the routines run on the host by default, with the game's exact results; saves, `longshot_options.ini` and the caches do not change. ## Checklist ### Ready for review - [x] One topic, on current `main`; the title is a Conventional Commit. - [ ] Tested on my own PC, in the game for a change the game runs, as Testing says. (The code ran in the game in a merged test tree on main 068fb8eb, in check mode and on; this head is not built or played.) - [ ] `mise run test` passes (any failure that also fails on `main` is named). (The 39 tests that read the changed files pass, the five failures are main's; the whole suite not run.) - [ ] Measured against `main`, or Measurements says why there is nothing to measure. (Measured with nine rewrite branches merged; this branch alone not yet.) - [x] No game files, build output or hand-edited generated files; `third_party/` untouched; nothing GPL-incompatible. ### When it applies - [ ] A switch added or changed: it ships at the default the measurements chose, with its row in `gpu/vk/switches.def`. (The rows are there; on by default, `=0` gives main's lifted code; this PR's own measurement can still turn it off.) - [x] A check under `tests/` that reads changed source is updated, not weakened. (`tests/test_native_visibility.py`, new; its "no other function was gated" slice finds the functions by their head lines.) - [ ] Host code builds on Windows (llvm-mingw) and Linux, with UTF-8 paths. (Windows: in the merged test tree. Linux: not built.) - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. ### Ready to merge - [ ] Everything in Still to test has run, its results in Tested. - [ ] Tier 1 hardware it can affect is tested (docs/dev/standards.md, "Hardware tiers"). - [ ] Approved in review, every comment answered. - [ ] CI passes on the head; on current `main`, measured again or a comment says why the numbers still hold. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
f_80029240 is the leaf the render calc's per-model visibility job
(f_8009bb80) calls for every model: the view's box against the model's
(FCMGT on four lanes, when the view's box is in use), then each plane against
the model box's corner on the plane's negative side (FCMGE, BSL, FMUL, the
lanes added 2+0 and 3+1, FADDP, the distance, FCMP); the model is culled (0)
at the first plane it is wholly in front of, else seen (1).

vis_box_80029240 does the same with SSE2, the planes four at a time with one
plane a lane (each lane gets the same operations on the same operands as the
game's), and returns every register the routine leaves (x0, x8, x9, x10,
v0..v3, NZCV). vis_box_80029240_scalar is the plain C one (one plane at a time).
Both answer only inside a class where the host's single precision is Arm's bit
for bit and no flag but IXC is raised: compared values not NaN or subnormal,
multiplied values and distances zero or in [2^-62, 2^62), every sum zero or not
tiny (an exact tiny sum is flushed under FZ on Arm). Anything else is declined,
writing nothing.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
nx2/oot_patches.py (supported build only) puts a gate first in the lifted
f_80029240, as the native maths units have: hos_vis_on_80029240 is 1 until the
first call has read the switch, then 0 unless it is on, so off it costs one
load and an untaken branch. host/oot_visibility.c answers with
host/visibility_kernels.c when HOS_NATIVE_VISIBILITY is on (off by default),
the lift's own float gate is open (RECOMP_FPX_OPEN: FPCR at its default but FZ,
FPSR.IXC set), the volume, its planes and the box are inside the guest window
and not across a 4 GB line, and the kernel takes the values; it writes every
register the routine leaves. HOS_NATIVE_VISIBILITY=check runs the kernel and
the lifted routine and logs a difference (the lifted result stays);
HOS_NATIVE_VISIBILITY_LOG=1 counts calls taken and declined. Another game build
leaves it off.

tests/test_native_visibility.py: the gate (once, idempotent, supported build
only); both kernels against a model of the game's instructions on 60,000
frusta, on-plane boxes, the view's box and plane counts, and their declines;
the hook off / on / check / another build, its declines, the 4 GB line.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
f_808226e0 runs once a frame from f_80822260: it copies the view into the
occlusion object, makes the column and row plane records of the occluder
height field and of the grid (f_80823e00, which ends in a logf), clears the
depth grid and the first tile level, projects each occluder column's eight
corners to a screen rectangle (a low column with its own depths, a tall one
with the view's), writes its depth and sub-cell coverage into the cells it
covers, folds the cells into the first tile level with the full / partial / far
counts, and reduces the tile levels into each other. The per-model visibility
job reads the grid and the tiles to cull models behind the occluders.

vis_occlusion_808226e0 does all of it in C with the game's operations in the
game's order (each FMADD / FMSUB / FNMADD / FNMSUB one fused multiply-add, the
compare-and-select pairs, FMINNM / FMAXNM with Arm's zeros, FCVTZS, SDIV with
Arm's zero divisor, the game's logf on the table in the image). Every float
operation is checked as it is made: operands zero or normal, results finite
and above the smallest normal or an exact zero, no division by zero, no
conversion out of range; then the host's IEEE arithmetic is Arm's and no flag
but IXC is raised. Every region it writes is copied aside first; a value
outside that class, the cycle counter's or the set-up's path, regions that
overlap or a build without FMA put everything back and leave the call to the
game.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
nx2/oot_patches.py gates f_808226e0 as it gates f_80029240. With the switch
on, host/oot_visibility.c answers the call with host/visibility_occlusion.c
when the lift's float gate is open and the kernel takes the call, marks every
region written for the GPU mirror, and clears v8..v15's upper halves (the
routine's LDP d8..d15 restore leaves them so; its one caller returns right
after the call). In check mode the kernel's regions are kept and put back, the
lifted routine runs, and a difference is logged. The counts are per routine.

tests/test_native_visibility.py: both gates; the grid's hook off / on / check,
an empty scene against the result worked out by hand, the path that is off, and
the declines leaving memory as it was.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The check mode's copy of the host's regions had room for 16; the grid's build
writes fewer, but a region past the 16th would have gone uncompared without a
word. The list now grows as needed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
nx2/split_giants.py cuts a giant lifted function into piece functions written before f_<addr>, so a gate in the first
piece must find f_<addr> declared above it (VIS_FWD, as on test/rewrites); the test checks the declaration comes
before the head.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
refactor(host): the visibility hooks declared with A64_NOESC; the test finds a function by its head line
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 16s
beanstalk/approval approved by llun (reviewed)
beanstalk/linux 2/1 fail
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
5f0c55d218
VIS_DECL declares hos_vis_<addr>(Ctx *A64_NOESC c, A64Fn lifted): without it the compiler takes the lifted
function's context for escaped at the gate's call, which costs the gated function with the switch off. Only the
declaration in the lifted code changes; the host code does not.

tests/test_native_visibility.py's "no other function was gated" slice looked for "void f_808226e0(", which now
first finds the file-scope declaration VIS_FWD puts at the file's head (the slice came out empty). It looks for
the head line "void f_<addr>(Ctx *__restrict A64_NOESC c) {".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Contributor

Tested in game: Ran in the game in a test tree of nine merged rewrite branches on main 068fb8eb (Windows, RTX 3090): in check mode in the Market and Hyrule Field the box test answered 6,291,456 of 6,291,456 calls and the grid build 4,095 of 4,096 with no difference from the lifted routines, and with the switch on in a 15-run CPU measurement at a 60 fps cap (every run held 60); this head itself is not built or played.

Tested in game: Ran in the game in a test tree of nine merged rewrite branches on main 068fb8eb (Windows, RTX 3090): in check mode in the Market and Hyrule Field the box test answered 6,291,456 of 6,291,456 calls and the grid build 4,095 of 4,096 with no difference from the lifted routines, and with the switch on in a 15-run CPU measurement at a 60 fps cap (every run held 60); this head itself is not built or played.
Author
Contributor

@Beanstalk linux: this branch with HOS_NATIVE_VISIBILITY unset against HOS_NATIVE_VISIBILITY=1 (the same build), and main against this branch with the switch unset, each in the child Market for a few minutes and in Hyrule Field by day for a few minutes. Measure the game's CPU seconds with the frame rate capped at 60 (on a PC that holds 60 there), standing and turning the camera, and the frame times uncapped. Expect less CPU time with the switch at 1, the same frame rate or better, no extra CPU time with the switch unset against main, the picture as without the switch (no model missing or popping), and a "visibility: the box-against-view test ... on the host" line in the log with the switch set.

@Beanstalk linux: this branch with HOS_NATIVE_VISIBILITY unset against HOS_NATIVE_VISIBILITY=1 (the same build), and main against this branch with the switch unset, each in the child Market for a few minutes and in Hyrule Field by day for a few minutes. Measure the game's CPU seconds with the frame rate capped at 60 (on a PC that holds 60 there), standing and turning the camera, and the frame times uncapped. Expect less CPU time with the switch at 1, the same frame rate or better, no extra CPU time with the switch unset against main, the picture as without the switch (no model missing or popping), and a "visibility: the box-against-view test ... on the host" line in the log with the switch set.
Member

@snaildev99 only members of BigBeans can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.

@snaildev99 only members of `BigBeans` can start the farm. One of them can approve this request by reacting 👍 to it or to this comment.
Member

@Beanstalk

@Beanstalk
Member
Beanstalk test farm · an earlier run: commit 5f0c55d (the newer results are in a comment below)

Commit 5f0c55d · approved by llun (reviewed) · dashboard

Summary, written by AI (gpt-6.1-terra) from the numbers below:

The change failed on Linux on both reported PCs, which the farm marked as outliers. With the visibility switch off, screenshot differences were 0.1422 on the RTX 5070 and 0.5873 on the RTX 5080, both above the 0.02 limit; on the 5080 with the switch on, the difference was 0.4989 and Market standing CPU time rose from 98.91 to 120.70 seconds (+22%, against a 10% limit). Several p99 frame-time comparisons were inconclusive because they lacked two usable starts per side. The switch-on scenario was marked crashed on the 5070, although its candidate recorded exit code 0.

Done (started 19:55 UTC, last report 23:56 UTC). Comment @Beanstalk to run it again.

Target Result Reports VRAM peak (MB) Crashes
linux ❌ fail (an outlier) 2 1 2 — none

visibility-route · frame time p99 (ms)

Target main HOS_NATIVE_VISIBILITY=0 (reference) HOS_NATIVE_VISIBILITY=1
linux 18.8 18.6 18.0 (−3%)

Why it failed

  • linux: visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) screenshot.max_diff: limit 0.0
  • linux: visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) field-standing.cpu_s: HOS_NATIVE_VISIBILITY=1 vs HOS_NATIVE_VISIBILITY=0 (PR): 69.1 → 75.2 (+8.7%); limit 5.0; the PR's own cells: HOS_NATIVE_VISIBILITY=1 against HOS_NATIVE_VISIBILITY=0 (HOS_NATIVE_VISIBILITY=0 (PR only))
  • linux: visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) market-standing.cpu_s.max_delta_pct: 98.9 → 120.7 (+22.0%); limit 10.0
  • linux: visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) screenshot.max_diff: limit 0.0
  • linux: visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) market-standing.cpu_s: HOS_NATIVE_VISIBILITY=1 vs HOS_NATIVE_VISIBILITY=0 (PR): 112.2 → 120.7 (+7.5%); limit 5.0; the PR's own cells: HOS_NATIVE_VISIBILITY=1 against HOS_NATIVE_VISIBILITY=0 (HOS_NATIVE_VISIBILITY=0 (PR only))
What was tested

What will be tested · AI plan (gpt-6.1-terra)

On Linux, the plan compares main with the PR’s visibility switch off and on in the child Market and Hyrule Field. It measures capped CPU time at two camera angles, uncapped frame times, and matching views for missing models or other visual changes; it also times lift builds. Crashes, failure to activate the switch, substantial CPU or frame-time regressions, or picture differences fail the plan. Gameplay takes roughly half an hour per tester, with timed builds counted separately.

  • Where: linux (1-2 reports).
  • Matrix: 2 cells, each on main and the PR: HOS_NATIVE_VISIBILITY 0 or 1 (PR only).
  • visibility-route: from a new save (settings frame_rate_cap=0), then teleport to Market, measure market-standing for 30 s (after 5 s warm-up), take the picture market-front to compare with main, move the stick, take the picture market-turned to compare with main, measure market-turned for 20 s (after 2 s warm-up), teleport to Hyrule Field, measure field-standing for 30 s (after 5 s warm-up), take the picture field-front to compare with main, take the picture field-turned to compare with main, measure field-turned for 20 s (after 2 s warm-up), set frame_rate_cap to 6, measure field-uncapped for 30 s (after 5 s warm-up), measure market-uncapped for 30 s (after 5 s warm-up); 1 run on each side for each of 2 cells (2 game starts a run).
  • Build times: main and the PR each built 2 times, alternating, each whole in a fresh folder: the lift again (no kept lift, main's compiled objects kept); every stage timed (shader preparation is not), and the lifted C compared.
  • Measured: crashes, frame times, stutters and hitches, memory and CPU time, numbers from the game's log.
  • The PR fails if: CPU time is more than 10% above main; or it crashes; or the PR's value_activation with HOS_NATIVE_VISIBILITY=1 is more than 100% below HOS_NATIVE_VISIBILITY=0; or the PR's CPU time in market-standing with HOS_NATIVE_VISIBILITY=1 is more than 5% above HOS_NATIVE_VISIBILITY=0; or the PR's CPU time in field-standing with HOS_NATIVE_VISIBILITY=1 is more than 5% above HOS_NATIVE_VISIBILITY=0; or a picture differs from main by more than 2%; or frame time (p99) is more than 15% above main.
  • Time: about 37 min of game runs per tester (4 game starts), plus the two builds and 4 timed builds (about 80 min).
Measurements on `linux`, for the PR description

Verdict: fail on linux (2 reports): visibility-route screenshot.max_diff; visibility-route field-standing.cpu_s: HOS_NATIVE_VISIBILITY=1 vs HOS_NATIVE_VISIBILITY=0 (PR) (+8.7%); visibility-route market-standing.cpu_s.max_delta_pct (+22.0%) failed.

Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 5070, 620.70.6 AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM; 2: linux+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 5080, 620.73.4 AMD Ryzen 7 7800X3D 8-Core Processor, 32 GB RAM
Scene: visibility-route, from a new save (frame_rate_cap=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); visibility-route: 1 game start a side, market-standing 30 s after 5 s warm-up, market-turned 20 s after 2 s warm-up, field-standing 30 s after 5 s warm-up, field-turned 20 s after 2 s warm-up, field-uncapped 30 s after 5 s warm-up, market-uncapped 30 s after 5 s warm-up; timed builds: main and this PR each built 2 times a cache, alternating, each a whole build in a fresh folder (lift; shader preparation not timed); each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (@Beanstalk on this PR runs it again)

Before: main @ a104182 After: this PR @ 5f0c55d
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · field-turned · CPU time (s) 32.3 33.6 (+4%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · market-turned · CPU time (s) 40.1 40.0 (−0%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · field-standing · CPU time (s) 62.5 58.2 (−7%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · field-uncapped · CPU time (s) 109.2 111.2 (+2%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · market-standing · CPU time (s) 118.4 112.0 (−5%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · market-uncapped · CPU time (s) 137.2 137.8 (+0%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · field-turned · frame time (p99) (ms) 18.8 18.6 (−1%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · market-turned · frame time (p99) (ms) 18.9 18.7 (−1%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · field-standing · frame time (p99) (ms) 18.8 19.4 (+4%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · field-uncapped · frame time (p99) (ms) 8.0 8.4 (+4%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · market-standing · frame time (p99) (ms) 19.7 19.8 (+1%)
visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only) · market-uncapped · frame time (p99) (ms) 11.7 11.7 (−0%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · field-turned · CPU time (s) 26.5 32.4 (+22%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · market-turned · CPU time (s) 35.7 39.7 (+11%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · field-standing · CPU time (s) 43.6 60.0 (+37%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · field-uncapped · CPU time (s) 109.0 109.3 (+0%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · market-standing · CPU time (s) 98.9 113.6 (+15%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · market-uncapped · CPU time (s) 141.3 137.4 (−3%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · field-turned · frame time (p99) (ms) 18.2 18.7 (+3%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · market-turned · frame time (p99) (ms) 18.6 19.0 (+2%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · field-standing · frame time (p99) (ms) 18.1 18.7 (+3%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · field-uncapped · frame time (p99) (ms) 6.8 8.2 (+20%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · market-standing · frame time (p99) (ms) 19.1 19.9 (+5%)
visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only) · market-uncapped · frame time (p99) (ms) 9.7 12.0 (+24%)

Build times (each a median of the timed builds; caches: lift = the lift again (no kept lift, main's compiled objects kept))

Before: main @ a104182 After: this PR @ 5f0c55d
lift · whole build (s) 475.7 550.7 (+16%)
lift · read (s) 1.1 1.0 (−17%)
lift · lift (s) 74.8 74.7 (−0%)
lift · compile (s) 389.5 464.8 (+19%)
lift · finish (s) 10.2 10.3 (+1%)
lift · lifted C main's differs

Flags and settings

Flag What it is Values tried Before this PR After this PR Effect (CPU time) Helps / doesn't help / untested
HOS_NATIVE_VISIBILITY switch the PR adds (set on the PR side only) 0, 1 not in main the PR's default visibility-route, main without it → the PR with 0: 32.3 → 33.6 (+4%) doesn't help

As Markdown for the PR description:

## Measurements

**Verdict:** fail on linux (2 reports): `visibility-route screenshot.max_diff`; `visibility-route field-standing.cpu_s: HOS_NATIVE_VISIBILITY=1 vs HOS_NATIVE_VISIBILITY=0 (PR) (+8.7%)`; `visibility-route market-standing.cpu_s.max_delta_pct (+22.0%)` failed.

Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5070, 620.70.6` `AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM`; 2: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5080, 620.73.4` `AMD Ryzen 7 7800X3D 8-Core Processor, 32 GB RAM`  
Scene: visibility-route, from a new save (frame_rate_cap=0); each cell's settings and switches as the table names them  
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); visibility-route: 1 game start a side, market-standing 30 s after 5 s warm-up, market-turned 20 s after 2 s warm-up, field-standing 30 s after 5 s warm-up, field-turned 20 s after 2 s warm-up, field-uncapped 30 s after 5 s warm-up, market-uncapped 30 s after 5 s warm-up; timed builds: main and this PR each built 2 times a cache, alternating, each a whole build in a fresh folder (lift; shader preparation not timed); each number the median of the runs, then of 2 reports  
How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again)

| | Before: main @ a104182 | After: this PR @ 5f0c55d |
|---|---|---|
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-turned · CPU time (s) | 32.3 | 33.6 (+4%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-turned · CPU time (s) | 40.1 | 40.0 (−0%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-standing · CPU time (s) | 62.5 | 58.2 (−7%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-uncapped · CPU time (s) | 109.2 | 111.2 (+2%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-standing · CPU time (s) | 118.4 | 112.0 (−5%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-uncapped · CPU time (s) | 137.2 | 137.8 (+0%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-turned · frame time (p99) (ms) | 18.8 | 18.6 (−1%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-turned · frame time (p99) (ms) | 18.9 | 18.7 (−1%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-standing · frame time (p99) (ms) | 18.8 | 19.4 (+4%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-uncapped · frame time (p99) (ms) | 8.0 | 8.4 (+4%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-standing · frame time (p99) (ms) | 19.7 | 19.8 (+1%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-uncapped · frame time (p99) (ms) | 11.7 | 11.7 (−0%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-turned · CPU time (s) | 26.5 | 32.4 (+22%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-turned · CPU time (s) | 35.7 | 39.7 (+11%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-standing · CPU time (s) | 43.6 | 60.0 (+37%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-uncapped · CPU time (s) | 109.0 | 109.3 (+0%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-standing · CPU time (s) | 98.9 | 113.6 (+15%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-uncapped · CPU time (s) | 141.3 | 137.4 (−3%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-turned · frame time (p99) (ms) | 18.2 | 18.7 (+3%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-turned · frame time (p99) (ms) | 18.6 | 19.0 (+2%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-standing · frame time (p99) (ms) | 18.1 | 18.7 (+3%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-uncapped · frame time (p99) (ms) | 6.8 | 8.2 (+20%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-standing · frame time (p99) (ms) | 19.1 | 19.9 (+5%) |
| visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-uncapped · frame time (p99) (ms) | 9.7 | 12.0 (+24%) |

Build times (each a median of the timed builds; caches: lift = the lift again (no kept lift, main's compiled objects kept))

| | Before: main @ a104182 | After: this PR @ 5f0c55d |
|---|---|---|
| lift · whole build (s) | 475.7 | 550.7 (+16%) |
| lift · read (s) | 1.1 | 1.0 (−17%) |
| lift · lift (s) | 74.8 | 74.7 (−0%) |
| lift · compile (s) | 389.5 | 464.8 (+19%) |
| lift · finish (s) | 10.2 | 10.3 (+1%) |
| lift · lifted C | main's | differs |

**Flags and settings**

| Flag | What it is | Values tried | Before this PR | After this PR | Effect (CPU time) | Helps / doesn't help / untested |
|---|---|---|---|---|---|---|
| `HOS_NATIVE_VISIBILITY` | switch the PR adds (set on the PR side only) | `0`, `1` | not in main | the PR's default | visibility-route, main without it → the PR with 0: 32.3 → 33.6 (+4%) | doesn't help |
The testers' hardware
Target Report Ran on
linux 1 linux+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 5070, 620.70.6 AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM
linux 2 linux+nvidia+nvidia-proprietary+dgpu NVIDIA GeForce RTX 5080, 620.73.4 AMD Ryzen 7 7800X3D 8-Core Processor, 32 GB RAM
<!-- beanstalk:previous --> <details><summary>Beanstalk test farm · an earlier run: commit <code>5f0c55d</code> (the newer results are in a comment below)</summary> Commit `5f0c55d` · approved by llun (reviewed) · [dashboard](https://beanstalk.magicbean.dev/prs/1902) > **Summary**, written by AI (`gpt-6.1-terra`) from the numbers below: > > The change failed on Linux on both reported PCs, which the farm marked as outliers. With the visibility switch off, screenshot differences were 0.1422 on the RTX 5070 and 0.5873 on the RTX 5080, both above the 0.02 limit; on the 5080 with the switch on, the difference was 0.4989 and Market standing CPU time rose from 98.91 to 120.70 seconds (+22%, against a 10% limit). Several p99 frame-time comparisons were inconclusive because they lacked two usable starts per side. The switch-on scenario was marked crashed on the 5070, although its candidate recorded exit code 0. Done (started 19:55 UTC, last report 23:56 UTC). Comment `@Beanstalk` to run it again. | Target | Result | Reports | VRAM peak (MB) | Crashes | | --- | --- | --- | --- | --- | | `linux` | ❌ fail (an outlier) | 2 [1](https://beanstalk.magicbean.dev/reports/rp_128ecda6a67c051f) [2](https://beanstalk.magicbean.dev/reports/rp_83d7bca060d28266) | — | none | **`visibility-route`** · frame time p99 (ms) | Target | main | `HOS_NATIVE_VISIBILITY=0` (reference) | `HOS_NATIVE_VISIBILITY=1` | | --- | --- | --- | --- | | `linux` | 18.8 | 18.6 | 18.0 (−3%) | **Why it failed** - `linux`: `visibility-route · HOS_NATIVE_VISIBILITY=0 (PR only)` `screenshot.max_diff`: `limit 0.0` - `linux`: `visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only)` `field-standing.cpu_s: HOS_NATIVE_VISIBILITY=1 vs HOS_NATIVE_VISIBILITY=0 (PR)`: `69.1 → 75.2 (+8.7%); limit 5.0; the PR's own cells: HOS_NATIVE_VISIBILITY=1 against HOS_NATIVE_VISIBILITY=0 (HOS_NATIVE_VISIBILITY=0 (PR only))` - `linux`: `visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only)` `market-standing.cpu_s.max_delta_pct`: `98.9 → 120.7 (+22.0%); limit 10.0` - `linux`: `visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only)` `screenshot.max_diff`: `limit 0.0` - `linux`: `visibility-route · HOS_NATIVE_VISIBILITY=1 (PR only)` `market-standing.cpu_s: HOS_NATIVE_VISIBILITY=1 vs HOS_NATIVE_VISIBILITY=0 (PR)`: `112.2 → 120.7 (+7.5%); limit 5.0; the PR's own cells: HOS_NATIVE_VISIBILITY=1 against HOS_NATIVE_VISIBILITY=0 (HOS_NATIVE_VISIBILITY=0 (PR only))` <details><summary>What was tested</summary> **What will be tested** · AI plan (`gpt-6.1-terra`) On Linux, the plan compares main with the PR’s visibility switch off and on in the child Market and Hyrule Field. It measures capped CPU time at two camera angles, uncapped frame times, and matching views for missing models or other visual changes; it also times lift builds. Crashes, failure to activate the switch, substantial CPU or frame-time regressions, or picture differences fail the plan. Gameplay takes roughly half an hour per tester, with timed builds counted separately. - **Where:** linux (1-2 reports). - **Matrix:** 2 cells, each on main and the PR: HOS_NATIVE_VISIBILITY `0` or `1` (PR only). - **visibility-route:** from a new save (settings frame_rate_cap=0), then teleport to Market, measure market-standing for 30 s (after 5 s warm-up), take the picture market-front to compare with main, move the stick, take the picture market-turned to compare with main, measure market-turned for 20 s (after 2 s warm-up), teleport to Hyrule Field, measure field-standing for 30 s (after 5 s warm-up), take the picture field-front to compare with main, take the picture field-turned to compare with main, measure field-turned for 20 s (after 2 s warm-up), set frame_rate_cap to 6, measure field-uncapped for 30 s (after 5 s warm-up), measure market-uncapped for 30 s (after 5 s warm-up); 1 run on each side for each of 2 cells (2 game starts a run). - **Build times:** main and the PR each built 2 times, alternating, each whole in a fresh folder: the lift again (no kept lift, main's compiled objects kept); every stage timed (shader preparation is not), and the lifted C compared. - **Measured:** crashes, frame times, stutters and hitches, memory and CPU time, numbers from the game's log. - **The PR fails if:** CPU time is more than 10% above main; or it crashes; or the PR's value_activation with HOS_NATIVE_VISIBILITY=1 is more than 100% below HOS_NATIVE_VISIBILITY=0; or the PR's CPU time in market-standing with HOS_NATIVE_VISIBILITY=1 is more than 5% above HOS_NATIVE_VISIBILITY=0; or the PR's CPU time in field-standing with HOS_NATIVE_VISIBILITY=1 is more than 5% above HOS_NATIVE_VISIBILITY=0; or a picture differs from main by more than 2%; or frame time (p99) is more than 15% above main. - **Time:** about 37 min of game runs per tester (4 game starts), plus the two builds and 4 timed builds (about 80 min). </details> <details><summary>Measurements on `linux`, for the PR description</summary> **Verdict:** fail on linux (2 reports): `visibility-route screenshot.max_diff`; `visibility-route field-standing.cpu_s: HOS_NATIVE_VISIBILITY=1 vs HOS_NATIVE_VISIBILITY=0 (PR) (+8.7%)`; `visibility-route market-standing.cpu_s.max_delta_pct (+22.0%)` failed. Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5070, 620.70.6` `AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM`; 2: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5080, 620.73.4` `AMD Ryzen 7 7800X3D 8-Core Processor, 32 GB RAM` Scene: visibility-route, from a new save (frame_rate_cap=0); each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); visibility-route: 1 game start a side, market-standing 30 s after 5 s warm-up, market-turned 20 s after 2 s warm-up, field-standing 30 s after 5 s warm-up, field-turned 20 s after 2 s warm-up, field-uncapped 30 s after 5 s warm-up, market-uncapped 30 s after 5 s warm-up; timed builds: main and this PR each built 2 times a cache, alternating, each a whole build in a fresh folder (lift; shader preparation not timed); each number the median of the runs, then of 2 reports How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) | | Before: main @ a104182 | After: this PR @ 5f0c55d | |---|---|---| | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-turned · CPU time (s) | 32.3 | 33.6 (+4%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-turned · CPU time (s) | 40.1 | 40.0 (−0%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-standing · CPU time (s) | 62.5 | 58.2 (−7%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-uncapped · CPU time (s) | 109.2 | 111.2 (+2%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-standing · CPU time (s) | 118.4 | 112.0 (−5%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-uncapped · CPU time (s) | 137.2 | 137.8 (+0%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-turned · frame time (p99) (ms) | 18.8 | 18.6 (−1%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-turned · frame time (p99) (ms) | 18.9 | 18.7 (−1%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-standing · frame time (p99) (ms) | 18.8 | 19.4 (+4%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-uncapped · frame time (p99) (ms) | 8.0 | 8.4 (+4%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-standing · frame time (p99) (ms) | 19.7 | 19.8 (+1%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-uncapped · frame time (p99) (ms) | 11.7 | 11.7 (−0%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-turned · CPU time (s) | 26.5 | 32.4 (+22%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-turned · CPU time (s) | 35.7 | 39.7 (+11%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-standing · CPU time (s) | 43.6 | 60.0 (+37%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-uncapped · CPU time (s) | 109.0 | 109.3 (+0%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-standing · CPU time (s) | 98.9 | 113.6 (+15%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-uncapped · CPU time (s) | 141.3 | 137.4 (−3%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-turned · frame time (p99) (ms) | 18.2 | 18.7 (+3%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-turned · frame time (p99) (ms) | 18.6 | 19.0 (+2%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-standing · frame time (p99) (ms) | 18.1 | 18.7 (+3%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-uncapped · frame time (p99) (ms) | 6.8 | 8.2 (+20%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-standing · frame time (p99) (ms) | 19.1 | 19.9 (+5%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-uncapped · frame time (p99) (ms) | 9.7 | 12.0 (+24%) | Build times (each a median of the timed builds; caches: lift = the lift again (no kept lift, main's compiled objects kept)) | | Before: main @ a104182 | After: this PR @ 5f0c55d | |---|---|---| | lift · whole build (s) | 475.7 | 550.7 (+16%) | | lift · read (s) | 1.1 | 1.0 (−17%) | | lift · lift (s) | 74.8 | 74.7 (−0%) | | lift · compile (s) | 389.5 | 464.8 (+19%) | | lift · finish (s) | 10.2 | 10.3 (+1%) | | lift · lifted C | main's | differs | **Flags and settings** | Flag | What it is | Values tried | Before this PR | After this PR | Effect (CPU time) | Helps / doesn't help / untested | |---|---|---|---|---|---|---| | `HOS_NATIVE_VISIBILITY` | switch the PR adds (set on the PR side only) | `0`, `1` | not in main | the PR's default | visibility-route, main without it → the PR with 0: 32.3 → 33.6 (+4%) | doesn't help | As Markdown for the PR description: ```markdown ## Measurements **Verdict:** fail on linux (2 reports): `visibility-route screenshot.max_diff`; `visibility-route field-standing.cpu_s: HOS_NATIVE_VISIBILITY=1 vs HOS_NATIVE_VISIBILITY=0 (PR) (+8.7%)`; `visibility-route market-standing.cpu_s.max_delta_pct (+22.0%)` failed. Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5070, 620.70.6` `AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM`; 2: linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5080, 620.73.4` `AMD Ryzen 7 7800X3D 8-Core Processor, 32 GB RAM` Scene: visibility-route, from a new save (frame_rate_cap=0); each cell's settings and switches as the table names them Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); visibility-route: 1 game start a side, market-standing 30 s after 5 s warm-up, market-turned 20 s after 2 s warm-up, field-standing 30 s after 5 s warm-up, field-turned 20 s after 2 s warm-up, field-uncapped 30 s after 5 s warm-up, market-uncapped 30 s after 5 s warm-up; timed builds: main and this PR each built 2 times a cache, alternating, each a whole build in a fresh folder (lift; shader preparation not timed); each number the median of the runs, then of 2 reports How to reproduce: the farm's plan above (`@Beanstalk` on this PR runs it again) | | Before: main @ a104182 | After: this PR @ 5f0c55d | |---|---|---| | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-turned · CPU time (s) | 32.3 | 33.6 (+4%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-turned · CPU time (s) | 40.1 | 40.0 (−0%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-standing · CPU time (s) | 62.5 | 58.2 (−7%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-uncapped · CPU time (s) | 109.2 | 111.2 (+2%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-standing · CPU time (s) | 118.4 | 112.0 (−5%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-uncapped · CPU time (s) | 137.2 | 137.8 (+0%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-turned · frame time (p99) (ms) | 18.8 | 18.6 (−1%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-turned · frame time (p99) (ms) | 18.9 | 18.7 (−1%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-standing · frame time (p99) (ms) | 18.8 | 19.4 (+4%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · field-uncapped · frame time (p99) (ms) | 8.0 | 8.4 (+4%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-standing · frame time (p99) (ms) | 19.7 | 19.8 (+1%) | | visibility-route · `HOS_NATIVE_VISIBILITY=0 (PR only)` · market-uncapped · frame time (p99) (ms) | 11.7 | 11.7 (−0%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-turned · CPU time (s) | 26.5 | 32.4 (+22%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-turned · CPU time (s) | 35.7 | 39.7 (+11%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-standing · CPU time (s) | 43.6 | 60.0 (+37%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-uncapped · CPU time (s) | 109.0 | 109.3 (+0%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-standing · CPU time (s) | 98.9 | 113.6 (+15%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-uncapped · CPU time (s) | 141.3 | 137.4 (−3%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-turned · frame time (p99) (ms) | 18.2 | 18.7 (+3%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-turned · frame time (p99) (ms) | 18.6 | 19.0 (+2%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-standing · frame time (p99) (ms) | 18.1 | 18.7 (+3%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · field-uncapped · frame time (p99) (ms) | 6.8 | 8.2 (+20%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-standing · frame time (p99) (ms) | 19.1 | 19.9 (+5%) | | visibility-route · `HOS_NATIVE_VISIBILITY=1 (PR only)` · market-uncapped · frame time (p99) (ms) | 9.7 | 12.0 (+24%) | Build times (each a median of the timed builds; caches: lift = the lift again (no kept lift, main's compiled objects kept)) | | Before: main @ a104182 | After: this PR @ 5f0c55d | |---|---|---| | lift · whole build (s) | 475.7 | 550.7 (+16%) | | lift · read (s) | 1.1 | 1.0 (−17%) | | lift · lift (s) | 74.8 | 74.7 (−0%) | | lift · compile (s) | 389.5 | 464.8 (+19%) | | lift · finish (s) | 10.2 | 10.3 (+1%) | | lift · lifted C | main's | differs | **Flags and settings** | Flag | What it is | Values tried | Before this PR | After this PR | Effect (CPU time) | Helps / doesn't help / untested | |---|---|---|---|---|---|---| | `HOS_NATIVE_VISIBILITY` | switch the PR adds (set on the PR side only) | `0`, `1` | not in main | the PR's default | visibility-route, main without it → the PR with 0: 32.3 → 33.6 (+4%) | doesn't help | ``` </details> <details><summary>The testers' hardware</summary> | Target | Report | Ran on | | --- | --- | --- | | `linux` | [1](https://beanstalk.magicbean.dev/reports/rp_128ecda6a67c051f) | linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5070, 620.70.6` `AMD Ryzen 5 9600X 6-Core Processor, 32 GB RAM` | | `linux` | [2](https://beanstalk.magicbean.dev/reports/rp_83d7bca060d28266) | linux+nvidia+nvidia-proprietary+dgpu `NVIDIA GeForce RTX 5080, 620.73.4` `AMD Ryzen 7 7800X3D 8-Core Processor, 32 GB RAM` | </details> </details>
BeanBot left a comment

Recommendation: Request changes. The hooks have an initialization data race, and check mode can report success without comparing all claimed state. The performance gain and the cost of the default-off path are not established for this branch.

This adds optional host implementations of two visibility routines, with fallback to the lifted game code and a comparison mode. The potential CPU saving comes at the cost of substantial exactness and fallback code, synchronization risk, and an unmeasured change to the default path.

Does it help: Unproven. Three interleaved Windows runs of nine combined rewrite branches used 2.7% less process CPU standing and 2.4% less turning than an older main; they do not isolate this PR. Offline timings show the native box path is slower for calls that stop before the planes, while the measured share of those calls is unknown. This head has no isolated game measurement against its base, including frame-time and 1% low results.
Other systems: The gates run on module workers even with the switch unset; the grid also marks guest memory for the GPU mirror. The reported capture-copy thread increase of 0.010–0.014 cores has not been attributed. Linux, AMD, Steam Deck, other Tier 1 configurations and this head on Windows remain untested. The reported Unicorn comparisons and earlier Windows check run support exactness for their inputs, but do not establish coverage on those systems.
Switches: HOS_NATIVE_VISIBILITY is new and defaults off; HOS_NATIVE_VISIBILITY_LOG is new and defaults off, with logging forced in check mode. Both have table rows. The performance switch's default was chosen because the change is unmeasured, rather than from platform measurements as the project requires; its default-off overhead also needs measurement.

Reviewed: 5f0c55d218 against a104182b40: Read the repository rules, contribution guide, PR template, standards, commits, changed host files and headers, and the changed sections of the patcher and switch table. Reviewed the test fixture and its stated coverage, and inspected guest-memory marking and gate dependencies. The remainder of the large patcher, the test file and unrelated callers were skimmed; the unavailable sibling Tooling rules and live Forgejo state could not be inspected. No generated files or translations changed.; nothing run (a code review).
Checked and fine:

  • The gates are restricted to the supported build and the native kernels decline unsupported inputs.
  • The grid snapshots writable regions and restores them on a decline; its successful native writes are marked for the GPU mirror.
  • The new game-free fixture exercises both box implementations, fallback paths and a hand-derived empty-grid case.
  • Both new environment variables have switch-table entries; third_party and generated files are untouched.

Blocking (to change before merge)

  1. host/oot_visibility.c:66: The initialization writes vis_mode and the volatile gate variables while other module workers read them without atomic synchronization. Two workers entering the box test during first initialization can race even when the switch is unset; volatile does not make the accesses safe, and a worker that observes vis_mode directly does not acquire the published state. Use atomic gate loads/stores and ensure every reader acquires completed initialization before reading the mode and logging state.
  2. host/oot_visibility.c:132: Check mode never compares the box result's NZCV, even though the native path writes it. A flag-only mismatch is counted as taken with no DIFFERENT line, so the advertised in-game register comparison can miss a divergence. Compare NZCV and add a fixture that deliberately changes only a flag.
  3. host/oot_visibility.c:161: If realloc or malloc fails while saving a grid region, occ_keep silently omits it and check mode can report a match without comparing that region. Record the failure and decline or explicitly report the check as incomplete; test allocation failure.
  4. nx2/oot_patches.py:392: The default-off gates alter the lifted functions used by every player, and the offline approximation reports up to 10% and 7% more time for the two routines. The combined process measurement cannot rule out a regression at this head. Measure this branch alone against its merge base, including default-off cost, enabled benefit, frame pacing and the suspected GPU mirror cost; revise or remove the change if the cost outweighs its demonstrated gain.
  5. gpu/vk/switches.def:107: The new performance switch ships off solely because it has not been measured alone, contrary to the project's switch rule. Measure affected platforms and select the default from those results, or defer the switch and implementation until that choice is supported.

Suggestions (the author may take or leave)

  • In check mode, compare the grid's relevant saved registers as well as memory if the description continues to claim register-for-register checking.
  • Complete the head's Windows and Linux game runs, Tier 1 coverage, full test suite and CI before merge.

Testing: No commands that build, execute project code or run tests were used, as requested. The PR reports game-free checks and an earlier Windows game run; this head has not been built or played.
Overlaps and merge order: No overlapping PRs could be established from the provided material; network access is closed.


Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The hooks have an initialization data race, and check mode can report success without comparing all claimed state. The performance gain and the cost of the default-off path are not established for this branch. This adds optional host implementations of two visibility routines, with fallback to the lifted game code and a comparison mode. The potential CPU saving comes at the cost of substantial exactness and fallback code, synchronization risk, and an unmeasured change to the default path. **Does it help:** Unproven. Three interleaved Windows runs of nine combined rewrite branches used 2.7% less process CPU standing and 2.4% less turning than an older main; they do not isolate this PR. Offline timings show the native box path is slower for calls that stop before the planes, while the measured share of those calls is unknown. This head has no isolated game measurement against its base, including frame-time and 1% low results. **Other systems:** The gates run on module workers even with the switch unset; the grid also marks guest memory for the GPU mirror. The reported capture-copy thread increase of 0.010–0.014 cores has not been attributed. Linux, AMD, Steam Deck, other Tier 1 configurations and this head on Windows remain untested. The reported Unicorn comparisons and earlier Windows check run support exactness for their inputs, but do not establish coverage on those systems. **Switches:** HOS_NATIVE_VISIBILITY is new and defaults off; HOS_NATIVE_VISIBILITY_LOG is new and defaults off, with logging forced in check mode. Both have table rows. The performance switch's default was chosen because the change is unmeasured, rather than from platform measurements as the project requires; its default-off overhead also needs measurement. **Reviewed:** 5f0c55d21895 against a104182b4037: Read the repository rules, contribution guide, PR template, standards, commits, changed host files and headers, and the changed sections of the patcher and switch table. Reviewed the test fixture and its stated coverage, and inspected guest-memory marking and gate dependencies. The remainder of the large patcher, the test file and unrelated callers were skimmed; the unavailable sibling Tooling rules and live Forgejo state could not be inspected. No generated files or translations changed.; nothing run (a code review). **Checked and fine:** - The gates are restricted to the supported build and the native kernels decline unsupported inputs. - The grid snapshots writable regions and restores them on a decline; its successful native writes are marked for the GPU mirror. - The new game-free fixture exercises both box implementations, fallback paths and a hand-derived empty-grid case. - Both new environment variables have switch-table entries; third_party and generated files are untouched. **Blocking** (to change before merge) 1. host/oot_visibility.c:66: The initialization writes vis_mode and the volatile gate variables while other module workers read them without atomic synchronization. Two workers entering the box test during first initialization can race even when the switch is unset; volatile does not make the accesses safe, and a worker that observes vis_mode directly does not acquire the published state. Use atomic gate loads/stores and ensure every reader acquires completed initialization before reading the mode and logging state. 2. host/oot_visibility.c:132: Check mode never compares the box result's NZCV, even though the native path writes it. A flag-only mismatch is counted as taken with no DIFFERENT line, so the advertised in-game register comparison can miss a divergence. Compare NZCV and add a fixture that deliberately changes only a flag. 3. host/oot_visibility.c:161: If realloc or malloc fails while saving a grid region, occ_keep silently omits it and check mode can report a match without comparing that region. Record the failure and decline or explicitly report the check as incomplete; test allocation failure. 4. nx2/oot_patches.py:392: The default-off gates alter the lifted functions used by every player, and the offline approximation reports up to 10% and 7% more time for the two routines. The combined process measurement cannot rule out a regression at this head. Measure this branch alone against its merge base, including default-off cost, enabled benefit, frame pacing and the suspected GPU mirror cost; revise or remove the change if the cost outweighs its demonstrated gain. 5. gpu/vk/switches.def:107: The new performance switch ships off solely because it has not been measured alone, contrary to the project's switch rule. Measure affected platforms and select the default from those results, or defer the switch and implementation until that choice is supported. **Suggestions** (the author may take or leave) - In check mode, compare the grid's relevant saved registers as well as memory if the description continues to claim register-for-register checking. - Complete the head's Windows and Linux game runs, Tier 1 coverage, full test suite and CI before merge. **Testing:** No commands that build, execute project code or run tests were used, as requested. The PR reports game-free checks and an earlier Windows game run; this head has not been built or played. **Overlaps and merge order:** No overlapping PRs could be established from the provided material; network access is closed. --- <sub>Automated code review by BeanBot (gpt-6.1-terra), on the pull request's opening. Nothing was run; a maintainer decides.</sub>
@ -104,6 +104,8 @@ SW(HOS_MOVIE_PACE, kill, bool, "on: a 60 FPS cap is raised for a movie's picture
SW(HOS_MUTEX_TRACE, diag, int, "0: off", "Logs mutex waits longer than <ms>, with the thread that held the mutex.")
SW(HOS_MUTEX_WATCH, diag, string, "off", "Logs every operation on the mutex at that guest address (hex) and its condition variables' signals.")
SW(HOS_MW_WAKE, kill, enum, "only changes to objects linked into a multi-wait wake WaitAny", "=all: every event, semaphore or queue change wakes every WaitAny, as before.")
SW(HOS_NATIVE_VISIBILITY, attempt, enum, "off: the visibility routines run as lifted", "1 runs the render calc's visibility routines (the box-against-view test f_80029240, the occlusion grid's build f_808226e0) on host code where it gives the game's bits exactly, the rest lifted; check runs both on every call and logs any difference (slow; host/oot_visibility.c).")
Member

The default is off because this branch has not been measured alone. The switch standard requires a default chosen from platform measurements; please establish those results and choose the default accordingly.

The default is off because this branch has not been measured alone. The switch standard requires a default chosen from platform measurements; please establish those results and choose the default accordingly.
@ -0,0 +63,4 @@
hos_log("visibility: the box-against-view test (f_80029240) and the occlusion grid (f_808226e0) on the host%s "
"(HOS_NATIVE_VISIBILITY)",
m == VIS_CHECK ? ", each call checked against the lifted routine" : "");
vis_mode = m;
Member

Multiple module workers can read vis_mode and the volatile gates while this thread writes them, including with the switch unset. Please publish and read the gates atomically and acquire completed initialization before reading the mode and log state.

Multiple module workers can read vis_mode and the volatile gates while this thread writes them, including with the switch unset. Please publish and read the gates atomically and acquire completed initialization before reading the mode and log state.
@ -0,0 +129,4 @@
/* (what the lifted code writes: x0, x8, x9, x10, v0..v3; it keeps only the NZCV bits it needs) */
VisBoxRegs g;
regs_from(c, &g);
if (g.x0 != r.x0 || g.x8 != r.x8 || g.x9 != r.x9 || g.x10 != r.x10 || memcmp(g.q, r.q, sizeof g.q)) {
Member

The native path writes NZCV, but check mode does not compare it. A flag-only mismatch is therefore reported as a match; please compare it and cover that case in the fixture.

The native path writes NZCV, but check mode does not compare it. A flag-only mismatch is therefore reported as a match; please compare it and cover that case in the fixture.
@ -0,0 +158,4 @@
OccCopy *k = (OccCopy *)ctx;
if (k->n == k->cap) {
const int cap = k->cap ? 2 * k->cap : 32;
OccRegion *r = (OccRegion *)realloc(k->r, (size_t)cap * sizeof *r);
Member

If this allocation fails, the region is silently omitted and the check can report a match without comparing it. Please propagate the failure and report or decline the incomplete check.

If this allocation fails, the region is silently omitted and the check can report a match without comparing it. Please propagate the failure and report or decline the incomplete check.
@ -386,0 +389,4 @@
# Each gets a gate first in its lifted function: hos_vis_on_<addr> is 1 until the first call has read the switch, then 0
# unless it is on (one load and an untaken branch when off); hos_vis_<addr>(c, f_<addr>) returns 1 when it answered.
VIS_GATES = ("80029240", "808226e0")
VIS_GATE = (" if (__builtin_expect(hos_vis_on_%s, 0) && hos_vis_%s(c, f_%s)) return; "
Member

These gates change the default-off lifted path too; the offline approximation shows up to 10% and 7% overhead for the two routines. Please measure this head alone against its merge base, including frame pacing and the suspected mirror cost, and revise the change if its cost outweighs the gain.

These gates change the default-off lifted path too; the offline approximation shows up to 10% and 7% overhead for the two routines. Please measure this head alone against its merge base, including frame pacing and the suspected mirror cost, and revise the change if its cost outweighs the gain.
Owner
@fix
Member

I changed this pull request by bringing it up to date: the new head is caca43f11c. I rebased it onto main (a764653726).

Conflicts resolved

  • gpu/vk/switches.def: Kept both visibility switch entries and the base branch’s updated Zstd entry.

Left for the author

  • Visibility initialization, comparison, and performance evidence: Changed host/oot_visibility.c:37 and nx2/oot_patches.py:392 to synchronize gate access and initialization; compare NZCV and decline incomplete grid checks. Added flag-only and allocation-failure cases in tests/test_native_visibility.py:280. The author must measure this head against its merge base on the supported platforms, including default-off overhead, frame times, 1% lows, and the suspected GPU mirror cost, then choose the switch default and assess whether the gain outweighs the cost. The existing combined-branch and offline timings do not establish those results. No builds or tests were r

Nothing was built or run: check the change. It will be reviewed again.


Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.

I changed this pull request by bringing it up to date: the new head is caca43f11c8e. I rebased it onto `main` (a764653726ba). **Conflicts resolved** - gpu/vk/switches.def: Kept both visibility switch entries and the base branch’s updated Zstd entry. **Left for the author** - **Visibility initialization, comparison, and performance evidence**: Changed host/oot_visibility.c:37 and nx2/oot_patches.py:392 to synchronize gate access and initialization; compare NZCV and decline incomplete grid checks. Added flag-only and allocation-failure cases in tests/test_native_visibility.py:280. The author must measure this head against its merge base on the supported platforms, including default-off overhead, frame times, 1% lows, and the suspected GPU mirror cost, then choose the switch default and assess whether the gain outweighs the cost. The existing combined-branch and offline timings do not establish those results. No builds or tests were r Nothing was built or run: check the change. It will be reviewed again. --- <sub>Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.</sub>
BeanBot force-pushed perf/native-visibility from 5f0c55d218
Some checks failed
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 16s
beanstalk/approval approved by llun (reviewed)
beanstalk/linux 2/1 fail
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to caca43f11c
Some checks failed
beanstalk/approval not approved yet
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 12s
PR description / Description follows the template (pull_request_target) Successful in 16s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 16s
CI / Workflow lint (actionlint) (pull_request) Successful in 18s
Third-party parts / As their notes say (pull_request) Successful in 19s
CI / Website build and checks (pull_request) Failing after 16s
2026-11-21 00:13:59 +00:00
Compare
BeanBot left a comment

Recommendation: Request changes. The production host build cannot see the floating-point gate used by both hooks, and the default-off path still has an initialization race. Check mode can report a match with state left unchecked, and the available runs leave the CPU gain and Linux regressions unresolved.

Adds opt-in host implementations for two rendering visibility routines, with a lifted fallback and a diagnostic comparison mode. The potential CPU saving comes with two implementations to maintain, a changed lifted path even when disabled, and per-frame grid snapshots and dirty-page marking.

Does it help: Unproven. Three interleaved Windows rounds with nine rewrites enabled used 2.7% less process CPU standing and 2.4% less turning in the capped Market. The combined off deltas of −0.3% and −0.2% were within the spread. Offline, box calls that exit before the plane tests took 28–35 ns versus 9–22 ns, and their in-game share is unknown. On the earlier Linux head, the RTX 5080 used 120.7 CPU seconds in the Market with visibility on versus 112.2 with it off and 98.9 on main. The current head lacks an isolated comparison.
Other systems: The hooks run on module workers across GPU vendors and can affect model visibility, the GPU mirror, and frame pacing. The earlier Windows check run found no reported differences among 6,291,456 native box calls and 4,095 native grid calls, within the fields that check mode compared. The earlier Linux farm flagged screenshot differences with the switch off on both NVIDIA systems, and several p99 comparisons lacked enough usable starts. AMD, Steam Deck, older CPUs without FMA, and the current Windows and Linux builds still need coverage. The diff adds no save, audio, input, or path-format changes.
Switches: HOS_NATIVE_VISIBILITY defaults off and accepts 1 or check. The default avoids the native kernels, but the gated lift still changes and the project requires a measured default. HOS_NATIVE_VISIBILITY_LOG defaults off and is forced on in check mode. That diagnostic default is appropriate. Both have switch-table rows.

Reviewed: caca43f11c against a764653726: Read AGENTS.md, CONTRIBUTING.md, developer standards, the measurement guidance, the Forgejo template, the six commits, the complete diff, and all new host files, headers, and tests. Read the changed patcher and switch-table sections and traced the lift, split, host build, guest memory, and GPU marking paths. Skimmed unrelated portions of the existing files. Read the supplied recorded runs, comments, and stale review. The review checklist file and sibling Tooling checkout were absent. No generated files or translations changed. The proprietary routine and offline harnesses were unavailable for independent inspection.; nothing run (a code review).
Checked and fine:

  • One visibility topic, with game-specific code in new host files and AI assistance recorded in commit trailers.
  • The generated gates are restricted to the supported build. The forward declarations cover the split-function reference.
  • The box kernel returns a decline without publishing its temporary registers. The grid frees its snapshot on success and decline.
  • The game-free fixture checks both box kernels against an instruction model and checks an empty grid against hand-calculated results.
  • No game files, generated outputs, third-party edits, shader or renderer interface changes, downloads, or network actions appear in the diff.

Blocking (to change before merge)

  1. host/oot_visibility.c:111: The production host compilation gets neither RECOMP_FPX_OPEN nor its definition from hos.h or the generated runtime a64.h. A normal Windows or Linux build compiling oot_visibility.c sees an undeclared call even with visibility off. The test injects the macro into its stand-in header and masks the failure. Expose the shared gate through a header used by the production host build and compile the hook with those headers in the fixture.
  2. host/oot_visibility.c:67: Two module workers entering visibility during initialization can race on the volatile gate variables and on vis_mode. A worker that sees vis_mode already set bypasses the acquire and can read unpublished logging state. This can happen with the switch unset. Use atomic accesses on both sides of each generated gate and acquire completed initialization before reading mode or logging state.
  3. host/oot_visibility.c:132: Box check mode ignores NZCV even though the native hook writes it. A flag-only difference passes, and the fixture's lifted stand-in never writes the model's flags back. Compare the flags using a reference that retains them, accounting for dead-flag elimination in the ordinary lift, and add a flag-only mismatch case.
  4. host/oot_visibility.c:166: If realloc or malloc fails while retaining a grid result, occ_keep silently drops that region. For example, a failed copy of the depth grid can hide a changed cell and count the check as taken. Record copy failure, free the partial capture, and decline or explicitly report an incomplete check. Cover this with allocation fault injection.
  5. host/oot_visibility.c:194: Grid check mode compares memory only. A native hook that fails to clear a saved vector register's upper half still gets counted as a match. Capture the expected register state before calling the lift and compare the state the replacement promises to preserve, including FPSR and the saved vector registers. Add a register-only mismatch case.
  6. host/visibility_occlusion.c:607: The snapshot reads entire capacity ranges before the routine establishes which arrays it uses. hos_guest_range only checks the reserved address window. An empty column strip with a null record pointer needs no record access in the lift, but the host snapshots a fabricated 48-byte record at guest address zero, which can fault on Windows. Skip unused ranges and validate every actual snapshot span before touching it. Add a fixture for an unused unmapped array.
  7. tests/test_native_visibility.py:592: The fixture treats compiler acceptance of -mfma as CPU support and applies it to both executables. An older x86 CPU can compile the fixture and then fault when the full-grid test executes FMA, although Longshot supports baseline builds on that CPU. Always exercise the baseline build, including the no-FMA decline, and run the FMA variant only after checking the executing CPU and OS support it.

Waiting on the author (a run, a measurement or a decision; no edit does it)

  • nx2/oot_patches.py:412: The default-off cost includes losing guest register localization in both hot functions, in addition to the gate. The offline approximation showed up to 10% extra box-call time and 7% extra grid-call time with visibility off before the declaration update. Repeat main versus default-off on the current code on Windows and Linux, including Tier 1 hardware, and resolve the earlier off-mode screenshot discrepancies using matched views. Preserve the optimized fallback if the cost persists.
  • The combined Windows run cannot establish this PR's gain or justify its default. The earlier Linux RTX 5080 run instead showed 7.5% more capped Market CPU time and 8.7% more Field CPU time with visibility on versus the same PR with it off. Use repeated interleaved Market and Field runs against the merge base and against PR-off to measure CPU time, average FPS, 1% lows, median and p99 frame times, and box early-exit frequency. Use the results to choose the default and decide whether both kernels earn their cost.
  • host/visibility_occlusion.c:927: The native grid marks whole capacity ranges, including pages where it performed no stores. The combined run saw the capture-copy thread use 10–18% more CPU, and the snapshot can itself allocate roughly 209 MiB at the permitted caps. On the isolated branch, measure dirty and copied pages, capture CPU, peak RAM, and frame pacing in a populated scene and under memory pressure. Narrow the writes and marks if the wider ranges add cost.
  • The description cites older game runs and partial checks. The current lift changes generated code, so record clean Windows and Linux build time and output size against the base, verify UTF-8 paths, and record lint, the full test suite, and CI on the updated head. Update Testing and Measurements with those results and distinguish the earlier farm findings from current-head results.

Suggestions (the author may take or leave)

  • Split the performance measurements by kernel to see whether the box test or grid builder independently justifies its maintenance cost.
  • Add a game-free populated-grid case with an independent expected result for coverage and depth. The current positive fixture has no occluder columns, and the grid check stand-in calls the kernel again.

Testing: Read-only review. No builds, tests, scripts, or game runs executed. The author reports earlier Windows gameplay, offline Unicorn comparisons, and partial game-free checks. The farm reports failures for 5f0c55d and a pending run for caca43f.
Overlaps and merge order: The Windows evidence combined eight sibling rewrites with this PR. No merge dependency or required order was identified in the checkout.


Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The production host build cannot see the floating-point gate used by both hooks, and the default-off path still has an initialization race. Check mode can report a match with state left unchecked, and the available runs leave the CPU gain and Linux regressions unresolved. Adds opt-in host implementations for two rendering visibility routines, with a lifted fallback and a diagnostic comparison mode. The potential CPU saving comes with two implementations to maintain, a changed lifted path even when disabled, and per-frame grid snapshots and dirty-page marking. **Does it help:** Unproven. Three interleaved Windows rounds with nine rewrites enabled used 2.7% less process CPU standing and 2.4% less turning in the capped Market. The combined off deltas of −0.3% and −0.2% were within the spread. Offline, box calls that exit before the plane tests took 28–35 ns versus 9–22 ns, and their in-game share is unknown. On the earlier Linux head, the RTX 5080 used 120.7 CPU seconds in the Market with visibility on versus 112.2 with it off and 98.9 on main. The current head lacks an isolated comparison. **Other systems:** The hooks run on module workers across GPU vendors and can affect model visibility, the GPU mirror, and frame pacing. The earlier Windows check run found no reported differences among 6,291,456 native box calls and 4,095 native grid calls, within the fields that check mode compared. The earlier Linux farm flagged screenshot differences with the switch off on both NVIDIA systems, and several p99 comparisons lacked enough usable starts. AMD, Steam Deck, older CPUs without FMA, and the current Windows and Linux builds still need coverage. The diff adds no save, audio, input, or path-format changes. **Switches:** HOS_NATIVE_VISIBILITY defaults off and accepts 1 or check. The default avoids the native kernels, but the gated lift still changes and the project requires a measured default. HOS_NATIVE_VISIBILITY_LOG defaults off and is forced on in check mode. That diagnostic default is appropriate. Both have switch-table rows. **Reviewed:** caca43f11c8e against a764653726ba: Read AGENTS.md, CONTRIBUTING.md, developer standards, the measurement guidance, the Forgejo template, the six commits, the complete diff, and all new host files, headers, and tests. Read the changed patcher and switch-table sections and traced the lift, split, host build, guest memory, and GPU marking paths. Skimmed unrelated portions of the existing files. Read the supplied recorded runs, comments, and stale review. The review checklist file and sibling Tooling checkout were absent. No generated files or translations changed. The proprietary routine and offline harnesses were unavailable for independent inspection.; nothing run (a code review). **Checked and fine:** - One visibility topic, with game-specific code in new host files and AI assistance recorded in commit trailers. - The generated gates are restricted to the supported build. The forward declarations cover the split-function reference. - The box kernel returns a decline without publishing its temporary registers. The grid frees its snapshot on success and decline. - The game-free fixture checks both box kernels against an instruction model and checks an empty grid against hand-calculated results. - No game files, generated outputs, third-party edits, shader or renderer interface changes, downloads, or network actions appear in the diff. **Blocking** (to change before merge) 1. host/oot_visibility.c:111: The production host compilation gets neither RECOMP_FPX_OPEN nor its definition from hos.h or the generated runtime a64.h. A normal Windows or Linux build compiling oot_visibility.c sees an undeclared call even with visibility off. The test injects the macro into its stand-in header and masks the failure. Expose the shared gate through a header used by the production host build and compile the hook with those headers in the fixture. 2. host/oot_visibility.c:67: Two module workers entering visibility during initialization can race on the volatile gate variables and on vis_mode. A worker that sees vis_mode already set bypasses the acquire and can read unpublished logging state. This can happen with the switch unset. Use atomic accesses on both sides of each generated gate and acquire completed initialization before reading mode or logging state. 3. host/oot_visibility.c:132: Box check mode ignores NZCV even though the native hook writes it. A flag-only difference passes, and the fixture's lifted stand-in never writes the model's flags back. Compare the flags using a reference that retains them, accounting for dead-flag elimination in the ordinary lift, and add a flag-only mismatch case. 4. host/oot_visibility.c:166: If realloc or malloc fails while retaining a grid result, occ_keep silently drops that region. For example, a failed copy of the depth grid can hide a changed cell and count the check as taken. Record copy failure, free the partial capture, and decline or explicitly report an incomplete check. Cover this with allocation fault injection. 5. host/oot_visibility.c:194: Grid check mode compares memory only. A native hook that fails to clear a saved vector register's upper half still gets counted as a match. Capture the expected register state before calling the lift and compare the state the replacement promises to preserve, including FPSR and the saved vector registers. Add a register-only mismatch case. 6. host/visibility_occlusion.c:607: The snapshot reads entire capacity ranges before the routine establishes which arrays it uses. hos_guest_range only checks the reserved address window. An empty column strip with a null record pointer needs no record access in the lift, but the host snapshots a fabricated 48-byte record at guest address zero, which can fault on Windows. Skip unused ranges and validate every actual snapshot span before touching it. Add a fixture for an unused unmapped array. 7. tests/test_native_visibility.py:592: The fixture treats compiler acceptance of -mfma as CPU support and applies it to both executables. An older x86 CPU can compile the fixture and then fault when the full-grid test executes FMA, although Longshot supports baseline builds on that CPU. Always exercise the baseline build, including the no-FMA decline, and run the FMA variant only after checking the executing CPU and OS support it. **Waiting on the author** (a run, a measurement or a decision; no edit does it) - nx2/oot_patches.py:412: The default-off cost includes losing guest register localization in both hot functions, in addition to the gate. The offline approximation showed up to 10% extra box-call time and 7% extra grid-call time with visibility off before the declaration update. Repeat main versus default-off on the current code on Windows and Linux, including Tier 1 hardware, and resolve the earlier off-mode screenshot discrepancies using matched views. Preserve the optimized fallback if the cost persists. - The combined Windows run cannot establish this PR's gain or justify its default. The earlier Linux RTX 5080 run instead showed 7.5% more capped Market CPU time and 8.7% more Field CPU time with visibility on versus the same PR with it off. Use repeated interleaved Market and Field runs against the merge base and against PR-off to measure CPU time, average FPS, 1% lows, median and p99 frame times, and box early-exit frequency. Use the results to choose the default and decide whether both kernels earn their cost. - host/visibility_occlusion.c:927: The native grid marks whole capacity ranges, including pages where it performed no stores. The combined run saw the capture-copy thread use 10–18% more CPU, and the snapshot can itself allocate roughly 209 MiB at the permitted caps. On the isolated branch, measure dirty and copied pages, capture CPU, peak RAM, and frame pacing in a populated scene and under memory pressure. Narrow the writes and marks if the wider ranges add cost. - The description cites older game runs and partial checks. The current lift changes generated code, so record clean Windows and Linux build time and output size against the base, verify UTF-8 paths, and record lint, the full test suite, and CI on the updated head. Update Testing and Measurements with those results and distinguish the earlier farm findings from current-head results. **Suggestions** (the author may take or leave) - Split the performance measurements by kernel to see whether the box test or grid builder independently justifies its maintenance cost. - Add a game-free populated-grid case with an independent expected result for coverage and depth. The current positive fixture has no occluder columns, and the grid check stand-in calls the kernel again. **Testing:** Read-only review. No builds, tests, scripts, or game runs executed. The author reports earlier Windows gameplay, offline Unicorn comparisons, and partial game-free checks. The farm reports failures for 5f0c55d and a pending run for caca43f. **Overlaps and merge order:** The Windows evidence combined eight sibling rewrites with this PR. No merge dependency or required order was identified in the checkout. --- <sub>Automated code review by BeanBot (gpt-6.2-sol), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.</sub>
@ -0,0 +64,4 @@
"(HOS_NATIVE_VISIBILITY)",
m == VIS_CHECK ? ", each call checked against the lifted routine" : "");
vis_mode = m;
hos_vis_on_80029240 = hos_vis_on_808226e0 = m != VIS_OFF;
Member

The workers race on these volatile gates and can bypass the acquire through vis_mode. Make both sides of the generated gate atomic and acquire initialization before reading its state.

The workers race on these volatile gates and can bypass the acquire through vis_mode. Make both sides of the generated gate atomic and acquire initialization before reading its state.
@ -0,0 +80,4 @@
why_name[WHY_TAKEN], k[WHY_TAKEN], why_name[WHY_FLOAT_MODE], k[WHY_FLOAT_MODE], why_name[WHY_RANGE],
k[WHY_RANGE], why_name[WHY_INPUT], k[WHY_INPUT], why_name[WHY_PATH], k[WHY_PATH],
vis_mode == VIS_CHECK ? ", checked: " : "", vis_mode == VIS_CHECK ? why_name[WHY_DIFF] : "",
vis_mode == VIS_CHECK ? k[WHY_DIFF] : 0);
Member

When logging with visibility on, the final format argument appends a stray 0 to the preceding count. Emit the difference suffix only in check mode.

When logging with visibility on, the final format argument appends a stray 0 to the preceding count. Emit the difference suffix only in check mode.
@ -0,0 +108,4 @@
int hos_vis_80029240(Ctx *c, A64Fn lifted) {
if (vis_mode < 0) vis_start();
if (vis_mode == VIS_OFF || (vis_mode == VIS_CHECK && vis_inside)) return 0; /* (the thread-local only in check mode: it can be a call) */
if (!RECOMP_FPX_OPEN(c)) return vis_count(&stats_box, WHY_FLOAT_MODE), 0;
Member

The normal host build does not get RECOMP_FPX_OPEN from hos.h or runtime a64.h. The test injects it. Share the real definition and compile the fixture against the production interface.

The normal host build does not get RECOMP_FPX_OPEN from hos.h or runtime a64.h. The test injects it. Share the real definition and compile the fixture against the production interface.
@ -0,0 +129,4 @@
/* (what the lifted code writes: x0, x8, x9, x10, v0..v3; it keeps only the NZCV bits it needs) */
VisBoxRegs g;
regs_from(c, &g);
if (g.x0 != r.x0 || g.x8 != r.x8 || g.x9 != r.x9 || g.x10 != r.x10 || memcmp(g.q, r.q, sizeof g.q)) {
Member

A flag-only mismatch passes this comparison. Use a reference that retains NZCV and test a mismatch limited to one flag.

A flag-only mismatch passes this comparison. Use a reference that retains NZCV and test a mismatch limited to one flag.
@ -0,0 +163,4 @@
k->r = r, k->cap = cap;
}
uint8_t *mine = (uint8_t *)malloc((size_t)n);
if (!mine) return;
Member

A failed capture allocation drops a region and can report a false match. Treat the check as incomplete and exercise allocation failure in the fixture.

A failed capture allocation drops a region and can report a false match. Treat the check as incomplete and exercise allocation failure in the fixture.
@ -0,0 +191,4 @@
vis_inside = 0;
int diff = -1;
uint64_t at = 0;
for (int i = 0; i < keep.n; i++) {
Member

A grid result with incorrect saved vector state still counts as a match. Compare the promised register state and test a register-only mismatch.

A grid result with incorrect saved vector state still counts as a match. Compare the promised register state and test a register-only mismatch.
@ -0,0 +604,4 @@
enum { I_C1 = 2, I_R1 = 3, I_C2 = 4, I_R2 = 5, I_K = 6, I_LOG = 7 };
int i_g = -1, i_t = -1, i_l = -1, i_b = -1, i_h = -1, i_f = -1;
if (ok && ncells >= 1) i_g = k, ok = add_reg(reg, &k, range, ld64(O + 0x668), 8 * (uint64_t)(uint32_t)ncells, 1);
if (ok && levels) {
Member

An unused null record array still produces a 48-byte snapshot range. The host window check accepts address zero, which can fault on Windows. Skip unused arrays and validate the ranges the snapshot touches.

An unused null record array still produces a 48-byte snapshot range. The host window check accepts address zero, which can fault on Windows. Skip unused arrays and validate the ranges the snapshot touches.
@ -0,0 +924,4 @@
if (!((int64_t)lvl < (int64_t)lv - 1)) break;
}
}
const uint8_t top = O[0x6d9];
Member

This marks entire capacities, including untouched pages. Isolate the PR's effect on copied pages, capture CPU, peak RAM, and frame pacing.

This marks entire capacities, including untouched pages. Isolate the PR's effect on copied pages, capture CPU, peak RAM, and frame pacing.
@ -406,0 +409,4 @@
# Each gets a gate first in its lifted function: hos_vis_on_<addr> is 1 until the first call has read the switch, then 0
# unless it is on (one load and an untaken branch when off); hos_vis_<addr>(c, f_<addr>) returns 1 when it answered.
VIS_GATES = ("80029240", "808226e0")
VIS_GATE = (" if (__builtin_expect(hos_vis_on_%s, 0) && hos_vis_%s(c, f_%s)) return; "
Member

The gate also changes how both lifted functions handle registers when off. Measure the current default-off build against main and include matched screenshots for the earlier Linux failures.

The gate also changes how both lifted functions handle registers when off. Measure the current default-off build against main and include matched screenshots for the earlier Linux failures.
@ -0,0 +589,4 @@
exe = tmp / ("fixture.exe" if os.name == "nt" else "fixture")
occ = tmp / ("occ_fixture.exe" if os.name == "nt" else "occ_fixture")
# (-mfma where the compiler takes it: the grid's build needs a fused multiply-add, as the game's build has)
fma = ["-mfma"] if subprocess.run([cc, "-mfma", "-x", "c", "-c", "-o", os.devnull, "-"], input="int x;\n",
Member

Compiler support for -mfma does not establish CPU support. Run the baseline fixture everywhere and gate the FMA executable on runtime support.

Compiler support for -mfma does not establish CPU support. Run the baseline fixture everywhere and gate the FMA executable on runtime support.
Co-Authored-By: Codex Sol <[email protected]>
(cherry picked from commit 58dc9611db58a8e5f6b4b38ad47993f05b913ac0)
Track actual guest page boundaries and crossing tile tails. Validate uniform bitmap ranges before writes so bulk marking can use a cheap walk; mixed ranges run lifted unchanged.

Co-Authored-By: Codex (GPT-6) <[email protected]>
(cherry picked from commit 7371d6f3dfac21b6fb24b6677eaab7b57a7c0050)
fix(build): the host's native hooks test the watch bitmap too (A64_MARK_BITMAP)
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval approved by HumanBean (reviewed)
beanstalk/linux 0/1 pass, 2 running, 0 queued
beanstalk/windows 0/1 pass, 2 running, 0 queued
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
537011092f
Every host file is now compiled with MARK_BITMAP, the lifted code's own setting, instead of none (or, on the crowd
branch, only host/oot_crowd.c). A native hook's A64_MARK then tests the first page's watch bit exactly as the lifted
store it stands for does, and the hooks' #ifdef A64_GPU_PAGE guards (the uniform-bitmap preflights in reverb and
visibility) are compiled in. hos_mark_written / HOS_WRITTEN, the host's own marks, still mark every page.
RX_MARK_BITMAP=0 turns the bitmap off for the lifted code and the hooks alike.

tests/test_mark_bitmap_host.py reads nx2/build.py (no build) and fails if the lifted flags or a host compile job
lack an unconditional MARK_BITMAP.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Owner
@fix
Owner

@Beanstalk

@Beanstalk
Member
Beanstalk test farm · an earlier run: commit 5370110 (the newer results are in a comment below)

Commit 5370110 · approved by HumanBean (reviewed) · dashboard

⚠️ Note: The explicit off value exercises the default-off path; main ignores both values of the new switch.
⚠️ Note: Check mode and its per-routine coverage counts are omitted to keep the run short, so this plan does not establish bit-for-bit agreement or detect transient popping between screenshots.
⚠️ Note: Copied GPU-mirror pages and allocation-failure or register-only mismatch fixtures cannot be measured directly by these gameplay probes.

Started at 07:34 UTC.

Estimate: done in about 2 h 15 min, from the queue and the testers online now.

Target Result Reports VRAM peak (MB) Crashes
linux 🔄 running 0 of 1 — —
windows 🔄 running 0 of 1 — —

visibility-route · frame time p99 (ms)

Target main HOS_NATIVE_VISIBILITY=0 (reference) HOS_NATIVE_VISIBILITY=1
linux — — —
windows — — —
What was tested

What will be tested · AI plan (gpt-6.1-terra)

On Linux and Windows, the plan compares the default-off and enabled visibility paths in the Market and Hyrule Field, with matched pictures and capped and uncapped measurements. It records CPU time, peak memory and frame times, and also times lift builds; gameplay takes about 21 minutes per tester. Crashes, substantial performance regressions or screenshot differences fail the pull request.

  • Where: linux (1-2 reports), windows (1-2 reports).
  • Matrix: 2 cells, each on main and the PR: HOS_NATIVE_VISIBILITY 0 or 1 (PR only).
  • visibility-route: from the repository's save market-entrance (settings frame_rate_cap=0), then teleport to Market, take the picture market to compare with main, measure market-capped for 30 s (after 5 s warm-up), set frame_rate_cap to 6, move the stick, take the picture market-turned to compare with main, measure market-uncapped for 20 s (after 5 s warm-up), set frame_rate_cap to 0, teleport to Hyrule Field, take the picture field to compare with main, measure field-capped for 30 s (after 5 s warm-up), take the picture field-turned to compare with main, measure field-uncapped for 20 s (after 5 s warm-up); 1 run on each side for each of 2 cells (2 game starts a run).
  • Build times: main and the PR each built 2 times, alternating, each whole in a fresh folder: the lift again (no kept lift, main's compiled objects kept); every stage timed (shader preparation is not), and the lifted C compared.
  • Measured: crashes, frame times, stutters and hitches, memory and CPU time, draws and frame rate.
  • The PR fails if: it crashes; or the PR's CPU time with HOS_NATIVE_VISIBILITY=1 is more than 5% above HOS_NATIVE_VISIBILITY=0; or the PR's frame time (p99) with HOS_NATIVE_VISIBILITY=1 is more than 15% above HOS_NATIVE_VISIBILITY=0; or a picture differs from main by more than 10%; or peak memory is more than 15% above main; or frame time (p99) is more than 15% above main; or frame time (worst 1%) is more than 15% above main.
  • Time: about 28 min of game runs per tester (4 game starts), plus the two builds and 4 timed builds (about 80 min).
<!-- beanstalk:previous --> <details><summary>Beanstalk test farm · an earlier run: commit <code>5370110</code> (the newer results are in a comment below)</summary> Commit `5370110` · approved by HumanBean (reviewed) · [dashboard](https://beanstalk.magicbean.dev/prs/1902) ⚠️ **Note:** The explicit off value exercises the default-off path; main ignores both values of the new switch. ⚠️ **Note:** Check mode and its per-routine coverage counts are omitted to keep the run short, so this plan does not establish bit-for-bit agreement or detect transient popping between screenshots. ⚠️ **Note:** Copied GPU-mirror pages and allocation-failure or register-only mismatch fixtures cannot be measured directly by these gameplay probes. Started at 07:34 UTC. **Estimate:** done in about 2 h 15 min, from the queue and the testers online now. | Target | Result | Reports | VRAM peak (MB) | Crashes | | --- | --- | --- | --- | --- | | `linux` | 🔄 running | 0 of 1 | — | — | | `windows` | 🔄 running | 0 of 1 | — | — | **`visibility-route`** · frame time p99 (ms) | Target | main | `HOS_NATIVE_VISIBILITY=0` (reference) | `HOS_NATIVE_VISIBILITY=1` | | --- | --- | --- | --- | | `linux` | — | — | — | | `windows` | — | — | — | <details><summary>What was tested</summary> **What will be tested** · AI plan (`gpt-6.1-terra`) On Linux and Windows, the plan compares the default-off and enabled visibility paths in the Market and Hyrule Field, with matched pictures and capped and uncapped measurements. It records CPU time, peak memory and frame times, and also times lift builds; gameplay takes about 21 minutes per tester. Crashes, substantial performance regressions or screenshot differences fail the pull request. - **Where:** linux (1-2 reports), windows (1-2 reports). - **Matrix:** 2 cells, each on main and the PR: HOS_NATIVE_VISIBILITY `0` or `1` (PR only). - **visibility-route:** from the repository's save `market-entrance` (settings frame_rate_cap=0), then teleport to Market, take the picture market to compare with main, measure market-capped for 30 s (after 5 s warm-up), set frame_rate_cap to 6, move the stick, take the picture market-turned to compare with main, measure market-uncapped for 20 s (after 5 s warm-up), set frame_rate_cap to 0, teleport to Hyrule Field, take the picture field to compare with main, measure field-capped for 30 s (after 5 s warm-up), take the picture field-turned to compare with main, measure field-uncapped for 20 s (after 5 s warm-up); 1 run on each side for each of 2 cells (2 game starts a run). - **Build times:** main and the PR each built 2 times, alternating, each whole in a fresh folder: the lift again (no kept lift, main's compiled objects kept); every stage timed (shader preparation is not), and the lifted C compared. - **Measured:** crashes, frame times, stutters and hitches, memory and CPU time, draws and frame rate. - **The PR fails if:** it crashes; or the PR's CPU time with HOS_NATIVE_VISIBILITY=1 is more than 5% above HOS_NATIVE_VISIBILITY=0; or the PR's frame time (p99) with HOS_NATIVE_VISIBILITY=1 is more than 15% above HOS_NATIVE_VISIBILITY=0; or a picture differs from main by more than 10%; or peak memory is more than 15% above main; or frame time (p99) is more than 15% above main; or frame time (worst 1%) is more than 15% above main. - **Time:** about 28 min of game runs per tester (4 game starts), plus the two builds and 4 timed builds (about 80 min). </details> </details>
Member

I changed this pull request where an edit could (the rest needs runs, measurements or decisions): the new head is e1e0089257. I rebased it onto main (7486869e64).

Conflicts resolved

  • nx2/oot_patches.py: Kept both the base branch’s sprint warning and the pull request’s visibility gate warning.

Changed

  • Synchronize visibility initialization: Made gate accesses atomic and acquire completed initialization before reading published state in host/oot_visibility.c:37 and nx2/oot_patches.py:412.
  • Synchronize GPU bitmap reads: Added acquire loads for bitmap reads in nx2/rtgen.py:69 and rechecked written pages for mappings in host/oot_visibility.c:166.
  • Validate grid dimensions: Validate both dimension pairs and their wide products before acquiring ranges in host/visibility_occlusion.c:592; added overflow cases in tests/test_native_visibility.py:547.
  • Handle incomplete comparisons: Track allocation failures and decline to the lifted routine with an incomplete-comparison log in host/oot_visibility.c:179.
  • Check CPU FMA support: Probe CPU support with a baseline executable before using FMA, retaining baseline fixture checks in tests/test_native_visibility.py:639. Tests were not run.
  • Honor the C compiler: Honor CC, report an unavailable requested compiler, and search for C compilers in tests/test_native_visibility.py:618.

Left for the author

  • Isolated visibility performance and cost attribution: Measure this head alone against its merge base in representative scenes, including capped CPU time, uncapped frame times and 1% lows. Attribute the early-out and capture-copy costs; those results are needed before choosing which hook or grid costs to reduce.

Still for the author (no edit does these)

  • L8 (minor): Run this head in game on Windows and the affected Tier 1 Linux and GPU systems, including check mode and the default-off comparison. The reported game run predates the rebase and bitmap changes, and does not establish their behavior on those systems.

Nothing was built or run: check the change. It will be reviewed again.


Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.

I changed this pull request where an edit could (the rest needs runs, measurements or decisions): the new head is e1e008925700. I rebased it onto `main` (7486869e64b1). **Conflicts resolved** - nx2/oot_patches.py: Kept both the base branch’s sprint warning and the pull request’s visibility gate warning. **Changed** - **Synchronize visibility initialization**: Made gate accesses atomic and acquire completed initialization before reading published state in host/oot_visibility.c:37 and nx2/oot_patches.py:412. - **Synchronize GPU bitmap reads**: Added acquire loads for bitmap reads in nx2/rtgen.py:69 and rechecked written pages for mappings in host/oot_visibility.c:166. - **Validate grid dimensions**: Validate both dimension pairs and their wide products before acquiring ranges in host/visibility_occlusion.c:592; added overflow cases in tests/test_native_visibility.py:547. - **Handle incomplete comparisons**: Track allocation failures and decline to the lifted routine with an incomplete-comparison log in host/oot_visibility.c:179. - **Check CPU FMA support**: Probe CPU support with a baseline executable before using FMA, retaining baseline fixture checks in tests/test_native_visibility.py:639. Tests were not run. - **Honor the C compiler**: Honor CC, report an unavailable requested compiler, and search for C compilers in tests/test_native_visibility.py:618. **Left for the author** - **Isolated visibility performance and cost attribution**: Measure this head alone against its merge base in representative scenes, including capped CPU time, uncapped frame times and 1% lows. Attribute the early-out and capture-copy costs; those results are needed before choosing which hook or grid costs to reduce. **Still for the author** (no edit does these) - **L8** (minor): Run this head in game on Windows and the affected Tier 1 Linux and GPU systems, including check mode and the default-off comparison. The reported game run predates the rebase and bitmap changes, and does not establish their behavior on those systems. Nothing was built or run: check the change. It will be reviewed again. --- <sub>Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.</sub>
BeanBot force-pushed perf/native-visibility from 537011092f
Some checks failed
PR description / Description follows the template (pull_request_target) Successful in 7s
Website / Website on Cloudflare (pull_request_target) Has been skipped
beanstalk/approval approved by HumanBean (reviewed)
beanstalk/linux 0/1 pass, 2 running, 0 queued
beanstalk/windows 0/1 pass, 2 running, 0 queued
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
CI / Website build and checks (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to e1e0089257
Some checks failed
beanstalk/approval not approved yet
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 24s
PR description / Description follows the template (pull_request_target) Successful in 28s
CI / Workflow lint (actionlint) (pull_request) Successful in 37s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 37s
Third-party parts / As their notes say (pull_request) Successful in 47s
CI / Website build and checks (pull_request) Has been cancelled
2026-11-21 07:39:26 +00:00
Compare
Member

Beanstalk test farm

Commit 41c96c3 · approved by nobody yet · dashboard

A member of BigBeans can review this commit and comment @Beanstalk to test it.

<!-- beanstalk:sticky --> ### Beanstalk test farm Commit `41c96c3` · approved by nobody yet · [dashboard](https://beanstalk.magicbean.dev/prs/1902) A member of `BigBeans` can review this commit and comment `@Beanstalk` to test it.
BeanBot left a comment

Recommendation: Request changes. The native box test can read inaccessible memory that the lifted routine would never touch, and check mode can miss differences in the grid’s writes. The claimed performance gain for this branch and the cost of its changes with the switch off remain unproven.

This adds optional host implementations of two visibility routines, with lifted fallbacks and a comparison mode. It also changes GPU write tracking for host code and lifted stores regardless of the new switch; the potential CPU saving is not yet isolated from eight other rewrites.

@fix will do (6)

  1. L1 host/visibility_kernels.c:173: The SIMD path loads four planes before it knows which plane ends the test, while vis_range only checks the reserved guest window, not committed memory. (more on the line)
  2. L2 host/oot_visibility.c:224: Check mode compares only regions reported by the native implementation. (more on the line)
  3. L3: The description says nothing changes with the visibility switch unset and caches do not change, but the PR also changes the bitmap load in every lifted store and adds a compile flag to every host job. The latter changes host object-cache keys. Describe these switch-independent changes and their rebuild and runtime implications.
  4. L4: The head commit is authored by BeanBot but has no AI-assistance trailer, although the checklist says assistance is recorded. Add the appropriate Assisted-by or Co-Authored-By trailer for that commit.
  5. L5 (its part an edit can do): Reduce or bypass the hook’s fixed work on early-exit calls, or remove an implementation that does not show a gain.
  6. L6 (its part an edit can do) nx2/rtgen.py:69: Limit the host flag to files that need it and avoid changing the shared store path if an equivalent synchronized implementation costs less.

For the author (3) (a run, a measurement or a decision)

  • L5 (minor): Measure this head against its base and with the visibility switch on and off in the same build. Include CPU time, uncapped frame times and lows, and the proportion of early-exit box calls; the combined nine-branch run cannot establish this PR’s benefit or its default.
  • L6 (minor) nx2/rtgen.py:69: The atomic bitmap read runs on lifted stores even with visibility off, and the changed host flags invalidate cached host objects. (more on the line)
  • L7 (minor): Run this head in check and on modes on Windows and Linux, and cover affected Tier 1 AMD and Steam Deck hardware. The reported no-difference run used an earlier version on Windows/NVIDIA and does not establish current-head correctness across the supported systems.
More: does it help, other systems, switches, what was read

Does it help: Unproven. The Windows measurement reports 2.7% less whole-process CPU standing and 2.4% less turning, but nine branches were enabled together on an older base. The offline box timings also show early-exit calls slowing from 9–22 ns to 28–35 ns, with their in-game share unknown. No current-head comparison isolates this PR or reports uncapped frame-time and low-FPS effects.

Other systems: The visibility paths can affect model culling on both Windows and Linux, including Steam Deck and other CPU classes. The bitmap and generated-header changes affect GPU write tracking even with visibility disabled. The reported game check covered an older version on Windows/NVIDIA; this head, Linux, AMD and Steam Deck remain unverified. The host compile flag also changes object-cache keys and can cause recompilation.

Switches: HOS_NATIVE_VISIBILITY is new and defaults off; 1 attempts native execution and check compares with the lift. Off is conservative but has not been selected by the required isolated measurements. HOS_NATIVE_VISIBILITY_LOG is new and defaults off; it enables diagnostic counts and is also enabled by check mode. RX_MARK_BITMAP retains its default but now also controls host compilation, extending its scope.

Reviewed: e1e0089257 against 7486869e64: Read AGENTS.md, CONTRIBUTING.md, the pull request template, relevant standards, the commit log and diff, the new host visibility files and tests, and the changed build and patch sections. Examined guest-range and bitmap definitions and their writers. The repository has no docs/review-checklist.md, and the adjacent Tooling rules were unavailable. The large unchanged portions of build.py, oot_patches.py and rtgen.py, and unrelated generated and translation files, were skimmed rather than read in full.; nothing run (a code review).

Checked and fine:

  • Both new environment variables have switch-table rows.
  • The initialization gate and diagnostic counters use atomic accesses, and check-mode recursion is thread-local.
  • The grid restores its saved writable regions on a declined full-build path and handles snapshot allocation failure by falling back.
  • The test probes CPU FMA support before executing its FMA build.
  • No third-party files or generated shader headers are changed; the renderer API interface is unchanged.

Testing: Read-only review; no builds, tests, project scripts or game runs were executed. The supplied CI reports no head-only failures.

Overlaps and merge order: Not checked: network access is closed.


Automated code review by BeanBot (gpt-6.1-terra), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.

**Recommendation: Request changes.** The native box test can read inaccessible memory that the lifted routine would never touch, and check mode can miss differences in the grid’s writes. The claimed performance gain for this branch and the cost of its changes with the switch off remain unproven. This adds optional host implementations of two visibility routines, with lifted fallbacks and a comparison mode. It also changes GPU write tracking for host code and lifted stores regardless of the new switch; the potential CPU saving is not yet isolated from eight other rewrites. **`@fix` will do (6)** 1. **L1** `host/visibility_kernels.c:173`: The SIMD path loads four planes before it knows which plane ends the test, while vis_range only checks the reserved guest window, not committed memory. (more on the line) 2. **L2** `host/oot_visibility.c:224`: Check mode compares only regions reported by the native implementation. (more on the line) 3. **L3**: The description says nothing changes with the visibility switch unset and caches do not change, but the PR also changes the bitmap load in every lifted store and adds a compile flag to every host job. The latter changes host object-cache keys. Describe these switch-independent changes and their rebuild and runtime implications. 4. **L4**: The head commit is authored by BeanBot but has no AI-assistance trailer, although the checklist says assistance is recorded. Add the appropriate Assisted-by or Co-Authored-By trailer for that commit. 5. **L5** (its part an edit can do): Reduce or bypass the hook’s fixed work on early-exit calls, or remove an implementation that does not show a gain. 6. **L6** (its part an edit can do) `nx2/rtgen.py:69`: Limit the host flag to files that need it and avoid changing the shared store path if an equivalent synchronized implementation costs less. **For the author (3)** (a run, a measurement or a decision) - **L5** (minor): Measure this head against its base and with the visibility switch on and off in the same build. Include CPU time, uncapped frame times and lows, and the proportion of early-exit box calls; the combined nine-branch run cannot establish this PR’s benefit or its default. - **L6** (minor) `nx2/rtgen.py:69`: The atomic bitmap read runs on lifted stores even with visibility off, and the changed host flags invalidate cached host objects. (more on the line) - **L7** (minor): Run this head in check and on modes on Windows and Linux, and cover affected Tier 1 AMD and Steam Deck hardware. The reported no-difference run used an earlier version on Windows/NVIDIA and does not establish current-head correctness across the supported systems. <details> <summary>More: does it help, other systems, switches, what was read</summary> **Does it help:** Unproven. The Windows measurement reports 2.7% less whole-process CPU standing and 2.4% less turning, but nine branches were enabled together on an older base. The offline box timings also show early-exit calls slowing from 9–22 ns to 28–35 ns, with their in-game share unknown. No current-head comparison isolates this PR or reports uncapped frame-time and low-FPS effects. **Other systems:** The visibility paths can affect model culling on both Windows and Linux, including Steam Deck and other CPU classes. The bitmap and generated-header changes affect GPU write tracking even with visibility disabled. The reported game check covered an older version on Windows/NVIDIA; this head, Linux, AMD and Steam Deck remain unverified. The host compile flag also changes object-cache keys and can cause recompilation. **Switches:** HOS_NATIVE_VISIBILITY is new and defaults off; 1 attempts native execution and check compares with the lift. Off is conservative but has not been selected by the required isolated measurements. HOS_NATIVE_VISIBILITY_LOG is new and defaults off; it enables diagnostic counts and is also enabled by check mode. RX_MARK_BITMAP retains its default but now also controls host compilation, extending its scope. **Reviewed:** e1e008925700 against 7486869e64b1: Read AGENTS.md, CONTRIBUTING.md, the pull request template, relevant standards, the commit log and diff, the new host visibility files and tests, and the changed build and patch sections. Examined guest-range and bitmap definitions and their writers. The repository has no docs/review-checklist.md, and the adjacent Tooling rules were unavailable. The large unchanged portions of build.py, oot_patches.py and rtgen.py, and unrelated generated and translation files, were skimmed rather than read in full.; nothing run (a code review). **Checked and fine:** - Both new environment variables have switch-table rows. - The initialization gate and diagnostic counters use atomic accesses, and check-mode recursion is thread-local. - The grid restores its saved writable regions on a declined full-build path and handles snapshot allocation failure by falling back. - The test probes CPU FMA support before executing its FMA build. - No third-party files or generated shader headers are changed; the renderer API interface is unchanged. **Testing:** Read-only review; no builds, tests, project scripts or game runs were executed. The supplied CI reports no head-only failures. **Overlaps and merge order:** Not checked: network access is closed. </details> --- <sub>Automated code review by BeanBot (gpt-6.1-terra), asked for by BeanBot, after its fix. Nothing was run; a maintainer decides.</sub>
@ -0,0 +221,4 @@
vis_inside = 0;
int diff = -1;
uint64_t at = 0;
for (int i = 0; i < keep.n; i++) {
Member

L2: Check mode compares only regions reported by the native implementation. If a native path omits a store that the lifted routine makes outside those regions, no difference is logged, so a successful check can conceal precisely the missing write that would leave the native result stale. Snapshot and compare all potentially written allocations, including bytes the native path did not report, and add a case where the lifted stand-in writes outside the native region.

**L2**: Check mode compares only regions reported by the native implementation. If a native path omits a store that the lifted routine makes outside those regions, no difference is logged, so a successful check can conceal precisely the missing write that would leave the native result stale. Snapshot and compare all potentially written allocations, including bytes the native path did not report, and add a case where the lifted stand-in writes outside the native region.
@ -0,0 +170,4 @@
game's for each plane, so each lane's bits are that plane's. Planes past the count (or past 6) are zero lanes and
not looked at. Out: T (the sums plus distance), D (the distances), bad (a bit a plane out of class), gt (t > 0) */
static inline void plane_group(const uint8_t *obj, uint32_t first, uint32_t n, __m128 mn, __m128 mx, __m128 *T, __m128 *D,
int *bad, int *gt) {
Member

L1: The SIMD path loads four planes before it knows which plane ends the test, while vis_range only checks the reserved guest window, not committed memory. If the first plane culls the box and a later plane lies on an uncommitted page, the lifted routine returns after the first plane but this path faults while preloading the later one. Validate that every speculatively read page is committed, or avoid reading planes beyond the one the lifted routine would test.

**L1**: The SIMD path loads four planes before it knows which plane ends the test, while vis_range only checks the reserved guest window, not committed memory. If the first plane culls the box and a later plane lies on an uncommitted page, the lifted routine returns after the first plane but this path faults while preloading the later one. Validate that every speculatively read page is committed, or avoid reading planes beyond the one the lifted routine would test.
@ -67,3 +67,3 @@
before is caught by its first byte; read-only on the store path */
extern uint8_t a64_gpu_pages[A64_DIRTY_PAGES / 8];
#define A64_GPU_PAGE(a) ((a64_gpu_pages[A64_DIRTY_AT(a) >> 3] >> (A64_DIRTY_AT(a) & 7)) & 1)
#define A64_GPU_PAGE(a) ((__atomic_load_n(&a64_gpu_pages[A64_DIRTY_AT(a) >> 3], __ATOMIC_ACQUIRE) >> (A64_DIRTY_AT(a) & 7)) & 1)
Member

L6 (minor): The atomic bitmap read runs on lifted stores even with visibility off, and the changed host flags invalidate cached host objects. Measure current-head switch-off frame times and clean and incremental build time against the base; the older combined run predates these changes.

**L6** (minor): The atomic bitmap read runs on lifted stores even with visibility off, and the changed host flags invalidate cached host objects. Measure current-head switch-off frame times and clean and incremental build time against the base; the older combined run predates these changes.
perf(host): HOS_NATIVE_VISIBILITY is on by default; =0 runs the visibility routines lifted
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
beanstalk/approval not approved yet
PR description / Description follows the template (pull_request_target) Successful in 17s
41c96c3a84
docs/dev/standards.md: a switch ships at its value from the branch's first push, never off because it is new.
The row's kind is kill, unset reads as 1 (env_on_or), and the test runs unset as on and 0 / off as off.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
snaildev99 changed title from WIP: perf(host): the box-against-view test and the occlusion grid's build on host code, bit for bit (HOS_NATIVE_VISIBILITY, off by default) to WIP: perf(host): the box-against-view test and the occlusion grid's build on host code, bit for bit (HOS_NATIVE_VISIBILITY, on by default) 2026-11-21 07:46:14 +00:00
Author
Contributor

HOS_NATIVE_VISIBILITY is now on by default (41c96c3a, on top of BeanBot's e1e00892), as docs/dev/standards.md asks: a switch ships at its value from the first push, not off because it is new. HOS_NATIVE_VISIBILITY=0 gives main's lifted code. The row's kind is kill, unset reads as on (env_on_or), and the test runs it unset and =1 (on), =0 and =off (off) and check; it fails when the old read is put back. Title and description updated to match.

`HOS_NATIVE_VISIBILITY` is now on by default (41c96c3a, on top of BeanBot's e1e00892), as docs/dev/standards.md asks: a switch ships at its value from the first push, not off because it is new. `HOS_NATIVE_VISIBILITY=0` gives main's lifted code. The row's kind is `kill`, unset reads as on (`env_on_or`), and the test runs it unset and `=1` (on), `=0` and `=off` (off) and `check`; it fails when the old read is put back. Title and description updated to match.
HumanBean left a comment

Recommendation: Request changes. The grid build and the box test's host code are the game's Arm routines transcribed one instruction at a time, which CONTRIBUTING.md's Licensing section rules out, the same problem that blocks sibling #1911. Separately, the switch ships on although the only measurement of this branch alone leans the other way.

For a player this would mean a little less CPU time in the per-model culling job and the occlusion grid build each frame (0.46 % and 0.69 % of the game's code time in the Market). The cost is ~1,500 lines of bit-exact float code plus a gate in two lifted functions. As it stands, the code doesn't come out better on net.

Does it help: Unproven, and the one isolated run leans No.

  • The Windows numbers (-2.7 % / -2.4 % process CPU) come from nine rewrite branches merged together on an older main, so they don't isolate this PR.
  • The farm ran this branch on its own (at 5f0c55d, Linux, RTX 5070 and RTX 5080, one start a side). Market standing CPU was 112.2 → 120.7 s (+7.5 %) and Field standing 69.1 → 75.2 s (+8.7 %) with the switch on against =0 in the same build. Uncapped p99 was +20 % / +24 % against main.
  • Those are single runs, and the farm's own main column differs by up to 30 % between cells (Field standing 62.5 against 43.6). So this is noise-level evidence, but it points the wrong way and there is nothing isolated pointing the right way.
  • The farm run at 5370110 was superseded before it reported. Offline, the box hook is slower than the lift on calls that end before the planes (28-35 ns against 9-22 ns), and their share in the game is unknown.

Other systems:

  • The gates sit in two lifted functions for everyone, whatever the switch says. The farm's =0 against main is inside its noise.
  • At this head, BeanBot's acquire load in nx2/rtgen.py A64_GPU_PAGE changes every lifted store's mark and rt/a64.h, so every player's whole lift recompiles, with the switch on or off. It is unmeasured and outside this PR's topic.
  • nx2/build.py now gives every host file -DA64_MARK_BITMAP. No host file uses A64_MARK itself today (only comments), so behaviour is unchanged, but every host object's cache key changes.
  • The build-time rows at 5f0c55d say +16 % whole build (+75 s compile), which nothing in the diff explains. Players compile locally, so this needs measuring again.
  • On Windows, the grid's snapshot of an unused record array at a null pointer is not a crash: commit_on_touch in host/mkw_glue.c commits the page and it reads zero. AMD, Intel, the Steam Deck and CPUs without FMA (every grid call declines there) are not run.

Switches:

  • HOS_NATIVE_VISIBILITY (kill, on by default, =0 lifted, check): the on default is not backed by a measurement of this PR. The one isolated run shows more CPU with it on. It should not ship on until an isolated, repeated, interleaved measurement shows a gain past the noise on Linux and Windows. If that measurement shows no gain, the change doesn't earn its code.
  • HOS_NATIVE_VISIBILITY_LOG (diag, off; always on in check mode): the right default.

Reviewed: 41c96c3a8 against 7486869e6 (its merge base), all 12 commits.

  • Read in full: host/oot_visibility.c, host/visibility_kernels.c/.h, host/visibility_occlusion.c/.h, the changed parts of nx2/oot_patches.py, nx2/build.py, nx2/rtgen.py and gpu/vk/switches.def, and tests/test_mark_bitmap_host.py.
  • Skimmed: tests/test_native_visibility.py, for what it runs (it compiles and runs fixtures in a temp dir and touches nothing else).
  • I did not check bit-exactness against the game's routine. Nothing was run (a code review).

Fixed in this review: nothing landed on the head. A rebase onto main b62932a96 was clean commit by commit, with no conflicts. A one-line commit putting A64_GPU_PAGE back to main's plain load was made, but the push to the head branch was refused. Both are left as suggestions below.

Checked and fine:

  • The gates are only in the supported build, and they are declared at file scope for split pieces.
  • vis_start publishes the mode with release/acquire. A worker that loses the race spins until the mode is published.
  • The SSE box kernel only reads planes inside the 0xe8 bytes vis_range checked, and lanes past the count are masked out of both gts and bads. Its x8/x9/x10 match the scalar loop's.
  • The grid's written() regions cover every store on the success path: O+0x288..0x4ea, both strip headers, 0x690..0x6a9, 0x6da, the used plane records, the grid, the tiles by page and the bounds.
  • A decline restores every writable region, and check mode declines when an allocation fails.
  • Both switches have rows, and nothing touches third_party/.

Blocking (to change before merge)

  1. Licensing: the host code is a transcription of the game's code, not code written from its behaviour.
    • host/visibility_occlusion.c:3 says "The game's operations in the game's order: every Arm FMADD / FMSUB / FNMADD / FNMSUB is one fused multiply-add here".
    • planes_80823e00 (host/visibility_occlusion.c:226-300, :331-420) is one statement per instruction, with the temporaries named after the routine's registers: float s17 = f_sub(bad, p58, p4c); (:233), float s22 = -f_mul(bad, s20, s29); (:244), s31b, s30b, s28b (:265). Comments like /* (s8*s8 + s31*s31 + s9*s9) */ record the register order.
    • low_corners / tall_corners (:483-560) and the grid loops follow the same pattern.
    • The box kernel's comments are its disassembly: host/visibility_kernels.c:57 "LDR q0, a; LDR q1, b; FCMGT v0.4s; ADDV s0; FMOV w8, s0", :121 "EXT #8 then FADD .2s", :122 "FADDP", :134 "LDR s3".
    • The hooks write back dead register values (x8, x9, x10, v2, v3 and NZCV at host/oot_visibility.c:10; v8..v15's upper halves from "its LDP d8..d15 restore" at :246). Those can only come from the routine's instruction listing.
    • CONTRIBUTING.md allows host code written from facts about behaviour (what a routine computes, its inputs and outputs) in your own words. It does not allow the game's instructions rendered as C.
    • game_logf follows the public optimized-routines logf and reads the game's table from guest memory at run time, which is fine on its own. The rest needs rewriting from a description of what the routines compute, or dropping.
  2. gpu/vk/switches.def:108: on by default with no isolated measurement, while the only isolated run shows +7.5 % / +8.7 % CPU with it on (above). Measure this branch alone, against its base and against =0 in the same build, with several interleaved runs a side on Linux and Windows: capped CPU, uncapped frame times and 1 % lows, and the share of early-exit box calls. Choose the default from those runs.
  3. nx2/rtgen.py:69: the __ATOMIC_ACQUIRE load (BeanBot's e1e00892) changes every lifted store in the game, recompiles the whole lift and was never measured, and it has nothing to do with visibility. Put main's line back: #define A64_GPU_PAGE(a) ((a64_gpu_pages[A64_DIRTY_AT(a) >> 3] >> (A64_DIRTY_AT(a) & 7)) & 1). If the bitmap's race should be fixed, that belongs in its own measured PR, and a relaxed load would be enough there.
  4. Build time: the farm measured +16 % whole build (+19 % compile) at 5f0c55d. Measure it again at the new head and explain it, or remove the cause, since players build locally.

Suggestions (the author may take or leave)

  • Give -DA64_MARK_BITMAP only to oot_visibility.c, the one host file that tests A64_GPU_PAGE, rather than to every host compile job, so the other host objects keep their cache.
  • host/oot_visibility.c:136: box check mode still doesn't compare NZCV, and grid check mode compares only the regions the host reports, not registers. Either compare them or narrow the "register for register" wording in the description.
  • Rebase onto current main: it is clean commit by commit, with no conflicts.

Testing:

  • The only in-game run of this code was at 38f0ad29, in a merged test tree on Windows/NVIDIA, where check mode found no difference.
  • This head is not built or played on any OS. Linux has only the farm's noisy run at 5f0c55d, which failed (CPU and screenshot limits; the screenshot differences also appear with =0, so they look like scene noise).
  • CI is not looked at here. All of this holds the merge.

Overlaps and merge order:

  • The siblings #1898, #1899, #1900, #1903, #1904, #1905, #1909, #1910, #1911, #2001 and #2036 all add rows next to these in gpu/vk/switches.def and hooks in nx2/oot_patches.py. Expect textual conflicts in whatever merges second.
  • #1904 (libm, logf): this PR carries its own game_logf. If 1904 lands a host logf, use one copy.
  • #1911 has the same licensing blocker. The Windows measurement is of all the rewrite branches together, so none of them can claim it alone.
**Recommendation: Request changes.** The grid build and the box test's host code are the game's Arm routines transcribed one instruction at a time, which CONTRIBUTING.md's Licensing section rules out, the same problem that blocks sibling https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1911. Separately, the switch ships on although the only measurement of this branch alone leans the other way. For a player this would mean a little less CPU time in the per-model culling job and the occlusion grid build each frame (0.46 % and 0.69 % of the game's code time in the Market). The cost is ~1,500 lines of bit-exact float code plus a gate in two lifted functions. As it stands, the code doesn't come out better on net. **Does it help:** Unproven, and the one isolated run leans No. - The Windows numbers (-2.7 % / -2.4 % process CPU) come from nine rewrite branches merged together on an older main, so they don't isolate this PR. - The farm ran this branch on its own (at 5f0c55d, Linux, RTX 5070 and RTX 5080, one start a side). Market standing CPU was 112.2 → 120.7 s (+7.5 %) and Field standing 69.1 → 75.2 s (+8.7 %) with the switch on against `=0` in the same build. Uncapped p99 was +20 % / +24 % against main. - Those are single runs, and the farm's own main column differs by up to 30 % between cells (Field standing 62.5 against 43.6). So this is noise-level evidence, but it points the wrong way and there is nothing isolated pointing the right way. - The farm run at 5370110 was superseded before it reported. Offline, the box hook is slower than the lift on calls that end before the planes (28-35 ns against 9-22 ns), and their share in the game is unknown. **Other systems:** - The gates sit in two lifted functions for everyone, whatever the switch says. The farm's `=0` against main is inside its noise. - At this head, BeanBot's acquire load in `nx2/rtgen.py` A64_GPU_PAGE changes every lifted store's mark and `rt/a64.h`, so every player's whole lift recompiles, with the switch on or off. It is unmeasured and outside this PR's topic. - `nx2/build.py` now gives every host file `-DA64_MARK_BITMAP`. No host file uses `A64_MARK` itself today (only comments), so behaviour is unchanged, but every host object's cache key changes. - The build-time rows at 5f0c55d say +16 % whole build (+75 s compile), which nothing in the diff explains. Players compile locally, so this needs measuring again. - On Windows, the grid's snapshot of an unused record array at a null pointer is not a crash: `commit_on_touch` in `host/mkw_glue.c` commits the page and it reads zero. AMD, Intel, the Steam Deck and CPUs without FMA (every grid call declines there) are not run. **Switches:** - `HOS_NATIVE_VISIBILITY` (kill, **on** by default, `=0` lifted, `check`): the on default is not backed by a measurement of this PR. The one isolated run shows more CPU with it on. It should not ship on until an isolated, repeated, interleaved measurement shows a gain past the noise on Linux and Windows. If that measurement shows no gain, the change doesn't earn its code. - `HOS_NATIVE_VISIBILITY_LOG` (diag, off; always on in check mode): the right default. **Reviewed:** 41c96c3a8 against 7486869e6 (its merge base), all 12 commits. - Read in full: `host/oot_visibility.c`, `host/visibility_kernels.c`/`.h`, `host/visibility_occlusion.c`/`.h`, the changed parts of `nx2/oot_patches.py`, `nx2/build.py`, `nx2/rtgen.py` and `gpu/vk/switches.def`, and `tests/test_mark_bitmap_host.py`. - Skimmed: `tests/test_native_visibility.py`, for what it runs (it compiles and runs fixtures in a temp dir and touches nothing else). - I did not check bit-exactness against the game's routine. Nothing was run (a code review). **Fixed in this review:** nothing landed on the head. A rebase onto main b62932a96 was clean commit by commit, with no conflicts. A one-line commit putting `A64_GPU_PAGE` back to main's plain load was made, but the push to the head branch was refused. Both are left as suggestions below. **Checked and fine:** - The gates are only in the supported build, and they are declared at file scope for split pieces. - `vis_start` publishes the mode with release/acquire. A worker that loses the race spins until the mode is published. - The SSE box kernel only reads planes inside the 0xe8 bytes `vis_range` checked, and lanes past the count are masked out of both `gts` and `bads`. Its x8/x9/x10 match the scalar loop's. - The grid's `written()` regions cover every store on the success path: O+0x288..0x4ea, both strip headers, 0x690..0x6a9, 0x6da, the used plane records, the grid, the tiles by page and the bounds. - A decline restores every writable region, and check mode declines when an allocation fails. - Both switches have rows, and nothing touches `third_party/`. **Blocking** (to change before merge) 1. Licensing: the host code is a transcription of the game's code, not code written from its behaviour. - `host/visibility_occlusion.c:3` says "The game's operations in the game's order: every Arm FMADD / FMSUB / FNMADD / FNMSUB is one fused multiply-add here". - `planes_80823e00` (`host/visibility_occlusion.c:226-300`, `:331-420`) is one statement per instruction, with the temporaries named after the routine's registers: `float s17 = f_sub(bad, p58, p4c);` (:233), `float s22 = -f_mul(bad, s20, s29);` (:244), `s31b, s30b, s28b` (:265). Comments like `/* (s8*s8 + s31*s31 + s9*s9) */` record the register order. - `low_corners` / `tall_corners` (:483-560) and the grid loops follow the same pattern. - The box kernel's comments are its disassembly: `host/visibility_kernels.c:57` "LDR q0, a; LDR q1, b; FCMGT v0.4s; ADDV s0; FMOV w8, s0", `:121` "EXT #8 then FADD .2s", `:122` "FADDP", `:134` "LDR s3". - The hooks write back dead register values (x8, x9, x10, v2, v3 and NZCV at `host/oot_visibility.c:10`; v8..v15's upper halves from "its LDP d8..d15 restore" at `:246`). Those can only come from the routine's instruction listing. - CONTRIBUTING.md allows host code written from facts about behaviour (what a routine computes, its inputs and outputs) in your own words. It does not allow the game's instructions rendered as C. - `game_logf` follows the public optimized-routines logf and reads the game's table from guest memory at run time, which is fine on its own. The rest needs rewriting from a description of what the routines compute, or dropping. 2. `gpu/vk/switches.def:108`: on by default with no isolated measurement, while the only isolated run shows +7.5 % / +8.7 % CPU with it on (above). Measure this branch alone, against its base and against `=0` in the same build, with several interleaved runs a side on Linux and Windows: capped CPU, uncapped frame times and 1 % lows, and the share of early-exit box calls. Choose the default from those runs. 3. `nx2/rtgen.py:69`: the `__ATOMIC_ACQUIRE` load (BeanBot's e1e00892) changes every lifted store in the game, recompiles the whole lift and was never measured, and it has nothing to do with visibility. Put main's line back: `#define A64_GPU_PAGE(a) ((a64_gpu_pages[A64_DIRTY_AT(a) >> 3] >> (A64_DIRTY_AT(a) & 7)) & 1)`. If the bitmap's race should be fixed, that belongs in its own measured PR, and a relaxed load would be enough there. 4. Build time: the farm measured +16 % whole build (+19 % compile) at 5f0c55d. Measure it again at the new head and explain it, or remove the cause, since players build locally. **Suggestions** (the author may take or leave) - Give `-DA64_MARK_BITMAP` only to `oot_visibility.c`, the one host file that tests `A64_GPU_PAGE`, rather than to every host compile job, so the other host objects keep their cache. - `host/oot_visibility.c:136`: box check mode still doesn't compare NZCV, and grid check mode compares only the regions the host reports, not registers. Either compare them or narrow the "register for register" wording in the description. - Rebase onto current main: it is clean commit by commit, with no conflicts. **Testing:** - The only in-game run of this code was at 38f0ad29, in a merged test tree on Windows/NVIDIA, where check mode found no difference. - This head is not built or played on any OS. Linux has only the farm's noisy run at 5f0c55d, which failed (CPU and screenshot limits; the screenshot differences also appear with `=0`, so they look like scene noise). - CI is not looked at here. All of this holds the merge. **Overlaps and merge order:** - The siblings https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1898, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1899, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1900, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1903, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1904, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1905, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1909, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1910, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1911, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/2001 and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/2036 all add rows next to these in `gpu/vk/switches.def` and hooks in `nx2/oot_patches.py`. Expect textual conflicts in whatever merges second. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1904 (libm, logf): this PR carries its own `game_logf`. If 1904 lands a host logf, use one copy. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1911 has the same licensing blocker. The Windows measurement is of all the rewrite branches together, so none of them can claim it alone.
@ -105,6 +105,8 @@ SW(HOS_MOVIE_PACE, kill, bool, "on: a 60 FPS cap is raised for a movie's picture
SW(HOS_MUTEX_TRACE, diag, int, "0: off", "Logs mutex waits longer than <ms>, with the thread that held the mutex.")
SW(HOS_MUTEX_WATCH, diag, string, "off", "Logs every operation on the mutex at that guest address (hex) and its condition variables' signals.")
SW(HOS_MW_WAKE, kill, enum, "only changes to objects linked into a multi-wait wake WaitAny", "=all: every event, semaphore or queue change wakes every WaitAny, as before.")
SW(HOS_NATIVE_VISIBILITY, kill, enum, "on: the visibility routines on host code where it gives the game's bits exactly, the rest lifted", "=0 (or false, off, no): the visibility routines run as lifted, as on main; 1 (or unset) runs the render calc's visibility routines (the box-against-view test f_80029240, the occlusion grid's build f_808226e0) on host code where it gives the game's bits exactly, the rest lifted; check runs both on every call and logs any difference (slow; host/oot_visibility.c).")
Owner

On by default, but the only isolated measurement (farm, 5f0c55d, Linux) shows +7.5 % Market / +8.7 % Field CPU with it on against =0, from single runs. Choose the default from repeated, interleaved runs of this branch alone.

On by default, but the only isolated measurement (farm, 5f0c55d, Linux) shows +7.5 % Market / +8.7 % Field CPU with it on against =0, from single runs. Choose the default from repeated, interleaved runs of this branch alone.
@ -0,0 +133,4 @@
/* (what the lifted code writes: x0, x8, x9, x10, v0..v3; it keeps only the NZCV bits it needs) */
VisBoxRegs g;
regs_from(c, &g);
if (g.x0 != r.x0 || g.x8 != r.x8 || g.x9 != r.x9 || g.x10 != r.x10 || memcmp(g.q, r.q, sizeof g.q)) {
Owner

Nit: check mode doesn't compare NZCV here, and the grid check compares only the regions the host reports. Compare them, or narrow the 'register for register' wording.

Nit: check mode doesn't compare NZCV here, and the grid check compares only the regions the host reports. Compare them, or narrow the 'register for register' wording.
@ -0,0 +54,4 @@
}
static inline uint32_t nzcv_fcmp0(float s) { return s < 0.0f ? 8u : s == 0.0f ? 6u : 2u; }
/* one of the two box steps: LDR q0, a; LDR q1, b; FCMGT v0.4s; ADDV s0; FMOV w8, s0. 0 when a lane is out of class */
Owner

Licensing: the comments are the routine's disassembly (here, and at :121, :122, :134), and the hook writes back its dead registers. Write it from the routine's behaviour, not its instruction listing.

Licensing: the comments are the routine's disassembly (here, and at :121, :122, :134), and the hook writes back its dead registers. Write it from the routine's behaviour, not its instruction listing.
@ -0,0 +230,4 @@
const float t = f_add(bad, off, s7);
const float p4c = ldf(P + 0x4c), p54 = ldf(P + 0x54), p58 = ldf(P + 0x58), p7c = ldf(P + 0x7c);
const float p84 = ldf(P + 0x84), p88 = ldf(P + 0x88), p8c = ldf(P + 0x8c), p90 = ldf(P + 0x90);
float s17 = f_sub(bad, p58, p4c);
Owner

Licensing: one statement per Arm instruction, with temporaries named after the routine's registers (s17, s28, s30, s7b...). This is the game's code as C, which CONTRIBUTING.md's Licensing section rules out. Rewrite it from a description of what the routine computes, or drop it.

Licensing: one statement per Arm instruction, with temporaries named after the routine's registers (s17, s28, s30, s7b...). This is the game's code as C, which CONTRIBUTING.md's Licensing section rules out. Rewrite it from a description of what the routine computes, or drop it.
@ -67,3 +67,3 @@
before is caught by its first byte; read-only on the store path */
extern uint8_t a64_gpu_pages[A64_DIRTY_PAGES / 8];
#define A64_GPU_PAGE(a) ((a64_gpu_pages[A64_DIRTY_AT(a) >> 3] >> (A64_DIRTY_AT(a) & 7)) & 1)
#define A64_GPU_PAGE(a) ((__atomic_load_n(&a64_gpu_pages[A64_DIRTY_AT(a) >> 3], __ATOMIC_ACQUIRE) >> (A64_DIRTY_AT(a) & 7)) & 1)
Owner

Outside this PR's topic: an acquire load in every lifted store's mark changes rt/a64.h, so the whole lift recompiles for everyone, switch on or off, and it was never measured. Put main's plain load back: #define A64_GPU_PAGE(a) ((a64_gpu_pages[A64_DIRTY_AT(a) >> 3] >> (A64_DIRTY_AT(a) & 7)) & 1). A fix for the race goes in its own measured PR (relaxed would be enough).

Outside this PR's topic: an acquire load in every lifted store's mark changes rt/a64.h, so the whole lift recompiles for everyone, switch on or off, and it was never measured. Put main's plain load back: `#define A64_GPU_PAGE(a) ((a64_gpu_pages[A64_DIRTY_AT(a) >> 3] >> (A64_DIRTY_AT(a) & 7)) & 1)`. A fix for the race goes in its own measured PR (relaxed would be enough).
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
CI / Website build and checks (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
beanstalk/approval not approved yet
PR description / Description follows the template (pull_request_target) Successful in 17s
This pull request is marked as a work in progress.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'perf/native-visibility:snaildev99-perf/native-visibility'
git switch 'snaildev99-perf/native-visibility'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!1902
No description provided.