WIP: perf(lift): RX_LOOP_SINK=1, a loop's guest-register stores sunk to its exits (compile only, off) #1978
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1978
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/lift-loop-sink"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
LLVM keeps a lifted loop's stores of its guest registers inside the loop while the loop holds a call it cannot prove
returns: the interrupt poll's
a64_poll_cur. WithRX_LOOP_SINK=1the native lifted code and the shared A64 helpersare built with
-mllvm -force-attribute=a64_poll_cur:willreturn(true: the poll returns when the thread runs again),so the stores move to the loop's exits. The page marks stay main's release stores, each after the guest store it
marks; the context is the thread's own (
restrict,noescape), never read by the collector or the checkpoint. Theemulator-hosted DLL keeps main's flags. Off by default: main's flags, and
a64.his main's in either mode. Compileonly; the build's log names the mode, and
dev.py packbuilds the game when it is on. From 7d85292a on a localperformance branch, reworked in review: the switch first also made the marks plain byte stores on x86-64; the reviewers
asked to keep release marks (a plain mark racing with the collector's acquire exchange is a data race the compiler may
exploit), so that half is dropped.
Read first:
loop_sink_flagsandnative_flagsinnx2/build.py,docs/dev/reference/loop-stores.md,tests/test_loop_sink.py.Testing
Tested: Windows 11, i7-9700, RTX 4060, llvm-mingw clang 23.1.5, Python 3.14.7 (the Builder's); game-free checks
only. This change has not run in the game on this PC yet, with the switch on or off (Still to test). Each check ran in
a hidden console of its own, through the PC's job queue (job fb2-1978-probe3), on this change's tree before its
rebase onto
main@f1f2dc5b(whose one commit touches none of its files) and before comment and doc edits (head5114aeaeafter them), the test run with no arguments (its compiler found throughLONGSHOT_MINGW, astests/run_all.pygives it):python -I -B tests/test_loop_sink.py: exit 0, "all passed": the flags for every off and on value, read at eachbuild, the DLL's unchanged, the header the same either way, the pack routes; in the IR at O1 and O2, bitmap off and
on, "copy CCGMMCC, load-only CCC" with the switch off and "copy GMM, load-only" (no context store) with it on (G a
guest store, M a release mark, C a context store); seeds 1 to 3, O1 and O2, bitmap off and on, the same digest,
polls and marks either way (e.g. "seed 3, O2, bitmap on: off 3b699ce7951861dc 7657 3701, on 3b699ce7951861dc 7657
3701"), the exact marks and polls checked in every round.
The same test with
tests/test_vk_pgo.pyandtests/test_light_passes.pyis queued on5114aeaeitself (jobfb2-1978-final); its result goes in a comment.
git diff --quiet origin/main HEAD -- nx2/rtgen.pyholds: the generated header is main's.Earlier, at 89531648 (the first form, plain marks),
python -I -B tests/run_all.py --strict --changed origin/mainonmain@2e1f6c32: 98 passed, 17 failed (named in that run's comment as failing onmaintoo; two of them,test_pipeline_states.pyandtest_recorder_sync.py, had found no Vulkan headers in that run (RX_CPMunset) and compiled nothing on either side), 7 skipped. That run wasnot repeated on this head. Those two, run again with the headers at
5114aeaeand onmain@f1f2dc5b:test_pipeline_states.pypasses on both;test_recorder_sync.pyfails onmaintoo with the same lines.Round 2 (
dev.py baseand the shared base): the base is keyed by its commit alone, anddev.py basepassed thecaller's environment to its build, so a base first built with
RX_LOOP_SINK=1became the off side of every later A/B. Nowthe shared base's build leaves the switch out (
scripts/devtools/build.pycmd_base,packs.GAME_COMPILE_SWITCHES). The IR order check now also covers the bitmap-on loop: each mark block runs after the guest store (the store's block dominates it).The branch was rebased unchanged onto
main@9dd41326(git range-diff: "=" for the PR's commit). Through the PC's jobqueue (job fb2-lift-r2,
RX_CPMset, each check in a hidden console of its own, 20 minutes at most):python -I -B tests/test_loop_sink.pyat the test commitdb1993b4: exit 1, "1 failed": "dev.py base, RX_LOOP_SINK=1 then unset: itsbuild leaves RX_LOOP_SINK out, as main builds it"; at the fix, head
57ae9f39: exit 0, "all passed" (both orders, one base build,without the switch).
tests/test_vk_pgo.pyandtests/test_light_passes.pyat57ae9f39: exit 0 each.Round 3 (
dev.py packs baseand a base you name): round 2 left the switch out only indev.py base. Thedocumented
dev.py packs baserunspacks.cmd_basedirectly, whose build still tookRX_LOOP_SINKfrom the caller'senvironment, so a named base built with it on was what a later
dev.py packs rtwith it unset copied. Nowpacks.cmd_base, where both commands converge, builds with the game's compile-only experiments(
GAME_COMPILE_SWITCHES) taken out of the environment afterpacks.runmerges the caller's; the wrapper's own strip isgone. A pack with the switch on still builds its own game in that mode. On
main@9dd41326(main has not moved; theearlier commits are unchanged), through the PC's job queue (job fb2-lift-r3b, light, two checks at once at most,
RX_CPMset, each check in a hidden console of its own, 20 minutes at most):python -I -B tests/test_loop_sink.pyat the test commit0100d3c5: exit 1, "2 failed": "dev.py packs base, RX_LOOP_SINK=1 first,unset second: the base's build (make_game.py's environment) has no RX_LOOP_SINK, as main builds it" and "...: dev.py packs
rt with the switch unset and no pack runs a game built off (got on)"; at the fix, head
947af7d2: exit 0, "all passed"(
dev.py packs baseanddev.py base, each with the switch on first and unset second and the other way round, thendev.py packs rtwith no pack, anddev.py packs packandrtwith the switch on).tests/test_dev_packs.pyandtests/test_dev_build.pyat947af7d2: one failure each, the same onmain@9dd41326in the same job ("every file the renderer keeps beside the game is classified (unknown:
['longshot_fsr4_pipelines.bin'])";
Linux.test_folders:C:\Users\User\GamesagainstC:\Users\User/Games).In the game, on this PC (Windows 11, i7-9700, RTX 4060), 2026-11-22: Kokiri Forest 800x600,
RX_LOOP_SINKunset againstset at build, both at
947af7d2, a warm-up a side then 8 interleaved runs a side, every run in the same scene state: mediancalm game time a frame 10.26 against 10.10 ms with it on, 1.6 % lower, no gain shown (Measurements).
Still to test:
RX_LOOP_SINKunset and with it set, its build time and theexecutable's size, expecting figures for Measurements (by hand)
python3 -I -B tests/test_loop_sink.py, expecting it to pass; and the game built with the switch unset and set, played inKokiri Forest, expecting no crash and frame times alike (by hand)
RX_LOOP_SINKunset and set, Kokiri Forest or the Market, a few interleaved runs a side, expectingno crash and frame times alike (by hand)
main@9dd41326, expecting the same lifted code; held until the farm pairs this head with9dd41326(the Test Farm request already on this PR)Measurements
Verdict:
RX_LOOP_SINKships off (the maintainer's request, while the enabled and disabled results are gathered). In play,no loss, and no gain shown: the median calm game time a frame is 1.6 % lower with the switch on (10.10 against 10.26 ms), inside the spread. Eight runs a side resolve roughly 1.5 to 2 % on the median (an estimate: identical runs of one build spread
about 5 % end to end in calm game time). The switch stays off.
In play: Kokiri Forest, off against on at
947af7d2, 8 runs a side (the current reading)Setup: Windows 11, i7-9700, RTX 4060, run alone on the PC on 2026-11-22, 06:05 to 06:48 UTC; Kokiri Forest (child), 800x600
(CPU-paced), frame cap off, 60 s a run; a warm-up a side, then 8 interleaved runs a side (off on on off, four times). The
teleport into the scene checked as taken in every run, so every run is in the same (light) scene state; the GPU cooled
to 70 °C before each run, its median clock over each measured stretch its full 2,745 to 2,760 MHz (the few dips are in
the clk_min column). Calm game time: the measured 60 s in six
10 s windows, each window whose median game time is more than 20 % above the run's lowest dropped (a burst), then the
median game time of the frames left; blocked: the game thread's wait over the same frames.
Baseline: this PR with
RX_LOOP_SINKunset (off), which builds withmain's flags and header, so it stands formain@9dd41326(the merge base); on = set at build.
947af7d2947af7d2Each run (the run's table; game and calm in ms a frame; clock in MHz over the measured stretch):
Clean build time and size, Linux and the Steam Deck are not measured.
Earlier: 4 runs a side, before the wrapper checked the teleport (2026-11-22, 00:15 to 01:03 UTC)
Reading then: not decided, no loss seen. Every run sits in one of two scene states, and both sides land in both.
Setup: Windows 11, i7-9700, RTX 4060, run alone on the PC on 2026-11-22; Kokiri Forest (child), 800x600 (CPU-paced),
frame cap off, 60 s a run; one warm-up a side, then 4 interleaved runs a side (off on on off off on on off). off =
RX_LOOP_SINKunset, whose lifted code, object keys and executable are those of
main@9dd41326(the merge base); on = set at build.Range: off 67.6 to 76.4 fps, on 70.3 to 82.3. The two states: light about 1,100 textures a frame and 76 to 83 fps, heavy
about 1,380 and 67 to 73. The run wrapper asks the Debug teleport into Kokiri Forest, where the save already puts Link; asked
within about 22 frames of entering play, the game refuses it and nothing reloads (heavy), asked later it is taken (light).
Within one state the on side leads in each pairing (heavy: off 67.6 and 69.4 fps, on 70.3 and 72.6; light: off 75.9 and 76.4, on 81.5 and 82.3), but that is two runs a cell, too few to call a gain.
Game-free checks
Setup: Windows 11, i7-9700, llvm-mingw clang 23.1.5;
tests/test_loop_sink.pythrough the PC's job queueBaseline: this head with the switch unset, which builds the lifted code with main's flags and main's
a64.h, so itstands for main in the fixture's IR.
5114aeae, switch off (as main)5114aeae, RX_LOOP_SINK=1Risks
The marks keep main's release order; only the poll's call is declared to return. With the switch on, a loop's context stores move to its exits, also in loops that
contain marked guest stores, so during such a loop the context in memory lags the loop's registers. Not verified: what
a fault, an interrupt or a sampler reading the context in the middle of such a loop sees. Off by default: unset, the build is
main's. Turning it off is unsetting it and building again: an executable keeps what it was built with. Changing it
compiles the lifted code again, and translates the game again where the translated C is gone (a player's build folder;
a development checkout and
LONGSHOT_KEEP_BUILD_FILES=1keep it). Nothing a player keeps (saves,longshot_options.ini) changes.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Recommendation: Request changes. The opt-in build removes the atomic publication the renderer relies on, and the default build invalidates every cached lifted object on update. The reported Windows fixture shows loop-store sinking, but the performance gain in play is unproven.
The build experiment can keep guest registers in host registers until eligible loops exit, reducing stores during play. It changes the write-watch protocol across lifted and host code and adds a full lifted-code rebuild on the first update even for players who leave it off.
@fixwill do (12)nx2/rtgen.py:71: A plain dirty-byte store can race with another writer or the atomic acquire exchange in host/rx_backend.c. (more on the line)nx2/rtgen.py:64: The generated a64.h changes with RX_LOOP_SINK unset. (more on the line)docs/dev/reference/loop-stores.md:83: The table reports outputs counted from IR and assembly lifted from the game's hottest functions. (more on the line)docs/dev/reference/loop-stores.md:61: Changing the switch keeps the lift key, but a normal player build may already have discarded code_*.c. (more on the line)nx2/build.py:1757: The emulator-hosted DLL gets the experimental mark flag, but A64_POLL_CTX routes its polls through a64_poll(c), so the forced attribute on a64_poll_cur cannot help that path. Keep the DLL on its existing mark and poll flags unless that mode has its own supported optimization and coverage.tests/test_loop_sink.py:176: The stand-in writes c.poll through cur_ while the lifted function accesses the same field through its restrict-qualified argument. (more on the line)tests/test_loop_sink.py:155: The IR check only compares the first guest store with the first mark in textual block order. (more on the line)tests/test_loop_sink.py:37: The suite hardcodes --cc clang, which bypasses the fixture's CC and llvm-mingw fallback. (more on the line)tests/test_loop_sink.py:34: Importing build before setting up the fixture captures the ambient RX_MARK_BITMAP, RX_MARCH, RX_TARGET, and LONGSHOT_SYSROOT. (more on the line)tests/test_loop_sink.py:115: The IR parser requires the exact prefix define dso_local void. (more on the line)For the author (1) (a run, a measurement or a decision)
Checks: Still running:
PR description / Description follows the template (pull_request_target),CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven. The reported Windows run at 89531648 matched digests and poll counts for six synthetic comparisons, covering three seeds at each of O1 and O2. The base fixture stopped at a missing Python helper. The historical experimental-branch figures use a different base and a scratch workload. There are no interleaved gameplay results against
6e1521a70aor clean build time and size results.Other systems: The dirty-byte changes reach Vulkan replay and submit-time capture on every GPU vendor, including Steam Deck. Native polls reach the virtual-core scheduler on Windows and Linux. The emulator-hosted DLL still uses its context-taking callback. Both native builders hash the changed header in their object keys. The diff leaves the renderer API, shaders, save and options formats, audio, and input code untouched. Gameplay coverage for the affected systems is pending.
Switches: RX_LOOP_SINK is a build-only developer switch. Unset, empty, 0, false, off, and no disable it. Enabling it adds A64_MARK_PLAIN, effective on x86-64, and asks LLVM to mark a64_poll_cur willreturn for polling builds. Off is the safe runtime default for this patch, and the default build must also preserve cached objects. The build-switch reference has a row. The runtime switches.def table does not cover build-only variables.
Reviewed:
8ffcba44d1against6e1521a70a: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, the standards, the commit and complete diff, both changed documentation files, nx2/rtgen.py, and tests/test_loop_sink.py in full. Read the changed sections and relevant callers in nx2/build.py, nx2/make_game.py, and scripts/devtools/build.py. Traced dirty marks, capture and replay, the poll and scheduler, the DLL shim, toolchain selection, object and lift caching, and test discovery. Skimmed recorded results and nearby fixtures. No local review checklist or sibling Tooling checkout was available. There were no changed translations or checked-in generated files.; nothing run (a code review).Checked and fine:
Testing: Read-only review. The author reported a passing Windows fixture on 89531648 and 17 changed-suite failures also seen on the base. The supplied CI checks are pending. Linux fixture results, gameplay, and build measurements are outstanding.
Overlaps and merge order: None found in local history. Forgejo was unavailable for overlap checks.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -0,0 +58,4 @@- At a poll, the context in memory may now be older than the loop's registers. `hos_vcore_checkpoint` is alreadyrequired never to read it (`host/hos_thread.c`); loop loads were already kept in host registers across polls.Unset, `0` or `off`: release marks and the plain poll, the flags main has. The switch is compile-only: a build thatL4 (minor): Changing the switch keeps the lift key, but a normal player build may already have discarded code_*.c. Its retained objects cannot be reused after a compiler flag change, so the builder can translate again. Qualify the promise here, in the build reference, and in the PR summary. For an A/B, explain how to keep the translated C and build both variants before launching them.
@ -0,0 +80,4 @@Each half alone, at `-O2`, in dynamic instructions: plain marks alone -0.3%, the poll's `willreturn` alone 0.0%,both -5.3%.Static counts for the game's 46 hottest functions under 150 KB of C (context loads and stores counted in the IR,L3: The table reports outputs counted from IR and assembly lifted from the game's hottest functions. The hot-function trial below also quotes derived outputs. Remove the game-derived figures and use a fixture authored independently of the dump. Correct the PR's no-game-output checklist entry accordingly.
@ -61,18 +61,27 @@ MARK_NATIVE_64 = r"""extern uint8_t a64_dirty[A64_DIRTY_PAGES];/* (Tried 2026-10-21: reading a mark first and writing it only when clear, against cache lines moving between cores:game threads +1.1 ms a frame, the code 18 MB bigger. Not kept.) */#define A64_DIRTY_AT(a) (((uint64_t)(a) >> 12) & (A64_DIRTY_PAGES - 1))/* (nx2/rtgen.py) A64_MARK_PLAIN (nx2/build.py loop_sink_flags) on x86-64: a mark is a plain byte store. LLVM's LICM takes anL2: The generated a64.h changes with RX_LOOP_SINK unset. build.py includes its bytes in every lifted object's dependencies. An ordinary player update with only retained objects returns KEPT_STALE, then make_game.py translates the game again and recompiles every lifted chunk. Generate the base header byte for byte on the off path and select the experimental header only for the enabled build.
@ -64,0 +68,4 @@stores, because the data's address is a guest integer made into a pointer, which it cannot prove does not point intoa64_dirty. Without the switch, or on other CPUs, the release store as before. */#if defined(A64_MARK_PLAIN) && (defined(__x86_64__) || defined(_M_X64))#define A64_MARK_ONE(p) (*(p) = 1)L1: A plain dirty-byte store can race with another writer or the atomic acquire exchange in host/rx_backend.c. x86 store ordering applies to emitted instructions, and possible aliasing does not prevent the compiler from eliminating a repeated plain mark to the same page. If the renderer clears the byte between two guest stores, an eliminated publication can leave a capture using stale data. Keep release-atomic marks on shared pages and restrict register promotion to paths where it can preserve that publication.
@ -0,0 +31,4 @@sys.path.insert(0, os.path.join(ROOT, "nx2"))sys.path.insert(0, HERE)import kit # noqa: E402,F401 (a64-kit on the path)import build # noqa: E402L9: Importing build before setting up the fixture captures the ambient RX_MARK_BITMAP, RX_MARCH, RX_TARGET, and LONGSHOT_SYSROOT. For example, RX_MARK_BITMAP=0 removes all coverage of the bitmap branch and can still pass, while RX_TARGET=windows with llvm-mingw on Linux can produce a PE file that the fixture tries to execute. Use a controlled native target and baseline CPU, and run both bitmap variants explicitly.
@ -0,0 +34,4 @@import build # noqa: E402import rtgen # noqa: E402RUN_ALL_ARGS = ["--cc", "clang"]L8: The suite hardcodes --cc clang, which bypasses the fixture's CC and llvm-mingw fallback. With LLVM-MinGW installed through LONGSHOT_MINGW and no clang on PATH, the strict suite fails before the fixture can run. Let the fixture resolve an explicit compiler, a compatible CC, or the configured toolchain, and derive its target from the chosen compiler so the generated program runs on the test host.
@ -0,0 +112,4 @@def loop_blocks(ir):"""the IR of f_80000000's loop: its blocks from the first one a branch goes back to, to the last such branch"""body = ir[ir.index("define dso_local void @f_80000000"):]L10 (minor): The IR parser requires the exact prefix define dso_local void. Clang on Linux can omit dso_local for a preemptible definition, for example with the fixture's --cflags set to -fPIC. Locate the definition by symbol, tolerate optional linkage and visibility attributes, and report a failed assertion if the function or loop is missing.
@ -0,0 +152,4 @@check("willreturn" not in attrs, "%s: a64_poll_cur as declared" % what)check(marks and all(o == "release" for _, o in marks), "%s: release marks: %s" % (what, marks))check(ctx_st >= 2, "%s: the loop stores its registers (%d)" % (what, ctx_st))guest = re.search(r"store i64 [^\n]*, align 1\n", loop)L7: The IR check only compares the first guest store with the first mark in textual block order. The executable checks final state after a single writer finishes, and every eight-byte destination is aligned, so no write crosses a page and no collector clears a mark during the loop. Add a cross-page case with expected marks and exercise a clear between writes to the same page. Check that each publication follows the data it covers.
@ -0,0 +173,4 @@void a64_poll_cur(void) {polls_++;(void)rand(); /* (a C library call) */if (++npoll_ %% 3) __atomic_store_n(&cur_->poll, 1, __ATOMIC_RELAXED);L6: The stand-in writes c.poll through cur_ while the lifted function accesses the same field through its restrict-qualified argument. That violates the restrict contract during the call and makes the poll-count comparison unreliable under optimization. Have the stand-in record the call without accessing Ctx. Test repeated requests through the scheduler's atomic request path.
Recommendation: Request changes. Preserve atomic dirty-page publication in the enabled build and preserve retained-object reuse in the default build.
nx2/rtgen.py:71races with the collector’s acquire exchange. The compiler can remove a repeated plain mark that a concurrent collector would have cleared between guest writes. Keep release publications for pages the collector can clear and test that interleaving. Existing inline findingnx2/build.pystill hashes the changed generated header into every lifted object. A player update that retained only objects falls back to translating and compiling again. Preserve the off-mode header bytes and test retained-object reuse. Existing inline findingDoes it help? Unproven. The author reported six matching Windows synthetic comparisons before rebasing. The same-head CI snapshot showed the description check passing and source checks pending. Gameplay, Linux, and clean build time and size are outstanding. Compare the merge base, PR default, and PR enabled builds on Windows and Linux, including Steam Deck, with interleaved gameplay that exercises Vulkan capture and virtual-core polling.
Switch and farm.
RX_LOOP_SINKis build-only and defaults to off. The stock farm plan cannot inject it at compile time, so enabling it takes a separately built executable. The current head has no farm approval or plan. After the code blockers are resolved, any farm run needs two distinct BigBeans approvals for the exact head because the diff touchesnx2/. Farm stateOverlaps. The current patches conflict with the dirty-page mask PR and the poll calling-convention PR. Recheck the shared generator and compile flags if either lands first.
Reviewed
8ffcba44d195against6e1521a70a32. Read all seven changed files and verified the write-watch, scheduler, DLL callback, and build-cache paths using revision-matched local Graphify maps and source. No PR code ran. Existing full review8ffcba44d15114aeaed4Pushed
5114aeae(one commit, onmain@f1f2dc5b; it replaces8ffcba44). The maintainer's review decided the scope: the plain marks are dropped, every mark stays main's release store, andRX_LOOP_SINK=1only declares the poll's callwillreturn, for the native build (the DLL keeps main's flags).nx2/rtgen.pyis main's again, soa64.his the same in either mode and retained objects keep their keys. In the fixture (clang 23.1.5, O1 and O2, bitmap off and on) that alone is enough: a copy loop with release marks and a load-only loop keep no context store with the switch on.dev.py packbuilds the game when it is on; the build's log names the mode; the counts over the game's lifted functions are removed (squashed so that no pushed doc keeps them). Tested:tests/test_loop_sink.pyall passed on this code before the rebase and comment edits; the run on5114aeaeitself is queued and its result follows here. Still to test: the in-game A/B and Linux.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62742: addressed in
5114aeae(one commit, rebased onmain@f1f2dc5b), and your first point decided the scope: the plain marks are dropped, every mark stays main's release store, andRX_LOOP_SINK=1now only marks the poll's callwillreturn(native build); in the fixture (clang 23.1.5) a copy loop with release marks and a load-only loop still keep no context store with it on.a64.his main's byte for byte again (git diff --quiet origin/main HEAD -- nx2/rtgen.pyholds), so retained objects keep their keys in either mode. The game-derived counts are removed; the fixture now checks in the IR that each guest store is followed by its two release marks, and at run time stores across pages and has the poll take every mark between writes with an acquire exchange, checking the exact marks left after the last one. The A/B in play (merge base, off, on) is pending with the opener.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62550 (L1): fixed in
5114aeaeby dropping the plain marks, as the maintainer's review also asked: every mark is main's release store with the switch either way. The test checks it in the IR (each guest store followed by twostore atomic ... releasemarks, bitmap off and on) and at run time with a poll that takes the marks between writes to the same page, checking the marks left exactly.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62551 (L2): fixed in
5114aeae. With the plain marks gone,nx2/rtgen.pyis main's again anda64.his the same with the switch unset and set (the test compares the two headers); the switch is only an LLVM option on the lifted chunks' command line, so the default build's object keys are main's.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62552 (L3): removed in
5114aeaeas asked. The static counts over the game's hottest lifted functions (and the derived figures of the tried variants) are gone from the doc, the commit message and the description; the page says the per-function counts were taken locally and are not published, and the claims it keeps rest on the authored loops intests/test_loop_sink.py. The description's no-game-output entry stands for this head.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62553 (L4): fixed in
5114aeae. The doc, the build reference row and the description now say a change of the switch translates the game again where the translated C is gone (a player's build folder), and that an A/B builds withLONGSHOT_KEEP_BUILD_FILES=1or in a development checkout, the game folder copied between the two builds.(L5 in https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62559): fixed in
5114aeae. The attribute is added bynative_flags(gen, rt64, dll)for the native build only; the emulator-hosted DLL keeps-DA64_POLL_CTXand main's flags (the test checks it), and the mark flag no longer exists.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62554 (L6): fixed in
5114aeae. The poll stand-in no longer touches the context; the copy loop raises the request itself throughcevery eighth iteration, so repeated requests go through the lifted code's own path, and the driver checks the exact polls taken per call.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62555 (L7): fixed in
5114aeae. The IR check now takes every store of the loop in order and requires each guest store to be followed by its two release marks before the next; at run time destinations are unaligned (stores cross pages) and the poll takes every mark with an acquire exchange between writes, and the marks left must be exactly the pages stored to after the last poll.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62556 (L8): fixed in
5114aeae.RUN_ALL_ARGSis gone and the test resolves the compiler itself:--cc, thenCC, clang on PATH, thenLONGSHOT_MINGW's clang, with that compiler's own target; a--targetin--cflagsruns the IR checks only.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62557 (L9): fixed in
5114aeae.nx2/build.pyis imported withRX_*,LONGSHOT_*andOOT_*cleared,RX_MARCH=x86-64and no process allowed to start; the target comes from the compiler, and IR and programs are built with the bitmap off and on explicitly.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62558 (L10): fixed in
5114aeae. The loop is found by its symbol (^define ...@f_80000000(, any linkage or visibility), and a missing function or loop is a reported failure, not an exception.(L11 in https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62559): the edit part is fixed in
5114aeae: the build's log printsbuild: RX_LOOP_SINK=1: ...when the executable has it, anddev.py packbuilds the game when it is on even with no file changed. The in-game A/B against the merge base (Windows, Linux, Deck, build time and size) is pending with the opener.(L12 in https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62559): in a later round. The earlier
--changedrun is not repeated on this head; its counts and skips will be reconciled from the job's output when the full checks run again.@Beanstalk This adds a compile switch, RX_LOOP_SINK, off by default (nx2/build.py, nx2/rtgen.py). The farm cannot build with it set, so only the default build is to test: with the switch unset, the lift and its C must be main's. Please time the lift against main's and check the lifted C is the same, on a Windows PC and on a Linux PC. No game scenario is needed: unset, the game's code is main's.
@siantm only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was for5114aea, and the PR has changed since (57ae9f3), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-62742, to be exact about
5114aeae: the marks stay main's release stores, in place after the guest store each one marks, and withRX_LOOP_SINK=1the context stores of a loop that contains marked guest stores are sunk to the loop's exits too: intests/test_loop_sink.pythe copy loop's stores go fromCCGMMCCan iteration (C a context store, G the guest store, M a release mark) toGMM, at O1 and O2 with the bitmap off and on. So inside such a loop the context in memory lags the loop's registers while the marks are published.Risks now names what is not verified: with the switch on, a loop's context stores move to its exits also in loops with marked guest stores, and what a fault, an interrupt or a sampler reading the context mid-loop then sees has not been checked. No code change; the head stays
5114aeae.The queue run on
5114aeaeitself (job fb2-1978-final, Windows 11, i7-9700, llvm-mingw clang 23.1.5, each check in a hidden console of its own) ended:python -I -B tests/test_loop_sink.py(no arguments, the compiler throughLONGSHOT_MINGW) "all passed", with the same IR as before the rebase (copy loopCCGMMCCoff,GMMon; load-only loopCCCoff, no store on), andtests/test_vk_pgo.py,tests/test_light_passes.pyexit 0 each.Tested corrected: an earlier run on this PC had no Vulkan headers for the renderer checks (
RX_CPMunset), so the ones it listed as also failing onmainhad compiled nothing on either side. They ran again with the headers at5114aeaeand onmain(queue job fb2-vk-1); the description now gives those results. No check fails at this head alone; nothing in the branch changed.Recommendation: Request changes. The earlier dirty-page and retained-object blockers are fixed. Keep the new compile switch out of the default shared base so an A/B actually compares the two build modes.
dev.py basepasses the caller’s environment to the build and keys the shared base by commit alone. If the first build of a commit containing this switch runs withRX_LOOP_SINK=1, a later run with the switch unset reuses the enabled executable as its baseline. The new pack route also lets an unchanged-tree off pack inherit that executable. Strip this switch from shared default-base builds or include the effective mode in the base identity. Exercise an enabled-first, unset-second sequence in a game-free test.Does it help? Unproven. The author’s exact-head Windows fixture passed and showed guest-register stores moving to loop exits while keeping release marks. The poll-only setting showed no instruction-count gain on the earlier experimental branch and compiler. Compare the actual merge base with this head off and on using verified executables, interleaved gameplay, and clean build time and size. Include Linux and Steam Deck. The crash reporter reads saved guest registers, so check what it reports when a fault occurs inside a loop eligible for promotion. Required source CI is pending.
Test Farm. The requested switch-off lifted C and lift-timing check is supported. The dashboard currently pairs
5114aeaewith6e1521a7while this head’s merge base isf1f2dc5b. Hold the pending request for the source fix and a corrected pair. The enabled compiler mode needs a separate build outside the current farm schema. Author’s request · Farm statusReviewed
5114aeaeagainstf1f2dc5band compared with the previous review head using revision-matched local maps and source. The runtime header generator matches the base, the normal native and DLL flags match with the switch off, and the renderer’s release/acquire protocol is intact. The patches still conflict with the build and pack changes in PR 1979 and PR 1985. Previous review@ -445,0 +448,4 @@the renderer; RX_PGO (the game's hot pieces), LONGSHOT_TRACY (its client is in the game's executable) and aGAME_COMPILE_SWITCHES experiment turned on the game"""if environ.get("RX_PGO") or environ.get("LONGSHOT_TRACY", "0") not in ("", "0") or \any(environ.get(k, "").strip().lower() not in ("", "0", "false", "off", "no") for k in GAME_COMPILE_SWITCHES):The enabled pack case still needs an off baseline. If RX_LOOP_SINK=1 is set on the first dev.py base for a commit that contains this switch, cmd_base forwards it and caches the enabled executable by commit alone. A later unset run treats that executable as the baseline, and an unchanged-tree off pack builds nothing. Keep the switch out of shared default-base builds or key the base by the effective compile mode.
@ -0,0 +222,4 @@scan = stores(loop_blocks(ir, "f_80000100", what))print("%s: the loops' stores in order: copy %s, load-only %s (G guest, M release mark, C context)" % (what, copy, scan))if bitmap: # (a mark only where the bitmap says: in a block of its own)check("G" in copy and "M" in copy and "m" not in copy, "%s: the copy loop's marks are release stores (%s)" % (what, copy))The bitmap-on assertion accepts any ordering of G and M, so it would pass an IR regression that publishes the mark before the guest store. Check that the guest store executes before the conditional mark block, or scope the documented order assertion to the bitmap-off case.
5114aeaed457ae9f39caPushed
57ae9f39(onmain@9dd41326; the PR's commit rebased unchanged): futondestroyer's second look.dev.py basenow builds the commit-keyed shared base withoutRX_LOOP_SINK, so the base is the default executable whichever order the builds come in; the test commitdb1993b4fails on it and the fix57ae9f39passes (job fb2-lift-r2). Testing has the round's results.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-65503: fixed in
57ae9f39(the test first, failing, indb1993b4).dev.py base(cmd_base) now leaves the game's compile-only experiments (packs.GAME_COMPILE_SWITCHES, hereRX_LOOP_SINK) out of the shared base's build and says so, so the commit-keyed base is main's mode whichever order the builds come in; a pack with the switch on still routes to its own game build.tests/test_loop_sink.pyrunscmd_baseat one revision enabled-first then unset, and unset-first then enabled: one build each time, without the switch, and an unchanged-tree pack uses that base unset and builds its own game set. At the test commit the test fails ("dev.py base, RX_LOOP_SINK=1 then unset: its build leaves RX_LOOP_SINK out"); at57ae9f39it passes (job fb2-lift-r2).https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-65504: fixed in
db1993b4. With the bitmap on the IR check now requires the same order as with it off (each guest store, then its two release marks, in block order), and a new check that every block holding a mark runs after the guest store: the store is earlier in the same block, or in a block that dominates the mark's block from the loop's header. A hand-written self-check shows the test passes a conditional mark block after the store and fails one reached before it. It passed at57ae9f39(job fb2-lift-r2).https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-65505: addressed in
db1993b4(test) and57ae9f39(fix), rebased unchanged ontomain@9dd41326(range-diff "=" for the PR's commit). The shared base no longer inherits the experiment (inline 65503), and the bitmap-on IR order is checked by dominance (inline 65504). What the crash reporter shows for a fault inside a promoted loop is not checked yet: Risks names it as not verified, and it goes with the in-game A/B (merge base, off, on), which is pending with the opener; the farm request is for the base pair to be corrected to this head's merge base.@Beanstalk This adds a compile switch, RX_LOOP_SINK, off by default (nx2/build.py, nx2/rtgen.py). The farm cannot build with it set, so only the default build is to test: with the switch unset, the lift and its C must be main's. This head is on main
9dd41326: please compare with main at that merge base. Please time the lift against main's and check the lifted C is the same, on a Windows PC and on a Linux PC. No game scenario is needed: unset, the game's code is main's.@siantm only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was for57ae9f3, and the PR has changed since (947af7d), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.Recommendation: Request changes. The queued
dev.py basenow clearsRX_LOOP_SINK, but the documenteddev.py packs basecommand still inherits it. A named base built with the switch enabled stays enabled whendev.py packs rtlater assembles the baseline with the switch unset. Apply the default-mode rule when the named base launches the compiler and add a regression test through that entry point. The enabled pack should continue to build its own executable. Documented A/B commandsThe queued-base regression test failed at the test-only commit and passed on the current Windows build, along with two related checks, according to the author. The bitmap-on fixture now checks control flow from the guest store to its release marks. Author’s update · Previous review
Does it help? Unproven. The focused fixture confirms the intended compiler change in a synthetic loop. The earlier probe measured no instruction-count gain from the retained poll-only change. Compare
9dd41326with verified switch-off and switch-on builds of this head in warmed, interleaved gameplay. Record frame times, clean build time and size, and what the crash reporter shows for a fault inside a promoted loop. Linux and Steam Deck coverage, the full checks, and required CI are still open.Test Farm. The current request asks for default lifted-C parity and lift timing on Windows and Linux. The dashboard shows head
57ae9f39paired withf1f2dc5b, while the actual merge base and requested baseline is9dd41326. Hold the request until the dashboard shows the correct pair. The enabled compiler mode still needs separately built executables. Author’s request · Farm statusReviewed
57ae9f39against9dd41326and compared the changes since5114aeae. The default flags and generated runtime header still match the base. The atomic page-mark protocol and DLL callback are intact. The PR remains WIP.@ -392,1 +394,4 @@py, env = environment()# (a shared base is keyed by its commit alone, and is the off side of every A/B on it: built as main builds it, without# the game's compile-only experiments; a pack with one on builds its own game, packs.py pack_route)left = [k for k in packs.GAME_COMPILE_SWITCHES if env.pop(k, None) is not None]This clears the switch for the queued wrapper. The documented direct dev.py packs base dispatches to packs.py:600, where the build at line 620 still inherits RX_LOOP_SINK from run(). A named baseline built with RX_LOOP_SINK=1 is then copied by no-pack dev.py packs rt even after the switch is unset. Apply the default-mode rule at the named-base build boundary after the environment merge, and cover enabled-first through that entry point. Keep the pack build's explicit opt-in.
Pushed
947af7d2(onmain@9dd41326; the earlier commits unchanged): futondestroyer's third look.dev.py packs basenow builds a named base withoutRX_LOOP_SINKtoo: the rule sits inpacks.cmd_base, where it anddev.py baseconverge, and the wrapper's own strip is gone. The test commit0100d3c5fails without it and the fix947af7d2passes (job fb2-lift-r3b). Testing has the round's results; Measurements says the switch stays off by default at the maintainer's request.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-67210: fixed in
947af7d2(the test first, failing, in0100d3c5).packs.cmd_base, whichdev.py packs baseruns directly anddev.py basethrough the queue, now builds the base withRX_LOOP_SINKtaken out of the environment afterpacks.runmerges the caller's, and the wrapper's strip inscripts/devtools/build.pyis gone; a pack with the switch on still builds its own game in that mode.tests/test_loop_sink.pynow drives both entry points throughdev.py's command line, enabled first then unset and the reverse, checking the base build's environment and thatdev.py packs rtwith no pack runs an off game while the enabled pack's runtime runs an on one: 2 failures at0100d3c5, all passed at947af7d2(job fb2-lift-r3b).https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1978#issuecomment-67211: addressed in
0100d3c5(test) and947af7d2(fix), onmain@9dd41326with the earlier commits unchanged. Round 2 covered only thedev.py basewrapper; this round covers the directdev.py packs basecommand, with the default applied inpacks.cmd_basewhere both converge (inline 67210). Testing in the description has the round's results (67293); the gameplay comparison and the Test Farm pairing are still open.@Beanstalk This adds a compile switch, RX_LOOP_SINK, off by default (nx2/build.py, nx2/rtgen.py). The farm cannot build with it set, so only the default build is to test: with the switch unset, the lift and its C must be main's. Please time the lift against main's and check the lifted C is the same, on a Windows PC and on a Linux PC. No game scenario is needed: unset, the game's code is main's.
@siantm only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.Re #1978 (comment) ("Does it help?"): the warmed, interleaved off/on comparison in play has run at
947af7d2, and Measurements now has it run by run. off is the switch unset, whose lifted code, object keys and executable are main 9dd41326's; on is set at build. Kokiri Forest, 800x600, 4 runs a side after a warm-up: off 67.6 to 76.4 fps, on 70.3 to 82.3. Reading: not decided, no loss seen. Every run sits in one of two scene states (a refused or a taken teleport at the run's start, about 1,380 against 1,100 textures a frame) and both sides land in both; within a state the on side leads in each pairing, at two runs a cell. The run wrapper is fixed, and a rerun of 8 a side, classed by state, follows. Clean build time and size, the crash reporter in a promoted loop, Linux and Steam Deck are not measured yet. The switch stays off.Re #1978 (comment) ("Does it help?"): the warmed, interleaved in-game comparison has run at
947af7d2, 8 runs a side in Kokiri Forest (800x600, CPU-paced), every run checked in the same scene state, GPU cooled before each, on this PC alone; off is the switch unset (main's flags and header). Calm game time a frame, median (range): off 10.26 ms (9.97 to 10.45), on 10.10 (9.97 to 10.33), 1.6 % lower with the switch on but inside the spread; fps 87.4 against 88.3, and no loss in the tail. Eight a side resolve roughly 1.5 to 2 % on the median (an estimate), so play shows no gain for it. The switch stays off; whether it stays as an off-by-default experiment or is dropped is your call and the author's. Measurements has every run; clean build time and size, Linux and the Steam Deck are not measured.Testing brought in line with Measurements: the Kokiri Forest off/on comparison of 2026-11-22 (8 a side at
947af7d2, no gain shown) is now under Tested; Still to test lists what is still open: clean build time and size, Linux, the Steam Deck and the Test Farm run.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.