WIP: perf(lift): RX_MARK_DIRECT=1, page marks reach their table directly, not through a .refptr slot (compile only, off) #1979
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1979
Loading…
Reference in a new issue
No description provided.
Delete branch "siantm/Longshot:perf/lift-mark-mask"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
A page mark is a shift, a mask into the soft write watch's table and the byte store, plus the table's address, which
MinGW code loads from its
.refptrslot, again after guest stores that might alias the slot. WithRX_MARK_DIRECT=1the generated
a64.hdeclares the table hidden (clang; it is the executable's own), so the lifted code and the host'sfiles reach it directly. The index keeps its mask and the marks stay release stores: only the table's address changes.
Unset or off,
a64.his main's byte for byte, so no lifted object's key changes. Compile only; the build's log namesthe mode, and
dev.py packbuilds the game when it is on. From 1bf5203e on a local performance branch, reworked inreview: the switch was
RX_MARK_MASK=0and also dropped the mask; the reviewers asked to keep the bounds, so that halfis gone and the switch is renamed for what it still does.
The bitmap lookup's plain load (
A64_GPU_PAGE, whichhos_gpu_pages_setupdates atomically) is main's and this PR nolonger touches it: making it atomic changes the default
a64.hfor every player, so it is left for a change of its own.Read first:
rt_headerinnx2/build.py,MARK_DIRECTinnx2/rtgen.py,docs/dev/reference/page-marks.md,tests/test_mark_direct.py.Testing
Tested: Windows 11, i7-9700, RTX 4060, llvm-mingw clang 23.1.5, Python 3.14.7 (the Builder's); game-free checks
only. This change has not run in the game on this PC yet, with the switch on or off (Still to test). Each check ran in
a hidden console of its own, through the PC's job queue (job fb2-1979-probe), on this change's tree before its rebase
onto
main@f1f2dc5b(whose one commit touches none of its files; head327c4f8aafter it), the test run with noarguments (its compiler found through
LONGSHOT_MINGW, astests/run_all.pygives it):python -I -B tests/test_mark_direct.py: exit 0, "all passed": the header main's for every off value and differentin the table's declaration alone for every on value; the table hidden in the IR with the switch on alone, release
marks each after its guest store;
.refptr.a64_dirtyin main's MinGW assembly and none with the switch; seeds 1 to 3,O1 and O2, bitmap off and on, the same digest, polls and marks either way (e.g. "seed 3, O2, bitmap on: off
3b699ce7951861dc 7657 3701, on 3b699ce7951861dc 7657 3701"), the index inside the table at the window's edge.
The same test with
tests/test_vk_pgo.py,tests/test_split_giants.pyandtests/test_light_passes.pyis queued on327c4f8aitself (job fb2-1979-final); its result goes in a comment.Earlier, at dd39b843 (the first form,
RX_MARK_MASK=0),python -I -B tests/run_all.py --strict --changed origin/mainonmain@2e1f6c32: 98 passed, 17 failed (named in that run's comment as failing onmaintoo; two of them,test_pipeline_states.pyandtest_recorder_sync.py, had found no Vulkan headers in that run (RX_CPMunset) and compiled nothing on either side),7 skipped. That run was not repeated on this head. Those two, run again with the headers at
327c4f8aand onmain@f1f2dc5b:test_pipeline_states.pypasses on both;test_recorder_sync.pyfails onmaintoo with the same lines.Round 2 (
dev.py baseand the shared base): the base is keyed by its commit alone, anddev.py basepassed thecaller's environment to its build, so a base first built with
RX_MARK_DIRECT=1became the off side of every later A/B. Nowthe shared base's build leaves the switch out (
scripts/devtools/build.pycmd_base,packs.GAME_COMPILE_SWITCHES).The branch was rebased unchanged onto
main@9dd41326(git range-diff: "=" for the PR's commit). Through the PC's jobqueue (job fb2-lift-r2,
RX_CPMset, each check in a hidden console of its own, 20 minutes at most):python -I -B tests/test_mark_direct.pyat the test commitdfa8408e: exit 1, "1 failed": "dev.py base, RX_MARK_DIRECT=1 then unset: itsbuild leaves RX_MARK_DIRECT out, as main builds it"; at the fix, head
de30a7f9: exit 0, "all passed" (both orders, one base build,without the switch).
tests/test_vk_pgo.pyandtests/test_light_passes.pyatde30a7f9: exit 0 each.Round 3 (
dev.py packs baseand a base you name): round 2 left the switch out only indev.py base. Thedocumented
dev.py packs baserunspacks.cmd_basedirectly, whose build still tookRX_MARK_DIRECTfrom the caller'senvironment, so a named base built with it on was what a later
dev.py packs rtwith it unset copied. Nowpacks.cmd_base, where both commands converge, builds with the game's compile-only experiments(
GAME_COMPILE_SWITCHES) taken out of the environment afterpacks.runmerges the caller's; the wrapper's own strip isgone. A pack with the switch on still builds its own game in that mode. On
main@9dd41326(main has not moved; theearlier commits are unchanged), through the PC's job queue (job fb2-lift-r3b, light, two checks at once at most,
RX_CPMset, each check in a hidden console of its own, 20 minutes at most):python -I -B tests/test_mark_direct.pyat the test commitfa3212a5: exit 1, "2 failed": "dev.py packs base, RX_MARK_DIRECT=1 first,unset second: the base's build (make_game.py's environment) has no RX_MARK_DIRECT, as main builds it" and "...: dev.py packs
rt with the switch unset and no pack runs a game built off (got on)"; at the fix, head
4388b51c: exit 0, "all passed"(
dev.py packs baseanddev.py base, each with the switch on first and unset second and the other way round, thendev.py packs rtwith no pack, anddev.py packs packandrtwith the switch on).tests/test_dev_packs.pyandtests/test_dev_build.pyat4388b51c: one failure each, the same onmain@9dd41326in the same job ("every file the renderer keeps beside the game is classified (unknown:
['longshot_fsr4_pipelines.bin'])";
Linux.test_folders:C:\Users\User\GamesagainstC:\Users\User/Games).In the game, on this PC (Windows 11, i7-9700, RTX 4060), 2026-11-22: Kokiri Forest 800x600,
RX_MARK_DIRECTunset againstset at build, both at
4388b51c, a warm-up a side then 8 interleaved runs a side, every run in the same scene state: mediancalm game time a frame 10.05 against 10.23 ms with it on, 1.8 % higher, no gain shown (leaning to a loss) (Measurements).
Still to test:
RX_MARK_DIRECTunset and with it set, its build time and theexecutable's size, expecting figures for Measurements (by hand)
python3 -I -B tests/test_mark_direct.py, expecting it to pass; and the game built with the switch unset and set, played inKokiri Forest, expecting no crash and frame times alike (by hand)
RX_MARK_DIRECTunset and set, Kokiri Forest or the Market, a few interleaved runs a side, expectingno crash and frame times alike (by hand)
main@9dd41326, expecting the same lifted code; held until the farm pairs this head with9dd41326(the Test Farm request already on this PR)expecting the medians within about 2 % of each other or the lean confirmed (by hand)
Measurements
Verdict:
RX_MARK_DIRECTships off (the maintainer's request, while the enabled and disabled results are gathered). In play,no gain: the median calm game time a frame is 1.8 % higher with the switch on (10.23 against 10.05 ms), with more long-tail runs on that side (five of eight with p99 above 24 ms, one at 24.3, against one of eight off); inside the spread, but leaning to a loss. Eight runs a side resolve roughly 1.5 to 2 % on the median (an estimate: identical runs of one build spread
about 5 % end to end in calm game time). The switch stays off.
In play: Kokiri Forest, off against on at
4388b51c, 8 runs a side (the current reading)Setup: Windows 11, i7-9700, RTX 4060, run alone on the PC on 2026-11-22, 06:48 to 07:33 UTC; Kokiri Forest (child), 800x600
(CPU-paced), frame cap off, 60 s a run; a warm-up a side, then 8 interleaved runs a side (off on on off, four times). The
teleport into the scene checked as taken in every run, so every run is in the same (light) scene state; the GPU cooled
to 70 °C before each run, its median clock over each measured stretch its full 2,745 to 2,760 MHz (the few dips are in
the clk_min column). Calm game time: the measured 60 s in six
10 s windows, each window whose median game time is more than 20 % above the run's lowest dropped (a burst), then the
median game time of the frames left; blocked: the game thread's wait over the same frames.
Baseline: this PR with
RX_MARK_DIRECTunset (off), which builds withmain's flags and header, so it stands formain@9dd41326(the merge base); on = set at build.
4388b51c4388b51cEach run (the run's table; game and calm in ms a frame; clock in MHz over the measured stretch):
Clean build time and size, Linux and the Steam Deck are not measured.
Earlier: 4 runs a side, before the wrapper checked the teleport (2026-11-22, 00:15 to 01:03 UTC)
Reading then: not decided, no loss seen. Every run sits in one of two scene states, and both sides land in both.
Setup: Windows 11, i7-9700, RTX 4060, run alone on the PC on 2026-11-22; Kokiri Forest (child), 800x600 (CPU-paced),
frame cap off, 60 s a run; one warm-up a side, then 4 interleaved runs a side (off on on off off on on off). off =
RX_MARK_DIRECTunset, whose lifted code, object keys and executable are those of
main@9dd41326(the merge base); on = set at build.Range: off 69.1 to 80.3 fps, on 69.4 to 83.2. The two states: light about 1,100 textures a frame and 76 to 83 fps, heavy
about 1,380 and 67 to 73. The run wrapper asks the Debug teleport into Kokiri Forest, where the save already puts Link; asked
within about 22 frames of entering play, the game refuses it and nothing reloads (heavy), asked later it is taken (light).
Within one state the two sides overlap (heavy: off 69.1 and 70.5 fps, on 69.4; light: off 78.3 and 80.3, on 80.3, 81.7 and 83.2): inside the spread.
Game-free checks
Setup: Windows 11, i7-9700, llvm-mingw clang 23.1.5;
tests/test_mark_direct.pythrough the PC's job queuef1f2dc5b327c4f8atests/test_mark_direct.pyrt_headeron main).refptr.a64_dirtyin main's MinGW code, none with the switchRisks
Only the table's declaration changes: the index keeps its mask, the marks their release order. Off by default: unset,
the build and its header are main's. Turning it off is unsetting it and building again: an executable keeps what it
was built with. Changing it compiles the lifted code and the host's files again, and translates the game again where
the translated C is gone (a player's build folder; a development checkout and
LONGSHOT_KEEP_BUILD_FILES=1keep it).Nothing a player keeps (saves,
longshot_options.ini) changes.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.Recommendation: Request changes. The opt-in path can index past both write-watch tables, and the generated header forces existing installations to recompile the lifted game even with the experiment disabled. The Windows fixture shows the intended code generation, but there is no measured gain in play.
Adds a developer build switch that removes the page-index mask and lets Clang address the dirty table directly. The reported MinGW fixture eliminates the
.refptrreference at O1 and O2 and produces matching digests for three seeds. The switch couples direct addressing to a separate bounds change and adds a rebuild cost for players who leave it unset.@fixwill do (12)nx2/rtgen.py:73: The raw page index is used for both a64_dirty and a64_gpu_pages. (more on the line)nx2/rtgen.py:82: The bitmap lookup whose index changes here is a plain load, while hos_gpu_pages_set atomically updates the same bytes. A game thread storing to one page can read a byte while another thread registers a neighboring page in that byte. The race already exists on main and carries into the new path. Use an atomic byte load. Relaxed ordering suffices if this read only tests the bit and mapping publication is synchronized separately.nx2/rtgen.py:60: Keep the generated default header byte-for-byte compatible and select the experimental variant only when requested, or preserve proven-compatible default object keys.docs/dev/reference/page-marks.md:9: The exact instruction and mark counts here come from the game's lifted output, and the weighted figures also use a sampled game profile. (more on the line)tests/test_mark_mask.py:36: The suite always passes --cc clang, so find_cc never consults CC or its llvm-mingw fallback. (more on the line)tests/test_mark_mask.py:33: Importing build also runs host discovery outside the fixture. (more on the line)tests/test_mark_mask.py:159: The poll callback writes c.poll through cur_ while the loop reads and writes the same member through its restrict-qualified c parameter. (more on the line)tests/test_mark_mask.py:137: The assembly check is skipped whenever the masked build lacks the literal .refptr name, even on MinGW. (more on the line)tests/test_mark_mask.py:180: The fixture compares two digests over low addresses and aligned eight-byte stores. (more on the line)docs/dev/reference/page-marks.md:3: The introduction says every guest store marks and implies one byte store per distinct page. (more on the line)For the author (2) (a run, a measurement or a decision)
nx2/rtgen.py:60: The emitted a64.h changes even with RX_MARK_MASK unset. (more on the line)Checks: Still running:
PR description / Description follows the template (pull_request_target),CI / Conventional Commit title (pull_request),CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request),CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request),CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request),CI / Workflow lint (actionlint) (pull_request),CI / Website build and checks (pull_request),Guide screenshots / Guide images and local UI source review (pull_request),Third-party parts / As their notes say (pull_request),Third-party parts / The same as upstream (pull_request).More: does it help, other systems, switches, what was read
Does it help: Unproven. The author reports matching digests and poll counts for three seeds at O1 and O2 on Windows, with
.refptr.a64_dirtypresent in the masked assembly and absent in the unmasked assembly. Those checks ran on dd39b843 before the reported unchanged rebase tob4a03790. There are no interleaved gameplay or clean-build measurements against6e1521a7.Other systems: The changed header reaches lifted code, shared A64 helpers, native host stores and the emulator-hosted DLL build. The Vulkan renderer consumes marks through host callbacks, so GPU imports, mirrored buffers and captures can all see the effect of missing marks on either OS, including Steam Deck. The collector's existing non-atomic word-walk reads were outside this diff. Windows and Linux standalone links retain the image-span guard for RIP-relative references. The Launcher and AppImage can hit the object-cache invalidation through the normal update path. The diff changes no renderer API, shader, save format, audio or input code.
Switches: RX_MARK_MASK is compile-time only. Unset, empty or any value other than 0, off, false or no keeps the mask and default visibility. Those four values enable A64_MARK_UNMASKED, which also hides the dirty symbol under Clang. Keeping the masked path as the default preserves the current bounds protection. No measurements justify shipping the experiment. The build switch is documented in build.md, which is the repository's table for build-only switches. switches.def covers runtime switches. The test adds --cc and --cflags, defaulting to compiler discovery and no extra flags, but the suite overrides discovery with --cc clang.
Reviewed:
b4a0379028against6e1521a70a: Read AGENTS.md, CONTRIBUTING.md, docs/dev/standards.md and the Forgejo pull request template in full. Reviewed every hunk in all seven changed files and read nx2/rtgen.py, docs/dev/reference/page-marks.md and tests/test_mark_mask.py in full. Read the relevant surrounding sections of the other four changed files and traced the generated header, store helpers, bitmap writer, collector, build and lift caches, pack routing, compiler selection and test runner. Other sections of those larger files were skimmed or outside the reviewed paths. docs/review-checklist.md and the sibling Tooling checkout were absent. Reviewed the supplied test report and CI status. No comments, reviews, raw run logs or generated assembly were available.; nothing run (a code review).Checked and fine:
Testing: Read-only review. git diff --check passed. The author reported the earlier Windows fixture passed and named 17 broader failures that also occurred on its earlier base. The PR lists seven skips. The rebased head, Linux fixture, game runs and build timings have no reported results. The supplied CI checks are still running.
Overlaps and merge order: None found in the supplied Forgejo record or local history. The offline checkout did not provide the open pull request list.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -0,0 +1,74 @@# Page marks: the index and the table's address (RX_MARK_MASK)Every guest store the lifted code makes marks the 4 KB pages it touches in `a64_dirty`, the soft write watch's tableL10 (minor): The introduction says every guest store marks and implies one byte store per distinct page. The bitmap skips stores on untracked pages, the lifter can coalesce stores or omit stack marks, and an ordinary mark stores both endpoints even when they share a page. Describe the default path before using its instruction counts to explain the change.
@ -0,0 +6,4 @@## What a mark costsCounted in the clang `-O3` code of the game's 20 hottest lifted functions, on the local performance branch thisL4: The exact instruction and mark counts here come from the game's lifted output, and the weighted figures also use a sampled game profile. The repository's no-game-derived-output rule covers those results. Remove the game-derived counts and percentages throughout this page, including the discarded trials. Use results from an authored game-free fixture for committed examples.
@ -58,3 +58,3 @@#define A64_MARK(a, n) (__atomic_store_n(&a64_dirty[(uint32_t)(a) >> 12], 1, __ATOMIC_RELEASE), \__atomic_store_n(&a64_dirty[(uint32_t)((a) + (n) - 1) >> 12], 1, __ATOMIC_RELEASE))"""MARK_NATIVE_64 = r"""extern uint8_t a64_dirty[A64_DIRTY_PAGES];MARK_NATIVE_64 = r"""/* (nx2/rtgen.py) A64_MARK_UNMASKED (nx2/build.py RX_MARK_MASK=0): the table is declared hidden, as it is theL3: The emitted a64.h changes even with RX_MARK_MASK unset. build.py hashes the header into the lifted object keys, so an existing player's update loses the cached lifted objects. If cleanup already removed the lifted C, the kept-object check also forces a new translation unless the lift can be restored. Measure an update from the merge base as well as a clean build.
@ -61,2 +70,4 @@/* (Tried 2026-10-21: reading a mark first and writing it only when clear, against cache lines moving between cores:game threads +1.1 ms a frame, the code 18 MB bigger. Not kept.) */#if defined(A64_MARK_UNMASKED)#define A64_DIRTY_AT(a) ((uint64_t)(a) >> 12)L1: The raw page index is used for both a64_dirty and a64_gpu_pages. A bad guest address can resolve to writable host memory outside the reservation, so a completed data store does not prove that either metadata index is valid. The bitmap read or the two endpoint marks can then fault or access unrelated host memory. Keep bounded indexing on every path, or check both endpoints before touching either table. The hidden-symbol change can be tested independently.
@ -0,0 +30,4 @@sys.path.insert(0, os.path.join(ROOT, "nx2"))sys.path.insert(0, HERE)import kit # noqa: E402,F401 (a64-kit on the path)import build # noqa: E402L6: Importing build also runs host discovery outside the fixture. On Linux it can invoke pkg-config and probe another Clang, and inherited RX_TARGET, LONGSHOT_SYSROOT, RX_MARCH and RX_MARK_BITMAP change the test flags. For example, RX_TARGET=windows with LONGSHOT_MINGW set makes the Linux run build Windows executables, and RX_MARK_BITMAP=0 silently skips bitmap coverage. Keep this test's flag selection free of build startup probes, set a runnable CPU baseline, and choose the target and both bitmap modes explicitly.
@ -0,0 +33,4 @@import build # noqa: E402import rtgen # noqa: E402RUN_ALL_ARGS = ["--cc", "clang"]L5: The suite always passes --cc clang, so find_cc never consults CC or its llvm-mingw fallback. A supported Windows setup with LONGSHOT_MINGW pointing to the Builder toolchain and no clang on PATH fails before any fixture compiles. Let the suite use the test's discovery and select the compiler and target together, honoring an explicit CC or --cc.
@ -0,0 +134,4 @@check(r.returncode == 0, "the loop's assembly (%s): %s" % (what, r.stderr[-1500:]))if r.returncode == 0:refptr[value] = ".refptr.a64_dirty" in open(base + ".s", encoding="utf-8").read()if refptr.get(""): # (a MinGW target: the default reaches the table through the slot)L8 (minor): The assembly check is skipped whenever the masked build lacks the literal .refptr name, even on MinGW. A compiler or target change can therefore make both variants omit that spelling and still pass without proving direct access. Select the assertion by target, and verify the unmasked instruction addresses a64_dirty directly. The advertised Windows target example also needs an assembly-only mode on Linux because program_checks executes what it builds.
@ -0,0 +156,4 @@void a64_poll_cur(void) {polls_++;(void)rand();if (++npoll_ %% 3) __atomic_store_n(&cur_->poll, 1, __ATOMIC_RELAXED);L7: The poll callback writes c.poll through cur_ while the loop reads and writes the same member through its restrict-qualified c parameter. That violates the fixture's aliasing contract. The real native callback leaves the poll member alone. Have the fixture count polls without modifying the context through a second pointer, or arrange repeated polls from the driver between loop invocations.
@ -0,0 +177,4 @@f_80000000(&c);for (int i = 0; i < 31; i++) digest = mix(digest, c.x[i]);digest = mix(digest, npoll_);for (int i = 0; i < 17; i++) { digest = mix(digest, a64_dirty[i]); a64_dirty[i] = 0; }L9 (minor): The fixture compares two digests over low addresses and aligned eight-byte stores. Both sides could stop marking the last byte's page or lower the marks to relaxed stores and still agree. Add expected marks for a deliberate page-crossing store, exercise addresses near the window limit, and check for atomic release stores in the IR.
Recommendation: Request changes. Keep page-mark indexes bounded, preserve the masked build's update path, and make sure a switch-only development pack actually contains the experimental executable.
The same-source A/B path can silently produce a masked test run.
dev.py packroutes from the files changed against the selected base.COMPILE_ONLY_ENVhere only changes the lift-key diagnostic, andpacks.pyhas no route override forRX_MARK_MASK. A clean checkout matching its base produces an empty pack withRX_MARK_MASK=0, sodev.py rtuses the base executable. A renderer-only source change can also skip the game build. Route an enabled experiment to the game build and cover the clean-base case in the pack fixture. Pack routingThe first and last bytes of a mark must stay within the dirty table, and the first-page lookup must stay within the bitmap. Completing a guest store does not establish either bound. Keep the existing bounds protection and measure direct symbol addressing independently. Generating a different
a64.hwith the switch unset also invalidates existing lifted object keys and can send an installation that retained only compiled objects back through translation. Preserve the default object keys or measure the actual update cost. Replace the committed game-derived counts with measurements from an authored game-free fixture. Bounds · Update cache · DocumentationDoes it help? Unproven. The author reports that the earlier Windows fixture removed the
.refptr.a64_dirtyreference at O1 and O2 and produced matching digests across three seeds. Its setup has compiler-selection and poll-aliasing issues, and its aligned stores leave page crossings untested. Add assertions for the expected first and last marks across a page boundary, exercise both bitmap settings, and check release stores in the generated IR. Once corrected, compare the base and final masked and enabled builds in warmed, interleaved gameplay. Report average FPS, 1% lows, median and p99 frame times, and compare clean-build and update cost. The author's Windows game run, a Linux fixture run, and affected Tier 1 gameplay are outstanding. The description check passed and the remaining source checks were pending on this head. Beanstalk shows no jobs or reports. Fixture discussion · BeanstalkChecked and fine. The flag reaches the lifted chunks, shared A64 helpers and native host sources. With the generated C retained, toggling the flag leaves the lift key alone and separates compiled object keys by flag. The mark stores still use release ordering and the collector takes the byte with an acquire exchange. The link-size guard still applies to the standalone executable and the emulator-hosted DLL. The build-variable entry belongs in the build reference and the flag currently defaults to masked.
Merge order. The loop-store and poll-convention experiments conflict in shared build wiring. Each can be evaluated independently. If either lands first, rebase the shared hunks and recheck that each switch still selects its own compiled mode. Loop-store PR · Poll-convention PR
Reviewed
b4a03790against6e1521a7. Read all seven changed files and traced the generated store helpers, mark bitmap, renderer collector, object-cache keys, kept-translation fallback and development pack routing with revision-matched Graphify maps and source. No PR code was run.@ -49,1 +49,3 @@COMPILE_ONLY_ENV = ("RX_HOT_OPT", "RX_PGO", "RX_VK_PGO", "RX_VK_PGO_FILE", "RX_A64_RT_O2", "RX_GAME_ONLY") # (make_game.py's: left out of the lift key)# (make_game.py's: left out of the lift key)COMPILE_ONLY_ENV = ("RX_HOT_OPT", "RX_PGO", "RX_VK_PGO", "RX_VK_PGO_FILE", "RX_A64_RT_O2", "RX_GAME_ONLY","RX_MARK_MASK")This entry only suppresses the lift-key diagnostic. On a clean checkout matching its shared base,
RX_MARK_MASK=0leavespack_routeatNonebecause no source files changed, sodev.py packcreates an empty pack anddev.py rtstill uses the masked base. On a renderer-only diff the game also stays at base. Route the enabled switch to a game build, as the pack path already does for RX_PGO, and add a clean-base routing case.b4a0379028327c4f8a93WIP: perf(lift): RX_MARK_MASK=0, page marks index the table unmasked and reach it directly (compile only, off)to WIP: perf(lift): RX_MARK_DIRECT=1, page marks reach their table directly, not through a .refptr slot (compile only, off)Pushed
327c4f8a(one commit, onmain@f1f2dc5b; it replacesb4a03790). The reviews' bounds point decided the scope: the unmasked index is dropped, both tables keep main's masked index, and the switch is renamedRX_MARK_DIRECT=1(title, docs, build reference, test) for what it still does, the hidden declaration of the marks' table. With the switch unseta64.his main's byte for byte;dev.py packbuilds the game when it is on; the build's log names the mode; the counts over the game's lifted functions are removed (the commit is squashed so that no pushed doc keeps them). The fixture stores across pages with exact expected marks, takes the marks at each poll, checks release marks in the IR, the assembly by target and the index at the window's edge. Tested:tests/test_mark_direct.pyall passed on the tree before the rebase (main's new commit touches none of its files); the run on327c4f8aitself is queued and its result follows here. Still to test: the in-game A/B and Linux.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62850: fixed in
327c4f8a.scripts/devtools/packs.pypack_routenow builds the game when the switch is on (nx2/kit.pyenv_on's rule), for a clean checkout matching its base and for a renderer-only change;tests/test_mark_direct.pycovers both routes and the off values. The switch is nowRX_MARK_DIRECT=1(see the review reply), and the build's log printsbuild: RX_MARK_DIRECT=1: ...when the executable has it.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62851: addressed in
327c4f8a(one commit, rebased onmain@f1f2dc5b), and your bounds point decided the scope: the unmasked index is dropped, both tables keep main's bounded index, and the switch is renamedRX_MARK_DIRECT=1for what it still does, the hidden table declaration alone (title, docs, build reference and test renamed;RX_MARK_MASKis left only in the doc's note on the dropped form). With the switch unseta64.his main's byte for byte, so retained objects keep their keys; the pack route is fixed (62850); the game-derived counts are removed. The fixture now stores across pages with exact expected marks, takes the marks at each poll, checks release marks after their data in the IR and the index at the window's edge, both bitmap modes. The gameplay and build-cost comparisons are pending with the opener.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62540 (L1): fixed in
327c4f8aby dropping the unmasked index, as the maintainer's review also asked:A64_DIRTY_ATis main's masked index on every path, for both tables. The test checks the index at the guest window's edge (the last byte, one past, far past) in every program. The switch now only declares the table hidden and is renamedRX_MARK_DIRECT=1.(L2 in https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62549): not changed here. The plain bitmap load is main's (
A64_GPU_PAGE), and after this round the PR no longer touches it or the index; making it an atomic load changes the defaulta64.h, which is what L3 asks us not to do in this PR (every player's lifted objects would rebuild). It is left for a change of its own, and the description says so.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62541 (L3): fixed in
327c4f8a.rtgen.writeadds the hidden declaration only withmark_direct=True(nx2/build.pyrt_header, switch on); unset or off,a64.his main's byte for byte, so no object key changes and no update cost is added. The test compares the header withrtgen.write's default for every off value.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62542 (L4): removed in
327c4f8aas asked. The counts and weighted percentages over the game's lifted functions are gone from the page (the discarded trials too), the commit message and the description; the page says those counts were taken locally and are not published, and the test is the authored fixture for the claims kept. The synthetic probe's figure stays, labelled as belonging to the dropped unmasked form.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62543 (L5): fixed in
327c4f8a.RUN_ALL_ARGSis gone and the test resolves the compiler itself:--cc, thenCC, clang on PATH, thenLONGSHOT_MINGW's clang, with that compiler's own target. The final run found the Builder's clang throughLONGSHOT_MINGWwith no arguments.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62544 (L6): fixed in
327c4f8a.nx2/build.pyis imported withRX_*,LONGSHOT_*andOOT_*cleared,RX_MARCH=x86-64and no process allowed to start (pkg-config, a compiler probe); the target comes from the compiler and every program is built with the bitmap off and on explicitly.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62545 (L7): fixed in
327c4f8a. The poll stand-in touches no context; the loop raises the request through its owncevery eighth iteration and the driver checks the exact polls taken per call.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62546 (L8): fixed in
327c4f8a. The assembly check is chosen by the compiler's target (-dumpmachine): with the switch on the code must namea64_dirtyitself with no.refptrslot or GOT entry, on any target; off, a MinGW target must load.refptr.a64_dirty. A--targetin--cflagsnow runs the IR and assembly checks only, never a program.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62547 (L9): fixed in
327c4f8a. Destinations are unaligned so stores cross pages, and each run checks the exact marks left (first and last byte's pages, where the bitmap says) after the last poll took the others; the index is checked at the window's limit; the IR check requires each guest store to be followed by tworeleasemark stores before the next.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62548 (L10): fixed in
327c4f8a. The page now describes the default path first: only stores starting on a GPU page mark (the bitmap), two release marks a store even on one page, SP and frame-pointer stores unmarked, coalesced stores, the host's own marks.(L11 in https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62549): fixed in
327c4f8a. Risks and the page now say turning it off is unsetting it and building again (an executable keeps what it was built with), and that lift reuse holds only where the translated C is kept (LONGSHOT_KEEP_BUILD_FILES=1or a development checkout).(L12 in https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-62549): the edit part is fixed in
327c4f8a: the bounds change is gone, so the switch is the visibility change alone, and the build's log records the compiled mode. The A/B against the merge base in play, with clean build and update cost, is pending with the opener.@Beanstalk This adds a compile switch, RX_MARK_DIRECT, off by default: with RX_MARK_DIRECT=1 the lifted code's page marks index the soft write watch's table without the mask and reach it directly (nx2/build.py, nx2/rtgen.py). Setting it keeps the lift; only the compile changes. The farm cannot build with it set, so only the default build is to test: with the switch unset, the lift and its C must be main's. Please time the lift against main's and check the lifted C is the same, on a Windows PC and on a Linux PC. No game scenario is needed: unset, the game's code is main's.
@siantm only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was for327c4f8, and the PR has changed since (de30a7f), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.A correction to 63220: "the final run" there is the queue run of this change's tree before its rebase onto
main@f1f2dc5b(job fb2-1979-probe, uncommitted; the rebase touched none of its files). The run on327c4f8aitself (fb2-1979-final) is still queued; its result will follow here.The queue run on
327c4f8aitself (job fb2-1979-final, Windows 11, i7-9700, llvm-mingw clang 23.1.5, each check in a hidden console of its own) ended:python -I -B tests/test_mark_direct.py(no arguments, the compiler throughLONGSHOT_MINGW) "all passed", andtests/test_vk_pgo.py,tests/test_split_giants.py,tests/test_light_passes.pyexit 0 each.Tested corrected: an earlier run on this PC had no Vulkan headers for the renderer checks (
RX_CPMunset), so the ones it listed as also failing onmainhad compiled nothing on either side. They ran again with the headers at327c4f8aand onmain(queue job fb2-vk-1); the description now gives those results. No check fails at this head alone; nothing in the branch changed.Recommendation: Wait for the enabled gameplay and build comparison. The revision keeps both page-index masks and the release stores. With
RX_MARK_DIRECTunset, the generator follows the original header path. Turning the switch on now routes an unchanged or renderer-only development pack through the game build. The new fixture checks cross-page stores, both bitmap modes and release stores in the generated IR. Earlier reviewThe author reports the new fixture and three related checks passed on exact head
327c4f8ain Windows. The change gets the intended direct symbol access in the fixture. The earlier instruction-count result came from a now-removed unmasked variant. Measure the retained direct declaration with a verified enabled executable and a default executable from the same revision, plus the merge-base build, in warmed, interleaved runs. Linux validation, author gameplay, build and update cost, and required CI remain open.The existing farm request can check default lifted C and lift timing with a matching base. It still describes the removed unmasked path, and the dashboard lists
6e1521a7as the base while this revision branches fromf1f2dc5b. Hold it until the request and pair match. The enabled compile-time switch needs a separate build. Recheck the shared build and pack wiring if #1978 or #1985 lands first. Existing farm request · Farm statusReviewed exact
327c4f8aandf1f2dc5b, the interdiff withb4a03790, and the affected build and page-mark paths using both revision maps and source.Recommendation: Request changes. I missed a shared-base cache issue in the earlier Wait review.
dev.py base <rev>forwardsRX_MARK_DIRECT=1when it builds a base, but it keys that base by commit alone. If the enabled build populates the base first, a later invocation with the switch unset reuses it. An unchanged-tree pack then has no game executable to overlay, so both sides of the comparison can run with direct marks enabled. Earlier review · Shared base pathBuild ordinary shared bases with the experimental switch off, or include the effective mode in the base identity and selection. Cover both build orders at the same revision and verify the executable that the default runtime actually starts. The bounds, default header and focused Windows fixture findings from the earlier review still apply. The farm request remains held for its baseline mismatch.
327c4f8a93de30a7f9a6Pushed
de30a7f9(onmain@9dd41326; the PR's commit rebased unchanged): futondestroyer's second look.dev.py basenow builds the commit-keyed shared base withoutRX_MARK_DIRECT, so the base is the default executable whichever order the builds come in; the test commitdfa8408efails on it and the fixde30a7f9passes (job fb2-lift-r2). Testing has the round's results.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-65506: fixed in
de30a7f9(the test first, failing, indfa8408e), rebased unchanged ontomain@9dd41326.dev.py base(cmd_base) now leavesRX_MARK_DIRECT(packs.GAME_COMPILE_SWITCHES) out of the shared base's build and says so, so the commit-keyed base is the default executable whichever order the builds come in, and a pack with the switch on builds its own game.tests/test_mark_direct.pyrunscmd_baseat one revision in both orders: one build, without the switch, and the default runtime of an unchanged-tree pack is that base's game. At the test commit the test fails ("dev.py base, RX_MARK_DIRECT=1 then unset: its build leaves RX_MARK_DIRECT out"); atde30a7f9it passes (job fb2-lift-r2).https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-65478: the comparison you ask for is queued with the opener: two packs at one revision, mdir-off (
RX_MARK_DIRECTunset) and mdir-on (RX_MARK_DIRECT=1), rebuilt on this round's head, plusmain@f1f2dc5bbuilt by its own lift build (build time and executable size logged); gameplay as one held segment, warm-ups then main, off, on, on, off, main, in Market. A separate update-cost build is not planned. Withde30a7f9the shared base no longer takes the switch (65506), so the off arm is the default executable; the farm request will be redone for this head's merge base, describing the hidden declaration only.@Beanstalk This adds a compile switch, RX_MARK_DIRECT, off by default: with RX_MARK_DIRECT=1 the soft write watch's table is declared hidden (the executable's own), so the lifted code's page marks reach it directly instead of through an address slot (nx2/build.py, nx2/rtgen.py). The farm cannot build with it set, so only the default build is to test: with the switch unset, the lift and its C must be main's. This head is on main
9dd41326: please compare with main at that merge base. Please time the lift against main's and check the lifted C is the same, on a Windows PC and on a Linux PC. No game scenario is needed: unset, the game's code is main's.@siantm only members of
BigBeanscan start the farm.One of them can approve this request by reacting 👍 to it or to this comment.This request was forde30a7f, and the PR has changed since (4388b51), so a 👍 no longer approves it. Comment@Beanstalkagain to ask for the new commit.Recommendation: Request changes. The high-level
dev.py basecommand now stripsRX_MARK_DIRECT, but the documenteddev.py packs basecommand still inherits it. With the flag enabled when that named base is built,dev.py packs rtlater copies its enabled game even with the flag unset. An unchanged off-mode pack also supplies no game override. Enforce the default in the common base builder, and cover the direct command’s environment in the fixture. Enabled packs should still build their own game. Documented A/B routeThe new fixture covers both high-level construction orders at the same revision. The author reports the enabled-first case failing at
dfa8408eand passing atde30a7f9. The test replaces the queue call, so it never checks the separate direct entry point. Previous review · Author’s regression resultThe retained experiment changes how the table is addressed. It preserves both page-index masks and the acquire/release protocol. The author’s exact-head Windows fixture covers direct addressing, page-crossing stores and both bitmap settings. Gameplay benefit remains unproven. The earlier instruction-count result was for the removed unmasked variant. Compare the off and enabled executables at
de30a7f9and the actual merge-base build at9dd41326, using warmed interleaved play and clean build time and size. Linux gameplay and source validation remain open.The existing farm request now describes the direct declaration and asks for the right merge base. The dashboard still shows
f1f2dc5b. Hold the off-only lift parity request until the farm pins9dd41326. The enabled compile mode needs separately built executables. Farm request · DashboardReviewed exact
de30a7f9and9dd41326, the interdiff against327c4f8a, and the affected build and page-mark paths with both revision maps and source. Required CI remains pending and the PR remains WIP.@ -392,1 +394,4 @@py, env = environment()# (a shared base is keyed by its commit alone, and is the off side of every A/B on it: built as main builds it, without# the game's compile-only experiments; a pack with one on builds its own game, packs.py pack_route)left = [k for k in packs.GAME_COMPILE_SWITCHES if env.pop(k, None) is not None]This strips RX_MARK_DIRECT when invoked through dev.py base. The documented dev.py packs base command calls packs.cmd_base directly and still inherits RX_MARK_DIRECT=1 through make_game. A later packs rt with the flag unset copies that already-enabled executable. Enforce the default in the common base builder and test the direct entry point as well as the wrapper.
Pushed
4388b51c(onmain@9dd41326; the earlier commits unchanged): futondestroyer's third look.dev.py packs basenow builds a named base withoutRX_MARK_DIRECTtoo: the rule sits inpacks.cmd_base, where it anddev.py baseconverge, and the wrapper's own strip is gone. The test commitfa3212a5fails without it and the fix4388b51cpasses (job fb2-lift-r3b). Testing has the round's results; Measurements says the switch stays off by default at the maintainer's request.https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-67213: fixed in
4388b51c(the test first, failing, infa3212a5).packs.cmd_base, whichdev.py packs baseruns directly anddev.py basethrough the queue, now builds the base withRX_MARK_DIRECTtaken out of the environment afterpacks.runmerges the caller's, and the wrapper's strip inscripts/devtools/build.pyis gone; a pack with the switch on still builds its own game in that mode.tests/test_mark_direct.pynow drives both entry points throughdev.py's command line, enabled first then unset and the reverse, checking the base build's environment and thatdev.py packs rtwith no pack runs an off game while the enabled pack's runtime runs an on one: 2 failures atfa3212a5, all passed at4388b51c(job fb2-lift-r3b).https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/1979#issuecomment-67214: addressed in
fa3212a5(test) and4388b51c(fix), onmain@9dd41326with the earlier commits unchanged. Round 2 covered only thedev.py basewrapper; this round covers the directdev.py packs basecommand, with the default applied inpacks.cmd_basewhere both converge (inline 67213). Testing in the description has the round's results (67295); the gameplay comparison and the Test Farm pairing are still open.@Beanstalk This adds a compile switch, RX_MARK_DIRECT, off by default: with RX_MARK_DIRECT=1 the soft write watch's table is declared hidden (the executable's own), so the lifted code's page marks reach it directly instead of through an address slot (nx2/build.py, nx2/rtgen.py). The farm cannot build with it set, so only the default build is to test: with the switch unset, the lift and its C must be main's. Please time the lift against main's and check the lifted C is the same, on a Windows PC and on a Linux PC. No game scenario is needed: unset, the game's code is main's.
@siantm only members of
BigBeanscan start the farm. One of them can approve this request by reacting 👍 to it or to this comment.Re #1979 (comment) (gameplay benefit): the warmed, interleaved off/on comparison in play has run at
4388b51c, and Measurements now has it run by run. off is the switch unset, whose lifted code, object keys and executable are those of the merge base9dd41326; on is set at build. Kokiri Forest, 800x600, 4 runs a side after a warm-up: off 69.1 to 80.3 fps, on 69.4 to 83.2. Reading: not decided, no loss seen. Every run sits in one of two scene states (a refused or a taken teleport at the run's start, about 1,380 against 1,100 textures a frame) and both sides land in both; within a state the two sides overlap. The run wrapper is fixed, and a rerun of 8 a side, classed by state, follows. Clean build time and size, Linux and Steam Deck are not measured yet. The switch stays off.Re #1979 (comment) (gameplay benefit): the warmed, interleaved in-game comparison has run at
4388b51c, 8 runs a side in Kokiri Forest (800x600, CPU-paced), every run checked in the same scene state, GPU cooled before each, on this PC alone; off is the switch unset (main's flags and header). Calm game time a frame, median (range): off 10.05 ms (9.96 to 10.79), on 10.23 (10.05 to 10.46), 1.8 % higher with the switch on; fps 90.7 against 85.2, and five of eight on-runs had p99 above 24 ms against one off-run. That is inside what eight a side resolve (roughly 1.5 to 2 %, an estimate) but leans to a loss: play shows no gain for it. The switch stays off; whether it stays as an off-by-default experiment or is dropped is your call and the author's. Measurements has every run; clean build time and size, Linux and the Steam Deck are not measured.Testing brought in line with Measurements: the Kokiri Forest off/on comparison of 2026-11-22 (8 a side at
4388b51c, no gain shown) is now under Tested; Still to test lists what is still open: clean build time and size, Linux, the Steam Deck and the Test Farm run, and a repeat of the Kokiri comparison only if the reviewer wants its lean settled.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.