WIP: perf(lift): the lane-permute idioms are written one line, not five memcpys #1202
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1202
Loading…
Reference in a new issue
No description provided.
Delete branch "Marcuss2/Longshot:perf/lift-compact-emit"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: low · Tested on: Linux (RADV, RX 9070 XT) · Needs: none
Why
The per-instruction translator writes an
EXTby 8 bytes and aDUP(element) as runs ofmemcpythrough two 16-byte stack buffers: five copies for the EXT, two for the DUP, more for the byte splats. Each is a plain permutation of the 16 bytes of one vector register, so each has a one- or two-statement form on the register's twouint64_thalves (a64.h'svreg[32][2]):There are 66,915 EXT and 41,487 DUP sites in the game's lifted C, and where a function is dense in them this is the hardest stretch of it to read: the idiom's five lines name two stack buffers that the register's bytes pass through, and the actual permutation has to be worked out from the offsets. The lifted C is what a person reads to find the next problem, and what an agent pays for by the token; one line instead of five at every one of those sites is the point of this change.
What changed
A pass over the lifted C, right after the lift, rewrites the lane moves the translator writes as runs of
memcpythrough 16-byte stack buffers — EXT, INS, DUP, UMOV, SMOV and the two FMOV directions — into one or two statements on the register's two halves, through the lift's own whole-register store (a64_vset128/a64_vset64, the opaque 16-byte storewiden_vstoresexists for — the rewrite keeps it, it never splits the store into two 8-byte halves); a shape it does not recognize is left exactly as written, and the pass is idempotent. It is text only and exact: the same bytes move, which is what the compiled test checks against the build's guest model.A64_NO_COMPACT=1writes the emitter's text unchanged.nx2/compact_emit.py: the pass, one rule per shape of the family, each written on the register's twouint64_thalves and stored through the lift's opaque whole-register store (a64_vset128/a64_vset64, never two 8-byte halves — the store-forwarding stallwiden_vstoresexists to avoid). Covered, by the instruction that writes them: EXT by 8 (66,915), DUP (element) (41,487), INS (element) (15,815), INS (general) from XZR (12,399), INS (general) (2,574), DUP (general) (3,241), DUP (scalar) (8,050), UMOV (2,326), SMOV (21), FMOV general → FP (27,732), FMOV FP → general (9,409) — 189,969 sites in 93 files. A shape whose buffers would be read or written out of bounds, whose source lane crosses the source register's two halves, or whose copies are not contiguous elements stays exactly as the emitter wrote it.A64_NO_COMPACT=1writes the emitter's text unchanged.nx2/make_game.py: the pass runs right after the lift (besidesplit_hot.py), andLIFT_SCRIPTSnames it, so a change to the rewrite invalidates a stored translation instead of stamping old C with a new key.tests/test_compact_emit.py: 886 shape variants — for each, a text check (rewritten, one 16-byte store, the rest untouched, idempotent) and a compiled check that runs the old idiom and the new statement against the build's own guest model from 200 random contexts, comparing every byte of every vector and general register (0 differ; a mutation check that drops the masks gives 1,798 differences, so the comparison bites).tests/test_lift_cleanup_error.py: itsrundouble now recognizes the new lift step; its assertions are unchanged.Measurements
Verdict: no change to the game. The lifted C is 10 MB smaller (1564.7 → 1554.7 MB, −0.6 %) and the
longshotexe 837 KB smaller than main's, and the frame time and CPU time are within this PC's run-to-run spread (fps +0.75 %, p = 0.43; CPU −0.57 %, p = 0.19 over 5 arms each). This is a readability change, and it is a readability change only.Hardware: AMD Ryzen 9 9950X3D (16 cores, 32 threads), 62 GB, NVMe; RX 9070 XT (RADV) for the game runs.
Scene: none for the build rows; in game, Kokiri Forest from the repo save
kokiri-start(the same scene for every arm), 1920x1080, V-Sync off, frame generation off,HOS_FPS_MAX=1000.Method: from scratch (
rm -rf build-linux,LONGSHOT_OBJ_STORE=0 LONGSHOT_LIFT_STORE=0,dev_build.py build), one build at a time through the PC's queue. In game: one warm-up pair per build, then ten arms through the queue in the mirrored orderA B B A A B B A A B, 40 s each; fps from the control bridge's frame counter and CPU from/proc/<pid>/stat, medians and an exact permutation test on the medians. The build wall has a spread wider than the difference on this PC (an identical second build measured 3:50 against 3:19), so only the compile phase and the per-chunk times are quoted as timings.How to reproduce: the build command above, then
grep compact_emitin the log for the site counts andpython -I nx2/compact_emit.py <gen folder>to run the pass alone; the game arms arebuild-linux/ab/ab_seq.shwithRTA/RTBset to the two game folders.71b1edd8dabe2efacode_*.c)longshotThe store shape is the part that needed care: the emitter's
a64_vst16is not a plainmemcpy— it is_mm_loadu_si128+A64_OPAQUE+_mm_storeu_si128, one 16-byte store the compiler may not narrow, andwiden_vstoresexists because two 8-byte stores followed by a 16-byte read stall on store forwarding (a quarter of the hottest Castle function's samples). Every form here goes througha64_vset128/a64_vset64and keeps that, and a text check in the test fails if a rewrite ever emits two halves.For the record, where the rewrite does move a compile time: the one hot function that carries 15 % of these sites went from 325.6 s to 227.4 s while the function was still compiled whole — but once it is split into pieces (PR #1146) that function compiles in 27.4 s, and the rewrite moves it to 27.3 s. On the densest ordinary chunk it is 49.0 s → 48.7 s, and a 24 MB and a 33 MB chunk compile the same with the old and the new text. The superlinear cost was the giant flow graph, not the copies; this change is for reading, not for building.
Flags and settings
One lift-time switch is added; it is off by default, so a build rewrites the idioms and a reviewer who needs the emitter's own text can ask for it.
A64_NO_COMPACT1writes the emitter's text unchangednx2/compact_emit.pyruns after the lift; the lift key holds it, so switching it re-lifts1only for testingTesting
Tested:
python -I -B tests/run_all.pyon Linux — 587 passed, 17 failed, 74 skipped, the 17 also failing onmain @ 71b1edd8;tests/test_compact_emit.py(text and compiled exactness, 200 random contexts per shape) passes, andtests/test_lift_cleanup_error.pypasses with itsrundouble recognizing the new step. In the game: the compact build loaded the repo save and played it twice (2,472 and 2,530 draws a frame, no crash, no assertion; the 10 failed pipelines in that scene are the same on amainbuild), and a 4-arm A/B againstmain's build ran both without a fault.Not tested: a Windows build (the pass is Python and the emitted C is plain C; no Windows build was made), a scene where these sites dominate the frame, and a hot-list build (the pass is independent of
A64_HOT_FUNCS; the split branch's build was measured separately).Must be tested: none.
Risks
What could break: the rewrite writes C, so a wrong shape would change what the game computes — the compiled test covers every shape the pass rewrites against the build's own guest model, byte for byte, and any shape it does not recognize is left as the emitter wrote it. It is a lift step, so a mistake shows up as the generated C, not as a crash:
A64_NO_COMPACT=1writes the emitter's text unchanged, andgit revertis the other way back.Turning it off:
A64_NO_COMPACT=1at lift time (documented).Compatibility: nothing for players: the C is different, the game is not. A build with an existing translation re-lifts once, because the lift key now holds
nx2/compact_emit.py.Issues
None found in the tracker or the open pull requests that this replaces; it is a readability pass over the lift's output, independent of #1146 (the split) and #1142/#1140 (the flags).
Checklist
Everyone
type(scope): description) of the status line's type.main(71b1edd8).python -I -B tests/run_all.pypasses (the 17 that fail also fail onmain).third_party/untouched (a new folder with aPROVENANCE.mdinstead). The rewrite is outside it on purpose: the emitter is vendored and checked.When it applies
Assisted-by:trailer.A64_NO_COMPACTdefaults off, i.e. the rewrite on).main(there is nonx2/compact_emit.pythere, so the test's import fails onmain; the test is the pass's own).tests/that reads changed source updated in this PR, not weakened (tests/test_lift_cleanup_error.py, whoserundouble now knows the new step).Tested in game: Linux (RADV, RX 9070 XT, Ryzen 9 9950X3D), a no-hot-list build with the compact lift: Kokiri Forest from the repo save, two 40 s runs (2,472 and 2,530 draws a frame), no crash, no assertion, the same 10 failed pipelines as a main build in that scene; a 4-arm A/B against main's build ran both without a fault
WIP: perf(lift): the lane-permute idioms are written one line, not five memcpysto perf(lift): the lane-permute idioms are written one line, not five memcpysperf(lift): the lane-permute idioms are written one line, not five memcpysto WIP: perf(lift): the lane-permute idioms are written one line, not five memcpys26124afed76a0ac1d110WIP: perf(lift): the lane-permute idioms are written one line, not five memcpysto perf(lift): the lane-permute idioms are written one line, not five memcpysThanks to a review catch on the store shape, this PR's rewrite changed after its first push: the emitter's is not but + + — one opaque 16-byte store, which exists to guarantee (two 8-byte stores followed by a 16-byte read stall on store forwarding, a quarter of the hottest Castle function's samples). The first form wrote the two halves directly and so dropped that guarantee (byte-identical result, but the wrong store shape); the shipped form goes through /, which are the same opaque store, and now fails if the rewrite ever emits two halves again.\n\nMeasured after the fix (10 arms, 5 each, mirrored, from scratch):\n\n| | main @
71b1edd8| this PR @6a0ac1d1|\n|---|---|---|\n| | 543,806,840 B | 542,702,328 B (−1.1 MB, −0.20 %) |\n| lifted C | 1564.7 MB | 1556.9 MB (−0.5 %) |\n| fps, median of 5 | 88.20 | 88.40 (+0.23 %, p = 0.71) |\n| CPU ms a frame, median of 5 | 43.614 | 43.270 (−0.79 %, p = 0.29) |\n\nSo the verdict stands: no game-visible change, a smaller exe and a shorter C. For completeness, the pre-fix form measured +0.12 % fps (p = 1.00) and −0.42 % CPU (p = 0.24) on the same protocol, so the store-shape correction is not a measurable regression either way — it is the correct thing to emit, and it is now checked.Extended the pass to the rest of the family. One commit on top of
6a0ac1d1, no force-push:dabe2efaonperf/lift-compact-emit.Catalogue
Measured with a script over the kept gen folder (
build-linux/dev-work/sa/gen, 93code_*.c; it is the old pass'soutput, so every memcpy shape still in it is one the old pass did not rewrite):
{ uint8_t _s[16],_d[16]; memcpy(_s,vreg[S],16); memcpy(_d,vreg[D],16); memcpy(_d+OD,_s+OS,W); a64_vst16(vreg[D],_d); }{ uint8_t _d[16]; memcpy(_d,vreg[D],16); uint64_t _e=(uint64_t)0; memcpy(_d+OD,&_e,W); a64_vst16(vreg[D],_d); }... uint64_t _e=c->x[G]; memcpy(_d+OD,&_e,W); a64_vst16(...){ uint64_t _e=c->x[G]; ... memset(_d,0,16); for(_i<L) memcpy(_d+_i*ST,&_e,W); a64_vst16(...){ uint8_t _s[16]; ... uint64_t _e=0; memcpy(&_e,_s+OS,W); a64_vset64(vreg[D],_e); }{ uint8_t _s[16]; ... memcpy(&_e,_s+OS,W); c->x[G]=_e; }... uint64_t _r=(uint64_t)(int64_t)(intN_t)_e; [mask] c->x[G]=_r;{ a64_vset64(vreg[D],0); memcpy(&vreg[D][0],&c->x[G],W); }{ uint64_t _r=0; memcpy(&_r,&vreg[S][0],W); c->x[G]=_r; }{ uint8_t _s[16],_k[16],_t[16]; ...recomp_aes_sbox... }{ float _a,_b,_r; memcpy(&_a,&vreg[S][h],N); (void)_b; _r = <float op>; a64_vset64(vreg[D],0); memcpy(&vreg[D][h],&_r,N); }The last two rows are the shapes whose destination buffer is not
_dand whose two memcpys have a computationbetween them: the AES rounds (xor, a byte permutation and a table lookup in a loop) and the scalar float temporaries.
A bit permutation must not assume a float assignment is a bit copy, so they keep the emitter's text.
There are no
a64_vext*,a64_vzip*,a64_vuzp*ora64_vtrnsites in this game's C (grep over the gen: 0), and theonly helper-built vector write in the family is
a64_vset_lo, which appears with per-lane arithmetic (BIT, CLZ, ...),never with a permutation. Two shapes do not use
a64_vst16-- the FMOV pair -- and both are in the table above.Implemented
For every shape above, in one or two statements on the register's two
uint64_thalves, through the lift's own opaquewhole-register store. Examples of the generated text:
Hard rules kept: the whole-register store is one opaque 16-byte store (
a64_vset128/a64_vset64, never two assignmentsto
vreg[D][0]/[1]); every source half is read as a call argument or in a local before the store, soS == Dsites areright; a shape that cannot be expressed exactly (buffers read or written out of bounds, a source lane crossing the
source register's two halves, a DUP (general) whose stride is not its element size) is left byte for byte as the emitter
wrote it. Idempotent, and
A64_NO_COMPACT=1still writes the emitter's text unchanged. TheA64_NO_COMPACTrow indocs/development.mdis updated with the new scope and counts.Counts and size
On the shapes added here alone: C 1556.9 MB -> 1554.7 MB (-2.15 MB, 81,567 sites, 26 B a site on average).
Build and exe, from scratch, one at a time (
rm -rf build-linux, the command in the task)71b1edd86a0ac1d1The first identical build of this commit measured 3:50 (compile 162 s) and the exe is the same 542,969,736 B, so the
build wall moves between runs on this PC by more than the difference; a 24 MB chunk compiles in 53.2/54.6 s with the
old pass's text and 52.9/52.9 s with this one, and a 33 MB chunk in 33.3/33.5 s vs 33.5/33.5 s. The exe is +267 KB over
the two shapes' build and still 837 KB under main: reported, not claimed as a win.
Tests
python -I -B tests/test_compact_emit.py --cc clang-- ok. 886 shape variants: every shape over each size, lanecount and byte offset the game's C has, plus the edges; for each a text check (rewritten, exactly one 16-byte store,
other text untouched, idempotent) and a compiled check that runs the old idiom and the new statement against the
build's guest model from 200 random contexts and compares every byte of every vector register and every general
register -- 0 cases differ. (Mutation check: dropping the element masks makes 1,798 of those contexts differ, so
the check bites.)
python -I -B tests/run_all.py-- 587 passed, 17 failed (the known ones), 74 skipped in 58 s.In game: 5+5 A/B against main @
71b1edd8Harness:
ab_pair.shwarm-up thenab_seq.sh(mirrored order A B B A A B B A A B, 40 s samples), baselinedsh-main-bt/build-linux/base, this buildbuild-linux/perm2. Scene: the repo savekokiri-start(Kokiri Forest,the save's load point), no teleport, the same
options.iniand save on both sides.71b1edd8dabe2efaNo change within the spread. This is a readability change; the medians differ by less than one arm of either side
and neither is significant.
What I would put in the description
perf(lift): the lane-permute idioms are written one line, not five memcpysto WIP: perf(lift): the lane-permute idioms are written one line, not five memcpysView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.