fix(vk): shader programs over 1 MiB, or with a stray branch after their end, are no longer left out #427
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
beanstalk: approved @ 1697603
beanstalk: approved @ 2c5a62e
beanstalk: approved @ 4fd82b7
beanstalk: approved @ 64d6d09
beanstalk: approved @ 6f91fae
beanstalk: approved @ 70a78f9
beanstalk: approved @ 8d361dc
beanstalk: approved @ 9d8356e
beanstalk: approved @ abf2227
beanstalk: approved @ d82f202
beanstalk: approved @ e5f1e40
beanstalk: approved @ ee8ab5f
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!427
Loading…
Reference in a new issue
No description provided.
Delete branch "mooki/Longshot:fix/shader-branch-past-end"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: fix · Risk: medium · Tested on: macOS 26.7.1, Apple M4 Max (game-free checks on arm64 and x86_64; a macOS game run with this PR's commits, run E9a) · Needs: Windows or Linux, NVIDIA GPU
Why
A shader program the game uses is left out on every platform: the shader reader refuses it before translating it.
This is for players on every platform and GPU. Compute shader program
79116b290d46542fis never translated, on anyrenderer path. Its stage is refused before translation for "a branch outside the program". Two causes are fixed here:
The GPU has no instruction count of its own. The limit is now the largest code area the renderer reads, 16 MiB, and a
lowered branch beyond Maxwell's 8 MiB relative reach takes its absolute form.
How to reproduce
Seen on: macOS 26.7.1, Apple M4 Max, MoltenVK: run 18, an integrated build of the macOS port with this PR's first
commit (its
longshot_log.txt, below). The capture and the size limits are the same code on every OS and renderer path:the changed code has no platform branch. Game-free: the new test against main 6c9edffe's files on this Mac.
Settings and switches: defaults.
Steps: start the game through the shader preparation and look for
79116b290d46542finlongshot_log.txt.Game-free: from this branch's checkout,
python -I -B tests/test_capture_past_end.py --cxx clang++ --source <a checkout of main>.Expected: the program translated: the line for its key,
vk: NVN2 program 79116b290d46542f (stage 5) …, with nonot translatedline.Actual: it is refused before translation for "a branch outside the program". Run 18's line:
The test against main 6c9edffe's files: exit 1, 33 of its 39 checks and all 5 capture loops fail.
Why main refuses it:
The capture reads past a program's end.
parse_code(gpu/vk/vk_shader.cpp) reads an NVN2 stage's Ampereinstructions up to the code length (+0x38). After each one,
ProgramCaptureState::Observedecides whether theprogram has ended. Its end rules are narrower than the lowering's, which decides the real end (
finish,source_size):lower_to_maxwell.h, "the closingself-branch"), which the capture takes as the end only after a computed branch;
So the capture can read past the end. A word there that decodes as a branch with an impossible target (negative,
misaligned, or past the largest program) refused the whole program. Nothing reaches such a word.
Programs over 1 MiB were refused. The capture refused any target at or past 1 MiB (
program_capture.h,MaximumInstructions = 65536). So did the lowering (code.size() > 65536, "invalid native program size") and thenative address map (
count > 65536). Run 18, an integrated build with the first commit of this PR, shows that thisis 79116b29's case (the line above). 0x104680 bytes is 66,664 instructions.
Eden's
Maxwell3D::Regs::Pipeline::offset(src/video_core/engines/maxwell_3d.h) andKeplerCompute'sLaunchParams::program_start(kepler_compute.h). Its relative branches reach 24 bits (±8 MiB) and its absoluteones 32 bits:
frontend/maxwell/instruction.h,BitField<20, 24, s64> offsetandBitField<20, 32, u64> absolute. An Ampere BRA's offset has 50 bits, as the capture decodes it.TryFindSize,MAXIMUM_SIZE = 0x100000,shader_environment.cpp). When that fails they build the control-flow graph to get thereal size ("Slow path … Build a control flow graph to get the real shader size",
ShaderCache::MakeShaderInfo,shader_cache.cpp). Ryujinx sizes a program by its furthest reachable block (Translator.cs:maxEndAddress).What changed
The capture loop now ends before a word it reads as a branch outside every program, and the lowering decides whether the
code before that word stands on its own. A program's size limit is now the largest code area the renderer reads, 16 MiB,
instead of 1 MiB, and a lowered branch beyond Maxwell's 8 MiB relative reach takes its absolute form.
Read first:
program_address_map.h(the limit) and thefinishlambda inlower_to_maxwell.h(the absolute forms),then
parse_code's capture loop. The suyu copies and the other four loops are the same change.The capture loop ends before a word it reads as a branch outside every program. This is in
parse_code, and inthe four other copies of the loop:
gpu/render/rx_shader.cppread_nvn_code;gpu/shaderbatch.cppbuild_pkgandnx2/bake_ds/bake_ds.cpp;gpu/shadertool.cpp.The lowering then decides:
vk: NVN2 program K (stage S) ends at +E, before the word at +X that would branch outside it (L code bytes);main's text and adds the code sizeand the lowering's reason.
One limit for a program's size,
MaximumNativeInstructions(frontend/ampere/program_address_map.h): 16 MiB ÷ 16= 1,048,576 instructions. That is as many as fit in the largest code area the renderer reads (
control_endrefusesan area that ends past 16 MiB). The capture's "outside" test, the lowering's size check and the address map's count
check all use it, in
gpu/vk/edenand in the WebGPU renderer'sgpu/third_party/suyucopy.lower_to_maxwell.hnowincludes
program_address_map.h.Far branches in the lowering (
LowerToMaxwellWith), both copies. The lowered Maxwell code holds each branch in the24-bit field at bits 20-43. A target more than 8 MiB of lowered code away (about 786,000 lowered instructions) was
refused ("lowered branch distance", "native computed branch range"). It now takes the absolute form, which the
frontend reads through
IsAbsoluteJump(control_flow.cpp):AnalyzeBRX's target is the table value itself, as with BRX.The predicate and the flow test stay as they were. Branches within reach are emitted exactly as before.
tests/test_capture_past_end.py(new) compilesvk_shader.cpp'shash_bytes,Code,code_keyandparse_codewith the recompiler's headers (--sourceselects a checkout), over packages it builds:BSSY before the start, and a CALL between two instructions. Each runs in compute and in graphics, and must translate
to the same code as the loop alone, with one log line;
mainparses: unchanged;@P0 BRAat instruction 16404 to its EXIT, past 1 MiB, and a@P1 BRAback. Ittranslates, each BRA lands on its target, and the address map finds both ends;
apart: the branches become JMPs and the call a JCAL, each holding its target's address, the predicate kept;
Unchanged:
third_party/;native; aprogram under 8 MiB of lowered code emits the same words.
Sources read:
eden-emulator/mirror@4a2c9234:shader_environment.cpp,shader_cache.cpp,maxwell/control_flow.cppandinstruction.h,engines/maxwell_3d.handkepler_compute.h;alula/Ryujinx@4930dced:Decoders/Decoder.csandTranslation/Translator.cs.GitHub's yuzu, suyu and Ryujinx repositories are blocked. Written from LS's own source; nothing was copied.
Measurements
Verdict: fixed in the game-free test, not yet seen in the game. The new test fails on main (33 of its 39 checks and
all 5 capture loops) and passes here (39 of 39, 5 of 5); the previous head of this PR failed its 6 new checks. Among them
is a program of 79116b29's size: on the previous head it is refused with run 18's line, word for word as far as that line
was quoted. The full game-free suite changes only by the new test and the load-sensitive
test_gpu_hang_watch. No gamerun of this head yet.
Commits: main
6c9edffe5d(2026-11-14); this PRfix/shader-branch-past-end@32d7759026(rebased fromd2111b26; previous head3c1b5551, on main5e072fc8).Hardware: Apple M4 Max, 16 cores, 64 GiB, macOS 26.7.1 (25G241); Apple clang 21. No GPU or driver involved.
Scene: none. Game-free test, object compiles and the game-free suite; the game line is run 18's.
Method: the new test at this head, and with
--sourceonmain's and the previous head's files (git archive);vk_shader.cppobjects with the renderer's flags (-mmacosx-version-min=14.4, arm64 and x86_64);tests/run_all.py,the whole suite, compared by test name with a
mainrun on the same Mac.How to reproduce:
python -I -B tests/test_capture_past_end.py --cxx clang++, then the same with--source <a checkout of main>.6c9edffe32d77590tests/test_capture_past_end.pygoes outside the programvk_shader.cppobject, arm64 / x86_64environdeclaration, whichmainlacks on Darwin)tests/run_all.py, 6 at once)test_capture_past_endnew and passing;test_gpu_hang_watchload-sensitive, under Testing)The previous head of this PR,
3c1b5551(on main5e072fc8), with the same test (--source):3c1b5551tests/test_capture_past_end.pyvk_shader.cppobject, arm64 / x86_64environdeclaration)The whole suite's summary lines, on the same Mac (the runner labels macOS
linux):Screenshots: none; the game run's evidence is its log lines (Testing).
Flags and settings
None.
Testing
Tested:
32d77590, and with--sourceon main 6c9edffe's files, whose 40 FAIL and 6 PASS lines areword for word those of main
5e072fc8before the rebase; and, before the rebase, onmacos/integrated1c53cff1 withthis PR's second commit applied (exit 0).
vk_shader.cppobjects on arm64 and x86_64, at32d77590and on main6c9edffe(with the compile-onlyenvirondeclaration on both), and before the rebase on that integrated tree.
(exit 0, at
32d77590:limit 1048576, capture 1, lowered 1 (1 words)).tests/run_all.py, 6 at once, on the same Mac (the runner labels macOSlinux), at32d77590andmain
6c9edffe:test_capture_past_end, absent onmain, passes here;
test_gpu_hang_watch: pass on main, FAIL here.test_gpu_hang_watchis load-sensitive (a time bound, 3.8 s against its 3.4 s; load averages at the run's start14.38 22.01 21.80) and is accepted without a rerun. This PR does not touch
vk_replay.cpp.requests, some are checks main added or changed since
5e072fc8):test_appimage_package_retry,test_appimage_player_build,test_collect_wide,test_control,test_cpm_discovery,test_crash_dialog,test_crash_record,test_env_on,test_game_layout,test_lift_cleanup_error,test_linux_preflight,test_longshot_ctl,test_low_memory_notice,test_make_game_cache,test_mem_gate,test_mem_gate_children,test_mirror_trim,test_oot_state_tap,test_pcache_header_read,test_pipeline_cache_file,test_pipeline_cache_save,test_pipeline_play_cache,test_pipeline_tour,test_prof_symbols,test_program_cache_cleanup,test_program_cache_one_variant,test_readback_late,test_report_run,test_split_giants,test_vk_pgo,test_vram_budget_row,test_walk_key.recorder_sync_test.exe-2026-11-14-180744.ips).
tests/test_bake_mimo.py, which builds the recompiler,gpu/shaderbatch.cppand the eden Ampere headers, and runsthem;
tests/test_lower_end_at_closing_branch.py,tests/test_bake_ds.py,tests/test_log_levels.pyandtests/test_compute_local_memory.py.aa95d35f, at33c427bb(32d77590before the rebases ontoe090cc48andaa95d35f):Sweep3's checks of this head (the C/C++ it changes compiled, and its own tests run, on arm64 and x86_64 where they
apply) gave the same results as before;
python -I -B tests/run_all.pyon wave S1's merge11a5b4e7, which carriesthis head: 568/22/17 (passed/failed/skipped) against main aa95d35f's 543/33/17 on this Mac, no new failure
(
reports/2026-11-15-rebase-sweep-3.md).5f1c612c,86f13743, carried in the macOS integration I4178ec1c47), runE9a: arm64, the field-dawn save, a 1,197 s hold, clean end (
reports/2026-11-15-mac-perf-E9.md).and 46501:
vk: slow compile: compute 79116b290d46542f …, 26.2 s: made (0)). There is nonot translatedline for it.not translatedlines in that run area network left out on this card, as designed on non-NVIDIA cards.63fc3d2calso carries both commits and ended clean.Not tested:
platform; the request below covers NVIDIA, where the program reaches its driver. The line for its key settles it:
vk: NVN2 program 79116b290d46542f (stage 5) …with nonot translatedline: translated;[trace] … a network left out on this card, as designed.lowered words. The frontend's handling was read, not run:
control_flow.cpptakes JMP/JCAL/JMX throughIsAbsoluteJump, and a block ends before its branch. A JMX needs a jump table, which no test builds. A program needsmore than 8 MiB of lowered code for any of them to appear.
rx_shader.cpp(needs wgpu-native's headers),bake_ds.cpp(fmt and boost) andshadertool.cpp(MSVC with the Eglantine clone). Their edits mirrorvk_shader.cpp's.Must be tested:
33c427bb) and mainaa95d35fthe same way,start each once through the shader preparation to the title and search
longshot_log.txtfor79116b290d46542f,expecting no
not translatedline for it on this PR, where main logsvk: program 79116b290d46542f not translated: … goes outside the program(by hand: test request T-C78E).Risks
What could break:
mainrefuses (a stray word, or a program over 1 MiB) and programs with morethan 8 MiB of lowered code. Others get the same words. A program
mainleft out now reaches the driver, which no gamerun of this head has shown yet.
inside the program. An indirect branch's targets are checked when it translates.
mainor with the other macOS-port PRs rebased on6c9edffe(git merge-treeof thenew heads, 19:49 CST).
Turning it off: revert the commits. There is no switch.
Compatibility:
vk_shader.cppand the eden Ampere headers (gpu/vk/eden) are hashed intoLONGSHOT_SHADER_STAMP(the translator stamp's files, listed in
nx2/translator_files.pysince main's #370), so the program cache, the preparerecord and the Vulkan pipeline cache files are rebuilt once: every player translates every program again at the first
start after updating. The WebGPU renderer's
gpu/third_party/suyucopy is not in the stamp. Saves, the options file andthe vk ABI are unchanged.
Issues
See #420 (found in the macOS port's start runs; this fix is for every platform).
Checklist
Everyone
type(scope): description) of the status line's type.main: onaa95d35f(#761, 2026-11-15), head33c427bb, without a conflict; this PR's owncommits, old → new:
71ed77da→cf23d72e,32d77590→33c427bb; the commit names elsewhere in this descriptionare from before the rebase.
python -I -B tests/run_all.pypasses (any failure that also fails onmainis named). On macOS: 531 passed, 17skipped, and 32 failures that also fail on main
6c9edffeon this Mac, named under Testing, plus the load-sensitivetest_gpu_hang_watchtiming (Testing). After the rebase onto mainaa95d35f(head33c427bb): wave S1's merge11a5b4e7, 568/22/17 against main aa95d35f's 543/33/17, no new failure.third_party/untouched (a new folder with aPROVENANCE.mdinstead).gpu/third_party/suyu's Ampere headers areLS's own lowering, changed in step with
gpu/vk/eden's (as earlier LS commits did); top-levelthird_party/isuntouched.
When it applies
Assisted-by:/Co-Authored-By:trailers.mainand passing here.tests/that reads changed source updated in this PR, not weakened. None needed a change: thechecks that read the changed files (
tests/test_bake_mimo.py,tests/test_bake_ds.py,tests/test_lower_end_at_closing_branch.py,tests/test_log_levels.py,tests/test_compute_local_memory.py) passunchanged.
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 [email protected]
WIP: fix(vk): a word after a shader program's end that reads as a stray branch no longer leaves the program outto WIP: fix(vk): shader programs over 1 MiB, or with a stray branch after their end, are no longer left outIndependent Windows baseline evidence for your shader target: main9c9d69ed, RTX5090 driver617.42, Windows11 build26300, matched API22 build. Preparation logs program79116b290d46542f not translated at NVN stage5 because a branch is outside the program; the summary reports38809 programs and1 failed. This strongly matches the summary failure, although the log does not explicitly pair them. Preparation exits0 and no noted/compute precompile pipeline is reported not made. I have not run this branch, so this is baseline reproduction only; no gameplay-effect or after-fix claim.
d2111b26a532d775902632d775902619fd58215e19fd58215e33c427bb8dWIP: fix(vk): shader programs over 1 MiB, or with a stray branch after their end, are no longer left outto fix(vk): shader programs over 1 MiB, or with a stray branch after their end, are no longer left outApproving on code. This is a code-only review: I read the diff at
33c427bband ran nothing here.Stray branch after the end. The capture now stops before the word and leaves it out of
native. Nothing past the program gets read. The lowering only walksnative.finishrefuses any direct branch or call aimed past the instruction it ends at. The address map is built fromsource_instruction_offsets, which runs only up to that end, so an indirect target landing on or past the dropped word throws inTranslateProgramAddress; it can't index out of bounds.Env::ReadInstructionis bounds-checked. An emptynative(the word at instruction 0) fails as "invalid native program size". The~0usentinel can't collide, becauseistays under 1M. Programsmainalready parsed get the samenative: they never hitInvalidTarget, and the capture's end rules are unchanged.The 16 MiB limit. It follows from
control_end's 16 MiB area cap, sonativeis at most 16 MiB and the address map at most 8 MiB. That is per translation, not per frame. I found no fixed buffer or size field on the program path that the larger size could overflow. The newwhy[256]is snprintf'd, so a long message is cut short, not overrun.code_keyhashes the whole area and the control section as before, so the program cache keys don't change. The translator stamp rebuilds the caches once, as the description says.Far branches. I checked these against the frontend.
address_ofis in the same address space asLocation: it includes the 0x50 header, and graphics and compute both start the CFG at 0x50. So JMP/JCAL withaddress_of(target)in bits 20-51 lands where the BRA would. JMX with offset -8 givesAnalyzeBRXthe table value plus 0 plus 8, the same target as the BRX form, andbranch_offsetcomes out as 0. JMP and JCAL end their block before the branch, as BRA and CAL do, so their throwing translator visitors are never reached. One non-blocking note is inline.monsterlag's Windows RTX 5090 baseline (main
9c9d69ed:79116b290d46542f, stage 5, "branch outside the program", 1 of 38,809 programs failed) is the program this PR targets. Onmainthat refusal comes from the real@P0 BRAat instruction 16404 to a target past 1 MiB. Run 18 shows the first commit alone doesn't clear it. The second commit raises the limit, which does. On macOS, run E9a with both commits translated and built it (two variants, about 1.23M SPIR-V words, 26.2 s each), with nonot translatedline.Before merge: it still needs one game run on Windows or Linux with an NVIDIA card, as the description's Must be tested says (test request T-C78E). That run should show no
not translatedline for79116b290d46542f, and that the driver gets through compiling a program this large without stalling preparation. The macOS run shows the translation is right; only NVIDIA hardware shows this program reaching its driver.@ -3329,0 +3338,4 @@// (longshot) BRX goes to the table's value plus its own address plus its offset, which reaches back only 8 MiB// (24 bits). Past that, JMX: the value plus the offset alone, -8 (the frontend's AnalyzeBRX adds the 8 back)const s64 back = -s64{address_of(static_cast<u32>(result.instructions.size()))} - 8;const bool absolute = back < -0x800000;Non-blocking. The JMX form reaches the frontend correctly, but in practice only through the IMNMX and MOV32I table bounds. The measured-table path in
indirect_branch_table_track.cpp(table_count) accepts an entry only ifrel < 0x40000andtarget >= block_begin.Offset(), and the second compares a native table value with a lowered block address. At more than 8 MiB of lowered code,block_beginis past 8 MiB, so no entry qualifies. Tracking then throws "Failed to track indirect branch" and the program is left out, as before this PR: a safe refusal, not a wrong translation. It's worth a line in Not tested or a follow-up issue so a far jump table there isn't taken for covered.