WIP: fix(vk): the program cache's stamp holds only the switches the translation reads, in name order #2144
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2144
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:fix/translator-stamp-env"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Setting
RX_VK_MESH_CONV, the renderer'sHOS_ABswitch, at any value, made the game drop its whole shader program cache. It then translated every program again and prepared the shaders from cold: on this PC, 214 s to the first frame instead of 6 s. Unsetting it did the same again. The program cache's stamp took every variable beginning withRX_VK_MESH, and with it seven switches that change no translated program:RX_VK_MESH_ARGS,_CONV,_COUNT_WORD,_FIRST_WORDand_IND_COUNT_WORD;_LANESand_OPT.The stamp also hashed the switches in the environment's order. That order comes from
longshot_env.txt's lines and from_putenv, which appends, so the same switches in another order dropped every cache too.The stamp now holds the nine
RX_VK_MESH_switches the translation reads, by name, and hashes the entries in name order. The list moves out ofvk_shader.cppintogpu/vk/translator_env.h. No translator stamp covers that file (nx2/translator_files.py), so a later change to the list moves the stamp only for runs that set a name it takes or leaves, not every player's cache.tests/test_program_stamp_knobs.pychecks the list both ways:EDEN_reads throughEnvSwitchand the nameskey_listedreads, which it did not see before;RX_VK_GPLare not held.It also compiles the header alone.
On main: start the game with
RX_VK_MESH_CONV=1. The log saysvk: program cache …: 0 program keys keptand the shaders are prepared before the first frame. Start it again without the switch, and the same happens.Read first:
gpu/vk/translator_env.h,translator_stampingpu/vk/vk_shader.cpp,tests/test_program_stamp_knobs.py.Rebased onto main
a2dc53859. Main's #2103 had since leftRX_VK_GP_TABLEout of the loop this PR moves: its value reaches the stamp asgraphics_table_loads, in the flags, and by its name too, setting it to its default prepared every shader again. It istranslator_env.h'sheld_elsewherenow, and the check holds each such name to what intranslator_stampholds it (HELD_BY). The runs and checks below were made before the rebase, at5b146577f; they have not run again.Overlaps:
RX_VK_MESH_FB_LAYOUT, which stays held by name: its description's "theRX_VK_MESHsweep" is this list now.RX_VK_NATIVE_NARROWis held by theRX_VK_NATIVEfamily.translator_stamp's ABI words, not these lines.sizeof(Program), another of its inputs; not this.vk_progcache.cpp, not by these lines. A stamp that moves less often keeps its store's programs longer.Testing
Tested: Windows 11, RTX 3080 (NVIDIA 621.14), from the
kokiri-startsave with Skip title screen, through the PC's job queue. Two runtimes over main971f77f52's base, each prepared at its making: main's alone, and this branch's pack with the switch table PR's change beside it (fix/switch-table-misses, a separate code path). Each run went to the 900th present, in the order: without the switch,RX_VK_MESH_CONV=1, without it again (Measurements).RX_VK_MESH_CONV=1and then going back to the default each gave0 program keys kept, 0 KBand a preparation before the first frame.47867 program keys kept, 145508 KBin all three, with no preparation.0 program keys kept, then 47,867 programs in about 3 minutes.tests/test_program_stamp_knobs.pypasses: 19 switches held, 2 log only, none of the 8. WithRX_VK_MESHput back as a family, it fails and names the seven.test_prepared_stamps,test_mesh_fallback_layout(a Linux check) andtest_switch_tableread the list from the header now.vk_shader.cppcompiles (syntax, llvm-mingw).tests/run_all.py, llvm-mingw, both PRs' commits together): 770 passed, 108 failed, 59 skipped. Main971f77f52on this PC: 767 passed, 112 failed. No check failed here that passed on that main. Main's #2171 has since repaired many of its checks, so this comparison is against971f77f52only; the suite has not run on current main.1101b6127e's own push too (the guide's review hashes for menu files main changed); not this PR's.Still to test:
RX_VK_MESH_CONV=1unset and set, expecting the build to pass and both starts to reach the title (@Beanstalk: the farm warms each start apart, so the cache kept across starts is Windows' result above)Measurements
Verdict: a run that sets one of the seven keeps the program cache: 6 s to the first frame instead of 214 s, and back to the default 13 s instead of 203 s.
Setup: Windows 11, RTX 3080 (NVIDIA 621.14), 24 preparation workers. Both runtimes were over the base of main
971f77f52, each prepared at its making. Each run went from thekokiri-startsave to the 900th present. The column is the time from launch to the first frame, and the program cache's line in the log.971f77f525b146577fRX_VK_MESH_CONV=1No switch is added and no default changes:
translator_env.hnames nine existingRX_VK_MESH_switches (RX_VK_MESH_FB_LAYOUT,_FIRST,_GLOBAL_ATTRS,_NOCULL,_PAGES,_SHADERS,_SHARED_WARP,_SHARED_WARP_CHECK,_STRICT_FMUL) one by one, where the stamp took them by their family before.Risks
vk_shader.cpp, a translator source, so itsLONGSHOT_SHADER_STAMPmoves once. At the first start after updating, every player translates and prepares again, as with any release that touches the translator; the driver's pipelines and the notes carry over (#1927). Later changes to the list move no player's stamp.RX_VK_MESH_LANESandRX_VK_MESH_OPTshape mesh pipelines but no program. Flipping one no longer prepares again from cold; the pipelines it changes are compiled at their first draw. No record holds them yet, as for the other pipeline-shaping switches (RX_VK_NOOPT,RX_VK_COMPUTE_PIN).RX_VK_MESH_translator switch must be named intranslator_env.h; the check fails otherwise.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.tests/that reads changed source is updated, not weakened.Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Tested in game: Windows 11, RTX 3080, NVIDIA 621.14, from the kokiri-start save to the 900th present, over main 971f77f52's base. Runs in the order without the switch, RX_VK_MESH_CONV=1, without it again: main kept 47,867 programs, then 0 and 0, preparing from cold twice (214 s and 203 s to the first frame); this PR kept 47,867 in all three (68 s, 6 s, 13 s).
Possibly related, for context:
@Beanstalk Linux, any GPU: build this PR and start the game three times: without switches, with RX_VK_MESH_CONV=1, and without it again, each to the title or the first save. Expecting the build to pass,
vk: program cache …: N program keys keptwith the same N at all three starts once the first start has prepared, and no shader preparation at the second and third start.Beanstalk test farm
Commit
b5bf87d· approved by nobody yet · dashboardA member of
BigBeanscan review this commit and comment@Beanstalkto test it.5b146577fbb5bf87d0b1@dyst — this one looks ready for a second pair of eyes: it is clean on current main
a2dc53859, the switch list is held by name and in name order, and the Windows numbers (47,867 programs kept, 6 s vs 214 s to the first frame) are the right kind of evidence for a high-priority cache bug.Two notes so we don't duplicate anything:
For the record, #1906 is the object-cache report, not this stamp; and #1542's commit 4 edits translator_stamp's ABI words, not this list — so neither covers #2144. Ping us if you want the extra review or the run.
Read of
b5bf87d0(the rebase plusheld_elsewhere): correctness and what a Linux run still needs. The core fix is right: the nine held names are exactly the sixteenRX_VK_MESH_*rows minus the seven draw/pipeline ones,entries()sorts by name and still dropsswitches::find_gonenames (gpu/vk/translator_env.h:64-76), andheld()/find_goneare plainstrncmp/strcmp._environis used exactly as main used it (gpu/vk/vk_shader.cpp:1473), so nothing in the diff is platform-specific.One correctness gap in
held_elsewhere.HELD_BYasserts thatgraphics_table_loadsholdsRX_VK_GP_TABLE(tests/test_program_stamp_knobs.py:48), and the header leaves the name out on that strength (gpu/vk/translator_env.h:41-43). But the translator reads the name itself too:gpu/vk/eden/shader_recompiler/frontend/maxwell/translate_program.cpp:202isstatic const bool table_check{std::getenv("RX_VK_GP_TABLE") != nullptr && std::atoi(...) == 2};, and with it true the program gets a storage descriptor at c0+0xff0 (:204-213). The flag only holdstable_loads_wanted(...)(gpu/vk/vk_coopmat.cpp:672-675), which returns 0 whenever the device has no host-memory import or no buffer device addresses — the case the code logs for at:698-700. There,RX_VK_GP_TABLE=2and unset both put0in the flags atvk_shader.cpp:1429, yettable_checkis true for one and false for the other, so they store differentProgram::Info.storage_buffers_descriptorsunder one stamp. main skipped the name with a!strncmptoo, so the gap is not introduced here; what is new is the test asserting the premise. Fix: foldtable_checkinto the flags besidegraphics_table_loads, or leave the name in the stamp (it is a diagnostic, so a cold translate when it is set costs only that run).The reference page now under-describes the list.
docs/dev/reference/switches.md:79-83says the stamp holds "families ... and names one by one" and that the check keeps "a draw or pipeline switch" out. It does not mentionheld_elsewhere, so a reader adding anRX_VK_GP_*switch believes the family holds it whenRX_VK_GP_TABLEis the exception; and the "draw or pipeline switch must not" half is only checked for the eightNOT_HELDnames. A sentence namingheld_elsewhere(and the eight, as today's check) would make the page true.Linux start: not verifiable from code, and not run at this head. Nothing in the diff is platform-specific (above), and the header ships:
scripts/linux/appimage.sh:78copiesgpu/vk/whole, so no packaging list needs it. But the description's suite and runs are from5b146577f, before the rebase, andheld_elsewherewith its fixture case has run nowhere. A Linux run would need:python -I -B tests/test_program_stamp_knobs.py --cxx g++atb5bf87d0: the fixture compilestranslator_env.halone, and with glibc's g++ that is the only Linux-only compile surface in the diff.RX_VK_MESH_CONV=1, unset — reading thevk: program cache … N program keys keptline each time. The first start after the update is the cold one (the stamp moves once); the second and third are the ones that prove the fix.The Test Farm warms each start apart, so it can prove the build and that both settings reach the title, not that the cache was kept. That half is a single machine's run (the job queue), which is what the Windows table already shows and Linux still does not.
Read-only review — from the diff at the head, main
a2dc5385and the sources. No checks or game run from me (the PC is held); the PR is WIP, so this is a comment, not an approval.Scope, today. I found no missed translation input, and the seven draw/pipeline switches are correctly out.
RX_VK_GP_TABLEis out by name and its value is inflags(graphics_table_loads);RX_VK_MISS_LOGandRX_VK_BLOCK_PROBEare correctly held — they change the emitted program (the c0+0xff0 descriptor, 16 KB slices). The other inputs are held by value where the name is not (WARP64,NO_F16,FRAG_SHUFFLE_FALLBACK,HANDLE_TABLES,SIM_TURF_POOLS,FORCE_MESH_FALLBACKand the MMA/NaN/invariant values).The gap I would close before ready. The Risk says "A new
RX_VK_MESH_translator switch must be named intranslator_env.h; the check fails otherwise." For 7 of the 9 names it would not fail:tests/test_program_stamp_knobs.pyscansgpu/vk/eden/shader_recompiler/**and only the two names inSHADER_SWITCHESfromvk_shader.cpp(pluskey_listed)._PAGES,_GLOBAL_ATTRS,_SHARED_WARPand_SHARED_WARP_CHECKare read ingpu/vk/vk_coopmat.cpp:708-716;_SHADERSingpu/vk/vk_main.cpp:416;_STRICT_FMULthroughgpu/vk/device_policy.h:741;_FB_LAYOUTingpu/vk/vk_shader.cpp:1280. Only_FIRST(spirv_emit_context.cpp:580) and_NOCULL(vk_shader.cpp) are seen;_FB_LAYOUThas its own presence check intest_mesh_fallback_layout.py.RX_VK_MESH_*that changes the translation but is read invk_coopmat.cpp,vk_main.cpporvk_draw.cpppasses this check and leaves a stale cache — the case the old family caught. Scanninggpu/vkforRX_VK_MESH_(or allRX_VK_/EDEN_) reads and requiring each to be in the list orNOT_HELDwould close it, and would also prove the nine are reads.Smaller, no stale cache. Five of the nine (
_PAGES,_GLOBAL_ATTRS,_SHARED_WARP,_SHARED_WARP_CHECK,_STRICT_FMUL) already reach the stamp inflags; with the name held too, setting one to the value the device already uses still drops the whole cache — the needless drop this PR removes for_CONV.held_elsewhere/HELD_BYis the mechanism (it is whyGP_TABLEis there).Checked and sound. Name order:
entries()sorts on the name before=, andstd::string::compareis byte-wise, so it is locale-independent; the fixture'sEDEN… < RX_VK…order matches, and the comparator is a strict weak order (only duplicateenvironnames are equivalent). Old stamp: a mismatch drops the cache whole, and the stamp moves once for every player becausevk_shader.cppis a translator source, so the name order adds no second move.translator_env.his not innx2/translator_files.py'sDIRS/FILES, so later list edits move no cache. #1542 commit 4 edits other lines of the same function — no conflict, but if both land close together it is two re-preparations, not one (one line in Risks); #2177 is the test runner; #2199 sees one moregpu/vk/header invk_shader.cpp's depfile, one recompile. Linux: no locale, no Vulkan, no pointer-size assumption; the header compiles alone under-std=c++20 -Wall -Werror, as the fixture does. A Linux start proves only the end-to-end "cache kept across starts" (theN program keys keptline), not the stamp itself.Evidence on the head. The description says the runs and checks were made at
5b146577fand "have not run again" after the rebase toa2dc5385, and the rebase changed the very check (HELD_BY/graphics_table_loads).python -I -B tests/test_program_stamp_knobs.pyis game-free and takes seconds;test_mesh_fallback_layout,test_prepared_stampsandtest_switch_tableare the other changed ones. Measurements are on971f77f52; the status line already warns about that.Correction to my review just above (07:14). It says
RX_VK_GP_TABLEis held bygraphics_table_loads, and that I found no missed translation input. The review at 07:10 is right and mine is wrong on that point: the translator also reads the name itself (gpu/vk/eden/shader_recompiler/frontend/maxwell/translate_program.cpp:202), and on a device without host-memory import or buffer device addressesgraphics_table_loadsis 0 for bothRX_VK_GP_TABLE=2and unset whiletable_checkdiffers, so two different programs share one stamp.My MESH-scope finding stands on its own (the knob test sees only 2 of the 9
RX_VK_MESH_*reads); theheld_elsewheregap at 07:10 is the stronger one. Please read the two together.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.