feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing #737
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!737
Loading…
Reference in a new issue
No description provided.
Delete branch "dyst/Longshot:feat/shader-cache-store"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: feat · Risk: medium · Tested on: Windows 11, RTX 3080, in the game (two game folders, one store) and game-free · Needs: a Linux run in the game (the flock lock, the AppImage's longshot-launch)
Why
Every game folder prepares its shaders on its own, about 25 minutes with Compile shaders All on a fast PC, even when another folder on the same PC translated the same shaders with the same translator a minute before: a stable and a nightly side by side, a rebuild into a new folder, each
dev_packs.pyruntime (which also carries a 10 GB copy of the caches). The caches are already stamped by what they are good for (program_cache_stamp), so a store keyed by that stamp lets every folder with the same translation use them in place.The design was agreed with the maintainers, including moving these files out of the game folder, where #321 put them. It complements MagicBeanDev/LongshotLauncher#57 (an update in place keeps its
cache/anyway) and replaces copying caches between folders (MagicBeanDev/LongshotLauncher#78). It mirrors the object and lift stores (#371, #372). No Longshot issue fits (the tracker searched).What changed
With
LONGSHOT_SHADER_CACHE=<folder>(or--shader-cache <folder>), the game keeps its shader caches in<folder>/<stamp>/for what one translation made and<folder>/shared/for the lists good for any build, so a game folder whose stamp is in the store starts with its programs and pipelines and prepares nothing. Unset, empty or0, every file stays in the game folder'scache/as before, and the stamp is not taken any sooner. Nothing sets it for players yet: a launcher PR follows, setting it beside its builds and cleaning up the stamps no build uses.gpu/vk/shader_store.h(new, standard library and the platform's file lock, no Vulkan):open_stamped(at the first stamped file asked for, where the game took the stamp before) andopen_shared(at the first list); each part held by<part>/.lockfor the process's life: open with no sharing on Windows,flockelsewhere, so a tool can tell a part in use whatever it calls the path. A second run of the same stamp keeps that part's files in its owncache/for that run.consume: a game folder'scache/of the same stamp is taken in at a start that holds the store. Files the store lacks are moved in (a rename, else a copy as.partrenamed in); the program-side files only whencache/'s program cache carries this stamp, the pipeline cache only when its stamp file does. Pipeline notes and lists the store has get the records it lacks (merge_records, by their heads: the notes' stamp and record size, each list's magic and layout). A pipeline cache the store has a main file for goes in as a play file, merged at the next start byplay_files_merge. A second run's program cache is left in itscache/: merging two program caches record by record is not done here; the log says so, with its size, so those files can go. The precompile's record (longshot_pipelines.bin.built) moves with the notes alone: notes taken in without their pipeline cache take the store's record away (asstate_list_prepare's notes without pipelines do), soRX_VK_LOAD=createdoes not trust it and the next start builds them.folder_key: the compiler's version and the C++ library's), not the stamp alone: the program cache keeps its records'Shader::Infoas it lies in memory, and the stamp has the library's version for libc++ and Microsoft's but none for libstdc++ (HumanBean's point on MagicBeanDev/LongshotLauncher#78). Before the store each version had its folder and its tools, so a cache never met another toolchain's; now one never does in the store either. The stamp itself is unchanged, so nobody's caches go.hand_over): each game folder registers in the stamp it uses (<stamp>/users/, a file per folder naming it) and records it (cache/longshot_shader_store.txt). When a folder starts under another stamp (a new translator, another toolchain), the stamp it used before is renamed aside at once and deleted on a thread of its own, unless another game folder that is still there and still on it uses it, a running game holds it, or keeping was asked for (LONGSHOT_SHADER_CACHE_KEEP=1,--shader-cache-keep: both kept, for an A/B). So a folder holds one set of caches at a time, as without the store. Its old files are not reused under the new stamp: they are of no use to another translator, and a toolchain's change keeps the stamp while the records' layout may differ. A folder an earlier run set aside is swept at a later start. On Linux, where a rename is not refused under a running game as on Windows,let_gotakes the stamp's.lockand keeps it over the rename, andholdchecks the.lockit locked is still the one at its path (else it makes the folder again), so a game starting meanwhile never runs on a folder going aside;scripts/shader_store.py prunedoes the same. A Fallback start (another Longshot holds the stamp) leaves the folder's record as it was, so the stamp it used before is still let go at the next start that holds the store.<stamp>/longshot_last_usedsay what each folder uses and when a stamp was last used.fb603f47):translation_settings_readyset right aftercoopmat_init(vk_main.cpp);program_cache_stamp()doesn't keep a stamp asked for before it (said once, taken again), andshader_cache_pathdoesn't open the store's part before it (a file asked for then iscache/'s, said once with its name). The loads' marker (longshot_load_start.txt) is the game folder's own now, oncache_pathbeside the driver's faults, as is its removal at a preparation.gpu/vk/vk_paths.hshader_cache_path/shared_cache_path.vk_draw.cpp(the pipeline notes and what goes with them, the preparation's record, the pipeline cache's folder and its stamp check, the compute and state lists),vk_progcache.cppdisk_init(the program cache),vk_programs.cpp(the module ids) take their files through them. The module ids now go by path beside the programs: before, they usedprogram_cache_path()'s folder, which is "" with the program cache off or held by another Longshot, and the file landed in the working folder. The game folder's own files stay oncache_path: the preparation-skipped marker, the driver's faults, the loader's notes go with the notes.host/hos_main.c--shader-cache <folder|0>setsLONGSHOT_SHADER_CACHEfor the run, over the environment (on Windows as long as the path is, not cut atMAX_PATH + 32bytes), and--shader-cache-keepsetsLONGSHOT_SHADER_CACHE_KEEP=1; both entry points skip their values, andhost/options_path.c's scan of the arguments (#729) no longer takes either for the image.nx2/make_game.pyGAME_ENV_KEEP: the build's shader preparation keepsLONGSHOT_SHADER_CACHEandLONGSHOT_SHADER_CACHE_KEEP(the otherLONGSHOT_names still go), so the preparation fills the store the game will use;builder/linux/longshot-launchkeeps it too.scripts/dev_packs.pygives every game it starts one store (~/.cache/longshot-shaders,%LOCALAPPDATA%\Longshot\shaders) unlessLONGSHOT_SHADER_CACHEsays otherwise;scripts/shader_store.py(new)lslists a store's stamps (size, last use, in use) andprune --keep Nlets go of the stamps no running game holds and no game folder still uses (its users/ and record;--game <folder>names more), the newest N of the rest kept.docs/development.mdThe shader cache store (the layout, what the game does with it, the A/B case, clean-up), the switch's row, the dev_packs bullet; the README's argument table;docs/design.md's cache note.tests/pipeline_play_cache_test.cpp's stand-ins haveshader_cache_pathbesidecache_path(it compilespcache_initand the stamp check cut fromvk_draw.cpp).tests/test_shader_store.pywithtests/test_shader_store.cpp(the header against temporary folders: off and on, the parts, whatcache/gives the store, the record, a part held; and the renderer's call sites,game_env, dev_packs' names),tests/test_shader_store_tool.py(prune against a lock a child process holds as the game does, and the users' markers). The hand-over's cases intests/test_shader_store.cpp: the old stamp let go; kept for another folder still on it, with keep, or held; a stale marker not counted; a set-aside folder swept.tests/test_options_path.py: neither argument is taken for the image (it fails before the fix).tests/test_prepare_at_build.pyfollowsGAME_ENV_KEEPin both the Python and the shell launcher.Measurements
Verdict: works in the game on Windows. A second game folder of the same build, its
cache/empty, found the stamp's caches in the store and reached play in 78.9 s, preparing nothing; the same folder run beside the first (so on its owncache/) prepared every program, 1983 s. Two folders at once, another stamp, let go, kept: each did what Testing says.In the game (this PR @
680d1c83on main @6c8d6e6a, Windows 11 Pro, Ryzen 9 3900X, 64 GB, RTX 3080, a hard drive; the game started headless from the repository'skokiri-startsave withlongshot_ctl.py launch --profile test,LONGSHOT_SHADER_CACHEset; time from launch to play):6c8d6e6a(no store: each folder its owncache/)680d1c83(the log:vk: shader store: ...; time to play)cache/emptycache/as on main)cache/cache/; 297.8 s (prepared: see the note)cache/cache/RX_VK_MISS_LOG=1, in the translator's stamp)LONGSHOT_SHADER_CACHE_KEEP=1shader_store.py ls: "nothing uses it"cache/; "a program cache of this stamp stays in <B's folder>/cache/ (7440 MB): the store's own is read instead, and those files can be deleted"Fixed since (
fb603f47): folder A's first start took in the lists but not the stamped files because this PR took the program cache's stamp too early. The store's part opens at the first stamped file asked for, and the first was the loads' marker (load_fallback_path, beside the notes), asked for byprograms_load_decideright aftervkCreateDeviceand beforecoopmat_initsets the translation's per-GPU settings the stamp holds. So the run's stamp was no build's, and a store never took in a folder's caches from a build. Measured on main @615b2498: main's build preparedcache/with stamp 063fb3887f21b453; this PR before the fix took 74bd18fe7410cdf1, kept nothing and prepared everything; with the fix it takes 063fb3887f21b453, "took in 7 files of this stamp from <the folder's> cache/", and its program cache kept 20,603 keys (no preparation). Not the test profile, as an earlier comment guessed.The game-free part follows.
Hardware: Windows 11 Pro, Ryzen 9 3900X, 64 GB; llvm-mingw clang 23 (the Builder's v2.0.14 tools)
Scene: none (game-free checks; a renderer-only build,
RX_VK_ONLY)Method:
tests/test_shader_store.py --cxx <clang++>andtests/test_shader_store_tool.pyon this branch; the checks pass or fail, so there is nothing to take a median ofHow to reproduce: the two commands above;
RX_VK_ONLY=1 python -I nx2/build.py --standalonefor the rendererd9486a0870503f8acache/when the store is first usedRX_VK_ONLY)longshot_vk.dlllinkedFlags and settings
One switch and one argument for the same thing, off by default: the launcher PR turns it on for the builds it makes.
LONGSHOT_SHADER_CACHE=<folder>/--shader-cache <folder>cache/cache/stays anyway); untested: in the gameLONGSHOT_SHADER_CACHE_KEEP=1/--shader-cache-keepTesting
Tested: the stamp fix in the game on Windows (
fb603f47on main @615b2498: main's build-preparedcache/taken in by the store at the first start, the same stamp, no preparation); the 282 checks that read this diff on that head, not the whole suite (run_all.py --changed longshot/main): 240 passed, 32 skipped, 10 failed, the same 10 failing on a clean main @615b2498(test_walk_key,test_auto_handheld,test_failed_compile_keeps_objects,test_missing_nup,test_tested_configurations,test_job_queue,test_make_game_cache,test_linux_preflight,test_vk_occlusion_queries,test_crash_dialog);test_shader_storefails with the oldload_fallback_pathput back. Before it, in the game on Windows (Measurements: two game folders and one store, both at once, another stamp let go and kept, this PR @680d1c83on main @6c8d6e6a).tests/test_shader_store.py(the C++ checks with the Builder's llvm-mingw clang) andtests/test_shader_store_tool.pypass, Windows 11.python -I -B tests/run_all.pyon this branch (before its rebase ontoa00c485c, which adds only a skill file) and on main @9e6f231d, with the Builder's llvm-mingw clang (12 at once): this branch 542 passed, 18 failed, 35 skipped; main 540 passed, 18 failed, 35 skipped; the same 18 fail on both (test_appimage_player_build, test_cpm_discovery, test_cpp_preflight, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_options_budget_once, test_readback_late, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_walk_key).tests/test_shader_store.pyalso with g++ (libstdc++), for the library's branch of the folder's name. The renderer builds alone (RX_VK_ONLY). The Linux launcher's filter run under a shell: it dropsLONGSHOT_Xand keepsLONGSHOT_SHADER_CACHE.On the rebased head (
fea52a0b, HumanBean's rebase onto main @7bacf846): Windows with the Builder's llvm-mingw clang (6 at once), the 225 checks that read this diff (the tests that name a changed file, and the tests it adds or changes), not the whole suite: 189 passed, 14 skipped (tools this PC lacks, the Vulkan headers among them), 22 failed, each also failing on main:test_appimage_player_build,test_cpp_preflight,test_crash_dialog,test_lift_cleanup_error,test_linux_preflight,test_make_game_cache,test_make_game_prepare,test_missing_nup,test_oot_state_tap,test_tested_configurations,test_vk_pgo,test_walk_keyin the full run on main @1310ecc6; andtest_driver_cache_folder,test_linux_driver_fault_guard,test_linux_tools,test_linux_update,test_longshot_launch,test_longshot_saves,test_menu_save_atomic,test_pipeline_tour,test_udmabuf_warning,test_vk_occlusion_querieson main @7bacf846(run there alone: they fail the same way).test_shader_storeandtest_shader_store_toolpass.The 225 checks run
nx2/test_cbuf_table_extent.py,nx2/test_dump_ssbo_read.py,nx2/test_mesh_dynamic_state.py,nx2/test_nup_stand_in.py,nx2/test_nup_zstd_heads.py,tests/test_amd_fragment_warp.py,tests/test_appimage_headers.py,tests/test_appimage_player_build.py,tests/test_appimage_prompt_fonts.py,tests/test_appimage_scripts.py,tests/test_aspect_default.py,tests/test_audio_output_menu.py,tests/test_auto_settings.py,tests/test_bake_ds.py,tests/test_bank_gen.py,tests/test_bc_raw_targets.py,tests/test_bound_keys.py,tests/test_bug_report.py,tests/test_build_cleanup_scope.py,tests/test_build_hash_cache.py,tests/test_build_log.py,tests/test_build_times.py,tests/test_builder_drive_error.py,tests/test_camera_speed.py,tests/test_cheat_items.py,tests/test_check_no_compiler.py,tests/test_check_pr_description.py,tests/test_chunks_save_folder.py,tests/test_commit_room.py,tests/test_copy_writable.py,tests/test_cpp_preflight.py,tests/test_cpu_target.py,tests/test_crash_dialog.py,tests/test_crash_record.py,tests/test_damaged_check_progress.py,tests/test_debug_tools_linux.py,tests/test_desc_clock.py,tests/test_desc_counts.py,tests/test_dev_packs.py,tests/test_device_lost_exit.py,tests/test_disk_warning.py,tests/test_dispatch_clock.py,tests/test_display_modes.py,tests/test_display_startup_failure.py,tests/test_dlss_checks.py,tests/test_draw_binds.py,tests/test_draw_side_effects.py,tests/test_draw_stream_visibility.py,tests/test_drawn_page_map.py,tests/test_driver_cache_folder.py,tests/test_driver_cache_room.py,tests/test_driver_faults.py,tests/test_dummy_views.py,tests/test_env_file.py,tests/test_equal_depth_margin.py,tests/test_failed_compile_keeps_objects.py,tests/test_failure_reason.py,tests/test_fallback_buffer_cleanup.py,tests/test_fallback_depth_hold.py,tests/test_fallback_loaded.py,tests/test_fallback_pipeline_state.py,tests/test_fallback_size_policy.py,tests/test_ffx_sdk.py,tests/test_finish_output.py,tests/test_flat_map.py,tests/test_flight_yields_to_events.py,tests/test_forge_pr.py,tests/test_framegen_preset.py,tests/test_fsr_uniform_ownership.py,tests/test_global_table_loop.py,tests/test_gpl_cold_library.py,tests/test_gpl_library_cache.py,tests/test_handle_lookup.py,tests/test_handle_table_arrays.py,tests/test_hle_stats.py,tests/test_hold_gpu.py,tests/test_intel_compiler_gate.py,tests/test_intel_quick_compute.py,tests/test_jobs_failure.py,tests/test_jobs_wait_runs_queued.py,tests/test_kept_translation.py,tests/test_launcher_capture.py,tests/test_lift_cache.py,tests/test_lift_cleanup_error.py,tests/test_lift_cut_short.py,tests/test_lift_store.py,tests/test_link_path_comma.py,tests/test_linux_driver_fault_guard.py,tests/test_linux_preflight.py,tests/test_linux_pressure.py,tests/test_linux_saves_move.py,tests/test_linux_tools.py,tests/test_linux_update.py,tests/test_load_default.py,tests/test_longshot_ctl.py,tests/test_longshot_launch.py,tests/test_longshot_saves.py,tests/test_low_memory_notice.py,tests/test_make_game_cache.py,tests/test_make_game_ngx.py,tests/test_make_game_prepare.py,tests/test_make_game_writable.py,tests/test_mark_groups.py,tests/test_masked_clear_failure.py,tests/test_mem_trim.py,tests/test_menu_env_on.py,tests/test_menu_restart.py,tests/test_menu_save_atomic.py,tests/test_mesh_args_cleanup.py,tests/test_mesh_fallback.py,tests/test_mesh_fallback_failed_once.py,tests/test_mesh_query_feature.py,tests/test_met_wait_capture.py,tests/test_missing_nup.py,tests/test_module_ids.py,tests/test_movement_speed.py,tests/test_native_float_atomic_add.py,tests/test_no_release_lookup.py,tests/test_no_webgpu.py,tests/test_object_cache_prune.py,tests/test_object_store.py,tests/test_ocarina_note_bindings.py,tests/test_ocarina_notes.py,tests/test_oot_state_tap.py,tests/test_opening_skip.py,tests/test_options_path.py,tests/test_output_resolution.py,tests/test_pad_layout.py,tests/test_parallel_dry_marks.py,tests/test_parallel_rb_refresh.py,tests/test_pass_hazard.py,tests/test_pcache_fetch_lock.py,tests/test_pcache_header_read.py,tests/test_pcache_stamp.py,tests/test_pipeline_cache_save.py,tests/test_pipeline_notes_open.py,tests/test_pipeline_play_cache.py,tests/test_pipeline_share.py,tests/test_pipeline_state_computes.py,tests/test_pipeline_state_counts.py,tests/test_pipeline_state_portable.py,tests/test_pipeline_states.py,tests/test_pipeline_tour.py,tests/test_portable_builder_startup.py,tests/test_precompile_compute_guard.py,tests/test_precompile_pipeline_cleanup.py,tests/test_predict_resume.py,tests/test_predict_sees_loaded.py,tests/test_predict_types.py,tests/test_predicted_check.py,tests/test_prepare_at_build.py,tests/test_prepare_cache_save.py,tests/test_prepare_cpus.py,tests/test_prepare_seed_stamp.py,tests/test_prepare_stall.py,tests/test_prepare_workers.py,tests/test_prepared_stamps.py,tests/test_prof_sampling.py,tests/test_program_cache_cleanup.py,tests/test_program_cache_format.py,tests/test_program_cache_index.py,tests/test_program_cache_one_variant.py,tests/test_program_cache_reader.py,tests/test_program_life.py,tests/test_program_load_wait.py,tests/test_program_record_head.py,tests/test_program_stamp_knobs.py,tests/test_prompt_skip.py,tests/test_psychov35_dlss.py,tests/test_psychov35_taau.py,tests/test_push_state.py,tests/test_quick_pipelines.py,tests/test_radv_mesh_optimizer.py,tests/test_reactive_mask.py,tests/test_readme_menu_rows.py,tests/test_recomplauncher_json.py,tests/test_recorder_own_idle.py,tests/test_recorder_sync.py,tests/test_recording_admission.py,tests/test_replay_copies.py,tests/test_rumble_menu.py,tests/test_same_draw.py,tests/test_same_dyn.py,tests/test_sampler_admission.py,tests/test_shader_census_cmd.py,tests/test_shader_list_skip.py,tests/test_shader_module_guard.py,tests/test_shader_stamp_cache.py,tests/test_shader_stamp_inputs.py,tests/test_shader_store.py,tests/test_shader_store_tool.py,tests/test_shadow_mask_guard.py,tests/test_ship_state_list.py,tests/test_skip_title.py,tests/test_split_before_draw.py,tests/test_stale_hot_pieces.py,tests/test_startup_progress.py,tests/test_state_list_plain.py,tests/test_stick_dead_zone.py,tests/test_switches_removed.py,tests/test_system_header_cache.py,tests/test_taau_env_lookup.py,tests/test_taau_fxaa.py,tests/test_taau_typed_dispatch.py,tests/test_table_track.py,tests/test_teleport_minimap_flag.py,tests/test_tested_configurations.py,tests/test_texture_handle_banks.py,tests/test_texture_squeeze.py,tests/test_third_party_objects.py,tests/test_threaded_rendering.py,tests/test_tls_objects.py,tests/test_tree_shadow_mask.py,tests/test_ub_threads.py,tests/test_udmabuf_warning.py,tests/test_upscale_fallback.py,tests/test_upscaler_rows.py,tests/test_utf8_paths.py,tests/test_vertex_bind.py,tests/test_vk_occlusion_queries.py,tests/test_vk_pgo.py,tests/test_walk_key.py,tests/test_warm_bounded.py,tests/test_widescreen_shapes.py,tests/test_work_lock.pyOn this head (
70503f8a: rebased onto main @d9486a08, with the answers to HumanBean's review): Windows, the 59 checks that read this diff (the tests that name a changed file), not the whole suite: 46 passed, 13 failed, each also failing on main @d9486a08when run there alone (test_bug_report,test_crash_dialog,test_crash_record,test_driver_cache_folder,test_driver_env_choose,test_log_open_failure,test_menu_save_atomic,test_oot_state_tap,test_pipeline_tour,test_udmabuf_warning,test_vk_occlusion_queries,test_vk_pgo,test_walk_key).tests/test_shader_store.pypasses with g++ 10.3 and the Builder's llvm-mingw clang++ on Windows, and with g++ 14.2 on Linux (Dockerpython:3.12: theflockpath ofhold,let_goand a held stamp), andtests/test_shader_store_tool.pyon Linux. Its new cases (the record a Fallback start leaves,.builtafter notes taken in with and without their pipeline cache, the program cache left incache/said) fail 5 checks against the header before this commit.host/hos_main.ccompiles to an object for Windows with llvm-mingw clang, with the same 3 warnings as before (the change is Windows-only).Host build (before the review's commit): compiled to objects on
fea52a0band on main @7bacf846, all clean on both:host/hos_main.candhost/options_path.con Windows with the Builder's llvm-mingw clang (x86_64-w64-mingw32) and on Linux with gcc 14.2 and clang 19 (Dockerpython:3.12,host/posixon the include path);gpu/vk/vk_draw.cpp,gpu/vk/vk_progcache.cppandgpu/vk/vk_programs.cppwith llvm-mingw clang++ and, on Linux, clang++ 19 (g++ refusesgpu/render/rx_nvn.h's_Static_assertin C++, on main too; the Linux build uses clang). The changed headers (shader_store.h,vk_paths.h) were compiled through those files. build.py's flags for one object, the runtime headers a lift makes (a64.h,a64_suyu.h,a64_fpx.h) copied from a local work folder, and for the renderer files the Builder's CPM Vulkan, SPIR-V and VMA headers. It covers the files this PR changes, not a whole game build.Not tested:
host/hos_main.c's--shader-cachecompiles (above), and the first run under Must be tested builds and runs it.flocklock in the game (the store's checks run it game-free, above), andtests/test_longshot_launch.py(it runs only there).Must be tested:
<store>/<stamp>/.lockheld by the first (theflocklock), the second running from its owncache/; and the AppImage's Play button (longshot-launch) starting the game with the switch still set.Risks
What could break: with the switch set, a folder's old stamp is deleted when it moves to another and no other existing folder still uses it: a folder whose record or marker went missing could let go of a stamp another build still wanted (that build then prepares again; the shared lists stay). With the switch set, a game that read or wrote a shader cache file in the wrong place would prepare again, or lose what it compiled in play; the call sites are all checked by text, and unset nothing moves. Two runs of one stamp could clash: each part is held for the process's life, and the second run uses its own
cache/. A store on another drive than the game folder makes the first take-in a copy (several GB) instead of a rename. The store grows by a whole set of caches for each translation until something lets the unused ones go: the launcher PR for players,scripts/shader_store.py prunefor development stores.Turning it off: unset
LONGSHOT_SHADER_CACHE, or start with--shader-cache 0: the game folder'scache/again. A store's files stay in the store until pruned.Compatibility: nothing changes for players until the launcher sets the switch: unset, every file is where it was, and nothing takes the stamp sooner than before. With it set, a game folder's existing caches of the same stamp move into the store at its first start. Saves, the options file and the vk ABI are unchanged.
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
b34e61c30b0e8569be74Rebased onto main
9e6f231d(past #729, which added options_path beside the store's paths in vk_paths.h: both kept). New: a stamp's folder is named by the stamp and the toolchain (folder_key: the compiler's version and the C++ library's), so caches never cross a toolchain in the store; HumanBean's point on LongshotLauncher#78 about libstdc++ (the stamp carries libc++'s and Microsoft's library version, not libstdc++'s) applies here too. The stamp itself is unchanged, so nobody's caches go. The record now says both (stamp=, translation=). Full suite again on this branch and main: no new failure; the store's check passes with clang and g++.0e8569be74a45e8c89d0Pushed
a45e8c89(rebased onto maina00c485c): an update in place, in the store. Each game folder registers in the stamp it uses (/users/) and records it; when a folder starts under another stamp, its old one is renamed aside and deleted in the background unless another game folder still on it uses it, a running game holds it, or LONGSHOT_SHADER_CACHE_KEEP=1 / --shader-cache-keep asks to keep both (an A/B). So a folder holds one set of caches at a time, as without the store; the old files are not reused under the new stamp (another toolchain keeps the stamp, not the records' layout). Also fixed: host/options_path.c (#729) took --shader-cache's value for the image; it now skips both arguments, with a case in test_options_path.py that fails before. scripts/shader_store.py prune now reads the stamps' users itself. Renderer builds; full suite: no failure main doesn't have. Still WIP for the whole build (the host change) and the runs in the game under Must be tested.a45e8c89d0fea52a0be3Rebased onto
main@7bacf846(no conflicts); pushed asfea52a0b. No tests were run here; the test boxes stay yours.Tested HumanBean's rebase (
fea52a0b): the 225 checks that read this diff, 189 passed, 14 skipped, 22 failed, each failing on main too; and the host build box ticked: host/hos_main.c, host/options_path.c, gpu/vk/vk_draw.cpp, vk_progcache.cpp and vk_programs.cpp compile to objects onfea52a0band on main7bacf846for Windows (llvm-mingw clang) and Linux (gcc and clang 19 in Docker); changed files only, not a whole game build.WIP: feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothingto feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothingA measurement that bears on "the old files are not reused under the new stamp", for the pipeline cache (
longshot_vk_pipelines.bin) only.On this PC (Ryzen 5 5600, RX 6700 XT, AMD 26.10.1, Windows 11), the launcher's update from nightly-130 (
0caf0618) to nightly-189 (aa95d35f) changed the translator stamp. The new folder's preparation therefore started without a pipeline cache and took 85 min: about 20 min of programs and compute pipelines, then 65 min of the 32,623 noted graphics pipelines.I ran that preparation again in a fresh nightly-189 folder, with nightly-130's
longshot_vk_pipelines.binand its play files copied intocache/andRX_VK_PCACHE_STAMP=0. The exe path was new, so AMD's own driver cache was cold, as in an update. The command was make_game.py's:--prepare-shaders --compile-shaders most.The stamp changed, but almost none of the SPIR-V did: one stamped commit landed in between (#400), and 480 sampled programs were byte-identical in both folders' program caches. The driver finds entries by their contents, so the old ones still hit. The 30 min left are most likely the pipelines new since nightly-130, mostly #563's list.
So keeping the previous stamp's pipeline cache (merged in as a play file, say) would save most of an update's preparation on AMD; dropping it costs the whole preparation. The program-side files are right to go with the stamp.
The cost is growth: entries for SPIR-V that no longer exists stay in the file. One possible bound is to carry it across one stamp change only. Another is to carry it only while it stays under some multiple of a fresh cache's size. Here the file grew from 1.49 GB (main file plus play files) to 1.98 GB. The update's freshly built one is 1.93 GB, so the stale entries cost only about 50 MB this time.
This is one run on one PC, so the minutes will vary. Would you rather have this in #737, or as a follow-up once it lands? I can write either.
Code review only: nothing was built or run here. I read the whole diff (shader_store.h, the call sites in vk_draw/vk_progcache/vk_programs, vk_paths.h, the host's arguments, make_game/longshot-launch, dev_packs, shader_store.py, the docs and the tests) and the comments.
Verdict: comment. The design holds up and I found nothing that has to change before merge. The store stays off unless
LONGSHOT_SHADER_CACHEis set, and nothing sets it for players yet. It isn't an approval because none of the in-game runs under Must be tested has been done, on Windows or Linux. On Linux theflocklock and the AppImage'slongshot-launchhave never run at all.What I checked and found right:
consumereads match what the game writes. The program cache's stamp is at offset 8 (disk_init). The notes' head is the stamp thensizeof(PipeNote), compared as 16 bytes.pcache_stamp.hkeeps 8 bytes at offset 0. The play-file numbers from 1000000 up stay withinplay_files_merge's 9 digits.shader_cache_pathcall (which takesprogram_cache_stamp()and caches it) comes afterreplay_start/create_deviceon the paths I followed:started_up,prepare_guess,pcache_init,ids_ready, the loader'sload_fallback_path. So the stamp isn't taken before the device exists. With the store unset, the stamp isn't taken here at all.*Afunctions,fopenandstd::filesystemall depend on the exe manifest's UTF-8 code page, as the rest of host/ and gpu/vk already do. NoCP_ACP.options_path.cand both argument scans inhos_main.cskip--shader-cache <v>and--shader-cache-keep.GAME_ENV_KEEPand longshot-launch's awk filter agree, andtest_prepare_at_build.pychecks both.0/empty mean off.dev_packs.pyturns it on for development only, withsetdefault.Suggestions, inline:
hand_overchecksheld_elsewhere, then renames. A game that takes the lock in between keeps running on a folder that has been renamed out from under it.hos_main.c: the--shader-cachevalue is silently cut atMAX_PATH + 32.cache/,.builtafter a merge), inline.Overlaps (merge order): none of these duplicates this PR, but two will break
tests/test_shader_store.pyonce both land, so the second to merge has to update it:driver_env_path() { return cache_path("longshot_driver_env.txt"); }tovk_draw.cpp.check_sites' regex rejects anycache_path("...")there other than the driver faults, so it needs that name added (it is the game folder's own, like the faults).driver_faults_pathtogame_folder_path()andlongshot_load_start.txtbeside the settings. Thedriver_faults_pathneedle incheck_sitesthen fails, and the load-start marker would leave the store's stamp folder.tests/pipeline_play_cache_test.cpp's stand-ins. That conflict is probably textual only.pcache_dir/pipe_notes_path(), so with this PR those files follow the store without changes. #623 and #33 don't overlap.madebyjordan's measurement (keeping the old stamp's
longshot_vk_pipelines.binsaves about 50 of 85 minutes of an update's preparation on AMD): main already drops that cache when the stamp changes (pcache_drop_stale), so letting it go here is no regression. I'd do it as the follow-up they offered, bounded to one stamp change, rather than grow this PR.@ -0,0 +252,4 @@if (n > 0) s.said.push_back("took in " + std::to_string(n) + " pipeline notes from " + from);continue;}if (!exists(to + name)) {When the store already has a program cache, a second run's
longshot_programs.bin/.idx(several GB) stays incache/for good. Nothing later deletes it, and every Store start reads its head again. You say in the description that the record-by-record merge isn't done here. Could a line in the log (orshader_store.py ls) point it out, so a player or tester knows those GB can go?A question:
precompile_covers_notescompares only the stamp in.built, not the count. Notes merged in here come from a run whose pipelines went into its own pipeline cache, so they're covered when that cache comes in as a play file. When it doesn't come in (its stamp file is missing or different), the merged notes are uncovered, but.builtstill says they're built andRX_VK_LOAD=createtrusts it.state_list_prepareremoves.builtwhen it adds notes without pipelines. Should the merge here do the same whenpcacheis false?@ -0,0 +495,4 @@if (FILE* f = fopen((s.stamp_dir + kLastUsed).c_str(), "wb")) fclose(f); // (its time: the launcher keeps the newest)s.said.push_back("stamp " + hex(folder_key(stamp)) + ": the shader caches are in " + s.stamp_dir);}write_record(s);write_recordalso runs in the Fallback case, when another Longshot holds this stamp, andhand_overdidn't run. The record then names the new stamp, so at the next Store startprevious == nameand the stamp this folder really used before is never let go by the game. Its users/ marker no longer counts, so only the launcher orprunewill remove it. Writing the record only whenstamped == Where::Storewould keep the update-in-place clean-up working across a Fallback run. In Fallback, this run's files are incache/anyway.@ -0,0 +439,4 @@s.said.push_back("stamp " + previous + ", this folder's before, kept (LONGSHOT_SHADER_CACHE_KEEP)");else if (used_by_others(prev_dir, previous, std::string()))s.said.push_back("stamp " + previous + ", this folder's before, stays: another game folder uses it");else if (held_elsewhere(prev_dir))On Linux this check is separate from the rename below. If a game on the old stamp starts between
held_elsewhere(which drops theflockat once) andlet_go's rename, the rename still succeeds under it: Linux doesn't refuse it the way Windows does. That game then keeps running on a renamed folder that a detached thread is deleting. Its open descriptors keep working, but every later open bystamp_dirpath fails. The window is small, but closing it is cheap: keep theflocktaken here until after the rename (return the descriptor, or do the rename inside it). A game that starts after that makes a fresh folder and lock.@ -1840,0 +1844,4 @@for (int i = 1; i + 1 < argc; i++)if (!strcmp(argv[i], "--shader-cache")) {#ifdef _WIN32static char set[MAX_PATH + 32];A store path longer than
MAX_PATH + 32bytes is cut short here without a word. A UTF-8 path with non-Latin characters reaches that sooner than its character count suggests._putenv_s("LONGSHOT_SHADER_CACHE", argv[i + 1])has no buffer to cut.--compile-shadersabove uses the same pattern, but its values are short.@ -0,0 +61,4 @@# (no shader cache file left on cache_path beside those two: one would split from its stamp's folder)for name, src in (("vk_draw.cpp", draw), ("vk_progcache.cpp", prog), ("vk_programs.cpp", progs)):for m in re.finditer(r'(?<![_a-z])cache_path\("([^"]*)"\)', src):if m.group(1) not in ("longshot_driver_faults.txt",):Heads-up on merge order: #902 adds
cache_path("longshot_driver_env.txt")to vk_draw.cpp, which this check then refuses. #345 movesdriver_faults_pathoffcache_path, which breaks the needle at line 56. Whichever of these lands second updates this check.fea52a0be370503f8ae5Pushed
70503f8a: rebased onto maind9486a08(one conflict, the switch table indocs/development.md: #900'sRX_VK_HALF_RATE_SHADINGrow and the store's both kept), plus one commit for @HumanBean's review. Thank you for the careful read.held_elsewhereand the rename: fixed both ways.let_gonow returns Gone / Held / Open. On Linux it takes the stamp's.lockwithflockand keeps it over the rename, closing it after.holdchecks that the.lockit locked is still the file at its path (fstatagainststat). If a game opened the.lockbefore the rename and locked it after, the folder is made again and its new.locktaken. Windows keeps the check-then-rename, since a game that takes the.lockin between makes the rename fail ("a file in it is open").scripts/shader_store.py prunekeeps the lock over its rename too.write_recordruns only whenstamped == Where::Store, so a Fallback run leaves the record naming the stamp before, and the next Store start lets that one go.test_shader_store.cpp's held-part case now checks the record is unchanged.--shader-cachecut atMAX_PATH + 32: the string is now allocated to fit the path and kept for the process's life (no_putenv_s, so nothing depends on which CRT declares it).cache/:consumenow logsa program cache of this stamp stays in <cache/> (N MB): the store's own is read instead, and those files can be deleted..builtafter merged notes: yes..builtnow moves into the store only with the notes file. When notes are merged in and no pipeline cache of theirs comes in (main file or play file), the store's.builtis removed, asstate_list_preparedoes, and the log says the next start builds them.cache/'s own.builtgoes with its merged notes. Both cases (with and without the pipeline cache) are in the test.The new checks fail 5 times against the header before this commit and pass after: Windows with g++ 10.3 and llvm-mingw clang++, and Linux with g++ 14.2 in Docker, which runs the
flockpath. The 59 checks that read the diff: 46 pass, and the 13 that fail also fail on maind9486a08. Details are in Testing.Merge order with #902 and #345: noted. Whichever of them lands after this PR updates
check_sitesintests/test_shader_store.py. If this one lands second, I'll do it here.@madebyjordan, thank you for the measurement: 85 to 33 minutes is a lot to leave on the table. I'd take it as the follow-up you offered, once this lands, bounded to one stamp change as you suggest. A follow-up keeps this PR's take-in rules as they are, and the play-file merge is already the way in for it. Main drops that cache at a stamp change today (
pcache_drop_stale), so nothing here is lost against main in the meantime.Still not run in the game: the runs under Must be tested are next.
70503f8ae53ded310de23ded310de2680d1c836dRun in the game on Windows (this PR @
680d1c83, rebased onto main @6c8d6e6awith one conflict, the two new sections ofdocs/development.mdboth kept), Windows 11, RTX 3080; the Must be tested steps, through the job queue. Every step is in Measurements.cache/, found the stamp's caches in the store and reached play in 78.9 s. Run beside folder A (so on its owncache/, as every folder is on main), the same B prepared every program from nothing: 1983 s.cache/; its files went into the store at its next start alone ("took in 1 pipeline notes ... 5 files"), and the 7.4 GB program cache it made is named in the log as deletable (the review's point 4).RX_VK_MISS_LOG=1is in the translator's stamp, so a start with it is a start under another translation): kept while another folder used it, kept withLONGSHOT_SHADER_CACHE_KEEP=1, let go once no folder used it and keep wasn't asked for.shader_store.py lsagreed after each step.One thing I didn't expect, an open question rather than a bug in this PR: folder A's first start took in the lists but not its stamped files. Its
cache/was prepared at build time, under the game folder's own settings;longshot_ctl.py launch --profile test(an emptyHOS_ENV_FILE, a user folder of its own) starts the game with another program cache stamp, so those files were not the stamp's, and main'sdisk_initwould have prepared again too. @HumanBean: should the test profile keep the folder's env file for the switches that are in the stamp, so a test start meets the build's caches? That would be its own change, tolongshot_ctl.py.The Windows item is off Must be tested; the Linux run (the
flocklock, the AppImage'slongshot-launch) is still wanted.680d1c836dfb603f47daCorrection to my last comment, and a fix (
fb603f47, rebased onto main @615b2498): the test profile was not the cause; this PR was. @HumanBean, please disregard the question I asked about--profile test.The bug: with the store set, the stamp's part opens at the first stamped file asked for and takes
program_cache_stamp()then, which keeps it for the run. The first such file was the loads' marker:load_fallback_path()built its path frompipe_notes_path(), andprograms_load_decide()asks for it right aftervkCreateDevice, beforecoopmat_init()sets the translation's per-GPU settings thattranslator_stamp()hashes (the invariant positions, the NaN guard, the matrix layouts). So every run with the store had a stamp no build of the same main computes, and the take-in never matched a build'scache/. That's why folder A's first start took in only the lists, and why all of this PR's runs agreed with each other but never with main. It would have hit every player's first start with the store after an update: the old caches never taken in, a full preparation each time.Measured (main @
615b2498, Windows 11, RTX 3080): main's build preparedcache/with stamp063fb3887f21b453. This PR before the fix took74bd18fe7410cdf1: "0 program keys kept", a full preparation. With the fix it takes063fb3887f21b453, "took in 7 files of this stamp from <the folder's> cache/", and the program cache kept 20,603 keys, with no preparation.The fix:
cache_path, beside the driver's faults), and so is its removal at a preparation;translation_settings_readyis set right aftercoopmat_init();program_cache_stamp()doesn't keep a stamp asked for before it (it says so once in the log);shader_cache_path()doesn't open the store's part before it (a file asked for then iscache/'s, said once with its name).So a later early caller is named in the log instead of silently splitting the caches.
test_shader_storechecks all four, and fails with the old marker path put back. The 282 checks that read the diff: 240 pass, 32 skip, and 10 fail that fail on a clean main too (Testing).LONGSHOT_JOBSfiles at once, not one #1147We'll run this on a Steam Deck OLED (SteamOS, RADV) for the Linux item under Must be tested: the
flocklock and a second folder that prepares nothing. The Deck matters here: a full preparation takes 25-40 minutes on it, and today every stamp change (anyRX_VK_MESH*switch, for one) throws the caches away in both directions.What we'll run. Your four commits cherry-picked onto main
71b1edd8, our Deck base (no conflicts; the added and removed lines matchgit diff 615b2498 fb603f47). The control is71b1edd8itself. Both builds have translator stamp2b1aa955fa54ed89, so a cache the control prepares is valid for the PR too. Every run gets its own game folder and store and its ownXDG_*, so nothing is shared with the Deck's installed game.RX_VK_MESH_NOCULL=2c56dc1006942f74(a full re-preparation), and a flip back (another one).cache/of the same stamp should move it into the store, with the same program-key count and no preparation.LONGSHOT_SHADER_CACHE_KEEP=1(expect one preparation for the new stamp and none after). Without KEEP: expect the old stamp let go and one set kept.cache/, should prepare nothing. At once: the second should get the Fallback lines, the first should hold<stamp>/.lock(read from/proc/locks), and both should reach play. Then the second alone, to check the late take-in and the "can be deleted" line, and that no program cache error lines appear afterwards.We'll post the times (to title, to play, translation and pipeline stages), disk use and the log lines here when it's done.
From reading the diff, and the checks on Linux:
tests/test_pipeline_tour.pyfails on this head and passes on its merge base615b2498(Linux), so it isn't one of the failures main shares. It guardsscripts/pipeline_tour.py, which movescache/longshot_pipelines.binand the state lists aside before a tour. WithLONGSHOT_SHADER_CACHEset, the game reads those from the store, so a tour would run with the store's notes in place. Either the tour learns the store (or starts its games with--shader-cache 0) and the check follows, or the check gets updated with a reason.cache/holds another stamp's caches (say, an update that changes the translator and turns the store on together):consumeskips them (shader_store.h:260-261), andpcache_drop_staleanddisk_initnow look in the store, so the old program cache (about 8 GB on the Deck) and its pipeline cache stay incache/for good, without a log line. On main they're dropped in place. Could the start that holds the store remove, or at least name,cache/'s stamped files of another stamp, as it already does for a second run's program cache?hold()gives up at the firstEWOULDBLOCK(shader_store.h:148).scripts/shader_store.py ls/prune(in_use) and another game'slet_gotake each stamp's.lockfor a moment, so a game starting right then falls back to its owncache/, and with an empty one it prepares everything. A few retries over a second or so inholdwould close that.cache/'s module ids,longshot_prepared.binand the predictions stay behind without a word (shader_store.h:278-284). Only the program cache gets the "can be deleted" line.python -I -B tests/run_all.py --changedagainst maina413bf1con this head (Linux, Vulkan and SPIR-V headers given): 278 checks, 265 passed, 4 skipped, 13 failed. Eleven of those fail on615b2498too (test_bug_report,test_failed_compile_keeps_objects,test_fsr_uniform_ownership,test_linux_preflight,test_linux_update,test_make_game_cache,test_missing_nup,test_nonuniform_image_arrays,test_replay_write_order,test_vk_occlusion_queries,test_walk_key).test_pipeline_cache_savefailed under load and passes alone. That leavestest_pipeline_tour, above.test_shader_store(g++ on Linux, theflockpath),test_shader_store_tool,test_options_path,test_prepare_at_buildandtest_pipeline_play_cachepass, on this head and on the cherry-pick. This head still merges cleanly into maina413bf1c.@dyst, @HumanBean: no action needed before the run. The first two points might be worth settling before merge.
Code-and-evidence review of
fb603f47daonly; no PR code, tests, builds or game runs were executed here.Verdict: request changes. Sharing prepared caches can spare a second game folder a lengthy preparation. The early translator-stamp capture and the earlier lock/record/
.builtreview points have been addressed; the new Windows first-start evidence demonstrates matching and taking in a build's existing cache.user_marker(inline). Lowercasing every path aliases case-distinct game folders, so changing one folder's stamp can remove the other folder's ownership marker and delete caches it still needs.test_pipeline_tourfailing here and passing on merge base615b2498. This is separate from the listed baseline failures.pipeline_tour.pyinheritsLONGSHOT_SHADER_CACHE, but moves aside, observes and exports only the game folder'scache/lists. With a store enabled the renderer now uses the store instead, so the tour can include old notes and miss its new state list. Force the store off for tour launches or make the tour explicitly manage the store, and update the existing check accordingly. This is a behavior problem, not merely a stale source-text assertion.Evidence still needed: Windows 11/RTX 3080 runs cover two folders, simultaneous fallback, stamp hand-over/KEEP and the current-head stamp fix. The Linux/SteamOS
flock, simultaneous-folder behavior and AppImage Play-button environment run is still only announced in the linked comment, not a completed result. Keep this not ready until that result is posted. Skips and main's failures are not passing coverage. The reported 78.9 s versus 1983.3 s illustrates cache reuse; it is not an interleaved performance estimate for the current head. The same comment also identifies old-stamp files stranded incache/; settle their cleanup or explicit reporting before claiming disk cleanup is complete. Keeping a previous stamp's driver cache is a separate follow-up: main already drops it, so its loss is not introduced here.Overlaps/order: coordinate #902 and #345 with this PR: the second to land must reconcile path ownership and
test_shader_store.pyassertions. #517 also changes preparation/notes lifetime and needs integration review. The earlier review's play-cache stand-in overlap with #416 should be rechecked if that work is carried forward. No duplicate implementation is established. Resolve the current mergeability state and revalidate the resulting head before merge.@ -0,0 +382,4 @@/// Where a game folder's marker goes in a stamp's users/: named by the folder's path, holding itinline std::string user_marker(const std::string& stamp_dir, const std::string& game) {uint64_t h = 0xCBF29CE484222325ull;for (const char ch : game) h = (h ^ (uint8_t)(ch == '\\' ? '/' : tolower((unsigned char)ch))) * 1099511628211ull;[P2] Preserve case-sensitive game-folder identities on Linux.
user_markerlowercases paths on every platform, so distinct folders such as/games/A/and/games/a/get the same marker. After both use stamp S, starting either folder under stamp T removes that shared marker from S inhand_over; with the other game stopped,used_by_othersfinds no owner andlet_godeletes S even though the other existing folder still records S. Its next start must prepare all shaders again. Keep path case (and literal backslashes) on case-sensitive platforms and add a Linux two-folder hand-over case proving the other marker/cache survives.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.