feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing #737

Open
dyst wants to merge 4 commits from dyst/Longshot:feat/shader-cache-store into main
Member

Type: feat · Risk: medium · Tested on: Windows 11, RTX 3080, in the game (two game folders, one store) and game-free · Needs: a Linux run in the game (the flock lock, the AppImage's longshot-launch)

Why

Every game folder prepares its shaders on its own, about 25 minutes with Compile shaders All on a fast PC, even when another folder on the same PC translated the same shaders with the same translator a minute before: a stable and a nightly side by side, a rebuild into a new folder, each dev_packs.py runtime (which also carries a 10 GB copy of the caches). The caches are already stamped by what they are good for (program_cache_stamp), so a store keyed by that stamp lets every folder with the same translation use them in place.

The design was agreed with the maintainers, including moving these files out of the game folder, where #321 put them. It complements MagicBeanDev/LongshotLauncher#57 (an update in place keeps its cache/ anyway) and replaces copying caches between folders (MagicBeanDev/LongshotLauncher#78). It mirrors the object and lift stores (#371, #372). No Longshot issue fits (the tracker searched).

What changed

With LONGSHOT_SHADER_CACHE=<folder> (or --shader-cache <folder>), the game keeps its shader caches in <folder>/<stamp>/ for what one translation made and <folder>/shared/ for the lists good for any build, so a game folder whose stamp is in the store starts with its programs and pipelines and prepares nothing. Unset, empty or 0, every file stays in the game folder's cache/ as before, and the stamp is not taken any sooner. Nothing sets it for players yet: a launcher PR follows, setting it beside its builds and cleaning up the stamps no build uses.

  • runtime: gpu/vk/shader_store.h (new, standard library and the platform's file lock, no Vulkan):
    • open_stamped (at the first stamped file asked for, where the game took the stamp before) and open_shared (at the first list); each part held by <part>/.lock for the process's life: open with no sharing on Windows, flock elsewhere, so a tool can tell a part in use whatever it calls the path. A second run of the same stamp keeps that part's files in its own cache/ for that run.
    • consume: a game folder's cache/ of the same stamp is taken in at a start that holds the store. Files the store lacks are moved in (a rename, else a copy as .part renamed in); the program-side files only when cache/'s program cache carries this stamp, the pipeline cache only when its stamp file does. Pipeline notes and lists the store has get the records it lacks (merge_records, by their heads: the notes' stamp and record size, each list's magic and layout). A pipeline cache the store has a main file for goes in as a play file, merged at the next start by play_files_merge. A second run's program cache is left in its cache/: merging two program caches record by record is not done here; the log says so, with its size, so those files can go. The precompile's record (longshot_pipelines.bin.built) moves with the notes alone: notes taken in without their pipeline cache take the store's record away (as state_list_prepare's notes without pipelines do), so RX_VK_LOAD=create does not trust it and the next start builds them.
    • A stamp's folder is named by the program cache's stamp and the toolchain (folder_key: the compiler's version and the C++ library's), not the stamp alone: the program cache keeps its records' Shader::Info as it lies in memory, and the stamp has the library's version for libc++ and Microsoft's but none for libstdc++ (HumanBean's point on MagicBeanDev/LongshotLauncher#78). Before the store each version had its folder and its tools, so a cache never met another toolchain's; now one never does in the store either. The stamp itself is unchanged, so nobody's caches go.
    • An update in place, by default (hand_over): each game folder registers in the stamp it uses (<stamp>/users/, a file per folder naming it) and records it (cache/longshot_shader_store.txt). When a folder starts under another stamp (a new translator, another toolchain), the stamp it used before is renamed aside at once and deleted on a thread of its own, unless another game folder that is still there and still on it uses it, a running game holds it, or keeping was asked for (LONGSHOT_SHADER_CACHE_KEEP=1, --shader-cache-keep: both kept, for an A/B). So a folder holds one set of caches at a time, as without the store. Its old files are not reused under the new stamp: they are of no use to another translator, and a toolchain's change keeps the stamp while the records' layout may differ. A folder an earlier run set aside is swept at a later start. On Linux, where a rename is not refused under a running game as on Windows, let_go takes the stamp's .lock and keeps it over the rename, and hold checks the .lock it locked is still the one at its path (else it makes the folder again), so a game starting meanwhile never runs on a folder going aside; scripts/shader_store.py prune does the same. A Fallback start (another Longshot holds the stamp) leaves the folder's record as it was, so the stamp it used before is still let go at the next start that holds the store.
    • The rest of the clean-up is the launcher's (a deleted build's stamp, kept ones past what the player keeps): the record and <stamp>/longshot_last_used say what each folder uses and when a stamp was last used.
  • runtime: the stamp only once the device's translation settings are in (fb603f47): translation_settings_ready set right after coopmat_init (vk_main.cpp); program_cache_stamp() doesn't keep a stamp asked for before it (said once, taken again), and shader_cache_path doesn't open the store's part before it (a file asked for then is cache/'s, said once with its name). The loads' marker (longshot_load_start.txt) is the game folder's own now, on cache_path beside the driver's faults, as is its removal at a preparation.
  • runtime: gpu/vk/vk_paths.h shader_cache_path / shared_cache_path. vk_draw.cpp (the pipeline notes and what goes with them, the preparation's record, the pipeline cache's folder and its stamp check, the compute and state lists), vk_progcache.cpp disk_init (the program cache), vk_programs.cpp (the module ids) take their files through them. The module ids now go by path beside the programs: before, they used program_cache_path()'s folder, which is "" with the program cache off or held by another Longshot, and the file landed in the working folder. The game folder's own files stay on cache_path: the preparation-skipped marker, the driver's faults, the loader's notes go with the notes.
  • host: host/hos_main.c --shader-cache <folder|0> sets LONGSHOT_SHADER_CACHE for the run, over the environment (on Windows as long as the path is, not cut at MAX_PATH + 32 bytes), and --shader-cache-keep sets LONGSHOT_SHADER_CACHE_KEEP=1; both entry points skip their values, and host/options_path.c's scan of the arguments (#729) no longer takes either for the image.
  • build: nx2/make_game.py GAME_ENV_KEEP: the build's shader preparation keeps LONGSHOT_SHADER_CACHE and LONGSHOT_SHADER_CACHE_KEEP (the other LONGSHOT_ names still go), so the preparation fills the store the game will use; builder/linux/longshot-launch keeps it too.
  • dev: scripts/dev_packs.py gives every game it starts one store (~/.cache/longshot-shaders, %LOCALAPPDATA%\Longshot\shaders) unless LONGSHOT_SHADER_CACHE says otherwise; scripts/shader_store.py (new) ls lists a store's stamps (size, last use, in use) and prune --keep N lets go of the stamps no running game holds and no game folder still uses (its users/ and record; --game <folder> names more), the newest N of the rest kept.
  • docs: docs/development.md The shader cache store (the layout, what the game does with it, the A/B case, clean-up), the switch's row, the dev_packs bullet; the README's argument table; docs/design.md's cache note.
  • tests: tests/pipeline_play_cache_test.cpp's stand-ins have shader_cache_path beside cache_path (it compiles pcache_init and the stamp check cut from vk_draw.cpp). tests/test_shader_store.py with tests/test_shader_store.cpp (the header against temporary folders: off and on, the parts, what cache/ gives the store, the record, a part held; and the renderer's call sites, game_env, dev_packs' names), tests/test_shader_store_tool.py (prune against a lock a child process holds as the game does, and the users' markers). The hand-over's cases in tests/test_shader_store.cpp: the old stamp let go; kept for another folder still on it, with keep, or held; a stale marker not counted; a set-aside folder swept. tests/test_options_path.py: neither argument is taken for the image (it fails before the fix). tests/test_prepare_at_build.py follows GAME_ENV_KEEP in both the Python and the shell launcher.

Measurements

Verdict: works in the game on Windows. A second game folder of the same build, its cache/ empty, found the stamp's caches in the store and reached play in 78.9 s, preparing nothing; the same folder run beside the first (so on its own cache/) prepared every program, 1983 s. Two folders at once, another stamp, let go, kept: each did what Testing says.

In the game (this PR @ 680d1c83 on main @ 6c8d6e6a, Windows 11 Pro, Ryzen 9 3900X, 64 GB, RTX 3080, a hard drive; the game started headless from the repository's kokiri-start save with longshot_ctl.py launch --profile test, LONGSHOT_SHADER_CACHE set; time from launch to play):

Start Before: main @ 6c8d6e6a (no store: each folder its own cache/) After: this PR @ 680d1c83 (the log: vk: shader store: ...; time to play)
folder B: a second game folder of the same build, its cache/ empty prepares every program from nothing: 1983.3 s (B run beside A, so on its own cache/ as on main) the stamp's caches in the store: 78.9 s, nothing prepared
folder A, the first start with the store its own cache/ the stamp's caches in the store; took in 3 lists from its cache/; 297.8 s (prepared: see the note)
folder A again its own cache/ the stamp's caches in the store; 54.1 s
A and B at once each its own cache/ A: the store's, 33.3 s. B: "this run's shader caches stay in <B's folder>/cache/ (another Longshot uses the store's; taken in at a later start)"
A under another stamp (RX_VK_MISS_LOG=1, in the translator's stamp) the old stamp "stays: another game folder uses it"
B under it too, LONGSHOT_SHADER_CACHE_KEEP=1 the old stamp "kept (LONGSHOT_SHADER_CACHE_KEEP)"; shader_store.py ls: "nothing uses it"
A back the other stamp stays: B uses it
B back, no keep the other stamp "let go"; took in B's notes and 5 files from its cache/; "a program cache of this stamp stays in <B's folder>/cache/ (7440 MB): the store's own is read instead, and those files can be deleted"

Fixed since (fb603f47): folder A's first start took in the lists but not the stamped files because this PR took the program cache's stamp too early. The store's part opens at the first stamped file asked for, and the first was the loads' marker (load_fallback_path, beside the notes), asked for by programs_load_decide right after vkCreateDevice and before coopmat_init sets the translation's per-GPU settings the stamp holds. So the run's stamp was no build's, and a store never took in a folder's caches from a build. Measured on main @ 615b2498: main's build prepared cache/ with stamp 063fb3887f21b453; this PR before the fix took 74bd18fe7410cdf1, kept nothing and prepared everything; with the fix it takes 063fb3887f21b453, "took in 7 files of this stamp from <the folder's> cache/", and its program cache kept 20,603 keys (no preparation). Not the test profile, as an earlier comment guessed.

The game-free part follows.

Hardware: Windows 11 Pro, Ryzen 9 3900X, 64 GB; llvm-mingw clang 23 (the Builder's v2.0.14 tools)
Scene: none (game-free checks; a renderer-only build, RX_VK_ONLY)
Method: tests/test_shader_store.py --cxx <clang++> and tests/test_shader_store_tool.py on this branch; the checks pass or fail, so there is nothing to take a median of
How to reproduce: the two commands above; RX_VK_ONLY=1 python -I nx2/build.py --standalone for the renderer

Before: main @ d9486a08 After: this PR @ 70503f8a
shader caches of two game folders with the same translation two sets, each prepared (store: none) one set in the store (test_shader_store's checks)
a game folder's existing cache/ when the store is first used n/a moved in, not prepared again (the checks)
renderer build (RX_VK_ONLY) builds builds: 0 failed, longshot_vk.dll linked
shader store: ok
test_shader_store: ok
test_shader_store_tool: Ran 4 tests ... OK
compiled in 35s, 0 failed
linked ...\work\sa\build\longshot_vk.dll: 13 MB (0s)

Flags and settings

One switch and one argument for the same thing, off by default: the launcher PR turns it on for the builds it makes.

Flag What it is Before this PR After this PR Effect Helps / doesn't help / untested Recommendation
LONGSHOT_SHADER_CACHE=<folder> / --shader-cache <folder> the shader cache store (none) unset: the game folder's cache/ set: the shader caches shared by every game folder with the same stamp; a folder's old stamp let go when it moves to another helps: side-by-side builds, rebuilds into new folders, dev_packs runtimes; doesn't help: one folder updated in place (its cache/ stays anyway); untested: in the game off for now; set by the Longshot Launcher beside its builds (its PR), and by dev_packs.py
LONGSHOT_SHADER_CACHE_KEEP=1 / --shader-cache-keep keep a folder's old stamp (none) unset: let go when no other folder uses it set: kept, so going back to it prepares nothing helps: A/B of two translations in one folder; costs a whole set of caches each off: for development; the launcher's "keep older shader caches" sets it

Testing

Tested: the stamp fix in the game on Windows (fb603f47 on main @ 615b2498: main's build-prepared cache/ taken in by the store at the first start, the same stamp, no preparation); the 282 checks that read this diff on that head, not the whole suite (run_all.py --changed longshot/main): 240 passed, 32 skipped, 10 failed, the same 10 failing on a clean main @ 615b2498 (test_walk_key, test_auto_handheld, test_failed_compile_keeps_objects, test_missing_nup, test_tested_configurations, test_job_queue, test_make_game_cache, test_linux_preflight, test_vk_occlusion_queries, test_crash_dialog); test_shader_store fails with the old load_fallback_path put back. Before it, in the game on Windows (Measurements: two game folders and one store, both at once, another stamp let go and kept, this PR @ 680d1c83 on main @ 6c8d6e6a). tests/test_shader_store.py (the C++ checks with the Builder's llvm-mingw clang) and tests/test_shader_store_tool.py pass, Windows 11. python -I -B tests/run_all.py on this branch (before its rebase onto a00c485c, which adds only a skill file) and on main @ 9e6f231d, with the Builder's llvm-mingw clang (12 at once): this branch 542 passed, 18 failed, 35 skipped; main 540 passed, 18 failed, 35 skipped; the same 18 fail on both (test_appimage_player_build, test_cpm_discovery, test_cpp_preflight, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_options_budget_once, test_readback_late, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_walk_key). tests/test_shader_store.py also with g++ (libstdc++), for the library's branch of the folder's name. The renderer builds alone (RX_VK_ONLY). The Linux launcher's filter run under a shell: it drops LONGSHOT_X and keeps LONGSHOT_SHADER_CACHE.

On the rebased head (fea52a0b, HumanBean's rebase onto main @ 7bacf846): Windows with the Builder's llvm-mingw clang (6 at once), the 225 checks that read this diff (the tests that name a changed file, and the tests it adds or changes), not the whole suite: 189 passed, 14 skipped (tools this PC lacks, the Vulkan headers among them), 22 failed, each also failing on main: test_appimage_player_build, test_cpp_preflight, test_crash_dialog, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_tested_configurations, test_vk_pgo, test_walk_key in the full run on main @ 1310ecc6; and test_driver_cache_folder, test_linux_driver_fault_guard, test_linux_tools, test_linux_update, test_longshot_launch, test_longshot_saves, test_menu_save_atomic, test_pipeline_tour, test_udmabuf_warning, test_vk_occlusion_queries on main @ 7bacf846 (run there alone: they fail the same way). test_shader_store and test_shader_store_tool pass.

The 225 checks run

nx2/test_cbuf_table_extent.py, nx2/test_dump_ssbo_read.py, nx2/test_mesh_dynamic_state.py, nx2/test_nup_stand_in.py, nx2/test_nup_zstd_heads.py, tests/test_amd_fragment_warp.py, tests/test_appimage_headers.py, tests/test_appimage_player_build.py, tests/test_appimage_prompt_fonts.py, tests/test_appimage_scripts.py, tests/test_aspect_default.py, tests/test_audio_output_menu.py, tests/test_auto_settings.py, tests/test_bake_ds.py, tests/test_bank_gen.py, tests/test_bc_raw_targets.py, tests/test_bound_keys.py, tests/test_bug_report.py, tests/test_build_cleanup_scope.py, tests/test_build_hash_cache.py, tests/test_build_log.py, tests/test_build_times.py, tests/test_builder_drive_error.py, tests/test_camera_speed.py, tests/test_cheat_items.py, tests/test_check_no_compiler.py, tests/test_check_pr_description.py, tests/test_chunks_save_folder.py, tests/test_commit_room.py, tests/test_copy_writable.py, tests/test_cpp_preflight.py, tests/test_cpu_target.py, tests/test_crash_dialog.py, tests/test_crash_record.py, tests/test_damaged_check_progress.py, tests/test_debug_tools_linux.py, tests/test_desc_clock.py, tests/test_desc_counts.py, tests/test_dev_packs.py, tests/test_device_lost_exit.py, tests/test_disk_warning.py, tests/test_dispatch_clock.py, tests/test_display_modes.py, tests/test_display_startup_failure.py, tests/test_dlss_checks.py, tests/test_draw_binds.py, tests/test_draw_side_effects.py, tests/test_draw_stream_visibility.py, tests/test_drawn_page_map.py, tests/test_driver_cache_folder.py, tests/test_driver_cache_room.py, tests/test_driver_faults.py, tests/test_dummy_views.py, tests/test_env_file.py, tests/test_equal_depth_margin.py, tests/test_failed_compile_keeps_objects.py, tests/test_failure_reason.py, tests/test_fallback_buffer_cleanup.py, tests/test_fallback_depth_hold.py, tests/test_fallback_loaded.py, tests/test_fallback_pipeline_state.py, tests/test_fallback_size_policy.py, tests/test_ffx_sdk.py, tests/test_finish_output.py, tests/test_flat_map.py, tests/test_flight_yields_to_events.py, tests/test_forge_pr.py, tests/test_framegen_preset.py, tests/test_fsr_uniform_ownership.py, tests/test_global_table_loop.py, tests/test_gpl_cold_library.py, tests/test_gpl_library_cache.py, tests/test_handle_lookup.py, tests/test_handle_table_arrays.py, tests/test_hle_stats.py, tests/test_hold_gpu.py, tests/test_intel_compiler_gate.py, tests/test_intel_quick_compute.py, tests/test_jobs_failure.py, tests/test_jobs_wait_runs_queued.py, tests/test_kept_translation.py, tests/test_launcher_capture.py, tests/test_lift_cache.py, tests/test_lift_cleanup_error.py, tests/test_lift_cut_short.py, tests/test_lift_store.py, tests/test_link_path_comma.py, tests/test_linux_driver_fault_guard.py, tests/test_linux_preflight.py, tests/test_linux_pressure.py, tests/test_linux_saves_move.py, tests/test_linux_tools.py, tests/test_linux_update.py, tests/test_load_default.py, tests/test_longshot_ctl.py, tests/test_longshot_launch.py, tests/test_longshot_saves.py, tests/test_low_memory_notice.py, tests/test_make_game_cache.py, tests/test_make_game_ngx.py, tests/test_make_game_prepare.py, tests/test_make_game_writable.py, tests/test_mark_groups.py, tests/test_masked_clear_failure.py, tests/test_mem_trim.py, tests/test_menu_env_on.py, tests/test_menu_restart.py, tests/test_menu_save_atomic.py, tests/test_mesh_args_cleanup.py, tests/test_mesh_fallback.py, tests/test_mesh_fallback_failed_once.py, tests/test_mesh_query_feature.py, tests/test_met_wait_capture.py, tests/test_missing_nup.py, tests/test_module_ids.py, tests/test_movement_speed.py, tests/test_native_float_atomic_add.py, tests/test_no_release_lookup.py, tests/test_no_webgpu.py, tests/test_object_cache_prune.py, tests/test_object_store.py, tests/test_ocarina_note_bindings.py, tests/test_ocarina_notes.py, tests/test_oot_state_tap.py, tests/test_opening_skip.py, tests/test_options_path.py, tests/test_output_resolution.py, tests/test_pad_layout.py, tests/test_parallel_dry_marks.py, tests/test_parallel_rb_refresh.py, tests/test_pass_hazard.py, tests/test_pcache_fetch_lock.py, tests/test_pcache_header_read.py, tests/test_pcache_stamp.py, tests/test_pipeline_cache_save.py, tests/test_pipeline_notes_open.py, tests/test_pipeline_play_cache.py, tests/test_pipeline_share.py, tests/test_pipeline_state_computes.py, tests/test_pipeline_state_counts.py, tests/test_pipeline_state_portable.py, tests/test_pipeline_states.py, tests/test_pipeline_tour.py, tests/test_portable_builder_startup.py, tests/test_precompile_compute_guard.py, tests/test_precompile_pipeline_cleanup.py, tests/test_predict_resume.py, tests/test_predict_sees_loaded.py, tests/test_predict_types.py, tests/test_predicted_check.py, tests/test_prepare_at_build.py, tests/test_prepare_cache_save.py, tests/test_prepare_cpus.py, tests/test_prepare_seed_stamp.py, tests/test_prepare_stall.py, tests/test_prepare_workers.py, tests/test_prepared_stamps.py, tests/test_prof_sampling.py, tests/test_program_cache_cleanup.py, tests/test_program_cache_format.py, tests/test_program_cache_index.py, tests/test_program_cache_one_variant.py, tests/test_program_cache_reader.py, tests/test_program_life.py, tests/test_program_load_wait.py, tests/test_program_record_head.py, tests/test_program_stamp_knobs.py, tests/test_prompt_skip.py, tests/test_psychov35_dlss.py, tests/test_psychov35_taau.py, tests/test_push_state.py, tests/test_quick_pipelines.py, tests/test_radv_mesh_optimizer.py, tests/test_reactive_mask.py, tests/test_readme_menu_rows.py, tests/test_recomplauncher_json.py, tests/test_recorder_own_idle.py, tests/test_recorder_sync.py, tests/test_recording_admission.py, tests/test_replay_copies.py, tests/test_rumble_menu.py, tests/test_same_draw.py, tests/test_same_dyn.py, tests/test_sampler_admission.py, tests/test_shader_census_cmd.py, tests/test_shader_list_skip.py, tests/test_shader_module_guard.py, tests/test_shader_stamp_cache.py, tests/test_shader_stamp_inputs.py, tests/test_shader_store.py, tests/test_shader_store_tool.py, tests/test_shadow_mask_guard.py, tests/test_ship_state_list.py, tests/test_skip_title.py, tests/test_split_before_draw.py, tests/test_stale_hot_pieces.py, tests/test_startup_progress.py, tests/test_state_list_plain.py, tests/test_stick_dead_zone.py, tests/test_switches_removed.py, tests/test_system_header_cache.py, tests/test_taau_env_lookup.py, tests/test_taau_fxaa.py, tests/test_taau_typed_dispatch.py, tests/test_table_track.py, tests/test_teleport_minimap_flag.py, tests/test_tested_configurations.py, tests/test_texture_handle_banks.py, tests/test_texture_squeeze.py, tests/test_third_party_objects.py, tests/test_threaded_rendering.py, tests/test_tls_objects.py, tests/test_tree_shadow_mask.py, tests/test_ub_threads.py, tests/test_udmabuf_warning.py, tests/test_upscale_fallback.py, tests/test_upscaler_rows.py, tests/test_utf8_paths.py, tests/test_vertex_bind.py, tests/test_vk_occlusion_queries.py, tests/test_vk_pgo.py, tests/test_walk_key.py, tests/test_warm_bounded.py, tests/test_widescreen_shapes.py, tests/test_work_lock.py

On this head (70503f8a: rebased onto main @ d9486a08, with the answers to HumanBean's review): Windows, the 59 checks that read this diff (the tests that name a changed file), not the whole suite: 46 passed, 13 failed, each also failing on main @ d9486a08 when run there alone (test_bug_report, test_crash_dialog, test_crash_record, test_driver_cache_folder, test_driver_env_choose, test_log_open_failure, test_menu_save_atomic, test_oot_state_tap, test_pipeline_tour, test_udmabuf_warning, test_vk_occlusion_queries, test_vk_pgo, test_walk_key). tests/test_shader_store.py passes with g++ 10.3 and the Builder's llvm-mingw clang++ on Windows, and with g++ 14.2 on Linux (Docker python:3.12: the flock path of hold, let_go and a held stamp), and tests/test_shader_store_tool.py on Linux. Its new cases (the record a Fallback start leaves, .built after notes taken in with and without their pipeline cache, the program cache left in cache/ said) fail 5 checks against the header before this commit. host/hos_main.c compiles to an object for Windows with llvm-mingw clang, with the same 3 warnings as before (the change is Windows-only).

Host build (before the review's commit): compiled to objects on fea52a0b and on main @ 7bacf846, all clean on both: host/hos_main.c and host/options_path.c on Windows with the Builder's llvm-mingw clang (x86_64-w64-mingw32) and on Linux with gcc 14.2 and clang 19 (Docker python:3.12, host/posix on the include path); gpu/vk/vk_draw.cpp, gpu/vk/vk_progcache.cpp and gpu/vk/vk_programs.cpp with llvm-mingw clang++ and, on Linux, clang++ 19 (g++ refuses gpu/render/rx_nvn.h's _Static_assert in C++, on main too; the Linux build uses clang). The changed headers (shader_store.h, vk_paths.h) were compiled through those files. build.py's flags for one object, the runtime headers a lift makes (a64.h, a64_suyu.h, a64_fpx.h) copied from a local work folder, and for the renderer files the Builder's CPM Vulkan, SPIR-V and VMA headers. It covers the files this PR changes, not a whole game build.

Not tested:

  • The host in a whole game build: host/hos_main.c's --shader-cache compiles (above), and the first run under Must be tested builds and runs it.
  • Linux: the flock lock in the game (the store's checks run it game-free, above), and tests/test_longshot_launch.py (it runs only there).
  • A second run's program cache merged into the store (not done here, see What changed).

Must be tested:

  • Linux, any GPU, by hand: the same steps, expecting the same log lines and a second folder that prepares nothing; and, with both folders started at once, <store>/<stamp>/.lock held by the first (the flock lock), the second running from its own cache/; and the AppImage's Play button (longshot-launch) starting the game with the switch still set.

Risks

What could break: with the switch set, a folder's old stamp is deleted when it moves to another and no other existing folder still uses it: a folder whose record or marker went missing could let go of a stamp another build still wanted (that build then prepares again; the shared lists stay). With the switch set, a game that read or wrote a shader cache file in the wrong place would prepare again, or lose what it compiled in play; the call sites are all checked by text, and unset nothing moves. Two runs of one stamp could clash: each part is held for the process's life, and the second run uses its own cache/. A store on another drive than the game folder makes the first take-in a copy (several GB) instead of a rename. The store grows by a whole set of caches for each translation until something lets the unused ones go: the launcher PR for players, scripts/shader_store.py prune for development stores.

Turning it off: unset LONGSHOT_SHADER_CACHE, or start with --shader-cache 0: the game folder's cache/ again. A store's files stay in the store until pruned.

Compatibility: nothing changes for players until the launcher sets the switch: unset, every file is where it was, and nothing takes the stamp sooner than before. With it set, a game folder's existing caches of the same stamp move into the store at its first start. Saves, the options file and the vk ABI are unchanged.

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** medium · **Tested on:** Windows 11, RTX 3080, in the game (two game folders, one store) and game-free · **Needs:** a Linux run in the game (the flock lock, the AppImage's longshot-launch) ## Why Every game folder prepares its shaders on its own, about 25 minutes with Compile shaders All on a fast PC, even when another folder on the same PC translated the same shaders with the same translator a minute before: a stable and a nightly side by side, a rebuild into a new folder, each `dev_packs.py` runtime (which also carries a 10 GB copy of the caches). The caches are already stamped by what they are good for (`program_cache_stamp`), so a store keyed by that stamp lets every folder with the same translation use them in place. The design was agreed with the maintainers, including moving these files out of the game folder, where https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/321 put them. It complements https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/57 (an update in place keeps its `cache/` anyway) and replaces copying caches between folders (https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/78). It mirrors the object and lift stores (https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/371, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/372). No Longshot issue fits (the tracker searched). ## What changed With `LONGSHOT_SHADER_CACHE=<folder>` (or `--shader-cache <folder>`), the game keeps its shader caches in `<folder>/<stamp>/` for what one translation made and `<folder>/shared/` for the lists good for any build, so a game folder whose stamp is in the store starts with its programs and pipelines and prepares nothing. Unset, empty or `0`, every file stays in the game folder's `cache/` as before, and the stamp is not taken any sooner. Nothing sets it for players yet: a launcher PR follows, setting it beside its builds and cleaning up the stamps no build uses. - runtime: `gpu/vk/shader_store.h` (new, standard library and the platform's file lock, no Vulkan): - `open_stamped` (at the first stamped file asked for, where the game took the stamp before) and `open_shared` (at the first list); each part held by `<part>/.lock` for the process's life: open with no sharing on Windows, `flock` elsewhere, so a tool can tell a part in use whatever it calls the path. A second run of the same stamp keeps that part's files in its own `cache/` for that run. - `consume`: a game folder's `cache/` of the same stamp is taken in at a start that holds the store. Files the store lacks are moved in (a rename, else a copy as `.part` renamed in); the program-side files only when `cache/`'s program cache carries this stamp, the pipeline cache only when its stamp file does. Pipeline notes and lists the store has get the records it lacks (`merge_records`, by their heads: the notes' stamp and record size, each list's magic and layout). A pipeline cache the store has a main file for goes in as a play file, merged at the next start by `play_files_merge`. A second run's program cache is left in its `cache/`: merging two program caches record by record is not done here; the log says so, with its size, so those files can go. The precompile's record (`longshot_pipelines.bin.built`) moves with the notes alone: notes taken in without their pipeline cache take the store's record away (as `state_list_prepare`'s notes without pipelines do), so `RX_VK_LOAD=create` does not trust it and the next start builds them. - A stamp's folder is named by the program cache's stamp and the toolchain (`folder_key`: the compiler's version and the C++ library's), not the stamp alone: the program cache keeps its records' `Shader::Info` as it lies in memory, and the stamp has the library's version for libc++ and Microsoft's but none for libstdc++ (HumanBean's point on https://git.magicbean.dev/MagicBeanDev/LongshotLauncher/pulls/78). Before the store each version had its folder and its tools, so a cache never met another toolchain's; now one never does in the store either. The stamp itself is unchanged, so nobody's caches go. - An update in place, by default (`hand_over`): each game folder registers in the stamp it uses (`<stamp>/users/`, a file per folder naming it) and records it (`cache/longshot_shader_store.txt`). When a folder starts under another stamp (a new translator, another toolchain), the stamp it used before is renamed aside at once and deleted on a thread of its own, unless another game folder that is still there and still on it uses it, a running game holds it, or keeping was asked for (`LONGSHOT_SHADER_CACHE_KEEP=1`, `--shader-cache-keep`: both kept, for an A/B). So a folder holds one set of caches at a time, as without the store. Its old files are not reused under the new stamp: they are of no use to another translator, and a toolchain's change keeps the stamp while the records' layout may differ. A folder an earlier run set aside is swept at a later start. On Linux, where a rename is not refused under a running game as on Windows, `let_go` takes the stamp's `.lock` and keeps it over the rename, and `hold` checks the `.lock` it locked is still the one at its path (else it makes the folder again), so a game starting meanwhile never runs on a folder going aside; `scripts/shader_store.py prune` does the same. A Fallback start (another Longshot holds the stamp) leaves the folder's record as it was, so the stamp it used before is still let go at the next start that holds the store. - The rest of the clean-up is the launcher's (a deleted build's stamp, kept ones past what the player keeps): the record and `<stamp>/longshot_last_used` say what each folder uses and when a stamp was last used. - runtime: the stamp only once the device's translation settings are in (fb603f47): `translation_settings_ready` set right after `coopmat_init` (`vk_main.cpp`); `program_cache_stamp()` doesn't keep a stamp asked for before it (said once, taken again), and `shader_cache_path` doesn't open the store's part before it (a file asked for then is `cache/`'s, said once with its name). The loads' marker (`longshot_load_start.txt`) is the game folder's own now, on `cache_path` beside the driver's faults, as is its removal at a preparation. - runtime: `gpu/vk/vk_paths.h` `shader_cache_path` / `shared_cache_path`. `vk_draw.cpp` (the pipeline notes and what goes with them, the preparation's record, the pipeline cache's folder and its stamp check, the compute and state lists), `vk_progcache.cpp` `disk_init` (the program cache), `vk_programs.cpp` (the module ids) take their files through them. The module ids now go by path beside the programs: before, they used `program_cache_path()`'s folder, which is "" with the program cache off or held by another Longshot, and the file landed in the working folder. The game folder's own files stay on `cache_path`: the preparation-skipped marker, the driver's faults, the loader's notes go with the notes. - host: `host/hos_main.c` `--shader-cache <folder|0>` sets `LONGSHOT_SHADER_CACHE` for the run, over the environment (on Windows as long as the path is, not cut at `MAX_PATH + 32` bytes), and `--shader-cache-keep` sets `LONGSHOT_SHADER_CACHE_KEEP=1`; both entry points skip their values, and `host/options_path.c`'s scan of the arguments (#729) no longer takes either for the image. - build: `nx2/make_game.py` `GAME_ENV_KEEP`: the build's shader preparation keeps `LONGSHOT_SHADER_CACHE` and `LONGSHOT_SHADER_CACHE_KEEP` (the other `LONGSHOT_` names still go), so the preparation fills the store the game will use; `builder/linux/longshot-launch` keeps it too. - dev: `scripts/dev_packs.py` gives every game it starts one store (`~/.cache/longshot-shaders`, `%LOCALAPPDATA%\Longshot\shaders`) unless `LONGSHOT_SHADER_CACHE` says otherwise; `scripts/shader_store.py` (new) `ls` lists a store's stamps (size, last use, in use) and `prune --keep N` lets go of the stamps no running game holds and no game folder still uses (its users/ and record; `--game <folder>` names more), the newest N of the rest kept. - docs: `docs/development.md` *The shader cache store* (the layout, what the game does with it, the A/B case, clean-up), the switch's row, the dev_packs bullet; the README's argument table; `docs/design.md`'s cache note. - tests: `tests/pipeline_play_cache_test.cpp`'s stand-ins have `shader_cache_path` beside `cache_path` (it compiles `pcache_init` and the stamp check cut from `vk_draw.cpp`). `tests/test_shader_store.py` with `tests/test_shader_store.cpp` (the header against temporary folders: off and on, the parts, what `cache/` gives the store, the record, a part held; and the renderer's call sites, `game_env`, dev_packs' names), `tests/test_shader_store_tool.py` (prune against a lock a child process holds as the game does, and the users' markers). The hand-over's cases in `tests/test_shader_store.cpp`: the old stamp let go; kept for another folder still on it, with keep, or held; a stale marker not counted; a set-aside folder swept. `tests/test_options_path.py`: neither argument is taken for the image (it fails before the fix). `tests/test_prepare_at_build.py` follows `GAME_ENV_KEEP` in both the Python and the shell launcher. ## Measurements **Verdict:** works in the game on Windows. A second game folder of the same build, its `cache/` empty, found the stamp's caches in the store and reached play in **78.9 s, preparing nothing**; the same folder run beside the first (so on its own `cache/`) prepared every program, **1983 s**. Two folders at once, another stamp, let go, kept: each did what Testing says. **In the game** (this PR @ 680d1c83 on main @ 6c8d6e6a, Windows 11 Pro, Ryzen 9 3900X, 64 GB, RTX 3080, a hard drive; the game started headless from the repository's `kokiri-start` save with `longshot_ctl.py launch --profile test`, `LONGSHOT_SHADER_CACHE` set; time from launch to play): | Start | Before: main @ 6c8d6e6a (no store: each folder its own `cache/`) | After: this PR @ 680d1c83 (the log: `vk: shader store: ...`; time to play) | |---|---|---| | **folder B**: a second game folder of the same build, its `cache/` empty | prepares every program from nothing: 1983.3 s (B run beside A, so on its own `cache/` as on main) | the stamp's caches in the store: **78.9 s, nothing prepared** | | folder A, the first start with the store | its own `cache/` | the stamp's caches in the store; took in 3 lists from its `cache/`; 297.8 s (prepared: see the note) | | folder A again | its own `cache/` | the stamp's caches in the store; 54.1 s | | A and B at once | each its own `cache/` | A: the store's, 33.3 s. B: "this run's shader caches stay in <B's folder>/cache/ (another Longshot uses the store's; taken in at a later start)" | | A under another stamp (`RX_VK_MISS_LOG=1`, in the translator's stamp) | | the old stamp "stays: another game folder uses it" | | B under it too, `LONGSHOT_SHADER_CACHE_KEEP=1` | | the old stamp "kept (LONGSHOT_SHADER_CACHE_KEEP)"; `shader_store.py ls`: "nothing uses it" | | A back | | the other stamp stays: B uses it | | B back, no keep | | the other stamp "let go"; took in B's notes and 5 files from its `cache/`; "a program cache of this stamp stays in <B's folder>/cache/ (7440 MB): the store's own is read instead, and those files can be deleted" | Fixed since (fb603f47): folder A's first start took in the lists but not the stamped files because this PR took the program cache's stamp too early. The store's part opens at the first stamped file asked for, and the first was the loads' marker (`load_fallback_path`, beside the notes), asked for by `programs_load_decide` right after `vkCreateDevice` and before `coopmat_init` sets the translation's per-GPU settings the stamp holds. So the run's stamp was no build's, and a store never took in a folder's caches from a build. Measured on main @ 615b2498: main's build prepared `cache/` with stamp 063fb3887f21b453; this PR before the fix took 74bd18fe7410cdf1, kept nothing and prepared everything; with the fix it takes 063fb3887f21b453, "took in 7 files of this stamp from <the folder's> cache/", and its program cache kept 20,603 keys (no preparation). Not the test profile, as an earlier comment guessed. The game-free part follows. Hardware: Windows 11 Pro, Ryzen 9 3900X, 64 GB; llvm-mingw clang 23 (the Builder's v2.0.14 tools) Scene: none (game-free checks; a renderer-only build, `RX_VK_ONLY`) Method: `tests/test_shader_store.py --cxx <clang++>` and `tests/test_shader_store_tool.py` on this branch; the checks pass or fail, so there is nothing to take a median of How to reproduce: the two commands above; `RX_VK_ONLY=1 python -I nx2/build.py --standalone` for the renderer | | Before: main @ d9486a08 | After: this PR @ 70503f8a | |---|---|---| | shader caches of two game folders with the same translation | two sets, each prepared (store: none) | one set in the store (test_shader_store's checks) | | a game folder's existing `cache/` when the store is first used | n/a | moved in, not prepared again (the checks) | | renderer build (`RX_VK_ONLY`) | builds | builds: 0 failed, `longshot_vk.dll` linked | ``` shader store: ok test_shader_store: ok test_shader_store_tool: Ran 4 tests ... OK compiled in 35s, 0 failed linked ...\work\sa\build\longshot_vk.dll: 13 MB (0s) ``` ### Flags and settings One switch and one argument for the same thing, off by default: the launcher PR turns it on for the builds it makes. | Flag | What it is | Before this PR | After this PR | Effect | Helps / doesn't help / untested | Recommendation | |---|---|---|---|---|---|---| | `LONGSHOT_SHADER_CACHE=<folder>` / `--shader-cache <folder>` | the shader cache store | (none) | unset: the game folder's `cache/` | set: the shader caches shared by every game folder with the same stamp; a folder's old stamp let go when it moves to another | helps: side-by-side builds, rebuilds into new folders, dev_packs runtimes; doesn't help: one folder updated in place (its `cache/` stays anyway); untested: in the game | off for now; set by the Longshot Launcher beside its builds (its PR), and by dev_packs.py | | `LONGSHOT_SHADER_CACHE_KEEP=1` / `--shader-cache-keep` | keep a folder's old stamp | (none) | unset: let go when no other folder uses it | set: kept, so going back to it prepares nothing | helps: A/B of two translations in one folder; costs a whole set of caches each | off: for development; the launcher's "keep older shader caches" sets it | ## Testing **Tested:** the stamp fix in the game on Windows (fb603f47 on main @ 615b2498: main's build-prepared `cache/` taken in by the store at the first start, the same stamp, no preparation); the 282 checks that read this diff on that head, not the whole suite (`run_all.py --changed longshot/main`): 240 passed, 32 skipped, 10 failed, the same 10 failing on a clean main @ 615b2498 (`test_walk_key`, `test_auto_handheld`, `test_failed_compile_keeps_objects`, `test_missing_nup`, `test_tested_configurations`, `test_job_queue`, `test_make_game_cache`, `test_linux_preflight`, `test_vk_occlusion_queries`, `test_crash_dialog`); `test_shader_store` fails with the old `load_fallback_path` put back. Before it, in the game on Windows (Measurements: two game folders and one store, both at once, another stamp let go and kept, this PR @ 680d1c83 on main @ 6c8d6e6a). `tests/test_shader_store.py` (the C++ checks with the Builder's llvm-mingw clang) and `tests/test_shader_store_tool.py` pass, Windows 11. `python -I -B tests/run_all.py` on this branch (before its rebase onto a00c485c, which adds only a skill file) and on main @ 9e6f231d, with the Builder's llvm-mingw clang (12 at once): this branch 542 passed, 18 failed, 35 skipped; main 540 passed, 18 failed, 35 skipped; the same 18 fail on both (test_appimage_player_build, test_cpm_discovery, test_cpp_preflight, test_crash_dialog, test_env_on, test_fpx_sticky, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_oot_state_tap, test_options_budget_once, test_readback_late, test_spawn_emit, test_tested_configurations, test_vk_pgo, test_walk_key). `tests/test_shader_store.py` also with g++ (libstdc++), for the library's branch of the folder's name. The renderer builds alone (`RX_VK_ONLY`). The Linux launcher's filter run under a shell: it drops `LONGSHOT_X` and keeps `LONGSHOT_SHADER_CACHE`. **On the rebased head (fea52a0b, HumanBean's rebase onto main @ 7bacf846):** Windows with the Builder's llvm-mingw clang (6 at once), the 225 checks that read this diff (the tests that name a changed file, and the tests it adds or changes), not the whole suite: 189 passed, 14 skipped (tools this PC lacks, the Vulkan headers among them), 22 failed, each also failing on main: `test_appimage_player_build`, `test_cpp_preflight`, `test_crash_dialog`, `test_lift_cleanup_error`, `test_linux_preflight`, `test_make_game_cache`, `test_make_game_prepare`, `test_missing_nup`, `test_oot_state_tap`, `test_tested_configurations`, `test_vk_pgo`, `test_walk_key` in the full run on main @ 1310ecc6; and `test_driver_cache_folder`, `test_linux_driver_fault_guard`, `test_linux_tools`, `test_linux_update`, `test_longshot_launch`, `test_longshot_saves`, `test_menu_save_atomic`, `test_pipeline_tour`, `test_udmabuf_warning`, `test_vk_occlusion_queries` on main @ 7bacf846 (run there alone: they fail the same way). `test_shader_store` and `test_shader_store_tool` pass. <details><summary>The 225 checks run</summary> `nx2/test_cbuf_table_extent.py`, `nx2/test_dump_ssbo_read.py`, `nx2/test_mesh_dynamic_state.py`, `nx2/test_nup_stand_in.py`, `nx2/test_nup_zstd_heads.py`, `tests/test_amd_fragment_warp.py`, `tests/test_appimage_headers.py`, `tests/test_appimage_player_build.py`, `tests/test_appimage_prompt_fonts.py`, `tests/test_appimage_scripts.py`, `tests/test_aspect_default.py`, `tests/test_audio_output_menu.py`, `tests/test_auto_settings.py`, `tests/test_bake_ds.py`, `tests/test_bank_gen.py`, `tests/test_bc_raw_targets.py`, `tests/test_bound_keys.py`, `tests/test_bug_report.py`, `tests/test_build_cleanup_scope.py`, `tests/test_build_hash_cache.py`, `tests/test_build_log.py`, `tests/test_build_times.py`, `tests/test_builder_drive_error.py`, `tests/test_camera_speed.py`, `tests/test_cheat_items.py`, `tests/test_check_no_compiler.py`, `tests/test_check_pr_description.py`, `tests/test_chunks_save_folder.py`, `tests/test_commit_room.py`, `tests/test_copy_writable.py`, `tests/test_cpp_preflight.py`, `tests/test_cpu_target.py`, `tests/test_crash_dialog.py`, `tests/test_crash_record.py`, `tests/test_damaged_check_progress.py`, `tests/test_debug_tools_linux.py`, `tests/test_desc_clock.py`, `tests/test_desc_counts.py`, `tests/test_dev_packs.py`, `tests/test_device_lost_exit.py`, `tests/test_disk_warning.py`, `tests/test_dispatch_clock.py`, `tests/test_display_modes.py`, `tests/test_display_startup_failure.py`, `tests/test_dlss_checks.py`, `tests/test_draw_binds.py`, `tests/test_draw_side_effects.py`, `tests/test_draw_stream_visibility.py`, `tests/test_drawn_page_map.py`, `tests/test_driver_cache_folder.py`, `tests/test_driver_cache_room.py`, `tests/test_driver_faults.py`, `tests/test_dummy_views.py`, `tests/test_env_file.py`, `tests/test_equal_depth_margin.py`, `tests/test_failed_compile_keeps_objects.py`, `tests/test_failure_reason.py`, `tests/test_fallback_buffer_cleanup.py`, `tests/test_fallback_depth_hold.py`, `tests/test_fallback_loaded.py`, `tests/test_fallback_pipeline_state.py`, `tests/test_fallback_size_policy.py`, `tests/test_ffx_sdk.py`, `tests/test_finish_output.py`, `tests/test_flat_map.py`, `tests/test_flight_yields_to_events.py`, `tests/test_forge_pr.py`, `tests/test_framegen_preset.py`, `tests/test_fsr_uniform_ownership.py`, `tests/test_global_table_loop.py`, `tests/test_gpl_cold_library.py`, `tests/test_gpl_library_cache.py`, `tests/test_handle_lookup.py`, `tests/test_handle_table_arrays.py`, `tests/test_hle_stats.py`, `tests/test_hold_gpu.py`, `tests/test_intel_compiler_gate.py`, `tests/test_intel_quick_compute.py`, `tests/test_jobs_failure.py`, `tests/test_jobs_wait_runs_queued.py`, `tests/test_kept_translation.py`, `tests/test_launcher_capture.py`, `tests/test_lift_cache.py`, `tests/test_lift_cleanup_error.py`, `tests/test_lift_cut_short.py`, `tests/test_lift_store.py`, `tests/test_link_path_comma.py`, `tests/test_linux_driver_fault_guard.py`, `tests/test_linux_preflight.py`, `tests/test_linux_pressure.py`, `tests/test_linux_saves_move.py`, `tests/test_linux_tools.py`, `tests/test_linux_update.py`, `tests/test_load_default.py`, `tests/test_longshot_ctl.py`, `tests/test_longshot_launch.py`, `tests/test_longshot_saves.py`, `tests/test_low_memory_notice.py`, `tests/test_make_game_cache.py`, `tests/test_make_game_ngx.py`, `tests/test_make_game_prepare.py`, `tests/test_make_game_writable.py`, `tests/test_mark_groups.py`, `tests/test_masked_clear_failure.py`, `tests/test_mem_trim.py`, `tests/test_menu_env_on.py`, `tests/test_menu_restart.py`, `tests/test_menu_save_atomic.py`, `tests/test_mesh_args_cleanup.py`, `tests/test_mesh_fallback.py`, `tests/test_mesh_fallback_failed_once.py`, `tests/test_mesh_query_feature.py`, `tests/test_met_wait_capture.py`, `tests/test_missing_nup.py`, `tests/test_module_ids.py`, `tests/test_movement_speed.py`, `tests/test_native_float_atomic_add.py`, `tests/test_no_release_lookup.py`, `tests/test_no_webgpu.py`, `tests/test_object_cache_prune.py`, `tests/test_object_store.py`, `tests/test_ocarina_note_bindings.py`, `tests/test_ocarina_notes.py`, `tests/test_oot_state_tap.py`, `tests/test_opening_skip.py`, `tests/test_options_path.py`, `tests/test_output_resolution.py`, `tests/test_pad_layout.py`, `tests/test_parallel_dry_marks.py`, `tests/test_parallel_rb_refresh.py`, `tests/test_pass_hazard.py`, `tests/test_pcache_fetch_lock.py`, `tests/test_pcache_header_read.py`, `tests/test_pcache_stamp.py`, `tests/test_pipeline_cache_save.py`, `tests/test_pipeline_notes_open.py`, `tests/test_pipeline_play_cache.py`, `tests/test_pipeline_share.py`, `tests/test_pipeline_state_computes.py`, `tests/test_pipeline_state_counts.py`, `tests/test_pipeline_state_portable.py`, `tests/test_pipeline_states.py`, `tests/test_pipeline_tour.py`, `tests/test_portable_builder_startup.py`, `tests/test_precompile_compute_guard.py`, `tests/test_precompile_pipeline_cleanup.py`, `tests/test_predict_resume.py`, `tests/test_predict_sees_loaded.py`, `tests/test_predict_types.py`, `tests/test_predicted_check.py`, `tests/test_prepare_at_build.py`, `tests/test_prepare_cache_save.py`, `tests/test_prepare_cpus.py`, `tests/test_prepare_seed_stamp.py`, `tests/test_prepare_stall.py`, `tests/test_prepare_workers.py`, `tests/test_prepared_stamps.py`, `tests/test_prof_sampling.py`, `tests/test_program_cache_cleanup.py`, `tests/test_program_cache_format.py`, `tests/test_program_cache_index.py`, `tests/test_program_cache_one_variant.py`, `tests/test_program_cache_reader.py`, `tests/test_program_life.py`, `tests/test_program_load_wait.py`, `tests/test_program_record_head.py`, `tests/test_program_stamp_knobs.py`, `tests/test_prompt_skip.py`, `tests/test_psychov35_dlss.py`, `tests/test_psychov35_taau.py`, `tests/test_push_state.py`, `tests/test_quick_pipelines.py`, `tests/test_radv_mesh_optimizer.py`, `tests/test_reactive_mask.py`, `tests/test_readme_menu_rows.py`, `tests/test_recomplauncher_json.py`, `tests/test_recorder_own_idle.py`, `tests/test_recorder_sync.py`, `tests/test_recording_admission.py`, `tests/test_replay_copies.py`, `tests/test_rumble_menu.py`, `tests/test_same_draw.py`, `tests/test_same_dyn.py`, `tests/test_sampler_admission.py`, `tests/test_shader_census_cmd.py`, `tests/test_shader_list_skip.py`, `tests/test_shader_module_guard.py`, `tests/test_shader_stamp_cache.py`, `tests/test_shader_stamp_inputs.py`, `tests/test_shader_store.py`, `tests/test_shader_store_tool.py`, `tests/test_shadow_mask_guard.py`, `tests/test_ship_state_list.py`, `tests/test_skip_title.py`, `tests/test_split_before_draw.py`, `tests/test_stale_hot_pieces.py`, `tests/test_startup_progress.py`, `tests/test_state_list_plain.py`, `tests/test_stick_dead_zone.py`, `tests/test_switches_removed.py`, `tests/test_system_header_cache.py`, `tests/test_taau_env_lookup.py`, `tests/test_taau_fxaa.py`, `tests/test_taau_typed_dispatch.py`, `tests/test_table_track.py`, `tests/test_teleport_minimap_flag.py`, `tests/test_tested_configurations.py`, `tests/test_texture_handle_banks.py`, `tests/test_texture_squeeze.py`, `tests/test_third_party_objects.py`, `tests/test_threaded_rendering.py`, `tests/test_tls_objects.py`, `tests/test_tree_shadow_mask.py`, `tests/test_ub_threads.py`, `tests/test_udmabuf_warning.py`, `tests/test_upscale_fallback.py`, `tests/test_upscaler_rows.py`, `tests/test_utf8_paths.py`, `tests/test_vertex_bind.py`, `tests/test_vk_occlusion_queries.py`, `tests/test_vk_pgo.py`, `tests/test_walk_key.py`, `tests/test_warm_bounded.py`, `tests/test_widescreen_shapes.py`, `tests/test_work_lock.py` </details> **On this head (70503f8a: rebased onto main @ d9486a08, with the answers to HumanBean's review):** Windows, the 59 checks that read this diff (the tests that name a changed file), not the whole suite: 46 passed, 13 failed, each also failing on main @ d9486a08 when run there alone (`test_bug_report`, `test_crash_dialog`, `test_crash_record`, `test_driver_cache_folder`, `test_driver_env_choose`, `test_log_open_failure`, `test_menu_save_atomic`, `test_oot_state_tap`, `test_pipeline_tour`, `test_udmabuf_warning`, `test_vk_occlusion_queries`, `test_vk_pgo`, `test_walk_key`). `tests/test_shader_store.py` passes with g++ 10.3 and the Builder's llvm-mingw clang++ on Windows, and with g++ 14.2 on Linux (Docker `python:3.12`: the `flock` path of `hold`, `let_go` and a held stamp), and `tests/test_shader_store_tool.py` on Linux. Its new cases (the record a Fallback start leaves, `.built` after notes taken in with and without their pipeline cache, the program cache left in `cache/` said) fail 5 checks against the header before this commit. `host/hos_main.c` compiles to an object for Windows with llvm-mingw clang, with the same 3 warnings as before (the change is Windows-only). **Host build (before the review's commit):** compiled to objects on fea52a0b and on main @ 7bacf846, all clean on both: `host/hos_main.c` and `host/options_path.c` on Windows with the Builder's llvm-mingw clang (`x86_64-w64-mingw32`) and on Linux with gcc 14.2 and clang 19 (Docker `python:3.12`, `host/posix` on the include path); `gpu/vk/vk_draw.cpp`, `gpu/vk/vk_progcache.cpp` and `gpu/vk/vk_programs.cpp` with llvm-mingw clang++ and, on Linux, clang++ 19 (g++ refuses `gpu/render/rx_nvn.h`'s `_Static_assert` in C++, on main too; the Linux build uses clang). The changed headers (`shader_store.h`, `vk_paths.h`) were compiled through those files. build.py's flags for one object, the runtime headers a lift makes (`a64.h`, `a64_suyu.h`, `a64_fpx.h`) copied from a local work folder, and for the renderer files the Builder's CPM Vulkan, SPIR-V and VMA headers. It covers the files this PR changes, not a whole game build. **Not tested:** - The host in a whole game build: `host/hos_main.c`'s `--shader-cache` compiles (above), and the first run under Must be tested builds and runs it. - Linux: the `flock` lock in the game (the store's checks run it game-free, above), and `tests/test_longshot_launch.py` (it runs only there). - A second run's program cache merged into the store (not done here, see What changed). **Must be tested:** - Linux, any GPU, by hand: the same steps, expecting the same log lines and a second folder that prepares nothing; and, with both folders started at once, `<store>/<stamp>/.lock` held by the first (the `flock` lock), the second running from its own `cache/`; and the AppImage's Play button (`longshot-launch`) starting the game with the switch still set. ## Risks **What could break:** with the switch set, a folder's old stamp is deleted when it moves to another and no other existing folder still uses it: a folder whose record or marker went missing could let go of a stamp another build still wanted (that build then prepares again; the shared lists stay). With the switch set, a game that read or wrote a shader cache file in the wrong place would prepare again, or lose what it compiled in play; the call sites are all checked by text, and unset nothing moves. Two runs of one stamp could clash: each part is held for the process's life, and the second run uses its own `cache/`. A store on another drive than the game folder makes the first take-in a copy (several GB) instead of a rename. The store grows by a whole set of caches for each translation until something lets the unused ones go: the launcher PR for players, `scripts/shader_store.py prune` for development stores. **Turning it off:** unset `LONGSHOT_SHADER_CACHE`, or start with `--shader-cache 0`: the game folder's `cache/` again. A store's files stay in the store until pruned. **Compatibility:** nothing changes for players until the launcher sets the switch: unset, every file is where it was, and nothing takes the stamp sooner than before. With it set, a game folder's existing caches of the same stamp move into the store at its first start. Saves, the options file and the vk ABI are unchanged. ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] Switches documented (docs/development.md; the README for a menu row) and shipped at their recommended value. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
CI / Conventional Commit title (pull_request) Successful in 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
Third-party parts / As their notes say (pull_request) Successful in 9s
b34e61c30b
LONGSHOT_SHADER_CACHE=<folder> (or --shader-cache <folder>): the game keeps
its shader caches in a store every game folder shares (gpu/vk/shader_store.h).
<store>/<stamp>/ holds what one translation made (the programs and their
index, the module ids, the pipeline notes and predictions, the preparation's
record, the driver's pipeline cache and its play files), by the program
cache's stamp, so a game folder whose stamp is there starts with them: no
preparation, no copy. <store>/shared/ holds the lists good for any build (the
compute and pipeline-state lists). Unset, empty or 0: the game folder's
cache/, as before, and nothing takes the stamp any sooner.

- The renderer takes those files by shader_cache_path / shared_cache_path
  (vk_paths.h); the store's part for the stamp opens at the first of them,
  where the game took the stamp before. The game folder's own markers, the
  driver's faults and the button prompts stay in cache/. The module ids now
  go beside the programs by path, not by program_cache_path's folder, which
  was "" with the program cache off or held, and put the file in the
  working folder.
- One game at a time on each part (<part>/.lock: open with no sharing on
  Windows, flock elsewhere, so a tool can tell); a second run of the same
  stamp keeps that part's files in its own cache/.
- A game folder's cache/ of the same stamp is taken in at the start that
  holds the store: files it lacks moved in (rename, else copy), notes and
  lists merged record by record, a pipeline cache as a play file. A second
  run's program cache stays in its cache/ (no record-by-record merge yet).
- The game never deletes from the store: it records the stamp it used
  (cache/longshot_shader_store.txt) and the stamp's last use, for the
  launcher. scripts/shader_store.py ls / prune --keep N cleans a
  development store.
- make_game.py's preparation and builder/linux/longshot-launch keep
  LONGSHOT_SHADER_CACHE (GAME_ENV_KEEP), so a build's preparation fills the
  store the game uses. scripts/dev_packs.py gives its games one store.

tests/test_shader_store.py (+ .cpp: the header against temporary folders:
off and on, the parts, what cache/ gives, the record, a part held),
tests/test_shader_store_tool.py (prune against a lock a child holds);
test_prepare_at_build.py follows GAME_ENV_KEEP. Docs: development.md's
"The shader cache store", the switch row, the dev_packs bullet; the
README's argument; design.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dyst force-pushed feat/shader-cache-store from b34e61c30b
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 8s
CI / Conventional Commit title (pull_request) Successful in 9s
CI / Workflow lint (actionlint) (pull_request) Successful in 13s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 12s
Third-party parts / As their notes say (pull_request) Successful in 9s
to 0e8569be74
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 4s
Third-party parts / As their notes say (pull_request) Successful in 8s
2026-11-15 08:29:16 +00:00
Compare
Author
Member

Rebased onto main 9e6f231d (past #729, which added options_path beside the store's paths in vk_paths.h: both kept). New: a stamp's folder is named by the stamp and the toolchain (folder_key: the compiler's version and the C++ library's), so caches never cross a toolchain in the store; HumanBean's point on LongshotLauncher#78 about libstdc++ (the stamp carries libc++'s and Microsoft's library version, not libstdc++'s) applies here too. The stamp itself is unchanged, so nobody's caches go. The record now says both (stamp=, translation=). Full suite again on this branch and main: no new failure; the store's check passes with clang and g++.

Rebased onto main 9e6f231d (past #729, which added options_path beside the store's paths in vk_paths.h: both kept). New: a stamp's folder is named by the stamp and the toolchain (folder_key: the compiler's version and the C++ library's), so caches never cross a toolchain in the store; HumanBean's point on LongshotLauncher#78 about libstdc++ (the stamp carries libc++'s and Microsoft's library version, not libstdc++'s) applies here too. The stamp itself is unchanged, so nobody's caches go. The record now says both (stamp=<folder>, translation=<program cache stamp>). Full suite again on this branch and main: no new failure; the store's check passes with clang and g++.
dyst force-pushed feat/shader-cache-store from 0e8569be74
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 5s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 4s
Third-party parts / As their notes say (pull_request) Successful in 8s
to a45e8c89d0
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 16s
CI / Conventional Commit title (pull_request) Successful in 13s
CI / Workflow lint (actionlint) (pull_request) Successful in 20s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 18s
Third-party parts / As their notes say (pull_request) Successful in 29s
Website / Website on Cloudflare (pull_request_target) Has been skipped
2026-11-15 09:35:48 +00:00
Compare
Author
Member

Pushed a45e8c89 (rebased onto main a00c485c): an update in place, in the store. Each game folder registers in the stamp it uses (/users/) and records it; when a folder starts under another stamp, its old one is renamed aside and deleted in the background unless another game folder still on it uses it, a running game holds it, or LONGSHOT_SHADER_CACHE_KEEP=1 / --shader-cache-keep asks to keep both (an A/B). So a folder holds one set of caches at a time, as without the store; the old files are not reused under the new stamp (another toolchain keeps the stamp, not the records' layout). Also fixed: host/options_path.c (#729) took --shader-cache's value for the image; it now skips both arguments, with a case in test_options_path.py that fails before. scripts/shader_store.py prune now reads the stamps' users itself. Renderer builds; full suite: no failure main doesn't have. Still WIP for the whole build (the host change) and the runs in the game under Must be tested.

Pushed a45e8c89 (rebased onto main a00c485c): an update in place, in the store. Each game folder registers in the stamp it uses (<stamp>/users/) and records it; when a folder starts under another stamp, its old one is renamed aside and deleted in the background unless another game folder still on it uses it, a running game holds it, or LONGSHOT_SHADER_CACHE_KEEP=1 / --shader-cache-keep asks to keep both (an A/B). So a folder holds one set of caches at a time, as without the store; the old files are not reused under the new stamp (another toolchain keeps the stamp, not the records' layout). Also fixed: host/options_path.c (#729) took --shader-cache's value for the image; it now skips both arguments, with a case in test_options_path.py that fails before. scripts/shader_store.py prune now reads the stamps' users itself. Renderer builds; full suite: no failure main doesn't have. Still WIP for the whole build (the host change) and the runs in the game under Must be tested.
HumanBean force-pushed feat/shader-cache-store from a45e8c89d0
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 16s
CI / Conventional Commit title (pull_request) Successful in 13s
CI / Workflow lint (actionlint) (pull_request) Successful in 20s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 18s
Third-party parts / As their notes say (pull_request) Successful in 29s
Website / Website on Cloudflare (pull_request_target) Has been skipped
to fea52a0be3
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s
Third-party parts / As their notes say (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
2026-11-15 18:46:09 +00:00
Compare
Owner

Rebased onto main @ 7bacf846 (no conflicts); pushed as fea52a0b. No tests were run here; the test boxes stay yours.

Rebased onto `main` @ 7bacf846 (no conflicts); pushed as fea52a0b. No tests were run here; the test boxes stay yours.
Author
Member

Tested HumanBean's rebase (fea52a0b): the 225 checks that read this diff, 189 passed, 14 skipped, 22 failed, each failing on main too; and the host build box ticked: host/hos_main.c, host/options_path.c, gpu/vk/vk_draw.cpp, vk_progcache.cpp and vk_programs.cpp compile to objects on fea52a0b and on main 7bacf846 for Windows (llvm-mingw clang) and Linux (gcc and clang 19 in Docker); changed files only, not a whole game build.

Tested HumanBean's rebase (fea52a0b): the 225 checks that read this diff, 189 passed, 14 skipped, 22 failed, each failing on main too; and the host build box ticked: host/hos_main.c, host/options_path.c, gpu/vk/vk_draw.cpp, vk_progcache.cpp and vk_programs.cpp compile to objects on fea52a0b and on main 7bacf846 for Windows (llvm-mingw clang) and Linux (gcc and clang 19 in Docker); changed files only, not a whole game build.
dyst changed title from WIP: feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing to feat(vk): one shader cache store for every game folder, so a build whose translation is unchanged prepares nothing 2026-11-15 20:53:19 +00:00
Member

A measurement that bears on "the old files are not reused under the new stamp", for the pipeline cache (longshot_vk_pipelines.bin) only.

On this PC (Ryzen 5 5600, RX 6700 XT, AMD 26.10.1, Windows 11), the launcher's update from nightly-130 (0caf0618) to nightly-189 (aa95d35f) changed the translator stamp. The new folder's preparation therefore started without a pipeline cache and took 85 min: about 20 min of programs and compute pipelines, then 65 min of the 32,623 noted graphics pipelines.

I ran that preparation again in a fresh nightly-189 folder, with nightly-130's longshot_vk_pipelines.bin and its play files copied into cache/ and RX_VK_PCACHE_STAMP=0. The exe path was new, so AMD's own driver cache was cold, as in an update. The command was make_game.py's: --prepare-shaders --compile-shaders most.

Update as shipped Old pipeline cache kept
Programs + compute pipelines ~20 min 2.3 min (9,691 compute pipelines, all from the cache, 0 compiled)
Graphics pipelines (32,623 noted) ~65 min 30.6 min
Total 85 min 33.1 min

The stamp changed, but almost none of the SPIR-V did: one stamped commit landed in between (#400), and 480 sampled programs were byte-identical in both folders' program caches. The driver finds entries by their contents, so the old ones still hit. The 30 min left are most likely the pipelines new since nightly-130, mostly #563's list.

So keeping the previous stamp's pipeline cache (merged in as a play file, say) would save most of an update's preparation on AMD; dropping it costs the whole preparation. The program-side files are right to go with the stamp.

The cost is growth: entries for SPIR-V that no longer exists stay in the file. One possible bound is to carry it across one stamp change only. Another is to carry it only while it stays under some multiple of a fresh cache's size. Here the file grew from 1.49 GB (main file plus play files) to 1.98 GB. The update's freshly built one is 1.93 GB, so the stale entries cost only about 50 MB this time.

This is one run on one PC, so the minutes will vary. Would you rather have this in #737, or as a follow-up once it lands? I can write either.

A measurement that bears on "the old files are not reused under the new stamp", for the pipeline cache (`longshot_vk_pipelines.bin`) only. On this PC (Ryzen 5 5600, RX 6700 XT, AMD 26.10.1, Windows 11), the launcher's update from nightly-130 (0caf0618) to nightly-189 (aa95d35f) changed the translator stamp. The new folder's preparation therefore started without a pipeline cache and took 85 min: about 20 min of programs and compute pipelines, then 65 min of the 32,623 noted graphics pipelines. I ran that preparation again in a fresh nightly-189 folder, with nightly-130's `longshot_vk_pipelines.bin` and its play files copied into `cache/` and `RX_VK_PCACHE_STAMP=0`. The exe path was new, so AMD's own driver cache was cold, as in an update. The command was make_game.py's: `--prepare-shaders --compile-shaders most`. | | Update as shipped | Old pipeline cache kept | |---|---|---| | Programs + compute pipelines | ~20 min | 2.3 min (9,691 compute pipelines, all from the cache, 0 compiled) | | Graphics pipelines (32,623 noted) | ~65 min | 30.6 min | | **Total** | **85 min** | **33.1 min** | The stamp changed, but almost none of the SPIR-V did: one stamped commit landed in between (#400), and 480 sampled programs were byte-identical in both folders' program caches. The driver finds entries by their contents, so the old ones still hit. The 30 min left are most likely the pipelines new since nightly-130, mostly #563's list. So keeping the previous stamp's pipeline cache (merged in as a play file, say) would save most of an update's preparation on AMD; dropping it costs the whole preparation. The program-side files are right to go with the stamp. The cost is growth: entries for SPIR-V that no longer exists stay in the file. One possible bound is to carry it across one stamp change only. Another is to carry it only while it stays under some multiple of a fresh cache's size. Here the file grew from 1.49 GB (main file plus play files) to 1.98 GB. The update's freshly built one is 1.93 GB, so the stale entries cost only about 50 MB this time. This is one run on one PC, so the minutes will vary. Would you rather have this in #737, or as a follow-up once it lands? I can write either.
HumanBean left a comment

Code review only: nothing was built or run here. I read the whole diff (shader_store.h, the call sites in vk_draw/vk_progcache/vk_programs, vk_paths.h, the host's arguments, make_game/longshot-launch, dev_packs, shader_store.py, the docs and the tests) and the comments.

Verdict: comment. The design holds up and I found nothing that has to change before merge. The store stays off unless LONGSHOT_SHADER_CACHE is set, and nothing sets it for players yet. It isn't an approval because none of the in-game runs under Must be tested has been done, on Windows or Linux. On Linux the flock lock and the AppImage's longshot-launch have never run at all.

What I checked and found right:

  • The heads consume reads match what the game writes. The program cache's stamp is at offset 8 (disk_init). The notes' head is the stamp then sizeof(PipeNote), compared as 16 bytes. pcache_stamp.h keeps 8 bytes at offset 0. The play-file numbers from 1000000 up stay within play_files_merge's 9 digits.
  • The first shader_cache_path call (which takes program_cache_stamp() and caches it) comes after replay_start/create_device on the paths I followed: started_up, prepare_guess, pcache_init, ids_ready, the loader's load_fallback_path. So the stamp isn't taken before the device exists. With the store unset, the stamp isn't taken here at all.
  • UTF-8 paths: the *A functions, fopen and std::filesystem all depend on the exe manifest's UTF-8 code page, as the rest of host/ and gpu/vk already do. No CP_ACP.
  • options_path.c and both argument scans in hos_main.c skip --shader-cache <v> and --shader-cache-keep.
  • GAME_ENV_KEEP and longshot-launch's awk filter agree, and test_prepare_at_build.py checks both.
  • Defaults: off, and 0/empty mean off. dev_packs.py turns it on for development only, with setdefault.

Suggestions, inline:

  1. Linux: hand_over checks held_elsewhere, then renames. A game that takes the lock in between keeps running on a folder that has been renamed out from under it.
  2. A Fallback start writes the record as if it now used the new stamp, so the stamp it used before is never let go by the game.
  3. hos_main.c: the --shader-cache value is silently cut at MAX_PATH + 32.
  4. Two smaller questions (left in cache/, .built after a merge), inline.

Overlaps (merge order): none of these duplicates this PR, but two will break tests/test_shader_store.py once both land, so the second to merge has to update it:

  • #902 adds driver_env_path() { return cache_path("longshot_driver_env.txt"); } to vk_draw.cpp. check_sites' regex rejects any cache_path("...") there other than the driver faults, so it needs that name added (it is the game folder's own, like the faults).
  • #345 moves driver_faults_path to game_folder_path() and longshot_load_start.txt beside the settings. The driver_faults_path needle in check_sites then fails, and the load-start marker would leave the store's stamp folder.
  • #416 also edits tests/pipeline_play_cache_test.cpp's stand-ins. That conflict is probably textual only.
  • #903, #428 and #517 reach their files through pcache_dir/pipe_notes_path(), so with this PR those files follow the store without changes. #623 and #33 don't overlap.

madebyjordan's measurement (keeping the old stamp's longshot_vk_pipelines.bin saves about 50 of 85 minutes of an update's preparation on AMD): main already drops that cache when the stamp changes (pcache_drop_stale), so letting it go here is no regression. I'd do it as the follow-up they offered, bounded to one stamp change, rather than grow this PR.

Code review only: nothing was built or run here. I read the whole diff (shader_store.h, the call sites in vk_draw/vk_progcache/vk_programs, vk_paths.h, the host's arguments, make_game/longshot-launch, dev_packs, shader_store.py, the docs and the tests) and the comments. **Verdict: comment.** The design holds up and I found nothing that has to change before merge. The store stays off unless `LONGSHOT_SHADER_CACHE` is set, and nothing sets it for players yet. It isn't an approval because none of the in-game runs under *Must be tested* has been done, on Windows or Linux. On Linux the `flock` lock and the AppImage's `longshot-launch` have never run at all. What I checked and found right: - The heads `consume` reads match what the game writes. The program cache's stamp is at offset 8 (`disk_init`). The notes' head is the stamp then `sizeof(PipeNote)`, compared as 16 bytes. `pcache_stamp.h` keeps 8 bytes at offset 0. The play-file numbers from 1000000 up stay within `play_files_merge`'s 9 digits. - The first `shader_cache_path` call (which takes `program_cache_stamp()` and caches it) comes after `replay_start`/`create_device` on the paths I followed: `started_up`, `prepare_guess`, `pcache_init`, `ids_ready`, the loader's `load_fallback_path`. So the stamp isn't taken before the device exists. With the store unset, the stamp isn't taken here at all. - UTF-8 paths: the `*A` functions, `fopen` and `std::filesystem` all depend on the exe manifest's UTF-8 code page, as the rest of host/ and gpu/vk already do. No `CP_ACP`. - `options_path.c` and both argument scans in `hos_main.c` skip `--shader-cache <v>` and `--shader-cache-keep`. - `GAME_ENV_KEEP` and longshot-launch's awk filter agree, and `test_prepare_at_build.py` checks both. - Defaults: off, and `0`/empty mean off. `dev_packs.py` turns it on for development only, with `setdefault`. Suggestions, inline: 1. Linux: `hand_over` checks `held_elsewhere`, then renames. A game that takes the lock in between keeps running on a folder that has been renamed out from under it. 2. A Fallback start writes the record as if it now used the new stamp, so the stamp it used before is never let go by the game. 3. `hos_main.c`: the `--shader-cache` value is silently cut at `MAX_PATH + 32`. 4. Two smaller questions (left in `cache/`, `.built` after a merge), inline. **Overlaps (merge order):** none of these duplicates this PR, but two will break `tests/test_shader_store.py` once both land, so the second to merge has to update it: - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/902 adds `driver_env_path() { return cache_path("longshot_driver_env.txt"); }` to `vk_draw.cpp`. `check_sites`' regex rejects any `cache_path("...")` there other than the driver faults, so it needs that name added (it is the game folder's own, like the faults). - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/345 moves `driver_faults_path` to `game_folder_path()` and `longshot_load_start.txt` beside the settings. The `driver_faults_path` needle in `check_sites` then fails, and the load-start marker would leave the store's stamp folder. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/416 also edits `tests/pipeline_play_cache_test.cpp`'s stand-ins. That conflict is probably textual only. - https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/903, https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/428 and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/517 reach their files through `pcache_dir`/`pipe_notes_path()`, so with this PR those files follow the store without changes. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/623 and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/33 don't overlap. **madebyjordan's measurement** (keeping the old stamp's `longshot_vk_pipelines.bin` saves about 50 of 85 minutes of an update's preparation on AMD): main already drops that cache when the stamp changes (`pcache_drop_stale`), so letting it go here is no regression. I'd do it as the follow-up they offered, bounded to one stamp change, rather than grow this PR.
@ -0,0 +252,4 @@
if (n > 0) s.said.push_back("took in " + std::to_string(n) + " pipeline notes from " + from);
continue;
}
if (!exists(to + name)) {
Owner

When the store already has a program cache, a second run's longshot_programs.bin/.idx (several GB) stays in cache/ for good. Nothing later deletes it, and every Store start reads its head again. You say in the description that the record-by-record merge isn't done here. Could a line in the log (or shader_store.py ls) point it out, so a player or tester knows those GB can go?

When the store already has a program cache, a second run's `longshot_programs.bin`/`.idx` (several GB) stays in `cache/` for good. Nothing later deletes it, and every Store start reads its head again. You say in the description that the record-by-record merge isn't done here. Could a line in the log (or `shader_store.py ls`) point it out, so a player or tester knows those GB can go?
Owner

A question: precompile_covers_notes compares only the stamp in .built, not the count. Notes merged in here come from a run whose pipelines went into its own pipeline cache, so they're covered when that cache comes in as a play file. When it doesn't come in (its stamp file is missing or different), the merged notes are uncovered, but .built still says they're built and RX_VK_LOAD=create trusts it. state_list_prepare removes .built when it adds notes without pipelines. Should the merge here do the same when pcache is false?

A question: `precompile_covers_notes` compares only the stamp in `.built`, not the count. Notes merged in here come from a run whose pipelines went into its own pipeline cache, so they're covered when that cache comes in as a play file. When it doesn't come in (its stamp file is missing or different), the merged notes are uncovered, but `.built` still says they're built and `RX_VK_LOAD=create` trusts it. `state_list_prepare` removes `.built` when it adds notes without pipelines. Should the merge here do the same when `pcache` is false?
@ -0,0 +495,4 @@
if (FILE* f = fopen((s.stamp_dir + kLastUsed).c_str(), "wb")) fclose(f); // (its time: the launcher keeps the newest)
s.said.push_back("stamp " + hex(folder_key(stamp)) + ": the shader caches are in " + s.stamp_dir);
}
write_record(s);
Owner

write_record also runs in the Fallback case, when another Longshot holds this stamp, and hand_over didn't run. The record then names the new stamp, so at the next Store start previous == name and the stamp this folder really used before is never let go by the game. Its users/ marker no longer counts, so only the launcher or prune will remove it. Writing the record only when stamped == Where::Store would keep the update-in-place clean-up working across a Fallback run. In Fallback, this run's files are in cache/ anyway.

`write_record` also runs in the Fallback case, when another Longshot holds this stamp, and `hand_over` didn't run. The record then names the new stamp, so at the next Store start `previous == name` and the stamp this folder really used before is never let go by the game. Its users/ marker no longer counts, so only the launcher or `prune` will remove it. Writing the record only when `stamped == Where::Store` would keep the update-in-place clean-up working across a Fallback run. In Fallback, this run's files are in `cache/` anyway.
@ -0,0 +439,4 @@
s.said.push_back("stamp " + previous + ", this folder's before, kept (LONGSHOT_SHADER_CACHE_KEEP)");
else if (used_by_others(prev_dir, previous, std::string()))
s.said.push_back("stamp " + previous + ", this folder's before, stays: another game folder uses it");
else if (held_elsewhere(prev_dir))
Owner

On Linux this check is separate from the rename below. If a game on the old stamp starts between held_elsewhere (which drops the flock at once) and let_go's rename, the rename still succeeds under it: Linux doesn't refuse it the way Windows does. That game then keeps running on a renamed folder that a detached thread is deleting. Its open descriptors keep working, but every later open by stamp_dir path fails. The window is small, but closing it is cheap: keep the flock taken here until after the rename (return the descriptor, or do the rename inside it). A game that starts after that makes a fresh folder and lock.

On Linux this check is separate from the rename below. If a game on the old stamp starts between `held_elsewhere` (which drops the `flock` at once) and `let_go`'s rename, the rename still succeeds under it: Linux doesn't refuse it the way Windows does. That game then keeps running on a renamed folder that a detached thread is deleting. Its open descriptors keep working, but every later open by `stamp_dir` path fails. The window is small, but closing it is cheap: keep the `flock` taken here until after the rename (return the descriptor, or do the rename inside it). A game that starts after that makes a fresh folder and lock.
host/hos_main.c Outdated
@ -1840,0 +1844,4 @@
for (int i = 1; i + 1 < argc; i++)
if (!strcmp(argv[i], "--shader-cache")) {
#ifdef _WIN32
static char set[MAX_PATH + 32];
Owner

A store path longer than MAX_PATH + 32 bytes is cut short here without a word. A UTF-8 path with non-Latin characters reaches that sooner than its character count suggests. _putenv_s("LONGSHOT_SHADER_CACHE", argv[i + 1]) has no buffer to cut. --compile-shaders above uses the same pattern, but its values are short.

A store path longer than `MAX_PATH + 32` bytes is cut short here without a word. A UTF-8 path with non-Latin characters reaches that sooner than its character count suggests. `_putenv_s("LONGSHOT_SHADER_CACHE", argv[i + 1])` has no buffer to cut. `--compile-shaders` above uses the same pattern, but its values are short.
@ -0,0 +61,4 @@
# (no shader cache file left on cache_path beside those two: one would split from its stamp's folder)
for name, src in (("vk_draw.cpp", draw), ("vk_progcache.cpp", prog), ("vk_programs.cpp", progs)):
for m in re.finditer(r'(?<![_a-z])cache_path\("([^"]*)"\)', src):
if m.group(1) not in ("longshot_driver_faults.txt",):
Owner

Heads-up on merge order: #902 adds cache_path("longshot_driver_env.txt") to vk_draw.cpp, which this check then refuses. #345 moves driver_faults_path off cache_path, which breaks the needle at line 56. Whichever of these lands second updates this check.

Heads-up on merge order: https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/902 adds `cache_path("longshot_driver_env.txt")` to vk_draw.cpp, which this check then refuses. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/345 moves `driver_faults_path` off `cache_path`, which breaks the needle at line 56. Whichever of these lands second updates this check.
dyst force-pushed feat/shader-cache-store from fea52a0be3
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 4s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s
Third-party parts / As their notes say (pull_request) Successful in 10s
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 3s
to 70503f8ae5
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s
Third-party parts / As their notes say (pull_request) Successful in 11s
PR description / Description follows the template (pull_request_target) Successful in 4s
2026-11-16 00:47:26 +00:00
Compare
Author
Member

Pushed 70503f8a: rebased onto main d9486a08 (one conflict, the switch table in docs/development.md: #900's RX_VK_HALF_RATE_SHADING row and the store's both kept), plus one commit for @HumanBean's review. Thank you for the careful read.

  1. Linux, the gap between held_elsewhere and the rename: fixed both ways. let_go now returns Gone / Held / Open. On Linux it takes the stamp's .lock with flock and keeps it over the rename, closing it after. hold checks that the .lock it locked is still the file at its path (fstat against stat). If a game opened the .lock before the rename and locked it after, the folder is made again and its new .lock taken. Windows keeps the check-then-rename, since a game that takes the .lock in between makes the rename fail ("a file in it is open"). scripts/shader_store.py prune keeps the lock over its rename too.
  2. The record after a Fallback start: write_record runs only when stamped == Where::Store, so a Fallback run leaves the record naming the stamp before, and the next Store start lets that one go. test_shader_store.cpp's held-part case now checks the record is unchanged.
  3. --shader-cache cut at MAX_PATH + 32: the string is now allocated to fit the path and kept for the process's life (no _putenv_s, so nothing depends on which CRT declares it).
  4. A second run's program cache left in cache/: consume now logs a program cache of this stamp stays in <cache/> (N MB): the store's own is read instead, and those files can be deleted.
  5. .built after merged notes: yes. .built now moves into the store only with the notes file. When notes are merged in and no pipeline cache of theirs comes in (main file or play file), the store's .built is removed, as state_list_prepare does, and the log says the next start builds them. cache/'s own .built goes with its merged notes. Both cases (with and without the pipeline cache) are in the test.

The new checks fail 5 times against the header before this commit and pass after: Windows with g++ 10.3 and llvm-mingw clang++, and Linux with g++ 14.2 in Docker, which runs the flock path. The 59 checks that read the diff: 46 pass, and the 13 that fail also fail on main d9486a08. Details are in Testing.

Merge order with #902 and #345: noted. Whichever of them lands after this PR updates check_sites in tests/test_shader_store.py. If this one lands second, I'll do it here.

@madebyjordan, thank you for the measurement: 85 to 33 minutes is a lot to leave on the table. I'd take it as the follow-up you offered, once this lands, bounded to one stamp change as you suggest. A follow-up keeps this PR's take-in rules as they are, and the play-file merge is already the way in for it. Main drops that cache at a stamp change today (pcache_drop_stale), so nothing here is lost against main in the meantime.

Still not run in the game: the runs under Must be tested are next.

Pushed 70503f8a: rebased onto main d9486a08 (one conflict, the switch table in `docs/development.md`: #900's `RX_VK_HALF_RATE_SHADING` row and the store's both kept), plus one commit for @HumanBean's review. Thank you for the careful read. 1. **Linux, the gap between `held_elsewhere` and the rename:** fixed both ways. `let_go` now returns Gone / Held / Open. On Linux it takes the stamp's `.lock` with `flock` and keeps it over the rename, closing it after. `hold` checks that the `.lock` it locked is still the file at its path (`fstat` against `stat`). If a game opened the `.lock` before the rename and locked it after, the folder is made again and its new `.lock` taken. Windows keeps the check-then-rename, since a game that takes the `.lock` in between makes the rename fail ("a file in it is open"). `scripts/shader_store.py prune` keeps the lock over its rename too. 2. **The record after a Fallback start:** `write_record` runs only when `stamped == Where::Store`, so a Fallback run leaves the record naming the stamp before, and the next Store start lets that one go. `test_shader_store.cpp`'s held-part case now checks the record is unchanged. 3. **`--shader-cache` cut at `MAX_PATH + 32`:** the string is now allocated to fit the path and kept for the process's life (no `_putenv_s`, so nothing depends on which CRT declares it). 4. **A second run's program cache left in `cache/`:** `consume` now logs `a program cache of this stamp stays in <cache/> (N MB): the store's own is read instead, and those files can be deleted`. 5. **`.built` after merged notes:** yes. `.built` now moves into the store only with the notes file. When notes are merged in and no pipeline cache of theirs comes in (main file or play file), the store's `.built` is removed, as `state_list_prepare` does, and the log says the next start builds them. `cache/`'s own `.built` goes with its merged notes. Both cases (with and without the pipeline cache) are in the test. The new checks fail 5 times against the header before this commit and pass after: Windows with g++ 10.3 and llvm-mingw clang++, and Linux with g++ 14.2 in Docker, which runs the `flock` path. The 59 checks that read the diff: 46 pass, and the 13 that fail also fail on main d9486a08. Details are in Testing. **Merge order with #902 and #345:** noted. Whichever of them lands after this PR updates `check_sites` in `tests/test_shader_store.py`. If this one lands second, I'll do it here. @madebyjordan, thank you for the measurement: 85 to 33 minutes is a lot to leave on the table. I'd take it as the follow-up you offered, once this lands, bounded to one stamp change as you suggest. A follow-up keeps this PR's take-in rules as they are, and the play-file merge is already the way in for it. Main drops that cache at a stamp change today (`pcache_drop_stale`), so nothing here is lost against main in the meantime. Still not run in the game: the runs under Must be tested are next.
dyst force-pushed feat/shader-cache-store from 70503f8ae5
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 3s
CI / Workflow lint (actionlint) (pull_request) Successful in 6s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 5s
Third-party parts / As their notes say (pull_request) Successful in 11s
PR description / Description follows the template (pull_request_target) Successful in 4s
to 3ded310de2
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 15s
CI / Conventional Commit title (pull_request) Successful in 12s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 14s
CI / Workflow lint (actionlint) (pull_request) Successful in 15s
Third-party parts / As their notes say (pull_request) Successful in 13s
2026-11-16 07:28:58 +00:00
Compare
dyst force-pushed feat/shader-cache-store from 3ded310de2
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 15s
CI / Conventional Commit title (pull_request) Successful in 12s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 14s
CI / Workflow lint (actionlint) (pull_request) Successful in 15s
Third-party parts / As their notes say (pull_request) Successful in 13s
to 680d1c836d
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 15s
PR description / Description follows the template (pull_request_target) Successful in 4s
2026-11-16 11:44:10 +00:00
Compare
Author
Member

Run in the game on Windows (this PR @ 680d1c83, rebased onto main @ 6c8d6e6a with one conflict, the two new sections of docs/development.md both kept), Windows 11, RTX 3080; the Must be tested steps, through the job queue. Every step is in Measurements.

  • A second game folder prepares nothing. Folder B, the same build with an empty cache/, found the stamp's caches in the store and reached play in 78.9 s. Run beside folder A (so on its own cache/, as every folder is on main), the same B prepared every program from nothing: 1983 s.
  • Two at once: the second said "another Longshot uses the store's; taken in at a later start" and ran from its own cache/; its files went into the store at its next start alone ("took in 1 pipeline notes ... 5 files"), and the 7.4 GB program cache it made is named in the log as deletable (the review's point 4).
  • Another stamp (RX_VK_MISS_LOG=1 is in the translator's stamp, so a start with it is a start under another translation): kept while another folder used it, kept with LONGSHOT_SHADER_CACHE_KEEP=1, let go once no folder used it and keep wasn't asked for. shader_store.py ls agreed after each step.

One thing I didn't expect, an open question rather than a bug in this PR: folder A's first start took in the lists but not its stamped files. Its cache/ was prepared at build time, under the game folder's own settings; longshot_ctl.py launch --profile test (an empty HOS_ENV_FILE, a user folder of its own) starts the game with another program cache stamp, so those files were not the stamp's, and main's disk_init would have prepared again too. @HumanBean: should the test profile keep the folder's env file for the switches that are in the stamp, so a test start meets the build's caches? That would be its own change, to longshot_ctl.py.

The Windows item is off Must be tested; the Linux run (the flock lock, the AppImage's longshot-launch) is still wanted.

Run in the game on Windows (this PR @ 680d1c83, rebased onto main @ 6c8d6e6a with one conflict, the two new sections of `docs/development.md` both kept), Windows 11, RTX 3080; the Must be tested steps, through the job queue. Every step is in Measurements. - **A second game folder prepares nothing.** Folder B, the same build with an empty `cache/`, found the stamp's caches in the store and reached play in **78.9 s**. Run beside folder A (so on its own `cache/`, as every folder is on main), the same B prepared every program from nothing: **1983 s**. - **Two at once:** the second said "another Longshot uses the store's; taken in at a later start" and ran from its own `cache/`; its files went into the store at its next start alone ("took in 1 pipeline notes ... 5 files"), and the 7.4 GB program cache it made is named in the log as deletable (the review's point 4). - **Another stamp** (`RX_VK_MISS_LOG=1` is in the translator's stamp, so a start with it is a start under another translation): kept while another folder used it, kept with `LONGSHOT_SHADER_CACHE_KEEP=1`, let go once no folder used it and keep wasn't asked for. `shader_store.py ls` agreed after each step. One thing I didn't expect, an open question rather than a bug in this PR: folder A's first start took in the lists but not its stamped files. Its `cache/` was prepared at build time, under the game folder's own settings; `longshot_ctl.py launch --profile test` (an empty `HOS_ENV_FILE`, a user folder of its own) starts the game with another program cache stamp, so those files were not the stamp's, and main's `disk_init` would have prepared again too. @HumanBean: should the test profile keep the folder's env file for the switches that are in the stamp, so a test start meets the build's caches? That would be its own change, to `longshot_ctl.py`. The Windows item is off Must be tested; the Linux run (the `flock` lock, the AppImage's `longshot-launch`) is still wanted.
dyst force-pushed feat/shader-cache-store from 680d1c836d
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 8s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 11s
CI / Workflow lint (actionlint) (pull_request) Successful in 12s
Third-party parts / As their notes say (pull_request) Successful in 15s
PR description / Description follows the template (pull_request_target) Successful in 4s
to fb603f47da
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 15s
CI / Workflow lint (actionlint) (pull_request) Successful in 16s
Third-party parts / As their notes say (pull_request) Successful in 23s
PR description / Description follows the template (pull_request_target) Successful in 5s
2026-11-16 19:16:15 +00:00
Compare
Author
Member

Correction to my last comment, and a fix (fb603f47, rebased onto main @ 615b2498): the test profile was not the cause; this PR was. @HumanBean, please disregard the question I asked about --profile test.

The bug: with the store set, the stamp's part opens at the first stamped file asked for and takes program_cache_stamp() then, which keeps it for the run. The first such file was the loads' marker: load_fallback_path() built its path from pipe_notes_path(), and programs_load_decide() asks for it right after vkCreateDevice, before coopmat_init() sets the translation's per-GPU settings that translator_stamp() hashes (the invariant positions, the NaN guard, the matrix layouts). So every run with the store had a stamp no build of the same main computes, and the take-in never matched a build's cache/. That's why folder A's first start took in only the lists, and why all of this PR's runs agreed with each other but never with main. It would have hit every player's first start with the store after an update: the old caches never taken in, a full preparation each time.

Measured (main @ 615b2498, Windows 11, RTX 3080): main's build prepared cache/ with stamp 063fb3887f21b453. This PR before the fix took 74bd18fe7410cdf1: "0 program keys kept", a full preparation. With the fix it takes 063fb3887f21b453, "took in 7 files of this stamp from <the folder's> cache/", and the program cache kept 20,603 keys, with no preparation.

The fix:

  • the loads' marker is the game folder's own (cache_path, beside the driver's faults), and so is its removal at a preparation;
  • translation_settings_ready is set right after coopmat_init();
  • program_cache_stamp() doesn't keep a stamp asked for before it (it says so once in the log);
  • shader_cache_path() doesn't open the store's part before it (a file asked for then is cache/'s, said once with its name).

So a later early caller is named in the log instead of silently splitting the caches. test_shader_store checks all four, and fails with the old marker path put back. The 282 checks that read the diff: 240 pass, 32 skip, and 10 fail that fail on a clean main too (Testing).

Correction to my last comment, and a fix (fb603f47, rebased onto main @ 615b2498): the test profile was not the cause; this PR was. @HumanBean, please disregard the question I asked about `--profile test`. **The bug:** with the store set, the stamp's part opens at the first stamped file asked for and takes `program_cache_stamp()` then, which keeps it for the run. The first such file was the loads' marker: `load_fallback_path()` built its path from `pipe_notes_path()`, and `programs_load_decide()` asks for it right after `vkCreateDevice`, before `coopmat_init()` sets the translation's per-GPU settings that `translator_stamp()` hashes (the invariant positions, the NaN guard, the matrix layouts). So every run with the store had a stamp no build of the same main computes, and the take-in never matched a build's `cache/`. That's why folder A's first start took in only the lists, and why all of this PR's runs agreed with each other but never with main. It would have hit every player's first start with the store after an update: the old caches never taken in, a full preparation each time. **Measured** (main @ 615b2498, Windows 11, RTX 3080): main's build prepared `cache/` with stamp `063fb3887f21b453`. This PR before the fix took `74bd18fe7410cdf1`: "0 program keys kept", a full preparation. With the fix it takes `063fb3887f21b453`, "took in 7 files of this stamp from <the folder's> cache/", and the program cache kept 20,603 keys, with no preparation. **The fix:** - the loads' marker is the game folder's own (`cache_path`, beside the driver's faults), and so is its removal at a preparation; - `translation_settings_ready` is set right after `coopmat_init()`; - `program_cache_stamp()` doesn't keep a stamp asked for before it (it says so once in the log); - `shader_cache_path()` doesn't open the store's part before it (a file asked for then is `cache/`'s, said once with its name). So a later early caller is named in the log instead of silently splitting the caches. `test_shader_store` checks all four, and fails with the old marker path put back. The 282 checks that read the diff: 240 pass, 32 skip, and 10 fail that fail on a clean main too (Testing).
Contributor

We'll run this on a Steam Deck OLED (SteamOS, RADV) for the Linux item under Must be tested: the flock lock and a second folder that prepares nothing. The Deck matters here: a full preparation takes 25-40 minutes on it, and today every stamp change (any RX_VK_MESH* switch, for one) throws the caches away in both directions.

What we'll run. Your four commits cherry-picked onto main 71b1edd8, our Deck base (no conflicts; the added and removed lines match git diff 615b2498 fb603f47). The control is 71b1edd8 itself. Both builds have translator stamp 2b1aa955fa54ed89, so a cache the control prepares is valid for the PR too. Every run gets its own game folder and store and its own XDG_*, so nothing is shared with the Deck's installed game.

  1. Today's cost, on the control: a cold preparation, a warm start, a flip to RX_VK_MESH_NOCULL=2c56dc1006942f74 (a full re-preparation), and a flip back (another one).
  2. A player's first start with the store: a folder with a prepared cache/ of the same stamp should move it into the store, with the same program-key count and no preparation.
  3. The store in one folder: a cold preparation and a warm start, then flips back and forth with LONGSHOT_SHADER_CACHE_KEEP=1 (expect one preparation for the new stamp and none after). Without KEEP: expect the old stamp let go and one set kept.
  4. Two folders. One after the other: the second, with an empty cache/, should prepare nothing. At once: the second should get the Fallback lines, the first should hold <stamp>/.lock (read from /proc/locks), and both should reach play. Then the second alone, to check the late take-in and the "can be deleted" line, and that no program cache error lines appear afterwards.
  5. A Kokiri Forest screenshot from both builds, and fps over 45 s standing (1280x720, 50% render, Custom), to make sure nothing else moved.

We'll post the times (to title, to play, translation and pipeline stages), disk use and the log lines here when it's done.

From reading the diff, and the checks on Linux:

  • tests/test_pipeline_tour.py fails on this head and passes on its merge base 615b2498 (Linux), so it isn't one of the failures main shares. It guards scripts/pipeline_tour.py, which moves cache/longshot_pipelines.bin and the state lists aside before a tour. With LONGSHOT_SHADER_CACHE set, the game reads those from the store, so a tour would run with the store's notes in place. Either the tour learns the store (or starts its games with --shader-cache 0) and the check follows, or the check gets updated with a reason.
  • A first start with the store whose cache/ holds another stamp's caches (say, an update that changes the translator and turns the store on together): consume skips them (shader_store.h:260-261), and pcache_drop_stale and disk_init now look in the store, so the old program cache (about 8 GB on the Deck) and its pipeline cache stay in cache/ for good, without a log line. On main they're dropped in place. Could the start that holds the store remove, or at least name, cache/'s stamped files of another stamp, as it already does for a second run's program cache?
  • hold() gives up at the first EWOULDBLOCK (shader_store.h:148). scripts/shader_store.py ls/prune (in_use) and another game's let_go take each stamp's .lock for a moment, so a game starting right then falls back to its own cache/, and with an empty one it prepares everything. A few retries over a second or so in hold would close that.
  • Small: when the store already has them, cache/'s module ids, longshot_prepared.bin and the predictions stay behind without a word (shader_store.h:278-284). Only the program cache gets the "can be deleted" line.
  • Checks: python -I -B tests/run_all.py --changed against main a413bf1c on this head (Linux, Vulkan and SPIR-V headers given): 278 checks, 265 passed, 4 skipped, 13 failed. Eleven of those fail on 615b2498 too (test_bug_report, test_failed_compile_keeps_objects, test_fsr_uniform_ownership, test_linux_preflight, test_linux_update, test_make_game_cache, test_missing_nup, test_nonuniform_image_arrays, test_replay_write_order, test_vk_occlusion_queries, test_walk_key). test_pipeline_cache_save failed under load and passes alone. That leaves test_pipeline_tour, above. test_shader_store (g++ on Linux, the flock path), test_shader_store_tool, test_options_path, test_prepare_at_build and test_pipeline_play_cache pass, on this head and on the cherry-pick. This head still merges cleanly into main a413bf1c.

@dyst, @HumanBean: no action needed before the run. The first two points might be worth settling before merge.

We'll run this on a Steam Deck OLED (SteamOS, RADV) for the Linux item under *Must be tested*: the `flock` lock and a second folder that prepares nothing. The Deck matters here: a full preparation takes 25-40 minutes on it, and today every stamp change (any `RX_VK_MESH*` switch, for one) throws the caches away in both directions. **What we'll run.** Your four commits cherry-picked onto main 71b1edd8, our Deck base (no conflicts; the added and removed lines match `git diff 615b2498 fb603f47`). The control is 71b1edd8 itself. Both builds have translator stamp `2b1aa955fa54ed89`, so a cache the control prepares is valid for the PR too. Every run gets its own game folder and store and its own `XDG_*`, so nothing is shared with the Deck's installed game. 1. Today's cost, on the control: a cold preparation, a warm start, a flip to `RX_VK_MESH_NOCULL=2c56dc1006942f74` (a full re-preparation), and a flip back (another one). 2. A player's first start with the store: a folder with a prepared `cache/` of the same stamp should move it into the store, with the same program-key count and no preparation. 3. The store in one folder: a cold preparation and a warm start, then flips back and forth with `LONGSHOT_SHADER_CACHE_KEEP=1` (expect one preparation for the new stamp and none after). Without KEEP: expect the old stamp let go and one set kept. 4. Two folders. One after the other: the second, with an empty `cache/`, should prepare nothing. At once: the second should get the Fallback lines, the first should hold `<stamp>/.lock` (read from `/proc/locks`), and both should reach play. Then the second alone, to check the late take-in and the "can be deleted" line, and that no program cache error lines appear afterwards. 5. A Kokiri Forest screenshot from both builds, and fps over 45 s standing (1280x720, 50% render, Custom), to make sure nothing else moved. We'll post the times (to title, to play, translation and pipeline stages), disk use and the log lines here when it's done. **From reading the diff, and the checks on Linux:** - `tests/test_pipeline_tour.py` fails on this head and passes on its merge base 615b2498 (Linux), so it isn't one of the failures main shares. It guards `scripts/pipeline_tour.py`, which moves `cache/longshot_pipelines.bin` and the state lists aside before a tour. With `LONGSHOT_SHADER_CACHE` set, the game reads those from the store, so a tour would run with the store's notes in place. Either the tour learns the store (or starts its games with `--shader-cache 0`) and the check follows, or the check gets updated with a reason. - A first start with the store whose `cache/` holds another stamp's caches (say, an update that changes the translator and turns the store on together): `consume` skips them (`shader_store.h:260-261`), and `pcache_drop_stale` and `disk_init` now look in the store, so the old program cache (about 8 GB on the Deck) and its pipeline cache stay in `cache/` for good, without a log line. On main they're dropped in place. Could the start that holds the store remove, or at least name, `cache/`'s stamped files of another stamp, as it already does for a second run's program cache? - `hold()` gives up at the first `EWOULDBLOCK` (`shader_store.h:148`). `scripts/shader_store.py ls`/`prune` (`in_use`) and another game's `let_go` take each stamp's `.lock` for a moment, so a game starting right then falls back to its own `cache/`, and with an empty one it prepares everything. A few retries over a second or so in `hold` would close that. - Small: when the store already has them, `cache/`'s module ids, `longshot_prepared.bin` and the predictions stay behind without a word (`shader_store.h:278-284`). Only the program cache gets the "can be deleted" line. - Checks: `python -I -B tests/run_all.py --changed` against main a413bf1c on this head (Linux, Vulkan and SPIR-V headers given): 278 checks, 265 passed, 4 skipped, 13 failed. Eleven of those fail on 615b2498 too (`test_bug_report`, `test_failed_compile_keeps_objects`, `test_fsr_uniform_ownership`, `test_linux_preflight`, `test_linux_update`, `test_make_game_cache`, `test_missing_nup`, `test_nonuniform_image_arrays`, `test_replay_write_order`, `test_vk_occlusion_queries`, `test_walk_key`). `test_pipeline_cache_save` failed under load and passes alone. That leaves `test_pipeline_tour`, above. `test_shader_store` (g++ on Linux, the `flock` path), `test_shader_store_tool`, `test_options_path`, `test_prepare_at_build` and `test_pipeline_play_cache` pass, on this head and on the cherry-pick. This head still merges cleanly into main a413bf1c. @dyst, @HumanBean: no action needed before the run. The first two points might be worth settling before merge.
futondestroyer left a comment

Code-and-evidence review of fb603f47da only; no PR code, tests, builds or game runs were executed here.

Verdict: request changes. Sharing prepared caches can spare a second game folder a lengthy preparation. The early translator-stamp capture and the earlier lock/record/.built review points have been addressed; the new Windows first-start evidence demonstrates matching and taking in a build's existing cache.

  • [P2] Preserve distinct Linux folder identities in user_marker (inline). Lowercasing every path aliases case-distinct game folders, so changing one folder's stamp can remove the other folder's ownership marker and delete caches it still needs.
  • [P2] Existing integration blocker, confirmed by source: the latest Linux review reports test_pipeline_tour failing here and passing on merge base 615b2498. This is separate from the listed baseline failures. pipeline_tour.py inherits LONGSHOT_SHADER_CACHE, but moves aside, observes and exports only the game folder's cache/ lists. With a store enabled the renderer now uses the store instead, so the tour can include old notes and miss its new state list. Force the store off for tour launches or make the tour explicitly manage the store, and update the existing check accordingly. This is a behavior problem, not merely a stale source-text assertion.

Evidence still needed: Windows 11/RTX 3080 runs cover two folders, simultaneous fallback, stamp hand-over/KEEP and the current-head stamp fix. The Linux/SteamOS flock, simultaneous-folder behavior and AppImage Play-button environment run is still only announced in the linked comment, not a completed result. Keep this not ready until that result is posted. Skips and main's failures are not passing coverage. The reported 78.9 s versus 1983.3 s illustrates cache reuse; it is not an interleaved performance estimate for the current head. The same comment also identifies old-stamp files stranded in cache/; settle their cleanup or explicit reporting before claiming disk cleanup is complete. Keeping a previous stamp's driver cache is a separate follow-up: main already drops it, so its loss is not introduced here.

Overlaps/order: coordinate #902 and #345 with this PR: the second to land must reconcile path ownership and test_shader_store.py assertions. #517 also changes preparation/notes lifetime and needs integration review. The earlier review's play-cache stand-in overlap with #416 should be rechecked if that work is carried forward. No duplicate implementation is established. Resolve the current mergeability state and revalidate the resulting head before merge.

Code-and-evidence review of fb603f47da34d7ee20f12430a2137143a7f0ef70 only; no PR code, tests, builds or game runs were executed here. **Verdict: request changes.** Sharing prepared caches can spare a second game folder a lengthy preparation. The early translator-stamp capture and the earlier lock/record/`.built` review points have been addressed; the new Windows first-start evidence demonstrates matching and taking in a build's existing cache. - **[P2] Preserve distinct Linux folder identities in `user_marker`** (inline). Lowercasing every path aliases case-distinct game folders, so changing one folder's stamp can remove the other folder's ownership marker and delete caches it still needs. - **[P2] Existing integration blocker, confirmed by source:** [the latest Linux review](https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/737#issuecomment-38672) reports `test_pipeline_tour` failing here and passing on merge base 615b2498. This is separate from the listed baseline failures. `pipeline_tour.py` inherits `LONGSHOT_SHADER_CACHE`, but moves aside, observes and exports only the game folder's `cache/` lists. With a store enabled the renderer now uses the store instead, so the tour can include old notes and miss its new state list. Force the store off for tour launches or make the tour explicitly manage the store, and update the existing check accordingly. This is a behavior problem, not merely a stale source-text assertion. **Evidence still needed:** Windows 11/RTX 3080 runs cover two folders, simultaneous fallback, stamp hand-over/KEEP and the current-head stamp fix. The Linux/SteamOS `flock`, simultaneous-folder behavior and AppImage Play-button environment run is still only announced in the linked comment, not a completed result. Keep this not ready until that result is posted. Skips and main's failures are not passing coverage. The reported 78.9 s versus 1983.3 s illustrates cache reuse; it is not an interleaved performance estimate for the current head. The same comment also identifies old-stamp files stranded in `cache/`; settle their cleanup or explicit reporting before claiming disk cleanup is complete. Keeping a previous stamp's driver cache is a separate follow-up: main already drops it, so its loss is not introduced here. **Overlaps/order:** coordinate https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/902 and https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/345 with this PR: the second to land must reconcile path ownership and `test_shader_store.py` assertions. https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/517 also changes preparation/notes lifetime and needs integration review. The earlier review's play-cache stand-in overlap with https://git.magicbean.dev/MagicBeanDev/Longshot/pulls/416 should be rechecked if that work is carried forward. No duplicate implementation is established. Resolve the current mergeability state and revalidate the resulting head before merge.
@ -0,0 +382,4 @@
/// Where a game folder's marker goes in a stamp's users/: named by the folder's path, holding it
inline std::string user_marker(const std::string& stamp_dir, const std::string& game) {
uint64_t h = 0xCBF29CE484222325ull;
for (const char ch : game) h = (h ^ (uint8_t)(ch == '\\' ? '/' : tolower((unsigned char)ch))) * 1099511628211ull;
Member

[P2] Preserve case-sensitive game-folder identities on Linux. user_marker lowercases paths on every platform, so distinct folders such as /games/A/ and /games/a/ get the same marker. After both use stamp S, starting either folder under stamp T removes that shared marker from S in hand_over; with the other game stopped, used_by_others finds no owner and let_go deletes S even though the other existing folder still records S. Its next start must prepare all shaders again. Keep path case (and literal backslashes) on case-sensitive platforms and add a Linux two-folder hand-over case proving the other marker/cache survives.

[P2] Preserve case-sensitive game-folder identities on Linux. `user_marker` lowercases paths on every platform, so distinct folders such as `/games/A/` and `/games/a/` get the same marker. After both use stamp S, starting either folder under stamp T removes that shared marker from S in `hand_over`; with the other game stopped, `used_by_others` finds no owner and `let_go` deletes S even though the other existing folder still records S. Its next start must prepare all shaders again. Keep path case (and literal backslashes) on case-sensitive platforms and add a Linux two-folder hand-over case proving the other marker/cache survives.
Some checks failed
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been skipped
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been skipped
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been skipped
Third-party parts / The same as upstream (pull_request) Has been skipped
Website / Website on Cloudflare (pull_request) Has been skipped
CI / Conventional Commit title (pull_request) Successful in 9s
Guide screenshots / Guide images and local UI source review (pull_request) Failing after 15s
CI / Workflow lint (actionlint) (pull_request) Successful in 16s
Third-party parts / As their notes say (pull_request) Successful in 23s
PR description / Description follows the template (pull_request_target) Successful in 5s
This pull request has changes conflicting with the target branch.
  • docs/development.md
  • gpu/vk/vk_draw.cpp
  • gpu/vk/vk_paths.h
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'feat/shader-cache-store:dyst-feat/shader-cache-store'
git switch 'dyst-feat/shader-cache-store'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!737
No description provided.