perf(vk): a start reads the pipeline notes file once, not twice (about 0.28 s off every start) #1873
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!1873
Loading…
Reference in a new issue
No description provided.
Delete branch "klimovich008/Longshot:perf/note-refs-once"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Every start of the game read the whole pipeline notes file,
cache/longshot_pipelines.bin, twice for the same list: once for the driver-cache check (notes_lost_by_driver) and again for the load at create (precompile_at_create), and three times when the warm pass ran too. On my PC that file is 1.0 GB with 37,474 notes, and a walk takes about 0.28 s when the file is in the OS's cache. This PR walks it once a start and hands the same list to each step, so every start is about 0.28 s shorter. Nothing a player sees changes, apart from that.The list is reused only when it is certainly the file's current content: the walk read the whole file, and the file's size, last write time and first 16 bytes are the same as when it was walked. Otherwise the step walks again, as before. Nothing in the game writes the notes file between these steps (the state list writes it before, play appends after the load at create has taken the list).
Read first:
start_note_refsandnotes_file_idingpu/vk/vk_draw.cpp;pipe_notes_refsnow also says whether it read the whole file (whole). Each of the three steps takes the list throughstart_note_refs; the load at create takes it for good, and a start that does not load at create lets it go (precompile_run).No switch: the steps get the same notes in the same order as before, and any case where the file could differ walks again. The shader caches are not invalidated (no file of the shader-cache stamp changed:
vk_draw.cppandvk_pipelines.hare not innx2/translator_files.py).Testing
Tested: Windows 11, RX 9070 XT, AMD 26.11.2, Ryzen 7 5700X3D, the notes file on a SATA SSD. 34 starts into Hyrule Field (the standard test's settings, uncapped) from the commits here before the merge with main, main's renderer and this PR's swapped into one game folder before each start, alternating: 26 on the first version (
bc3cca59) and 8 ona87d6d5e. Two of them, one a side, ran the warm pass first (the marker of a finished warm removed), as a first start after an update does. Every PR start logged one walk of the notes file; main walks two or three times. The game played normally: game clock 0.998 to 1.002, no new error lines, and the notes file stayed byte-identical across all starts (same SHA-256 as before them); the other cache files changed the same way in both arms.Checks (on the merged head, Windows, llvm-mingw):
tests/test_note_refs_once.py(new): a synthetic notes file (no game data) in 13 shapes (duplicate keys, notes refused, a tail cut in five places, an empty file, a wrong stamp or record size, no file): the new walk gives exactly main's list. Then the start's shapes (normal, driver lost the notes,--prepare-shaders) walk once, and a file that grew, was rewritten (new time or new size alone), could not be opened, was locked against reads, or failed a read or seek part way is walked again. Nine broken variants each fail their own case. It fails on main. Passes with clang++ (llvm-mingw), g++ (MSYS2) and g++ on Linux (WSL).tests/test_start_progress_at_create.pyandtests/test_quit_not_crash.pycut these functions from the source: each gets one stub line for the new state; both pass, as doestests/test_cold_cache_check.py.tests/run_all.py --jobs 4: 762 passed, 66 failed, 54 skipped on Windows; a clean checkout of main89c3d21eon the same PC: 761 passed, the same 66 failed (65 FAIL andtest_vbytes.py's 600 s timeout), 54 skipped. The extra pass is the new test. Those 66 fail on this PC's Windows setup on main too (CI runs the suite on Linux):the 66, failing on main as well
test_appimage_player_build.py, test_auto_handheld.py, test_bug_report_defaults.py, test_capture_read_once.py, test_capture_readback_visibility.py, test_crash_dialog.py, test_crt_options.py, test_damage_debug_menu.py, test_dev_ab.py, test_dev_build.py, test_dev_migrate.py, test_dev_packs.py, test_display_startup_failure.py, test_dlss_feature_size.py, test_draw_binds.py, test_draw_stream_visibility.py, test_early_split_before_draw.py, test_failed_compile_keeps_objects.py, test_frame_overlap.py, test_fsr_image_retirement.py, test_fsr_uniform_ownership.py, test_game_menu_mouse_buttons.py, test_guest_imported.py, test_high_fps_fixes.py, test_hold_gpu.py, test_host_guest_writes.py, test_hud_layout_row.py, test_job_queue.py, test_libc_hooks.py, test_lift_cache.py, test_linux_preflight.py, test_linux_pressure.py, test_longshot_ctl.py, test_low_vram_texture_mips.py, test_make_game_ngx.py, test_mise_build.py, test_missing_nup.py, test_mods_list.py, test_ngx_sdk.py, test_pads_after_menu.py, test_pass_barrier.py, test_pending_guest_import.py, test_pipeline_play_cache.py, test_prepare_cache_save.py, test_present_queue_default.py, test_present_source_crop.py, test_profiling_mode.py, test_psychov35_fsr3_fg.py, test_readback_early.py, test_readback_late.py, test_readback_timeout_source.py, test_recording_admission.py, test_replay_write_order.py, test_sampler_admission.py, test_sdiv_overflow.py, test_slow_frames.py, test_song_note_cue.py, test_split_before_draw.py, test_staging_burst.py, test_texel_buffer_ownership.py, test_ub_update_batch.py, test_upscaler_rows.py, test_vbytes.py, test_vcore_escape.py, test_vk_occlusion_queries.py, test_walk_key.py
Linux:
vk_draw.cppcompiled with the Linux build's flags (clang 21, syntax only) here; the Test Farm's Linux testers built this PR's headcefaf33dfrom source and ran it, 2 starts a side against main89c3d21e, no crash (NVIDIA: GTX 1070, RTX 4070; RADV: Steam Machine NAVI33).Still to test:
boot_time(and the start's shader seconds) about 0.25-0.3 s shorter with this PR and the same game afterwards (@Beanstalk)the notes' file walked 1 times this startand no new error lines (by hand)Measurements
Verdict: every start about 0.28 s shorter: the second walk of the notes file is gone. A cold walk (the file not in the OS's cache) is not shortened: both sides pay one; main then pays a warm one on top.
Setup: Windows 11, RX 9070 XT, AMD 26.11.2, Ryzen 7 5700X3D, SATA SSD; the game's own log lines in 34 alternating starts (17 a side; 4 with the notes file dropped from the OS's cache first), over main
0de25c3e. 26 starts ran the first version (bc3cca59); the later commits only add the checks for a failed or changed walk (about 0.1 ms a start, timed game-free), and the merge with main changes none of this code.0de25c3ea87d6d5e--prepare-shadersin the game.Risks
longshot_pipelines.binat the same size with the same last write time (or within one time tick of a FAT or exFAT drive) between the driver check and the load at create is not seen, and the load at create would look for notes at the old places. Nothing in the game writes the file there. Main's list was already exposed the same way for the rest of the session; this adds only the driver check's 40-180 ms (or the warm pass) to that window. A wrong place is caught bypipe_note_at's check on every note it reads: that pipeline is not loaded at create, as on a first start.(the notes' file walked N times this start, in M ms: start_note_refs), and the warm pass's summary has; the notes' file walked N times this start so farinside its last parenthesis. No script in the repository reads past those lines' times; a parser of your own anchored on the old endings would need a look.pipelineCreationCacheControlskip the driver check, so they already walked once on a normal start: no change there.longshot_options.iniis touched, and no shaders are prepared again.Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
tests/that reads changed source is updated, not weakened.cefaf33dfrom source, reports rp_a723f9e4a05d5bea, rp_2c0bd8afb270d731, rp_8df7ec4cbef3f6da.)Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
89c3d21einto perf/note-refs-onceTested in game: Windows 11, RX 9070 XT, AMD 26.11.2, Ryzen 7 5700X3D: 34 alternating starts into Hyrule Field (17 with main's renderer, 17 with this PR's, over main
0de25c3e). Every PR start logged one walk of the notes file; the load-at-create line went from 276-323 ms to 19-26 ms, 'the shaders took' from 4.3 s to 4.0 s (medians, warm starts). Game clock 0.998-1.002, no new error lines, the notes file byte-identical after every start. Two starts (one a side) ran the warm pass first, as after an update: one walk on the PR side.@Beanstalk Start-up time, this PR against main: the PR reads the pipeline notes file once a start instead of twice, about 0.28 s less on an RX 9070 XT with AMD's Windows driver. Nothing changes in play. What this PC can't show: Linux and NVIDIA.
vram_gb = ">= 6"on Windows targets), as many as are free.kokiri-start, uncapped and windowed:frame_rate_cap = 6,vsync = 2,frame_generation = 0,fullscreen = 0.wait_s = 10before the teleport to Hyrule Field,timeout_s = 120on its map wait, then measure 20 s.boot_timeandshader_compile(the start'svk: precompile:seconds) are what this PR changes. Also alog_valueof the load-at-create line's milliseconds (load at create \(RX_VK_LOAD=create\).*; in (\d+) ms), which holds main's second walk.boot_time_s,shader_compile_s,fps_mean,frametime_p99_ms,run_hitch_countandrun_frametime_max_msare report-only (one start's time is noisy, and play should be unchanged).Beanstalk test farm · ✅ passed on every target
Commit
cefaf33· approved by klimovich008 (reviewed) · dashboardDone (started 14:01 UTC, last report 14:55 UTC). Comment
@Beanstalkto run it again.linux+nvidialinux+radvwindows+nvidia+vram-ge6gb(6 GB VRAM or more)Couldn't be judged: the tester's PC could not finish the test. The farm tries another PC, or the same one after a fix; it says nothing about the PR on its own.
windows+nvidia+vram-ge6gb, report 1:the main base failed (exit 1): build.py stopped (exit status 1): fatal error: too many errors emitted, stopping now [-ferror-limit=]What was tested
What will be tested · AI plan (
gpt-6.1-terra)Compare two alternating starts per side on Linux NVIDIA, Linux RADV and Windows NVIDIA PCs, then teleport from Kokiri Forest to Hyrule Field. Each tester takes about 22 minutes; startup, shader preparation, load-at-create time and gameplay frame metrics are reported without performance thresholds. A crash or failure to reach Hyrule Field within the timeout fails the test. Matching hardware must be available for each target; otherwise that target waits.
kokiri-start(settings vsync=2, fullscreen=0, frame_rate_cap=6, frame_generation=0), then teleport to Hyrule Field, measure field for 20 s (after 10 s warm-up); 2 runs on each side.Measurements on `linux+nvidia`, for the PR description
Verdict: pass on linux+nvidia (2/1 report): no rule failed and no PR run crashed.
Hardware: 1: linux+nvidia+nvidia-proprietary+dgpu
NVIDIA GeForce GTX 1070, 580.196.7Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz, 32 GB RAM; 2: linux+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 4070, 615.71.9AMD Ryzen 7 5800X3D 8-Core Processor, 64 GB RAMScene: startup-field, from the repository's save kokiri-start (vsync=2, fullscreen=0, frame_rate_cap=6, frame_generation=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); startup-field: 2 game starts a side, field 20 s after 10 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)89c3d21cefaf33Flags and settings
None: the plan varied no setting, switch or build option.
As Markdown for the PR description:
Measurements on `linux+radv`, for the PR description
Verdict: pass on linux+radv (1/1 report): no rule failed and no PR run crashed.
Hardware: 1: linux+amd+radv+dgpu
AMD Steam Machine (RADV NAVI33), 26.1.99AMD Custom CPU 1772, 16 GB RAMScene: startup-field, from the repository's save kokiri-start (vsync=2, fullscreen=0, frame_rate_cap=6, frame_generation=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); startup-field: 2 game starts a side, field 20 s after 10 s warm-up; each number the median of the runs, then of 1 report
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)89c3d21cefaf33Flags and settings
None: the plan varied no setting, switch or build option.
As Markdown for the PR description:
Measurements on `windows+nvidia+vram-ge6gb`, for the PR description
Verdict: pass on windows+nvidia+vram-ge6gb (2/1 report): no rule failed and no PR run crashed.
Hardware: 1: windows+nvidia+nvidia-proprietary+dgpu
NVIDIA GeForce RTX 4080, 621.1.0AMD Ryzen 7 9800X3D 8-Core Processor, 32 GB RAM; 2: windows+nvidia+nvidia-proprietary+dgpuNVIDIA GeForce RTX 3090, 621.1.0AMD Ryzen 9 5900X 12-Core Processor, 128 GB RAMScene: startup-field, from the repository's save kokiri-start (vsync=2, fullscreen=0, frame_rate_cap=6, frame_generation=0); each cell's settings and switches as the table names them
Method: the Beanstalk test farm: main and this PR built and run on the same PC in one session, game starts alternating (ABBA); startup-field: 2 game starts a side, field 20 s after 10 s warm-up; each number the median of the runs, then of 2 reports
How to reproduce: the farm's plan above (
@Beanstalkon this PR runs it again)89c3d21cefaf33Flags and settings
None: the plan varied no setting, switch or build option.
As Markdown for the PR description:
The testers' hardware
linux+nvidiaNVIDIA GeForce GTX 1070, 580.196.7Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz, 32 GB RAMlinux+nvidiaNVIDIA GeForce RTX 4070, 615.71.9AMD Ryzen 7 5800X3D 8-Core Processor, 64 GB RAMlinux+radvAMD Steam Machine (RADV NAVI33), 26.1.99AMD Custom CPU 1772, 16 GB RAMwindows+nvidia+vram-ge6gbNVIDIA GeForce RTX 4080, 621.1.0AMD Ryzen 7 9800X3D 8-Core Processor, 32 GB RAMwindows+nvidia+vram-ge6gbNVIDIA GeForce RTX 3090, 621.1.0AMD Ryzen 9 5900X 12-Core Processor, 128 GB RAM🤖 The Test Farm's start-up numbers, from the reports (main
89c3d21evs this PRcefaf33d, 2 starts a side, alternating main, PR, PR, main; one start each per cell, so n=1). "1st" is each side's first start (main 0, PR 1), "2nd" the second (PR 2, main 3).boot_timeand the start's shader seconds. The expected ~0.3 s is far below these starts' spread (the first start compiles pipelines for 60-810 s, and the sides differ by up to 20 s either way), so they show neither a gain nor a loss.WIP: perf(vk): a start reads the pipeline notes file once, not twice (about 0.28 s off every start)to perf(vk): a start reads the pipeline notes file once, not twice (about 0.28 s off every start)