perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.8 GB less) #416
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No project
No assignees
5 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!416
Loading…
Reference in a new issue
No description provided.
Delete branch "tonesi/Longshot:perf/388-pcache-low-ram"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Type: perf · Risk: medium · Tested on: Linux, RTX 3070 Ti · Needs: Linux, NVIDIA, 16 GB of RAM (a long played session)
Why
On a 16 GB Linux PC the game's memory outgrows RAM and swap fills as areas change
(#388). About 0.8 GB of it is the pipeline cache's file: it is read
whole at every start and the driver keeps its copy in the game's private memory for the session, although NVIDIA's own
shader cache already holds the same pipelines. Not reading it there takes that off the game's footprint on such a PC.
What changed
On Linux with NVIDIA's own driver, 20 GB of RAM or less and programs loaded at their create calls, the start no longer reads
cache/longshot_vk_pipelines.binwhen NVIDIA's own shader cache is on and holds 512 MB or more; everywhere else it is readas before. With the file not read, nothing is written over it, and play's pipelines are still let go at each save.
pipeline cache:
pcache_load_off()ingpu/vk/vk_draw.cppdecides throughgpu/vk/pcache_load.h(new, no Vulkan in it).Where
RX_VK_PCACHE_LOADis unset, the file is not read only when all of these hold:programs_load_at_create());dev.driver_id == VK_DRIVER_ID_NVIDIA_PROPRIETARY: not NVK);sysconf; unknown reads the file);pcache_load::nvidia_cache()): the folder NVIDIA'sdriver uses, as its README gives it:
__GL_SHADER_DISK_CACHE_PATH'sGLCache, else$XDG_CACHE_HOME/nvidia/GLCacheor~/.cache/nvidia/GLCache, or the earlier drivers'.nv/GLCachewhere it is there. Only that folder counts, not Mesa's.__GL_SHADER_DISK_CACHE=0, no folder for it, a folder that can't be walked through, or under 512 MB: the file is read;--prepare-shadersrun.One log line gives the decision and its reason on every PC (
vk: pipeline cache file read: ...ornot read: ...), withthe RAM and NVIDIA's folder and size where they were looked at.
pipeline cache: with the file not read (this rule or
RX_VK_PCACHE_LOAD=0):play_files_merge()no longer merges the play files into the empty cache, nor writes that cache over the file at 16 ofthem (a session measured on main wrote 38 MB over a 1.2 GB file);
pipeline_cache_save()skips only the cache's own part (never fetched nor written) and still callsplay_cache_save(),which lets play's generation go at each save, the ordinary one every 300 presents included, writing nothing. Before the
review it returned at once, so play's pipelines stayed in one generation until the close. The log says
play's generation <n> let go, not writtenfor the first four and every 16th.pcache_drop_stale()drops the files another translator made whether or not the file is read (before, with the fileunread, 1.2 GB of dead files stayed on disk after every update that changed the translator;
RX_VK_PCACHE_STAMP=0still keeps them).
tests:
tests/test_pcache_load.py(new): the rule's table, andpcache_load_off()itself cut out ofvk_draw.cppand runagainst real folders (sparse files): an empty NVIDIA folder beside a full Mesa folder reads the file, the
path/XDG/HOME/
.nvchoice, a cache off, missing, not a folder or not readable, NVK, more RAM, start mode, a preparation,and the switch over a preparation.
tests/test_pipeline_play_cache.pygainsno-save(the play files left alone) andno-save-periodic(the realpipeline_cache_save(false)into the real play caches: the generation freed past thethreshold, new work in the next one, no file written); both fail on
main.tests/test_pipeline_cache_save.pygainsno-save.tests/test_pcache_stamp.pychecks the stale files go with the file unread.docs: the
RX_VK_PCACHE_LOADrow indocs/development.md's GPU switches.Measurements
Verdict: it helps and passed. On the 16 GB PC the game's private memory + swap at the end of the tour is 0.78-0.82 GB lower (medians on both bases; every PR run below every main run), its peak (resident + swapped) 0.45-0.82 GB lower and swap's peak 1.1-1.4 GB lower; fps and frames over 100 ms are the same. The cost is on the load workers: a program loaded at create takes 5.0-5.1 ms instead of 1.9-2.0 (median), and draws waited for a load 6-7 times a tour instead of 1-4 (195-226 ms in all). No listed pipeline was left in no cache. Lowest MemAvailable is higher too (+80 and +219 MB) but varies a lot from run to run: an observation, not a claim that the pressure is solved.
Hardware: Pop!_OS 24.04 (COSMIC), kernel 7.2.15; NVIDIA GeForce RTX 3070 Ti 8 GB (also drives the desktop), driver 595.104.02;
Intel Core i7-12700K; 16 GB RAM (15,757 MB usable), swap 15.4 GB zram + 4 GB partition
Scene: issue 388's teleport tour from a child save at Zora's River: Zora's Domain, Hyrule Field, Kakariko, the Market, Kokiri
Forest, Lake Hylia, Death Mountain Trail, Goron City, 40 s each with a scripted walk and camera turns (about 7 minutes).
Windowless (headless renderer), Quality preset Auto (this PC: tier 3 here, the reporter's
memory_budget=3), output2560x1440, no DLSS (a source build without it), programs loaded at create (the default with mesh shaders)
Method: two series. First, main
414f56e2built in full into its own game folder and prepared; the PR's folder is a copy of it after that (caches,NVIDIA's cache, options; romfs hard-linked) with the PR's
longshot_vk.soswapped in, built withRX_VK_ONLY=1 RX_RELEASE=1 nx2/build.py --standaloneand the committed renderer profile (RX_VK_PGO, asmake_game.pypasses it: built so from mainthe same way, the
.sois byte for byte the full build's). The PR changes the renderer alone and none of the shader stamp'sfiles, so an update keeps these caches. Three interleaved runs a side (main, PR, main, PR, main, PR), each after RAM had
settled to 9 GB+ available. Then main moved on to
82886e89(the render thread, on by default here): main's folder was updated in place by a build of82886e89and the PR's folder made from it again with the rebased branch's renderer (f0963cdc), and three more pairs ran; the first is a warm-up (main 82886e89's translator stamp changed withsizeof(Program),b2911d33, so each folder's first start prepared every shader again: the PR's in its tour, 14.6 GB peak), the other two compare. maince9d49b4, the base now, changes onlyscripts/longshot_ctl.py's Windows console flag. The game's memory every 2 s from/proc/<pid>/statusand/proc/<pid>/smaps_rollup(resident +swapped, file pages left out), the system's from
/proc/meminfo; loads and waits from the log's close census; fps from therenderer's frame log from 60 s on, an observation on the desktop's card, not a timing run
How to reproduce: on a 16 GB Linux PC with NVIDIA and a prepared folder, the same tour (
scripts/longshot_ctl.py teleport),with and without
RX_VK_PCACHE_LOAD=1; compare the game'sRssAnon + RssShmem + smaps SwapandMemAvailable82886e89(2 runs)414f56e2(3 runs)Medians; each run, main
82886e89/ this PR f0963cdc (the comparable pairs q2, q3):Each run, main
414f56e2/ this PR 9ea38a42:The PR's folder kept its 1.2 GB
longshot_vk_pipelines.binand the play files it was handed through all its tours, nonewritten, merged or folded; main's wrote 5-13 play files a tour and folded 22 into its file at one start. Every run reached
all eight areas, with no device lost and no kernel or NVRM line. Measured commits fetchable from the fork:
measured/416-9ea38a42,measured/416-f0963cdc(and the first measurement'smeasured/416-999bc37f) onhttps://git.magicbean.dev/tonesi/Longshot.
The first measurement (main
0caf0618against 999bc37f, before the review's fixes, three runs a side on the same PC and tour)gave -831 MB peak, -733 MB private + swap at the end, +329 MB lowest MemAvailable; its runs are in
docs/tested/too.Flags and settings
RX_VK_PCACHE_LOADgets an automatic default where it is unset;LONGSHOT_PREPARE_ONLYis now read by the rule too.RX_VK_PCACHE_LOAD=0or10), or read as before (1); either way whatever the rule says, a--prepare-shadersrun included1reads it on the small PC, for an A/BLONGSHOT_PREPARE_ONLY--prepare-shaderssets for its own process--prepare-shaderspcache_load_off(), so a preparation reads and saves the fileTesting
Tested:
414f56e2(the PR at 9ea38a42), then three on main
82886e89(the PR at f0963cdc; the first pair a warm-up). Every run reached alleight areas, with no device lost and 0 kernel or NVRM lines.
not read, NVIDIA's cache 1.2 GB), and theplay's generation <n> let golines asplay went on (the ordinary save letting play's pipelines go: review point 1).
__GL_SHADER_DISK_CACHE=0reads the file; a fresh, empty__GL_SHADER_DISK_CACHE_PATHbeside a Mesa folder of 600 MB (MESA_SHADER_CACHE_DIR, a sparse file) reads the file:the start's own driver-cache line shows 600 MB in all, which the guard before the review summed past 512 MB (review
point 2).
414f56e2→82886e89changed the translator stamp. The PR's folder prepared every shader againat its first start, with the file not read, and loaded at create from then on with 0 listed pipelines in no cache. The
head's stale-file fix then dropped the 14 files of the old translator (1219 MB) at a start, still not reading the file.
That short start (4 minutes, idle at the title) logged four NVRM lines 3 s before its timeout ended it
(
dmaAllocMapping_GM107: can't alloc VA space for mapping,NV_ERR_NO_MEMORYfrom the mapping reuse callback), with nodevice lost and nothing in the game's log; none of the other 21 game runs today, main's or the PR's, logged any. Not
understood yet.
tests/test_pcache_load.py(the table, andpcache_load_off()against real folders: 20 cases),tests/test_pipeline_play_cache.py(14 cases;no-saveandno-save-periodicfail on main,no-save-periodicon thebranch before the review at line 493, the retirement),
tests/test_pipeline_cache_save.py(no-save) andtests/test_pcache_stamp.py, with GCC and Clang.python -I -B tests/run_all.py --jobs 8):538 passed, 10 failed, 30 skipped. main
82886e89the same way: 537 passed and the same 10 failed:test_appimage_player_build,test_cpm_discovery,test_env_on(main'sRX_VK_PRESENT_AFTER_SIGNAL),test_lift_cleanup_error,test_linux_preflight,test_make_game_cache,test_readback_late,test_vram_budget_row,test_walk_key(nohos_oot_autoload_inputstub), andtest_crash_record(passes alone on both: slow under the suite'sload).
longshot_vk.dll) of the head compiled and linked on this PC as CI's build-windows.yml builds it,with the pinned llvm-mingw 20261104 (
RX_TARGET=windows RX_VK_ONLY=1 RX_RELEASE=1 nx2/build.py --standalone): 0 failed.The three changed test programs compiled for Windows (the workbench's fixture syntax check, and the adapter test linked
with llvm-mingw), not run there.
Not tested:
there: a start on a new driver finds the driver's part of the shaders made for another driver (
warmed_stampholdsdriverVersion) and makes it again before play (prepare_run), from the source, not run here.the file on RADV, arrivals a little rougher on 4 cores; a Mesa rule would read the folder
MESA_SHADER_CACHE_DIRnames).Must be tested:
(the default with mesh shaders), then the same with
RX_VK_PCACHE_LOAD=1; expecting the log'spipeline cache file not readline,play's generation ... let golines as play goes on, nolisted pipelines ... in no cacheat the close, andplay no rougher than with
RX_VK_PCACHE_LOAD=1(by hand: test request T-XTVW, waiting for thetesting wantedlabel;free -mduring play both ways)Risks
What could break: on the PCs the rule covers, a program loaded at create takes its pipelines from NVIDIA's disk cache
instead of the file's in-memory copy: each load is slower on the load workers and draws wait for a load a little more often
(the table). If NVIDIA's cache holds 512 MB or more that doesn't answer this game (another program's shaders, or an old
driver's), pipelines the file would have answered are compiled when first needed; more than 500 listed misses write
longshot_load_start.txt, and the next start loads programs before the title, where the rule reads the file again.Turning it off:
RX_VK_PCACHE_LOAD=1reads the file again; reverting the commit is enough.Compatibility: nothing to redo after updating: the shader stamp's files are untouched, so the program cache and the
pipeline cache files stay. Where the file isn't read it and its play files stay on disk as they were, for a start that reads
it (they are neither updated nor written over), unless another translator made them: those go at the start, as where it is
read. The first start after an update that changes the translator prepares as on main, and on these PCs keeps what it builds
in NVIDIA's own cache, not in the file. No
MKWVK_API_VERSIONchange, no options or save change.Issues
See #388
See #223
See #415
See https://github.com/NotRexed/Hookshot/pull/2206
Checklist
Everyone
type(scope): description) of the status line's type.main.python -I -B tests/run_all.pypasses (any failure that also fails onmainis named).third_party/untouched (a new folder with aPROVENANCE.mdinstead).When it applies
Assisted-by:/Co-Authored-By:trailers.docs/tested/for each new measured run (format indocs/tested-configurations.md).mainand passing here.tests/that reads changed source updated in this PR, not weakened.🤖 Generated with Claude Code
Description completed: the measured commit named on the 'this PR' line (999bc37f, rebased since onto
5e072fc8unchanged, and why the numbers still hold), LONGSHOT_PREPARE_ONLY in the flags table, the Windows back end compiled and linked with the pinned llvm-mingw (Linux build, as CI does), and test request T-XTVW for a longer session on another 16 GB Linux NVIDIA PC.WIP: perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.7-0.8 GB less)to perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.7-0.8 GB less)Read head
4fd82b75against main5e072fc8: the complete eight-file diff, description, sole comment, empty reviews, issue 388 and both comments, cache readers/writers, program-load dispatch and the relevant changes in 415 and 370; ran nine game-free check invocations across five files, all passed, with GCC and Clang for the compiled checks. No game, full build or timing run.What's good
vkCreatePipelineCache; file-backed input avoids a large temporary byte vector, not the driver's retained cache. The reported memory reduction is consistent with removing that retained input.play_files_merge()prevents an empty base cache plus 16 rolling files from replacing the larger base file. The newno-savecase passes with both compilers and preserves the authored base and rolling files.Must change
Reach generation retirement from the ordinary save entry point. In
gpu/vk/vk_draw.cpp, lines 945-949, the new no-save branch destroys retired play caches, but the periodic caller cannot reach it:vk_replay.cpp, line 4099, callspipeline_cache_save(false), whose first condition atvk_draw.cpp, line 10343, returns whenpcache_no_saveis true.play_cache_begin()still creates a generation, andcreate_cached()keeps adding play pipelines to it. On the PCs newly selected by this default, those additions therefore stay in the same generation until close, instead of being retired at the periodic saves as on main. Route generation retirement around the file-write prohibition while keeping the current user-count synchronization. Extend the fixture throughpipeline_cache_save(false), after enough new pipelines to trigger a save, rather than callingplay_cache_save()directly. It should prove that the old cache is destroyed, new work still succeeds, and no file is written. Traced through the source; the passingno-savecase does not exercise this caller. No long-session memory run.Use the selected driver's cache for the cold-cache guard. In
gpu/vk/vk_draw.cpp, lines 1055-1057, the new decision treatsdriver_cache_bytes(nullptr)as NVIDIA's cache size. That helper, lines 9619-9633, adds NVIDIA's and Mesa's folders together. A fresh, empty__GL_SHADER_DISK_CACHE_PATHwith at least 512 MiB left in Mesa's folder therefore selects “file not read”, even though NVIDIA's cache is empty. This is not just the disclosed case of other programs filling NVIDIA's cache: an unrelated driver's bytes defeat the explicit cold-cache fallback. The base and rolling files are then ignored, and listed pipelines which neither live cache can answer are deferred to their first draw or dispatch (vk_programs.cpp, lines 240-255). Count only the selected driver's cache for this decision and treat an unavailable scan as unknown/cold. Add an adapter-level case with an empty NVIDIA folder and a populated Mesa folder, requiring the file to be read. Traced through the source, not reproduced in a game; the new table test supplies a number directly and never exercises folder selection.What the condition actually selects
#ifndef _WIN32, and the GPU check isvendorID == 0x10DE. On the supported platforms this means Linux with an NVIDIA GPU, not specifically NVIDIA's proprietary driver. Mesa NVK is not excluded by that check;dev.driver_idis already available if proprietary-driver-only is the intended scope.sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGE_SIZE), shifted right by 20. It is total RAM reported by the OS, not current free RAM, VRAM or a process memory allowance. Unknown RAM falls back to reading the file. The inclusive limit is 20,480 MiB, or 20 GiB, with the byte count rounded down to MiB. A typical 16 GiB PC qualifies, including the reported 15,757 MiB machine. Typical 24 and 32 GiB PCs do not; unusually large hardware reservations can change the reported total.programs_load_decide()runs after device creation and before this choice. The existing load mode matters: native mesh normally selects create, while the fallback, compressed program cache or an existing load-start marker can select start. This PR does not help the issue's original load-before-title configuration by default.RX_VK_PCACHE_LOAD=0,off,falseornoforces the skip;1or another non-off value forces loading. Empty is treated as automatic. The switch wins over preparation too:choose(0, true, ...)skips the file, and the new test explicitly requires that. The documentation's “always by --prepare-shaders” needs that exception.Cold, disabled or unwritable driver caches
A genuinely empty aggregate scan falls back to main's file-loading path. A first start without a valid warm marker also has the existing up-front preparation/warm pass, so it is not correct to say every cold first start necessarily compiles everything in play.
The size check does not establish that this game's entries are present, compatible or usable. It does not check
__GL_SHADER_DISK_CACHE=0, and it does not establish usability on a read-only home. Read-only storage alone does not prove existing entries cannot be read. NVIDIA also documents that its cached binaries are driver-version-specific: https://download.nvidia.com/XFree86/Linux-x86_64/570.124.04/README/openglenvvariables.html. These cases need separate evidence, not a directory-size inference.With an existing warm marker, a skipped file and driver misses, loads at create only probe the caches. Missing pipelines are compiled when first needed in play. More than 500 listed misses can write
longshot_load_start.txt, making the next run load before the title and disabling this automatic memory saving. The description discloses unrelated NVIDIA entries, but “this costs nothing beyond main” is not established for disabled caches, read-only storage or other drivers.Files retained for the next run
When the file is skipped, neither the base file nor new rolling files are written. Existing published files and their stamp are left alone; old temporary-file cleanup still runs. A later run that reads the file can reuse the retained base and rolling files if their translator stamp and Vulkan vendor/device/cache UUID still match. Otherwise the normal stamp/header checks reject them. This preserves an existing valid cache, but does not keep it updated with the skipped run's new pipelines or make an old-driver file valid for a new driver.
The explicit
RX_VK_PCACHE_LOAD=0overwrite protection does work independently of the new default. The new retirement branch also stops the close path from writing a rolling file. The periodic-retirement problem in finding 1 is about retained memory, not a remaining overwrite path.Measurements
The reported warm-cache comparison meets the important paired-memory conditions: named builds, the same 16 GB PC and settings, and three alternating tours per side with medians. Arithmetic from the committed run rows gives:
The first two ranges are disjoint and their median gaps exceed their within-side spreads. That supports the claimed memory reduction on this PC and tour. MemAvailable has overlapping ranges and substantial variation; it does not establish that memory pressure is solved. The reported fps difference is smaller than the spread and was explicitly not collected as a controlled timing comparison. There is no demonstrated speed gain. The reported program-load median worsens from 1.8 to 5.0 ms, and total first-use waits from 52 to 206 ms; retain all three runs' load and wait rows to show their spread, not just one excerpt and the medians. No overall startup-time improvement is shown.
A cold-driver-cache run was measured, once per side, with 10,143 and 10,901 graphics misses. However, the candidate read the file in that run. It tests the cold fallback, not the behavior of skipping the file while the relevant driver cache is cold or disabled. It also lacks repeated cold pairs.
Correct the rebase explanation. Main's intervening change includes
gpu/vk/eden/shader_recompiler/ir_opt/scene_tonemap_pass.h.nx2/build.py, lines 1178-1182, hashes every file underedeninto the shader stamp. That change therefore does go through cache invalidation, even though the pipeline-loading code is unchanged. Keep the numbers labelled as measured on999bc37f; explain the changed cache identity and distinguish warmed steady-state results from the first start after updating. The measured commit was not available from the fetched refs or the public commit endpoints, so its claimed identical replay was not independently established here. Full per-run logs and memory samples are not linked in the thread.Overlaps and remaining checks
pcache_init(). This PR deliberately keeps generations. Do not combine those approaches unchanged: 415's guard would prevent the retirement design here from running at all and send new pipelines into the base in-memory cache. Their switch documentation and fixture cases also need one combined definition. No combined build or test was run.vk_api.hdoes not exclude the changed Eden header above. Installing its new formula causes a one-time invalidation. While this PR skips loading, the old pipeline files and stamp remain untouched; on a later loading run, normal stamp rejection still applies. Do not relabel old warm-cache evidence as a combined-stack update test.test_pcache_load,test_pipeline_play_cache(13 cases),test_pcache_stampandtest_pcache_fetch_lock, each with GCC and Clang, plustest_gpl_library_cache. No test was weakened. These are rule, file-model and source checks, not a native NVIDIA-cache reproduction.The same idea measured on AMD with RADV, a Steam Deck OLED, in case the rule goes past NVIDIA. Not this PR's code: the switch main already has,
RX_VK_PCACHE_LOAD=0, against unset.Setup. Steam Deck OLED (AMD Custom APU 0932, RADV VANGOGH, Mesa 26.2.0-devel), 16 GB shared with the GPU, SteamOS 3.9.5, kernel 7.2.7-valve2. A renderer built from main
514591a3with renderer-only changes of other pull requests (#238, #384, #387, #391, #223's commit), programs loaded at create (RX_VK_LOAD=create), Minimum preset, handheld profile, no screen (headless). The pipeline cache file is 307 MB there. Each run starts the game and loads a save by script; memory is read from/proc/<pid>/statusand/proc/meminfoat the end of the measured span, KiB divided by 1,024.What it gives back (
RssAnon, MiB, file read then not read):~/.cache(1,489 MB)About 200 to 300 MiB for a 307 MB file.
MemAvailablefollows (heavy view 2,488 / 2,652 against 2,762 / 2,802 MiB). GPU memory (drm-memory-gtt) does not move, and standing still the frame rate is the same (29.28 / 29.57 against 29.46 / 29.36 fps).What it costs here.
RX_VK_FRAME_TRACE=1, time spent in frames over 60 ms): Kokiri Forest 5.5 s read against 3.1 and 6.0 s; heavy view 3.2 against 3.4 s; Hyrule Field 2.5 / 2.8 / 3.5 / 2.6 s read against 3.0 / 3.4 / 3.8 / 3.5 / 2.6 s. In Hyrule Field the longest frame is about 1 s in 4 of the 5 runs without the file (514 to 1,011 ms) against 1 of the 4 with it (529 to 1,118 ms). Small samples, but on this 4-core machine the workers building pipelines seem to cost the replay while it uploads the area's textures.For the driver cache guard (the review's point 2). On a Deck in Game Mode, Steam points Mesa at a cache of the game's own,
steamapps/shadercache/<the shortcut's id>/mesa_shader_cache_sf(a single-file cache, throughMESA_SHADER_CACHE_DIR), not at~/.cache/mesa_shader_cache. On this Deck the first holds 187 MB and the second 1,489 MB (filled by headless runs started over SSH). The 187 MB one answered every pipeline in the runs above: the copy grew by 3 MB in its first run and not after. So a 512 MB threshold on a folder's size would look at the wrong folder here, and would turn down a cache that is complete. If the rule goes to Mesa, the folder to read is the oneMESA_SHADER_CACHE_DIRnames when it is set.Not tested: this PR's code; a start from Steam itself (its cache was a copy, read through
MESA_SHADER_CACHE_DIRandMESA_DISK_CACHE_SINGLE_FILE=1in a headless run); a cold Mesa cache; a long session. The game folder's cache files were copied aside and put back after each series.perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.7-0.8 GB less)to WIP: perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.7-0.8 GB less)@Beanstalk
Beanstalk test farm
Commit
0863e13· tier untrusted · approved by nobody yet · dashboardA member of
BigBeanscan review this commit and comment@Beanstalkto test it.4fd82b75591433d4d393WIP: perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.7-0.8 GB less)to WIP: perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.8 GB less)Thanks for the careful review, @mooki. Both must-change points are fixed, plus one more thing the re-measure turned up (stale files, below). The branch is rebased onto main
ce9d49b4and measured again on414f56e2and on82886e89. Point by point:1. Generation retirement from the ordinary save (
61e672fd). Fixed as you described:pipeline_cache_save()no longer returns whenpcache_no_saveis set. It skips only the cache's own part (never fetched nor written) and still callsplay_cache_save(), with the same user-count wait, so the ordinary save every 300 presents takes play's generation away and frees it, writing nothing. The fixture goes through the real entry point now:tests/test_pipeline_play_cache.pyno-save-periodicextractspipeline_cache_savetoo and runspipeline_cache_save(false)into the real play caches. Under the threshold nothing happens. Past it, the generation is destroyed on the save's thread, new work goes into the next one, and the folder holds only the base file afterwards (no play file, no temporary). It fails on the previous head4fd82b75at the retirement (line 495) and on main, and passes with GCC and Clang. It runs over main's newbetweenpath too (b2911d33), which the rebase kept as main has it. The game logsplay's generation <n> let go, not writtenfor the first four and every 16th, and it does in the PR's tours (for exampleplay's generation 2 let go, not written (the file not read): 21 pipelines, 9 from the driver's cache, 12 compiled).2. The selected driver's cache for the cold-cache guard (
03fd3c83). The rule counts NVIDIA's own folder alone now, the one its driver uses per its README:__GL_SHADER_DISK_CACHE_PATH'sGLCache, else$XDG_CACHE_HOME/nvidia/GLCacheor~/.cache/nvidia/GLCache, or the older drivers'$XDG_CACHE_HOME/.nv/GLCache/~/.nv/GLCachewhere they exist (the README says the driver reuses them). Mesa's folders don't count. A cache that is off (__GL_SHADER_DISK_CACHEan off value, or none of the three variables set, where the README says the driver keeps none), a folder that can't be walked through, one that isn't a folder, or one under 512 MB: the file is read. A folder that isn't there counts as 0.driver_cache_bytes()stays for the start's log line only.The adapter-level case is in
tests/test_pcache_load.py:pcache_load_off()itself is cut out ofvk_draw.cpp, compiled with stand-ins for the driver ID, the load mode and the RAM, and run once a case against real folders (sparse 600 MB files).mesa-onlyis your case: a fresh, empty__GL_SHADER_DISK_CACHE_PATHwith 600 MB in Mesa's folders, which reads the file. 20 cases in all, with GCC and Clang. In the game, on the PR's folder:What the condition selects:
dev.driver_id == VK_DRIVER_ID_NVIDIA_PROPRIETARY, so Mesa's NVK reads the file (annvkcase in the test).RX_VK_PCACHE_LOAD=0behaved before), and the docs row now says the switch wins over--prepare-shaderstoo.switch-onandswitch-off-preparecheck that).Stale files (
da116f0e). Updating main from414f56e2to82886e89changed the translator stamp (sizeof(Program),b2911d33). Main dropped its 14 pipeline cache files (1219 MB), but the PR's folder kept them, becausepcache_drop_stale()returned early whenever the file wasn't read. On the PCs this rule covers, that would leave 1.2 GB of dead files on disk after every release. Those files now go whether or not the file is read (RX_VK_PCACHE_STAMP=0still keeps them;tests/test_pcache_stamp.pychecks it). The game showed it: the head's renderer on the PR's folder logged14 files made for another translator dropped (1219 MB)and still didn't read the file. The same update is also the nearest thing to your driver-update case I could run here (a translator change, not a driver change): with the file unread, the PR's folder prepared every shader again at its first start and loaded at create from then on with 0 listed pipelines in no cache.Cold, disabled or unwritable caches: disabled (
__GL_SHADER_DISK_CACHE=0) and unreadable now read the file. A read-only cache isn't a reason to read, as you say, so the rule doesn't look at it. On driver updates: agreed that the size doesn't establish usability. What I found in the source, not run here (no second driver to try):warmed_stamp()holdsdriverVersion, so the first start on a new driver finds the driver's part of the shaders made for another driver and makes it again before play (prepare_run, "made again"), which is what fills NVIDIA's new-driver cache. The >500-misses fallback you describe is real. If NVIDIA's folder holds 512 MB that doesn't answer this game, misses are compiled at first use, and past 500 the marker moves the next start to loading before the title, where this rule reads the file. I took "costs nothing beyond main" out of the description; Risks now says exactly this.Files retained: agreed with your reading. The description's Compatibility says the file and its play files stay untouched where it isn't read, for a later start that reads it, and that they aren't updated.
Measurements: measured again, three interleaved tours a side on main
414f56e2, then two comparable pairs on main82886e89after a warm-up pair. Every run's load and wait row is in the description now, not just one excerpt. Private memory + swap at the end: -775 MB on414f56e2(3 + 3) and -816 MB on82886e89(2 + 2), every PR run below every main run. Peak game total: -452 and -816 MB. Swap peak: -1.07 and -1.37 GB. Loads at create: 1.9-2.0 → 5.0-5.1 ms. Waits: 1-4 → 6-7 a tour. MemAvailable: I've kept it as an observation and don't claim the memory pressure is solved. fps isn't a timing comparison (the desktop's card), and the description says no speed gain.pcache_no_saveis false, soplay_files_merge()and both saves run as on main. Repeated cold pairs would compare main with main, so I ran the two decisions in the game instead (above).0caf0618and5e072fc8does change the stamp. The new numbers are from builds of main and the PR on it, with main's prepared caches handed to the PR's folder. That's right for an update to this PR because the PR changes none of the translator stamp's inputs: none oftranslator_files.py's stamped files (eden/,sirit/,compat/,vk_shader.cpp,vk_bank.cpp,vk_state.h), and none of the records whose sizestranslator_stamp()folds in. The first start after updating main itself can translate again, which is main's change, not this PR's (82886e89did). The measured commits are on the fork asmeasured/416-9ea38a42andmeasured/416-f0963cdc(andmeasured/416-999bc37ffor the first measurement), so they can be fetched. The last rebase, ontoce9d49b4, brings in onlyscripts/longshot_ctl.py's Windows console flag. The old runs stay indocs/tested/labelled with 999bc37f and0caf0618.Overlaps: #415 is still open. I haven't combined them. Whichever lands second has to take one design: this PR keeps the generations and frees them at each save, while 415 stops them from starting (so play's pipelines go into the base cache, which isn't saved, and stay for the session). I'd keep this one's, since it gives the memory back. Then 415's merge guard in
pcache_init()is redundant withplay_files_merge()'s own early return here, and itsplay_cache_begin()guard would undo the retirement. #370 is merged. The new runs were built on it and nothing is relabelled.Checks on the head's code: the whole suite on Linux gives 538 passed, 10 failed, 30 skipped. Main
82886e89run the same way gives 537 passed and the same 10 failures, listed under Testing (test_crash_recordpasses alone on both). The Windows back end compiled and linked with the pinned llvm-mingw 20261104, and the three changed test programs compiled for Windows (syntax and one link, not run).One more thing, in the interest of full disclosure: the short start that checked the stale-file drop (4 minutes, idle at the title) logged four NVRM lines 3 s before its timeout ended it:
dmaAllocMapping_GM107: can't alloc VA space for mapping, andNV_ERR_NO_MEMORYfrom the mapping reuse callback. There was no device lost and nothing in the game's log. None of the other 21 game runs today, main's or the PR's, logged any. I don't understand it yet, so it's under Testing as it is.@p0lux, thank you for measuring it on the Deck. Two things I took from it for this PR:
__GL_SHADER_DISK_CACHE_PATH'sGLCache, else$XDG_CACHE_HOME/nvidia/GLCacheor~/.cache/nvidia/GLCache, or the older drivers'.nv/GLCachewhere it exists). Mesa's folders no longer count toward it. So the case you describe can't come up with this PR, but you're right that a Mesa version of the rule would have to read the folderMESA_SHADER_CACHE_DIRnames, single-file cache included, and not a size under~/.cache. I've noted it in the PR as what a Mesa follow-up needs.WIP: perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.8 GB less)to perf(vk): skip the pipeline cache file on Linux NVIDIA PCs with 20 GB of RAM or less (0.8 GB less)This conflicts with
mainnow:docs/development.mdandtests/test_pipeline_play_cache.py. #415 added anRX_VK_PCACHE_LOADswitch of its own (with its own table row andload-*cases in that test), which overlaps this PR'sRX_VK_PCACHE_LOAD=0 or 1row and itsno-savecases, so reconciling them is a design choice I haven't made for you. Could you rebase onto currentmain?Code review of head
1433d4d3against current maina55fc6cf. Read only: nothing built or run.Verdict: comment. Not mergeable until the conflict with #415 is settled, and the hardware run it asks for is posted.
Earlier request for changes (mooki,
4fd82b75)Both must-change points are answered by the new head, as the diff shows:
pipeline_cache_save()no longer returns onpcache_no_save. It skips only the cache's own fetch and write, and still callsplay_cache_save(), which frees the generation and writes nothing. The newno-save-periodiccase goes throughpipeline_cache_save(false).pcache_load::nvidia_cache(). Mesa's folders never count, and a cache that is off, unreadable, not a folder or under 512 MB reads the file. NVK is out throughdriver_id == VK_DRIVER_ID_NVIDIA_PROPRIETARY. Themesa-onlycase intests/test_pcache_load.pyis the one mooki described.The measurements were also redone against newer mains, with per-run rows. That review is stale and can be dismissed once the conflict is settled.
The conflict is not a basic one
Main now has #415 (
35465d12), which fixes the sameRX_VK_PCACHE_LOAD=0problem the opposite way. It stops play's generations from starting (play_cache_begin()returns onpcache_no_save), guardsplay_files_merge()inpcache_init(), and adds its own docs row andload-*cases. This PR keeps the generations and frees them at each save. Merged together unchanged, 415's guard inplay_cache_begin()means no generation is ever made. Play's pipelines then pile into the in-memory base cache for the whole session, which is the retention this PR's second fix removes, andno-save-periodicwould fail. Resolving it means dropping or rewriting 415's guard and its cases, and merging the twoRX_VK_PCACHE_LOADrows indocs/development.md. That is a design choice for the author (the PR already said it would keep its own design), so I have not pushed a fix. Please rebase onto main, keep one retirement design, and update 415'sload-*cases intests/test_pipeline_play_cache.pyinstead of weakening them.The default, for Linux NVIDIA (Tier 1)
I think it is right for everyone it covers, and everyone else reads the file exactly as before:
--prepare-shadersrun. Anything unknown (RAM, folder, a walk error) reads the file.longshot_load_start.txt, so the next start loads before the title and reads the file again.RX_VK_PCACHE_LOAD=1or a later start on a bigger PC still finds it. Stale files from another translator now go in both cases, astests/test_pcache_stamp.pychecks.hos_main.cmakes the folder that__GL_SHADER_DISK_CACHE_PATHnames and gives the cache 10 GiB, so the rule's folder is the one the driver uses. Without that variable the driver's own default is walked.Points for the author
dmaAllocMapping_GM107,NV_ERR_NO_MEMORY) on the stale-file start are disclosed and not understood. The long-session run should watch for them.Testing
test_pcache_load,test_pipeline_play_cacheno-saveandno-save-periodic,test_pipeline_cache_save,test_pcache_stamp). The author's full suite shows the same 10 failures as main. I ran none of them.RX_VK_PCACHE_LOAD=1) is not yet reported. The description's own Must be tested names it, so I am not approving before it is posted. The conflict also leaves no head that has been run.82886e89while main is now far past it. The change touches none of the translator stamp's inputs, so I think the numbers still hold, but a re-run after the rebase is cheap.Merge order: settle with 415 (already merged) first, so this PR rebases onto it.
Tested on Linux NVIDIA. This PR's 7 commits are rebased onto
main@43f40b87:HumanBean:rebase/pr416-on-main, if you want it. Two conflicts came up, both resolved keeping both sides:docs/development.md'sRX_VK_PCACHE_LOADrow;tests/test_pipeline_play_cache.py's cases.One thing needs your call. #415 landed on main with
pcache_no_save ||inplay_cache_begin(), so with the file unread no play generations start. This PR's design keeps them and lets them go at each save, which is the opposite. Itsno-savetests fail against that guard. To measure the PR as you did, my local build also removed that one guard. That commit is not in the branch above, because it would break #415'sload-off*tests. The two need reconciling in the PR.Hardware: RTX 3090, NVIDIA 620.70.06, Ryzen 9 7900X, 30 GB RAM, CachyOS.
Method: the automatic rule doesn't apply at 30 GB, so it's forced with
RX_VK_PCACHE_LOAD=0against=1, on one build and one game folder. The folder holds a real 221 MBlongshot_vk_pipelines.bin, copied from a played folder. Programs load at create, interleaved, two sessions each.Route: DLAA 2560x1440, Hyrule Field then the Market (teleports), walking and turning. Memory is the renderer's
private N MBlines, plus RssAnon+RssShmem+VmSwap sampled every 2 s.=1)=0)privatepeak (MB)On this PC, leaving the file unread saved no memory I could measure. Both counters are the same within run-to-run noise. It did cost what you measured: the median load at create went from about 1.3 ms to about 3 ms, and 10 s more in all. fps is the same. With 30 GB the page cache may keep things that a 16 GB PC under pressure would drop, so this doesn't rule out your 0.8 GB on a 16 GB PC. It only means the saving didn't reproduce here.
🤖 Generated with Claude Code
A reconciliation with #415, for you to take or leave:
HumanBean:rebase/pr416-on-mainnow has one more commit on the rebased branch (a690b1ea).play_cache_begin()skips the play generations only whenRX_VK_PCACHE_LOADitself is off (0/off/...), which is #415's A/B with nothing kept. The file left unread by your automatic rule (pcache_load.h) keeps them, andplay_cache_savelets each go, as you designed. So #415'sload-off*cases (set through the switch) and yourno-savecases (withpcache_no_saveset and the switch unset) should both hold.tests/pipeline_play_cache_test.cpphad kept both sides'bool pcache_no_saveandstd::atomic<size_t> pcache_saved_sizedeclarations, which wouldn't compile. Each is declared once now.I haven't run the checks on this commit (
tests/test_pipeline_play_cache.py), since I can't run tests on this PC. CI on your PR will, once it's in your branch. This doesn't change the measurement above: forced on with a real 221 MB file on a 30 GB PC, it saved no memory and slowed loads at create from about 1.3 to 3 ms.🤖 Generated with Claude Code
1433d4d393a690b1ea9aa690b1ea9a0863e13f53I've pushed onto this branch (at the maintainer's request), so you'll want to pull before working on it again. It now holds your 7 commits rebased onto today's
main(71b1edd8), plus one of mine. Head is0863e13f, and Forgejo shows it mergeable.Conflicts resolved:
docs/development.md'sRX_VK_PCACHE_LOADrow andtests/test_pipeline_play_cache.py's cases: kept both sides.pipeline_cache_save(): main has since addedpcache_save_outcome(kNoCache,kOff, ...). I kept those lines and your rule. With the file unread the save still runs to let play's generations go, and it recordskOff, whichtests/test_precompile_save.pylooks for.My commit, the #415 reconciliation (described in my comment above):
play_cache_begin()skips the play generations only whenRX_VK_PCACHE_LOADitself is off, which is #415's A/B. Your automatic rule keeps them. The rebase had also leftpcache_no_saveandpcache_saved_sizedeclared twice intests/pipeline_play_cache_test.cpp, so that's one of each now.I haven't run the checks on it, since I can't run tests on my PC. CI's run on this head will say. If you'd rather have your own history back, it was
1433d4d3.🤖 Generated with Claude Code
Tested for issue 388 on Linux, RTX 3090 (NVIDIA 620.70.06), with the game in a 13 GB memory cgroup (
systemd-run --user --scope -p MemoryMax=13G), roughly a 16 GB PC less its desktop. This PC has 30 GB, so this PR's gate (20 GB or less) doesn't switch on here. Instead,main@ac778751ran with the pipeline cache file read (its default) againstRX_VK_PCACHE_LOAD=0, which is what this PR makes the default on such PCs.Route: DLAA 2560x1440, threading on Auto, a save loaded, then Hyrule Field and the Market, walking then turning, about 3 minutes of play. 2 runs each, interleaved M C C M. The cgroup's memory and swap were sampled every 2 s.
Verdict: no difference I can measure on this route. Both arms sit at the limit and swap under 1 GB. fps is the same, and the hitches are within run-to-run noise (one main run had a 171 ms frame).
This doesn't reproduce issue 388's growth. That needs the long tour: about 11 minutes through area changes, where the report's swap climbed to 15 GB. The cgroup's peak also counts page cache, which the kernel drops before it swaps. So this says the file costs nothing measurable over a short session at 13 GB, not that it doesn't matter over a long one. The memory tour of the original report (Zora's River, Zora's Domain, Hyrule Field, Kakariko) under the same limit is the test that would settle it.
🤖 Generated with Claude Code
The long tour for issue 388 on Linux, RTX 3090 (NVIDIA 620.70.06), with the game in a 13 GB memory cgroup (
systemd-run --user --scope -p MemoryMax=13G), roughly a 16 GB PC less its desktop.main@ac778751ran with the pipeline cache file read (its default) againstRX_VK_PCACHE_LOAD=0, which is what this PR makes the default on Linux NVIDIA PCs with 20 GB or less.Route: DLAA 2560x1440, threading on Auto, a save loaded, then nine areas by the debug teleport: Zora's River, Zora's Domain, Hyrule Field, Kakariko, Death Mountain Trail, Goron City, the Market, the Lost Woods, Kokiri Forest. About 9 minutes each, interleaved M C C M. The cgroup's memory and swap were sampled every 2 s.
Verdict: this route reproduces the issue, but this PR doesn't change it.
The swap comes from elsewhere. The original report's own numbers point at the private memory the game builds up while a save and its areas load (8.3 GB
RssAnonat 0:31). Samplingsmapsby mapping during those loads, under the same limit, is the next step: it would show which allocations they are.🤖 Generated with Claude Code