WIP: fix(vk): give memory back under Linux memory pressure and after the precompile #223
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/upscaling
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
testing wanted
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!223
Loading…
Reference in a new issue
No description provided.
Delete branch "dukenheimer/Longshot:brett-d/legendary-broccoli"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
On a 16 GB Steam Deck OLED, the game's guest memory (4.7-6.0 GB shmem), the renderer's anonymous heap (up to 8.9 GB on
main) and kernel-pinned VMA blocks share one 16 GB pool while RADV's device-heap budget (~7.5 GB) reports VRAM under 50% used: loading into Kokiri Forest droveMemAvailabledown to 354-390 MB with 2.8-4.6 GB in swap and terminated all 3 starts onmain. This releases the precompile's shader modules on<= 20 GBRAM systems once their pipelines are built and gives back texture blocks, idle staging/capture buffers and idle guest-pool imports under Linux memory pressure beforeearlyoomor direct reclaim acts.What changed
gpu/vk/linux_pressure.h): readsMemAvailable,SwapFreeandMemTotalfrom/proc/meminfoon Linux, computes the shortfall belowRX_VK_AVAIL_RESERVE_MB(default1536MB) andRX_VK_COMMIT_RESERVE_MB(default2560MB, matching Windows' commit rule), gates post-precompile module release (should_drop_modules: on by default for<= 20 GBRAM or active pressure, off on 32-64 GB desktops), and providesPressureHold(cuts the texture budget from the texture total at pressure entry with a 512 MB floor, holds it without per-frame ratcheting while VMA compaction frees blocks, and leaves pressure only once both reserves clear by 256 MB).texture_frame()ingpu/vk/vk_texture.cpp): samples/proc/meminfoevery 30 presents and stores the shortfall inlinux_shortfall_bytes(soimports_give_back()reads the atomic instead of opening/proc/meminfounderpools_mu). WhilePressureHoldis active, applies the held budget tobudgetandrule_budget, drains the render-target pool (pool_pressed = true), wakes VRAM compaction at its pressed thresholds (compact.pressed = true, 192 MB free space, 120-present rest), frees idle stagingarenason pressure entry and after 5 s idle (instead of every pressed frame), checksmalloc_trim(0)every 600 presents only when the previous trim freed>= 16 MBor free arena bytes grew by>= 64 MB, and logsvk: memory pressure on Linux:at most every 600 frames.cap_free_trim(),imports_give_back()ingpu/vk/vk_memory.cpp): under Linux memory pressure (linux_pressure_shortfall()), trims spare capture buffers (cap_free) down to 32 MB and adds the shortfall (linux_pressure::import_want(over, dead_import_bytes)) toimports_give_back()so idle unmirrored guest-pool imports are unpinned even on< 12 GBsystem-heap budgets.program_cache_read(),program_cache_drop_modules(),program_cache_complete_graphics()ingpu/vk/vk_progcache.cpp;precompile_run(),graphics_pipeline(),pipeline_cache_play()ingpu/vk/vk_draw.cpp;gpu/vk/vk_shader.h): records each uncompressed record's(record_at, record_size)and each stage's(code_at, code_words)inlongshot_programs.bin(no change toLONGSHOT_SHADER_STAMPorkProgramCacheFormat). On Linux with<= 20 GBRAM or active pressure (should_drop_modules), onceprecompile_run()finishes (pipeline_cache_play()), destroys theVkShaderModulehandles of ready programs whose pipelines were built in the precompile (modules_dropped = true), remakes them on demand fromlongshot_programs.binviaprogram_cache_complete_graphics()(checking the record head atrecord_atand markingp.failed = trueand leaving the pipeline unbuilt on any read orvkCreateShaderModulefailure), and callsmalloc_trim(0).tests/test_linux_pressure.py,tests/linux_pressure_test.cpp): game-free source and compiled C++ checks for/proc/meminfoparsing (MemAvailable,SwapFree,MemTotal),should_drop_modules,PressureHoldhysteresis/floor/no-ratchet,import_want, andprogram_cache_drop_modules()/program_cache_complete_graphics()(including failedmake_moduleand stale record head checks).docs/development.md,docs/tested-configurations.md): documentsRX_VK_AVAIL_RESERVE_MB,RX_VK_DROP_MODULESandRX_VK_KEEP_MODULES, updatesRX_VK_COMMIT_RESERVE_MBfor Linux, and adds the Steam Deck OLED rows.Flags and settings
RX_VK_AVAIL_RESERVE_MB=<MB>MemAvailablein/proc/meminfo, sampled every 30 presents) the renderer leaves free on Linux before trimming the texture budget (PressureHold), pools, staging/capture spares and idle imports;0turns theMemAvailablecheck offMemAvailable >= 1536 MB/proc/meminfo), 32+ GB Linux PCs with ampleMemAvailableRX_VK_COMMIT_RESERVE_MB=<MB>MemAvailable + SwapFreein/proc/meminfo) alongside Windows (ullAvailPageFile);0turns the check offMemAvailable + SwapFree >= 2560 MBRX_VK_DROP_MODULES=0or1pipeline_cache_play()lets go of ready programs'VkShaderModulehandles afterprecompile_run()on Linux (1forces on any RAM size;0keeps them)<= 20 GBRAM or active pressure, off on> 20 GBdesktops=1)RX_VK_KEEP_MODULES=1RX_VK_DROP_MODULES=0: keeps ready programs'VkShaderModulehandles resident afterprecompile_run()on Linux0): for A/B testingMeasurements
main:
08771847(2026-11-13; this PR's base is7315957c, a few commits newer with no memory changes)this PR: brett-d/legendary-broccoli @
05f4c4aa(6931f715withdocs/tested-configurations.mdupdated)Hardware: Steam Deck OLED (16 GB shared RAM, 8 GB swap file, AMD Custom GPU 0932, RADV Mesa 26.2.0-devel, SteamOS 3.9.5, kernel 7.2.7-valve2, Gaming Mode 1280x720, stock clocks, mesh shaders on compute fallback)
Scene: Kokiri Forest moving route; Quality preset Auto (this PC), each build in its own folder and caches, without the Steam Deck profile (#225) so both keep all 25,303 graphics pipelines ready (
RX_VK_LOAD=start)Method: Full Linux build of host + renderer from source; memory sampled once a second from
/proc(RSS,RssAnon, processVmSwap,MemAvailable,SwapFree) with the low-memory guard on (kills whenMemAvailable < 500 MBfor 5 s with< 4 GBSwapFree, or< 250 MBwith< 2 GBSwapFree). One cold and one warm run onthis PR @ 05f4c4aa; one cold and two warm runs onmain @ 08771847(mainwas killed by the guard in all 3 starts before reaching a measured window, so real fps are not comparable; full report in #223 (comment)).How to reproduce: On a Steam Deck OLED in Gaming Mode (1280x720, stock clocks) with Quality preset Auto (this PC), run the Kokiri Forest moving route on a cold and a warm cache with
main @ 08771847andthis PR @ 6931f715.08771847(2 warm runs)05f4c4aa(1 warm run)RssAnon)VmSwap)MemAvailablemin (SwapFreethen)< 4 GBswap free -> killed)< 500 MB, all with> 4 GBswap free)VkShaderModules let go after precompilevk: memory pressure on Linux:lines2,371 -> 1,465 -> 1,465 MBtexture budget; no ratcheting)p95)p9590 ms)mainkilled at 2 min)08771847(1 cold run)05f4c4aa(1 cold run)RssAnon)VmSwap)MemAvailablemin (SwapFreethen)< 4 GBswap free -> killed)vk: memory pressure on Linux:lines1,732 MBx3, then recovered to2,560 MBonce memory returned)p95)p9596 ms)mainkilled at 6 min)Testing
Tested:
main @ 08771847vsthis PR @ 05f4c4aa(#223 (comment)), confirming41,044shader modules let go on warm start (RssAnondown from8.9 GBto2.95 GB, swap down from2.8-3.5 GBto0.95 GB),PressureHoldholding the texture budget without ratcheting (2,371 -> 1,465 -> 1,465 MBwarm;1,732 MBx3 ->2,560 MBrecovery cold), and 0 kills across both starts; plus a warm moving-route run combined with #225 (5,028compute/fallback modules let go while vertex/fragment modules stayed resident,RssAnonpeak2,496 MB, swap peak247 MB,MemAvailable >= 1,110 MB).clang++andg++, llvm-mingw) and in the Linux Ubuntu 22.04 build container (clang++andg++):tests/test_linux_pressure.py,tests/test_commit_limit.py,tests/test_vram_compact.py,tests/test_retired_give_back.py,tests/test_mesh_fallback.py,tests/test_prepare_cache_save.py,tests/test_precompile_pipeline_cleanup.py,tests/test_pipeline_play_cache.py,tests/test_oot_wide_save_picture.py,tests/test_mesh_args_cleanup.py.python -I -B tests/run_all.pyon6931f715, rebased onmain @ 7315957c): 488 passed, 6 failed, 34 skipped. All 6 failures also fail onmainwithout this change (test_fpx_sticky,test_texture_staging_publication,test_display_pointer,test_check_third_party,test_spawn_emit,test_release).Not tested:
should_drop_modulesandPressureHoldstay idle above 20 GB RAM).VkShaderModulewas dropped atpipeline_cache_play()and remade viaprogram_cache_complete_graphics()on a 16 GB desktop GPU withRX_VK_LOAD=start(tests/test_linux_pressure.pycovers the exact C++ reload and failure paths).Needs other testers:
vk: programs: ... shader modules let go after the precompileat start-up and no out-of-memory kill or new pipeline hitches (request to open).Screenshots
Nothing on screen.
Risk and rollback
MemAvailable < 1536 MB),PressureHoldlowers the texture cache budget (with a 512 MB floor and 256 MB clear hysteresis) and unpins idle unmirrored guest-pool imports, which can cause extra texture streaming or re-imports on a 16 GB system that would otherwise be killed byearlyoom.RX_VK_AVAIL_RESERVE_MB=0andRX_VK_COMMIT_RESERVE_MB=0turn the Linux pressure checks off.<= 20 GBLinux systems, if a precompiled program needs a new pipeline state variant in play after itsVkShaderModulewas dropped atpipeline_cache_play(),program_cache_complete_graphics()re-reads its SPIR-V fromlongshot_programs.binon the pipeline build worker (and marks the program failed if the read orvkCreateShaderModulefails).RX_VK_KEEP_MODULES=1(orRX_VK_DROP_MODULES=0) keeps allVkShaderModulehandles resident as before. Reverting the commit is enough.Reviewer notes
gpu/vk/linux_pressure.h, post-precompileVkShaderModulerelease on<= 20 GBLinux systems) and is kept separate fromd4nkn3ss's work on Windows play-time memory growth across 12-area laps (worklist 5.6, https://github.com/NotRexed/Hookshot/issues/993) andHumanBean's shader-preparation set changes (Compile shaders: All / Most / None, #158).precompile_run()setsp.graphics_built = trueonly on programs whose graphics pipelines were built before the title (wanted.count(p.key)). Onmainalone, that keeps un-noted mesh-fallback programs' vertex/fragment modules resident; when combined with #225 (wherewantedis empty when graphics pipelines are left to first draw),program_cache_drop_modules()drops only the compute and mesh-fallback computeVkShaderModules and keeps all vertex and fragmentVkShaderModules in memory so first draws in play build from memory without readinglongshot_programs.bin.Issues
See #91
See #54
See https://github.com/NotRexed/Hookshot/issues/993
See https://github.com/NotRexed/Hookshot/issues/1000
Checklist
For everyone
type(scope): description).main.python -I -B tests/run_all.pyrun and passing (any failure that also fails onmainis named above).third_party/untouched (a new folder with aPROVENANCE.mdinstead).If applicable
Assisted-by:/Co-Authored-By:trailers.docs/tested-configurations.mdfor each measured run.mainand passing here.Thanks. The Deck numbers are convincing: the guard killed
mainand this branch got through the route. I read the code only, without building or running it. Two things need changing before this leaves WIP, and a few smaller ones are below.Needs changing
program_cache_complete_graphics()can return false: the read fromlongshot_programs.binfails, orvkCreateShaderModulefails, which is most likely exactly when memory is short. All three callers ingraphics_pipeline()(sync, GPL, worker job) ignore the result, sobuild_graphics()passesVK_NULL_HANDLEas a stage module. That is invalid usage and can crash the driver.program_cache_complete()marks the program failed so its draws are left out; this path needs to do the same (inline comments).pipeline_cache_play()drops the modules unlessRX_VK_KEEP_MODULES=1is set, whatever the RAM. On a 32-64 GB desktop, every new pipeline variant of a precompiled program in play now pays a disk read and avkCreateShaderModule, to save memory that machine doesn't need. The title says "under Linux memory pressure", and the PR lists desktops as untested. Please either gate the drop (on total RAM, on the handheld profile from #225, or on pressure), or measure in-play hitches on a desktop and say so in the PR.Worth a look
MemAvailablehasn't moved yet, then return to the full budget as soon asMemAvailablecrosses the line. Idle staging arenas are also freed every pressed frame and made again at the next load. Entering at the reserve and leaving at the reserve plus a margin would avoid the flapping./proc/meminfois read every present intexture_frame(), and again every 16 submissions insideimports_give_back()withpools_muheld. One reading per N frames that both of them use would be cheaper, and would keep file I/O out from underpools_mu.RX_VK_PRECOMPILE_GRAPHICS=none, which exists only in #225, not inmainor here. Please say in the PR that this depends on it, or word those comments so they hold without it.Checked and fine: Windows behaves as before.
read_meminforeturns false off Linux,imports_give_backstays off by default there, and the drop is#ifdef __linux__. With ampleMemAvailablethe pressure checks do nothing.code_atpoints at the right SPIR-V bytes for uncompressed records. The capture spares, idle staging arenas and idle imports that get freed are all ones the GPU has finished with, the same as before.Context: #91, #54.
@ -2249,6 +2254,7 @@ PipeEntry* graphics_pipeline(const Program& p, const NvnState& s, const Targets&job->predicted = predicted;jobs_push([job] {// (over a linked one, which is left alone: draws recorded with it may still be on their way)program_cache_complete_graphics(*job->p);The return value is ignored here and at lines 2224 and 2233. When the SPIR-V read or
make_modulefails,build_graphics()runs withVK_NULL_HANDLEstage modules, which is invalid usage and can crash the driver. On false, leave the entry not ready and the program failed, asprogram_cache_complete()does for dispatches, instead of building.@ -9742,0 +9755,4 @@// programs without a built graphics pipeline (!graphics_built, as under RX_VK_PRECOMPILE_GRAPHICS=none) keep their vertex// and fragment modules so in-play pipeline builds do not read the disk. pcache is kept so in-play builds still hit the// start's pipeline cache and RADV's disk cache without copying every pipeline into play_cache. RX_VK_KEEP_MODULES=1 keeps allif (!env_on("LONGSHOT_PREPARE_ONLY") && !prepare_left_workers()) {This runs on every Linux machine, whatever its RAM, so a 64 GB desktop also pays a disk read and a module creation for each new pipeline variant in play. Please gate it (total RAM, the Deck profile, or pressure) or measure in-play hitches on a desktop. As it stands it doesn't match the title ("under Linux memory pressure").
@ -841,0 +849,4 @@for (const Stage& st : prog.stages) {if (prog.fallback && (st.vk_stage & (VK_SHADER_STAGE_COMPUTE_BIT | VK_SHADER_STAGE_TASK_BIT_EXT | VK_SHADER_STAGE_MESH_BIT_EXT)))continue;if (!st.module && st.code_words) need = true;st.moduleis read here withoutlight_mu, while another worker building a different variant of the same program may be writing it under the lock below. That is a data race, harmless on x86 and ARM in practice but UB. Take the lock for the check, or make the check cheap some other way (an atomic flag on the Program that the drop sets).@ -841,0 +866,4 @@ok = fread(spirv.data(), 4, spirv.size(), disk_in) == spirv.size();}if (ok) st.module = make_module(spirv);if (!st.module) return false;This returns false without marking
p.failed, so the next draw tries again. Under memory pressure that is a disk read and a module creation on every attempt. Also, nothing checks that the bytes atcode_atare still this program's record:program_cache_readchecks the record head against the key. A cheap check of the head atcode_atwould turn a stale offset into a clean failure instead of a module made from wrong bytes.@ -4156,2 +4165,2 @@if (frame_no % 1800 == 0 && !busy.exchange(true))jobs_push([] {const bool log_malloc = frame_no % 600 == 0 && frame_no % 1800 == 0;const bool check_malloc = log_malloc || (linux_pressed && frame_no % 120 == 0);Under sustained pressure this calls
malloc_trim(0)every 120 presents once 64 MB is free. glibc locks each arena while it trims (your log shows 47 ms), so allocating threads can stall every 2 s. Worth a minimum interval or a check that the last trim actually gave memory back. Nit:frame_no % 600 == 0 && frame_no % 1800 == 0on line 4165 is just% 1800.@ -4383,0 +4404,4 @@{uint64_t avail_bytes = 0, swap_free_bytes = 0;const uint64_t avail_res = linux_avail_reserve(), commit_res = commit_reserve();if ((avail_res || commit_res) && linux_pressure::read_meminfo(&avail_bytes, &swap_free_bytes)) {This opens and parses
/proc/meminfoon every present, andimports_give_back()reads it again withpools_muheld. Read it once every ~30 frames here and keep the result in an atomic thatimports_give_back()reads, so there is no file I/O underpools_mu.@ -4383,0 +4408,4 @@const uint64_t over = linux_pressure::shortfall(avail_bytes, swap_free_bytes, avail_res, commit_res);linux_pressed = over != 0;if (over) {budget = std::min(budget, total_bytes > over ? total_bytes - over : 0);There is no hysteresis and no floor here. The shortfall comes off the current total every frame, and memory only goes back to the OS when compaction frees a whole block, so this can keep cutting while
MemAvailablehasn't caught up yet. It then goes straight back to the full budget onceMemAvailablecrosses the reserve. Suggest leaving pressure only at the reserve plus a margin (say 256 MB) and holding the budget taken at the start of pressure, rather than recomputing it fromtotal_byteseach frame.5e2d92f8b61e709d9b2dThanks for the review! Addressed every point in
6f6c631c(rebased ontomain @ 9f64464b):Failed
program_cache_complete_graphics()remake ingraphics_pipeline()(gpu/vk/vk_draw.cpp,gpu/vk/vk_progcache.cpp):program_cache_complete_graphics()now setsp.failed = trueand logs on failure (matchingprogram_cache_complete()), so subsequent draws skip the failed program immediately instead of retrying disk reads andvkCreateShaderModuleon every draw.graphics_pipeline()(sync,gpl_wanted, and the worker lambda) now check the return value ofprogram_cache_complete_graphics()and, onfalse, marke->ready.store(true)withe->pipeline == VK_NULL_HANDLEand return without callingbuild_graphics()orgpl_link().Program::record_atandProgram::record_sizeand verified the 16-byte record header (head[0] == p.key && head[1] == p.record_size) atrecord_at - head_sizebefore readingst.code_at, turning any stale offset into a clean failure.tests/linux_pressure_test.cpp(bothmake_modulefailure and a stalerecord_atkey mismatch) andtests/test_linux_pressure.py.Lock-free fast path in
program_cache_complete_graphics()without data race onst.module(gpu/vk/vk_shader.h,gpu/vk/vk_progcache.cpp):std::atomic<bool> modules_dropped{false}onProgram, set withmemory_order_releasebyprogram_cache_drop_modules()and cleared withmemory_order_releaseonceprogram_cache_complete_graphics()finishes underlight_mu.program_cache_complete_graphics()now checksprog.modules_dropped.load(std::memory_order_acquire)before takinglight_muinstead of readingst.moduleoutside the lock.Gate post-precompile
VkShaderModuledrop so 32-64 GB Linux desktops are untouched (gpu/vk/linux_pressure.h,gpu/vk/vk_draw.cpp):linux_pressure::should_drop_modules(total_ram_bytes, current_shortfall, keep_env, drop_env):pipeline_cache_play()now drops modules by default only when physical RAM (MemTotalfrom/proc/meminfo) is<= 20 GB(kModuleDropMaxRam, covering 16 GB handhelds and PCs) or whenshortfall > 0at start-up. 32-64 GB Linux desktops keep their modules by default (RX_VK_DROP_MODULES=1forces the drop on any RAM size;RX_VK_KEEP_MODULES=1orRX_VK_DROP_MODULES=0keeps them everywhere).malloc_trim(0)call inpipeline_cache_play()onif (dropped_modules).Hysteresis and floor for the texture budget hold, and staging-arena flapping (
gpu/vk/linux_pressure.h,gpu/vk/vk_texture.cpp):linux_pressure::PressureHold: on entering pressure, capturesbaseline_total = total_bytesand holdsheld_budget = max(512 MB floor, baseline_total - over)rather than subtractingoverfrom the already-shrinkingtotal_byteson every frame while VMA compaction is still emptying blocks. IfMemAvailabledrops deeper while active (over > last_over),held_budgetsteps down frombaseline_total; it leaves pressure only once bothMemAvailableandMemAvailable + SwapFreeclear their reserves bykClearMargin(256 MB).linux_entered_pressure) and after 5 s of inactivity while under pressure (wait_s = 5), rather than on every pressed frame.Shared
/proc/meminfosampling every 30 frames, outsidepools_mu(gpu/vk/vk_texture.cpp,gpu/vk/vk_memory.cpp,gpu/vk/vk_draw.h):texture_frame()now reads/proc/meminfoonce every 30 presents and stores the shortfall instd::atomic<uint64_t> linux_shortfall_bytes(linux_pressure_shortfall()).imports_give_back()readslinux_pressure_shortfall()directly, performing zero file I/O underpools_mu.malloc_trim(0)frequency and effectiveness check (gpu/vk/vk_texture.cpp):log_malloctoframe_no % 1800 == 0.malloc_trimat most every600presents (~10-40 s instead of 120 presents), and skips the call when the previous trim freed< 16 MBof private memory unless free arena bytes (fordblks) have grown by at least64 MBsince that trim.Comments and PR description links (
gpu/vk/vk_progcache.cpp,gpu/vk/vk_shader.h,docs/development.md, PR description):docs/development.mdso they stand on their own onmainwithout assumingRX_VK_PRECOMPILE_GRAPHICS=nonefrom #225, and updated the PR description to use full URLs for #91, #54, #158, #225, https://github.com/NotRexed/Hookshot/issues/993, and https://github.com/NotRexed/Hookshot/issues/1000.1e709d9b2dabe9e348adabe9e348ad05f4c4aafbSteam Deck:
05f4c4acompletes where main is killed for running out of memory.Setup: Steam Deck OLED (16 GB, 8 GB swap file), SteamOS 3.9.5, kernel 7.2.7-valve2, RADV Mesa 26.2.0-devel. Programs load at the start: the GPU draws mesh programs through the compute fallback. Each build in its own folder and caches, the user's options (Quality preset Auto (this PC)), stock clocks, Gaming Mode 1280x720. The same moving Kokiri Forest route each time.
05f4c4aon main7315957) and main0877184, both without the Steam Deck profile (#225), so both keep every graphics pipeline ready. The two mains differ by a few commits. Neither touches memory.08771840877184087718405f4c4a05f4c4aThe warm start keeps far less:
vk: programs: 41044 shader modules let go after the precompile, 47797 program keys kept, start-up 8.2 s. Anon falls from main's 8.9 GB to 2.95 GB.vk: memory pressure on Linux:lines:Not covered:
0877184, not7315957.05f4c4aafb6931f7155b6931f7155bebaad175e1View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.