fix(host): the game's memory stays in RAM when the PC runs short: a soft working-set minimum (HOS_WS_FLOOR_MB) #2007
No reviewers
Labels
No labels
area/accessibility
area/audio
area/input
area/install
area/menu
area/renderer
area/saves
area/tooling
area/upscaling
beanstalk: approved
beanstalk: failed
beanstalk: needs approval
beanstalk: running
beanstalk: tested
ci: approved
good first issue
gpu/amd
gpu/igpu
gpu/intel
gpu/nvidia
help wanted
kind
bug
kind
chore
kind
docs
kind
feature
kind
performance
kind
question
needs measurements
platform/linux
platform/macos
platform/steam-deck
platform/windows
preview
priority
critical
priority
high
priority
low
regression
risk
high
risk
low
risk
medium
status
blocked
status
confirmed
status
duplicate
status
invalid
status
needs-info
status
wontfix
ci: approved
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
MagicBeanDev/Longshot!2007
Loading…
Reference in a new issue
No description provided.
Delete branch "GummiNaomi/Longshot:fix/ws-floor"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
On a Windows PC with other programs open, a player's game stuttered in play: frames of 35-150 ms, often several in a row,
with the renderer idle and nothing uploaded. In the frame trace the game's main thread was blocked for nearly the whole
frame (
game_msaboutmain_blocked_ms, its CPU time and its waits for a virtual core near 0). In that session Windowshad trimmed the game's working set to about 1.1 GB of its 20 GB of private memory: the log's
vk: memory: working setlines fall from 5.8 GB to 1.1 GB over the session while the private memory stays at 20 GB. Every touch of the game's
memory then faulted it back in. The main thread waits on the frame's job threads (the ModuleSystem pool,
f_8007fe60)and on the submit's memory capture, and those were the threads faulting: 24 such frames a thousand while under 2 GB was
resident, 3.3-3.6 a thousand while more was.
Windows keeps at least a process's working-set minimum when it trims, and the game's was Windows' default, a few hundred
KB, so under memory pressure the largest process, the game, was trimmed first and furthest. The host now asks for a
minimum at start: a quarter of the RAM, at most 4096 MB (
host/ws_floor.h, called fromhost/hos_main.cbesideno_power_throttling). It is a soft minimum (QUOTA_LIMITS_HARDWS_MIN_DISABLE): Windows still trims below it when memoryis critically short, so it cannot starve the PC; until then it trims other processes and the game's pages above the
minimum first. Raising it needs
SeIncreaseWorkingSetPrivilege, which Windows gives every account, disabled; it is enabledon the process token for the call. The maximum stays soft. The log says
mem: the working set's minimum is 4096 MB of the PC's 31620 MB .... Linux (and the Steam Deck): nothing changes.The switch:
HOS_WS_FLOOR_MB(kind hw, its row ingpu/vk/switches.def): unset, the rule above;<MB>another minimum(at most half the RAM);
0(or off, no, false) Windows' own minimum, as on main.How to see it on main: a Windows PC short of RAM (a browser, Discord and another program or two open beside the game on
32 GB), then walk and turn in Castle Town's market: runs of 35-150 ms frames with no uploads, and the log's working set
falling to 1-2 GB. Holding the game's working set to 1100 MB with any tool that sets a process's maximum working set shows
the same frames on a PC with free memory.
Read first:
host/ws_floor.h.Testing
Tested: Windows 11, Ryzen 9 9950X3D (SMT off), 32 GB, RTX 4070 SUPER (NVIDIA 617.42), headless, uncapped, a walk
route: a teleport, then about 100 s walking and turning in the market and about 200 s in Kokiri Forest. With the game's
working set capped at 1100 MB (a check of the cause, not of this change) the player's stalls appear; with a helper
process holding the PC's free RAM less 2 GB and touching it every 2 s, the minimum kept the game at 4.0-4.1 GB resident
in both runs, against 0.95-2.8 GB without it, and the log's line says the minimum was set. With plenty of RAM nothing
changed.
tests/test_ws_floor.py(the rule; on Windows the call in a process of its own, and the minimum and soft flagsWindows reports back) fails on main and passes here, on Windows and on Linux (WSL, gcc: the rule only).
tests/test_switch_table.pyandtests/test_doc_anchors.pypass.tests/run_all.py: the whole run on Windows (8 at once): 279 passed, 71 failed, 556 skipped. 70 of the failures fail on main @f1f2dc5bon this PC too (51 of them because this PC has no C compiler on PATH:FileNotFoundError), listed below; the 71st,tests/test_longshot_ctl.py, failed once on a Windows file lock while removing its temp folder and passed three reruns alone.The 70 checks that fail on main here too
tests/test_alt_enter.py,tests/test_appimage_player_build.py,tests/test_audio_deadline.py,tests/test_audio_output_errors.py,tests/test_audio_wave_buffer_release.py,tests/test_bc_levels.py,tests/test_bind_memo.py,tests/test_capture_copy_helpers.py,tests/test_cheat_vm.py,tests/test_collect_wide.py,tests/test_dev_ab.py,tests/test_dev_build.py,tests/test_dev_migrate.py,tests/test_dev_packs.py,tests/test_dlss_actor.py,tests/test_dlss_checks.py,tests/test_dlss_math.py,tests/test_env_file.py,tests/test_failed_compile_keeps_objects.py,tests/test_flush_nothing_held.py,tests/test_fps_overlay_order.py,tests/test_fps_rate.py,tests/test_fps_stats.py,tests/test_framegen_auto.py,tests/test_framegen_gate.py,tests/test_fsr3_math.py,tests/test_gpu_choice.py,tests/test_hd_rumble.py,tests/test_hle_guest_ranges.py,tests/test_host_guest_writes.py,tests/test_import_span.py,tests/test_jobs_wait_runs_queued.py,tests/test_libc_hooks.py,tests/test_lift_cache.py,tests/test_log_redact.py,tests/test_make_game_ngx.py,tests/test_menu_mouse_scroll.py,tests/test_mirror_inherit.py,tests/test_mise_build.py,tests/test_missing_nup.py,tests/test_movie_upscale_policy.py,tests/test_ngx_sdk.py,tests/test_nvn_texture_handle.py,tests/test_oot_jitter.py,tests/test_pool_span.py,tests/test_present_ahead.py,tests/test_prof_clock.py,tests/test_profiling_mode.py,tests/test_program_cache_cleanup.py,tests/test_prompt_auto_face.py,tests/test_quick_pipelines.py,tests/test_renhdrum.py,tests/test_report_words.py,tests/test_rt_pool_minimum.py,tests/test_rumble_mapping.py,tests/test_rumble_menu.py,tests/test_rumble_refresh.py,tests/test_same_bind.py,tests/test_seed_reader.py,tests/test_skip_code.py,tests/test_slow_frames.py,tests/test_sparse_bc_usage.py,tests/test_status_line.py,tests/test_submitted_page.py,tests/test_tex_spans_once_a_present.py,tests/test_texture_layout_sizes.py,tests/test_ub_gap.py,tests/test_upscale_independent.py,tests/test_upscale_label.py,tests/test_wb_pages.pyStill to test:
HOS_WS_FLOOR_MB=0, expecting noruns of game-side stalls with it on (by hand: the two minimum-on pressure runs here lost their market stretch to a
teleport the game refused, so that comparison is not made yet)
more and the other programs to stay usable (by hand)
Measurements
Verdict: the cause is shown (the trimmed working set), and the minimum holds the game's memory in RAM under real
pressure: 4.0-4.1 GB resident against 0.95-2.8 GB. That this removes the market's stalls under pressure follows from
the capped runs (at 4 GB nothing was added) but is not measured directly yet: both minimum-on pressure runs missed the
market. No cost seen with plenty of RAM (one run with the minimum on).
Setup: as Tested. The off side is this PR's build with
HOS_WS_FLOOR_MB=0, which makes no call: main's behaviour. Builtand measured at 5cb3b97c on main
00e0dc93; rebased ontof1f2dc5bsince as this head, with the code unchanged (only thetest now finds its C compiler on its own). Hitches: frames of 33 ms or more; game-side: those with no uploads and the
main thread blocked for at least half the frame; per area, the first 300 frames after each arrival left out. Pressure:
off, on, off, on in turn, 21 Nov 03:32-04:00.
00e0dc93(HOS_WS_FLOOR_MB=0)What chose the 4096 MB default, the game's working set held to a maximum on main @
00e0dc93(the same route, one run each,plenty of RAM): no maximum 21 and 23 hitches (6 and 6 game-side); 4000 MB 17 (5); 2000 MB 25 (11); 1100 MB, the player's,
91 (47, 3.2 s). The stalls start below about 2 GB; 4 GB is past any effect.
The player's session (main @
3df5c13e, their own play, ~6 minutes, the frame trace and the log): game-side frames 3.6 athousand while 4 GB or more was resident, 3.3 at 2-4 GB, 24.0 under 2 GB (10 of the log's 600-present windows, 144 frames).
Risks
A game that keeps 4 GB resident leaves other programs less when the PC is short of RAM: they page instead of the game
(by design; the soft minimum gives way when memory is critically short, so nothing is starved). On an 8 GB PC the minimum
is 2 GB. If the privilege cannot be enabled or the call is refused, the log says so and Windows' own minimum stays, as on
main.
HOS_WS_FLOOR_MB=0turns it off;HOS_WS_FLOOR_MB=<MB>sets another value. After updating: nothing changes forsaves, options or the caches.
Checklist
Ready for review
main; the title is a Conventional Commit.mise run testpasses (any failure that also fails onmainis named).main, or Measurements says why there is nothing to measure.third_party/untouched; nothing GPL-incompatible.When it applies
mainand passes here.gpu/vk/switches.def.tests/that reads changed source is updated, not weakened. (None needed a change: the checks thatread these files pass or fail as on
main,tests/test_switch_table.pyamong the passes.)and run. Linux: not built here; someone else builds Linux. The Linux side of the change is only the header's rule,
compiled by the test with gcc.)
gpu/vk/switches.def, the switches' reference.)Assisted-by:/Co-Authored-By:trailers.Ready to merge
main, measured again or a comment says why the numbers still hold.🤖 Generated with Claude Code
Possibly related, for context:
Recommendation: Request changes. The startup call can remove an existing Windows working-set cap and leaves a process privilege enabled after the call. The reported runs show higher game residency under pressure, but they have yet to show that the default fixes the market stalls or keeps competing apps responsive.
Longshot asks Windows to keep more of the game resident so memory pressure causes fewer stalls. The host startup placement keeps the policy out of the frame loop and the renderer, but the implementation also changes the maximum working set and can shift several GiB of memory pressure to the rest of the PC.
@fixwill do (13)host/ws_floor.h:62: The setter clears both hard-limit flags and can raise the maximum. (more on the line)host/ws_floor.h:54: Closing the token handle leaves SeIncreaseWorkingSetPrivilege enabled on the process token, including when the working-set setter subsequently fails. (more on the line)host/ws_floor.h:59: A failed working-set query falls through with hi and flags still zero, so the setter can replace an unknown existing policy with the synthesized maximum and report success. (more on the line)host/ws_floor.h:27: The parser takes any leading decimal prefix. (more on the line)gpu/vk/switches.def:239: The switch row and startup log read as a promise of residency, and the header and Risks section infer that a soft minimum protects the rest of the PC from starvation. (more on the line)tests/test_ws_floor.py:123: The startup check can pass ifws_floor();is commented out or put in an unreachable branch. (more on the line)tests/test_ws_floor.py:132: Compiler selection ignores CC, and a missing compiler prints SKIP followed by PASS and exits zero. (more on the line)tests/test_ws_floor.py:69: The live fixture requires Windows to grant the full machine-dependent request. (more on the line)host/ws_floor.h:25: Choose the smallest floor that measurably fixes the stalls and add a lower-floor or release policy for setups where it costs too much shared headroom.00e0dc93, while Setup says that the pressure control ran the PR binary with the switch off. Label that control with the binary's actual commit and separate it from the capped runs on main. Include the supporting log excerpts and medians, and update the checklist to reflect the reported suite failure and pending Linux host build.PR description / Description follows the template (pull_request_target)fails on this head and passes on the base branch: Failing after 10s.Guide screenshots / Guide images and local UI source review (pull_request)fails on this head and passes on the base branch: Failing after 14s.CI / Website build and checks (pull_request)fails on this head and passes on the base branch: Failing after 19s.For the author (2) (a run, a measurement or a decision)
host/hos_main.c:2371: The target market stall was never measured with the floor enabled. (more on the line)host/ws_floor.h:25: The pressure test measures only what Longshot keeps. (more on the line)More: does it help, other systems, switches, what was read
Does it help: Unproven for stutter. The PR reports 4.0–4.1 GB resident with the policy versus 0.95–2.8 GB with it off in two pressure runs per side. Both enabled runs missed the market, where the disabled runs recorded 3.2 and 37.4 game-side hitches per 10,000 frames. Forest results overlapped at 0.4 and 4.8 off versus 3.6 and 3.9 on. The 1100 MB cap increased hitches from 21–23 to 91 on the older base and shows the cost of a small working set. The pressure A/B used the older PR build with a switch-off control. A matched market comparison on the current base and head is still missing.
Other systems: The Windows policy covers the whole process, including guest memory, renderer host allocations, and audio and input buffers. It can also affect competing apps and shared-memory GPU workloads through physical-memory pressure. The reported run covers one Windows 11 NVIDIA desktop. Windows 10, AMD, low-RAM and shared-memory setups, and foreground and background responsiveness remain unverified. The Linux and Steam Deck runtime call is excluded at compile time. Shader preparation and crash-helper launches return before it, and the diff leaves saves, caches, and the renderer API alone.
Switches: HOS_WS_FLOOR_MB is a Windows startup-only hardware switch. Unset requests one quarter of usable physical RAM, capped at 4096 MiB. A numeric override is capped at half of RAM, and 0 or lowercase off, no, or false requests no policy change. Its switch-table row is present. The parser needs to reject malformed numbers, and the default needs player-facing and competing-app measurements before it can be judged suitable.
Reviewed:
d7591451abagainstf1f2dc5bd2: Read AGENTS.md, CONTRIBUTING.md, the Forgejo template, the developer standards and switch reference, the relevant fix and measurement guidance, the single commit, and the complete diff. Read host/ws_floor.h and tests/test_ws_floor.py in full. Read the changed regions of host/hos_main.c and gpu/vk/switches.def and followed startup, environment loading, token use, linking, test discovery, and the working-set log producer. Skimmed unrelated regions of the two existing files and supporting workflows. Reviewed the supplied PR text and check status. Raw runs and the measured PR commit were unavailable in this checkout. No local review checklist, changed translations, or changed generated files.; nothing run (a code review).Checked and fine:
Testing: Read-only review. The author reports the targeted Windows and WSL fixture passes plus switch-table and doc-anchor passes. The Windows suite reported 279 passes, 71 failures, and 556 skips. Seventy failures were reported on the base too, and the remaining failure passed three isolated reruns. A full Linux host build and the pending in-game comparisons have yet to be reported. Git's whitespace check was clean.
Overlaps and merge order: No overlapping working-set change found in the available checkout history. Live Forgejo pull requests were outside the checkout.
Automated code review by BeanBot (gpt-6.2-sol), on the pull request's opening. Nothing was run; a maintainer decides.
@ -236,6 +236,7 @@ SW(HOS_WINDOW_NOACTIVATE, diag, bool, "off: the window opens in front", "=1 openSW(HOS_WINDOW_SIZE, tune, string, "1280x720", "Sets the window's client size as WxH (each at least 64).")SW(HOS_WINDOW_TITLE, tune, string, "'Longshot'", "The window's title on Linux (SDL).")SW(HOS_WM_CLASS, tune, string, "HOS_APP_ID", "The window's WM class / app id on Linux (SDL).")SW(HOS_WS_FLOOR_MB, hw, int, "Windows: a soft working-set minimum of a quarter of the RAM, at most 4096 MB: Windows keeps that much of the game in RAM before it trims below it (host/ws_floor.h)", "<MB>: that minimum instead (at most half the RAM); 0 (or off, no, false): Windows' own minimum, as before, so memory pressure can trim the game to a fraction of what its frames touch.")L5 (minor): The switch row and startup log read as a promise of residency, and the header and Risks section infer that a soft minimum protects the rest of the PC from starvation. Windows may trim below a soft target, and an accepted target says nothing about desktop responsiveness. Describe it as a request in the row, log, header, and PR text. For the 16 GB test, expect the logged request to follow the quarter-of-usable-RAM rule, which can yield less than 4096 MiB.
@ -2351,6 +2368,7 @@ static int game_main(int argc, char **argv) {#ifdef _WIN32timeBeginPeriod(1); /* 1 ms timer resolution: the 60 Hz vsync and the game's frame pacing sleep in short steps */no_power_throttling();ws_floor(); /* (the game's memory kept in RAM under memory pressure: host/ws_floor.h) */L9 (minor): The target market stall was never measured with the floor enabled. Run the same market route from a save in the area on the current base and head, with synchronized pressure, player settings and at least two interleaved runs per side. Report game-side hitches, p99 frame time and 1% lows. Add a repeated no-pressure comparison to check the default's cost in ordinary play.
@ -0,0 +22,4 @@/* The minimum in MB for a PC with ram_mb of RAM and the switch's value env (NULL or empty: not set); 0: none */static inline uint64_t ws_floor_mb(uint64_t ram_mb, const char *env) {uint64_t mb = ram_mb / 4 < 4096 ? ram_mb / 4 : 4096;L10: The pressure test measures only what Longshot keeps. An 8 or 16 GB Windows PC with active browser or Discord windows, especially one with a shared-memory GPU, can keep the game smooth by moving paging or GPU residency pressure to those other workloads. Measure app-switch latency, page-ins and available memory alongside game frame times on those setups, and cover the affected Windows Tier 1 hardware, including AMD.
@ -0,0 +24,4 @@static inline uint64_t ws_floor_mb(uint64_t ram_mb, const char *env) {uint64_t mb = ram_mb / 4 < 4096 ? ram_mb / 4 : 4096;if (env && *env) {if (*env >= '0' && *env <= '9') mb = strtoull(env, NULL, 10);L4: The parser takes any leading decimal prefix. On a 32 GiB PC,
16000MiBsilently requests 16000 MiB instead of the documented 4096 MiB fallback for an invalid value. Require a complete decimal value before accepting the override, handle out-of-range input explicitly, and reuse the host's case-insensitive off-value convention soOFFalso disables it.@ -0,0 +51,4 @@tp.PrivilegeCount = 1;tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;if (LookupPrivilegeValueA(NULL, "SeIncreaseWorkingSetPrivilege", &tp.Privileges[0].Luid))AdjustTokenPrivileges(tok, FALSE, &tp, 0, NULL, NULL);L2: Closing the token handle leaves SeIncreaseWorkingSetPrivilege enabled on the process token, including when the working-set setter subsequently fails. Other threads then run with the changed token for the rest of the session. Capture the previous privilege state, check for ERROR_NOT_ALL_ASSIGNED, and restore the state on both setter outcomes while preserving the setter's error.
@ -0,0 +56,4 @@}SIZE_T lo = 0, hi = 0;DWORD flags = 0;GetProcessWorkingSetSizeEx(GetCurrentProcess(), &lo, &hi, &flags);L3 (minor): A failed working-set query falls through with hi and flags still zero, so the setter can replace an unknown existing policy with the synthesized maximum and report success. Return the query error before changing the token or the limits.
@ -0,0 +59,4 @@GetProcessWorkingSetSizeEx(GetCurrentProcess(), &lo, &hi, &flags);const SIZE_T want = (SIZE_T)*mb << 20, room = (SIZE_T)256 << 20;const SIZE_T top = hi > want + room ? hi : want + room; /* (the maximum at least the minimum: the call's rule) */if (SetProcessWorkingSetSizeEx(GetCurrentProcess(), want, top, QUOTA_LIMITS_HARDWS_MIN_DISABLE | QUOTA_LIMITS_HARDWS_MAX_DISABLE))L1: The setter clears both hard-limit flags and can raise the maximum. If a launcher or memory tool sets a hard 1100 MiB cap before Longshot starts, the default can replace it with a soft maximum of at least 4352 MiB. Respect existing hard limits and preserve the maximum and its policy wherever the requested floor fits. If raising a soft maximum is necessary, make that change explicit and cover the existing-cap case in the fixture.
@ -0,0 +66,4 @@CHECK(hi >= lo);CHECK(fl & QUOTA_LIMITS_HARDWS_MIN_DISABLE);CHECK(fl & QUOTA_LIMITS_HARDWS_MAX_DISABLE);CHECK(ws_floor_apply(NULL, &mb, &ram, &err) == 1 && mb == ws_floor_mb(ram, NULL));L8 (minor): The live fixture requires Windows to grant the full machine-dependent request. A constrained runner or account whose policy removes the privilege can refuse it even when Longshot's fallback works. Test the default and refusal paths deterministically with Win32 stand-ins, and keep a live smoke case that reports the required Windows capability and checks the result when available.
@ -0,0 +120,4 @@if needle not in src:print("FAIL: hos_main.c: %s (%s)" % (what, needle))ok = Falsestart = src.find("no_power_throttling();\n", src.find("timeBeginPeriod(1);"))L6: The startup check can pass if
ws_floor();is commented out or put in an unreachable branch. The compiled fixture invokes the helper directly, so it misses that regression. Exercise the actual host startup wiring with stand-ins and an explicitly controlled switch value, and assert the Windows call and opt-out by behavior. Put that check before the fix in the branch history so the base fails for missing behavior instead of a missing header. Check the getter results and compare the minimum, maximum, and flags in the off case.@ -0,0 +129,4 @@def main():ap = argparse.ArgumentParser()ap.add_argument("--cc", default="")L7: Compiler selection ignores CC, and a missing compiler prints SKIP followed by PASS and exits zero. For example, a checkout with a valid compiler available only through CC can show a green result under
run_all.py --strictwithout compiling anything. Honor CC after--cc, use the supported x64 compiler for the Windows fixture, and make missing-tool handling visible to the runner or fail the check.@fix
I didn't fix this pull request: it doesn't allow edits from maintainers (the author can turn on Allow edits from maintainers).
Automated fix by BeanBot (gpt-6.1-terra), asked for by HumanBean. Nothing was run; a maintainer decides.