feat(scripts): memtour.py on Windows: the game's commit split into the guest window, VRAM blocks and the rest #992

Open
d4nkn3ss wants to merge 1 commit from d4nkn3ss/Longshot:feat/memtour-windows into main
Contributor

Type: feat · Risk: low · Tested on: Windows 11, RTX 3080 · Needs: none

Why

The most reported problem (worklist row 5.6) is memory that fills up over play, ending in "Out of memory (RAM plus the page file)". Most of those reports come from Windows players, and scripts/memtour.py (#570), the one-command memory tour, read /proc and stopped on Windows ("Linux only"). The crash reports in #894, #881, #861 and #899 give 20–30 GB of "private" memory but no breakdown of it. On Windows it splits into three parts that need very different fixes:

  • the guest window's committed pages;
  • the VRAM blocks: on NVIDIA every device-local allocation is charged to the process's commit, though it uses no RAM;
  • the host's and the driver's own heaps.

See #388 (the same question on Linux) and #939.

What changed

On Windows, memtour now samples the game's commit charge and splits it into those three parts, alongside the working set, the system's commit and its limit, and the RAM available. The summary reports the same for each area. Linux is unchanged. The new --image and --romfs options run it on a development build (work/sa/build), which has no image beside it.

  • scripts: scripts/memtour.py:
    • split_regions(): the split, a pure function over VirtualQueryEx's regions. The guest window is the allocation at A64_HOST_BASE; write-combined is committed private memory mapped PAGE_WRITECOMBINE.
    • win_api() / win_sample(): the ctypes calls (OpenProcess with PROCESS_QUERY_INFORMATION only, VirtualQueryEx, GetProcessMemoryInfo, GetPerformanceInfo). They read the address space's layout, never its contents.
    • Sampler takes a row from either OS and stops once the process is gone. summarize() hands Windows rows to summarize_windows().
    • main() runs on Windows.
  • tests: tests/test_memtour.py:
    • the split over made-up regions (a reservation, an image and a mapped file left out);
    • a Windows row, and no row once the process is gone;
    • the sampler writing Windows rows and stopping;
    • the Windows summary;
    • a live sample of the test's own process on Windows.
      The Linux sampler check now asks for the Linux sampler by name.
  • docs: docs/development.md's memtour row says what it samples on each OS.

Why write-combined stands for the VRAM blocks: a probe of 32 unmapped 32 MB vkAllocateMemorys per memory type on the RTX 3080 (NVIDIA 621.14, Windows 11) raised PrivateUsage by 1,026 MB for every device-local type. Each showed up as a committed private write-combined region, with the working set unchanged, and freeing gave it back. Over a 43-minute tour that part tracks the log's vk: memory heaps: heap 0 (device) blocks (7.0–7.5 GB of blocks, 7.3–7.8 GB write-combined). gpu/vk/vk_texture.cpp's commit reserve (#346) already assumes VRAM blocks are charged; this measures how much. On AMD and Intel the column may mean something else, which is why it is named for what it is, not "VRAM".

Measurements

Verdict: nothing to measure: a script and its checks. What the new memtour showed on main is in a comment below: no growth on main over a second lap, and about 20 GB of commit, ~8 GB of it VRAM blocks.

Flags and settings

None

Testing

Tested:

  • python -I -B tests/test_memtour.py: 21 pass on Windows 11.
  • python -I -B tests/run_all.py from PowerShell with llvm-mingw from the Longshot Builder: 584 passed, 26 failed, 42 skipped on this PR @ 079131a9. main @ fd2e3e98 gives the same counts with the same 26 failures: test_appimage_player_build, test_auto_handheld, test_cpp_preflight, test_crash_dialog, test_crt_options, test_env_on, test_fsr_uniform_ownership, test_guest_imported, test_import_reach, test_lift_cleanup_error, test_linux_preflight, test_make_game_cache, test_make_game_prepare, test_missing_nup, test_options_budget_once, test_pads_after_menu, test_parallel_init_order, test_pending_guest_import, test_readback_late, test_release, test_replay_write_order, test_run_all_changed, test_tested_configurations, test_vk_occlusion_queries, test_vk_pgo and test_walk_key. Some have open PRs, among them #804, #806, #144, #954, #937 and #625.
  • memtour run end to end on main's build, above.

Not tested: Linux. Its sampler is unchanged apart from being chosen by name, and its checks pass here on the made-up /proc. AMD and Intel on Windows, where the write-combined column may not be the VRAM blocks.

Must be tested:

  • Linux, any GPU: python -I scripts/memtour.py <game folder> --save <save> --dests 1,2 --dwell 30, expecting mem.csv with the Linux columns and the summary as before (by hand: summary.md).

Risks

What could break: memtour on Linux, if the sampler's choice went wrong. The game is untouched.

Turning it off: revert the commit.

Compatibility: Nothing changes for players. On Linux, memtour's files are as before.

Issues

See #388, #939

Checklist

Everyone

  • One topic.
  • The title is a Conventional Commit (type(scope): description) of the status line's type.
  • Why gives the reason plainly and links the issues that fit (the tracker searched).
  • Measurements has a verdict, or one line saying why there is nothing to measure.
  • Testing names what was tested, what wasn't, and what must still be tested, where and how.
  • Rebased on current main.
  • python -I -B tests/run_all.py passes (any failure that also fails on main is named).
  • No game files, dumps, build output or generated files.
  • third_party/ untouched (a new folder with a PROVENANCE.md instead).
  • The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

When it applies

  • AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
  • A check under tests/ that reads changed source updated in this PR, not weakened.
  • Docs updated.

🤖 Generated with Claude Code

**Type:** feat · **Risk:** low · **Tested on:** Windows 11, RTX 3080 · **Needs:** none ## Why The most reported problem (worklist row 5.6) is memory that fills up over play, ending in "Out of memory (RAM plus the page file)". Most of those reports come from Windows players, and `scripts/memtour.py` (#570), the one-command memory tour, read `/proc` and stopped on Windows ("Linux only"). The crash reports in #894, #881, #861 and #899 give 20–30 GB of "private" memory but no breakdown of it. On Windows it splits into three parts that need very different fixes: - the guest window's committed pages; - the VRAM blocks: on NVIDIA every device-local allocation is charged to the process's commit, though it uses no RAM; - the host's and the driver's own heaps. See #388 (the same question on Linux) and #939. ## What changed On Windows, memtour now samples the game's commit charge and splits it into those three parts, alongside the working set, the system's commit and its limit, and the RAM available. The summary reports the same for each area. Linux is unchanged. The new `--image` and `--romfs` options run it on a development build (`work/sa/build`), which has no image beside it. - scripts: `scripts/memtour.py`: - `split_regions()`: the split, a pure function over VirtualQueryEx's regions. The guest window is the allocation at `A64_HOST_BASE`; write-combined is committed private memory mapped `PAGE_WRITECOMBINE`. - `win_api()` / `win_sample()`: the ctypes calls (`OpenProcess` with `PROCESS_QUERY_INFORMATION` only, `VirtualQueryEx`, `GetProcessMemoryInfo`, `GetPerformanceInfo`). They read the address space's layout, never its contents. - `Sampler` takes a row from either OS and stops once the process is gone. `summarize()` hands Windows rows to `summarize_windows()`. - `main()` runs on Windows. - tests: `tests/test_memtour.py`: - the split over made-up regions (a reservation, an image and a mapped file left out); - a Windows row, and no row once the process is gone; - the sampler writing Windows rows and stopping; - the Windows summary; - a live sample of the test's own process on Windows. The Linux sampler check now asks for the Linux sampler by name. - docs: `docs/development.md`'s memtour row says what it samples on each OS. Why write-combined stands for the VRAM blocks: a probe of 32 unmapped 32 MB `vkAllocateMemory`s per memory type on the RTX 3080 (NVIDIA 621.14, Windows 11) raised PrivateUsage by 1,026 MB for every device-local type. Each showed up as a committed private write-combined region, with the working set unchanged, and freeing gave it back. Over a 43-minute tour that part tracks the log's `vk: memory heaps: heap 0 (device)` blocks (7.0–7.5 GB of blocks, 7.3–7.8 GB write-combined). `gpu/vk/vk_texture.cpp`'s commit reserve (#346) already assumes VRAM blocks are charged; this measures how much. On AMD and Intel the column may mean something else, which is why it is named for what it is, not "VRAM". ## Measurements **Verdict:** nothing to measure: a script and its checks. What the new memtour showed on main is in a comment below: no growth on main over a second lap, and about 20 GB of commit, ~8 GB of it VRAM blocks. ### Flags and settings None ## Testing **Tested:** - `python -I -B tests/test_memtour.py`: 21 pass on Windows 11. - `python -I -B tests/run_all.py` from PowerShell with llvm-mingw from the Longshot Builder: 584 passed, 26 failed, 42 skipped on this PR @ 079131a9. main @ fd2e3e98 gives the same counts with the same 26 failures: `test_appimage_player_build`, `test_auto_handheld`, `test_cpp_preflight`, `test_crash_dialog`, `test_crt_options`, `test_env_on`, `test_fsr_uniform_ownership`, `test_guest_imported`, `test_import_reach`, `test_lift_cleanup_error`, `test_linux_preflight`, `test_make_game_cache`, `test_make_game_prepare`, `test_missing_nup`, `test_options_budget_once`, `test_pads_after_menu`, `test_parallel_init_order`, `test_pending_guest_import`, `test_readback_late`, `test_release`, `test_replay_write_order`, `test_run_all_changed`, `test_tested_configurations`, `test_vk_occlusion_queries`, `test_vk_pgo` and `test_walk_key`. Some have open PRs, among them #804, #806, #144, #954, #937 and #625. - memtour run end to end on main's build, above. **Not tested:** Linux. Its sampler is unchanged apart from being chosen by name, and its checks pass here on the made-up /proc. AMD and Intel on Windows, where the write-combined column may not be the VRAM blocks. **Must be tested:** - Linux, any GPU: `python -I scripts/memtour.py <game folder> --save <save> --dests 1,2 --dwell 30`, expecting `mem.csv` with the Linux columns and the summary as before (by hand: `summary.md`). ## Risks **What could break:** memtour on Linux, if the sampler's choice went wrong. The game is untouched. **Turning it off:** revert the commit. **Compatibility:** Nothing changes for players. On Linux, memtour's files are as before. ## Issues See #388, #939 ## Checklist ### Everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`) of the status line's type. - [x] Why gives the reason plainly and links the issues that fit (the tracker searched). - [x] Measurements has a verdict, or one line saying why there is nothing to measure. - [x] Testing names what was tested, what wasn't, and what must still be tested, where and how. - [x] Rebased on current `main`. - [x] `python -I -B tests/run_all.py` passes (any failure that also fails on `main` is named). - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### When it applies - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [x] A check under `tests/` that reads changed source updated in this PR, not weakened. - [x] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(scripts): memtour.py on Windows: the game's commit split into the guest window, VRAM blocks and the rest
Some checks failed
PR description / Description follows the template (pull_request_target) Has been cancelled
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
e66783ea2c
memtour.py read /proc only, so the Windows players who report "Out of memory
(RAM plus the page file)" could not run it. On Windows it now samples the
game's commit charge (PrivateUsage) and splits it with VirtualQueryEx by
where it is: the guest window's committed pages, memory mapped
write-combined (on NVIDIA every device-local allocation is charged to the
process this way, using no RAM) and the rest; with the working set, the
system's commit and its limit, and the RAM available. Its summary says the
same per area. --image and --romfs run it on a development build.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
Contributor

What memtour's split showed on main, for row 5.6. These numbers are observations of one run on each build, not a measurement of this PR.

Verdict: nothing to measure for the change itself: a script with its checks. What follows is what it showed on main.

Hardware: Windows 11 Pro 26300, RTX 3080 10 GB (NVIDIA 621.14), i9-12900K, 64 GB, system-managed page file
Scene: adult save; teleports to memtour's default destinations 13, 1, 2, 3, 0, 7, 9, 11, two laps of 150 s each with a walk; defaults, windowed
Method: the commit split sampled every 60 s from outside the game (the same split as split_regions()), one run on each build, lap 1's end against lap 2's
How to reproduce: python -I scripts/memtour.py work/sa/build --image work/sa/image/image.bin --romfs <dump>\romfs --save <save folder> --laps 2 --dwell 150 --env HOS_OOT_CHEATS=1

MB, at the end of each lap main @ a8a30c22, lap 1 main @ a8a30c22, lap 2 main @ dc390e59 (Nov 14), lap 1 main @ dc390e59, lap 2
private (commit charge) 20,355 20,035 19,390 19,781
... guest window 7,601 7,639 7,417 7,470
... write-combined (VRAM blocks) 8,124 7,819 7,278 7,174
... the rest 4,346 4,294 4,328 4,769
working set 11,230 10,802 10,929 11,295
vk: imports: live at the end 116 (1,852 MB) 127 (1,808 MB)

On main nothing grows in lap 2, and the rest holds at 4.3 GB. A commit of ~20 GB is the fixed size of a play session on a 10 GB card here, and ~8 GB of it is VRAM that uses no RAM. On dc390e59 the rest still grew by ~440 MB a lap. My Nov 12 tour log had imports at 430 live (5.4 GB) and private memory at 27 GB, so the import give-back merged since then shows.

The end-to-end run of this PR's memtour (--dests 1,2 --dwell 30) on main @ a8a30c22:

| peak private memory (the commit charge, against RAM plus the page file) | 20,519 |
| ... the guest window's, at the end | 7,391 |
| ... write-combined (on NVIDIA the VRAM blocks), at the end | 6,015 |
| ... the rest (the host's and the driver's heaps), at the end | 3,470 |
| peak system commit (its limit then) | 47,780 (69,408) |

Posted by an AI agent (Claude Code) for d4nkn3ss.

What memtour's split showed on main, for row 5.6. These numbers are observations of one run on each build, not a measurement of this PR. **Verdict:** nothing to measure for the change itself: a script with its checks. What follows is what it showed on main. Hardware: Windows 11 Pro 26300, RTX 3080 10 GB (NVIDIA 621.14), i9-12900K, 64 GB, system-managed page file Scene: adult save; teleports to memtour's default destinations 13, 1, 2, 3, 0, 7, 9, 11, two laps of 150 s each with a walk; defaults, windowed Method: the commit split sampled every 60 s from outside the game (the same split as `split_regions()`), one run on each build, lap 1's end against lap 2's How to reproduce: `python -I scripts/memtour.py work/sa/build --image work/sa/image/image.bin --romfs <dump>\romfs --save <save folder> --laps 2 --dwell 150 --env HOS_OOT_CHEATS=1` | MB, at the end of each lap | main @ a8a30c22, lap 1 | main @ a8a30c22, lap 2 | main @ dc390e59 (Nov 14), lap 1 | main @ dc390e59, lap 2 | |---|---|---|---|---| | private (commit charge) | 20,355 | 20,035 | 19,390 | 19,781 | | ... guest window | 7,601 | 7,639 | 7,417 | 7,470 | | ... write-combined (VRAM blocks) | 8,124 | 7,819 | 7,278 | 7,174 | | ... the rest | 4,346 | 4,294 | 4,328 | 4,769 | | working set | 11,230 | 10,802 | 10,929 | 11,295 | | `vk: imports:` live at the end | | 116 (1,852 MB) | | 127 (1,808 MB) | On main nothing grows in lap 2, and the rest holds at 4.3 GB. A commit of ~20 GB is the fixed size of a play session on a 10 GB card here, and ~8 GB of it is VRAM that uses no RAM. On dc390e59 the rest still grew by ~440 MB a lap. My Nov 12 tour log had imports at 430 live (5.4 GB) and private memory at 27 GB, so the import give-back merged since then shows. The end-to-end run of this PR's memtour (`--dests 1,2 --dwell 30`) on main @ a8a30c22: ``` | peak private memory (the commit charge, against RAM plus the page file) | 20,519 | | ... the guest window's, at the end | 7,391 | | ... write-combined (on NVIDIA the VRAM blocks), at the end | 6,015 | | ... the rest (the host's and the driver's heaps), at the end | 3,470 | | peak system commit (its limit then) | 47,780 (69,408) | ``` *Posted by an AI agent (Claude Code) for d4nkn3ss.*
d4nkn3ss force-pushed feat/memtour-windows from e66783ea2c
Some checks failed
PR description / Description follows the template (pull_request_target) Has been cancelled
CI / Conventional Commit title (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Has been cancelled
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Has been cancelled
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Has been cancelled
CI / Workflow lint (actionlint) (pull_request) Has been cancelled
Guide screenshots / Guide images and local UI source review (pull_request) Has been cancelled
Third-party parts / As their notes say (pull_request) Has been cancelled
Third-party parts / The same as upstream (pull_request) Has been cancelled
to 079131a913
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 16s
2026-11-16 01:11:32 +00:00
Compare
Author
Contributor

Rebased on main @ fd2e3e98 (now 079131a9). run_all.py gives the same 26 failures on both, and test_memtour.py passes (21). The description is updated.

Posted by an AI agent (Claude Code) for d4nkn3ss.

Rebased on main @ fd2e3e98 (now 079131a9). `run_all.py` gives the same 26 failures on both, and `test_memtour.py` passes (21). The description is updated. *Posted by an AI agent (Claude Code) for d4nkn3ss.*
d4nkn3ss changed title from WIP: feat(scripts): memtour.py on Windows: the game's commit split into the guest window, VRAM blocks and the rest to feat(scripts): memtour.py on Windows: the game's commit split into the guest window, VRAM blocks and the rest 2026-11-16 01:16:14 +00:00
Some checks are pending
CI / Conventional Commit title (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
CI / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
CI / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
CI / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Guide screenshots / Guide images and local UI source review (pull_request) Blocked by required conditions
Third-party parts / As their notes say (pull_request) Blocked by required conditions
Third-party parts / The same as upstream (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request) Blocked by required conditions
Website / Website on Cloudflare (pull_request_target) Has been skipped
PR description / Description follows the template (pull_request_target) Successful in 16s
This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.
Some workflows are waiting to be reviewed.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'feat/memtour-windows:d4nkn3ss-feat/memtour-windows'
git switch 'd4nkn3ss-feat/memtour-windows'
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
MagicBeanDev/Longshot!992
No description provided.