WIP: feat(runtime): HOS_COUNTERS, the memory and cache counters as one JSON line every N seconds

dyst commented

2026-11-13 01:06:51 +00:00

First-time contributor

Why

To see whether memory leaks across a scene or a loop, today you collect lines from several places logged at different rates: the status line every 600 frames, the VMA heaps, memory_stats, and the process's memory, which only the bug report shows. This writes them all side by side, one JSON line every N seconds, with a small reader.

What changed

renderer: counters_tick in gpu/vk/vk_replay.cpp, called from status_tick: with HOS_COUNTERS=<seconds> it writes one JSON line per interval to HOS_COUNTERS_FILE, else longshot_counters.jsonl beside the game. The first line names the format and the build, {"counters":1,"version":"...","every_s":5}. Each line has:
- the runtime's counters: the guest heap and its use, the address regions, the process's private memory and working set;
- the video memory and the system heap the driver allows (VMA's budgets: vram_usage, system_heap_usage);
- the texture cache (count, MB), the mirrors and their MB, the bytes synced and written back so far (memory_stats);
- draws, pipelines (draw_stats) and the program cache's hits so far (st_cache_hits, vk_progcache.h).
Every counter is read where it is already kept, on the replay thread that owns the texture cache's. Unset, nothing is read or written.
interface: MkwVkHost.host_counters in gpu/vk/vk_api.h, the runtime's part of a line as JSON members; MKWVK_API_VERSION 21 → 22.
host: host_counters in host/rx_backend.c (the heap, its use, the regions, hos_process_memory's private memory and working set), last in vk_host; hos_mem_regions() in host/hos_mem.c gives the region count.
renderer: build_version() in gpu/vk/vk_menu.cpp (declared in vk_menu.h) gives the version for the first line; system_heap_usage is declared in vk_internal.h.
scripts: scripts/counters.py (new, standard library): a table; --diff [A B], line B against line A (by default the first and the last); --grew, the levels that went up at every line and never down, the leak check across a loop of the same scene. Counters "so far" (presents, draws, synced bytes...) are left out of --grew. A file of another format is refused.
tests: tests/test_counters.py (new).

Not included: the virtual-core counters, whose stats reset every 5 s when logged and need a getter that doesn't reset them; and HOS_MEM_MAP, which walks the whole address space and is too slow to sample.

Flags and settings

Flag	New or changed	What it does	Default now	Goal: on by default?	Helps on	Doesn't help on	Untested on	Untested, but should help on
`HOS_COUNTERS`	New	Every so many seconds, a JSON line of the memory and cache counters	unset (nothing read or written)	no (a developer switch)	-	-	Linux	-
`HOS_COUNTERS_FILE`	New	The file `HOS_COUNTERS` writes to	unset (`longshot_counters.jsonl` beside the game)	no (a developer switch)	-	-	Linux	-

Measurements

Run on a Longshot build of this branch (this PR was made on Longshot; there is no GitHub original). There is no main-side measurement: main has no HOS_COUNTERS, and the run checks the counters' file, not a speed. The two columns are two points of the same run (counters.py --diff), not two builds.

main: 82de567a (2026-11-12) (this branch's base; nothing was run on it)
this PR: feat/counters @ 03608e3a
Hardware: RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11
Scene: from the start through a save into Kokiri Forest
Method: HOS_COUNTERS=2, one launch; scripts/counters.py --diff from the start to Kokiri Forest, and --grew
How to reproduce: set HOS_COUNTERS=2, play from the start into Kokiri Forest from a save, then python -I scripts/counters.py --diff and --grew on longshot_counters.jsonl

`counters.py --diff`	this PR (Longshot build), the start	this PR (Longshot build), Kokiri Forest
textures	3	1,247
texture MB	31	3,205
mirrors	4	75
mirror MB	16	1,128
video memory, MB	2,270	6,931
the process's private memory	16.5 GB	25.1 GB

37 lines over 74 s, the first naming the format and the build, then one every 2 s, each valid JSON.
--grew: only the mirrors went up at every line, as the scene loaded in.
No errors in the log; Kokiri Forest at 32-37 fps. This branch's build recompiled the host and renderer in 45 s.

Testing

Tested: the runs under Measurements (on this branch) and the game-free checks this description names.

Not tested: Linux (the checks and the game run were on Windows only).

Needs other testers:

Linux: HOS_COUNTERS=2, expecting valid JSON lines with the process's memory read from /proc (request to open)

Screenshots

Nothing on screen.

Risk and rollback

The renderer interface changes (MkwVkHost.host_counters, version 22), so a renderer built before this change and a runtime built after it (or the other way round) no longer have the same version.
Unset, counters_tick returns after one cached getenv. Set, it writes and flushes a line on the replay thread every N seconds.
Leaving HOS_COUNTERS unset turns it off; reverting the commit is enough.

Reviewer notes

Made on Longshot directly (no GitHub original). The dev-loop proposal's E3 (NotRexed/Hookshot#2636).
Checks run (on Longshot, not the whole run_all.py): every check that reads the changed files (125), through run_all.py, Windows: 124 pass. test_readback_wait_bounded fails the same way on Longshot's main on this PC.
tests/test_counters.py: counters_tick is cut out and run against stub counters through a clock. Unset, it writes nothing and makes no file. Set to 5, its first line names the format, build and interval, then one line every 5 s (not one per present), each valid JSON with the runtime's members merged in. HOS_COUNTERS_FILE, and a runtime that gives no counters, also work. host_counters is cut out and run: its members, wrapped in braces, are JSON, and a buffer too small gives 0. counters.py: the table, the difference, what only grew, and a file of another format refused. In the source: status_tick calls it; MkwVkHost ends with the member, version 22; the runtime's table gives it last. Two deliberate breaks fail it: the "unset, nothing" line removed, and a line written at every present.

Issues

None.

Checklist

For everyone

One topic.
The title is a Conventional Commit (type(scope): description).
Rebased on current main.
python -I -B tests/run_all.py run and passing (any failure that also fails on main is named above).
Tested in the game, or what wasn't is named under Testing.
Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests).
Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported.
No game files, dumps, build output or generated files.
third_party/ untouched (a new folder with a PROVENANCE.md instead).
The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled.

If applicable

AI assistance recorded with Assisted-by: / Co-Authored-By: trailers.
Flags table filled in; README / docs/development.md switch reference updated.
Screenshots for a change that shows on screen.
A row in docs/tested-configurations.md for each measured run.
MKWVK_API_VERSION bumped (gpu/vk/vk_api.h's MkwVkHost or MkwVkRenderer changed).
The SPIR-V headers regenerated with glslang (a shader changed).
A game-free test added for the fix, failing on main and passing here.
A check under tests/ that reads changed source updated in this PR, not weakened.
Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths.
Docs updated.

🤖 Generated with Claude Code

## Why To see whether memory leaks across a scene or a loop, today you collect lines from several places logged at different rates: the status line every 600 frames, the VMA heaps, `memory_stats`, and the process's memory, which only the bug report shows. This writes them all side by side, one JSON line every N seconds, with a small reader. ## What changed - renderer: `counters_tick` in `gpu/vk/vk_replay.cpp`, called from `status_tick`: with `HOS_COUNTERS=<seconds>` it writes one JSON line per interval to `HOS_COUNTERS_FILE`, else `longshot_counters.jsonl` beside the game. The first line names the format and the build, `{"counters":1,"version":"...","every_s":5}`. Each line has: - the runtime's counters: the guest heap and its use, the address regions, the process's private memory and working set; - the video memory and the system heap the driver allows (VMA's budgets: `vram_usage`, `system_heap_usage`); - the texture cache (count, MB), the mirrors and their MB, the bytes synced and written back so far (`memory_stats`); - draws, pipelines (`draw_stats`) and the program cache's hits so far (`st_cache_hits`, `vk_progcache.h`). Every counter is read where it is already kept, on the replay thread that owns the texture cache's. Unset, nothing is read or written. - interface: `MkwVkHost.host_counters` in `gpu/vk/vk_api.h`, the runtime's part of a line as JSON members; `MKWVK_API_VERSION` 21 → 22. - host: `host_counters` in `host/rx_backend.c` (the heap, its use, the regions, `hos_process_memory`'s private memory and working set), last in `vk_host`; `hos_mem_regions()` in `host/hos_mem.c` gives the region count. - renderer: `build_version()` in `gpu/vk/vk_menu.cpp` (declared in `vk_menu.h`) gives the version for the first line; `system_heap_usage` is declared in `vk_internal.h`. - scripts: `scripts/counters.py` (new, standard library): a table; `--diff [A B]`, line B against line A (by default the first and the last); `--grew`, the levels that went up at every line and never down, the leak check across a loop of the same scene. Counters "so far" (presents, draws, synced bytes...) are left out of `--grew`. A file of another format is refused. - tests: `tests/test_counters.py` (new). Not included: the virtual-core counters, whose stats reset every 5 s when logged and need a getter that doesn't reset them; and `HOS_MEM_MAP`, which walks the whole address space and is too slow to sample. ## Flags and settings | Flag | New or changed | What it does | Default now | Goal: on by default? | Helps on | Doesn't help on | Untested on | Untested, but should help on | |---|---|---|---|---|---|---|---|---| | `HOS_COUNTERS` | New | Every so many seconds, a JSON line of the memory and cache counters | unset (nothing read or written) | no (a developer switch) | - | - | Linux | - | | `HOS_COUNTERS_FILE` | New | The file `HOS_COUNTERS` writes to | unset (`longshot_counters.jsonl` beside the game) | no (a developer switch) | - | - | Linux | - | ## Measurements Run on a Longshot build of this branch (this PR was made on Longshot; there is no GitHub original). There is no main-side measurement: `main` has no `HOS_COUNTERS`, and the run checks the counters' file, not a speed. The two columns are two points of the same run (`counters.py --diff`), not two builds. main: 82de567a (2026-11-12) (this branch's base; nothing was run on it) this PR: feat/counters @ 03608e3a Hardware: RTX 3080 (NVIDIA 621.14), Ryzen 9 3900X, 64 GB, Windows 11 Scene: from the start through a save into Kokiri Forest Method: `HOS_COUNTERS=2`, one launch; `scripts/counters.py --diff` from the start to Kokiri Forest, and `--grew` How to reproduce: set `HOS_COUNTERS=2`, play from the start into Kokiri Forest from a save, then `python -I scripts/counters.py --diff` and `--grew` on `longshot_counters.jsonl` | `counters.py --diff` | this PR (Longshot build), the start | this PR (Longshot build), Kokiri Forest | |---|---|---| | textures | 3 | 1,247 | | texture MB | 31 | 3,205 | | mirrors | 4 | 75 | | mirror MB | 16 | 1,128 | | video memory, MB | 2,270 | 6,931 | | the process's private memory | 16.5 GB | 25.1 GB | - 37 lines over 74 s, the first naming the format and the build, then one every 2 s, each valid JSON. - `--grew`: only the mirrors went up at every line, as the scene loaded in. - No errors in the log; Kokiri Forest at 32-37 fps. This branch's build recompiled the host and renderer in 45 s. ## Testing **Tested:** the runs under Measurements (on this branch) and the game-free checks this description names. **Not tested:** Linux (the checks and the game run were on Windows only). **Needs other testers:** - Linux: `HOS_COUNTERS=2`, expecting valid JSON lines with the process's memory read from /proc (request to open) ## Screenshots Nothing on screen. ## Risk and rollback - The renderer interface changes (`MkwVkHost.host_counters`, version 22), so a renderer built before this change and a runtime built after it (or the other way round) no longer have the same version. - Unset, `counters_tick` returns after one cached `getenv`. Set, it writes and flushes a line on the replay thread every N seconds. - Leaving `HOS_COUNTERS` unset turns it off; reverting the commit is enough. ## Reviewer notes - Made on Longshot directly (no GitHub original). The dev-loop proposal's E3 ([NotRexed/Hookshot#2636](https://github.com/NotRexed/Hookshot/pull/2636)). - Checks run (on Longshot, not the whole `run_all.py`): every check that reads the changed files (125), through `run_all.py`, Windows: 124 pass. `test_readback_wait_bounded` fails the same way on Longshot's `main` on this PC. - `tests/test_counters.py`: `counters_tick` is cut out and run against stub counters through a clock. Unset, it writes nothing and makes no file. Set to 5, its first line names the format, build and interval, then one line every 5 s (not one per present), each valid JSON with the runtime's members merged in. `HOS_COUNTERS_FILE`, and a runtime that gives no counters, also work. `host_counters` is cut out and run: its members, wrapped in braces, are JSON, and a buffer too small gives 0. `counters.py`: the table, the difference, what only grew, and a file of another format refused. In the source: `status_tick` calls it; `MkwVkHost` ends with the member, version 22; the runtime's table gives it last. Two deliberate breaks fail it: the "unset, nothing" line removed, and a line written at every present. ## Issues None. ## Checklist ### For everyone - [x] One topic. - [x] The title is a Conventional Commit (`type(scope): description`). - [ ] Rebased on current `main`. - [ ] `python -I -B tests/run_all.py` run and passing (any failure that also fails on `main` is named above). - [x] Tested in the game, or what wasn't is named under Testing. - [x] Testing filled in: what was tested, what wasn't, and what needs other testers (with test requests). - [ ] Measured: main and this PR at named commits, on the same PC, scene and settings, at least 2 interleaved runs each, the median reported. - [x] No game files, dumps, build output or generated files. - [x] `third_party/` untouched (a new folder with a `PROVENANCE.md` instead). - [x] The license is OK: GPL-3.0-or-later, nothing incompatible and nothing decompiled. ### If applicable - [x] AI assistance recorded with `Assisted-by:` / `Co-Authored-By:` trailers. - [ ] Flags table filled in; README / docs/development.md switch reference updated. - [ ] Screenshots for a change that shows on screen. - [ ] A row in `docs/tested-configurations.md` for each measured run. - [x] `MKWVK_API_VERSION` bumped (`gpu/vk/vk_api.h`'s `MkwVkHost` or `MkwVkRenderer` changed). - [ ] The SPIR-V headers regenerated with glslang (a shader changed). - [x] A game-free test added for the fix, failing on `main` and passing here. - [ ] A check under `tests/` that reads changed source updated in this PR, not weakened. - [ ] Host code builds on both Windows (llvm-mingw) and Linux, with UTF-8 paths. - [ ] Docs updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dyst added 1 commit

2026-11-13 01:06:51 +00:00

feat(runtime): HOS_COUNTERS, the memory and cache counters as one JSON line every N seconds 03608e3a5c

The counters a leak or a memory question needs were logged at different rates in different places (the status line every
600 frames, the VMA heaps, memory_stats, the process's memory only in the bug report), so comparing two points of a run
meant collecting lines from several. HOS_COUNTERS=<seconds> writes them side by side as one JSON line per interval to
HOS_COUNTERS_FILE, else longshot_counters.jsonl beside the game, its first line naming the format and the build: the
runtime's (guest heap, regions, process memory; MkwVkHost.host_counters, so MKWVK_API_VERSION 22), the video memory and
system heap the driver allows, the texture cache, mirrors and synced and written-back bytes, draws, pipelines and program
cache hits. scripts/counters.py prints a table, two points' difference, and the levels that only grew. Unset, nothing is
read or written. The dev-loop proposal's E3 (NotRexed/Hookshot#2636).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

dyst changed title from ~~feat(runtime): HOS_COUNTERS, the memory and cache counters as one JSON line every N seconds~~ to WIP: feat(runtime): HOS_COUNTERS, the memory and cache counters as one JSON line every N seconds

2026-11-13 01:14:04 +00:00

dyst commented

2026-11-13 01:14:05 +00:00

Author

First-time contributor

Back to WIP under #48's process: this was carried over before the new template. Its description will be brought to the template and measured on Longshot (main against this PR, at named commits) when it's wanted for review; ask here if you'd like it sooner.

dyst commented

2026-11-13 01:58:30 +00:00

Author

First-time contributor

Description brought to Longshot's PR template (#48). Still WIP: it has a game run on this branch but no main-against-this-PR measurement yet, and it isn't rebased on current main.

dyst commented

2026-11-13 02:00:45 +00:00

Author

First-time contributor

Description moved to the template's Testing section and whole checklist (#72): what was tested, what wasn't, and what needs other testers (their test requests to open while this is WIP).

mooki referenced this pull request

2026-11-13 02:27:28 +00:00

Idea: system performance readings alongside the frame counter, with optional file output #92

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u 'feat/counters:dyst-feat/counters'

git switch 'dyst-feat/counters'

Rows
Columns

WIP: feat(runtime): HOS_COUNTERS, the memory and cache counters as one JSON line every N seconds #65

Why

What changed

Flags and settings

Measurements

Testing

Screenshots

Risk and rollback

Reviewer notes

Issues

Checklist

For everyone

If applicable

Checkout