Game: at Windows' commit limit the NVN layer's own allocations end the game ("recording handle set") with no wait and no out-of-memory notice #333

New issue

Open

opened 2026-11-14 01:26:28 +00:00 by llun · 0 comments

llun commented

2026-11-14 01:26:28 +00:00

Member

Longshot version

nightly-80-7315957c, a player's log. Longshot main (16047f55) has the same code; the lines are quoted below.

Channel

Nightly

What kind of problem

Crash, freeze or black screen

What happened

At Windows' commit limit (RAM plus the page file full), the game ended with a fatal error that doesn't say it ran out of memory:

[hos Presentation Thread] vk: commit limit near: 28 MB of commit left (reserve 2560 MB); the textures' budget down to 0 MB of their 2234 MB (RX_VK_COMMIT_RESERVE_MB)
...
[hos Presentation Thread] vk: image over budget, retrying without WITHIN_BUDGET
[hos Presentation Thread] vk: VRAM full: a 768x768 image (format 37) in system memory
[hos Resource Memory] mem: commit of 0x80000 bytes at the commit limit made after 50 ms (1 so far)
[hos host] vk: image over budget, retrying without WITHIN_BUDGET
[hos Resource Memory] mem: commit of 0x3f0000 bytes at the commit limit made after 200 ms (2 so far)
[hos Presentation Thread] vk: image over budget, retrying without WITHIN_BUDGET
[hos ModuleSystemWorker3] FATAL: native NVN: recording handle set (lr 0x80600de0 sp 0x10062cb960)
[hos] guest backtrace: lr 0x80600de0 0x800a2da0 0x8009ac4c 0x80081268 0x8007fe40 0x80df26c0 0x80df2608 0x834a2ec8 0

recording handle set is live_handle_add() in host/nvn_native.c (line 1416): when the handle set doubles, calloc fails and die() → hos_fatal() ends the game with ExitProcess(3).

The memory pressure itself is the known problem (the worklist's top row, memory filling over play; #324 and #77). This issue is about how it ends:

No wait. #77 made hos_commit and the first-touch handler (host/mkw_glue.c) wait up to 5 s at the commit limit, retrying while the texture cache gives commit back. In this log that wait worked twice, a moment before the crash. The host's own C allocations don't go through it: a malloc/calloc failing at the limit ends the game at once.
No out-of-memory notice. The two commits that waited succeeded, so commit_refused() never ran, and neither did its log line or hos_out_of_memory_box. The player saw a crash with no explanation, and the log's last line names a hash set. The player has tried several forks and assumed it was their GPU.
It isn't one call site. The same pattern is in host/nvn_native.c at command buffer table (1278, 1313), recording buffer (%zu bytes) (1351), recording handle set (1416) and recording copy (%zu bytes) (1576). Whichever allocation loses the race at the limit decides the message.

Expected: a host allocation that fails at the commit limit gets the same treatment as a guest commit: wait for commit to come back, and if it doesn't, say "out of memory (RAM plus the page file)" in the log and in the notice before stopping.

A possible shape (not tried): an xalloc/xcalloc in the host that, on failure, checks GlobalMemoryStatusEx headroom (or _doserrno/GetLastError() for 1455), sleeps and retries like commit_wait(), and on giving up calls the same path as commit_refused() before die(). Use it at the five sites above, and in other host code that die()s on a failed allocation.

Where in the game

Hyrule Field (child), loaded from a save, about halfway across. The player says it happens there every time, on every fork they tried.

How often

Every time (for this player, at this spot)

Graphics card and driver version

RTX 3060 12 GB, NVIDIA 617.14

CPU, RAM and operating system

12 cores / 20 threads, 16 GB RAM with a 16 GB page file (commit limit about 32.5 GB), Windows

Changes from the defaults

1920x1080 output, DLSS Quality, Longshot's frame interpolation on (Auto), V-Sync off. Nothing else visible in the log.

What the log shows about memory

The game's private commit: 16.5 GB at the first status, 22.0 GB a minute later in Hyrule Field, 23.7 GB at the last status before the crash. VRAM was never the limit (about 6 of 11.1 GB used).
The commit rule (#77, default reserve 2560 MB) was active for the whole session. From the first minute it held headroom near 2 GB and cut the texture budget to 0, then headroom dropped to 28 MB.
The VRAM full: ... in system memory lines at the end come from the commit limit, not VRAM: every VRAM block is charged to the process's commit on Windows.

longshot_log.txt

Not attached: it's a player's log passed on through Discord. The lines above are quoted from it.

Filed from a review of the player's log; the root cause (commit growth) is the memory row and #324, so this issue covers only the crash path and its message.

🤖 Generated with Claude Code

### Longshot version nightly-80-7315957c, a player's log. Longshot main (16047f55) has the same code; the lines are quoted below. ### Channel Nightly ### What kind of problem Crash, freeze or black screen ### What happened At Windows' commit limit (RAM plus the page file full), the game ended with a fatal error that doesn't say it ran out of memory: ``` [hos Presentation Thread] vk: commit limit near: 28 MB of commit left (reserve 2560 MB); the textures' budget down to 0 MB of their 2234 MB (RX_VK_COMMIT_RESERVE_MB) ... [hos Presentation Thread] vk: image over budget, retrying without WITHIN_BUDGET [hos Presentation Thread] vk: VRAM full: a 768x768 image (format 37) in system memory [hos Resource Memory] mem: commit of 0x80000 bytes at the commit limit made after 50 ms (1 so far) [hos host] vk: image over budget, retrying without WITHIN_BUDGET [hos Resource Memory] mem: commit of 0x3f0000 bytes at the commit limit made after 200 ms (2 so far) [hos Presentation Thread] vk: image over budget, retrying without WITHIN_BUDGET [hos ModuleSystemWorker3] FATAL: native NVN: recording handle set (lr 0x80600de0 sp 0x10062cb960) [hos] guest backtrace: lr 0x80600de0 0x800a2da0 0x8009ac4c 0x80081268 0x8007fe40 0x80df26c0 0x80df2608 0x834a2ec8 0 ``` `recording handle set` is `live_handle_add()` in `host/nvn_native.c` (line 1416): when the handle set doubles, `calloc` fails and `die()` → `hos_fatal()` ends the game with `ExitProcess(3)`. The memory pressure itself is the known problem (the worklist's top row, memory filling over play; #324 and #77). This issue is about how it ends: - **No wait.** #77 made `hos_commit` and the first-touch handler (`host/mkw_glue.c`) wait up to 5 s at the commit limit, retrying while the texture cache gives commit back. In this log that wait worked twice, a moment before the crash. The host's own C allocations don't go through it: a `malloc`/`calloc` failing at the limit ends the game at once. - **No out-of-memory notice.** The two commits that waited succeeded, so `commit_refused()` never ran, and neither did its log line or `hos_out_of_memory_box`. The player saw a crash with no explanation, and the log's last line names a hash set. The player has tried several forks and assumed it was their GPU. - **It isn't one call site.** The same pattern is in `host/nvn_native.c` at `command buffer table` (1278, 1313), `recording buffer (%zu bytes)` (1351), `recording handle set` (1416) and `recording copy (%zu bytes)` (1576). Whichever allocation loses the race at the limit decides the message. Expected: a host allocation that fails at the commit limit gets the same treatment as a guest commit: wait for commit to come back, and if it doesn't, say "out of memory (RAM plus the page file)" in the log and in the notice before stopping. A possible shape (not tried): an `xalloc`/`xcalloc` in the host that, on failure, checks `GlobalMemoryStatusEx` headroom (or `_doserrno`/`GetLastError()` for 1455), sleeps and retries like `commit_wait()`, and on giving up calls the same path as `commit_refused()` before `die()`. Use it at the five sites above, and in other host code that `die()`s on a failed allocation. ### Where in the game Hyrule Field (child), loaded from a save, about halfway across. The player says it happens there every time, on every fork they tried. ### How often Every time (for this player, at this spot) ### Graphics card and driver version RTX 3060 12 GB, NVIDIA 617.14 ### CPU, RAM and operating system 12 cores / 20 threads, 16 GB RAM with a 16 GB page file (commit limit about 32.5 GB), Windows ### Changes from the defaults 1920x1080 output, DLSS Quality, Longshot's frame interpolation on (Auto), V-Sync off. Nothing else visible in the log. ### What the log shows about memory - The game's private commit: 16.5 GB at the first status, 22.0 GB a minute later in Hyrule Field, 23.7 GB at the last status before the crash. VRAM was never the limit (about 6 of 11.1 GB used). - The commit rule (#77, default reserve 2560 MB) was active for the whole session. From the first minute it held headroom near 2 GB and cut the texture budget to 0, then headroom dropped to 28 MB. - The `VRAM full: ... in system memory` lines at the end come from the commit limit, not VRAM: every VRAM block is charged to the process's commit on Windows. ### longshot_log.txt Not attached: it's a player's log passed on through Discord. The lines above are quoted from it. --- Filed from a review of the player's log; the root cause (commit growth) is the memory row and #324, so this issue covers only the crash path and its message. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

llun referenced this issue from a pull request that will close it,

2026-11-14 02:39:52 +00:00

WIP: fix(hos): wait at the commit limit for the host's own allocations, then say out of memory #341