Prevent Castle Town guard-house crashes with a larger render-target pool #1202

2026-11-09T17:09:50Z

nathanaelries commented

2026-11-09 17:09:50 +00:00

(Migrated from github.com)

Talking to the soldier in Castle Town’s guard house while rotating the camera repeatedly crashed at 2560×1440 with the 305 MiB render-target pool. Reserving 512 MiB resolved the reported reproduction.

This gives the pool a 512 MiB minimum while preserving existing 4K/ultrawide growth and explicit HOS_RT_POOL_MB overrides. The existing graphics-heap/budget hooks use the same sizing function, so the backing allocation grows with it. The base reservation increases by 207 MiB.

The initial stop was native NVN: texture handle ids -1/264. The companion handle correction matches the original SDK: sign-extend both IDs, combine them, and return zero when the texture ID is zero. That correction alone did not resolve the room: it exposed a subsequent allocator crash in f_8005ed00. Disabling write-back diff and the separate present queue also did not resolve it. Raising the pool to 512 MiB with those changes retained did.

Validation

python -I -B tests/run_all.py --jobs 1 --strict nvn rt_pool: 5 passed.
New production-function fixtures: 12 pool-sizing cases and 13 handle cases, passing with GCC and Clang. The pool regression fails against current main’s old sizing; the handle regression fails against the old constructor.
Local differential check: the corrected source routine and the patched native routine matched the original SDK in 10,196 boundary/random argument pairs. SDK bytes and generated artifacts stay local; the committed fixtures contain authored constants only.
In-game confirmation: v2.0.8 (source ed212f62), Linux, RX 7800 XT/RADV, docked 2560×1440 at 100% resolution, frame generation off. The same soldier/camera action passed after setting the pool to 512 MiB. Steam’s overlay remained enabled for the working controller setup.

The complete game has not been rebuilt from this exact branch, and Windows runtime behavior has not been tested. The pool recommendation is based on the repeatable allocation failure and the successful larger-pool test; peak pool use has not been instrumented. No game files, saves, settings files, binaries, caches, or SDK code are included.

Talking to the soldier in Castle Town’s guard house while rotating the camera repeatedly crashed at 2560×1440 with the 305 MiB render-target pool. Reserving 512 MiB resolved the reported reproduction. This gives the pool a 512 MiB minimum while preserving existing 4K/ultrawide growth and explicit `HOS_RT_POOL_MB` overrides. The existing graphics-heap/budget hooks use the same sizing function, so the backing allocation grows with it. The base reservation increases by 207 MiB. The initial stop was `native NVN: texture handle ids -1/264`. The companion handle correction matches the original SDK: sign-extend both IDs, combine them, and return zero when the texture ID is zero. That correction alone did **not** resolve the room: it exposed a subsequent allocator crash in `f_8005ed00`. Disabling write-back diff and the separate present queue also did not resolve it. Raising the pool to 512 MiB with those changes retained did. ## Validation - `python -I -B tests/run_all.py --jobs 1 --strict nvn rt_pool`: **5 passed**. - New production-function fixtures: 12 pool-sizing cases and 13 handle cases, passing with GCC and Clang. The pool regression fails against current main’s old sizing; the handle regression fails against the old constructor. - Local differential check: the corrected source routine and the patched native routine matched the original SDK in 10,196 boundary/random argument pairs. SDK bytes and generated artifacts stay local; the committed fixtures contain authored constants only. - In-game confirmation: v2.0.8 (`source ed212f62`), Linux, RX 7800 XT/RADV, docked 2560×1440 at 100% resolution, frame generation off. The same soldier/camera action passed after setting the pool to 512 MiB. Steam’s overlay remained enabled for the working controller setup. The complete game has not been rebuilt from this exact branch, and Windows runtime behavior has not been tested. The pool recommendation is based on the repeatable allocation failure and the successful larger-pool test; peak pool use has not been instrumented. No game files, saves, settings files, binaries, caches, or SDK code are included.

hckrman101 (Migrated from github.com) approved these changes

2026-11-09 17:56:45 +00:00

hckrman101 (Migrated from github.com) left a comment

Thanks, this looks good to me. What I checked:

hos_rt_pool_size is the only sizing source for the pool config store, the Graphics heap growth, the module budget correction and the f_80b72340 heap trim in nx2/oot_patches.py. So the 512 MB floor applies to all four consistently. At 1440p those patches used to do nothing (growth was 0). Now they add 207 MB, which is less than the 382 MB the 4K path already uses, so it stays inside a configuration that already works.
The env override still wins, including values below the floor. 4K/ultrawide results are unchanged (687 / 1373). The 1080p output mode recently added on main gets 512, which is fine. It merges cleanly with that change.
I checked the handle change against the Vulkan side. An all-ones handle decodes to texture slot 0xfffff and sampler 0xfff. texture_of, sampler_of and texture_slot_version all bounds-check those, so it falls back to the dummy texture and default sampler instead of indexing out of range.
Both tests compile the real function bodies through the existing block() helper, they're picked up by run_all.py, and the main cases fail against the old code (die -> exit 86, and 305 instead of 512). I checked the expected numbers in the fixtures by hand.
No overlap with other open PRs (#1179 touches nn_applet.c, but in different hunks).

Two non-blocking notes are inline: (1) the floor is applied after scaling, so 4K doesn't get the same headroom the guard house needed at 1440p, and (2) a one-time log for negative texture ids would keep pool exhaustion visible now that the die() is gone.

Thanks, this looks good to me. What I checked: - `hos_rt_pool_size` is the only sizing source for the pool config store, the Graphics heap growth, the module budget correction and the f_80b72340 heap trim in `nx2/oot_patches.py`. So the 512 MB floor applies to all four consistently. At 1440p those patches used to do nothing (growth was 0). Now they add 207 MB, which is less than the 382 MB the 4K path already uses, so it stays inside a configuration that already works. - The env override still wins, including values below the floor. 4K/ultrawide results are unchanged (687 / 1373). The 1080p output mode recently added on main gets 512, which is fine. It merges cleanly with that change. - I checked the handle change against the Vulkan side. An all-ones handle decodes to texture slot 0xfffff and sampler 0xfff. `texture_of`, `sampler_of` and `texture_slot_version` all bounds-check those, so it falls back to the dummy texture and default sampler instead of indexing out of range. - Both tests compile the real function bodies through the existing `block()` helper, they're picked up by `run_all.py`, and the main cases fail against the old code (die -> exit 86, and 305 instead of 512). I checked the expected numbers in the fixtures by hand. - No overlap with other open PRs (#1179 touches nn_applet.c, but in different hunks). Two non-blocking notes are inline: (1) the floor is applied after scaling, so 4K doesn't get the same headroom the guard house needed at 1440p, and (2) a one-time log for negative texture ids would keep pool exhaustion visible now that the `die()` is gone.

host/nn_applet.c

					
				@ -134,1 +134,4 @@

				    const uint64_t scaled = px > 2560 * 1440 ? (game * px / (2560 * 1440) + 0xfffff) & ~0xfffffULL : game;

				    const uint64_t minimum = 512ULL << 20;

				    const uint64_t v = e ? strtoull(e, NULL, 0) << 20 : scaled < minimum ? minimum : scaled;

				    static int logged;

Something to consider: the floor is applied after scaling. If the guard house needs more than 305 MB at 1440p because of render targets, it would need roughly 2.25x that at 4K, but 4K still gets 687 MB (305 scaled). Applying the floor to the base first (game < minimum ? minimum : game, then scale) would give 4K the same relative headroom, at about +465 MB at 4K. Not tested at 4K either way, so I'm fine leaving it as is for now. It's just worth knowing if a 4K report of the same room comes in.

Something to consider: the floor is applied after scaling. If the guard house needs more than 305 MB at 1440p because of render targets, it would need roughly 2.25x that at 4K, but 4K still gets 687 MB (305 scaled). Applying the floor to the base first (`game < minimum ? minimum : game`, then scale) would give 4K the same relative headroom, at about +465 MB at 4K. Not tested at 4K either way, so I'm fine leaving it as is for now. It's just worth knowing if a 4K report of the same room comes in.

host/nvn_native.c

					
				@ -207,2 +207,4 @@

				       rather than stopping the game here. Keep the signed ids, including their upper bits. */

				    RET(t ? (uint64_t)t | (uint64_t)s << 20 : 0);

				}

				NV(nvnDeviceGetTexelFetchHandle) { RET((uint64_t)I(1)); }

Matching the SDK makes sense. One small thing: the old die("texture handle ids -1/...") was the only clear sign that the render-target pool had run out (issue 1031 was diagnosed from it, and the comment above hos_rt_pool_size still names it as the symptom). Maybe a one-time hos_log when t < 0, so a future pool exhaustion shows up in the log before the allocator crash in f_8005ed00?

Matching the SDK makes sense. One small thing: the old `die("texture handle ids -1/...")` was the only clear sign that the render-target pool had run out (issue 1031 was diagnosed from it, and the comment above `hos_rt_pool_size` still names it as the symptom). Maybe a one-time `hos_log` when `t < 0`, so a future pool exhaustion shows up in the log before the allocator crash in f_8005ed00?