fix(vk): validate capture dimensions before allocating readbacks #1551

Merged

Arkex-ai merged 3 commits from fix/capture-dimension-contract into main

2026-11-10 08:16:49 +00:00

Arkex-ai commented

2026-11-10 07:40:59 +00:00

(Migrated from github.com)

Screenshot dimensions are converted with atoi and narrowed before validation. Invalid text becomes zero, negative dimensions become huge unsigned values, and the RGBA byte expression can wrap. Requests can reach Vulkan before rejection, and an image that fits the device can still exceed the existing PNG encoder's single-IDAT limit.

Parse positive decimal dimensions without overflow, then validate device, readback and encoder bounds before any screenshot allocation. Reject unsupported requests once for that process, with a diagnostic; keep valid requested sizes and the1280x720 default. Window presentation continues, and headless mode retains its offscreen policy without producing a capture for an invalid request. Signed, whitespace-containing and partially numeric values are intentionally rejected rather than accepted as a prefix. This adds no arbitrary memory budget and does not guarantee that an otherwise valid request can be allocated.

Validation at c35bd37 on main 7d1562d:

Production setup plus actual capture-entry/presentation conditions run with a Vulkan allocation model across24 input/capability cases in both window/headless routes. Standard,43:18,32:9, odd/small sizes and defaults pass; device, conversion, uint64 wrap and PNG/deflate-overhead boundaries reject before allocation and report once. Large boundary cases allocate no pixel memory.
Original main fails the pre-allocation contract; seven protection-removal/corruption mutations fail by assertions. The real PNG writer's small multi-deflate-block output passes CRC/header/decompression/scanline checks. Windows LLVM, Linux GCC and GCC ASan/UBSan pass.
Full Windows suite224 passed/2 failed/20 skipped versus exact main223/2/20, with the same existing NUP privilege/native SDK limitations. Official clean guest generation and matching API12 host/backend build pass.
Native FG2: valid capture produces two fully decoded RGB3440x1440 PNGs. Width0 reports once, produces no PNG, and the game continues rendering and resizing. Both runs maximize/restore and exit0; original checkpoint hashes are unchanged. No large hardware allocation or GPU fault was induced.

Resource lifetime/readiness remains in #1472. A private CPU composition passes both production-code suites after its sliced setup fixture includes the real dimension guard and initialized device limits; validation is not stubbed out. This PR does not import1472's allocation changes. See docs/capture-dimensions.md, including the PNG format bound and integration note. No game data, captures, generated game code or binaries are included. Three exact-head fork workflows follow.

Current-head validation

Current head e20c42c, based on main ff69134. Earlier local/runtime results describe their recorded source hashes.

Integrated with main screenshot readiness/slot ownership/flash sampling. The sliced resource fixture includes the real dimension guard and initialized device limits. Dimension/resource/slot suites pass, seven dimension mutations remain causal, and Linux GCC/ASan/UBSan pass. The fixture also uses the production CRT compatibility definition under native Windows clang; warnings remain errors. Earlier native evidence stays attached to its recorded hash.

Fork runs are separate from upstream Actions approval; no maintainer approval gate is bypassed.

Screenshot dimensions are converted with `atoi` and narrowed before validation. Invalid text becomes zero, negative dimensions become huge unsigned values, and the RGBA byte expression can wrap. Requests can reach Vulkan before rejection, and an image that fits the device can still exceed the existing PNG encoder's single-IDAT limit. Parse positive decimal dimensions without overflow, then validate device, readback and encoder bounds before any screenshot allocation. Reject unsupported requests once for that process, with a diagnostic; keep valid requested sizes and the1280x720 default. Window presentation continues, and headless mode retains its offscreen policy without producing a capture for an invalid request. Signed, whitespace-containing and partially numeric values are intentionally rejected rather than accepted as a prefix. This adds no arbitrary memory budget and does not guarantee that an otherwise valid request can be allocated. Validation at `c35bd37` on main `7d1562d`: - Production setup plus actual capture-entry/presentation conditions run with a Vulkan allocation model across24 input/capability cases in both window/headless routes. Standard,43:18,32:9, odd/small sizes and defaults pass; device, conversion, uint64 wrap and PNG/deflate-overhead boundaries reject before allocation and report once. Large boundary cases allocate no pixel memory. - Original main fails the pre-allocation contract; seven protection-removal/corruption mutations fail by assertions. The real PNG writer's small multi-deflate-block output passes CRC/header/decompression/scanline checks. Windows LLVM, Linux GCC and GCC ASan/UBSan pass. - Full Windows suite224 passed/2 failed/20 skipped versus exact main223/2/20, with the same existing NUP privilege/native SDK limitations. Official clean guest generation and matching API12 host/backend build pass. - Native FG2: valid capture produces two fully decoded RGB3440x1440 PNGs. Width0 reports once, produces no PNG, and the game continues rendering and resizing. Both runs maximize/restore and exit0; original checkpoint hashes are unchanged. No large hardware allocation or GPU fault was induced. Resource lifetime/readiness remains in #1472. A private CPU composition passes both production-code suites after its sliced setup fixture includes the real dimension guard and initialized device limits; validation is not stubbed out. This PR does not import1472's allocation changes. See `docs/capture-dimensions.md`, including the PNG format bound and integration note. No game data, captures, generated game code or binaries are included. Three exact-head fork workflows follow.  ### Current-head validation Current head `e20c42c`, based on main `ff69134`. Earlier local/runtime results describe their recorded source hashes. Integrated with main screenshot readiness/slot ownership/flash sampling. The sliced resource fixture includes the real dimension guard and initialized device limits. Dimension/resource/slot suites pass, seven dimension mutations remain causal, and Linux GCC/ASan/UBSan pass. The fixture also uses the production CRT compatibility definition under native Windows clang; warnings remain errors. Earlier native evidence stays attached to its recorded hash. - [Linux AppImage](https://github.com/Arkex-ai/Hookshot/actions/runs/42779705198): success. - [Windows builder](https://github.com/Arkex-ai/Hookshot/actions/runs/42779701175): success. - [Game-free source regressions](https://github.com/Arkex-ai/Hookshot/actions/runs/42779697853): success. Fork runs are separate from upstream Actions approval; no maintainer approval gate is bypassed.