fix(linux): allow udmabuf when a sandbox hides its module limit #2431

Open

Arkex-ai wants to merge 2 commits from Arkex-ai/fix/linux-udmabuf-startup into main

Arkex-ai commented

2026-11-11 19:00:01 +00:00

(Migrated from github.com)

Summary

An accessible /dev/udmabuf goes unused when a sandbox hides its module-limit file. This lets automatic startup try the conservative 64 MB fallback already used by the renderer, addressing the selection gap in #2404 (tracked in #1000).

Draft: affected-runtime and discard-safety validation remain open. The memfd configuration reported in #2411 still crashes without the Vulkan renderer. The latest fault trace faults at host null outside the last 16 discarded ranges; it does not establish the discard as the cause. This startup proposal does not fix that crash or prove safe affected gameplay.

Technical changes

Use the fallback only when the limit file cannot be opened after the device opens successfully. Preserve readable positive values, zero/invalid limits, permission warnings and explicit on/off overrides.
Keep renderer import limits, imported-page guards and discard behavior unchanged. The 64 MB fallback is conservative, not a claim about the kernel's configured limit.
Exercise the production selection block with 16 scenarios and five causal mutations; document the sandbox boundary and provisional worklist item.

Validation

Current head 774c7f5f7a9c23e9178251cd97b5129c29103d38, rebased onto main 51da0464b4cf0c7f56b81ebd6bb309325835e175. All four proposal-file blobs are identical to 6718dd5e; the eight intervening upstream commits do not alter those files.

Linux: 7 focused checks passed, including startup selection, permission-warning behavior and affected build/runner checks. Windows: 6 applicable focused checks passed. The POSIX stderr fixture runs on Linux; no product/test check was weakened.
GCC and Clang pass all 16 selector cases. Five semantic mutations are rejected, and exact current main fails the hidden-sysfs case after successful compilation.
The complete Linux host translation unit passes GCC/Clang syntax checks with repository-generated headers and no game input. This is not a linked game build.
All applicable hosted checks passed on 774c7f5f: source/SPIR-V/lint, Windows backend/Builder and Linux AppImage. Release-asset publication is skipped for the PR. The previous head's release-fixture failure remains historical; this rebase does not change or fix that fixture.

Earlier full-suite evidence and remaining limits

Earlier 164f6c09, with identical proposal-file blobs: full Linux suite 467 / 0 / 24; Windows 451 / 4 / 36. All four Windows failures reproduced on the exact clean base. These are earlier-head source checks, not current hosted CI or gameplay.

Actual affected Flatpak/AMD imports, area transitions and memory-discard safety remain unverified. The #2404 reporter offered to test. A speculative zero-and-drop workaround is not included: it had a reported residency trade-off, and the new null-fault trace does not demonstrate a discarded-range dereference. Current main and this PR retain the existing policy.

## Summary An accessible `/dev/udmabuf` goes unused when a sandbox hides its module-limit file. This lets automatic startup try the conservative 64 MB fallback already used by the renderer, addressing the selection gap in #2404 (tracked in #1000). **Draft: affected-runtime and discard-safety validation remain open.** The memfd configuration reported in [#2411](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2411) still crashes without the Vulkan renderer. The [latest fault trace](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2411) faults at host null outside the last 16 discarded ranges; it does not establish the discard as the cause. This startup proposal does not fix that crash or prove safe affected gameplay. ## Technical changes - Use the fallback only when the limit file cannot be opened after the device opens successfully. Preserve readable positive values, zero/invalid limits, permission warnings and explicit on/off overrides. - Keep renderer import limits, imported-page guards and discard behavior unchanged. The 64 MB fallback is conservative, not a claim about the kernel's configured limit. - Exercise the production selection block with 16 scenarios and five causal mutations; document the sandbox boundary and provisional worklist item. ## Validation Current head **`774c7f5f7a9c23e9178251cd97b5129c29103d38`**, rebased onto main **`51da0464b4cf0c7f56b81ebd6bb309325835e175`**. All four proposal-file blobs are identical to `6718dd5e`; the eight intervening upstream commits do not alter those files. - Linux: **7 focused checks passed**, including startup selection, permission-warning behavior and affected build/runner checks. Windows: **6 applicable focused checks passed**. The POSIX stderr fixture runs on Linux; no product/test check was weakened. - GCC and Clang pass all 16 selector cases. Five semantic mutations are rejected, and exact current main fails the hidden-sysfs case after successful compilation. - The complete Linux host translation unit passes GCC/Clang syntax checks with repository-generated headers and no game input. This is not a linked game build. - All applicable hosted checks passed on `774c7f5f`: [source/SPIR-V/lint](https://github.com/NotRexed/Hookshot/actions/runs/43230868516), [Windows backend/Builder](https://github.com/NotRexed/Hookshot/actions/runs/43230868519) and [Linux AppImage](https://github.com/NotRexed/Hookshot/actions/runs/43230868409). Release-asset publication is skipped for the PR. The previous head's release-fixture failure remains historical; this rebase does not change or fix that fixture. <details> <summary>Earlier full-suite evidence and remaining limits</summary> Earlier `164f6c09`, with identical proposal-file blobs: full Linux suite **467 / 0 / 24**; Windows **451 / 4 / 36**. All four Windows failures reproduced on the exact clean base. These are earlier-head source checks, not current hosted CI or gameplay. Actual affected Flatpak/AMD imports, area transitions and memory-discard safety remain unverified. The #2404 reporter offered to test. A speculative zero-and-drop workaround is not included: it had a reported residency trade-off, and the new null-fault trace does not demonstrate a discarded-range dereference. Current main and this PR retain the existing policy. </details>

github-sync referenced this pull request

2026-11-12 06:23:06 +00:00

Linux: memfd guest-window discard crashes after the intro #2411

github-sync changed title from ~~WIP: fix(linux): allow udmabuf when a sandbox hides its module limit~~ to fix(linux): allow udmabuf when a sandbox hides its module limit

2026-11-12 06:23:10 +00:00

github-sync commented

2026-11-12 07:27:56 +00:00

Owner

@Arkex-ai commented on GitHub · 2026-11-12 06:29 UTC · original

Rebased on c544828f at 6718dd5e. Builds, packaging, SPIR-V/lint and the other three source shards passed. The remaining failure is test_release_scripts.py in Linux part 1: the simulated concurrent nightly runs finish with HTTP 404s and a checksum mismatch.

This PR changes neither that fixture nor the release scripts. The earlier full local Linux suite passed at 164f6c09 (467/0/24); the later rebase preserves all four proposal-file blobs, and the current focused checks pass. That earlier suite is not a claim that this hosted job passed.

Could a maintainer rerun the failed shard? The PR remains draft behind the #2411 discard-safety and affected-Flatpak validation gates.

> [@Arkex-ai](https://github.com/Arkex-ai) commented on GitHub · 2026-11-12 06:29 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2431#issuecomment-6816497407) Rebased on `c544828f` at `6718dd5e`. Builds, packaging, SPIR-V/lint and the other three source shards passed. The remaining failure is [`test_release_scripts.py` in Linux part 1](https://github.com/NotRexed/Hookshot/actions/runs/43166305483/job/131008870336): the simulated concurrent nightly runs finish with HTTP 404s and a checksum mismatch. This PR changes neither that fixture nor the release scripts. The earlier full local Linux suite passed at `164f6c09` (467/0/24); the later rebase preserves all four proposal-file blobs, and the current focused checks pass. That earlier suite is not a claim that this hosted job passed. Could a maintainer rerun the failed shard? The PR remains draft behind the #2411 discard-safety and affected-Flatpak validation gates.

github-sync commented

2026-11-12 14:42:35 +00:00

Owner

@Arkex-ai commented on GitHub · 2026-11-12 13:48 UTC · original

Rebased onto current main 51da0464 at 774c7f5f; all four proposal-file blobs are unchanged. Linux focused checks pass 7/7, Windows 6/6; both Linux compilers pass the selector controls and complete host syntax check. New exact-head CI is running separately.

@CIB, thanks for the rendererless trace. Since the null fault is outside all 16 recorded ranges, the 0.3 ms adjacency does not yet show a discarded-page access. The repeated three-page decommits are another lead, but I won't infer a cause from them.

If you run the second trace, could it also record the originating thread ID for each discard and the faulting native module/RVA plus available host-runtime symbols? No instruction bytes or memory dump are needed. That should help distinguish a repeated-decommit caller from an invalid/null pointer in the process-lifetime path. This PR stays draft while affected-runtime safety remains unproven.

> [@Arkex-ai](https://github.com/Arkex-ai) commented on GitHub · 2026-11-12 13:48 UTC · [original](https://github.com/NotRexed/Hookshot/pull/2431#issuecomment-6825389527) Rebased onto current main `51da0464` at `774c7f5f`; all four proposal-file blobs are unchanged. Linux focused checks pass 7/7, Windows 6/6; both Linux compilers pass the selector controls and complete host syntax check. New exact-head CI is running separately. @CIB, thanks for the [rendererless trace](https://git.magicbean.dev/MagicBeanDev/Hookshot/issues/2411). Since the null fault is outside all 16 recorded ranges, the 0.3 ms adjacency does not yet show a discarded-page access. The repeated three-page decommits are another lead, but I won't infer a cause from them. If you run the second trace, could it also record the originating thread ID for each discard and the faulting native module/RVA plus available host-runtime symbols? No instruction bytes or memory dump are needed. That should help distinguish a repeated-decommit caller from an invalid/null pointer in the process-lifetime path. This PR stays draft while affected-runtime safety remains unproven.

github-sync referenced this pull request

2026-11-12 15:49:19 +00:00

[Tracking] Linux and Steam Deck: GPU hangs, crashes, low FPS and build failures #1000

Commenting is not possible because the repository is archived.

No reviewers