MagicBeanDev/Hookshot

Archived

Watch

Fork 0

Linux, Flatpak Steam (Gaming Mode): guest memory falls back to host-pointer imports because /sys/module is hidden (half the frame rate on AMD) #2404

Closed

opened 2026-11-11 17:26:57 +00:00 by willcrisis · 3 comments

willcrisis commented

2026-11-11 17:26:57 +00:00

(Migrated from github.com)

Hookshot version

Measured on v2.0.14 (source f2796384); now on v2.1.0 (source e561602e), where the check below is unchanged (also on today's main). Both with a locally patched hookshot_vk.so (the value_rec fix from #997; unrelated to this).

What kind of problem

Performance

What happened

Started from Steam in Gaming Mode, the game runs at about half the frame rate it reaches in Desktop Mode on the same device and the same game folder. The log says why:

[hos host] mem: guest memory stays host-pointer imports for the GPU (slow submissions on AMD): udmabuf's size_limit_mb cannot be read
[hos sead::MainThread] vk: guest memory as dma-bufs: no (the window is anonymous memory)

On this distribution, steam is the Flatpak Steam (/usr/bin/steam runs flatpak run org.anatase.Steam), so a game started from Steam runs inside Steam's Flatpak sandbox. The sandbox has devices=all, so /dev/udmabuf opens, but Flatpak does not expose /sys/module, so /sys/module/udmabuf/parameters/size_limit_mb cannot be read. In host/hos_main.c the memfd window is only used when that file reads a non-zero limit (usable = limit_mb > 0), so the game falls back to host-pointer imports. Outside Steam (Desktop Mode), the same file reads 64 and dma-bufs are used.

Forcing the memfd with HOS_GUEST_MEMFD=1 in hookshot_env.txt fixes it. The renderer already assumes 64 MB when it cannot read the limit (gpu/vk/vk_memory.cpp: dmabuf_limit = 64ull << 20 before reading the file):

[hos host] mem: the guest window is a memfd's (12), shared with the GPU as dma-bufs
[hos sead::MainThread] vk: guest memory as dma-bufs: yes
[hos sead::MainThread] vk: dma-bufs of at most 64 MB (udmabuf's limit 64 MB); larger pools are several, in a sparse buffer

No pool failed to import in that run (0 not imported lines; the larger pools went in pieces into a sparse buffer). A 57 s Steam recording of that run (Death Mountain Trail, including a fight) shows no flashes or missing geometry.

Run	Memory for the GPU	Real fps in gameplay (median)	Slowest frame of 600
Gaming Mode, run 1	host-pointer imports	13.4	95 ms
Gaming Mode, run 2	host-pointer imports	12.3	145 ms
Gaming Mode, `HOS_GUEST_MEMFD=1`	dma-bufs	31.3	52 ms
Desktop Mode (no Steam)	dma-bufs	28.3	71 ms

"Real fps" is the replay's presents, N fps lines over frames with more than 300 draws; frame generation (2x) was on in all four, and these are the game's own frames. The two slow runs had Render resolution on Auto, the lighter setting on this GPU (it drops to its lowest size here); the forced-memfd run had it at 67%, and the Desktop Mode run started on Auto and was switched to 67% partway. So the slow runs did less GPU work (median GPU time 8.7 and 14.2 ms a frame, against 18.8 and 20.2 ms) and still ran at less than half the frame rate.

Suggested fix: when /dev/udmabuf opens but size_limit_mb cannot be opened, assume the kernel's default (64 MB) as the renderer does, rather than falling back to host pointers. A file that exists and reads 0 still means udmabuf refuses everything, so that case stays as it is:

FILE *f = fopen("/sys/module/udmabuf/parameters/size_limit_mb", "r");
if (f) {
    if (fscanf(f, "%llu", &limit_mb) != 1) limit_mb = 0;
    fclose(f);
} else {
    /* (a sandbox without /sys/module, such as Flatpak Steam's: udmabuf's default, as gpu/vk/vk_memory.cpp assumes) */
    limit_mb = 64;
    hos_log("mem: udmabuf's size_limit_mb cannot be read (a sandbox?): assuming its default, 64 MB");
}
usable = limit_mb > 0;

Possibly related: #2239 (pools over 64 MB not imported when sparse binding is not used). Not seen here: with the forced memfd, sparse binding worked on v2.0.14 and on v2.1.0 (a 54-minute Gaming Mode session on v2.1.0: larger pools are several, in a sparse buffer, 0 not imported lines). PR #2111 covers /dev/udmabuf permission errors, not this case: here the device opens fine.

Where in the game

Anywhere: the choice is made at start-up. Measured in gameplay; the forced-memfd run was on Death Mountain Trail.

How often

Every start from Steam in Gaming Mode.

Graphics card and driver version

AMD Ryzen Z1 Extreme (RADV PHOENIX), Mesa RADV, Vulkan 1.4, driver 0x6802003

CPU, RAM and operating system

ASUS ROG Ally X (Z1 Extreme), 24 GB (about 15.6 GB visible to Linux). Anatase (Fedora-based, immutable), Steam as a Flatpak (org.anatase.Steam), Gaming Mode on gamescope.

Changes from the defaults

RX_VK_VRAM_MB=3072; Render resolution 67%, Window scaling filter Lanczos, Frame generation 2x, Frame pacing on.

Other software hooked into the game

Steam's overlay and Fossilize layers and gamescope's WSI layer (Gaming Mode). MangoHud and LSFG-VK are installed as implicit layers; I did not check whether they loaded. They are the same in all four runs above.

hookshot_log.txt

Under Steam, the log goes to Steam's stderr (the journal), not to hookshot_log.txt; the relevant lines are quoted above. I can attach the full journal extract of each run if useful.

The analysis was done with AI assistance (Claude); the testing and recordings are mine.

### Hookshot version Measured on v2.0.14 (source f2796384); now on v2.1.0 (source e561602e), where the check below is unchanged (also on today's `main`). Both with a locally patched `hookshot_vk.so` (the `value_rec` fix from #997; unrelated to this). ### What kind of problem Performance ### What happened Started from Steam in Gaming Mode, the game runs at about half the frame rate it reaches in Desktop Mode on the same device and the same game folder. The log says why: ``` [hos host] mem: guest memory stays host-pointer imports for the GPU (slow submissions on AMD): udmabuf's size_limit_mb cannot be read [hos sead::MainThread] vk: guest memory as dma-bufs: no (the window is anonymous memory) ``` On this distribution, `steam` is the Flatpak Steam (`/usr/bin/steam` runs `flatpak run org.anatase.Steam`), so a game started from Steam runs inside Steam's Flatpak sandbox. The sandbox has `devices=all`, so `/dev/udmabuf` opens, but Flatpak does not expose `/sys/module`, so `/sys/module/udmabuf/parameters/size_limit_mb` cannot be read. In `host/hos_main.c` the memfd window is only used when that file reads a non-zero limit (`usable = limit_mb > 0`), so the game falls back to host-pointer imports. Outside Steam (Desktop Mode), the same file reads `64` and dma-bufs are used. Forcing the memfd with `HOS_GUEST_MEMFD=1` in `hookshot_env.txt` fixes it. The renderer already assumes 64 MB when it cannot read the limit (`gpu/vk/vk_memory.cpp`: `dmabuf_limit = 64ull << 20` before reading the file): ``` [hos host] mem: the guest window is a memfd's (12), shared with the GPU as dma-bufs [hos sead::MainThread] vk: guest memory as dma-bufs: yes [hos sead::MainThread] vk: dma-bufs of at most 64 MB (udmabuf's limit 64 MB); larger pools are several, in a sparse buffer ``` No pool failed to import in that run (0 `not imported` lines; the larger pools went in pieces into a sparse buffer). A 57 s Steam recording of that run (Death Mountain Trail, including a fight) shows no flashes or missing geometry. | Run | Memory for the GPU | Real fps in gameplay (median) | Slowest frame of 600 | |---|---|---|---| | Gaming Mode, run 1 | host-pointer imports | 13.4 | 95 ms | | Gaming Mode, run 2 | host-pointer imports | 12.3 | 145 ms | | Gaming Mode, `HOS_GUEST_MEMFD=1` | dma-bufs | **31.3** | 52 ms | | Desktop Mode (no Steam) | dma-bufs | 28.3 | 71 ms | "Real fps" is the replay's `presents, N fps` lines over frames with more than 300 draws; frame generation (2x) was on in all four, and these are the game's own frames. The two slow runs had Render resolution on Auto, the lighter setting on this GPU (it drops to its lowest size here); the forced-memfd run had it at 67%, and the Desktop Mode run started on Auto and was switched to 67% partway. So the slow runs did less GPU work (median GPU time 8.7 and 14.2 ms a frame, against 18.8 and 20.2 ms) and still ran at less than half the frame rate. **Suggested fix:** when `/dev/udmabuf` opens but `size_limit_mb` cannot be opened, assume the kernel's default (64 MB) as the renderer does, rather than falling back to host pointers. A file that exists and reads 0 still means udmabuf refuses everything, so that case stays as it is: ```c FILE *f = fopen("/sys/module/udmabuf/parameters/size_limit_mb", "r"); if (f) { if (fscanf(f, "%llu", &limit_mb) != 1) limit_mb = 0; fclose(f); } else { /* (a sandbox without /sys/module, such as Flatpak Steam's: udmabuf's default, as gpu/vk/vk_memory.cpp assumes) */ limit_mb = 64; hos_log("mem: udmabuf's size_limit_mb cannot be read (a sandbox?): assuming its default, 64 MB"); } usable = limit_mb > 0; ``` Possibly related: #2239 (pools over 64 MB not imported when sparse binding is not used). Not seen here: with the forced memfd, sparse binding worked on v2.0.14 and on v2.1.0 (a 54-minute Gaming Mode session on v2.1.0: `larger pools are several, in a sparse buffer`, 0 `not imported` lines). PR #2111 covers `/dev/udmabuf` permission errors, not this case: here the device opens fine. ### Where in the game Anywhere: the choice is made at start-up. Measured in gameplay; the forced-memfd run was on Death Mountain Trail. ### How often Every start from Steam in Gaming Mode. ### Graphics card and driver version AMD Ryzen Z1 Extreme (RADV PHOENIX), Mesa RADV, Vulkan 1.4, driver 0x6802003 ### CPU, RAM and operating system ASUS ROG Ally X (Z1 Extreme), 24 GB (about 15.6 GB visible to Linux). Anatase (Fedora-based, immutable), Steam as a Flatpak (`org.anatase.Steam`), Gaming Mode on gamescope. ### Changes from the defaults `RX_VK_VRAM_MB=3072`; Render resolution 67%, Window scaling filter Lanczos, Frame generation 2x, Frame pacing on. ### Other software hooked into the game Steam's overlay and Fossilize layers and gamescope's WSI layer (Gaming Mode). MangoHud and LSFG-VK are installed as implicit layers; I did not check whether they loaded. They are the same in all four runs above. ### hookshot_log.txt Under Steam, the log goes to Steam's stderr (the journal), not to `hookshot_log.txt`; the relevant lines are quoted above. I can attach the full journal extract of each run if useful. The analysis was done with AI assistance (Claude); the testing and recordings are mine.

Arkex-ai commented

2026-11-11 18:24:09 +00:00

(Migrated from github.com)

I've claimed the startup-selection investigation in #dev-work and reproduced the decision on current main (6df80a8d) with an authored C fixture: an accessible device plus missing sysfs selects anonymous memory; readable positive/zero/invalid limits and explicit overrides behave as expected. This is source-path evidence, not a reproduction of your FPS measurements.

Before changing automatic selection, I'm checking the host/renderer limit-policy mismatch and the separate memfd-discard report in #2411. I won't fold an unproven discard workaround into this fix.

Would you be able to test a candidate in the affected Flatpak setup once it's ready, with a current-version log? The useful checks would be actual dma-buf imports, explicit-off behavior and area transitions, rather than FPS alone. No game files or saves are needed.

I've claimed the startup-selection investigation in #dev-work and reproduced the decision on current main (`6df80a8d`) with an authored C fixture: an accessible device plus missing sysfs selects anonymous memory; readable positive/zero/invalid limits and explicit overrides behave as expected. This is source-path evidence, not a reproduction of your FPS measurements. Before changing automatic selection, I'm checking the host/renderer limit-policy mismatch and the separate memfd-discard report in #2411. I won't fold an unproven discard workaround into this fix. Would you be able to test a candidate in the affected Flatpak setup once it's ready, with a current-version log? The useful checks would be actual dma-buf imports, explicit-off behavior and area transitions, rather than FPS alone. No game files or saves are needed.

willcrisis commented

2026-11-11 19:51:27 +00:00

(Migrated from github.com)

Happy to test a candidate in the Flatpak Steam / Gaming Mode setup and report back with the log (dma-buf import lines, HOS_GUEST_MEMFD=0, and area transitions).

On #2411, a data point: I've run with the memfd window for about a day and a half of play (Desktop Mode by default, and Gaming Mode with HOS_GUEST_MEMFD=1), on v2.0.14 and v2.1.0, RADV PHOENIX, Mesa 26.2.3. That includes many area loads (Hyrule Field, Death Mountain Trail, Goron City). I saw no crash or hang after the intro.

Happy to test a candidate in the Flatpak Steam / Gaming Mode setup and report back with the log (dma-buf import lines, `HOS_GUEST_MEMFD=0`, and area transitions). On #2411, a data point: I've run with the memfd window for about a day and a half of play (Desktop Mode by default, and Gaming Mode with `HOS_GUEST_MEMFD=1`), on v2.0.14 and v2.1.0, RADV PHOENIX, Mesa 26.2.3. That includes many area loads (Hyrule Field, Death Mountain Trail, Goron City). I saw no crash or hang after the intro.

NotRexed commented

2026-11-11 21:29:59 +00:00

(Migrated from github.com)

Thanks for the report. Reports of the same problem are now collected in #1000, which lists this one with your details and links back here, so one fix closes them together and the open list stays readable. Follow #1000 for the fix, and add new details there (or here). If this turns out to be a separate problem it will be reopened. Pull request #2431 works on this.