Open the window on COSMIC's Wayland with SDL2 #1839

Merged

italloramon merged 2 commits from fix/wayland-libdecor-hang into main

2026-11-10 18:20:32 +00:00

italloramon commented

2026-11-10 17:55:07 +00:00

(Migrated from github.com)

On a COSMIC desktop (Pop!_OS) the game never opens its window on Wayland: it hangs at start-up with nothing drawn, and the only way in today is SDL_VIDEODRIVER=x11, which gives up the Wayland mailbox present that the game prefers. This makes it start on Wayland with no setting.

Reported in #992 and listed under Wayland in #1000.

Why it hangs

The hang is in the bundled SDL2 (2.32.10), before Hookshot touches the GPU: a small program that only opens an SDL2 window with these libraries hangs the same way, and HOS_HEADLESS=1 starts fine. The log stops after CountAddOnContent, with no vk: line, and the main thread waits in poll() on the Wayland socket forever.

SDL2 2.32 asks for the compositor's decorations (xdg-decoration) before the window's first configure, then waits for that configure in Wayland_ShowWindow on shell_surface.xdg.initial_configure_seen.
COSMIC answers CLIENT_SIDE before the configure. SDL's handler (handle_configure_zxdg_decoration) then hides the window, clears shell_surface and shows it again with libdecor, all inside that wait.
The inner show finishes, but shell_surface is a union: the outer loop still reads the xdg member's initial_configure_seen, which the switch cleared and which libdecor never sets. The outer loop dispatches forever.

SDL2 2.30 asked for decorations only after that wait, so the same switch did not hang. The SDL2 branch still has this code. SDL3 waits on one shell_surface_status instead and is not affected.

GNOME and weston do not offer xdg-decoration, and sway, KDE and Hyprland answer server-side, so none of them reaches the switch. That is why the reports so far are COSMIC only: a GNOME + NVIDIA laptop in #1000 opens directly. Hybrid GPUs are not the cause.

What changed

host/posix/display_sdl.c: with SDL2, hos_display_init sets SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=1 before SDL_Init, so SDL uses libdecor from the start and never makes the switch. libdecor still takes the compositor's decorations where they are offered. Without libdecor, SDL keeps its own path. SDL3 is left alone, and the environment variable still wins (SDL2 does not let a hint at normal priority override it), so SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=0 restores the old behaviour.
tests/test_wayland_libdecor_hint.py (RUN_ALL_ARGS names the compiler for tests/run_all.py): compiles the actual hos_display_init with SDL stubs and checks that the hint is set before SDL_Init for SDL2, and not for SDL3 or a headless start. It fails on main and passes here.

How it was tested

Pop!_OS 24.04, COSMIC (cosmic-comp 0.1, Wayland), Intel Raptor Lake iGPU + RTX 3050 6 GB Laptop, NVIDIA 595.104.02, libdecor 0.2.2 with its GTK plugin.

In the game: I rebuilt v2.0.13 with only this change (the v2.0.13 AppImage's tools plus this patch, --update), with no variables set.

	v2.0.13	v2.0.13 + this change
COSMIC, Wayland	hangs at start-up (32 log lines, no renderer)	title screen, `rx: renderer: Vulkan`
the same, `SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=0`	hangs	hangs (the variable wins)

On other compositors: I opened a window with the same flags as the game (SHOWN \| RESIZABLE \| ALLOW_HIGHDPI \| VULKAN) through the bundled libSDL2-2.0.so.0, with each compositor nested in COSMIC, and read the decoration handshake with WAYLAND_DEBUG=client.

Compositor	Before	After	Decorations, both before and after
COSMIC	hangs	opens	client (libdecor; COSMIC asks for it)
GNOME Shell 46	opens	opens	client (no `xdg-decoration`)
sway 1.9	opens	opens	server (`configure(2)`, no client frame)
weston 13	opens	opens	client (no `xdg-decoration`)
COSMIC / GNOME with libdecor hidden	opens, no frame	opens, no frame	none

Only COSMIC changes. On the others the window and who draws its frame are the same before and after.

python -I -B tests/run_all.py: the window checks pass (test_wayland_libdecor_hint, test_display_startup_failure, test_windows_display_startup). Ten checks fail on this machine on main too, because it has no clang++. They do not touch this code.

Screenshot

The title screen on COSMIC's Wayland after the change (before it, the window never drew anything):

🤖 Generated with Claude Code

On a COSMIC desktop (Pop!_OS) the game never opens its window on Wayland: it hangs at start-up with nothing drawn, and the only way in today is `SDL_VIDEODRIVER=x11`, which gives up the Wayland mailbox present that the game prefers. This makes it start on Wayland with no setting. Reported in #992 and listed under Wayland in #1000. ## Why it hangs The hang is in the bundled SDL2 (2.32.10), before Hookshot touches the GPU: a small program that only opens an SDL2 window with these libraries hangs the same way, and `HOS_HEADLESS=1` starts fine. The log stops after `CountAddOnContent`, with no `vk:` line, and the main thread waits in `poll()` on the Wayland socket forever. 1. SDL2 2.32 asks for the compositor's decorations (`xdg-decoration`) before the window's first `configure`, then waits for that configure in `Wayland_ShowWindow` on `shell_surface.xdg.initial_configure_seen`. 2. COSMIC answers `CLIENT_SIDE` before the configure. SDL's handler (`handle_configure_zxdg_decoration`) then hides the window, clears `shell_surface` and shows it again with libdecor, all inside that wait. 3. The inner show finishes, but `shell_surface` is a union: the outer loop still reads the `xdg` member's `initial_configure_seen`, which the switch cleared and which libdecor never sets. The outer loop dispatches forever. SDL2 2.30 asked for decorations only after that wait, so the same switch did not hang. The SDL2 branch still has this code. SDL3 waits on one `shell_surface_status` instead and is not affected. GNOME and weston do not offer `xdg-decoration`, and sway, KDE and Hyprland answer server-side, so none of them reaches the switch. That is why the reports so far are COSMIC only: a GNOME + NVIDIA laptop in #1000 opens directly. Hybrid GPUs are not the cause. ## What changed - `host/posix/display_sdl.c`: with SDL2, `hos_display_init` sets `SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=1` before `SDL_Init`, so SDL uses libdecor from the start and never makes the switch. libdecor still takes the compositor's decorations where they are offered. Without libdecor, SDL keeps its own path. SDL3 is left alone, and the environment variable still wins (SDL2 does not let a hint at normal priority override it), so `SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=0` restores the old behaviour. - `tests/test_wayland_libdecor_hint.py` (`RUN_ALL_ARGS` names the compiler for `tests/run_all.py`): compiles the actual `hos_display_init` with SDL stubs and checks that the hint is set before `SDL_Init` for SDL2, and not for SDL3 or a headless start. It fails on `main` and passes here. ## How it was tested Pop!_OS 24.04, COSMIC (cosmic-comp 0.1, Wayland), Intel Raptor Lake iGPU + RTX 3050 6 GB Laptop, NVIDIA 595.104.02, libdecor 0.2.2 with its GTK plugin. **In the game:** I rebuilt v2.0.13 with only this change (the v2.0.13 AppImage's tools plus this patch, `--update`), with no variables set. | | v2.0.13 | v2.0.13 + this change | |---|---|---| | COSMIC, Wayland | hangs at start-up (32 log lines, no renderer) | title screen, `rx: renderer: Vulkan` | | the same, `SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=0` | hangs | hangs (the variable wins) | **On other compositors:** I opened a window with the same flags as the game (`SHOWN \| RESIZABLE \| ALLOW_HIGHDPI \| VULKAN`) through the bundled `libSDL2-2.0.so.0`, with each compositor nested in COSMIC, and read the decoration handshake with `WAYLAND_DEBUG=client`. | Compositor | Before | After | Decorations, both before and after | |---|---|---|---| | COSMIC | **hangs** | opens | client (libdecor; COSMIC asks for it) | | GNOME Shell 46 | opens | opens | client (no `xdg-decoration`) | | sway 1.9 | opens | opens | server (`configure(2)`, no client frame) | | weston 13 | opens | opens | client (no `xdg-decoration`) | | COSMIC / GNOME with libdecor hidden | opens, no frame | opens, no frame | none | Only COSMIC changes. On the others the window and who draws its frame are the same before and after. `python -I -B tests/run_all.py`: the window checks pass (`test_wayland_libdecor_hint`, `test_display_startup_failure`, `test_windows_display_startup`). Ten checks fail on this machine on `main` too, because it has no `clang++`. They do not touch this code. ## Screenshot The title screen on COSMIC's Wayland after the change (before it, the window never drew anything): ![after: title screen on COSMIC Wayland](https://github.com/italloramon/Hookshot/raw/pr-assets/wayland-libdecor/after-cosmic-title.png) 🤖 Generated with [Claude Code](https://claude.com/claude-code)

hckrman101 (Migrated from github.com) approved these changes

2026-11-10 18:12:03 +00:00

hckrman101 (Migrated from github.com) left a comment

Looks good. Setting SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=1 only for SDL2, before SDL_Init, is a small and well-scoped way around the 2.32 xdg-decoration/initial_configure hang.

Other platforms: the hint is only read by SDL2's Wayland backend, so X11 and Windows are unaffected. Sway, KDE and Hyprland answer server-side, and GNOME and weston don't offer xdg-decoration, so none of them reaches the switch. It's set via hos_sdl_hint (normal priority), so an env value of SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=0 still wins, and without libdecor SDL keeps its own path. SDL3 is left alone. There's no COSMIC detection to get wrong, which I like. The nested-compositor table in the description covers the other cases well.

The test compiles the real hos_display_init with stubs and checks the hint is set early for SDL2 and not for SDL3 or headless. It declares RUN_ALL_ARGS, so it plugs into the new run_all layout. I didn't build or run it.

The fork branch has no CI runs, so a merge of main (or a push) would give CI a chance to run before this goes in, but nothing here conflicts.

Looks good. Setting SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=1 only for SDL2, before SDL_Init, is a small and well-scoped way around the 2.32 xdg-decoration/initial_configure hang. Other platforms: the hint is only read by SDL2's Wayland backend, so X11 and Windows are unaffected. Sway, KDE and Hyprland answer server-side, and GNOME and weston don't offer xdg-decoration, so none of them reaches the switch. It's set via hos_sdl_hint (normal priority), so an env value of SDL_VIDEO_WAYLAND_PREFER_LIBDECOR=0 still wins, and without libdecor SDL keeps its own path. SDL3 is left alone. There's no COSMIC detection to get wrong, which I like. The nested-compositor table in the description covers the other cases well. The test compiles the real hos_display_init with stubs and checks the hint is set early for SDL2 and not for SDL3 or headless. It declares RUN_ALL_ARGS, so it plugs into the new run_all layout. I didn't build or run it. The fork branch has no CI runs, so a merge of main (or a push) would give CI a chance to run before this goes in, but nothing here conflicts.