Crash under Xorg when built with Wayland support

[jbeich]

2b30756 (#225) unlike 18a63a2 (#109) still doesn't support building for both X11 and Wayland.

$ qmake5 -config release DEFINES+="X11 WAYLAND"
$ make
$ sudo make install

$ env -u WAYLAND_DISPLAY XDG_SESSION_TYPE=x11 vulkanCapsViewer
Segmentation fault

Process 1733 stopped
* thread #1, name = 'vulkanCapsViewer', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x40)
    frame #0: 0x0000000822e70adc libwayland-client.so.0`wl_proxy_get_version(proxy=0x0000000000000000) at wayland-client.c:2317:16
   2314 WL_EXPORT uint32_t
   2315 wl_proxy_get_version(struct wl_proxy *proxy)
   2316 {
-> 2317         return proxy->version;
                              ^
   2318 }
   2319
   2320 /** Get the id of a proxy object **/
(lldb) bt
* thread #1, name = 'vulkanCapsViewer', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x40)
  * frame #0: 0x0000000822e70adc libwayland-client.so.0`wl_proxy_get_version(proxy=0x0000000000000000) at wayland-client.c:2317:16
    frame #1: 0x00000000002fbeed vulkanCapsViewer`wl_display_get_registry(wl_display=0x0000000000000000) at wayland-client-protocol.h:1109:54
    frame #2: 0x00000000002f6fbd vulkanCapsViewer`VulkanCapsViewer::initVulkan(this=0x0000000820c3b0c0) at vulkancapsviewer.cpp:816:43
    frame #3: 0x00000000002f527f vulkanCapsViewer`VulkanCapsViewer::VulkanCapsViewer(this=0x0000000820c3b0c0, parent=0x0000000000000000) at vulkancapsviewer.cpp:234:10
    frame #4: 0x00000000002a81c6 vulkanCapsViewer`main(argc=1, argv=0x0000000820c3b988) at main.cpp:115:22
    frame #5: 0x000000082a641325 libc.so.7`__libc_start1(argc=1, argv=0x0000000820c3b988, env=0x0000000820c3b998, cleanup=<unavailable>, mainX=(vulkanCapsViewer`main at main.cpp:60)) at libc_start1.c:172:7
    frame #6: 0x00000000002a5890 vulkanCapsViewer`_start at crt1_s.S:83
(lldb) f 1
frame #1: 0x00000000002fbeed vulkanCapsViewer`wl_display_get_registry(wl_display=0x0000000000000000) at wayland-client-protocol.h:1109:54
   1106         struct wl_proxy *registry;
   1107
   1108         registry = wl_proxy_marshal_flags((struct wl_proxy *) wl_display,
-> 1109                          WL_DISPLAY_GET_REGISTRY, &wl_registry_interface, wl_proxy_get_version((struct wl_proxy *) wl_display), 0, NULL);
                                                                                  ^
   1110
   1111         return (struct wl_registry *) registry;
   1112 }
(lldb) f 2
frame #2: 0x00000000002f6fbd vulkanCapsViewer`VulkanCapsViewer::initVulkan(this=0x00000008212d67d0) at vulkancapsviewer.cpp:816:43
   813              };
   814
   815              const auto wayland_display = wl_display_connect(nullptr);
-> 816              const auto wayland_registry = wl_display_get_registry(wayland_display);
                                                  ^
   817              uint32_t wayland_compositor_name = 0;
   818              wl_registry_add_listener(wayland_registry, &WAYLAND_LISTENER, &wayland_compositor_name);
   819              wl_display_roundtrip(wayland_display);

[SaschaWillems]

I'm not much of a Linux expert. Can you elaborate a bit? Is this a general problem or something specific to a certain setup?

[HildarTheDorf]

Guessing the driver still reports support for KHR_wayland_surface despite there being no Wayland compositor at all.

Should just be a case of checking if the wl_display is null. I can take a look tomorrow.

[jbeich]

Is this a general problem or something specific to a certain setup?

Common to distros following "batteries included" policy in order for packages to be useful for the largest set of users. Very few distros are Xorg-only i.e., build Vulkan-Loader without Wayland WSI support.

still reports support for KHR_wayland_surface

Ditto KHR_xcb_surface but it doesn't crash likely due to probing order i.e., no need to check KHR_xcb_surface when KHR_wayland_surface is detected.

$ env -u WAYLAND_DISPLAY vulkaninfo | fgrep wayland
        VK_KHR_wayland_surface                 : extension revision 6
$ vulkaninfo | fgrep -B1 wayland
        VK_KHR_wayland_surface                 : extension revision 6
--
GPU id : 0 (Intel(R) HD Graphics 530 (SKL GT2)):
        Surface type = VK_KHR_wayland_surface
--
GPU id : 1 (llvmpipe (LLVM 15.0.7, 256 bits)):
        Surface type = VK_KHR_wayland_surface

$ env -u DISPLAY vulkaninfo | fgrep xcb
        VK_KHR_xcb_surface                     : extension revision 6
$ vulkaninfo | fgrep -C2 xcb
        VK_KHR_xcb_surface                     : extension revision 6
        VK_KHR_xlib_surface                    : extension revision 6
--
GPU id : 0 (Intel(R) HD Graphics 530 (SKL GT2)):
        Surface types: count = 2
                VK_KHR_xcb_surface
                VK_KHR_xlib_surface
--
GPU id : 1 (llvmpipe (LLVM 15.0.7, 256 bits)):
        Surface types: count = 2
                VK_KHR_xcb_surface
                VK_KHR_xlib_surface

Should just be a case of checking if the wl_display is null.

I did something similar downstream.