X11 denial of service by client crash

[zougloub]

Describe the bug

I've been running some program (https://www.st.com/en/development-tools/stm32cubeide.html) which sometimes crashes (the crash is unrelated to X) and this somehow causes a denial of service on the X11 interface.

Further X11 clients will hang trying to connect to the X socket:

connect(3, {sa_family=AF_UNIX, sun_path=@"/tmp/.X11-unix/X0"}

Which belongs to wayfire, which is otherwise responsive.

For restoring X11 functionality, I have no other workaround except restarting wayfire.

To Reproduce

Reproduction is not easy, I don't know what can cause the crash yet.

Expected behavior

• If a program crashes, X11 is still available.
• If an X11 server dies, maybe it can restart

Wayfire version
0.9.0

[ammen99]

There should be an automatic restart of the Xwayland process, unless the Xwayland process itself hangs (which seems likely based on your description). If that is correct, then the bug is a Xwayland bug, but you can kill the Xwayland process and Wayfire will start a new one. Any of the existing Xwayland clients will be killed though, but at least wayland-native clients will survive.

[zougloub]

Oh, I see that Xwayland is gone (but silently and I didn't get a core dump), somehow it didn't come back.

[mark-herbert42]

Have the same with SAPGui 7.80 which is java application mostly. If it crashes - Xwayland is gone and never come back. If I just kill Xwayland using kill command it restarts without any issue.

[ammen99]

It might be worth trying another wlroots-based compositor with the same wlroots version. Wlroots is actually in charge of tracking Xwayland and when to restart it and Wayfire has few things it can do to influence this behavior, so I suspect the bug is in wlroots. If you compile from scratch, you might also want to try wlroots 0.18 with this PR for Wayfire: #2452

[zougloub]

Quickly tried with merged track-wlroots over master (0.10.0-75430ce1 (Sep 28 2024) branch HEAD wlroots-0.18.0) and the issue still happens.

[soreau]

It seems to me that Xwayland is dying in a way that wlroots does not destroy the socket and fails to restart Xwayland. If you run with wayfire -d wlroots &> /tmp/wayfire.log and reproduce the problem, do you get any interesting (error) messages?

[zougloub]

Reproduced with the logging and we have:

EE 29-09-24 22:52:02.883 - [xwayland/xwm.c:1192] Failed to get window property                                                                                                                           
EE 29-09-24 22:52:02.883 - [xwayland/xwm.c:1192] Failed to get window property                                                                                                                           
DD 29-09-24 22:52:02.884 - [xwayland/xwm.c:1662] unhandled X11 event: 10                    
EE 29-09-24 22:52:02.885 - [xwayland/xwm.c:1641] xcb error: op 12:0, code 3, sequence 5010, value 8393241  
EE 29-09-24 22:52:02.885 - [xwayland/xwm.c:1641] xcb error: op 25:0, code 3, sequence 5011, value 8393241
EE 29-09-24 22:52:02.885 - [xwayland/xwm.c:1641] xcb error: op 18:0, code 3, sequence 5013, value 8393241
DD 29-09-24 22:52:02.886 - [xwayland/xwm.c:1662] unhandled X11 event: 10                    
EE 29-09-24 22:52:02.888 - [xwayland/xwm.c:1641] xcb error: op 18:0, code 3, sequence 5020, value 8393241
EE 29-09-24 22:52:02.888 - [xwayland/xwm.c:1641] xcb error: op 19:0, code 3, sequence 5021, value 8393241  
EE 29-09-24 22:52:02.888 - [xwayland/xwm.c:1641] xcb error: op 18:0, code 3, sequence 5023, value 8393241
EE 29-09-24 22:52:02.888 - [xwayland/xwm.c:1641] xcb error: op 19:0, code 3, sequence 5024, value 8393241
DD 29-09-24 22:52:03.316 - [xwayland/xwm.c:887] unhandled X11 property 300 (XdndAware) for window 8388623  
DD 29-09-24 22:52:03.316 - [xwayland/xwm.c:887] unhandled X11 property 307 (XdndProxy) for window 8388623      
DD 29-09-24 22:52:03.316 - [xwayland/xwm.c:887] unhandled X11 property 300 (XdndAware) for window 8393219        
DD 29-09-24 22:52:03.317 - [xwayland/xwm.c:887] unhandled X11 property 307 (XdndProxy) for window 8393219
DD 29-09-24 22:52:03.318 - [xwayland/xwm.c:887] unhandled X11 property 539 (_SUNW_JAVA_AWT_TIME) for window 8388623
DD 29-09-24 22:52:03.318 - [xwayland/xwm.c:887] unhandled X11 property 539 (_SUNW_JAVA_AWT_TIME) for window 8388623
DD 29-09-24 22:52:03.318 - [xwayland/xwm.c:887] unhandled X11 property 539 (_SUNW_JAVA_AWT_TIME) for window 8388623
DD 29-09-24 22:52:03.335 - [xwayland/xwm.c:887] unhandled X11 property 539 (_SUNW_JAVA_AWT_TIME) for window 8388623
DD 29-09-24 22:52:03.346 - [xwayland/xwm.c:887] unhandled X11 property 539 (_SUNW_JAVA_AWT_TIME) for window 8388623          
DD 29-09-24 22:52:03.346 - [xwayland/xwm.c:887] unhandled X11 property 539 (_SUNW_JAVA_AWT_TIME) for window 8388623
DD 29-09-24 22:52:03.347 - [xwayland/xwm.c:887] unhandled X11 property 539 (_SUNW_JAVA_AWT_TIME) for window 8388623
DD 29-09-24 22:52:03.348 - [xwayland/xwm.c:887] unhandled X11 property 539 (_SUNW_JAVA_AWT_TIME) for window 8388623
DD 29-09-24 22:52:03.422 - [xwayland/xwm.c:887] unhandled X11 property 325 (_NET_WM_ICON_NAME) for window 6295490
DD 29-09-24 22:52:03.423 - [xwayland/xwm.c:887] unhandled X11 property 37 (WM_ICON_NAME) for window 6295490
II 29-09-24 22:52:08.863 - [wayland] file descriptor expected, object (4), message create_pool(nhi)
II 29-09-24 22:52:08.863 - [wayland] error in client communication (pid 1642)
II 29-09-24 22:52:08.863 - [xwayland/server.c:217] Restarting Xwayland
EE 29-09-24 22:52:08.863 - [xwayland/server.c:314] socketpair failed: Too many open files
XWAYLAND: wl_display#1: error 1: invalid arguments for wl_shm#4.create_pool
(EE) failed to dispatch Wayland events: Invalid argument

So I see that create_pool issue, and the "too many open files" which I can check further.

[soreau]

I googled the message "[wayland] file descriptor expected, object (4), message create_pool(nhi)" and found quite a few relevant similar reports. As far as I can tell, you'll want Xwayland with this commit. So, the problem might be fixed by upgrading Xwayland. I do not believe this to be a wayfire issue since this bug affects all different types of compositors with Xwayland, so closing for now.

[zougloub]

The number of open files by wayfire is indeed increasing uncontrollably:

lrwx------ 1 cJ cJ 64 Sep 30 09:07 990 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 991 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 992 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 993 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 994 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 995 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 996 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 997 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 998 -> '/memfd:xwayland-shared (deleted)'*
lrwx------ 1 cJ cJ 64 Sep 30 09:07 999 -> '/memfd:xwayland-shared (deleted)'*

When using my killer program.

It's wlroots that can't reopen Xwayland due to max open fds.

I am running a recent Xwayland that includes the above patch.

[soreau]

@zougloub I found this sway commit which indicates that wayfire might want to raise the limit as well. So I am reopening this issue.

I will work on a patch, but is there any way you can test to see if this issue happens in sway?

[zougloub]

Note, in this case, I looks like that X client manages to raise, forever, the amount of fds (I found a way to get 100 fds open per second), and the number is not going down until I quit the program ; raising the limit may just be pushing the inevitable further.

It also means that I have found a workaround, I can watch the number of fds and notify when it's dangerously high, then I can close and reopen my program, and the number of open fds in Xwayland and wayfire has gone down, and all other X clients don't need to be killed.

I'll see about testing in sway.

[soreau]

Also, do you have latest Xwayland? What is the output of pkg-config --modversion xwayland?

[soreau]

Here is a patch to set max open files limit for wayfire, even though as you said, it might only buy you more time.

[ammen99]

This sounds like a bug in Xwayland or in wlroots tbh, though what Sway has will probably be useful in Wayfire too anyway, for cases where there are many open windows, not in your case.

[zougloub]

xwayland 24.1.2

Be back soon with sway results.

[zougloub]

Sadly the program doesn't render correctly under sway, so I can't reproduce the problem there.

[ammen99]

Sadly the program doesn't render correctly under sway, so I can't reproduce the problem there.

To me this indicates even more strongly that the bug is in the Xwayland server. I would try weston as well, or gnome-wayland, my guess is you will get the same issue, then you can report it to the Xwayland developers :/

[mark-herbert42]

Installed xwayland from git so latest of the latest, nothing changes from 24.1.3

Latest SAP GUI 8.10 for Linux/java. It does survive for 5 minutes max before crash and xwayland can not be restarted after that crash. Only complete wayfire restart helps. No coredumps or something like that, nothing in dmesg or journalctl