WSS port stops receiving/listening....

[jerry6021]

I have cloned the latest 1.13.9

I have a FS env with 8 different physical phones, grandstream/yealin/poly/etc that don't use the WSS port and they all don't have any issues with signalling or audio.

I use ctxSIP (SIP.js 0.7.8) as well as my own WebPhone (SIP.js 0.20.0). Both work perfectly 90% of the time.
Both use the WSS port - this is where the issue appears to be...

After a day of testing with 2 WebPhones constantly dialing, listening to MOH for 5 seconds, hangup and repeat, it seems to be the WSS port stops reading/listening.

I set SIP.js keepalives to 1 second.
I have TPORT_DEBUG=9
I have SOFIA_DEBUG=9
I have tcpdump in one window, watching for traffic on the WSS port - it never stops
I have fs_cli opened with and it comes to a complete stop on the receiving side.
After about 10-15 seconds, the receiving continues...

Looks like the receiving port gets blocked for a short bit then possibly timeout while reading and continues...

Here is the fs_cli output...

tport.c:3528 tport_send_msg() tport_vsend returned 731
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac802dd750 destroy wss transport 0x7fac802dd940.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac80323480 destroy wss transport 0x7fac80323670.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac80369310 destroy wss transport 0x7fac80369500.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac803af5d0 destroy wss transport 0x7fac803af7c0.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac803f5570 destroy wss transport 0x7fac803f5760.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac80aaa820 destroy wss transport 0x7fac80aaaa10.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=0
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac8043f2f0 destroy wss transport 0x7fac8043f4e0.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac80af3e50 destroy wss transport 0x7fac80af4040.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac80101c30 destroy wss transport 0x7fac80101e20.
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac805aa250 destroy wss transport 0x7fac805aa440.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac80633140 destroy wss transport 0x7fac80633330.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 1
ws.c:926 ws_close() ssl_error: = 1-SSL_ERROR_SSL
ws.c:937 ws_close() ssl_error: SSL_ERROR_SSL = YES errno=0
ws.c:938 ws_close() ws_close() errno=Operation not permitted
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac808e9d90 destroy wss transport 0x7fac808e9f80.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac808b9ab0 destroy wss transport 0x7fac808b9ca0.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport_type_ws.c:530 tport_ws_deinit_secondary() 0x7fac809b27a0 destroy wss transport 0x7fac809b2990.
ws.c:889 ws_close() wsh_ssl: 1
ws.c:901 ws_close() code: = -1
ws.c:902 ws_close() ssl_error: = 5
ws.c:924 ws_close() ssl_error: = 5-SSL_ERROR_SYSCALL
ws.c:931 ws_close() ssl_error: SSL_ERROR_SYSCALL = YES errno=9
ws.c:932 ws_close() ws_close() errno=Input/output error
tport.c:2777 tport_wakeup_pri() tport_wakeup_pri(0x7fac80006290): events IN
tport.c:2900 tport_recv_event() tport_recv_event(0x7fac80006290)

I added custom debug messages to ws_close to spit out those extra messages when things go wrong.

[jerry6021]

Forgot 1 thing - the physical phones do not show any issues even when the WebPhones over WSS are stuck...

[huongbn]

I have same issue with you, I'm searching to downgrade to 1.13.7 but not success

[jerry6021]

I have been able to reproduce the issue a little more reliably tonight.

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 1 0 A.B.C.D:7443 0.0.0.0:* LISTEN 12462/freeswitch

The Recv-Q gets to 1 plus and the websocket thread stops... I have to restart FS to correct.
The pure SIP based phones continue to work without and noticable impact.

the webphones, using WSS are done - FS is simply not receiving the packets.

I'm wondering if there is some buffer overflow happening... That'll have to wait a few days

[jerry6021]

A moment ago, FS stopped talking to the IPv4 clients (5060/7080/7443)

The ESL (8021) and the IPv6 ports seem OK.

I got some 1006 errors on the javascript side...

I waited 10 minutes but it did not correct itself.

Wed 05 Oct 2022 09:53:37 AM EDT
Proto Recv-Q Send-Q Local Address   Foreign Address         State       PID/Program name
tcp       27      0 [IP v4]:7080    0.0.0.0:*               LISTEN      25442/freeswitch
tcp       15      0 [IP v4]:5060    0.0.0.0:*               LISTEN      25442/freeswitch
tcp       65      0 [IP v4]:7443    0.0.0.0:*               LISTEN      25442/freeswitch
tcp        0      0 0.0.0.0:8021    0.0.0.0:*               LISTEN      25442/freeswitch
tcp6       0      0 [IP V6]:5060    :::*                    LISTEN      25442/freeswitch
tcp6       0      0 [IP V6]:7443    :::*                    LISTEN      25442/freeswitch

This is with the latest 1.13.9

[jerry6021]

does anyone have FS 1.10.6 with the latest libsofia-sip-au and ctxSIP working over WSS ?

[huongbn]

@jerry6021 can you fix this issue

[dilyanpalauzov]

#163 is about leaking websockets.

[themsley-voiceflex]

See PR #231 and issue #2283 for the reason why and my fix.

[themsley-voiceflex]

@andywolk this one can probably be closed too

[andywolk]

Fixed by #233