Socket.io disconnect with ping timeout randomly

[jitendrapawar]

we have a real time chat application, that uses socket.io, node.js, mongodb etc. Chat is working very fine except one issue & that is as follows

Some time in between while chatting user are getting disconnected with ping timeout. As I checked there was no problem with internet connection & also there is no logs on re-connection attempts. It directly gets disconnected.

OS - Ubuntu 14.04/AWS EC2
socket.io version on server - 1.6.0
Node version - v0.10.25

Please let us know what could be the problem. Also let me know if you need any other details

[dashcraft]

Have you tried listening to any emitted error events?
Here is a stack overflow article that contains a list of common events emitted by the socket..

http://stackoverflow.com/questions/24224287/list-of-socket-io-events

[avihaym]

I have the same problem did you figure this out ?

[darrachequesne]

@jitendrapawar are you able to reproduce consistently?

[jitendrapawar]

No it's not repeatedly, it's randomly getting disconnected with ping timeout.

…

On Jan 10, 2017 7:50 PM, "Damien Arrachequesne" ***@***.***> wrote: @jitendrapawar <https://github.com/jitendrapawar> are you able to reproduce consistently? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2769 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANP7JpiqxkTpZlc5CsayjIZf9GAqURLiks5rQ5OsgaJpZM4K9hrg> .

[darrachequesne]

• Do you send binary data? (Support for sending big messages #2666)
• Does it timeout after a given delay (could be proxy-related)?
• Is it related to a given browser?
• Are they properly reconnecting after the ping timeout?

[arctelix]

Same issue here. No other errors are emitted. Seems to be totally random.

[chengsu]

I disable the network, and enable it immediately. About 1 minute later, the server gets a 'ping timeout' problem.

[plh97]

  pingInterval: 25000,
  pingTimeout: 60000,

author said that: a client may have to wait up to pingTimeout + pingInterval ms before getting a disconnect event.
but i found while i F5 the webpage , i will get a new socket.id, am i disconnected ????i refresh my page less than 1s,
i can;t found this how to work

[jstelz]

I have just come across this issue as well. Our app has run fine without these disconnects for over a year, but now experiencing random disconnects. In debug, reason for disconnect is "ping timeout" and clients seem to have no issue reconnecting.

The only thing I can possibly narrow it down to so far is a new version of Google Chrome. Chrome 63 seems to have no issues, but users on 64 are the ones experiencing the disconnects and I have no idea why

[vladluca]

I'm facing the exact same problem and I'm on Chrome 63 @jstelz. Does anyone found the problem or a solution?

[darrachequesne]

That may be linked to #5082.

[sirudog]

@darrachequesne In my case it is definitely linked to the issue of client-side timers throttled in background tabs (at least in chrome, I have not yet tested it in other browsers).
I see the same symptoms as mentioned above by the others and it is very easy to reproduce. I just have to start my web app, make sure it connects, and open another tab to force my app running in the background. Eventually the timers will be throttled by chrome resulting in ping timeouts, which in the end will make the server disconnect my app as it thinks the app is not alive anymore.
There is one possible way of disabling this throttling in chrome => https://www.tenforums.com/tutorials/80233-enable-disable-google-chrome-background-tab-throttling-windows.html
But this is not perfect and sometimes not even applicable. Probably the only good solution would be what was also done in primus lib: to Reverse the direction of ping/pong messages (primus/primus#534)
This seems a rather big refactoring in engine.io affecting both server and client libs though...

[darrachequesne]

@sirudog well, that's quite a big breaking change! I think we'll have to do that though...

[Salzian]

I have this problem with a game I'm working on.

When I set the server to
pingInterval: 1000, pingTimeout: 500
my client sends out about every second a ping timeout and reconnects.

When I set the server to
pingInterval: 500, pingTimeout: 1000
I don't get timeouts but the server never detects a disconnect.

[RosenTomov]

For anyone suffering random timeouts, try changing the default engine.io "transport upgrade timeout" (upgradeTimeout).

After reconnecting and receiving large amount of data, the socket seemed unresponsive, debugging engine.io didn't show any errors, just the socket trying to upgrade transport every time the client reconnects.

example server:

// index.js
import io from 'socket.io';
import expressApp from './express/expressApp'; // express server
import socket from './socket/socket';

const port = 4000;
expressApp.set('port', process.env.Port || port);

const server = expressApp.listen(port, 'localhost');

const socketIo = io(server, {
  upgradeTimeout: 30000 // default value is 10000ms, try changing it to 20k or more
});

socket(socketIo);

// socket.js
export default socketIo =>
  socketIo.on('connection', socket => {
  //...return large amount of data
  }
}

You can read more about engine.io here:
https://github.com/socketio/engine.io

[kllr]

hi

For anyone suffering random timeouts, try changing the default engine.io "transport upgrade timeout" (upgradeTimeout).

After reconnecting and receiving large amount of data, the socket seemed unresponsive, debugging engine.io didn't show any errors, just the socket trying to upgrade transport every time the client reconnects.

example server:

// index.js
import io from 'socket.io';
import expressApp from './express/expressApp'; // express server
import socket from './socket/socket';

const port = 4000;
expressApp.set('port', process.env.Port || port);

const server = expressApp.listen(port, 'localhost');

const socketIo = io(server, {
  upgradeTimeout: 30000 // default value is 10000ms, try changing it to 20k or more
});

socket(socketIo);

// socket.js
export default socketIo =>
  socketIo.on('connection', socket => {
  //...return large amount of data
  }
}

You can read more about engine.io here:
https://github.com/socketio/engine.io

Hi,

I tried it but got still many disconnects:

15:10:57,270 urbanGUI INFO connected
15:10:57,549 urbanGUI INFO Client disconnected: 6830bd090575426f9c53dec108702e04
15:10:59,70 urbanGUI INFO connected
15:10:59,342 urbanGUI INFO Client disconnected: e2eb60d8a2aa445f94a17b6c6fc00e34
15:11:00,770 urbanGUI INFO connected
15:11:01,41 urbanGUI INFO Client disconnected: 62b2619d3b3a4c05aa9cfa0ad3df9524
15:11:02,655 urbanGUI INFO connected
15:11:02,940 urbanGUI INFO Client disconnected: a421da49b04c46f38c1ae8c08b14e5db
15:11:04,153 urbanGUI INFO connected
15:11:04,421 urbanGUI INFO Client disconnected: b935f53d393c42488411e2b6c3e5961a
15:11:05,359 urbanGUI INFO connected
15:11:05,630 urbanGUI INFO Client disconnected: ee8a42bb8b0e4bc8b9e813dd808c1dc3
15:11:07,232 urbanGUI INFO connected
15:11:07,513 urbanGUI INFO Client disconnected: 5c9804841d0a438e87536977c222c630
15:11:08,703 urbanGUI INFO connected
15:11:08,981 urbanGUI INFO Client disconnected: a5dfcc72eb5c4445b09fb28153220961
15:11:10,601 urbanGUI INFO connected
15:11:10,893 urbanGUI INFO Client disconnected: 394e409a10fa41a8aa7e7c747b4ea606
15:11:12,87 urbanGUI INFO connected
15:11:12,360 urbanGUI INFO Client disconnected: e83bacd1338d4306b8eaccc9597f7a0b
15:11:13,384 urbanGUI INFO connected
15:11:13,661 urbanGUI INFO Client disconnected: 52cf363b5d26431a8b9b6459551aedcd
15:11:14,389 urbanGUI INFO connected
15:11:14,664 urbanGUI INFO Client disconnected: f254c71fbd374cac9bd3d4161e53456f
15:11:15,995 urbanGUI INFO connected
15:11:16,275 urbanGUI INFO Client disconnected: ec7d67d7b4a04019b3496dcf40e0d41a
15:11:17,709 urbanGUI INFO connected
15:11:17,992 urbanGUI INFO Client disconnected: 68a9ea7e54f34cc3be9f9395b19943b7
15:11:18,720 urbanGUI INFO connected
15:11:19,1 urbanGUI INFO Client disconnected: e17cdcf179d9482dbf4f6befc2e0db47
15:11:19,808 urbanGUI INFO connected
15:11:20,91 urbanGUI INFO Client disconnected: 8f262794c9fc40bb90aa37f9fe20db80
15:11:21,430 urbanGUI INFO connected
15:11:21,705 urbanGUI INFO Client disconnected: 2769a86f041c422ea7806ce11177b88f
15:11:23,95 urbanGUI INFO connected
15:11:23,368 urbanGUI INFO Client disconnected: 3306fd1f321740e1a80d005291e1a801
15:11:24,662 urbanGUI INFO connected
15:11:24,943 urbanGUI INFO Client disconnected: ac6afdc9952440fc8d4846548adc50f2
15:11:25,774 urbanGUI INFO connected
15:11:26,58 urbanGUI INFO Client disconnected: 5b270718206d49aeba597361bbff8be2
15:11:27,673 urbanGUI INFO connected
15:11:27,952 urbanGUI INFO Client disconnected: ce9bba31d7494d19800c24cd68aedb2c
15:11:29,477 urbanGUI INFO connected
15:11:29,753 urbanGUI INFO Client disconnected: a29d9cd1903c422f9606a801e4685c2c
15:11:30,684 urbanGUI INFO Client disconnected: 48924b51684e46008558c7577c1f6269
15:11:31,59 urbanGUI INFO connected
15:11:31,325 urbanGUI INFO Client disconnected: f2e6c04f38a94617bb6ac8a331cde224
15:11:32,858 urbanGUI INFO connected
15:11:33,141 urbanGUI INFO Client disconnected: 139db6786631476ea1561b669a0b2431
15:11:34,558 urbanGUI INFO connected
15:11:34,844 urbanGUI INFO Client disconnected: 83940d8f836e48a5918e14794e19c0c8
15:11:35,547 urbanGUI INFO connected
15:11:35,815 urbanGUI INFO Client disconnected: 5f85ca0cc1254b74b19d41f5c321ac8a
15:11:37,316 urbanGUI INFO connected
15:11:37,593 urbanGUI INFO Client disconnected: 7554f2dd4fd54bad9b95300c1b0998ca
15:11:39,195 urbanGUI INFO connected
15:11:39,476 urbanGUI INFO Client disconnected: 8fead9b5e8e148e9956778b7a91e875b
15:11:40,481 urbanGUI INFO connected
15:11:40,767 urbanGUI INFO Client disconnected: 2579464ed45b418491bda2fc60b8bcc6
15:11:41,488 urbanGUI INFO connected
15:11:41,761 urbanGUI INFO Client disconnected: b8b25fec44584b288d12c297b21c72bd
15:11:42,686 urbanGUI INFO connected
15:11:42,966 urbanGUI INFO Client disconnected: 0f780d9ebc294e47bf3f5184c38c8836
15:11:44,388 urbanGUI INFO connected
15:11:44,668 urbanGUI INFO Client disconnected: 14999c3f00314bb285bbde4501e3cecb
15:11:45,581 urbanGUI INFO connected
15:11:45,849 urbanGUI INFO Client disconnected: ada765125a374b7fbb4174d7d97bb901
15:11:47,95 urbanGUI INFO connected
15:11:48,328 urbanGUI INFO connected
15:11:48,602 urbanGUI INFO joined room: 6e2f8d8f-d835-478b-12a3-70ba109603a1
15:12:15,248 urbanGUI INFO connected
15:12:15,523 urbanGUI INFO joined room: 6e2f8d8f-d835-478b-12a3-70ba109603a1
15:12:41,806 urbanGUI INFO connected
15:12:42,77 urbanGUI INFO Client disconnected: 1bfd592fe1274c4db14d37ee6e144726
15:12:43,582 urbanGUI INFO connected
15:12:43,855 urbanGUI INFO Client disconnected: 63a0a6abdb3240f598af9e5947520362
15:12:44,988 urbanGUI INFO connected
15:12:45,256 urbanGUI INFO Client disconnected: 930e46874fdd48888e6f2abac0e7f066

[RosenTomov]

@kllr
Change your disconnect event to show the reason:

socket.on('disconnect', reason => {
            console.log(`reason: ${reason}`);
});

Another thing you can try is to drop transport upgrade

const socketIo = io(server, {
  transports: ['websocket'],
  allowUpgrades: false
});

[vforv]

I get same problem with socket.io 2.2. It randomly brake after some period of time and start reconnecting. Error is ping timeout.

Is there any way to make sure connection is stabile and never brake? Myabe some browser or something like that?

[jbgomez21]

Hi,
The same problem here, I’m using socket.io 2.2.0, someone has a solution for this?

Greetings

[FaizanZahid]

same problem happening to me, i m on socket.io 2.2.0 , and when on my website when a client sends an image in base64 to the server, and image is big, then this causes the image sender client to get ping timeout error and disconnects the client. plz let us know if anyone found any fix.

[RosenTomov]

Can you guys try by settings the pingTimeout to be higher than the pingInterval, like suggested here:
#3259 (comment)

Server:

const socket = io(server, {
  pingTimeout: 30000
  // ...other props
});

Client:

const socket = io(`${config.server}`, {
    pingTimeout: 30000
    // ...other props
  });

Default pingInterval is 25000ms.
If that still doesn't work, try with websocket only:

transports: ['websocket'],
allowUpgrades: false,
pingTimeout: 30000

[llaakso]

There is one undocumented (?) problematic behavior regarding timeouts: if server sends big packet which takes longer than timeout interval to transmit, that leads to timeout and disconnect. Only way to fix this is to make sure you send only relatively small packets. This is a problem as it means that you need to pay (too) much attention on your server logic so the data is always within the limits - and - as the connection speed is unknown factor, it's hard to fix this completely.

I'd suggest to adjust the timeout logic slightly to overcome this.

[FaizanZahid]

Some more info about my issues.
Now i m sending smaller packets, now it is not getting disconnected on image uploads but it randomly throws ping timeout errors, it happens totally randomly so its hard to debug the cause of it now that the packets r smaller.

@IIaakso what is the recommended ping timeout settings, mine r these as of now:

My client js:

My server settings:

still facing same issue on "socket.io": "2.2.0",
"socket.io-client": "2.2.0",

socket.io-client:socket close (ping timeout) +22s

[vforv]

Same issue as @FaizanZahid . It just randomly disconnect with with ping timeout...

[kelexel]

Same issue, and my setup really doesn't differ from yours (forceNew: true, being the most important part imo)

I'm upgrading from an antique socket.io 1.x with node 6.x setup to socket.io 2.2 and node 12.x, on both sending and receiving servers of the test.

Prior to the upgrade I could launch around 12k concurrent connections from the source server (running the client) to the destination server (running the io server).

The obvious system limits, firewall limits etc were checked and tuned prior to the update, so both "sending" and "receiving" servers should not be cause for issues or limitations.

But after upgrading to socket.io 2.2, I get random ping timeouts when launching about 2.5k concurrent connections...

*edit: no data is being transmitted in my test app, so it's not a data length, or content type issue. the clients only connects to the server, which performs custom auth for the clients, and if auth is successful, the clients wait for a server event.

So any insight on possible cause would be highly appreciated ;)

[kelexel]

From the tests I performed on my setup (FreeBSD 11.2, node 12.3.x), here are the results I can observe and replicate:

socket.io-client version | socket.io version | result
2.2 | 2.2 | random ping timeouts after approximatively 2000 connections
1.3.5 | 2.2 | works perfectly with 4500 concurrent connections
2.2 | 1.3.5 | cannot connect
1.3.5 | 1.3.5 | works perfectly with 4500 concurrent connections

So the issue seems to be in the client section of socket-io-client 2.2, or at least 2.x client branch...

[vforv]

Any solution for this?? It is not problem with number of connections in my case, it just randomly disconnect after few hours of inactivity with ping timeout error...

[Rinat93]

I have a similar problem, is the reason known?

[mukesh-simform]

@berksafran I'm facing same in android only, Could you please let me know how did you fixed it ?

[berksafran]

Well, Unfortunately I couldn't fixed this issue but I saw that this problem is only in development process. When you release your application and try with react-native run-android --variant=release, your socket.io server will work correctly.

The other solution that I found it, is uploading your socket.io server to any cloud server. (heroku, digitalocean etc.) After uploading, you should set your connection address to url of cloud server. I have tried it and it's working now.

If any question, you can text me on here.

Good luck.

[robin3317]

Anyone find the solution ? Updated: I tested after deploying my app, it works.

[sukalyansen123]

Hi everyone,
I am developing an app using socket.io.
For client I am using socket.io-client 2.3.0
For server I am using socket.io version 2.3.0

My server is hosted on heroku. And I am facing these random ping timeouts. The ping timeouts occur even more frequently when I open my app on mobile. Any solution that I can try? Thanks

[RosenTomov]

@sukalyansen123 From my debugging long ago, I noticed that it fails to upgrade transport and it times out. Try with websockets only:

const socketIo = io(server, {
  transports: ['websocket'],
  allowUpgrades: false
});

[szarkowicz]

@sukalyansen123 - I had a similar issue and for me actually setting the default timeouts explicitly seemed to help.

SERVER SIDE:

BEFORE:
let io = require('socket.io')(server, { transports: ['websocket'], allowUpgrades: false });

AFTER:
`
let io = require('socket.io')(server,
{
transports: ['websocket'],
allowUpgrades: false,
pingInterval: 25000, // default - 25000
pingTimeout: 60000, // default - 60000

});
`

[sukalyansen123]

Thanks. I tried doing this but still doesnt help. Now again on mobile socket disconnection happens, sometimes due to ping timeout and sometimes due to transport close. I am not sure if this is due to drop in internet connections, but again how am I supposed to handle this.

[szarkowicz]

Here is what my client side looks like if this helps - I only connect using websocket protocol
`
this.socket = io(serverURL,
{
transports: ['websocket'],
autoConnect: true

        });

`
Hopefully this helps

[sukalyansen123]

Here is what my client side looks like if this helps - I only connect using websocket protocol
`
this.socket = io(serverURL,
{
transports: ['websocket'],
autoConnect: true

        });

`
Hopefully this helps

Thanks again. Also wanted to know how you're handling the disconnect event from the client side.

[sukalyansen123]

Here is what my client side looks like if this helps - I only connect using websocket protocol
`
this.socket = io(serverURL,
{
transports: ['websocket'],
autoConnect: true

        });

`
Hopefully this helps

This seems to help a bit. However right now sockets are getting disconnected if kept idle for sometime with transport close. Would sending heartbeats at regular time intervals be a good solution?

[pouyan4938]

i have same problem. but i using socket for my online game.
i test somethings and i found when ping time out be show!
reason is when client(user phone) be sleep or go out of my game, server will disconnect it.
but i didn't found any solution for fix it! yet.

[gopalkohli]

Try this code at server side

var fs = require('fs');
var pkey = fs.readFileSync('/etc/ssl/private/ssl.key'); //Replace the path of your SSL key
var pcert = fs.readFileSync('/etc/ssl/certs/ssl.crt');//Replace the path of your SSL cert
var options = {
  key: pkey,
  cert: pcert
};

var app = require('https').createServer(options);

var io = require('socket.io')(app, {'pingTimeout': 180000, 'pingInterval': 25000});

Here pingInterval is importantant, Keep it low, I tried various values and found 25 sec is good to keep socket keep pinging before it get timeout. Main issue is there if within 60 sec there is no ping/pong then it will disconnect and try to auto reconnect. Also I found pingTimeout at server side and timeout and client side not able to help to disconnect the socket in 60 sec. Its happening due to chrome latest version 83.*

[darrachequesne]

As suggested above, we will revert the ping/pong mechanism in Socket.IO v3: https://github.com/socketio/engine.io-protocol#difference-between-v3-and-v4

[ritikasib]

I have also started to face same issue. I am using version socket.io version2.0.4 and started reciveing ping timeout issue after implement sockets with multiple nodes using socket.io-redis. But it is working completely fine with single node without redis. Can someone help me with any possible reason on this ?

[PranKe01]

Our project just updated node to v12. After that update we also get the disconnection because of ping timeout :| Here is a list of times when the disconnects occured:

11:06:59
11:08:19 +80s
11:10:39 +140s
11:14:39 +240s
11:15:59 +80s
11:20:59 +300s
11:22:19 +80s
11:23:19 +60s
11:24:39 +80s
11:28:59 +260s

Our settings:
TimeOut: 30s
Interval: 20s
ReconnectionDelay: 10s

[dandamiangow3t]

Hey! had the same problem. Socket.io versions 2.3.0 (same versions on client and server).
My problem was with the server. After I checked how much it takes for the server to prepare the response sent over the socket, after 2 days of diggings I've found the problem. It was a method for which the execution time would increase linearly, and when it started to take 1s+ to execute, the server was just unable to answer to the client's ping and then the client would issue a connection close and connection reattempt, and from here you know the rest.
So, in conclusion, what I'm saying is that is not a socket.io problem but most often a server problem.
Hope this helps.

[MaximRG]

Matching the versions for server and client solves the problem for me. I've tested with V 3.0.4 & 2.3.0

[irvanherz]

I had a similar problem. This bug happened randomly.
From the client side I got an event saying that my connection was disconnected.
Fortunately, soon I realized that maybe this isn't a socket.io problem at all. It doesn't happen randomly either.
From my observations I can tell that the 'ping timeout' occurs only when I reload the page in the browser. After reloading, the socket.io server creates a new socket handle; and of course after that, socket session in the browser get the new one.
But, where did the old socket session go?
RIGHT! The session remain alive until the pingTimeout is reached. When that happens, we will see the ping timeout. And unfortunately, we wrongly guessed that it happened in the current socket session.
What you can do: try to console.log socket id each new socket created/connected AND disconnected.
I'm not an English native speaker. But I hope you understand. :D

[darrachequesne]

There was two changes in latest releases that should fix this issue:

• the ping-pong mechanism was reversed in v3 (so a delay of a timer on the client-side should no longer trigger a disconnection)

Reference: https://socket.io/blog/engine-io-4-release/#Heartbeat-mechanism-reversal

• the default value of pingTimeout was increased from 5000 to 20000 in v4 (to give each side a longer delay before closing the connection)

Reference: https://socket.io/docs/v4/migrating-from-3-x-to-4-0/#The-default-value-of-pingTimeout-was-increased

I think we can now close this. Please reopen if needed.

[zouyu01-sz]

匹配服务器和客户端的版本为我解决了这个问题。我用 V 3.0.4 & 2.3.0 测试过

V 3.0.4这个版本确实没有pingTimeout了，感谢宝贵经验

[darrachequesne]

For future readers:

Please check https://socket.io/docs/v4/troubleshooting-connection-issues/