Question: What could be the possible reasons for socket-io server returning 400 Bad Request via nginx?

[shamasis]

Hi,

We are facing a scenario where a good percentage of requests going to /Socket.io/?... is returning with 400.

Default nginx access log results are littered with:

10.0.0.82 - - [04/Dec/2015:11:43:27 +0000] "GET /socket.io/?__sails_io_sdk_version=0.11.0&__sails_io_sdk_platform=browser&__sails_io_sdk_language=javascript&EIO=3&transport=websocket HTTP/1.1" 400 45 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"

The requests are not even reaching our overlying SailsJS server. We suspect something going wrong at socket-io layer. Any thought?

[shamasis]

Interesting discovery: The upgrade header is blank for the requests having 400 response. Forcing upgrade header to websocket stops the 400, but causes intermittent 499 error on nginx.

[shamasis]

After a LOT of painstaking log-watching, this is our temporary solution for nginx config (to stop freaking out our servers.) The following directive checks for missing "Upgrade" header in request and returns with a 202 without hitting the node server.

location /socket.io {
    set $local_upgrade_failure "${arg___sails_io_sdk_platform}${arg_transport}${connection_upgrade}";
    if ( $local_upgrade_failure = "browserwebsocket" ) { return 202; break; }

 ... remain proxy configurations
}

Now, this does NOT fix the clients that are sending these requests, but surely frees up the server. Now, the question is @rauchg, why are some socket.io clients not sending the "upgrade" header? One pattern is that they are all Windows and a very few Linux machines. Did not spot a single OSX in one full hour of heavy usage.

[nkzawa]

Did you set up sticky sessions ? http://socket.io/docs/using-multiple-nodes/

[shamasis]

Not needed. We use websockets alone and its sticky anyway.

[julianlam]

A bit late of a reply, but seems related to socketio/engine.io#283... open since 2014, mind you.

[shamasis]

Just as a note, we are still facing this error.

• transport only websocket
• proxy of nginx configured correctly

We see that the request header is missing Upgrade "websocket" within nginx. If we force this using set_header, the soxket.io server closes connection and we get 499 from nginx.

[julianlam]

If using nginx, we worked around this by setting max_fails to 0. Not a good solution, but got us going again.

[shamasis]

@julianlam - I will research and try it out. If it works, lunch is on me if we ever meet. 😊

[shamasis]

Ok. That does not solve the problem. I am thinking of ditching as many network layers as possible to get to the root of this.

We have AWS ELB in TCP mode which proxies to EC2 instances (c4.2xlarge) and those instances have nginx and docker running in them. nginx is configured simply and nicely (all websocket related configs in place) to reverse proxy to a docker port. Inside docker pm2 runs our NodeJS server (SailsJS) driven in cluster mode. The kernel is tweaked to handle large number of connections.

This is so frustrating that I am going crazy.

[MasterJames]

Server tweak? To handle large number of connections? Whats that look like? Maybe turn that off and see if it's responsible.
Read somewhere (remote syslog site) ELBalancer is a bad bottle neck under "heavy load" burst conditions. I think they opted for DNS only solution.
Here it is...
https://www.loggly.com/blog/why-aws-route-53-over-elastic-load-balancing/

[MasterJames]

#1942 (comment)
More on the load balancer issue recommended against in this thread.

[shamasis]

Thanks for feedback.

The criteria of ELB feature failure in the Loggly articles seems a bit dated and does not apply to us. We are doing TCP load balancing with a predictable traffic on standard ports. Also, all other points in that article has been addressed in ELB over time - except the burst traffic drop and warm up (not relevant to us.) 😞

The server tweaks are simply increase of file ulimits, net somaxconn to modest 1024 and increase to internal port range.

I've tried turning each off. Making it a vanilla nginx with Websocket proxy tweaks - but to no use.

[MasterJames]

Oh well. Mayne I was thinking bad gateway.
400 is bad request. Maybe try to get better logging of the request content on both sides and see if you can see why it's "bad" aka malformed.

[shamasis]

I will then have to deep dive into socket.io - it's being returned by socket.io with no additional message -

{ code: 3, message: "badRequest"}