Websocket connection requests fail when highly concurrent

[bpiper]

Branch/Environment/Version

• Branch/Version: 2.9.1
• Environment: on-premise gateway in Docker

Describe the bug

When Tyk proxies multiple websocket connection requests simultaneously, occasionally one of them will fail, either with a 500 response, or a response containing an invalid Sec-WebSocket-Accept header. The following is logged by Tyk:

time="Nov 29 00:52:52" level=error msg="Hijack error: http: connection has been hijacked" origin=******** path="/api/some/endpoint"
time="Nov 29 00:52:52" level=error msg="http: proxy error: http: connection has been hijacked" api_id=my-api org_id=default prefix=proxy server_name=api.server.com user_id=-- user_ip="********" user_name=
2019/11/29 00:52:52 http: response.WriteHeader on hijacked connection from github.com/TykTechnologies/tyk/gateway.(*ErrorHandler).HandleError (handler_error.go:82)
2019/11/29 00:52:52 http: response.Write on hijacked connection from text/template.(*state).walk (exec.go:269)

I originally encountered this when running integration tests that (due to parallel execution) can open up to around 10 websocket connections simultaneously. We run two gateway servers behind an AWS load balancer with clustering.

Reproduction steps
Steps to reproduce the behavior:

1. Run Tyk gateway in front of a server with a websocket endpoint.
2. Open a whole bunch of websocket connections at once. If you have wscat installed, something like this will encounter a single connection failure at least 50% of the time:

for i in {1..40}; do wscat -c wss://api.server.com/api/some/endpoint < /dev/zero & done

Configuration (tyk config file):
See attached API configuration. Tyk config is just the default from tykio/tyk-gateway:v2.9.1 with the following env variables set:

TYK_GW_STORAGE_HOST = some.redis.endpoint
TYK_GW_STORAGE_PORT = <redis-port>
TYK_GW_COPROCESSOPTIONS_ENABLECOPROCESS = false
TYK_GW_HEALTHCHECK_ENABLEHEALTHCHECKS = true
TYK_GW_SECRET = ********
TYK_GW_STORAGE_ENABLECLUSTER = true

[bpiper]

tyk_api_config.json.txt

[candux]

We can confirm the problem. It's testable with wss://echo.websocket.org:443 and attached config
tyk_websocket.json.txt
for i in {1..10}; do wscat -c "wss://tyk-gw/realtime-ws2/" < /dev/zero & done

[buger]

We found a way to fix it #2747

Scheduled for 2.9.3

[maciejwojciechowski]

verified, fixed.

before:

tyk@MBP-Maciej tyk-develop-env (master) $ for i in {1..10}; do wscat -n -c "wss://tyk-gateway:8181/realtime-ws2/" < /dev/zero & done
[1] 23287
[2] 23288
[3] 23289
[4] 23290
[5] 23291
[6] 23292
[7] 23293
[8] 23294
[9] 23295
[10] 23296
                                       error: Unexpected server response: 500
error: Unexpected server response: 500
error: Unexpected server response: 500
error: Unexpected server response: 500
error: Unexpected server response: 500
error: Unexpected server response: 500
error: Unexpected server response: 500
error: Unexpected server response: 500
error: Invalid Sec-WebSocket-Accept header

after

tyk@MBP-Maciej tyk-develop-env (master) $ for i in {1..10}; do wscat -n -c "wss://tyk-gateway:8181/realtime-ws2/" < /dev/zero & done
[1] 23381
[2] 23382
[3] 23383
[4] 23384
[5] 23385
[6] 23386
[7] 23387
[8] 23388
[9] 23389
[10] 23390
                                       Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)
Connected (press CTRL+C to quit)