aiohttp server hangs when nginx try to mixin plain http request with failed websocket connection upgrade in one tcp stream.

[zTrix]

Long story short

aiohttp server hangs when nginx try to mixin plain http request with failed websocket connection upgrade in one tcp stream.

The following tcpdump capture shows what happens in detail.

Expected behaviour

Server returns something.

Actual behaviour

Server hangs forever without closing tcp connection. Nginx return 504 Gateway timeout to end user.

Steps to reproduce

1. write a simple api server using aiohttp (version 3.4.4 in my case), implement some /api/xxx for testing, leave /ws unimplemented.
2. using nginx to reverse proxy to this api server.
3. user open browser targeting a web page which contains both ws request and api request. Then api query will gateway timeout.

Example nginx.conf:

    ...
    location /api/ {
        proxy_pass http://server_backend;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /ws {
        proxy_pass http://server_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

aiohttp server serves /api/ request, but leave /ws unimplemented.

Then this issue could be reproduced.

Your environment

Docker: nginx:latest
Docker: aiohttp3.4.4 on python:3.7-alpine3.7

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #2920 (AIOHttp failing after some requests), #660 (aiohttp.request hangs on some URLs), #2024 (Preparing a websocket from a low-level-server request fails), #1768 (websocket connection is closing.), and #1541 (Aiohttp.web server timeout on long requests.).

[zTrix]

The weird thing here is why nginx still sends http/1.0 request to upstream backend server while proxy_http_version 1.1; specified in nginx.conf.

[kxepal]

Your nginx configuration works fine for me for a bit modified web_ws.py example. Could you share the code which reproduces the issue? And full nginx config could be useful as well.

[zTrix]

ws-api-mixed-in-one-tcp-stream-hang-bug-demo.tar.gz

Here attached a minimal environment to reproduce this issue.

to start the servers, run

$ docker-compose up -d

to reproduce the issue, follow the steps in http://your_server/index.html

[kxepal]

Thanks! I don't have docker, but without it everything works fine, nothing hangs.

[zTrix]

Maybe different nginx version behave differently?

BTW, you need to follow the steps described in index.html:

1. visit http://your_ip/api/null first to see everything ok
2. visit http://your_ip/ws to see a 404 page
3. visit http://your_ip/api/null again, which will get you a 504 Gateway timeout after around 1 minute

It works out on condition that, nginx did try to keepalive with upstream server, send the third request using the same tcp connection used by step 2.

[zTrix]

Here I attach a simpler way to produce this issue, using aiohttp client itself.

demo.tar.gz

Only server.py and client.py provided.

1. Run server.py using python server.py
2. Run client.py using python client.py

tcpdump capture screenshot:

[zTrix]

Running the client.py against a golang written demo server works just fine. Here is the demo code:

server.go

package main

import (
    "fmt"
    "log"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    fmt.Fprintf(w, "{\"err\":null}")
}

func main() {
    http.HandleFunc("/api/null", handler)
    log.Fatal(http.ListenAndServe(":9211", nil))
}

The tcpdump screen capture below:

[kxepal]

Maybe different nginx version behave differently?

1.14.0 on my side.

BTW, you need to follow the steps described in index.html:

Yes, I followed those instructions and repeat them several times.

[kxepal]

demo.tar.gz

This perfectly reproduces the issue. Thanks!

[kxepal]

Meanwhile issue couldn't be reproduced with uvloop. Hm.

[zTrix]

Any progress for this issue?

[asvetlov]

Not yet

[gjcarneiro]

I've had this bug in production, with latest aiohttp. Disabling upstream http connection keepalive in nginx fixed it.

Meanwhile issue couldn't be reproduced with uvloop. Hm.

Not so. My production code runs aiohttp with uvloop, and still experienced this issue.

[asvetlov]

I think the bug is fixed by #3971