HttpClient has many TIME_WAIT or CLOSE_WAIT connections

[ahmet8282]

I use AddHttpClient() dependency injection to add a client to a service. At times, when I execute netstat -a on the server, I see many connections open with TIME_WAIT or CLOSE_WAIT status. I believe that these connections take up so much resource that, other TCP connections are unable to operate. Is this possible? Is there a way to stop these, and is it safe?

[davidsh]

This usually means that your code using HttpClient is not disposing of the HttpResponseMessage objects etc.

In order to provide further guidance, you'll need to submit a code sample of how you are using HttpClient.

[ahmet8282]

It's very standard actually. I created a named http client. I have injected IHttpClientFactory a transient service. I created a httpClient with the IHttpClientFactory.Create("namedClient") in the service constructor. Then I just started posting requests via service methods. I just followed https://docs.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-2.2#named-clients

The only difference is that, I don't use baseUrl. I just have full url for posts.

[stephentoub]

That example should be disposing of the HttpResponseMessage that results from awaiting the SendAsync call. Can you try doing the same in your code and see if that makes a difference?

[davidsh]

That example should be disposing of the HttpResponseMessage that results from awaiting the SendAsync call. Can you try doing the same in your code and see if that makes a difference?

Example of a better recommended pattern:

public async Task<IEnumerable<GitHubIssue>> GetAspNetDocsIssues()
{
    HttpResponseMessage response = await Client.GetAsync(
        "/repos/aspnet/docs/issues?state=open&sort=created&direction=desc");

    try
    {
        response.EnsureSuccessStatusCode();

        var result = await response.Content
            .ReadAsAsync<IEnumerable<GitHubIssue>>();
    }
    finally
    {
        // Dispose of HttpResponseMessage to free up system resources and close network connections.
        response.Dispose();
    }

    return result;
}

[ahmet8282]

It looks like it worked. Thank you. Perhaps you should update the code on Microsoft site as well.
@glennc @rynowak @stevejgordon

[ahmet8282]

Nope. I spoke too early. Again I have over 4000 CLOSE_WAITS right now with response.Dispose() added. Do you think setting a Connection: close would help and is it necessary? @davidsh @stephentoub

[stephentoub]

Nope. I spoke too early. Again I have over 4000 CLOSE_WAITS right now with response.Dispose() added. Do you think setting a Connection: close would help and is it necessary?

Can you please share a repro?

[ahmet8282]

It will take some time to prepare the repro. But meanwhile if it will help. Below is the netstat -a shows. Over 4000 rows of it. I'm using it to send push notifications to Firebase Cloud Messaging. And as I stated, i don't set base url. I just do http post with full urls, if it makes a different. So maybe Google is doing some weird stuff with their load balancers and servers perhaps to keep the connections open?

tcp        1      0 master-memberserv:44653 ams16s32-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:32769 ams15s29-in-f106.1e:443 CLOSE_WAIT
tcp        1      0 master-memberserv:34231 ams15s32-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:45417 fra02s28-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:36765 ams16s31-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:34433 ams15s21-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:39045 fra02s28-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:33929 ams16s31-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:38447 ams15s30-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:36419 ams15s21-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:44855 ams15s40-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:42707 ams15s40-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:34921 ams15s30-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:42767 ams15s29-in-f106.1e:443 CLOSE_WAIT
tcp        1      0 master-memberserv:35237 ams16s29-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:45307 ams16s29-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:40095 ams16s30-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:46173 ams15s32-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:44443 ams16s29-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:38657 ams16s31-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:39739 ams15s33-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:34855 ams16s31-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:36435 ams15s32-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:46519 ams15s40-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:42839 ams16s30-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:38919 ams15s30-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:46211 ams15s29-in-f106.1e:443 CLOSE_WAIT
tcp        1      0 master-memberserv:33343 ams15s40-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:40581 ams16s32-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:33939 ams16s30-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:38275 fra02s28-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:46397 ams15s32-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:36333 ams15s30-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:42675 ams16s31-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:37631 ams16s29-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:35741 ams15s33-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:45223 ams16s31-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:33375 fra02s28-in-f10.1e1:443 CLOSE_WAIT
tcp        1      0 master-memberserv:33375 fra02s28-in-f10.1e1:443 CLOSE_WAIT

[ahmet8282]

@stephentoub I have created a test repo for this. https://github.com/ahmettahasakar/HttpClientTest
I made it so that there are 100 concurrent posts to Firebase Cloud Messaging. Even though I post blank payloads, the connections still seem to stay open.

In order for project to work, you need a Firebase authentication file. You can generate one easily from https://firebase.google.com/docs/admin/setup

[stephentoub]

Thanks. I'll take a look.

Even though I post blank payloads, the connections still seem to stay open.

It's expected that the connections stay open. You don't want them to stay open at all? By default connections are kept alive in a connection pool for several minutes in order to avoid the expense of creating and tearing them down for every request.

[ahmet8282]

@stephentoub I don't mind having the connections open. As you said, it's good to reuse them for performance. However, I think there is something else going on here.

Although I post requests to the same URL, I believe because of Google's load balancers and DNS settings, it connects to different server most of the time. So it rarely reuses the same opened connection. And when I do 1000 posts, instead of reusing the connections, it just opens new connections to a different server each time.

This is my theory by the way. I may be totally off, and all of this may be normal. But I don't think to have thousands of connections open with the state CLOSE_WAIT is normal. At times, I get errors about not being able to open any new TCP connections from other services. So these thousands of open connections that are just waiting to be reused, are clogging the sockets for other services.

[geoffkizer]

Are you sure you are using the same HttpClient instance each time?

Although I post requests to the same URL, I believe because of Google's load balancers and DNS settings, it connects to different server most of the time. So it rarely reuses the same opened connection.

If you are using the same URL and the same HttpClient instance, then connections should be reused, regardless of stuff like DNS round robin.

[ahmet8282]

@geoffkizer I initiliaze clients with the line below
_client = clientFactory.CreateClient(FirebaseServiceClient);

So my understanding from https://docs.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-2.2#named-clients is that the transient services are supposed to use the same client. Am I wrong? Isn't the whole point of IHttpClientFactory to manage HttpClients to be reused