System.Net.Http.HttpRequestException: Normally only one use of each socket address (protocol/network address/port) is allowed

[zhenlei520]

Scenario:
Continuously send a large number of short requests. Their request time is very short, but the number is large. They are requested through IP. The IP and port are fixed. The exception message System.Net.Http.HttpRequestException: Normally only one use of each socket address (protocol/network address/port) is allowed will appear from time to time

CallerClient is a global static

public class CallerClient : IDisposable
{
    private HttpClient _httpClient = null;

    public CallerClient()
    {
        _httpClient = new HttpClient(
            new HttpClientHandler()
            {
                AutomaticDecompression = System.Net.DecompressionMethods.GZip,
                MaxConnectionsPerServer = 65535,
                UseProxy = false,
                Proxy = null,
            });
        _httpClient.DefaultRequestHeaders.AcceptEncoding.Add(new System.Net.Http.Headers.StringWithQualityHeaderValue("gzip"));
        _httpClient.Timeout = TimeSpan.FromMinutes(15);
    }

    public async Task<string> GetString(string url, HttpMethod method, object options, int timeout, Dictionary<string, IEnumerable<string>> headers)
    {
        var resp = await GetResponse(url, method, options, timeout, headers).ConfigureAwait(false);
        return await resp.Content.ReadAsStringAsync().ConfigureAwait(false);
    }

    public async Task<HttpResponseMessage> GetResponse(string url, HttpMethod method, object options, int timeout, Dictionary<string, IEnumerable<string>> headers)
    {
        var resp = await GetResponseIgnoreHttpStatusCode(url, method, options, timeout, headers).ConfigureAwait(false);
        return resp.EnsureSuccessStatusCode();
    }

    public async Task<HttpResponseMessage> GetResponseIgnoreHttpStatusCode(string url, HttpMethod method, object options, int timeout, Dictionary<string, IEnumerable<string>> headers)
    {
        var opts = new RequestOptions
        {
            ContentBody = options,
            Timeout = timeout,
            Headers = headers,
        };
        return await GetResponseCoreAsync(url, method, opts).ConfigureAwait(false);
    }

    private async Task<HttpResponseMessage> GetResponseCoreAsync(string url, HttpMethod method, RequestOptions opts)
    {
        if (opts == null) throw new ArgumentNullException($"options");
        int timeout = opts.Timeout;
        object body = opts.ContentBody;
        Dictionary<string, IEnumerable<string>> headers = opts.Headers;
        if (timeout <= 0) timeout = Convert.ToInt32(_httpClient.Timeout.TotalMilliseconds);
        using (var cts = new CancellationTokenSource(timeout))
        {
            var buffer = body == null ? null : JsonSerializer.SerializeToUtf8Bytes(body);
            var requestUri = new Uri(url);

            var request = new HttpRequestMessage()
            {
                Method = method,
                RequestUri = requestUri,
            };

            if (buffer != null)
            {
                request.Content = new ByteArrayContent(buffer);
                request.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/json");;
            }

            if (headers != null && headers.Count > 0)
            {
                foreach (var kv in headers)
                {
                    request.Content.Headers.Add(kv.Key, kv.Value);
                }
            }

            var resp = await _httpClient.SendAsync(request, opts.CompletionOption, cts.Token).ConfigureAwait(false);

            return resp;
        }
    }

    private class RequestOptions
    {
        public object ContentBody { get; set; }
        public int Timeout { get; set; } = 0;
        public Dictionary<string, IEnumerable<string>> Headers { get; set; }
        public HttpCompletionOption CompletionOption { get; set; } = HttpCompletionOption.ResponseContentRead;
    }

    public void Dispose()
    {
        _httpClient.Dispose();
        _httpClient = null;
    }
}

[zhenlei520]

.net version: .net6.0
system: Windows Server 2022

[antonfirsov]

This looks like a symptom of port exhaustion.

MaxConnectionsPerServer = 65535

Note that this is the total number of TCP ports, the number of available dynamic ports is typically 16384. Moreover, it does not limit the total number of connections, only the ones that go to the same peer. If you want to resolve this by limiting MaxConnectionsPerServer you'd need a much lower number. I would recommend against this, since it may hurt throughput.

A couple of ideas:

1. .NET 6.0 will reach end of support on November 12, 2024. We strongly recommend to upgrade to .NET 8. Since there were many changes in the connection pooling implementation there is some chance that the issue will go away or occur less frequently.

2. If your server supports HTTP/2, you may try to use HTTP/2 in order to multiplex requests over the same TCP connection. This can be done eg. by setting client.DefaultRequestVersion = HttpVersion.Version20, and potentially setting DefaultVersionPolicy = HttpVersionPolicy.RequestVersionExact to prefer errors over HTTP/1.1 fallback.

3. Extend the dynamic port range on the machine: https://learn.microsoft.com/en-us/troubleshoot/windows-client/networking/tcp-ip-port-exhaustion-troubleshooting#default-dynamic-port-range-for-tcpip

4. Since you are on Windows, you can try to configure an auto-reuse port range on the machine. Note that this only helps if your HttpClient deals with multiple servers.
   https://devblogs.microsoft.com/dotnet/dotnet-6-networking-improvements/#handle-port-exhaustion-by-utilizing-auto-reuse-port-range-on-windows

[MihaZupan]

Is the server you're talking to in this case your YARP configuration described in dotnet/yarp#2427?
Since you're setting Connection: close on every response, that would explain why you'll quickly run into port exhaustion that Anton described.

[zhenlei520]

Is the server you're talking to in this case your YARP configuration described in microsoft/reverse-proxy#2427? Since you're setting Connection: close on every response, that would explain why you'll quickly run into port exhaustion that Anton described.

No, this is not the same problem. It seems that the problem is caused by the reuse of the client port when creating a connection. Today, I tried to change the registry HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters MaxUserPort=65534 and TcpTimedWaitDelay=5 to solve this problem. But in addition to this, I saw some netizens reuse the port by ServicePointManager.ReusePort = true, which can also avoid this problem, but I don’t know what risks this has.

https://stackoverflow.com/questions/44548444/how-does-servicepointmanager-reuseport-and-so-reuse-unicastport-alleviate-epheme

[MihaZupan]

Port reuse can help if you're talking to multiple different servers, it won't do anything if you're only establishing connections to a single server.

Note that ServicePointManager is obsolete and that option does nothing when using HttpClient, but you can set the relevant flags on the sockets themselves. See https://learn.microsoft.com/dotnet/fundamentals/networking/http/httpclient-migrate-from-httpwebrequest.

If you're establishing tons of connections to a single endpoint, it is expected that you may run out of available ports and observe these exceptions.

[antonfirsov]

Not sure how does the MaxUserPort registry key work, but using netsh to configure ephemeral ports is superior, since you are then in control of the entire port range. See point 3 in #109141 (comment).

SO_REUSE_UNICASTPORT (aka. the setting that is exposed as ServicePoint.ReusePort on .NET Framework) is automatically enabled on .NET 6+, but you need to tweak some global Windows settings to make it actually work, see point 4 in #109141 (comment).

Please let us know if any of the recommendations helped.

[zhenlei520]

Not sure how does the MaxUserPort registry key work, but using netsh to configure ephemeral ports is superior, since you are then in control of the entire port range. See point 3 in #109141 (comment).

SO_REUSE_UNICASTPORT (aka. the setting that is exposed as ServicePoint.ReusePort on .NET Framework) is automatically enabled on .NET 6+, but you need to tweak some global Windows settings to make it actually work, see point 4 in #109141 (comment).

Please let us know if any of the recommendations helped.

I am glad to receive your reply. The project is currently migrating from .net461 to .net8.0. After the migration is completed, we will try point 4 in #109141 (comment).

[dotnet-policy-service]

This issue has been marked needs-author-action and may be missing some important information.

[dotnet-policy-service]

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[dotnet-policy-service]

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.