HttpWebRequest hits 1s timeout when resolving http://localhost on Win10 (.NET Framework is fast)

[jimevans]

Consider the following method (error/exception handling omitted for brevity:

public virtual string MakeHttpRequest(Uri fullUri, string httpMethod, string requestBody)
{
    string responseString = string.Empty;
    HttpWebRequest request = HttpWebRequest.Create(fullUri) as HttpWebRequest;
    request.Method = httpMethod;
    request.Timeout = 30000;
    request.Accept = "application/json,image/png";
    if (request.Method == "POST")
    {
        string payload = requestBody;
        byte[] data = Encoding.UTF8.GetBytes(payload);
        request.ContentType = "application/json;charset=utf-8";
        Stream requestStream = request.GetRequestStream();
        requestStream.Write(data, 0, data.Length);
        requestStream.Close();
    }

    HttpWebResponse webResponse = request.GetResponse() as HttpWebResponse;
    using (Stream responseStream = webResponse.GetResponseStream())
    using (StreamReader responseStreamReader = new StreamReader(responseStream, Encoding.UTF8))
    {
        responseString = responseStreamReader.ReadToEnd();
    }
    return responseString;
}

Executing this method against a server running on localhost yields wildly different execution times when run via the full .NET Framework vs. .NET Core. I have a demo project that shows the discrepancy, making calls to a locally running HTTP server, that server being the Chromium project's chromedriver.exe, used to automate the Chrome browser. When running the executables created by that project, you can see the following typical outputs (as run on Windows 10, build 16288):

Run against .NET Core 2.0:

Starting test...
Starting ChromeDriver 2.32.498550 (9dec58e66c31bcc53a9ce3c7226f0c1c5810906a) on port 9515
Only local connections are allowed.
Started session 946394531499259b7442073c0e26060d
Navigating to http://www.google.com
Navigation complete
Making 10 HTTP calls to localhost, logging the elapsed time...
Elapsed time for HTTP call: 1038 milliseconds
Elapsed time for HTTP call: 1016 milliseconds
Elapsed time for HTTP call: 1015 milliseconds
Elapsed time for HTTP call: 1011 milliseconds
Elapsed time for HTTP call: 1012 milliseconds
Elapsed time for HTTP call: 1024 milliseconds
Elapsed time for HTTP call: 1013 milliseconds
Elapsed time for HTTP call: 1027 milliseconds
Elapsed time for HTTP call: 1013 milliseconds
Elapsed time for HTTP call: 1014 milliseconds
Test finished. Press to exit.

Running against the full .NET Framework 4.5:

Starting test...
Starting ChromeDriver 2.32.498550 (9dec58e66c31bcc53a9ce3c7226f0c1c5810906a) on port 9515
Only local connections are allowed.
Started session 81c2bbd21a0d89354b2dead8d52ee982
Navigating to http://www.google.com
Navigation complete
Making 10 HTTP calls to localhost, logging the elapsed time...
Elapsed time for HTTP call: 35 milliseconds
Elapsed time for HTTP call: 7 milliseconds
Elapsed time for HTTP call: 12 milliseconds
Elapsed time for HTTP call: 5 milliseconds
Elapsed time for HTTP call: 4 milliseconds
Elapsed time for HTTP call: 7 milliseconds
Elapsed time for HTTP call: 4 milliseconds
Elapsed time for HTTP call: 6 milliseconds
Elapsed time for HTTP call: 5 milliseconds
Elapsed time for HTTP call: 5 milliseconds
Test finished. Press to exit.

I would expect similar performance between framework versions. Absent that, I would expect there to be settings to modify to the HttpWebRequest object that would yield similar performance as a workaround.

[EDIT] C# syntax highlighing by @karelz

[jimevans]

I'll point out here, to forestall the inevitable comment mentioning it, that changing from System.Net.HttpWebRequest to System.Net.Http.HttpClient in the .NET Core case doesn't resolve the issue. The performance is the same. In the aforementioned demo project, there is an alternative implementation that can be used in the .NET Core case that utilizes the newer class. The performance is comparable.

[davidsh]

This is a duplicate of #23255.

[jimevans]

@davidsh While I'm more than happy to concede that it might be (and probably is) a duplicate, the statistics given in the report of #23255 seem to indicate individual requests via the full framework take longer than the same requests using .NET Core. This is the opposite behavior of what I've observed, hence this new issue report. I'm sure I'm missing some context or am misinterpreting what I'm reading in that report. Please feel free to educate me.

[qmfrederik]

Can you try changing your tests so that they use http://127.0.0.1 in the URL instead of http://localhost?

When using 127.0.0.1, I get response times of ~10ms, whereas localhost give me response times of ~1020ms, so it looks like there's a 1-second timeout you're hitting.

[Drawaes]

Which might suggest it's an issue with the dns lookup class rather than the http class. Which would be insightful

[jimevans]

Of course, @qmfrederik is correct. Using the IP address directly in the URL reduces the time required to make the HTTP call. However, as pointed out by @Drawaes, that does indicate an issue somewhere else in the network stack, most likely in DNS name resolution, and maybe specific to local loopback addresses. If someone with the appropriate permissions wants to edit the title of the issue report to better indicate what the issue actually is, that would be fine by me. Additionally, if this is still a duplicate of a different issue, and someone could point me to the proper one, I'd like that too.

[Drawaes]

Try using the dns class directly to resolve local host.

[Drawaes]

Also if you can try your test against ::1 It might be resolving localhost to ipv6 and that might be the issue.

[jimevans]

@Drawaes The server in question only listens on IPv4 in dual-stack environments. Attempting to change the URL to ::1 (predictably) yields a System.Net.WebException with the message, "An error occurred while sending the request. A connection with the server could not be established." Dns.GetHostEntry() returns both an IPv6 and an IPv4 address for the localhost host name in my environment. This leads me to three questions:

How do I force the HttpWebRequest (or the HttpClient, for that matter) to use the IPv4 address in that case?

How is it that the full framework version, which returns the same list of addresses from Dns.GetHostEntry(), and in the same order, is able to bypass the issue?

Is it not a valid issue that there is a negative perf difference between .NET Core and the .NET Framework for the same code?

[Drawaes]

@jimevans I don't work for MS so can't speak for them I am merely an interested party but let me try to answer you :)

1. I don't know other than to resolve the DNS entry yourself and call the IPv4 address (I have had to do this before on full framework, not for perf but other reasons).
2. That is a good question, it should at least be consistent that I agree with!
3. Absolutely, I hope I didn't come across as saying your issue wasn't valid. I was just trying to narrow it down for everyone (binary search of the problem space ;) ).

[qmfrederik]

What I think happens is that .NET resolves localhost to ::1 and 127.0.0.1. Because ::1 is first in the list, it tries to connect to ::1 (perhaps with a 1-second timeout), fails, and continues with the next entry, 127.0.0.1, which succeeds.

Looking at it this way, it seems the framework is behaving correctly. You could argue that using localhost to connect to an IPv4-only service on a dual-stack system is not optimal, and that you should use 127.0.0.1 instead.

As to why the difference, it seems Desktop .NET gives preference to IPv4 whereas .NET Core treats them equally. @davidsh or @stephentoub may be able to comment as to whether that is by design.

Out of curiosity, is there any reason using 127.0.0.1 instead of localhost would not work for your use case?

[jimevans]

@qmfrederik Your supposition of what's happening under the covers is what I suspect too, but given that the IP addresses appear in the same order in both Desktop and Core configurations, I would expect the underlying library to approach them the same way.

As to your question, the API that my project provides to end users includes the user providing a URL to a remote-end processor (like the chromedriver.exe mentioned in the repro), and this URL is often to the local machine. Knowing the level of sophistication of my average user, insisting on the loopback IP address means I have no elegant choices. I can tell all of my users to use an IP address instead of localhost. This would put the .NET language bindings at odds with every other language binding providing this API, as neither Java, Python, Ruby, nor JavaScript have this restriction. Or, I could put my language bindings in the business of parsing the URL I get, and magically attempting to do the right thing, which is going to be error-prone, and isn't a core competency of the project.

Moreover, I have no control over the remote-end behavior with respect to IPv4 vs. IPv6. It's entirely possible that the several remote ends that my project supports, which are provided by browser vendors (including Microsoft!) may differ between what support they offer for different network stacks.

[bofcarbon1]

I've started using .NET Core 2.0 with Entity Framework Core 2.0. A lot of changes from .NET 4.5 for me as I used the Http Response Message. Now I'm returning an 'object' that automatically gets converted to Json. I've received some response handshake errors that worry me. Thinking I'm getting timed out sometimes. I am sending requests via an observable from an Angular 2 node.js web app. My web page is loading without the results from the .NET api. Eventually it will load but sometimes it looks like it takes 25 or more seconds. Have not measured but it seems like a lifetime in web response time. I use some console.log commands and notice that they show up late. Granted I'm still learning the observable and am not certain about any implied REST asynchronous thing going on. As a user though I'm not going to want to see a page without data that suddenly appears later. I'd just start to reload the page after a few seconds. I use Google Chrome usually but I gave IE a try just to compare and its about the same. I am also using Visual Studio 2017 Community (free) run on IIS Express (localhost) and my Angular web app runs on the Cloud 9 IDE (free version) so my issue could be limited resources. I've also noticed that Visual Studio 2017 Community IDE is slow to respond. Just switching files takes a while. It often freezes and goes unresponsive. I'm leaning toward Visual Studio as the weak link. My Angular 2 web apps with an observable usually get a quick response back when calling Express against MongoDB. If I continue to have problems with Visual Studio I'm thinking that I may look to Java Spring REST as an alternative to Express. Java is still a big market and I don't want to stare at my laptop waiting for my back end tools to deliver services.

[vincywindy]

I use this simple code:

    class Program
    {
        static string url = "http://192.0.0.39:6277/check";
        static void Main(string[] args)
        {

            Do();
            Console.ReadLine();
        }
        public static async void Do()
        {
            while (true)
            {
                var g = Guid.NewGuid().ToString();
                Console.WriteLine(DateTime.Now.ToString()+ " 发送:" + g);
                var result= await HttpPostMath(url, g);
                Console.WriteLine(DateTime.Now.ToString()+" 接受:" + g);
                await Task.Delay(1000);
            }
        }
        public async static Task<string> HttpPostMath(string url, string paramsValue)
        {
            try
            {
                using (var client = new HttpClient())
                {
                    var content = new StringContent(paramsValue);
                    var response = await client.PostAsync(url, content);
                    var responseString = await response.Content.ReadAsStringAsync();
                    return responseString;
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
                return paramsValue;
            }
        }
    }

The server return the string that what I send.
Here is the result:
In Win10(I7 4770+16GBmen), it is no problem:

But in Win2008(E5-2650+16GBmen) it is slow:

This code accesses a LAN server without any complex operations.
I try another two Win10 and win server 2008 machine,Same result。
So I think something was wrong in Win7/2008 when use Http,Win10 doesn't have this problem.
By the way,when I use .net framework 4.6, 2008 and win10 both work well.

[EDIT] Fix code formatting by @karelz

[pieter-venter]

We were hit by this very same issue. We use Selenium extensively and noticed that on Windows 10 machines our tests slowed to a crawl when we switched to .Net core. What is strange is on virtual machines, running Windows 2016 they were still running really fast.

I found @jimevans's demo project and started experimenting.

I tried disabling IPv6 on the network adapter but it made no difference. It seems like "localhost" uses some special loopback adapter so disabling IPv6 on your other network adapters won't do anything!

This solved the problem for me:
https://superuser.com/questions/586144/disable-ipv6-loopback-on-windows-7-64-bit/681967#681967

I also confirmed that Windows 2016 servers already have this ::ffff:0:0/96 higher on the priority list, hence why they don't suffer from the 1000ms delay.

@jimevans It may be an option to make the ServerUrl in DriverService.cs read/write so that we can set it to 127.0.0.1 as a workaround.

[karelz]

Updated title
To sum up the problem: When using http://localhost, .NET Core can hit 1s timeout on Win10 resolving it to ::1 first, then to 127.0.0.1.

Workaround: Resolve IP yourself, don't use http://localhost.

Open question: Why is .NET Core different than .NET Framework? Can it be fixed?

[davidsh]

@Drawaes The server in question only listens on IPv4 in dual-stack environments. Attempting to change the URL to ::1 (predictably) yields a System.Net.WebException with the message, "An error occurred while sending the request. A connection with the server could not be established." Dns.GetHostEntry() returns both an IPv6 and an IPv4 address for the localhost host name in my environment. This leads me to three questions:

This is the root cause of the problem. Why does this server only listen on IPv4 addresses, especially for loopback? That is not optimal. And disabling IPv6 is not a good idea since IPv6 is really baked into Windows.

The problem, however, is not due to DNS resolution, per se. But rather trying to determine quickly which IP address to use for a given DNS name.

When a DNS name is resolved, there might be multiple results, i.e. multiple A records. Or in many cases, both A and AAAA (IPv6) records returned.

.NET Framework HTTP stack uses managed sockets and that has optimizations for parallel connection attempts on both IPv6 and IPv4 addresses for a given DNS name. In the old days, when IPv6 was new, some clients didn't support it. But if the DNS records return both IPv6 and IPv4 addresses, then each one needs to be tried before a connection is made. If IPv6 is enabled on the client machine (or server) then, the connect will fail (timeout). Then the IPv4 connection is tried. Doing this one at a time is slow. However, recent Windows OS versions can do the TCP connection for both IPv4 and IPv6 at the same time. And the first one that connects will win usually. So, it's much faster.

.NET Core on Windows uses a different HTTP stack (native WinHTTP). I think it is not using the winsock call that does parallel TCP connection lookups. So, that is why it might be slower in this case where IPv6 connections are disabled on the loopback listener side.

There is no way that I know of to disable how loopback resolves. It is built into the Windows TCP drivers. You can't suppress getting the "::1" and "127.0.0.1" addresses for a single "localhost" DNS lookup.

My recommendation on this is to make sure you listen on all loopback addresses (both IPv4 and IPv6).

[jimevans]

@davidsh

My recommendation on this is to make sure you listen on all loopback addresses (both IPv4 and IPv6).

While I don’t disagree that is the ideal solution, I feel compelled to reiterate that consumers of the .NET Core API of the project I hit this issue with have no control over how the server ends listen. The server in question can come from any of a list of major browser vendors (including Google via the Chromium project, Mozilla, Apple, and even Microsoft). If I could modify the behavior of the listening component, I absolutely agree that having it listen on all loopback addresses would be the correct option, but that is not the case.

[noobed7]

@davidsh

.NET Core on Windows uses a different HTTP stack (native WinHTTP). I think it is not using the winsock call that does parallel TCP connection lookups.

How do you know that? Could you point me to some code in the dotnet repository to see how things are laid out? I've tried to find any information on that, but with no luck.

[karelz]

@noobed7 see code in System.Net.Http and System.Net.Http.WinHttp:

https://github.com/dotnet/corefx/blob/c84ed430941fcf4e475b1149558498c55b4e295e/src/System.Net.Http/src/System.Net.Http.csproj#L184-L191

https://github.com/dotnet/corefx/tree/c84ed430941fcf4e475b1149558498c55b4e295e/src/System.Net.Http.WinHttpHandler/src/System/Net/Http

[karelz]

Triage: It seems there is not much we can do with current WinHTTP-based stack and there is workaround (use IP address / listen on both IPv4 and IPv6). We should be able to address the issue with ManagedHandler. Moving to Future.

[iSazonov]

@karelz Many thanks for the workaround!

[davidsh]

We should be able to address the issue with ManagedHandler. Moving to Future.

@geoffkizer @stephentoub Do we expect this issue to be resolved now with SocketsHttpHandler? Is it efficient in dealing with IPv4/IPv6 resolution especially IPv6 doesn't resolve correctly?

[stephentoub]

Is it efficient in dealing with IPv4/IPv6 resolution especially IPv6 doesn't resolve correctly?

SocketsHttpHandler simply relies on Socket.ConnectAsync:

https://github.com/dotnet/corefx/blob/ab8e81453fa869556821c464bd5824b588d679cd/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs#L57-L62

I've not tested in this context to see what impact that has on this case.

[geoffkizer]

Me either. I'm also not sure exactly what .net framework is doing here. This seems like something worth understanding better...

[karelz]

Triage: We believe it will be addressed by dotnet/corefx#29716.

[karelz]

Duplicate of dotnet/corefx#29716