SocketsHttpHandler retries with useless "Authorization" header when SystemNetworkCredential is in use

[mstefarov]

Description

When SocketsHttpHandler is configured to use CredentialCache.DefaultCredentials (i.e. SystemNetworkCredential), it will automatically retry with a useless "Authorization" header in response to WWW-Authenticate challenges. The content of the header is Og== (base-64 encoded colon character), meaning empty username and empty password.

The code in AuthenticationHelper that adds a "no username / no password" header ^

The auto-retry request seen in Fiddler ^

Reproduction Steps

using System.Net;
var handler = new SocketsHttpHandler();
handler.Credentials = CredentialCache.DefaultCredentials;

using var invoker = new HttpMessageInvoker(handler);
var request = new HttpRequestMessage(HttpMethod.Get, "https://www.geoportal.ch/services/wms/benken/wasserkorporation_benken/authenticate?service=WMS&request=GetCapabilities");
var response = await invoker.SendAsync(request, CancellationToken.None); // This causes 2 requests instead of 1

Expected behavior

SystemNetworkCredential should be special-cased when challenged for Basic and Digest authentication, because it only has meaning for NTLM and Negotiate. Only 1 request should be made and the original 401 response returned.

Actual behavior

An extra request is made. This request is unexpected and has no chance of succeeding. Response for the original (authorization-free) request is lost, and only the second response is returned -- which complicates working with some services that return a different response depending on whether or not request included "Authorization" header at all.

Regression?

The issue is not present in other HttpMessageHandler implementations, such as WebRequestHandler (.NET Framework) or HttpClientHandler (UWP). I am not sure if it was present in previous versions of dotnet core.

Known Workarounds

No response

Configuration

My repro targets net8.0-windows10.0.19041, runs on .NET 8.0.13, and is built with VS 17.13.2

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[MihaZupan]

The suggestion to ignore DefaultCredentials when doing password-based auth sounds reasonable to me.
I don't know that a server would care in practice to distinguish between no creds vs. empty creds.

@rzikm @wfurt thoughts?

[wfurt]

The Remark already says it is applicable only to NTLM & Kerberos
https://learn.microsoft.com/en-us/dotnet/api/system.net.credentialcache.defaultcredentials?view=net-8.0#system-net-credentialcache-defaultcredentials

I was under impression we had check for that somewhere but I guess not. By quick search this also impacts connection pool as we use separate pools for separate identities. We probably should doing that - just avoiding at extra attempt.

[MihaZupan]

Triage: Not critical for .NET 10 since this has been the behavior of SocketsHttpHandler since the start.
Changing the behavior as described makes sense, we'd welcome a contribution here.