Ignore invalid response header in HttpClient request

[EricXiexm]

In .net standard 2.0
When i use System.Net.Http.HttpClient to request url data,throw an exception like:

The url run in web browser work well,but its response headers like:

The reponse headers is not standardized,in .net core it will throw an exception,but in framework4.5 work well.

Any solution for this in .net core.

[davidsh]

The reponse headers is not standardized,in .net core it will throw an exception,but in .NET Framework 4.5 works well.

cc: @dotnet/ncl @stephentoub

This looks like another case where we need to adjust SocketsHttpHandler behavior to ignore (with logging) malformed header lines and not reject the response entirely.

[scalablecory]

I've just tested PlatformHandler/WinHTTP, it does not allow this. Will check Framework next.

[scalablecory]

I can't replicate this in 4.7.2 either. It gives the following misleading error (CR is indeed followed by LF):

WebException: The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF

However, Chrome, Firefox, and Curl do allow it. I think it would be reasonable (and trivial) to allow it in SocketsHttpHandler. Thoughts, @davidsh @wfurt @stephentoub?

[davidsh]

WebException: The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF

I'm curious if the parsing will work if you add the 'UseUnsafeHeaderParsing' config item. It affects .NET Framework HttpWebRequest stack and in theory will affect HttpClient. You need to put this into the app.config for .NET Framework application.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.7.2" />
    </startup>
  <system.net>
    <settings>
      <httpWebRequest useUnsafeHeaderParsing ="true"/>
    </settings>
  </system.net>
</configuration>

https://docs.microsoft.com/en-us/dotnet/api/system.net.configuration.httpwebrequestelement.useunsafeheaderparsing?view=netframework-4.8

However, Chrome, Firefox, and Curl do allow it. I think it would be reasonable (and trivial) to allow it in SocketsHttpHandler. Thoughts, @davidsh @wfurt @stephentoub?

However, if this isn't a regression in .NET Core 3.0 compared with .NET Core 2.x and if this doesn't easily repro in .NET Framework without extra config settings, then this wouldn't meet the bar for a .NET Core 3.0 change.

Also, see related issue about malformed CR-LF sequences: #25319

[scalablecory]

Works with useUnsafeHeaderParsing="true".

Does not work on 2.2. Moving to Future.

[karelz]

Triage: We should create a new setting or something like that (likely not API). Bonus: Use same switch name as .NET Framework has.
It will match .NET Framework and also browsers behavior (which @scalablecory tried).

In this specific case we should either ignore the line with date without header name (likely what .NET Framework does) or use the text prior to column ':' as key and the rest as value.

[FullstackSensei]

Having a similar issue with a webservice where the response headers contain a header with a '=' instead of '-' in the header name.

Any possible workarounds in Core 2.X/3.0 until the useUnsafeHeaderParsing switch is implemented? Tried with both System.Net.WebRequest and System.Net.Http.HttpClient, and get the same error.

[ChrisVanDijk]

I'm trying to connect to a State Server for storing data (yes, I know it's better to use SQL Server or Redis cache, but I can't install anything on the server or create tables on the SQL Server, and I need to survive application restarts), but the state server responds with an invalid header, it doesn't send the HTTP1.1 part, but just '200 OK'. Because of that, HttpClient throws an exception:

'Received an invalid status line: '200 OK'.'

So I need a way to ignore those errors. In the meantime I use TcpClient to send and receive the data, but I rather use HttpClient.

[karelz]

@ChrisVanDijk that is not invalid header, that is invalid response format of the first line.
This is not something covered by this issue.

Personally I don't think it is something we should be resilient to, unless we find out it is super-wide spread.
Do you know what the server is? Is there a chance to fix it instead?

[ChrisVanDijk]

@karelz it't the good old ASP.NET State Service, according to the documentation it should respond with a standard response format, but it doesn't. Maybe that's a bug, or they did it on purpose.

I don't think this happens a lot, and I hope nobody connects to the state service outside of the old ASP.NET world. I shouldn't use it either, but for this project I don't have another option. I've fixed it by using TcpClient, which works fine for cases like this.