Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProtocolException with iOS .NET 6.0 Application: The number of bytes available is inconsistent with the HTTP Content-Length header. #4903

Open
chamons opened this issue Sep 6, 2022 · 26 comments
Labels

Comments

@chamons
Copy link

chamons commented Sep 6, 2022

From @DuncWatts on Mon, 05 Sep 2022 12:20:03 GMT

I'm porting some code over from an existing Xamarin.Forms application however the WCF client I've written fails when run as a .NET 6.0 iOS application.

As WCF connected services don't work within iOS due to the restrictions on emitted code, I use a reference generated by the Silverlight tooling as a workaround and add System.ServiceModel.Http to use it.

Steps to Reproduce

I've found a random soap service to demonstrate this (http://www.dneonline.com/calculator.asmx?WSDL) and created a .NET Standard 2.0 class library that contains the WCF client and the code that calls it. I've then created a simple .NET 6.0 iOS application and a Xamarin.iOS application that executes that code that calls the webservice. In the Xamarin.iOS application it can correctly call the webservice while the .NET 6.0 iOS application fails with the stack trace below.

Expected Behavior

Web service is called correctly

Actual Behavior

Exception thrown when calling web service:

[0:] System.ServiceModel.ProtocolException: The number of bytes available is inconsistent with the HTTP Content-Length header.  There may have been a network error or the client may be sending invalid requests.
   at System.ServiceModel.Channels.HttpResponseMessageHelper.ReadBufferedMessageAsync(Task`1 inputStreamTask, TimeoutHelper timeoutHelper)
   at System.ServiceModel.Channels.HttpResponseMessageHelper.ParseIncomingResponse(TimeoutHelper timeoutHelper)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.<ReceiveReplyAsync>d__17[[System.ServiceModel.Channels.IRequestChannel, System.Private.ServiceModel, Version=4.10.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.ServiceModel.Channels.RequestChannel.RequestAsync(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.RequestAsyncInternal(Message message, TimeSpan timeout)
   at System.Runtime.AsyncResult.End[SendAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
   at System.ServiceModel.ClientBase`1.ChannelBase`1[[WcfExample.CalculatorSoap, WcfExample, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null],[WcfExample.CalculatorSoap, WcfExample, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].EndInvoke(String methodName, Object[] args, IAsyncResult result)
   at WcfExample.CalculatorSoapClient.CalculatorSoapClientChannel.EndAdd(IAsyncResult result) in C:\Users\me\source\repos\WcfExample\WcfExample\CalculatorService.cs:line 594
   at WcfExample.CalculatorSoapClient.WcfExample.CalculatorSoap.EndAdd(IAsyncResult result) in C:\Users\me\source\repos\WcfExample\WcfExample\CalculatorService.cs:line 258
   at System.Threading.Tasks.TaskFactory`1[[System.Int32, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].FromAsyncCoreLogic(IAsyncResult , Func`2 , Action`1 , Task`1 , Boolean )
--- End of stack trace from previous location ---
   at WcfExample.Example.AddAsync(Int32 a, Int32 b) in C:\Users\me\source\repos\WcfExample\WcfExample\Example.cs:line 14
   at WcfExample.iOS.Net6.AppDelegate.<>c.<<FinishedLaunching>b__4_0>d.MoveNext() in C:\Users\me\source\repos\WcfExample\WcfExample.iOS.Net6\AppDelegate.cs:line 29

Environment

Version information
Microsoft Visual Studio Enterprise 2022
Version 17.4.0 Preview 1.0
VisualStudio.17.Preview/17.4.0-pre.1.0+32804.182
Microsoft .NET Framework
Version 4.8.04084

Installed Version: Enterprise

Visual C++ 2022   00476-80000-00000-AA112
Microsoft Visual C++ 2022

ADL Tools Service Provider   1.0
This package contains services used by Data Lake tools

ASA Service Provider   1.0

ASP.NET and Web Tools   17.4.86.38009
ASP.NET and Web Tools

Azure App Service Tools v3.0.0   17.4.86.38009
Azure App Service Tools v3.0.0

Azure Data Lake Tools for Visual Studio   2.6.5000.0
Microsoft Azure Data Lake Tools for Visual Studio

Azure Functions and Web Jobs Tools   17.4.86.38009
Azure Functions and Web Jobs Tools

Azure Stream Analytics Tools for Visual Studio   2.6.5000.0
Microsoft Azure Stream Analytics Tools for Visual Studio

C# Tools   4.4.0-1.22403.13+ebbf56c257fb4d3128d3487ef525d92e9f94b412
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools   1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Extensibility Message Bus   1.4.1 (main@2ee106a)
Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

Microsoft Azure Hive Query Language Service   2.6.5000.0
Language service for Hive query

Microsoft Azure Stream Analytics Language Service   2.6.5000.0
Language service for Azure Stream Analytics

Microsoft Azure Tools for Visual Studio   2.9
Support for Azure Cloud Services projects

Microsoft JVM Debugger   1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Mono Debugging for Visual Studio   17.4.0 (d412e49)
Support for debugging Mono processes with Visual Studio.

NuGet Package Manager   6.4.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

Project System Tools   1.0
Tools for working with C#, VisualBasic, and F# projects.

Razor (ASP.NET Core)   17.0.0.2237602+b10895cdddd2286fe448ca44dfeb6f9c2c4f6abd
Provides languages services for ASP.NET Core Razor.

SQL Server Data Tools   17.0.62207.28050
Microsoft SQL Server Data Tools

ToolWindowHostedEditor   1.0
Hosting json editor into a tool window

TypeScript Tools   17.0.10727.2001
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools   4.4.0-1.22403.13+ebbf56c257fb4d3128d3487ef525d92e9f94b412
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools   17.4.0-beta.22376.4+0e8a2053cc2fc7d3801205a873e0afd3b6a1235e
Microsoft Visual F# Tools

Visual Studio IntelliCode   2.2
AI-assisted development for Visual Studio.

VisualStudio.DeviceLog   1.0
Information about my package

VisualStudio.Mac   1.0
Mac Extension for Visual Studio

VSPackage Extension   1.0
VSPackage Visual Studio Extension Detailed Info

Xamarin   17.4.0.95 (main@de68f64)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer   17.4.0.32 (remotes/origin/d17-4@439b92ed7b)
Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin Templates   17.4.1 (0b2b400)
Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms.

Xamarin.Android SDK   13.0.99.36 (main/b4998c8)
Xamarin.Android Reference Assemblies and MSBuild support.
    Mono: dffa5ab
    Java.Interop: xamarin/java.interop/main@032f1e71
    SQLite: xamarin/sqlite/3.39.2@40e8743
    Xamarin.Android Tools: xamarin/xamarin-android-tools/main@9c641b3


Xamarin.iOS and Xamarin.Mac SDK   15.13.0.6 (af0b05917)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

Build Logs

msbuild.zip

Example Project (If Possible)

WcfExample.zip

Copied from original issue xamarin/xamarin-macios#15866

@chamons
Copy link
Author

chamons commented Sep 6, 2022

From @chamons on Tue, 06 Sep 2022 15:07:56 GMT

Given all of the assemblies in the stack trace are BCL (Base Class Library), in particular System.ServiceModel.*, I believe this should be looked at by the dotnet folks first.

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@danmoseley danmoseley transferred this issue from dotnet/runtime Sep 6, 2022
@danmoseley
Copy link
Member

I think this belongs in dotnet/wcf

@DuncWatts
Copy link

Note this works fine in a .NET 6 WPF application, it's only the .NET 6 iOS build that has this issue so far. I've yet to try it on Android or MacOS.

@danmoseley
Copy link
Member

cc @radical @steveisok since this may be iOS specific and therefore not a WCF issue - perhaps they have seen something similar.

@chamons
Copy link
Author

chamons commented Sep 6, 2022

Given the lack of any Xamarin iOS types in the stack is the reason I bounced it over to the runtime at first.

@chamons
Copy link
Author

chamons commented Sep 6, 2022

I just checked, and this works fine on macOS Desktop with NET6.

@chamons
Copy link
Author

chamons commented Sep 6, 2022

This appears to be the location of the exception.

@steveisok
Copy link
Member

@simonrozsival Please give this a look and see if you can find anything.

@simonrozsival
Copy link
Member

I managed to reproduce the issue locally. When the exception that @chamons pointed out is throw, the state is:

  • bytesRead == 0
  • _contentLength == 348
  • offset == 325
  • count == 23
  • buffer contains 325 bytes corresponding to a UTF-8 encoded string <?xml version="1.0" encoding="utf-8"?><soap:Envelope ...</soap:Envelope>

The raw HTTP response is:

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/10.0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
X-Powered-By-Plesk: PleskWin
Date: Thu, 08 Sep 2022 07:57:57 GMT
Content-Length: 348

<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><AddResponse xmlns="http://tempuri.org/"><AddResult>3</AddResult></AddResponse></soap:Body></soap:Envelope>

As far as I can tell the actual size of the body is 325 bytes. I think the ProtocolException is the correct behavior in this case because the server responded with incorrect Content-Length (even though it is not consistent with legacy Xamarin which seems to ignore this mismatch). The code in WCF that checks the mismatch is 7 years old and the ReadAsync code in NSUrlSessionHandler+NSUrlSessionDataTaskStream hasn't changed in 4 years so this doesn't seem like a recent regression.

@chamons can you share the macOS desktop project so I can try to check why that behaves differently?

@DuncWatts
Copy link

@simonrozsival I've added a WPF .NET 6 application to my example that isn't throwing a ProtocolException if that's of any help.

WcfExampleWithWPF.zip

@DuncWatts
Copy link

I've run Fiddler against the WPF build and in this instance it's returning Content-Length: 325 so assume it must be something in how it's making the request.

Request headers:

POST /calculator.asmx HTTP/1.1
Host: www.dneonline.com
Cache-Control: no-cache, max-age=0
SOAPAction: "http://tempuri.org/Add"
Accept-Encoding: gzip, deflate
Content-Type: text/xml; charset=utf-8
Content-Length: 161

Response headers:

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=utf-8
Vary: Accept-Encoding
Server: Microsoft-IIS/10.0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
X-Powered-By-Plesk: PleskWin
Date: Thu, 08 Sep 2022 09:50:39 GMT
Content-Length: 325

@simonrozsival
Copy link
Member

simonrozsival commented Sep 8, 2022

@DuncWatts That's interesting. I ran the request a couple more times and I got three different Content-Length values for the same request (identical headers to yours) with a body length of 325 bytes: 262, 325, 348. It seems there's something wrong with that server.

Anyway, I found a workaround that might help. If you can, add this property to your iOS project: <UseNativeHttpHandler>false</UseNativeHttpHandler>. If you depend on the behavior of NSUrlSessionHandler somewhere, you can will need to instantiate it yourself and pass it to HttpClient explicitly through its constructor. This may not be suitable for all projects though.

The SocketsHttpHandler, that is used when the native one is disabled, behaves differently: the response's content.Headers.ContentLength doesn't have a value (even though the response contains it) so the ReadChunkedBufferedMessageAsync method is called instead of ReadBufferedMessageAsync. The ReadChunkedBufferedMessageAsync method ignores the Content-Length header and reads the whole response body.

if (!content.Headers.ContentLength.HasValue)
{
return ReadChunkedBufferedMessageAsync(contentStreamTask, timeoutHelper);
}
return ReadBufferedMessageAsync(contentStreamTask, timeoutHelper);

I'm not sure if this is the desired behavior of SocketsHttpHandler or not, but it's definitely an inconsistency between the platforms. There were some improvements to the Content-Length behavior recently (for example dotnet/runtime#62541) so I'll try it with the latest preview .NET 7 later as well.

EDIT: So I ran the example in a console app with the latest changes to the SocketsHttpHandler in .NET 7 P7 and the quirk with missing ContentLength in the response is still there.

@simonrozsival
Copy link
Member

@karelz is the behavior of SocketsHttpHandler and the HttpResponseMessage it returns described above the expected behavior or is that a bug?

@mconnew
Copy link
Member

mconnew commented Sep 13, 2022

WCF has the ability to enable modifying the HttpMessageHandler stack. There's a blog post here that demonstrates how to modify the behavior of the HttpClientHandler where you can wrap it if needed. If you need to keep using the native implementation, you could wrap it with an implementation which removes the Content-Length header once the response has been received before returning the response message to WCF. This will cause WCF to switch to a mode where it keeps reading the content until it either reaches the end, or it hits the configured MaxReceivedMessageSize byte limit.

@mconnew
Copy link
Member

mconnew commented Sep 13, 2022

@simonrozsival, @DuncWatts, @karelz, which repo is the best place to track this? I think it's been sufficiently investigated to know this is not an issue WCF's implementation or usage of HttpClient.

@simonrozsival
Copy link
Member

If @karelz confirms this is a bug then we should move it to dotnet/runtime.

@karelz
Copy link
Member

karelz commented Sep 14, 2022

@simonrozsival can you please restate what exactly is the SocketsHttpHandler behavior that you suspect may be a bug? (I tried to gather it from previous comment, but it was not 100% clear to me sadly :() Thanks!

If we find a bug in some component, I would recommend to create a new bug with specific details instead of this one which is IMO not super-clear at the moment. (there is of course always an option it is just me being slow ;))

@simonrozsival
Copy link
Member

@karelz when HttpClient is used with SocketsHttpHandler, the response message's content.Headers.ContentLength doesn't have a value even though the HTTP response contained the Content-Length header. This doesn't happen when NSUrlSessionHandler is used instead. I think I'll just go ahead and create a new issue in dotnet/runtime with the details.

@karelz
Copy link
Member

karelz commented Sep 14, 2022

Yeah, that looks like a bug to me.
Please file it.
With info if it is specific to iOS, or general. And ideally with a standalone repro for us to act on. Thanks!
cc @MihaZupan @ManickaP @CarnaViire

@DuncWatts
Copy link

Removing the Content-Length header using a DelegatingHandler works a charm thank you. The previous workaround of setting UseNativeHttpHandler to false did work but triggered a lot of errors with the linker which I was struggling to resolve.

@simonrozsival
Copy link
Member

tl;dr: The root of the problem is gzip compression of the response. The server works correctly. It's not a bug in .NET. It's possibly a bug in WCF and/or NSUrlSessionHandler.

@karelz I won't open any issue in dotnet/runtime after all. Sorry for the confusion 😄


I did some more digging and I realized that my previous conclusions were incorrect. I overlooked the fact that the response is gzipped and so the Content-Length contains the size of the compressed payload (interesting in this case the gzipped payload is 348 B instead of the original 325 B). WCF sets the AutomaticDecompression property of the handler and it leaves all the decompression implementation details to the handler.

The .NET implementation (``) makes the content.Headers.ContentLength value inaccessible, while the `NSUrlSessionHandler` implementation returns the size of the compressed payload. WCF assumes that when the `ContentLength` value is available, it's the size of the plaintext content and not necessarily the exact value of the `Content-Length` header value.

@mconnew The docs of the ContentLength property say that it should contain the value of Content-Length and the Mac/iOS implementation sticks to the exact wording of the docs. I think WCF should ignore the value if the payload was compressed instead of relying on .NET's internal implementation. Or is there some documentation that would suggest it's a bug in the Mac/iOS implementation.

@MihaZupan
Copy link
Member

wcf shouldn't be validating Content-Length here, it's either redundant or just wrong.

If decompression happened, or the server returned both the content length and chunked transfer encoding, or some handler somewhere changed the content/headers, this verification logic will be wrong.

SocketsHttpHandler will enforce the correctness of the response content length (it has to - it's the core of the protocol).
If the server sent us too many bytes, you won't see them when reading the content stream.
If the server sent us too few bytes, the ReadAsync on the content stream will throw.

@MihaZupan
Copy link
Member

Ah, I see you just commented that decompression was indeed the issue.
This should be fixed in wcf. You should let the underlying handler handle the protocol correctness.
WCF shouldn't be enforcing the value of the Content-Length as it can be different from the number of bytes read from the content.

@simonrozsival simonrozsival removed their assignment Sep 14, 2022
@mconnew
Copy link
Member

mconnew commented Sep 14, 2022

What was the behavior with HttpWebRequest? We are using the exact same logic on .NET Framework.

@psudodev
Copy link

I am facing similar issue on Maui Android on .net 7 @chamons did you find any work around for this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

10 participants