-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DotNetty.Codecs.CorruptedFrameException: negative pre-adjustment length field #3879
Comments
Hi @leonelcm We've had some other reports similar to this one and they've been difficult to reproduce, the biggest reason being that we don't even know what the message type is at this stage in the decoding pipeline. The reason why this is happening is because the 4 byte header we append to the front of each message (this is a standard socket message encoding technique) and in this instance the header appears to have some bad information in it. Here's the line that threw: This could have happened because this piece of math that happened just before: int actualLengthFieldOffset = input.ReaderIndex + this.lengthFieldOffset;
long frameLength = this.GetUnadjustedFrameLength(input, actualLengthFieldOffset, this.lengthFieldLength, this.byteOrder); Was wrong. I think what is likely happening in this case is that the |
Going forward, I can address this in one of two ways:
|
Wonder if these are related #3273 |
@Aaronontheweb, thank you very much for your feedback. Would the messages be dropped when this issue occur? What is the impact of these issues? |
I think this is going to be difficult to locate, specifically, without a really good reproduction spec, but I ran into this myself yesterday while working on some tests with Akka.Remote. I think the issue in this case is that DotNetty, for whatever reason, can occasionally corrupt frames using its decoding scheme (length frame.) I also think it has to do with how the input buffers are being re-used inside DotNetty as well. We're about to embark on a number of issues for Akka.NET v1.4.0 that might render this moot - namely moving off of DotNetty and implementing Artery for Akka.NET TCP. |
After running some detailed tests as part of #4106 I've determined that this is ultimately an issue with the WorkaroundThere is a temporary work-around for this problem which should have a fairly minimal performance impact (as determined by my numbers):
This will disable buffer pooling altogether and thus work around the source of this issue. SolutionThe solution is to programmatically either fix our use of reference-counting ( |
included this into Akka.NET v1.3.18: https://github.com/akkadotnet/akka.net/releases/tag/1.3.18 |
- Which Akka.Net version you are using
Akka.net: 1.3.13
DotNetty: 0.6.0
- On which platform you are using Akka.Net
Windows
- A list of steps to reproduce the issue. Or an gist or github repo which can be easily used to reproduce your case.
The following exceptions were generated last night while we were processing messages:
Error caught channel "[::ffff:165.33.24.232]:8910"->"[::ffff:165.34.44.22]:35690"
{
"Depth": 0,
"ClassName": "DotNetty.Codecs.CorruptedFrameException",
"Message": "negative pre-adjustment length field: -160",
"Source": "DotNetty.Codecs",
"StackTraceString": " at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input)\r\n at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input, List
1 output)\r\n at DotNetty.Codecs.ByteToMessageDecoder.CallDecode(IChannelHandlerContext context, IByteBuffer input, List
1 output)\r\n at DotNetty.Codecs.ByteToMessageDecoder.ChannelRead(IChannelHandlerContext context, Object message)\r\n at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelRead(Object msg)","RemoteStackTraceString": null,
"RemoteStackIndex": 0,
"ExceptionMethod": {
"Name": "Decode",
"AssemblyName": "DotNetty.Codecs",
"AssemblyVersion": "0.6.0.0",
"AssemblyCulture": "",
"ClassName": "DotNetty.Codecs.LengthFieldBasedFrameDecoder",
"Signature": "System.Object Decode(DotNetty.Transport.Channels.IChannelHandlerContext, DotNetty.Buffers.IByteBuffer)",
"MemberType": 8
},
"HResult": -2146233088,
"HelpURL": null
}
No response from remote for inbound association. Handshake timed out after [15000] ms
Error caught channel "[::ffff:165.33.24.232]:8910"->"[::ffff:165.34.44.22]:53650"
{
"Depth": 0,
"ClassName": "DotNetty.Codecs.TooLongFrameException",
"Message": "Adjusted frame length exceeds 50000: 721008 - discarded",
"Source": "DotNetty.Codecs",
"StackTraceString": " at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Fail(Int64 frameLength)\r\n at DotNetty.Codecs.LengthFieldBasedFrameDecoder.FailIfNecessary(Boolean firstDetectionOfTooLongFrame)\r\n at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input)\r\n at DotNetty.Codecs.LengthFieldBasedFrameDecoder.Decode(IChannelHandlerContext context, IByteBuffer input, List
1 output)\r\n at DotNetty.Codecs.ByteToMessageDecoder.CallDecode(IChannelHandlerContext context, IByteBuffer input, List
1 output)\r\n at DotNetty.Codecs.ByteToMessageDecoder.ChannelRead(IChannelHandlerContext context, Object message)\r\n at DotNetty.Transport.Channels.AbstractChannelHandlerContext.InvokeChannelRead(Object msg)","RemoteStackTraceString": null,
"RemoteStackIndex": 0,
"ExceptionMethod": {
"Name": "Fail",
"AssemblyName": "DotNetty.Codecs",
"AssemblyVersion": "0.6.0.0",
"AssemblyCulture": "",
"ClassName": "DotNetty.Codecs.LengthFieldBasedFrameDecoder",
"Signature": "Void Fail(Int64)",
"MemberType": 8
},
"HResult": -2146233088,
"HelpURL": null
}
Unfortunately I can't reproduce the issues as this happened in PROD and we process millions of messages a day. Any thoughts on what could be the issue and could be the impact?
Thank you,
Leonel
The text was updated successfully, but these errors were encountered: