Akka.Remote.EndpointException: Error while decoding incoming Akka PDU #3273

Aaronontheweb · 2018-01-16T16:28:39Z

Akka.NET v1.3.3 nightlies

Akka.Remote.EndpointException: Error while decoding incoming Akka PDU ---> Google.Protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
   at Google.Protobuf.CodedInputStream.SkipRawBytes(Int32 size)
   at Akka.Remote.Serialization.Proto.Msg.Payload.MergeFrom(CodedInputStream input)
   at Google.Protobuf.CodedInputStream.ReadMessage(IMessage builder)
   at Akka.Remote.Serialization.Proto.Msg.RemoteEnvelope.MergeFrom(CodedInputStream input)
   at Google.Protobuf.CodedInputStream.ReadMessage(IMessage builder)
   at Akka.Remote.Serialization.Proto.Msg.AckAndEnvelopeContainer.MergeFrom(CodedInputStream input)
   at Google.Protobuf.MessageExtensions.MergeFrom(IMessage message, ByteString data)
   at Google.Protobuf.MessageParser`1.ParseFrom(ByteString data)
   at Akka.Remote.Transport.AkkaPduProtobuffCodec.DecodeMessage(ByteString raw, RemoteActorRefProvider provider, Address localAddress)
   at Akka.Remote.EndpointReader.TryDecodeMessageAndAck(ByteString pdu)
   --- End of inner exception stack trace ---
   at Akka.Remote.EndpointReader.TryDecodeMessageAndAck(ByteString pdu)
   at Akka.Remote.EndpointReader.<Reading>b__11_1(InboundPayload inbound)
   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
   at Akka.Actor.UntypedActor.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Akka.Actor.ActorCell.HandleFailed(Failed f)
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)

Happened while the endpointManager actor was decoding a message.

Have also seen this occur on an endpointReader actor inside the same cluster:

[akka://SICluster/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2FSICluster%4010.0.17.97%3A10050-3442#554962129]
 Message
AssociationError [akka.tcp://SICluster@10.0.16.115:10050] <- akka.tcp://SICluster@10.0.17.97:10050: Error [Error while decoding incoming Akka PDU] [   at Akka.Remote.EndpointReader.TryDecodeMessageAndAck(ByteString pdu)
   at Akka.Remote.EndpointReader.<Reading>b__11_1(InboundPayload inbound)
   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
   at Akka.Actor.UntypedActor.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)]

The text was updated successfully, but these errors were encountered:

Aaronontheweb · 2018-01-16T22:58:01Z

cc @Horusiath looks like this was an issue with the wire format not being totally compatible between 1.3.2 and 1.3.3 once WeaklyUp is turned on. The cluster had some nodes running the 1.3.3 nightlies and others running 1.3.2 stable.

We had to modify the .proto files to support WeaklyUp, no?

Horusiath · 2018-01-17T07:03:59Z

@Aaronontheweb yes, new member status was added. However from what I was reading, it was marked as not breaking binary format compatibility.

Aaronontheweb · 2018-01-17T14:18:03Z

Yeah, that's what I thought upon looking at the changes to the .proto file too. Could also be that this cluster was using @nvivo's custom .NET Core / .NET Desktop intertop stuff.

nvivo · 2018-01-23T12:32:00Z

Some more info on this, I noticed this error appearing when having 2 different nightlies running: 1.3.3 beta-475 and another one from a few days ago, probably 472 or 470. After updating all nodes to the same version again, it stopped.

AndreSteenbergen · 2018-11-02T11:05:10Z

Can this still be an issue? I am on 1.3.9, and I am seeing this as well. Actorsystem running for about an hour (on 1 system) and it just seems to stop nodes as well.

AndreSteenbergen · 2018-11-02T11:24:08Z

Might be similar, or not related at all: (enable pooling was set to false (as seen on another similar issue))
[ERROR][11/2/18 11:16:19 AM][Thread 0008][remoting] Error while decoding incoming Akka PDU
Cause: Akka.Remote.EndpointException: Error while decoding incoming Akka PDU ---> Google.Protobuf.InvalidProtocolBufferException: SkipLastField called on an end-group tag, indicating that the corresponding start-group was missing

Aaronontheweb · 2018-11-02T18:00:29Z

This might still be an issue - mind submitting some information about your runtime environment @AndreSteenbergen ? It'd be helpful to know if it's Linux-specific or not.

AndreSteenbergen · 2018-11-02T19:21:20Z

Off course:
Ubuntu 16.04 LTS
Dotnet Core 2.0
Akka cluster 1.3.9 (until today the latest stable)

Problem is I can't seem to reproduce it simply, one time this happen after 5 minutes, another time oafter an hour, I am using MessagePack as serializer if that has anything to do with this

AndreSteenbergen · 2018-11-03T04:54:42Z

Can it be connected with gossip messages? I have let my cluster sit overnight without giving any tasks. I see this in the logs:

on one side (4062 port) I see this
[ERROR][11/3/18 4:52:07 AM][Thread 0006][[akka://claxe/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fclaxe%4010.0.0.30%3A4060-2/endpointWriter#1261779370]] AssociationError [akka.tcp://claxe@10.0.0.30:4062] <- akka.tcp://claxe@10.0.0.30:4060: Error [While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.] [ at Google.Protobuf.CodedInputStream.RefillBuffer(Boolean mustSucceed)

on the 4060 side I see this:

[ERROR][11/3/18 4:56:08 AM][Thread 0009][remoting] Error while decoding incoming Akka PDU
Cause: Akka.Remote.EndpointException: Error while decoding incoming Akka PDU ---> Google.Protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the mof a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.

AndreSteenbergen · 2018-11-03T05:02:28Z

If it is of any help, I migrated from Azure to a VPS the dotnet version is 2.0.9. Because of the dotnet core 2.1 issue. Can this also be a dotnetty thing?

AndreSteenbergen · 2018-11-03T07:19:01Z

Could it be my own deserializer? Not reading to the end?

[ERROR][11/3/18 7:14:53 AM][Thread 0008][remoting] Error while decoding incoming Akka PDU
Cause: Akka.Remote.EndpointException: Error while decoding incoming Akka PDU ---> Google.Protobuf.InvalidProtocolBufferException: Mismatched end-group tag. Started with field 12; ended with field 13
   at Google.Protobuf.CodedInputStream.SkipGroup(UInt32 startGroupTag)
   at Akka.Remote.Serialization.Proto.Msg.ActorRefData.MergeFrom(CodedInputStream input)
   at Google.Protobuf.CodedInputStream.ReadMessage(IMessage builder)
   at Akka.Remote.Serialization.Proto.Msg.RemoteEnvelope.MergeFrom(CodedInputStream input)
   at Google.Protobuf.CodedInputStream.ReadMessage(IMessage builder)
   at Akka.Remote.Serialization.Proto.Msg.AckAndEnvelopeContainer.MergeFrom(CodedInputStream input)
   at Google.Protobuf.MessageExtensions.MergeFrom(IMessage message, ByteString data)
   at Google.Protobuf.MessageParser`1.ParseFrom(ByteString data)
   at Akka.Remote.Transport.AkkaPduProtobuffCodec.DecodeMessage(ByteString raw, IRemoteActorRefProvider provider, Address localAddress)
   at Akka.Remote.EndpointReader.TryDecodeMessageAndAck(ByteString pdu)
   --- End of inner exception stack trace ---
   at Akka.Remote.EndpointReader.TryDecodeMessageAndAck(ByteString pdu)
   at Akka.Remote.EndpointReader.<Reading>b__11_1(InboundPayload inbound)
   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
   at Akka.Actor.UntypedActor.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Akka.Actor.ActorCell.HandleFailed(Failed f)
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)

AndreSteenbergen · 2018-11-03T07:25:26Z

Could this be an issue: I am also running squid on that machine. I used an old config from the machine I was migrating away from. It received constant forbiddens. Dotnet reported I was on ::1 instead of 127.0.0.1 I needed the extra acl. Could this be related? I mostly see these messages from local nodes, maybe an ipv6 issue somehow?

AndreSteenbergen · 2018-11-29T07:43:14Z

Is this related?

This is on a lighthouse instance, it's unlikely any of the messages from the actorsystem gets send to that instance.

[ERROR][11/29/18 7:34:31 AM][Thread 0009][remoting] While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
Cause: Google.Protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.

Later in the logs I see these kind of messages. To me it looks like a stream of data where one hick-up results in errors later in the stream. As these errors are quite contstant after a while. When I restart lighthouse, the errors seem to stop (for a while).

[ERROR][11/29/18 7:39:03 AM][Thread 0007][remoting] SkipLastField called on an end-group tag, indicating that the corresponding start-group was missing

[ERROR][11/29/18 7:35:09 AM][Thread 0007][remoting] Mismatched end-group tag. Started with field 6; ended with field 8

AndreSteenbergen · 2019-02-18T10:54:25Z

Just upgraded to .net core 2.2 on ubuntu 16.04. This issue came back. It is the Lighthouse service node which throws these messages. I have configured my lighthouse system to not create custom actor. So I don't really understand what is going on.

Aaronontheweb · 2019-02-18T11:03:44Z

I need to upgrade the lighthouse image - has to do with the old Akka.NET version running on it.

…

Sent from my iPhone

On Feb 18, 2019, at 11:54 AM, AndreSteenbergen ***@***.***> wrote: Just upgraded to .net core 2.2 on ubuntu 16.04. This issue came back. It is the Lighthouse service node which throws these messages. I have configured my lighthouse system to not create custom actor. So I don't really understand what is going on. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

AndreSteenbergen · 2019-02-18T13:29:47Z

Could this be an issue? I have set a max frame size of 256K

               dot-netty.tcp {
                        transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
                        applied-adapters = []
                        transport-protocol = tcp
                        #will be populated with a dynamic host-name at runtime if left uncommented
                        hostname = "0.0.0.0"
                        public-hostname = "10.0.0.31"
                        port = 10160
                        maximum-frame-size = 256000b
                }

Aaronontheweb · 2019-02-18T14:09:59Z

Nah, it's old DotNetty code. Just need to update the dependencies and rebuild the image.

…

Sent from my iPhone

On Feb 18, 2019, at 2:29 PM, AndreSteenbergen ***@***.***> wrote: Could this be an issue? I have set a max frame size of 256K dot-netty.tcp { transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote" applied-adapters = [] transport-protocol = tcp #will be populated with a dynamic host-name at runtime if left uncommented hostname = "0.0.0.0" public-hostname = "10.0.0.31" port = 10160 maximum-frame-size = 256000b } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

AndreSteenbergen · 2019-02-18T14:11:00Z

I am not using LightHouse from docker, I compiled one myself. I run native on linux, without docker.

Google.Protobuf.InvalidProtocolBufferException: Mismatched end-group tag. Started with field 12; ended with field 13

and:

Google.Protobuf.InvalidProtocolBufferException: SkipLastField called on an end-group tag, indicating that the corresponding start-group was missing

Aaronontheweb · 2019-02-18T14:38:50Z

What version of Akka.NET are you running on Lighthouse?

…

Sent from my iPhone

On Feb 18, 2019, at 3:11 PM, AndreSteenbergen ***@***.***> wrote: I am not using LightHouse from docker, I compiled one myself. I run native on linux, without docker. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

AndreSteenbergen · 2019-02-18T14:46:52Z

1.3.11

Aaronontheweb · 2019-02-18T16:38:33Z

That defeats my theory then

…

Sent from my iPhone

On Feb 18, 2019, at 3:46 PM, AndreSteenbergen ***@***.***> wrote: 1.3.11 Verzonden vanaf mijn Windows 10-apparaat Van: Aaron Stannard Verzonden: maandag 18 februari 2019 15:39 Aan: akkadotnet/akka.net CC: AndreSteenbergen; Mention Onderwerp: Re: [akkadotnet/akka.net] Akka.Remote.EndpointException: Error whiledecoding incoming Akka PDU (#3273) What version of Akka.NET are you running on Lighthouse? Sent from my iPhone > On Feb 18, 2019, at 3:11 PM, AndreSteenbergen ***@***.***> wrote: > > I am not using LightHouse from docker, I compiled one myself. I run native on linux, without docker. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

AndreSteenbergen · 2019-03-10T21:36:45Z

Issue resolved (I think) .... I Checked out LightHouse from the webcrawler example, with petabridge 0.4 something. Which is packed with Akka.Cluster 1.3.10, not 1.3.11. No build errors, because Cluster was part of the project. So I had a Cluster version mismatch ..... Sorry ..

Aaronontheweb · 2019-04-26T17:48:09Z

Related issue:

Exception thrown: 'Google.Protobuf.InvalidProtocolBufferException' in Google.Protobuf.dll
Additional information: Protocol message contained an invalid tag (zero). occurred

Google.Protobuf.dll!Google.Protobuf.CodedInputStream.ReadTag() + 0x1cd bytes
Akka.Remote.dll!Akka.Remote.Serialization.Proto.Msg.AckAndEnvelopeContainer.MergeFrom(Google.Protobuf.CodedInputStream input) + 0x12f bytes
Google.Protobuf.dll!Google.Protobuf.MessageExtensions.MergeFrom(Google.Protobuf.IMessage message, Google.Protobuf.ByteString data) + 0xa3 bytes
Google.Protobuf.dll!Google.Protobuf.MessageParser<Akka.Remote.Serialization.Proto.Msg.AckAndEnvelopeContainer>.ParseFrom(Google.Protobuf.ByteString data) + 0x84 bytes
Akka.Remote.dll!Akka.Remote.Transport.AkkaPduProtobuffCodec.DecodeMessage(Google.Protobuf.ByteString raw, Akka.Remote.IRemoteActorRefProvider provider, Akka.Actor.Address localAddress) + 0x6c bytes

Aaronontheweb · 2019-04-26T17:51:58Z

This looks to me like it has to be a message framing issue somewhere further up the food chain, but I'm skeptical about that because we've tested the hell out of the DotNetty message framing sitting in front of it.

Best idea is to probably add some additional Info logging inside the AkkaPduProtobuffCodec class on the type of message being deserialized when this happened.

Aaronontheweb · 2019-04-29T19:19:12Z

LOL welp, my fault - my issue was the result of a unit test I wrote intentionally injecting a mal-formed packet into the transport. Disregard my latest comments.

Aaronontheweb added confirmed bug akka-remote serialization labels Jan 16, 2018

Horusiath closed this as completed Jan 17, 2018

Horusiath reopened this Jan 17, 2018

Aaronontheweb mentioned this issue Aug 8, 2019

DotNetty.Codecs.CorruptedFrameException: negative pre-adjustment length field #3879

Closed

Aaronontheweb added this to the 1.4.0 milestone Feb 25, 2020

Aaronontheweb mentioned this issue Feb 25, 2020

disable buffer pooling in DotNetty transport #4252

Merged

Aaronontheweb closed this as completed in #4252 Feb 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Akka.Remote.EndpointException: Error while decoding incoming Akka PDU #3273

Akka.Remote.EndpointException: Error while decoding incoming Akka PDU #3273

Aaronontheweb commented Jan 16, 2018

Aaronontheweb commented Jan 16, 2018

Horusiath commented Jan 17, 2018

Aaronontheweb commented Jan 17, 2018

nvivo commented Jan 23, 2018

AndreSteenbergen commented Nov 2, 2018

AndreSteenbergen commented Nov 2, 2018

Aaronontheweb commented Nov 2, 2018

AndreSteenbergen commented Nov 2, 2018 •

edited

Loading

AndreSteenbergen commented Nov 3, 2018 •

edited

Loading

AndreSteenbergen commented Nov 3, 2018

AndreSteenbergen commented Nov 3, 2018

AndreSteenbergen commented Nov 3, 2018 •

edited

Loading

AndreSteenbergen commented Nov 29, 2018

AndreSteenbergen commented Feb 18, 2019

Aaronontheweb commented Feb 18, 2019 via email

AndreSteenbergen commented Feb 18, 2019

Aaronontheweb commented Feb 18, 2019 via email

AndreSteenbergen commented Feb 18, 2019 •

edited

Loading

Aaronontheweb commented Feb 18, 2019 via email

AndreSteenbergen commented Feb 18, 2019 via email •

edited

Loading

Aaronontheweb commented Feb 18, 2019 via email

AndreSteenbergen commented Mar 10, 2019

Aaronontheweb commented Apr 26, 2019

Aaronontheweb commented Apr 26, 2019

Aaronontheweb commented Apr 29, 2019

Akka.Remote.EndpointException: Error while decoding incoming Akka PDU #3273

Akka.Remote.EndpointException: Error while decoding incoming Akka PDU #3273

Comments

Aaronontheweb commented Jan 16, 2018

Aaronontheweb commented Jan 16, 2018

Horusiath commented Jan 17, 2018

Aaronontheweb commented Jan 17, 2018

nvivo commented Jan 23, 2018

AndreSteenbergen commented Nov 2, 2018

AndreSteenbergen commented Nov 2, 2018

Aaronontheweb commented Nov 2, 2018

AndreSteenbergen commented Nov 2, 2018 • edited Loading

AndreSteenbergen commented Nov 3, 2018 • edited Loading

AndreSteenbergen commented Nov 3, 2018

AndreSteenbergen commented Nov 3, 2018

AndreSteenbergen commented Nov 3, 2018 • edited Loading

AndreSteenbergen commented Nov 29, 2018

AndreSteenbergen commented Feb 18, 2019

Aaronontheweb commented Feb 18, 2019 via email

AndreSteenbergen commented Feb 18, 2019

Aaronontheweb commented Feb 18, 2019 via email

AndreSteenbergen commented Feb 18, 2019 • edited Loading

Aaronontheweb commented Feb 18, 2019 via email

AndreSteenbergen commented Feb 18, 2019 via email • edited Loading

Aaronontheweb commented Feb 18, 2019 via email

AndreSteenbergen commented Mar 10, 2019

Aaronontheweb commented Apr 26, 2019

Aaronontheweb commented Apr 26, 2019

Aaronontheweb commented Apr 29, 2019

AndreSteenbergen commented Nov 2, 2018 •

edited

Loading

AndreSteenbergen commented Nov 3, 2018 •

edited

Loading

AndreSteenbergen commented Nov 3, 2018 •

edited

Loading

AndreSteenbergen commented Feb 18, 2019 •

edited

Loading

AndreSteenbergen commented Feb 18, 2019 via email •

edited

Loading