v2.9.4: Only 4 nodes are supported in private net?

[efpies]

I'm trying to build NEO private net based on neo-privatenet-docker configuration with neo-cli v2.9.4.

I have 4 consensus nodes and 1 additional node, let's name it a wallet node. They all run in the same docker container.

When all 4 consensus nodes are running, wallet node can't connect because of this error:

[ERROR][02/28/2019 20:33:16][Thread 0005][akka://NeoSystem/system/IO-TCP/$e] Transport endpoint is not connected
Cause: System.Net.Sockets.SocketException (107): Transport endpoint is not connected
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.get_RemoteEndPoint()
   at Akka.IO.TcpConnection.CompleteConnect(IActorRef commander, IEnumerable`1 options)
   at Akka.IO.TcpOutgoingConnection.<>c__DisplayClass12_0.<Connecting>b__0(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[ERROR][02/28/2019 20:33:16][Thread 0005][akka://NeoSystem/system/IO-TCP/$b] Transport endpoint is not connected
Cause: System.Net.Sockets.SocketException (107): Transport endpoint is not connected
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.get_RemoteEndPoint()
   at Akka.IO.TcpConnection.CompleteConnect(IActorRef commander, IEnumerable`1 options)
   at Akka.IO.TcpOutgoingConnection.<>c__DisplayClass12_0.<Connecting>b__0(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[ERROR][02/28/2019 20:33:16][Thread 0005][akka://NeoSystem/system/IO-TCP/$d] Transport endpoint is not connected
Cause: System.Net.Sockets.SocketException (107): Transport endpoint is not connected
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.get_RemoteEndPoint()
   at Akka.IO.TcpConnection.CompleteConnect(IActorRef commander, IEnumerable`1 options)
   at Akka.IO.TcpOutgoingConnection.<>c__DisplayClass12_0.<Connecting>b__0(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[ERROR][02/28/2019 20:33:16][Thread 0005][akka://NeoSystem/system/IO-TCP/$c] Transport endpoint is not connected
Cause: System.Net.Sockets.SocketException (107): Transport endpoint is not connected
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.get_RemoteEndPoint()
   at Akka.IO.TcpConnection.CompleteConnect(IActorRef commander, IEnumerable`1 options)
   at Akka.IO.TcpOutgoingConnection.<>c__DisplayClass12_0.<Connecting>b__0(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[ERROR][02/28/2019 20:33:16][Thread 0004][akka://NeoSystem/system/IO-TCP/$d] Restarting not supported for connection actors.
Cause: [akka://NeoSystem/system/IO-TCP/$d#1133645249]: Akka.Actor.PostRestartException: Exception post restart (System.Net.Sockets.SocketException) ---> Akka.Pattern.IllegalStateException: Restarting not supported for connection actors.
   at Akka.IO.TcpConnection.PostRestart(Exception reason)
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.FinishRecreate(Exception cause, ActorBase failedActor)
   --- End of inner exception stack trace ---
[ERROR][02/28/2019 20:33:16][Thread 0004][akka://NeoSystem/system/IO-TCP/$b] Restarting not supported for connection actors.
Cause: [akka://NeoSystem/system/IO-TCP/$b#1459417403]: Akka.Actor.PostRestartException: Exception post restart (System.Net.Sockets.SocketException) ---> Akka.Pattern.IllegalStateException: Restarting not supported for connection actors.
   at Akka.IO.TcpConnection.PostRestart(Exception reason)
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.FinishRecreate(Exception cause, ActorBase failedActor)
   --- End of inner exception stack trace ---
[ERROR][02/28/2019 20:33:16][Thread 0004][akka://NeoSystem/system/IO-TCP/$e] Restarting not supported for connection actors.
Cause: [akka://NeoSystem/system/IO-TCP/$e#1918635427]: Akka.Actor.PostRestartException: Exception post restart (System.Net.Sockets.SocketException) ---> Akka.Pattern.IllegalStateException: Restarting not supported for connection actors.
   at Akka.IO.TcpConnection.PostRestart(Exception reason)
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.FinishRecreate(Exception cause, ActorBase failedActor)
   --- End of inner exception stack trace ---
[ERROR][02/28/2019 20:33:16][Thread 0004][akka://NeoSystem/system/IO-TCP/$c] Restarting not supported for connection actors.
Cause: [akka://NeoSystem/system/IO-TCP/$c#1574558595]: Akka.Actor.PostRestartException: Exception post restart (System.Net.Sockets.SocketException) ---> Akka.Pattern.IllegalStateException: Restarting not supported for connection actors.
   at Akka.IO.TcpConnection.PostRestart(Exception reason)
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.FinishRecreate(Exception cause, ActorBase failedActor)
   --- End of inner exception stack trace ---

If I stop any consensus node, the wallet node connects successfully. But after that, this consensus node isn't able to connect due to the same error. That means, only 4 nodes are being able to work together.

But if I downgrade all the way to neo-cli v2.9.0, everything is fine. Sometimes, this error pops up, but a neo-cli instance is able to recover after that. But v2.9.1+ have this issue. At the same time, I see that neo-privatenet-docker is built based on v2.9.0, so it must be a well-know issue preventing this repo from updating? I guess...

[jsolman]

Currently only 3 connections from the same ip are allowed. This is the reason for the behavior you are seeing.

[efpies]

Thanks, that explains why everything works perfectly when I use several containers instead of one.