Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ClusterReceptionist, RingOrdering - Unable to sort because the IComparer.Compare() method returns inconsistent results. #3840

Closed
hhko opened this issue Jul 7, 2019 · 4 comments · Fixed by #3866
Closed

ClusterReceptionist, RingOrdering - Unable to sort because the IComparer.Compare() method returns inconsistent results. #3840

hhko opened this issue Jul 7, 2019 · 4 comments · Fixed by #3866
Assignees
@Aaronontheweb
Labels
akka-cluster-tools confirmed bug critical
Milestone
1.4.0

Comments

@hhko
Copy link

hhko commented Jul 7, 2019 •
edited
Loading

  • Akka 1.3.13

After #2535 issue occurs, RingOrdering throws an exception(IComparer.Compare) repeatedly.
As a result, The 'Akka.Cluster.Tools.Client.ClusterReceptionist' is continuously restarting and Memory is increasing up to 20G.

[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c77] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c77] 

[2019/07/01 16:24:14.384][  212][ERROR]akka://mysystem/system/receptionist[Akka.Actor.OneForOneStrategy] An entry with the same key already exists. System.ArgumentException: An entry with the same key already exists.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
   at System.Collections.Generic.SortedDictionary`2.Add(TKey key, TValue value)
   at Akka.Routing.ConsistentHash.Create[T](IEnumerable`1 nodes, Int32 virtualNodesFactor)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c77] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c78] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c78] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c78] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c79] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c79] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c79] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c80] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c81] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c82] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c80] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.41:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c80] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c81] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c81] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c82] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c82] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c83] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c83] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c83] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c84] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c84] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c84] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c85] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c85] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c85] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c86] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c86] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c86] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c87] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c87] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c87] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c88] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c88] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c88] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c89] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c89] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c89] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c90] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c90] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c90] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c91] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c91] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c91] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c92] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c92] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c92] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c93] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c93] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c93] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c94] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c94] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c94] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c95] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c95] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c95] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c96] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c96] 
[2019/07/01 16:24:14.384][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c96] 
[2019/07/01 16:24:14.384][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c97] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c97] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c97] 
[2019/07/01 16:24:14.404][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c98] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c98] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c98] 
[2019/07/01 16:24:14.404][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c99] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c99] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c99] 
[2019/07/01 16:24:14.404][  212][DEBUG]LocalActorRefProvider(akka://mysystem)[LocalActorRefProvider(akka://mysystem)] [akka://mysystem/] Instantiating Remote Actor [akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c100] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/user/ApiMasterActor/singleton/InferenceManageActor -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c100] 
[2019/07/01 16:24:14.404][  212][DEBUG][akka://mysystem/system/remote-watcher#1726918254][Akka.Cluster.ClusterRemoteWatcher] Watching: [akka://mysystem/system/remote-deployment-watcher -> akka.tcp://mysystem@100.100.100.43:8931/remote/akka.tcp/mysystem@100.100.100.44:4064/user/ApiMasterActor/singleton/InferenceManageActor/c100] 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarting 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarted (Akka.Cluster.Tools.Client.ClusterReceptionist) 

[2019/07/01 16:24:14.404][  212][ERROR]akka://mysystem/system/receptionist[Akka.Actor.OneForOneStrategy] Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'. System.ArgumentException: Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'.
   at System.Collections.Generic.IntrospectiveSortUtilities.ThrowOrIgnoreBadComparer(Object comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at System.Collections.Immutable.ImmutableSortedSet`1.LeafToRootRefill(IEnumerable`1 addedItems)
   at System.Collections.Immutable.ImmutableSortedSet`1.Union(IEnumerable`1 other)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarting 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarted (Akka.Cluster.Tools.Client.ClusterReceptionist) 

[2019/07/01 16:24:14.404][  212][ERROR]akka://mysystem/system/receptionist[Akka.Actor.OneForOneStrategy] Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'. System.ArgumentException: Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'.
   at System.Collections.Generic.IntrospectiveSortUtilities.ThrowOrIgnoreBadComparer(Object comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at System.Collections.Immutable.ImmutableSortedSet`1.LeafToRootRefill(IEnumerable`1 addedItems)
   at System.Collections.Immutable.ImmutableSortedSet`1.Union(IEnumerable`1 other)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarting 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mysystem/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarted (Akka.Cluster.Tools.Client.ClusterReceptionist) 

[2019/07/01 16:24:14.404][  212][ERROR]akka://mysystem/system/receptionist[Akka.Actor.OneForOneStrategy] Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'. System.ArgumentException: Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'.
   at System.Collections.Generic.IntrospectiveSortUtilities.ThrowOrIgnoreBadComparer(Object comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at System.Collections.Immutable.ImmutableSortedSet`1.LeafToRootRefill(IEnumerable`1 addedItems)
   at System.Collections.Immutable.ImmutableSortedSet`1.Union(IEnumerable`1 other)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
@hhko hhko changed the title Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering - Unable to sort because the IComparer.Compare() method returns inconsistent results. ClusterReceptionist, RingOrdering - Unable to sort because the IComparer.Compare() method returns inconsistent results. Jul 7, 2019
@Aaronontheweb
Copy link
Member

@hhko thanks! We'll look into this.

@Aaronontheweb Aaronontheweb self-assigned this Jul 7, 2019
@kimbyungeun
Copy link

Me and @hhko is co-worker.

Let me explain about our environment.

  1. We operate many services with Akka.Cluster.
  2. Use Custer Singleton for Actor distribution.
  3. Use the Cluster Client to send messages outside the cluster.

There is an additional explanation for the error.
[2019/07/01 16:24:14.404] More than 26 error messages were output at the same time.
Most services consumed more than 20GB of memory as the same errors were output continuously.

[2019/07/01 16:24:14.404][  212][DEBUG]akka://mls-cluster-system/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarting 
[2019/07/01 16:24:14.404][  212][DEBUG]akka://mls-cluster-system/system/receptionist[Akka.Cluster.Tools.Client.ClusterReceptionist] Restarted (Akka.Cluster.Tools.Client.ClusterReceptionist) 
[2019/07/01 16:24:14.404][  212][ERROR]akka://mls-cluster-system/system/receptionist[Akka.Actor.OneForOneStrategy] Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'. System.ArgumentException: Unable to sort because the IComparer.Compare() method returns inconsistent results. Either a value does not compare equal to itself, or one value repeatedly compared to another value yields different results. IComparer: 'Akka.Cluster.Tools.Client.ClusterReceptionist+RingOrdering'.
   at System.Collections.Generic.IntrospectiveSortUtilities.ThrowOrIgnoreBadComparer(Object comparer)
   at System.Collections.Generic.GenericArraySortHelper`1.Sort(T[] keys, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
   at System.Collections.Immutable.ImmutableSortedSet`1.LeafToRootRefill(IEnumerable`1 addedItems)
   at System.Collections.Immutable.ImmutableSortedSet`1.Union(IEnumerable`1 other)
   at Akka.Cluster.Tools.Client.ClusterReceptionist.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)

@Aaronontheweb
Copy link
Member

Thanks folks - I'm going to get to work on fixing this soon. Flying back from India to the USA at the moment, but we'll take care of this. Sorry for not following up sooner - been traveling continuously since this issue was filed.

@Aaronontheweb
Copy link
Member

Figured this out in #2535 - it's the comparer that's bad. Working on a multi-node test to verify the fix and to ensure that old nodes properly get pruned out of the state.

@Aaronontheweb Aaronontheweb mentioned this issue Jul 26, 2019
Merged
Aaronontheweb added a commit that referenced this issue Jul 26, 2019
@Aaronontheweb
ClusterClient fixes (#3866)
211db8e 
Fixed broker `IComparer` for ClusterClient hash ring and ported over other handoff fixes.

Close #2535
Close #2312
Close #3840

* implemented akka/akka#24167

* implemented akka/akka#22992
@Aaronontheweb Aaronontheweb added this to the 1.4.0 milestone Jul 26, 2019
Aaronontheweb added a commit to Aaronontheweb/akka.net that referenced this issue Jul 26, 2019
@Aaronontheweb
ClusterClient fixes (akkadotnet#3866)
17386d3 
Fixed broker `IComparer` for ClusterClient hash ring and ported over other handoff fixes.

Close akkadotnet#2535
Close akkadotnet#2312
Close akkadotnet#3840

* implemented akka/akka#24167

* implemented akka/akka#22992
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Aaronontheweb Aaronontheweb

Labels
akka-cluster-tools confirmed bug critical
Projects
None yet
Milestone
1.4.0
Development

Successfully merging a pull request may close this issue.

ClusterClient fixes Aaronontheweb/akka.net
3 participants
@Aaronontheweb @hhko @kimbyungeun