Akka.Cluster.Sharding should automatically reference Akka.Cluster.DData configuration

[Aaronontheweb]

Version: 1.3.8

Ran into the following issue in a cluster when akka.cluster.sharding.state-store-mode=ddata

[ERROR][06/26/2018 14:58:13][Thread 0015][remoting] Cannot find serializer with id [12]. The most probable reason is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems.
Cause: System.Runtime.Serialization.SerializationException: Cannot find serializer with id [12]. The most probable reason is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems.

The serializer with ID 12 is one of the two DData serializers:

akka.net/src/contrib/cluster/Akka.DistributedData/reference.conf

Lines 88 to 101 in 41c116d

	akka.actor {
	serializers {
	akka-data-replication = "Akka.DistributedData.Serialization.ReplicatorMessageSerializer, Akka.DistributedData"
	akka-replicated-data = "Akka.DistributedData.Serialization.ReplicatedDataSerializer, Akka.DistributedData"
	}
	serialization-bindings {
	"Akka.DistributedData.IReplicatorMessage, Akka.DistributedData" = akka-data-replication
	"Akka.DistributedData.IReplicatedDataSerialization, Akka.DistributedData" = akka-replicated-data
	}
	serialization-identifiers {
	"Akka.DistributedData.Serialization.ReplicatedDataSerializer, Akka.DistributedData" = 11
	"Akka.DistributedData.Serialization.ReplicatorMessageSerializer, Akka.DistributedData" = 12
	}
	}

Looks like the reason why this is occurring is because the ActorSystem didn't have the default DData configuration referenced as a fallback, although they did have the default Akka.Cluster.Sharding configuration referenced.

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/ClusterSharding.cs

Lines 277 to 280 in 41c116d

	public static Config DefaultConfig()
	{
	return ConfigurationFactory.FromResource<ClusterSharding>("Akka.Cluster.Sharding.reference.conf");
	}

We should change this to have it automatically reference the default Cluster.DData config and the Cluster.Tools.Singleton configs, so users don't have to remember to do that.

One day in the future akkadotnet/HOCON#48 will eliminate the need for us to do this inside individual modules, but until then we should patch this.

[Aaronontheweb]

Actually.... @Horusiath, if all nodes are running with akka.cluster.sharding.state-store-mode=ddata the serializer IDs will be loaded automatically once the DistributedData plugin is called, right? So couldn't this issue only occur if some nodes were running with akka.cluster.sharding.state-store-mode=ddata but not others? i.e. a misconfiguration?

[Aaronontheweb]

So it looks like we actually do load things correctly right now - this issue was probably caused by nodes having different akka.cluster.sharding.state-store-mode configurations.

[Aaronontheweb]

Yeah, we already do this here:

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/ClusterSharding.cs

Lines 248 to 255 in 37eeb37

	public ClusterSharding(ExtendedActorSystem system)
	{
	_system = system;
	_system.Settings.InjectTopLevelFallback(DefaultConfig());
	_system.Settings.InjectTopLevelFallback(ClusterSingletonManager.DefaultConfig());
	_system.Settings.InjectTopLevelFallback(DistributedData.DistributedData.DefaultConfig());
	_cluster = Cluster.Get(_system);
	Settings = ClusterShardingSettings.Create(system);

[Aaronontheweb]

The cause of this issue must be, then, one node that isn't even running ClusterSharding at all trying to communicate with other nodes in the cluster.