Akka.Cluster.Sharding: dealing with `remember-entities` and actors who can't start up correctly

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.41
Which Akka.NET Modules? Akka.Cluster.Sharding

Describe the bug

So an interesting scenario:

1. Actor gets created via Akka.Cluster.Sharding, but is misconfigured (i.e. missing DI registration that can't be resolved when the resolver.Props gets fired up) - crashes immediately;
2. Actor hits its "10 crashes within 1s and die" limit on the SupervisionStrategy, gets killed off by the Shard
3. Shard notices that the remembered-entity actor is dead, re-creates it; and
4. The cycle repeats, creating millions of errors until human intervention arrives.

Expected behavior

IMHO, the remember-entities system should not bypass the supervision system - if the supervisor recommends perma-death to a repeatedly failing actor, we need some way of honoring that and un-remembering the actor.

Actual behavior

Infinite churn between the remember-entities system and the restarting actor.

[Arkatufus]

So the problem here is that the Shard remember entities system restart mechanism is totally isolated from the Shard actor SupervisorStrategy

• When the Shard receives a StartEntity request, it creates the entity actor, sets it to Active state, and set a death watch over it.
• Entity actor failed to start (in constructor or during recovery) multiple times, triggering the SupervisorStrategy "max number of retries" retry srategy, causing SupervisorStrategy to stop the actor using Context.Stop()
• Shard receives the EntityTerminated, and Shard restarts the entity actor, honoring the remember entity rule.
• There is no way to know the reason for the termination because all context is lost here.

So I guess the only way to do this is to create a bridge between the 2 system, we'll need to make a custom SupervisorStrategy for the Shard actors that can notifies the Shard when an entity was restarted excessively.

• Create a new message for this event; lets call it SupervisorStrategyTermination(IActorRef Child, Exception Cause)
• Create a new EntityState; lets call it PassivatingDueToErrors
• Create a custom SupervisorStrategy class by inheriting OneForOneStrategy, this class references the Shard actor; lets call it ShardingOneForOneStrategy
• Override the OneForOneStrategy.ProcessFailure() method. When an actor gets restarted excessively, we send a SupervisorStrategyTermination to the Shard before we call Context.Stop() on the entity actor.
• Shard actor should receive the SupervisorStrategyTermination message and sets the entity actor state to PassivatingDueToErrors
• When the EntityTerminated message arrives, Shard actor should have the entity actor state in the PassivatingDueToErrors state, and will remove (passivate) the remembered entity actor.

We can simplify the process by not using the PassivatingDueToErrors custom state and uses Passivating state instead, but we will lose the passivation context when the death watch termination signal comes. Using a custom entity state we can emit a better warning in the log when we passivate the actor.

[Aaronontheweb]

Additional idea / requirement for this mentioned by @jshave in Discord: Having different strategies would be nice. Maybe an exponential back off or something the developer could specify?

The thread I'd take from that: make it possible to configure the Shard's supervision strategy via the ShardingSettings class.

[Aaronontheweb]

we'll need to make a custom SupervisorStrategy for the Shard actors that can notifies the Shard when an entity was restarted excessively

I think this is the right idea - we already have a "thrashing-detection" system built into the OneForOneSupervisorStrategy base class:

akka.net/src/core/Akka/Actor/SupervisorStrategy.cs

Lines 222 to 223 in 65ba0f4

	private readonly int _maxNumberOfRetries;
	private readonly int _withinTimeRangeMilliseconds;

akka.net/src/core/Akka/Actor/SupervisorStrategy.cs

Lines 373 to 379 in f67e123

	public override void ProcessFailure(IActorContext context, bool restart, IActorRef child, Exception cause, ChildRestartStats stats, IReadOnlyCollection<ChildRestartStats> children)
	{
	if (restart && stats.RequestRestartPermission(MaxNumberOfRetries, WithinTimeRangeMilliseconds))
	RestartChild(child, cause, suspendFirst: false);
	else
	context.Stop(child);
	}

Here is what I would do:

1. Create a new message type that gets emitted by the Shard's supervision strategy back to the Shard whenever an entity actor is killed via the SupervisionStrategy - if Directive.Stop is issued for any reason, that always wins over remember-entities going forward. This message type that we're emitting to the Shard is what enforces this - "yes, we were remembering this actor, but it encountered a fatal error - stop remembering it until someone tries to contact it again in the future."
2. That's the end of the scope of this message - we're not doing edge cases, we're not doing anything else other than signaling "this entity has problems - we're going to log a very angry looking error message that explains why we're not remembering this actor any more." If someone tries to message the entity actor again in the future, we will treat it like every other new entity.
3. We will create a dedicated base class for ShardSupervisorStrategy that guarantees that this message is always sent back to the Shard whenever an entity actor is stopped. If there are non-entity actors that can be supervised by the Shard directly, such as the remember-entities infrastructure actors, we might need to check that before sending the new message type.
4. I'd expose this on the ShardSettings (using the ShardSupervisorStrategy base type) as an optional property so users can configure the restart count / duration intervals via C#.