Durable entity with null state persistence

[emreertugrul]

Hi,
I've created a durable entity function that does some action, and when a certain condition is met it deletes itself with Entity.Current.DeleteState()

When this is done the state is cleared (becomes null) but the entity still persists and is sent within the results when ListEntitiesAsync is called:
var allRunningEntities = await client.ListEntitiesAsync(new EntityQuery { EntityName = nameof(PlayingMatchEntity), FetchState = true, }, new System.Threading.CancellationToken());
Result:

  {
    "entityId": {
      "name": "playingmatchentity",
      "key": "4ilku4ut596biciph693j3pey"
    },
    "lastOperationTime": "2020-07-29T11:31:31.1441338Z",
    "state": null
  },

I cant find in any documentation on how to get rid of these null-state entities..

[ConnorMcMahon]

This has been discussed in the past in this issue. I believe the conclusion reached in that issue was that you should use the PurgeInstanceHistoryAsync() API to remove these from the table storage that the ListEntitiesAsync() calls from.

In general, this is something that we should probably document better, or at least make more discoverable. @cgillum, @anthonychu, @sebastianburckhardt thoughts?

[emreertugrul]

Yeah, the PurgeInstanceHistoryAsync seems to be available on IDurableOrchestrationClient which is not used (according to documentation at least) when you're dealing with only durable entities, instead IDurableEntityClient is used and the there arent any helpful methods there to purge..

[ConnorMcMahon]

@emreertugrul to clarify, if you use IDurableClient (which is the union of the IDurableOrchestrationClient and the IDurableEntityClient), you can use that method. The entity's "orchestration instance id" is @<entity-name>@<entity-key>.

Perhaps IDurableEntityClient deserves a method like PurgeEntityHistoryAsync() that allows the usage of entity ids instead of just strings that represent orchestration instance ids.

[sebastianburckhardt]

I think there are several issues here.

• the entities that don't have a state should be suppressed in the ListEntitiesAsync() results. This is currently being addressed in Allow ListEntitiesAsync to only return entities with non-null state #1421.
• Residual table entries (belonging to deleted entities) should be removed automatically when safe. This is tracked in Garbage Collection for Deleted Entities #1065.
• IDurableEntityClient should provide a DeleteEntityStatesAsync() that safely deletes the entity states (as Purge is not safe)

[MartinWickman]

Were are we with this? I'm currently in a situation where I have an entity which I tried to remove using TerminateAsync(). It's now in Terminated(5) state, but it's still there. The effect is that orchestrations using the entity just blocks/stops.

The output from ListEntitiesAsync() looks like this:

        {
            "entityId": {
                "name": "pagingentity",
                "key": "bci-monitoring-1"
            },
            "lastOperationTime": "2021-08-05T08:58:11.1974054Z",
            "state": null
        }

And the output from ListInstancesAsync() looks like this:

    {
        "name": "@pagingentity@bci-monitoring-1",
        "instanceId": "@pagingentity@bci-monitoring-1",
        "createdTime": "2021-08-05T08:54:55.7172325Z",
        "lastUpdatedTime": "2021-08-05T08:58:11.1974054Z",
        "input": {
            "exists": true,
            "state": "{\"state\":{\"Cursor\":\"610ba3770000000000000000\",\"Since\":\"2021-08-05T08:50:24.2657793Z\"}}",
        },
        "output": "about time",
        "runtimeStatus": 5,   // == Terminated
        "customStatus": {
            "entityExists": true
        },
        "history": null
    },

The problem is that it's no longer possible to use the entity. When my orchestration tries to read from the entity, it just stops with the log message @pagingentity@bci-monitoring-1: Discarding 1 dequeued history event(s): Instance is Terminated.

Here is the relevant code from the orchestration function which tries to load (the terminated) entity:

            log.LogInformation("10");
            var entityId = new EntityId(nameof(PagingEntity), "bci-monitoring-1");
            log.LogInformation("20");
            var entity = context.CreateEntityProxy<IPagingEntity>(entityId);
            log.LogInformation("30");
            var state = await entity.GetTheData();
            log.LogInformation("40"); // Never gets here

The log shows this:

[2021-08-05T09:34:35.263Z] Executing 'MonitorChangesLoop' (Reason='(null)', Id=9412432f-1427-4c7b-8873-2b8cb919ea49)
[2021-08-05T09:34:35.275Z] 10
[2021-08-05T09:34:35.276Z] 20
[2021-08-05T09:34:35.282Z] 30
[2021-08-05T09:34:35.305Z] Executed 'MonitorChangesLoop' (Succeeded, Id=9412432f-1427-4c7b-8873-2b8cb919ea49, Duration=57ms)
[2021-08-05T09:34:35.349Z] @pagingentity@bci-monitoring-1: Discarding 1 dequeued history event(s): Instance is Terminated

I guess is that the orchestration never gets a "reply" from the entity so the orchestration is forever stuck in "running" state waiting for a reply?

The only way around this is manually terminate the orchestration and create a new entity with a different name.

So:

1. How do I go about removing this entity?
2. What is the preferred way of deleting entities?
3. Is this "orchestration blocking forever" expected behavior when trying read from a terminated entity?

I'm running this locally in Azurite by the way, if that makes any difference.

[sebastianburckhardt]

Sorry for the confusion we have caused. I think we need to improve the docs and fix some remaining bugs to prevent this in the future.

The correct and safe way to delete an entity is to call Entity.Current.DeleteState, as you have observed.
This logically deletes the entity. Though I understand there remain some reasons of confusion:

• A deleted entity leaves behind some leftover-state in storage. That state is left there on purpose by our runtime for 30min after deletion to prevent out-of-order problems with message delivery. It does not contain the actual entity state, just some metadata. After that it can be safely removed. We have an API called CleanEntityStorage which removes all such leftover states.

• Queries can return deleted entities. This is a bug, see ListEntitiesAsync should not return deleted entities #1364. For some reason we have not fixed it yet, I apologize.

TerminateAsync should never be called on an entity, as it will permanently disable this entity as you have observed. Terminating an entity does not make sense because it terminates the orchestration that functions as the entity scheduler, which then means the entity becomes forever unresponsive.

To remove an entity that has gotten into a bad state (e.g. after TerminateAsync) you can call PurgeAsync which should remove all traces of the entity from storage. Under normal circumstances this is not necessary.