Support for Recycling Instance IDs

[cgillum]

Summary

This feature allows instance IDs to be reused when using the Azure Storage provider. For example, if there is an instance with ID "singleton" and it completes, it will be possible to create a brand-new instance with the same ID of "singleton".

Motivation

When creating long-running singletons, it's useful to have the client specify a well-known instance ID. If that instance should fail for any reason, it should be possible to recreate the instance using the same instance ID.

[SimonLuckenuik]

@cgillum Is this still in scope? Even for short lived scenarios this has value.

Example: we create a report generation workflow which is unique to a document, that runs for 5 minutes, and that cannot be executed multiple times in parallel. We would probably create a well known instance ID corresponding to the document identifier and if you want to generate the report you simply start a flow with the same instance. When requesting a startup of the workflow we can also check if already running. Otherwise, it means we need to track the instance ID and persist it somewhere in order to know the status of the processing.

[cgillum]

Yes, this is still in scope. In fact, it may already work. I've been told that the Durable Task Framework already knows how to recycle instance IDs for completed instances. I'm leaving this open until I/somebody gets around to doing more comprehensive testing to see what the exact behavior is.

[cgillum]

I finally got around to testing this. The current behavior seems to be that creating a new instance with an existing ID will restart that instance from scratch. This works well for completed instances, but could be problematic for running instances as I would expect this opens the door for weird race conditions. Ideally we would drop "StartNew" messages when an instance is already running. This is something that would need to be fixed in the Durable Task Framework layer (DurableTask.AzureStorage).

[ConnorMcMahon]

This is officially supported at this point.

On Durable 2.x, the behavior is that StartNew messages are not schedule if the state is Pending/Running, while they overwrite the existing orchestration if it is in a Completed/Failed/Terminated state.

There is still room to improve the scenario when two clients try to start the same orchestration at the same time, but that is being tracked #367.