Singleton Orchestration completes, but status stuck in pending

[ghost]

Description

A singleton orchestration I'm invoking appears to be stuck in pending, despite having run to completion. This is leading to an issue where our client is not starting a new timer instance of the singleton orchestration, since we do not run starter.StartNewAsync when an orchestration is pending.

Expected behavior

Orchestration should appear as completed/failed after execution terminates.

Actual behavior

Orchestration appears to be pending.

Relevant source code snippets

The function is quite simple and does nothing but call an underlying activity function. We're working on refactoring the code base to remove this redundant layer.

 public static async Task InvokeSqlFunctionAll([OrchestrationTrigger] IDurableOrchestrationContext context)
{
    if (context is null)
    {
        throw new ArgumentNullException(nameof(context));
    }

    (string function, AdlsFeed comb) = context.GetInput<(string function, AdlsFeed comb)>();
    await context.CallActivityAsync(nameof(InvokeSqlFunction), (function, comb));
}

Known workarounds

We currently manually re-deploy the application or manually change the instance tables. No automatic workaround is currently employed.

App Details

• Durable Functions extension version (e.g. v1.8.3): v2.1.1
• Azure Functions runtime version (1.0 or 2.0): 2.0
• Programming language used: C#

Screenshots

Here's the affected orchestration's history:

and here's how it looks on the instances table:

If deployed to Azure

• Timeframe issue observed: 2020-01-26 16:01-16:02 UTC
• Function App name: fraudhuntersfeeds-test
• Function name(s): InvokeSqlFunctionAll
• Azure region: East US 2
• Orchestration instance ID(s): Azure;PaymentAuth
• Azure storage account name: fraudhuntersfeedtest

[amdeel]

Hey @jublair, sorry for the delayed response! This seems to be a known issue where two orchestrators with the same instanceId are being started at the exact same time. This can lead to the status being overridden and other weird behavior.

[ghost]

Thanks for the information, @amdeel! Out of question, are there any appropriate workarounds to this issue at this time?

[ConnorMcMahon]

@jublair,

You could workaround this yourself in the meantime with the use of a blob lease.

Have a blob with the name of your singleton instance (i.e. Azure;PaymentAuth). Require that any clients trying to create your singleton instance have access to a lease on that blob. That ensures that you aren't trying to create two at the exact same time, which is when this race condition is being hit. Instead, the first client.StartNewAsync() will successfully create the singleton, and the second will attempt to create the singleton but throw an exception because there is already an instance with that id.

[ConnorMcMahon]

Also, if you have a single client function that starts your singleton instance, you may be able to use the [Singleton] attribute to ensure that only one instance of your client function can execute at a time. It looks like there are some caveats around this, especially regarding billing, but it may be worth a shot, and would be easier than the blob lease approach.

[ghost]

Sounds good, thanks Connor. We actually used to use both of these methods, but moved to Azure Durable Functions hoping to slim down our tech stack :D we haven't encountered this issue in a while, but we'll give your suggestions a shot if we continue to experience issues.