ArgumentException "The function 'XYZ' doesn't exist" when the functions host recycles

[cgillum]

Issue

Given sufficient load on a function app, it's possible that an orchestrator function will fail with the following exception during a host recycle:

{InstanceID}: Function 'XYZ (Orchestrator)' failed with an error. Reason: System.ArgumentException: The function 'ABC' doesn't exist, is disabled, or is not an activity function. Additional info: No activity functions are currently registered!
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.ThrowIfFunctionDoesNotExist(String name, FunctionType functionType)
   at Microsoft.Azure.WebJobs.DurableOrchestrationContext.CallDurableTaskFunctionAsync[TResult](String functionName, FunctionType functionType, String instanceId, RetryOptions retryOptions, Object input)

This is caused by an incorrect validation check that occurs when the functions host is shutting down. The result is that the orchestration transitions into a Failed state.

This issue is expected to impact all versions of the Durable Functions extension.

Workaround

No known workarounds at this time. Any failed orchestrations will need to be restarted from the beginning.

Technical Details

The problem is that the DurableTaskListener "deregisters" a function when its listener is stopped by the functions runtime. It's possible that an orchestrator function may still be running after the listener has been stopped, in which case the CallActivityAsync and CallSubOrchestratorAsync APIs will throw an exception because the target function is no longer listed.

The fix should be to relax this validation so that we don't throw in the case where a function has been "delisted", but rather allow the CallActivityAsync or CallSubOrchestratorAsync call to succeed. This will allow the message to be written to the queue, and once the listeners are startup up again, the function-call message can be picked up.

[olitomlinson]

I've just experienced this exact same exception under high-load.

**DF 1.7.1 **
Functions V2

e62d2d1a-084b-4285-82c2-d7cd1d3ebd65: Function 'Orchestrator (Orchestrator)' failed with an error. Reason: System.ArgumentException: The function 'forward_to_' doesn't exist, is disabled, or is not an activity function. Additional info: The following are the active activity functions: 'get_available_'.
at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.ThrowIfFunctionDoesNotExist(String name, FunctionType functionType) in C:\projects\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\DurableTaskExtension.cs:line 522

@cgillum would using the rewind API be a suitable remediation for this? it feels like it should

[olitomlinson]

Also had this for an Orchestrator around the same time :

The function 'Orchestrator' doesn't exist, is disabled, or is not an orchestrator function. Additional info: No orchestrator functions are currently registered!

[shiwani]

@cgillum I am using the latest .net core 2.0 ,still facing the same issue .Kindly help.

[cgillum]

@shiwani please open a new issue since this one is closed. Also, please include more information so that we can properly investigate.

[ParthPurani]

Clearing azure emulator data solved problem for me,

cd C:\Program Files (x86)\Microsoft SDKs\Azure\Storage Emulator
.\AzureStorageEmulator.exe clear all

Please educate me if this is the wrong approach.

[stefanrothnet]

@ParthPurani Perfect, thanks a lot! Worked for me!

[antogh]

@ParthPurani
Thanks, that helped.
My problem started when I renamed some functions with more meaningful names