Deadlock when a language worker restart crashes on initialization

[brettsam]

We have a deadlock in the language worker restart flow.

1. For example, for some reason (due to a timeout, say), we need to kill the process.
2. We restart the process and wait for the init call to come from the worker (waiting on a TaskCompletionSource to be completed):
   • https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs#L576
   • https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs#L135
   • https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs#L157
3. But the worker process crashes before connecting (which happened to your incident). Therefore, we never complete the _workerInitTask.
4. We see the error and try to restart again.
   • https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/Rpc/RpcWorkerProcess.cs#L68
5. We hit a lock in the restart loop that is meant to prevent multiple restarts from occurring simultaneously... but b/c we never completed that init task in step 1, it is never released. As far as it's concerned, 2 restarts are happening simultaneously.
   • https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs#L570

This can cause the process to effectively hang as it's waiting for these two events to complete.

Note: In some other flows, we do explicitly set the _workerInitTask -- https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs#L815

[alrod]

Possibly related:
Azure/azure-functions-nodejs-worker#482

[brettsam]

This was actually related to a fix already made by @alrod -- it took a while to realize this. Dupe of #7983, which has already been fixed and deployed. The incident mentioned above happened in 4.1.0 -- which was before this was deployed in 4.2.0.