Reliability improvements for worker-host communication

[liliankasem]

We have seen many CRIs where out-of-proc language workers fail to start or call back to the host within the defined timeout period. We need to identify the root-cause and fix all the potential reasons, however currently this is very difficult as we are dealing with a long tail of various issues from different components and levels. To do this, we need to improve the reliability of host-worker communication and fix potential bugs.

This epic will keep track of issues that could help us improve worker-host reliability:

• [Rpc][MultipleWorkers] Add check for memory consumption before spawning new processes #6704
• Optimize grpc to get rid of a throughput bottleneck  #4076
• [RPC] Shutdown functions host if there are no language worker processes that can handle invocation request #7462
• Recycle failing language workers #7292

Protobuf Message Implementations

• Design WorkerHeartbeat functionality #8164
• Add support to gracefully shutdown language worker #2308
• Provide out-of-proc support for cancellation tokens #2152

Logging

• Pass log-levels for Functions to the Language Worker #6879

Surface Customer Issues

• Report startup failures in the app Log Stream #8025
• Rethrow RpcException on invocation as InvocationException #4296
• Log user exceptions to application insights directly azure-functions-dotnet-worker#846

Investigation

• Investigate if the function dispatcher state is getting properly reset #8165