Function app gets into stuck state due to socket exhaustion

[paulbatum]

Function apps running the consumption plan are subject to limits on how many outbound connections each instance can have. Ideally when this happens the process restarts so that execution can continue. However there are scenarios where what happens instead is that the host restarts and indexing fails (because indexing might involve attempting to validate storage credentials, which itself can fail due to the inability to create a new outbound connection to storage). Once indexing fails, the host has started but none of the functions are running and so the app essentially goes into an idle state.

Timeframe where this failure occurred is 2017-09-29 01:43 to 2017-09-29 02:25

Details to look up app name:

• Timestamp: 2017-09-29 01:44:04.5972293 UTC
• Invocation ID: Function started (Id=e2a2461e-8350-4c4e-a38b-a86291ee05f0)
• Region: West US

Ideally, if there is an attempt to start a host and the process is already in a socket exhaustion state, we would just crash the process so that it can restart in a good state.

[nelak]

@paulbatum What's the recommended way for a function to behave and not exhaust the sockets?
Is there any way to tell what's the socket distribution on an instance?
E.g.: In node you may set the http.globalAgent.maxSockets or https.globalAgent.maxSockets but without any kind of telemetry is really hard to measure for a good value.
Also if you are running multiple workers you need to take this into account because they'll be running under the same instance right?