GrpcWorkerChannel.LoadResponse is not threadsafe

[paulbatum]

There is a race in GrpcWorkerChannel that results in failed function executions on this codepath:

System.ArgumentOutOfRangeException : capacity was less than the current size. (Parameter 'value')
at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument,ExceptionResource resource)
at System.Collections.Generic.List1.set_Capacity(Int32 value) at System.Collections.Generic.List1.AddWithResize(T item)
at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.LoadResponse(FunctionLoadResponse loadResponse) at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 835
at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.PendingItem.SetResult(InboundGrpcEvent message) at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 1777

The problem is here:

azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs

Line 835 in 94bd90c

_inputLinks.Add(disposableLink);

There can be concurrent calls to LoadResponse and they race to call List.Add(..). I suggest using a threadsafe collection for _inputLinks instead of List.

Investigative information

FunctionsLogs
| where PreciseTimeStamp >= datetime(2024-07-30T22:30:00.0000000Z) and PreciseTimeStamp <= datetime(2024-07-30T23:00:00.0000000Z)
| where RoleInstance == "5986acec-de97-44ab-852a-2f23dd8cb503"

Repro steps

Hard to repro due to the race - in general it seems to require very large scale before you get unlucky with the timing and hit this.