System.TimeoutException: The operation has timed out.

[chsakell]

Hello,

We are getting System.TimeoutException: The operation has timed out. exceptions when using QueueBind or UnBindQueue methods. The issue happens only in high traffic hours in an ASP.NET Core SignalR 3.1 application hosted in linux containers - OpenShift.

Exception:

System.TimeoutException: The operation has timed out.
   at RabbitMQ.Util.BlockingCell`1.WaitForValue(TimeSpan timeout)
   at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)
   at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body)
   at RabbitMQ.Client.Framing.Impl.Model.QueueUnbind(String queue, String exchange, String routingKey, IDictionary`2 arguments)
   at RabbitMQ.Client.Impl.AutorecoveringModel.QueueUnbind(String queue, String exchange, String routingKey, IDictionary`2 arguments)
   at Customer.PushService.Infrastructure.RabbitMq.Bindings.RabbitMqCustomerBindingsService.UnBindQueue(String routingKey) 

When a user subscribes to a Signalr method, a binding is created in RabbitMQ.

private void BindQueue(string routingKey)
{
    using var model = _connection.CreateModel();
    model.QueueBind(queueName, exchangeName, routingKey, null);
}

At some point though during high volume, it seems that everything is just stopped, thread pool starvation is created and the above errors are shown.

Searching the code, it seems that it throws the exception in the BlockingCell WaitForValue method.

We don't have any memory leak or cpu issue on the ASP.NET Core app, so can you tell a reason why this happens?
Of course there's a lock statement in the ModelBase, and the GetReply is the one that throws the exception, but what does this actually mean? Is it possible that multiple concurrent requests to QueueBind or QueueUnBind cause this issue?

Thanks

[sungam3r]

I think this problem has been already discussed before. See #860 and related issues with ThreadPool.SetMinThreads.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(