NullReferenceException in HeartbeatReadTimerCallback crashing application

[MichaelLogutov]

We just got another application crashed due to RabbitMQ library:

Application: Scheduler.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.NullReferenceException at RabbitMQ.Client.Framing.Impl.Connection.HeartbeatReadTimerCallback(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.TimerQueueTimer.CallCallback() at System.Threading.TimerQueueTimer.Fire() at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch()

I think that all exceptions inside separate RabbitMQ library threads should not propagate exceptions in any cases. If connection is down and auto recovery is not completed or failed, then Publish and Get methods should throw exceptions.

Anyway - what should I do now to stop RabbitMQ library from crashing my application?
We're using RabbitMQ 3.6.5.0

[kjnilsson]

Hi, which version of the dotnet client are you using?

[MichaelLogutov]

As I've said - we're using RabbitMQ 3.6.5.0
.NET 4.6.1

[kjnilsson]

Could you try the latest version of the client (currently 4.1.0)?

[MichaelLogutov]

Will do. Thanks.

[kjnilsson]

There may be a fix in this commit.

[michaelklishin]

First of all, there is no such RabbitMQ version as 3.6.5.0. What .NET client library version is used?

[MichaelLogutov]

https://www.nuget.org/packages/RabbitMQ.Client/3.6.5

[MichaelLogutov]

We've updated to 4.1.0 and got hit with the same crash this night:

Application: Scheduler.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.NullReferenceException at RabbitMQ.Client.Framing.Impl.Connection.HeartbeatReadTimerCallback(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.TimerQueueTimer.CallCallback() at System.Threading.TimerQueueTimer.Fire() at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch()

[kjnilsson]

@MichaelLogutov ok, looks like there is still an issue somewhere. Can you give a bit more details on how it is being used? Some kind of minimal repro obviously would be fantastic :)

[MichaelLogutov]

We're creating connection when application start using IConnectionFactory.CreateConnection:

var connection = this.connectionFactory.CreateConnection();
connection.AutoClose = false;

And then we're using CreateChannel:

            var channel = connection.CreateModel();

            if (this.configuration.PublishConfirmTimeout > TimeSpan.Zero)
                channel.ConfirmSelect();

            return channel;

And Publish or BasicGet: channel.BasicGet(queue: queueName, noAck: false)

The most curious thing is that this crash occurred in application that should not publish or receiving messages - and thus does not creates any channels. I could be wrong (it's really big solution and some part of it underneath can use queue to publish something) but maybe RabbitMQ.Client library spins some thread at any case?

[kjnilsson]

After looking at the code I could see one potential null ref scenario so have covered that in the linked PR.