AutorecoveringConnection Memory Leak

[NathanielAB]

Describe the bug

Hi we have noticed a large amount of TimerQueues when using RabbitMq latest version 7.1.1, leading to an increase in memory usage

Reproduction steps

Start multiple consumers in parallel threads and by time the memory usage increases

Expected behavior

No memory leaks when connecting to multiple consumers

Additional context

No response

[lukebakken]

@NathanielAB - it would be greatly appreciated if you took the time to do the following:

• Provide code to reproduce the issue. Yes, you described how ("Start multiple consumers in parallel threads and by time the memory usage increases"), but to ensure I reproduce the issue the same way, I would appreciate code I can clone, compile and run. A single sentence can be easily mis-interpreted.
• Describe the tool you're using to track allocations, and how to use it. I can't tell from your screenshot.
• If you can, provide a PR to address the issue.

[lukebakken]

@NathanielAB I can't make progress without more information.

Related issues / comments:

• Intermittent deadlock when closing a channel using CloseAsync in 7.x #1751
• Bug in IntervalList causes leak in locks. #1784
• Intermittent deadlock when closing a channel using CloseAsync in 7.x #1751 (comment)

So, it seems that #1784 didn't really fix the issue around timer leaks.

Pinging people who worked on this issue before to see if they can contribute:

@ZajacPiotr98
@DenisMayorko

[NathanielAB]

@NathanielAB - it would be greatly appreciated if you took the time to do the following:

• Provide code to reproduce the issue. Yes, you described how ("Start multiple consumers in parallel threads and by time the memory usage increases"), but to ensure I reproduce the issue the same way, I would appreciate code I can clone, compile and run. A single sentence can be easily mis-interpreted.
• Describe the tool you're using to track allocations, and how to use it. I can't tell from your screenshot.
• If you can, provide a PR to address the issue.

Hi sorry for the late reply, I will try and produce a small POC console application.

The tool I used was visual studio I got a memory dump (.gcdump) and analyzed it there. It shows the root (multiple references nested within each other of TimerQueue) and the referenced types, the types generating the TimerQueues, mostly RabbitMq. It seems to be happening when there is a recovery of a connection, because the specific method causing these instances is RecoverConnectionAsync.

[lukebakken]

Thank you @NathanielAB. It would also be great to know if you're using .NET core or a netstandard2.0-TFM environment.

[NathanielAB]

Thank you @NathanielAB. It would also be great to know if you're using .NET core or a netstandard2.0-TFM environment.

.NET Core

[lukebakken]

@NathanielAB if you aren't able to provide code to reproduce the issue, could you log all first-chance-exceptions when this problem presents itself?

I think what may be happening is that an exception is being thrown here:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/RabbitMQ.Client/Impl/AutorecoveringConnection.cs#L297-L298

...and this prevents this cancellation token source from being disposed:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/RabbitMQ.Client/Impl/AutorecoveringConnection.cs#L303

I'm making some code changes and can have a 7.1.3 alpha release available soon, as well.

[lukebakken]

@NathanielAB - you can get 7.1.3-alpha.0 from here:

https://www.myget.org/feed/rabbitmq-dotnet-client/package/nuget/RabbitMQ.Client

It includes code from PR #1814.