AutorecoveringConnection does not synchronize access to all recorded entity collections

[Mattish]

We've run into some Unobserved task exceptions being thrown from AutorecoveringConnection.RecoverExchanges() due to the collection being modified.

Unobserved task exception occurred 
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute. 
at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext() 
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.RecoverExchanges() in d:\work\RabbitMQ\projects\client\RabbitMQ.Client\src\client\impl\AutorecoveringConnection.cs:line 911 
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.PerformAutomaticRecovery() in d:\work\RabbitMQ\projects\client\RabbitMQ.Client\src\client\impl\AutorecoveringConnection.cs:line 374 
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.<>c__DisplayClass10.<BeginAutomaticRecovery>b__f() in d:\work\RabbitMQ\projects\client\RabbitMQ.Client\src\client\impl\AutorecoveringConnection.cs:line 352 
at System.Threading.Tasks.Task.Execute() 
--- End of inner exception stack trace --- 
---> (Inner Exception #0) System.InvalidOperationException: Collection was modified; enumeration operation may not execute. 
at System.Collections.Generic.Dictionary`2.ValueCollection.Enumerator.MoveNext() 
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.RecoverExchanges() in d:\work\RabbitMQ\projects\client\RabbitMQ.Client\src\client\impl\AutorecoveringConnection.cs:line 911 
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.PerformAutomaticRecovery() in d:\work\RabbitMQ\projects\client\RabbitMQ.Client\src\client\impl\AutorecoveringConnection.cs:line 374 
at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.<>c__DisplayClass10.<BeginAutomaticRecovery>b__f() in d:\work\RabbitMQ\projects\client\RabbitMQ.Client\src\client\impl\AutorecoveringConnection.cs:line 352 
at System.Threading.Tasks.Task.Execute()<--- 

In RecoverQueues() we lock on m_recordedQueues but elsewhere in the class m_recordedQueues is locked by m_recordedEntitiesLock instead. As there is no synchronization in RecoverQueues(), the Dictionary constructor uses foreach so the dictionary clone isn't protected and can throw if elsewhere manipulated.

We believe this problem will also occur for RecoverExchanges() due to no locking around the foreach but there is elsewhere in the class, although we have no seen this thrown.

We've only had 2 single cases of this happening in the past year+ of using rabbit, but it does cause the service to fall over. We are running on 3.6.5, but I can see this set of locking still exists on master

[michaelklishin]

Please try 4.1.1 first.

[Mattish]

The Automatic Connection and Topology recovery logic has not changed at all in the 4.1.1 AutorecoveringConnection.cs from 3.6.5. The comments regarding the lack of synchronization for those collection affects both 4.1.1 tagged and master.

[michaelklishin]

A good candidate for 3.6.7.

[michaelklishin]

Closing since now stuff for 3.6.7, including #289, can be merged.