Add (optional) retry logic to topology recovery

[acogoluegnes]

Topology recovery could benefit from retry logic on failed operations for some specific cases, e.g. failed binding because auto-delete queues has been deleted during the topology recovery.

The retry logic would be:

• optional
• configurable to some extent: retry condition (which error, which recoverable entity), retry action, number of retries, backoff policy

[vikinghawk]

I like the approach here and gave the latest milestone a try. Looks like the default of 2 retries was enough to fix my previous tests recovery errors.

Some feedback:

• It would be nice to have visibility of the entity and exception in the logs when a failure has occurred and its going to retry
  • Log either in DefaultRecoveryHandler or TopologyRecoveryRetryLogic predicates?
  • Or maybe delegate to the connection's ExceptionHandler and let users decide what to log?
• When the queue not found error happens during consumer recovery, it needs to recover all the bindings on the newly recovered queue as well.
• Its not possible for users to write their own DefaultRetryHandler.RetryOperation implementations without using reflections to get at the package protected recoverBinding/recoverQueue/etc. methods on AutorecoveringConnection.
• It would be nice to have a helper method (perhaps on TopologyRecoveryRetryLogic) that creates common scenarios such as the queue not found [1] for users.
  • So all users had to do was something like connectionFactory.setTopologyRecoveryRetryHandler(TopologyRecoveryRetryLogic.retryOnQueueNotFound());
  • This would make it easier and less error prone to consume
  • It also makes it easier to keep code portable between the 4.x and 5.x releases as it would hide the compile time differences such as RetryCondition in 4.x vs the functional predicates used in 5.x. We have apps that need to run on Java 1.7 but others that run on 1.8 and use the micrometer metrics. So currently i compile my code once to 1.7 with the 4.x version of amqp-client but then use 5.x at runtime when running on java 8. The 4.x and 5.x versions have been compatible for everything I am using so far.

[1] https://github.com/rabbitmq/rabbitmq-java-client/blob/master/src/test/java/com/rabbitmq/client/test/functional/TopologyRecoveryRetry.java#L61

[acogoluegnes]

@vikinghawk Thanks for the feedback. I pushed a commit to address your remarks. Snapshots for 5.x and 4.x are available.

I made only recoverConsumer and recoverQueue public, as they contain logic that is worth re-using. For the other entities, calling RecordedEntity#recover should be enough.

[vikinghawk]

+1, this looks good. I'll try to test again today and let you know.