Automate AMQ Broker cross datacenter mirror replication feature: configuration, swapover smoke testing, and load test

[shuawest]

SUMMARY

AMQ Broker 7.11 supports a newer set of features for Mirroring broker messages and acknowledgments between remote datacenter or cloud sites at the broker level instead of the storage or infrastructure level. These are high-impact and sought-after features in a majority of customers and users who require Continuity of Business and Disaster Recovery considerations in their messaging architecture. As of 7.11, synchronous mirroring [1] is supported, and prior versions supported asynchronous mirroring (Note this is different than clustering of brokers within a single site).

Using ansible to automate the complex multi-site mirror configuration, as well as the validation and smoke testing of the topology after provisioning, would be highly valuable to the product, project, and user base.

[1]

Synchronous mirroring support
Starting in 7.11, you can configure synchronous mirroring between brokers to ensure that messages are written to the volumes of both brokers in the mirror at the same time. By using synchronous mirroring, you ensure that the mirrored broker is up-to-date for disaster recovery. For more information, see Configuring broker connections in Configuring AMQ Broker.

ISSUE TYPE

• Feature Idea

Requested Enhancement

• Configure the Mirror feature with the ability to choose between sync or async replication.
• Provide the option to use AMQ Interconnect (qpid) may also be required for connectivity across sites.
• Provide the option to mirror clusters of brokers on each side of the mirroring.
• Produce and consume messages on a primary site, and verify messages and acks are mirrored to remote site
• Produce messages and consume some. Swap or failover to the remote site. Ensure expected messages minus acked messages are consumable on remote site.
• Failback to primary site, and verify expected messages again.
• Provide a baseline messaging performance test harness that can be used to put the brokers under load, and measure broker performance, RTO, and RPO.

[shuawest]

Prior work has already gone into this here: https://github.com/RHEcosystemAppEng/amq-cob

Integrating this work in a consumable manner as part of the Ansilbe MW collections, and aligning with the some practices, will help improve the accessibility of the automation and adoption of amq-broker.