Backup and Restore of Hub Cluster Services

[benswinney]

The RHACM Hub cluster is the single source of truth across the pattern. It will hold the state of the GitOps implementations and the currently deploy/managed clusters and applications. This creates a single point of failure (SPOF), which is somewhat mitigated by the multi-zone deployment of OpenShift that that Hub cluster will reside on. Should the Cloud Provider running the Hub cluster suffer a regional outage, then the Hub cluster will no longer be accessible.

As this pattern is to provide multi-cloud operations, we can look to explore either a Hub of Hubs configuration (quite immature at this point in time) or we can explore the Cluster Backup Operator, which aims to provide a mechanism to Backup and Restore a Hub Cluster to another OpenShift environment.

https://github.com/stolostron/cluster-backup-operator

Proposal will be to leverage the Cluster Backup Operator to backup to an ODF S3 endpoint, then leverage VolSync to synchronize the S3 endpoint to another location, either on-premise or a n other Public Cloud and restore the Hub Cluster services there in the event of a failure.