[feat] [broker] PIP-188 support blue-green cluster migration [part-1]

[rdhabalia]

Motivation

It fixes : #16551
Cluster migration or Blue-Green cluster deployment is one of the proven solutions to migrate live traffic from one cluster to another. One of the examples is applications running on Kubernetes sometimes require a Kubernetes cluster upgrade which can cause downtime for the entire application during a Kubernetes cluster upgrade. Blue-green deployment is an application release model that gradually transfers user traffic from a previous version of an app or microservice to a nearly identical new release—both of which are running in production.

The old version can be called the blue environment while the new version can be known as the green environment. Once production traffic is fully transferred from blue to green, blue can standby in case of rollback or be pulled from production and updated to become the template upon which the next update is made.

We need such capability in Apache pulsar to migrate live traffic from the blue cluster to the green cluster so, eventually, the entire traffic moves from the blue cluster to the green cluster without causing downtime for the topics.

Modification

• Handle cluster migration for persistent/non-persistent topics
• Add CLI and REST API support to mark cluster migration

NOTE: This is the 1st part of the feature. 2nd PR will be submitted to handle Replicator changes which can handle cluster migration without compromising ordering and data persistent guarantee.

Result

User can migrate Pulsar cluster without user traffic interruption.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The binary protocol
• The REST endpoints
• The admin CLI options
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[dlg99]

@rdhabalia From what I understand, the change lgtm with caveat that it only works for the cases when old data is deleted after consumption (e.g. offloaded data is not preserved).

[rdhabalia]

@dlg99 yes, this PIP addresses the migration problem from one cluster to a brand-new cluster that will not have old data but should ensure persistent data is delivered before completely migrating to the new cluster. yes, right now, this PIP doesn't handle offloaded data which can be also accessible from a new cluster but that can be a feature added later on top of these changes and such things shouldn't be a problem for this fundamental approach of cluster migration.

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (master@5e6736a). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #17962   +/-   ##
=========================================
  Coverage          ?   46.88%           
  Complexity        ?    17932           
=========================================
  Files             ?     1574           
  Lines             ?   128518           
  Branches          ?    14136           
=========================================
  Hits              ?    60258           
  Misses            ?    62084           
  Partials          ?     6176           

Flag	Coverage Δ
unittests	46.88% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

[Anonymitaet]

1. It seems that you miss this note shown at the beginning of reference-pulsar-admin.md
   This markdown file is not shown on the Pulsar website from 2.11.x, so please do not update it manually because it's less likely for users to see it.

2. The pulsar-admin reference doc is generated automatically from the code.
   I've checked and found that several flags are missing on the reference doc. Can you take a look?