fed: setup the br-federation-manager

[csuzhangxc]

What problem does this PR solve?

• add a new br-federation-manager framework
• add a Helm chart for installing the br-federation-manager

What is changed and how does it work?

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test

1. setup two Kubernetes clusters, one for the control plane which runs the br-federation-manager, and another for the data plane, and ensure the network from the control plane to the data plane API server is working
2. create RBAC resources in the data plane which will be added into the federation KUBECONFIG used by the br-federation-manager to access the data plane, e.g (tidb-admin namespace should be existing)

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: br-federation-manager
     namespace: tidb-admin
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: br-federation-manager:br-federation-manager
   rules:
   - apiGroups:
     - pingcap.com
     resources:
     - backup
     - restore
     - backupschedule
     verbs:
     - '*'
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: br-federation-manager:br-federation-manager
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: br-federation-manager:br-federation-manager
   subjects:
   - kind: ServiceAccount
     name: br-federation-manager
     namespace: tidb-admin

3. generate a KUBECONFIG to the data plane cluster based on the above ServiceAccount's secret token, e.g.

   apiVersion: v1
   kind: Config
   current-context: dataplane-a
   clusters:
   - name: dataplane-a
     cluster:
       certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXdPREF5TWpneE1Wb1hEVE16TURVd05UQXlNamd4TVZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTm1BCm4wc1NVaW4xV0xqTTkxbUVMVXRMSWZwMHNaR0E0UDJ0TEhidmVabG9GbG56VExaNWYyMFYvM1BVK1lnR0JreU4KUzVRcXJ3NlJJSEdGWjUybGdnaWhsQldsN2FFM1h3ZkVyOGx6K1JkMG9PQW5XZ3Zlb2ZybUlJQXk5SHdEZ1NIQwo4MFM1ejhJeTBVK2ZXS2dqTnIvbU9oakQ4RDhJT2lHTDFncVRraHpDdGZ4d0dlRHdLZ0RhTmpLQzh6NFI2anp2ClBNOEdHRjltQk92NXQvREVaeE5UNDh0bWErb0hkanBHZUNsVFE3a2pqb0RkeFVPNG9HTG5oVUh4YjFqVXM2cGoKcG4yVlNjTjZ5S0JKbExPRVRkbnZ6azMxV2lMUzhnbDNBK1FuRDI1WXBRSlJ5WmZ2Z3l3UzJBNFJXdVJYUUU5MApzTFJEc281SjZ5RzJFVEI5aTg4Q0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZMcStJcDhOVHZDMkxjaENFUHBIemJhQS9vUWxNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFCbkY3aENmTlJlZGlkZXVid2xCZytQVVhrT1BsL1JYWFl2ekp5YkduaWtqWEloNHNySQpPLytRRkF3OXVHWEFXdUV1U0xUWS9RWkNXV215ZE9vbXNYc3JtME51TzdBRVhMbjRWODRTb2Z6bHU5TlloVjVhCjFpc3JWV0JBbWI0MzQ0SGlHb21JVE1WMEUrdjM1RXFya0Q5bWhEK2NtcDFMaSszTkpSV1d0MUVlNVdTTzA2ZkEKV0p3S2c3OVRWNzI0MVFIZnVGSDN3S3RIQmtqVW8rb0pBME04Q2J3QlBHQjFveUM0aXp3K0h0Nk1pSXQwakZ4LwpyTDN2cy90QklJQ3FXTjdFN3RVcjBUY1c4NU0wanlsWHMrMU1xNXJRRTlwTkFqS0RseWZKbmxCRGdVdTRhVEttCmova2NNczNLWVVUc3BEQWxQT3JVNkZ1OGI4VnFFOGJUYnRONgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
       server: https://172.20.0.2:6443
   contexts:
   - name: dataplane-a
     context:
       cluster: dataplane-a
       user: dataplane-a
   users:
   - name: dataplane-a
     user:
       token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklteFpSa0pTV0d0dVNuQlVZV2xMYTJ4S1lVZ3dYMWR0VG1OQkxUTTBhalJhVnpOVE1EWjFSMmxwVFZraWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUowYVdSaUxXRmtiV2x1SWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXpaV055WlhRdWJtRnRaU0k2SW1KeUxXWmxaR1Z5WVhScGIyNHRiV0Z1WVdkbGNpMTBiMnRsYmkwNU5XaGtlQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG01aGJXVWlPaUppY2kxbVpXUmxjbUYwYVc5dUxXMWhibUZuWlhJaUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sY25acFkyVXRZV05qYjNWdWRDNTFhV1FpT2lKa05UQTBaR00zTlMwd05UYzBMVFEwTlRjdFlUTmhNeTFoTkRjelpXSmlNV1E0WmpraUxDSnpkV0lpT2lKemVYTjBaVzA2YzJWeWRtbGpaV0ZqWTI5MWJuUTZkR2xrWWkxaFpHMXBianBpY2kxbVpXUmxjbUYwYVc5dUxXMWhibUZuWlhJaWZRLlFPVjVzYzdmVlAyS0txTFZ5WmkwVVlMVWN1aFFaTjB2aUlQbGszWFgxQUgtUjNNT0V3OV9QeXRmQ01SMzgyNGNTTXBXazNzWkRMbGd3WFYtQ2lHMzFTTkJ0UnlSUUdkeUtkWXpYZXVyRVF0SFR0V0ZaRTVnVTh3NDZCWmJfZ29DbWlVLThCYTViaThGSUlEUjdSa2ttUmVqYWRjR21vQ3JQNFRyeU1DMk0xcktId2EzSW16OVZ4VVVxWmZuVWRZaFZ6ZTBWSWxCYnZ0MmZPTjl1QlpUU3ZkSjdybzh1bkY1X21qRDJfZlpqa1hrMEVHR2Vxa2szS2ZPS2pSS1lnUE92SmhIR3NMQmxQY2ZNOUYxV18wMkoxY1lXX0h5VE5tR3pEV3U4dVdUV2w1RlF2TGFzWDA5SVVoLVhvVmV2RVRLMVhXenB0YURpWXVRQkxOeThVZHVRdw==

4. encode the above KUBECONFIG to base64 and add it into the federation KUBECONFIG secret in the control plane, e.g. (tidb-admin namespace should be existing), NOTE: if the name is changed to another value, do not forget to set it to .Values.brFederationManager.federationKubeconfigSecret when install the Hem chart later

   apiVersion: v1
   kind: Secret
   metadata:
     name: br-federation-kubeconfig
   type: Opaque
   data:
     dataplane-a: YXBpVmVyc2lvbjogdjEKa2luZDogQ29uZmlnCmN1cnJlbnQtY29udGV4dDogZGF0YXBsYW5lLWEKY2x1c3RlcnM6Ci0gbmFtZTogZGF0YXBsYW5lLWEKICBjbHVzdGVyOgogICAgY2VydGlmaWNhdGUtYXV0aG9yaXR5LWRhdGE6IExTMHRMUzFDUlVkSlRpQkRSVkpVU1VaSlEwRlVSUzB0TFMwdENrMUpTVU0xZWtORFFXTXJaMEYzU1VKQlowbENRVVJCVGtKbmEzRm9hMmxIT1hjd1FrRlJjMFpCUkVGV1RWSk5kMFZSV1VSV1VWRkVSWGR3Y21SWFNtd0tZMjAxYkdSSFZucE5RalJZUkZSSmVrMUVWWGRQUkVGNVRXcG5lRTFXYjFoRVZFMTZUVVJWZDA1VVFYbE5hbWQ0VFZadmQwWlVSVlJOUWtWSFFURlZSUXBCZUUxTFlUTldhVnBZU25WYVdGSnNZM3BEUTBGVFNYZEVVVmxLUzI5YVNXaDJZMDVCVVVWQ1FsRkJSR2RuUlZCQlJFTkRRVkZ2UTJkblJVSkJUbTFCQ200d2MxTlZhVzR4VjB4cVRUa3hiVVZNVlhSTVNXWndNSE5hUjBFMFVESjBURWhpZG1WYWJHOUdiRzU2VkV4YU5XWXlNRll2TTFCVksxbG5SMEpyZVU0S1V6VlJjWEozTmxKSlNFZEdXalV5YkdkbmFXaHNRbGRzTjJGRk0xaDNaa1Z5T0d4NksxSmtNRzlQUVc1WFozWmxiMlp5YlVsSlFYazVTSGRFWjFOSVF3bzRNRk0xZWpoSmVUQlZLMlpYUzJkcVRuSXZiVTlvYWtRNFJEaEpUMmxIVERGbmNWUnJhSHBEZEdaNGQwZGxSSGRMWjBSaFRtcExRemg2TkZJMmFucDJDbEJOT0VkSFJqbHRRazkyTlhRdlJFVmFlRTVVTkRoMGJXRXJiMGhrYW5CSFpVTnNWRkUzYTJwcWIwUmtlRlZQTkc5SFRHNW9WVWg0WWpGcVZYTTJjR29LY0c0eVZsTmpUalo1UzBKS2JFeFBSVlJrYm5aNmF6TXhWMmxNVXpobmJETkJLMUZ1UkRJMVdYQlJTbEo1V21aMlozbDNVekpCTkZKWGRWSllVVVU1TUFwelRGSkVjMjgxU2paNVJ6SkZWRUk1YVRnNFEwRjNSVUZCWVU1RFRVVkJkMFJuV1VSV1VqQlFRVkZJTDBKQlVVUkJaMHRyVFVFNFIwRXhWV1JGZDBWQ0NpOTNVVVpOUVUxQ1FXWTRkMGhSV1VSV1VqQlBRa0paUlVaTWNTdEpjRGhPVkhaRE1reGphRU5GVUhCSWVtSmhRUzl2VVd4TlFUQkhRMU54UjFOSllqTUtSRkZGUWtOM1ZVRkJORWxDUVZGQ2JrWTNhRU5tVGxKbFpHbGtaWFZpZDJ4Q1p5dFFWVmhyVDFCc0wxSllXRmwyZWtwNVlrZHVhV3RxV0Vsb05ITnlTUXBQTHl0UlJrRjNPWFZIV0VGWGRVVjFVMHhVV1M5UldrTlhWMjE1WkU5dmJYTlljM0p0TUU1MVR6ZEJSVmhNYmpSV09EUlRiMlo2YkhVNVRsbG9WalZoQ2pGcGMzSldWMEpCYldJME16UTBTR2xIYjIxSlZFMVdNRVVyZGpNMVJYRnlhMFE1YldoRUsyTnRjREZNYVNzelRrcFNWMWQwTVVWbE5WZFRUekEyWmtFS1YwcDNTMmMzT1ZSV056STBNVkZJWm5WR1NETjNTM1JJUW10cVZXOHJiMHBCTUUwNFEySjNRbEJIUWpGdmVVTTBhWHAzSzBoME5rMXBTWFF3YWtaNEx3cHlURE4yY3k5MFFrbEpRM0ZYVGpkRk4zUlZjakJVWTFjNE5VMHdhbmxzV0hNck1VMXhOWEpSUlRsd1RrRnFTMFJzZVdaS2JteENSR2RWZFRSaFZFdHRDbW92YTJOTmN6TkxXVlZVYzNCRVFXeFFUM0pWTmtaMU9HSTRWbkZGT0dKVVluUk9OZ290TFMwdExVVk9SQ0JEUlZKVVNVWkpRMEZVUlMwdExTMHRDZz09CiAgICBzZXJ2ZXI6IGh0dHBzOi8vMTcyLjIwLjAuMjo2NDQzCmNvbnRleHRzOgotIG5hbWU6IGRhdGFwbGFuZS1hCiAgY29udGV4dDoKICAgIGNsdXN0ZXI6IGRhdGFwbGFuZS1hCiAgICB1c2VyOiBkYXRhcGxhbmUtYQp1c2VyczoKLSBuYW1lOiBkYXRhcGxhbmUtYQogIHVzZXI6CiAgICB0b2tlbjogWlhsS2FHSkhZMmxQYVVwVFZYcEpNVTVwU1hOSmJYUndXa05KTmtsdGVGcFNhMHBUVjBkMGRWTnVRbFZaVjJ4TVlUSjRTMWxWWjNkWU1XUjBWRzFPUWt4VVRUQmhhbEpoVm5wT1ZFMUVXakZTTW14d1ZGWnJhV1pSTG1WNVNuQmpNMDFwVDJsS2NtUlhTbXhqYlRWc1pFZFdla3d6VG14amJscHdXVEpXYUZreVRuWmtWelV3U1dsM2FXRXpWbWxhV0VwMVdsaFNiR041TlhCaWVUbDZXbGhLTW1GWFRteFpWMDVxWWpOV2RXUkRPWFZaVnpGc1l6TkNhRmt5VldsUGFVb3dZVmRTYVV4WFJtdGlWMngxU1dsM2FXRXpWbWxhV0VwMVdsaFNiR041TlhCaWVUbDZXbGhLTW1GWFRteFpWMDVxWWpOV2RXUkRPWHBhVjA1NVdsaFJkV0p0Um5SYVUwazJTVzFLZVV4WFdteGFSMVo1V1ZoU2NHSXlOSFJpVjBaMVdWZGtiR05wTVRCaU1uUnNZbWt3TlU1WGFHdGxRMGx6U1cxME1WbHRWbmxpYlZZd1dsaE5kV0ZYT0haak1sWjVaRzFzYWxwWFJtcFpNamt4WW01UmRtTXlWbmxrYld4cVdsTXhhRmt5VG5aa1Z6VXdURzAxYUdKWFZXbFBhVXBwWTJreGJWcFhVbXhqYlVZd1lWYzVkVXhYTVdoaWJVWnVXbGhKYVV4RFNuSmtWMHBzWTIwMWJHUkhWbnBNYld4MlRETk9iR051V25CWk1sWm9XVEpPZG1SWE5UQk1NMDVzWTI1YWNGa3lWWFJaVjA1cVlqTldkV1JETlRGaFYxRnBUMmxLYTA1VVFUQmFSMDB6VGxNd2QwNVVZekJNVkZFd1RsUmpkRmxVVG1oTmVURm9Ua1JqZWxwWFNtbE5WMUUwV21wcmFVeERTbnBrVjBscFQybEtlbVZZVGpCYVZ6QTJZekpXZVdSdGJHcGFWMFpxV1RJNU1XSnVVVFprUjJ4cldXa3hhRnBITVhCaWFuQnBZMmt4YlZwWFVteGpiVVl3WVZjNWRVeFhNV2hpYlVadVdsaEphV1pSTGxGUFZqVnpZemRtVmxBeVMwdHhURlo1V21rd1ZWbE1WV04xYUZGYVRqQjJhVWxRYkdzeldGZ3hRVWd0VWpOTlQwVjNPVjlRZVhSbVEwMVNNemd5TkdOVFRYQlhhek56V2tSTWJHZDNXRll0UTJsSE16RlRUa0owVW5sU1VVZGtlVXRrV1hwWVpYVnlSVkYwU0ZSMFYwWmFSVFZuVlRoM05EWkNXbUpmWjI5RGJXbFZMVGhDWVRWaWFUaEdTVWxFVWpkU2EydHRVbVZxWVdSalIyMXZRM0pRTkZSeWVVMURNazB4Y2t0SWQyRXpTVzE2T1ZaNFZWVnhXbVp1VldSWmFGWjZaVEJXU1d4Q1luWjBNbVpQVGpsMVFscFVVM1prU2pkeWJ6aDFia1kxWDIxcVJESmZabHBxYTFock1FVkhSMlZ4YTJzelMyWlBTMnBTUzFsblVFOTJTbWhJUjNOTVFteFFZMlpOT1VZeFYxOHdNa294WTFsWFgwaDVWRTV0UjNwRVYzVTRkVmRVVjJ3MVJsRjJUR0Z6V0RBNVNWVm9MVmh2Vm1WMlJWUkxNVmhYZW5CMFlVUnBXWFZSUWt4T2VUaFZaSFZSZHc9PQo=
   

5. install the federation CRDs with kubectl apply -f ./manifests/federation-crd.yaml in the control plane
6. install the common previous CRDs with kubectl apply -f ./manifests/crd.yaml in the data plane
7. build the br-federation-manager docker image with make br-federation-docker (and retag from localhost:5000/pingcap/br-federation-manager to any other tag as you like)
8. push the docker image to a registry (docker hub) or load it into KIND with kind load docker-image localhost:5000/pingcap/br-federation-manager
9. enter the Helm chart directory (./charts/br-federation) and install the Helm chart with helm install --namespace tidb-admin br-federation-manager . --set image=${br-federation-image}
10. the br-federation-manager will be stared, some of logs

    I0508 06:20:08.908738       1 fed_volume_backup_schedule_controller.go:77] Starting volumeBackupSchedule controller
    I0508 06:20:08.908992       1 fed_volume_backup_controller.go:77] Starting volumeBackup controller
    I0508 06:20:08.909029       1 fed_volume_restore_controller.go:77] Starting volumeRestore controller
    

11. apply an empty volume backup into the control plane, e.g.

    apiVersion: federation.pingcap.com/v1alpha1
    kind: VolumeBackup
    metadata:
      name: basic-volumebackup
    spec: {}

12. the br-federation-manager will reconcile this VolumeBackup with log

    I0508 06:20:32.284766       1 backup_manager.go:63] sync VolumeBackup default/basic-volumebackup
    

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• WangLe1321

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[codecov-commenter]

Codecov Report

Merging #4996 (c1ed4db) into master (d14d404) will increase coverage by 1.06%.
The diff coverage is 1.52%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4996      +/-   ##
==========================================
+ Coverage   59.12%   60.19%   +1.06%     
==========================================
  Files         228      247      +19     
  Lines       26441    30537    +4096     
==========================================
+ Hits        15634    18382    +2748     
- Misses       9287    10574    +1287     
- Partials     1520     1581      +61     

Flag	Coverage Δ
e2e	34.99% <2.33%> (?)
unittest	57.31% <1.40%> (-1.82%)	⬇️

[BornChanger]

/cherry-pick release-1.5

[ti-chi-bot]

@BornChanger: The label(s) /release-1.5 cannot be applied, because the repository doesn't have them.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@BornChanger: new pull request created to branch release-1.5: #5105.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.