feat: support pdb with v1 and v1beta version to control pod's deletion

[jiuker]

support pdb

• create tenant
• expand tenant
• decomm pool from tenant
• delete tenant

Test steps:
1.start a tenant (4*4)
2. wait for tenant's status are ready
3.kubectl get pdb -A you will see that.
4. And delete the all pod. You will see delete the pod one by one.(For MinAvailable are 3)

[shtripat]

@jiuker please add verification steps

[shtripat]

I followed the below steps

1. TAG=minio/operator:pdb make all
2. kind create cluster --config kind-config.yaml
3. kind load docker-image docker.io/minio/operator:pdb
4. k kustomize . > operator-new.yaml
5. Changed the operator image value in operator-new.yaml with docker.io/minio/operator:pdb
6. k apply -f operator-new.yaml
7. Once operator pods ready, created a tenant from operator console
8. k logs minio-operator-7db874b9b-4pw9z -n minio-operator shows error as below

I0718 11:00:37.918185       1 pdb.go:259] PodDisruptionBudget: v1
E0718 11:00:37.921171       1 main-controller.go:676] error syncing 'tenant-ns/tenant-1': poddisruptionbudgets.policy "tenant-1-pool-0" is forbidden: User "system:serviceaccount:minio-operator:minio-operator" cannot get resource "poddisruptionbudgets" in API group "policy" in the namespace "tenant-ns"

Also there are no pdb created

$ k get pdb -A
No resources found

[jiuker]

@shtripat I have fixed the ClusterRole. Test again plz. And you should update it.

[shtripat]

Below scenarios work fine

1. Create tenant
2. Delete tenant
3. Expand tenant

And PDBs get created for pools of the tenant.

Tried below scenario manually

1. Edited tenant and removed one pool from the list
2. deleted the stateful set for corresponding pool
3. PDB doesnt get deleted and error in operator pod error syncing 'tenant-ns/tenant-1': poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:minio-operator:minio-operator" cannot deletecollection resource "poddisruptionbudgets" in API group "policy" in the namespace "tenant-ns"

[shtripat]

LGTM. Verified below scenarios using kustomize plugin

1. Create tenant (pdb creates)
2. Expand tenant (pdb cerates for new pool)
3. Decomm pool by editing YAML in console (pdb gets deleted for decomm'ed pool)
4. Delete tenant (all pdbs get deleted)

[jiuker]

@harshavardhana @pjuarezd PATL