[BUG]All pods restart after hscale out / hscale in

[ahjing99]

➜ ~ kbcli version
Kubernetes: v1.27.8-gke.1067004
KubeBlocks: 0.9.0-alpha.7
kbcli: 0.9.0-alpha.2

1. hscale out, every pod restart after hscale out the new pod

➜  ~ kbcli cluster create mytest --cluster-definition=apecloud-mysql  --set cpu=100m,memory=0.5Gi,replicas=3,storage=1Gi
Info: --cluster-version is not specified, ClusterVersion ac-mysql-8.0.30 is applied by default
Cluster mytest created

➜  ~ kbcli cluster describe mytest
Name: mytest	 Created Time: Mar 28,2024 17:03 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   Delete

Endpoints:
COMPONENT   MODE        INTERNAL                                      EXTERNAL
mysql       ReadWrite   mytest-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE         ROLE       STATUS    AZ              NODE                                                CREATED-TIME
mysql       mytest-mysql-0   follower   Running   us-central1-c   gke-yjtest-default-pool-d5896c86-7mbr/10.128.0.24   Mar 28,2024 17:03 UTC+0800
mysql       mytest-mysql-1   follower   Running   us-central1-c   gke-yjtest-default-pool-d5896c86-xgv2/10.128.0.20   Mar 28,2024 17:03 UTC+0800
mysql       mytest-mysql-2   leader     Running   us-central1-c   gke-yjtest-default-pool-d5896c86-332f/10.128.0.28   Mar 28,2024 17:03 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       100m / 100m          512Mi / 512Mi           data:1Gi       kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20231215.ge77d836.14

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default mytest

➜  ~ kbcli cluster hscale  mytest --components mysql --replicas 4
Please type the name again(separate with white space when more than one): mytest
OpsRequest mytest-horizontalscaling-vgvvc created successfully, you can view the progress:
	kbcli cluster describe-ops mytest-horizontalscaling-vgvvc -n default

➜  ~ k get pod | grep mytest
mytest-mysql-0                                           5/5     Running       0          80s
mytest-mysql-1                                           5/5     Running       0          22s
mytest-mysql-2                                           5/5     Terminating   0          7m32s
mytest-mysql-3                                           5/5     Running       0          2m37s

2. Hscale in also has the same problem

➜  ~ kbcli cluster hscale  mytest --components mysql --replicas 3
Please type the name again(separate with white space when more than one): mytest
OpsRequest mytest-horizontalscaling-mbl9l created successfully, you can view the progress:
	kbcli cluster describe-ops mytest-horizontalscaling-mbl9l -n default

➜  ~  k get pod | grep mytest
mytest-mysql-0                                           4/5     Terminating   0          2m56s
mytest-mysql-1                                           5/5     Running       0          118s
mytest-mysql-2                                           5/5     Running       0          65s
mytest-mysql-3                                           4/5     Terminating   0          4m13s

➜  ~ kbcli report cluster   --with-logs --all-containers mytest
reporting cluster information to report-cluster-mytest-2024-03-28-17-17-10.zip
processing manifests                               OK
processing events                                  OK
process pod logs                                   OK
➜  ~ kbcli report kubeblocks --with-logs --all-containers --output yaml
reporting KubeBlocks information to report-kubeblocks-2024-03-28-17-17-40.zip
processing manifests                               OK
processing events                                  OK
process pod logs                                   OK

report-kubeblocks-2024-03-28-17-17-40.zip
report-cluster-mytest-2024-03-28-17-17-10.zip