Fix to allow etcd-druid to patch STS when there is only a change of mount-path even if all pods are not up and running

[unmarshall]

How to categorize this PR?

/area control-plane
/kind bug

What this PR does / why we need it:
The issue is described well in #908. This PR allows etcd-druid to patch the StatefulSet allowing the unhealthy pod to come up eventually if what has changed is only TLS volume mount path.

NOTE: If the TLS has been enabled for peer URL and is still not been reflected in StatefulSet then to ensure that the cluster comes up post update of StatefulSet, it is essential that all members be up and running in a 3 member cluster. Therefore that restriction will still be in place. What this PR does is relax this restriction in case only the mount paths have changed.
v0.23.x changes the mount paths for TLS volumes.

Additionally it also now adds an additional label to the etcd client and peer service to ensure that only statefulset pods are selected by it.

Which issue(s) this PR fixes:
Fixes #908 and #914

Release note:

Fixes etcd client and peer service label selector, ensuring that only Etcd statefulset pods are selected.

etcd controller now differentiates between TLS configuration change and peer TLS enablement. Only if peer TLS has been enabled and not yet reflected it will wait for all pods to come up else it will allow patching of statefulset.

[ishan16696]

/assign @ishan16696 @seshachalam-yv

[unmarshall]

/retest

[unmarshall]

/retest

[ishan16696]

I have mentioned the steps to reproduce the bug and to test the fix : #908 (comment)

[seshachalam-yv]

/lgtm

[ishan16696]

I have also tested the compaction job fix:

Before the fix etcd compaction pod is also added to endpointslices due to label selector of svc.

$kubectl describe endpointslices.discovery.k8s.io etcd-main-client-qght6
Name:         etcd-main-client-qght6
....
Endpoints:
  - Addresses:  10.244.0.19
    Conditions:
      Ready:    true
    Hostname:   <unset>
    TargetRef:  Pod/etcd-main-0
    NodeName:   kind-control-plane
    Zone:       <unset>
  - Addresses:  10.244.0.20
    Conditions:
      Ready:    true
    Hostname:   <unset>
    TargetRef:  Pod/etcd-main-compactor-stvkj
    NodeName:   kind-control-plane
    Zone:       <unset>
Events:         <none>

$ kubectl describe endpointslices.discovery.k8s.io etcd-main-peer-cbc4h

Name:         etcd-main-peer-cbc4h
...
Endpoints:
  - Addresses:  10.244.0.19
    Conditions:
      Ready:    true
    Hostname:   etcd-main-0
    TargetRef:  Pod/etcd-main-0
    NodeName:   kind-control-plane
    Zone:       <unset>
  - Addresses:  10.244.0.20
    Conditions:
      Ready:    true
    Hostname:   <unset>
    TargetRef:  Pod/etcd-main-compactor-stvkj
    NodeName:   kind-control-plane
    Zone:       <unset>
Events:         <none>

Then I deployed druid with this PR image and trigger the compaction pod, etcd compaction pod is not added to endpointSlices as now services are selecting the only etcd statefulset pods.

NAME                          READY   STATUS    RESTARTS   AGE
etcd-main-0                   2/2     Running   0          9m16s
etcd-main-compactor-88zsb     1/1     Running   0          90s

k describe endpointslices.discovery.k8s.io etcd-main-client-qght6
Name:         etcd-main-client-qght6
...
Endpoints:
  - Addresses:  10.244.0.19
    Conditions:
      Ready:    true
    Hostname:   <unset>
    TargetRef:  Pod/etcd-main-0
    NodeName:   kind-control-plane
    Zone:       <unset>
Events:         <none>

k describe endpointslices.discovery.k8s.io etcd-main-peer-cbc4h
Name:         etcd-main-peer-cbc4h
...
Endpoints:
  - Addresses:  10.244.0.19
    Conditions:
      Ready:    true
    Hostname:   etcd-main-0
    TargetRef:  Pod/etcd-main-0
    NodeName:   kind-control-plane
    Zone:       <unset>
Events:         <none>

[ishan16696]

LGTM!!

[unmarshall]

/test pull-etcd-druid-e2e-kind-nondistroless-etcd