etcd client and peer service label selector causes etcd compaction job pods to be included in its endpoint slices

[unmarshall]

How to categorize this issue?

/area control-plane
/kind bug

What happened:
Compaction controller creates a Job to run compaction based on event threshold. The labels on the pod that gets created has labels (see code).
When etcd controller creates resources for an Etcd cluster, then it creates a client service and a peer service.
The client and peer service has the following label selector:

  selector:
    app.kubernetes.io/managed-by: etcd-druid
    app.kubernetes.io/part-of: etcd-test

The above labels are also present on the pods that are started via the compaction Job. This results in compaction Job getting included in the endpoint slices as it gets selected due to the above selector along with the etcd statefulset pods. This causes API server to fail as it can potentially connect to the compaction job pod for etcd requests.

What you expected to happen:
etcd compaction job pods should never be included in the endpoint slices for etcd client and peer services.

How to reproduce it (as minimally and precisely as possible):
Deploy etcd-druid version v0.23.2. Trigger a compaction job and allow it to run for some time. In the mean time kill the API server pod and let it restart again and hope that it now connects to the compaction job pod.

Environment:

• etcd-druid version: v0.23.2