Fix liveness and readiness probes

[timuthy]

How to categorize this PR?

/area quality
/kind bug

What this PR does / why we need it:
This PR fixes several issues for the currently used liveness and readiness checks and also adds a startup probe for single- and multi-node etcds.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
Issue with current liveness probe:

• When passing multiple arguments to /bin/sh -ec then only the first argument is considered, i.e. /bin/sh -ec ETCDCTL_API=3 is always successful.

            - /bin/sh
            - -ec
            - ETCDCTL_API=3
            - etcdctl
            - --cert=/var/etcd/ssl/client/client/tls.crt
            - --key=/var/etcd/ssl/client/client/tls.key
            - --cacert=/var/etcd/ssl/client/ca/ca.crt
            - --endpoints=https://etcd-aws-local:2379/
            - get
            - foo
            - --consistency=s

Issue with current readiness probe:

• The exec command did not evaluate the return code of the HTTP response and thus the container was considered ready even though the /health(z) endpoint returned != 200.

            - /usr/bin/curl
            - --cert
            - /var/etcd/ssl/client/client/tls.crt
            - --key
            - /var/etcd/ssl/client/client/tls.key
            - --cacert
            - /var/etcd/ssl/client/ca/ca.crt
            - https://etcd-aws-local:8080/healthz

For single-node it's possible to switch to an http probe to solve the explained issue.

For multi-node it's necessary to use an exec probe because the /health endpoint of etcd is protected by mutual TLS (also see etcd-io/etcd#12370) and providing a client cert is not supported via Kubernetes http probes.

The new liveness probe is now accurate, but still fails after few seconds due to the backup sidecar requiring a long time to promote its etcd member from learner. This leads to the etcd container being restarted, and the backup sidecar's initialization fails, and begins another initialization when the etcd container comes back up. This cycle continues, and the etcd pods never become ready. This problem is solved by using a startup probe of 2 minutes to allow the initialization to complete without interruptions due to etcd container restarts.

Release note:

An issue has been fixed that caused the `liveness` and `readiness` probes of `etcd` to always succeed even though an error was reported. This prevented defective etcd pods from being restarted automatically and caused unready candidates being considered as ready to serve traffic via the `etcd service`.

A `startup` probe has been added to `etcd` to allow 2 minutes of initialization time before checking for etcd liveness.

Add support for running envtest on M1 Macbooks.

[shreyas-s-rao]

@timuthy Thanks for the quick fix! Overall LGTM except for a small nit.
/lgtm

[aaronfern]

Thanks for the PR @timuthy!

One comment from me

[timuthy]

Thanks for the reviews @shreyas-s-rao and @aaronfern 🚀 I fixed the type, PTAL.

[shreyas-s-rao]

@timuthy the unit tests for sts component are failing because the probes in validateEtcd and validateEtcdWithDefaults functions in the test need to be adapted. Can you please make that change? Thanks!

[shreyas-s-rao]

/hold

[aaronfern]

/lgtm