Add "safe-to-evict" autoscaler annotations for components with emptyDir mounted

[pkosiec]

Description

Changes proposed in this pull request:

• Add "safe-to-evict" autoscaler annotations for components with emptyDir mounted

Prometheus, Alertmanager and Neo4j have emptyDir mounted which prevent them to scale down clusters that use Cluster Autoscaler, e.g. our long-running one on GCP:

While I didn't see such errors for kubed, argo server and grafana, they also use emptyDir for cache, so I annotated them just to be safe.

Notes

If we upgraded to GKE 1.22 we wouldn't need such annotations on the pods:
https://cloud.google.com/kubernetes-engine/docs/release-notes#October_27_2021

There's already open task for that: #611

Testing

Run make dev-cluster and observe new annotations:

kubectl get po -n capact-system > /tmp/pods.yaml

Open the file in a text editor and verify that for all pods with emptyDir the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "true" is properly set.

Related links

• https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node
• How do I gain access to the cluster autoscaler on GKE? kubernetes/autoscaler#966

[mszostok]

tested with helm template. LGTM 👍