Components that are eligible for leaderelection should use anti-affinity

[mattmoor]

/area API
/kind feature
/kind good-first-issue

I noticed this in Contour's controller yaml, and I think it's probably a good pattern to follow:

      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: contour
              topologyKey: kubernetes.io/hostname
            weight: 100

Basically, if one of the HA scenarios you want to handle is disruptions due to node failures, then we should make sure that replicas of each component aren't scheduled on the same node as one another.

cc @pmorie for thoughts / alternatives
cc @lionelvillard @markusthoemmes @mgencur who have expressed interest in this space in other contexts FYI

[mattmoor]

We should probably also consider PodDisruptionBudget, but AIUI those create problems for single-replica controllers because they can no longer receive updates(?).

cc @tcnghia who IIRC hit this with Istio

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.