Drop PodDisruptionBudget for calico-kube-controllers

[timebertt]

How to categorize this PR?

/area networking
/kind bug

What this PR does / why we need it:

Drop PodDisruptionBudget for calico-kube-controllers, because

• it can anyways only be a single replica:

  gardener-extension-networking-calico/charts/internal/calico/templates/kube-controller/deployment-calico-kube-controllers.yaml

  Lines 14 to 15 in 1e73ca7

  	# The controllers can only have a single active instance.
  	replicas: 1

• it can prevent the VPA from updating calico-kube-controllers in case it is in CrashLoopBackOff because of OOM kills.

A few references for context:

• Don't consider Crashloopbackoff pods as part of PDB machine-controller-manager#684
• [wip] Always allow an unready pod to be evicted. kubernetes/kubernetes#105296 (comment)
• [VPA] Feature Request: Delete OOM Pods kubernetes/autoscaler#4730

Which issue(s) this PR fixes:

calico-kube-controllers was in CrashLoopBackOff because of OOM kills:

$ ks get po -l k8s-app=calico-kube-controllers
NAME                                       READY   STATUS             RESTARTS   AGE
calico-kube-controllers-7d557d76c6-dqq8k   0/1     CrashLoopBackOff   59         4h43m

Hence, the PDB had status.disruptionsAllowed=0:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: calico-kube-controllers
  namespace: kube-system
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
status:
  currentHealthy: 0
  desiredHealthy: 0
  disruptionsAllowed: 0
  expectedPods: 1
  observedGeneration: 1

Which caused kube-apiserver to deny the eviction by VPA:

I0519 01:18:03.188662       1 update_priority_calculator.go:114] quick OOM detected in pod kube-system/calico-kube-controllers-7d557d76c6-g7lv2, container calico-kube-controllers
I0519 01:18:03.188675       1 update_priority_calculator.go:143] pod accepted for update kube-system/calico-kube-controllers-7d557d76c6-g7lv2 with priority 0.37876744005850843
I0519 01:18:03.188692       1 updater.go:215] evicting pod calico-kube-controllers-7d557d76c6-g7lv2
E0519 01:18:03.191952       1 pods_eviction_restriction.go:141] failed to evict pod kube-system/calico-kube-controllers-7d557d76c6-g7lv2, error: Cannot evict pod as it would violate the pod's disruption budget.
W0519 01:18:03.191970       1 updater.go:218] evicting pod calico-kube-controllers-7d557d76c6-g7lv2 failed: Cannot evict pod as it would violate the pod's disruption budget.

The issue is actually a more general one. However, removing PDBs for singleton pods is the best thing we can do for now to ensure VPA keeps scaling up components on CrashLoopBackOff because of OOM kills.

Special notes for your reviewer:

Release note:

The `PodDisruptionBudget` for `calico-kube-controllers` is removed, as it is a singleton and can prevent VPA from scaling it up.

[DockToFuture]

/lgtm

[timebertt]

@DockToFuture @ScheererJ should we merge this PR?

[timebertt]

ping @DockToFuture @ScheererJ

[ScheererJ]

/lgtm