Check for misconfigured PDBs on Node drain and set proper error description

[ialidzhikov]

/area ops-productivity
/kind enhancement

Currently the Machine status does not clearly indicate pod eviction failures that are caused by misconfigured PodDisruptionBudgets.

With this PR the Machine .status.lastOperation.description clearly indicates for misconfigured PodDisruptionBudgets:

  lastOperation:
    description: 'Drain failed due to - [error while evicting pod "nginx-deployment-6b474476c4-2mck6":
      pod disruption budget default/pdb is misconfigured and requires zero voluntary
      evictions, error while evicting pod "nginx-deployment-6b474476c4-8b8hl": pod
      disruption budget default/pdb is misconfigured and requires zero voluntary evictions].
      Will retry in next sync. Initiate node drain'
    lastUpdateTime: "2021-01-26T21:12:56Z"
    state: Failed
    type: Delete

This will allow components like extension controllers to match .status.lastOperation.description and to properly flag it as configuration problem - ref gardener/gardener#3020.

Which issue(s) this PR fixes:
Needed for gardener/gardener#3020

Release note:

machine-controller-manager now checks for misconfigured PodDisruptionBudgets (ones that require zero voluntary evictions and make impossible the graceful Node drain) and sets better Machine `.status.lastOperation.description` for such Machines. This change is breaking as out-of-tree providers need new RBAC permissions - list and watch access for PodDisruptionBudgets in the target cluster.

[prashanth26]

/assign

[prashanth26]

Thank you for this change @ialidzhikov . A very meaningful change.

Minor comment: This change is only on the OOT branch of MCM and this change wouldn't be backward compatible with any controllers (example: extension-provider-azure or other external controllers using in-tree) currently using in-tree MCM. This is fine as we anyways would deprecate the in-tree MCM code eventually move to OOT code, just wanted to mention/remind you.

Changes /lgtm otherwise

[prashanth26]

Just a reminder: We will have to update all the extensions to adapt the cluster-roles to add this post merging of this PR.

[prashanth26]

/lgtm

[ialidzhikov]

Minor comment: This change is only on the OOT branch of MCM and this change wouldn't be backward compatible with any controllers (example: extension-provider-azure or other external controllers using in-tree) currently using in-tree MCM. This is fine as we anyways would deprecate the in-tree MCM code eventually move to OOT code, just wanted to mention/remind you.

Sure, I have in mind to update the RBAC of the gardener provider extensions when we update the corresponding MCM provider version in the extension that vendors this PR. For the gardener provider extensions that still use in-tree MCM, there is no action required.

[prashanth26]

Sure, I have in mind to update the RBAC of the gardener provider extensions when we update the corresponding MCM provider version in the extension that vendors this PR. For the gardener provider extensions that still use in-tree MCM, there is no action required.

Sure sound good. We will also need to vendor the library into the MCM OOT providers and then make releases for this change to kick in. Which is fine for now. So i shall go ahead and merge this?

[ialidzhikov]

Sure sound good. We will also need to vendor the library into the MCM OOT providers and then make releases for this change to kick in. Which is fine for now.

Yep.

So i shall go ahead and merge this?

If it also looks good to you, then we can proceed with it.