Centralize Node Cordon & Drain

[michaelgugino]

/kind feature

Describe the solution you'd like
Currently, provider Actuators may (or may not) implement Node cordon & drain before delete. We should do this in the machine-controller.

Anything else you would like to add:
To be provider neutral, we may need to add an extension to the Actuator interface such as "preDrain()" that allows a provider to do things like label/annotation VMs (such as in an auto scale group) or similar action.

Here's how OpenShift is doing it today: https://github.com/openshift/cluster-api/blob/openshift-4.0-cluster-api-0.0.0-alpha.4/pkg/controller/machine/controller.go#L195

[detiber]

The main impediment to this today is the lack of remote node references. It'll work for self-managed clusters, but if a separate management cluster is used it will not work today.

[ncdc]

Remote node ref issue: #520

[detiber]

/assign @michaelgugino

[michaelgugino]

/active

[ncdc]

@michaelgugino FYI it's slash-lifecycle active - thanks!

[michaelgugino]

/lifecycle active

[dlipovetsky]

There was a #cluster-api Kubernetes slack thread on this topic on 09/24/19-09/25/19.

Quoting myself here:

If drain is imperative and client-side (which it is today, AFAIK), then a CAPI controller can/should implement drain. But if drain becomes declarative and server-side (e.g. something that happens as the result of Taints applied to a Node), then CAPI controller can/should apply the Taints.

Moving drain server-side is in progress upstream: kubernetes/kubernetes#25625. There is a Node Maintenance Controller KEP in progress right now: kubernetes/enhancements#1080.