Support node taints in cluster config

[jiayiwang7]

What would you like to be added:

Add node taints option in cluster config for both control plane and worker node groups. So that the defined taints can be attached to the node(s) automatically during cluster creation/upgrade.

cluster config

We add taints option at the cluster level, eg:

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: cluster-1
spec:
  ...
  controlPlaneConfiguration:
    count: 2
    endpoint:
      host: "198.18.100.27"
    machineGroupRef:
      kind: VSphereMachineConfig
      name: cluster-1-cp
    taints: # optional
    - key: key1
      value: val1 # optional
      effect: NoSchedule # Valid effects are NoSchedule, PreferNoSchedule and NoExecute.
  workerNodeGroupConfigurations:
  - count: 2
    machineGroupRef:
      kind: VSphereMachineConfig
      name: cluster-1
    taints: # optional
    - key: key2
      value: val2 # optional
      effect: PreferNoSchedule # Valid effects are NoSchedule, PreferNoSchedule and NoExecute.

implementation

Cluster API already supports taints for both control plane and machine deployment:

• CP: https://github.com/kubernetes-sigs/cluster-api/blob/master/controlplane/kubeadm/config/crd/bases/controlplane.cluster.x-k8s.io_kubeadmcontrolplanes.yaml#L708 and https://github.com/kubernetes-sigs/cluster-api/blob/master/controlplane/kubeadm/config/crd/bases/controlplane.cluster.x-k8s.io_kubeadmcontrolplanes.yaml#L895
• MD: https://github.com/kubernetes-sigs/cluster-api/blob/master/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigtemplates.yaml#L656

We might easily pass down the taints option to the CAPI spec before applying.

The CAPI taints option does not work for worker nodes (kubeadmconfigtemplates). We https://github.com/kubernetes-sigs/cluster-api/blob/main/controlplane/kubeadm/controllers/controller.go#L267might add taints for worker nodes in the CLI workflow, or as a postKubeadmCommands defined in the kubeadmconfigtemplates.

Why is this needed:

User requests to allow the retention of node taints during an upgrade. Currently the rolling update recreates VMs and the new nodes joined do not have the taints defined in the original nodes. User needs to manually attach those taints back after upgrade.

We'd like to provide a descriptive way to let user define node taints in the cluster config, so that the nodes being created/upgraded can have the proper taints (if defined) on the node(s) automatically without manual setup.

TODO list

• cp taints POC Support adding taints on control plane nodes #359
• worker taints POC Worker node taints #977
• better unit test coverage for worker node taints implementation #1127
• Validate at least one worker node group configuration is free from NoSchedule or NoExecute taints #1126
• doc update for taints #1120
• e2e tests for taints #1121
• Wait for the controlplane endpoints to be ready before waiting for the kubeconfig secret #629
• Add unit tests for BR taints config code in the kubeadm bootstrap controller #1085
• Taints Support Cleanup #1125
• unit tests for the kubectl exe methods introduced in Taints validation #1225
• e2e tests for CP node taints #1226
• e2e tests for BR taints #1227

TBD:

• support worker nodes taints update without VM recreate

[danbudris]

an update on this issue:

In order to provide controller support for taints, we'll need to add a mechanism by which we generate new kubeadmconfigtemplates with an update joinConfiguration containing the requisite taints, and then modify the machineDeployemnts which reference those kubeadmconfigtemplates to reference the new kubeadmconfigtemplate; this way we can trigger a rollout of the machinedeployments, causing new machines to be launched, and the updated joinConfiguration in the kubeadmconfigtemplate to go into effect. This will also address SSH key rolling.

As of right now, upstream intentionally does not support 'immutable' kubeadmconfigtemplates, but the official guidance is to treat them in the same way we treat the immutable machineTemplates -- e.g. create a new one with the updates, modify the machineDeployment to use it, trigger rollout & node recreation.

See:

• Make KubeadmConfig and the KubeadmConfigTemplate immutable kubernetes-sigs/cluster-api#4910
• https://cluster-api.sigs.k8s.io/tasks/updating-machine-templates.html

[danbudris]

this feature is released in 0.7.1