kubelet-config-x.y ConfigMap logic improvements

[neolit123]

tracking issue in k/e + KEP: kubernetes/enhancements#2915

1.23 Alpha:

• deprecate the old format and introduce the new one controlled by UnversionedKubeletConfigMap:
  kubeadm: introduce the UnversionedKubeletConfigMap feature gate kubernetes#105741
• add e2e tests in k/kubeadm:
  kinder: add e2e test for UnversionedKubeletConfigMap (Alpha) #2601

1.24 Beta:

• update KEP:
  kubeadm: update 2915-kubeadm-replace-kubelet-config-x.y for 1.24 enhancements#3120
• flip the FG to enabled and graduate it to Beta, update the appropriate Beta TODOs introduced in the original PR:
  kubeadm: switch UnversionedKubeletConfigMap to true kubernetes#108027
• other k/k fixes:
  test/e2e/framework: include the new control plane taint for "--non-blocking-taints" kubernetes#108336
  Automated cherry pick of #108127: test/e2e_kubeadm: fix matching UnversionedKubeletConfigMap kubernetes#108128
  test/e2e_kubeadm: fix matching UnversionedKubeletConfigMap defaults kubernetes#108127
• cleanup k/kubeadm / update e2e tests:
  kinder: add handling of the new/old kubelet-config map #2654
• docs: cleanup any kubelet-config-x.yy instances in at k/website
  kubeadm: update legacy kubelet config map information and document feature gates website#31687

1.25 GA:

• update KEP to GA:
  sig-cl/kubeadm/2915: update KEP for 1.25 enhancements#3316
• remove logic related to the feature gate and lock it to true. update the appropriate GA TODOs introduced in the original PR:
  kubeadm: graduate the UnversionedKubeletConfigMap FG to GA kubernetes#110327
  kubeadm: UnversionedKubeletConfigMap is GAed in v1.25 kubernetes#111146
• remove e2e tests in k/kubeadm as the default e2e tests will exercise this
  kubeadm: remove unversioned-kubelet-config-map test job test-infra#26461
  kinder: remove workflows for UniversionedKubeletConfigMap #2702
• k/website docs (update kubeadm-init page FG section)
  kubeadm: update unversioned kubelet config map fg info website#34159

1.26 post-GA:

• remove the feature gate
  remove UnversionedKubeletConfigMap feature gate: GAed in v1.25 kubernetes#113448

---

EDIT: original issue description below

when a control-plane node is created by init using kubernetesVersion: X.Y, kubeadm will create a "versioned" ConfigMap called kubelet-config-X.Y.

when a worker node joins the cluster, it uses the version of the local kubelet to fetch this ConfigMap, but if the version of the local kubelet differs from the control-plane version (X.Y) used to originally create the CM, the join process will just fail.

both init and upgrade use the version of the control-plane to manage this ConfigMap,
so it makes some sense for the join process to use the version of the control-plane as well.
but this raises the question whether this internal config stored in the CM will be compatible with the local kubelet.

however we have a version skew topic to discuss here as well.
Kubernetes itself supports X-1 nodes joining a X cluster, but kubeadm does not.

in our documentation we have no official mentions if e.g a kubeadm 1.13 and a kubelet 1.13 worker node can join a cluster with a CP that is 1.14:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#version-skew-policy

[SataQiu]

Maybe we can get the cluster version through kubeadm-config ConfigMap when joining new work node.

# kubectl describe cm kubeadm-config -n kube-system
Name:         kubeadm-config
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
ClusterConfiguration:
----
apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.13.0
networking:
  dnsDomain: cluster.local
  podSubnet: ""
  serviceSubnet: 10.96.0.0/12
scheduler: {}

ClusterStatus:
----
apiEndpoints:
  test-01:
    advertiseAddress: 192.168.137.139
    bindPort: 6443
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterStatus

Events:  <none>

The ClusterConfiguration contains kubernetesVersion: v1.13.0.

[neolit123]

@SataQiu
the PR you sent is something we've discussed yesterday - use kubernetesVersion.
so maybe we can merge that as a start in 1.16.

but the issue has deeper roots and we possibly need to discuss all the implications here.
e.g. versioning the kubelet-config CM, the usage of internal types and the component skew as a whole.

[rosti]

kubelet's component config is a versioned API. It is at beta level now and I don't expect catastrophic incompatibility issues between a couple of consecutive kubelet versions, especially on join.
So I am all in for having equivalent behavior between the different kubeadm sub-commands.

[rosti]

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[neolit123]

/lifecycle frozen

[pacoxu]

For beta in 1.24, is cleanup any kubelet-config-x.yy instances in kubeadm too aggressive?
In my mind, the cleanup should happen when the feature is GA. Or users cannot roll back easily.

[neolit123]

For beta in 1.24, is cleanup any kubelet-config-x.yy instances in kubeadm too aggressive?
In my mind, the cleanup should happen when the feature is GA. Or users cannot roll back easily.

i think i mean clearing docs / code comments.
the code path for the x.yy variant will still be supported.

[pacoxu]

Oops. Got it.

[neolit123]

looks like kubernetes/kubernetes#108027 broke https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-latest-on-1-23

that is a skew test.
maybe we need to one day make the k/k -> /test/e2e_kubeadm suite match the kubeadm version and not the k8s version.
if it matches the k8s version it also means it must support the kubeadm skew of k8s N and N-1.

the fix is to properly default the FG value to match the kubeadm features.go based on k8s MINOR version.

the fix here is to make test/e2e_kubeadm/kubelet_config_test.go tollerate both the old and new object names in the 1.23 branch of k/k.

i will send a PR later.

[pacoxu]

The CI is green again.

v1.24 tasks are finished and the milestone can move to 1.25.

[neolit123]

the feature gate was removed for 1.26. this is complete.