Define upgrade path for ComponentConfig

[fabriziopandini]

kubeadm allow usage of component config for defining settings for kubelet and kube-proxy; component configs are stored in ConfigMaps and reused when joining nodes or doing upgrades.

However, as of today kubeadm does not yet define a procedure for upgrading component configs in the case such API change format e.g. change from kube-proxy/config/v1alpha1 to kube-proxy/config/v1alpha2.

As an initial step, we should evaluate if this is the scope of kubeadm because it requires a certain degree of coordination across nodes (there should not be old components referring to a new/not supported component config), and this can't be achieved by kubeadm alone.
If this is the case, it should be documented what is the expected procedure higher-level tools/user's should take care of.

It is also important to keep in mind nuances of component config management for kubelet (1CM per version, copied locally) and kube-proxy (1CM, linked by all instances).

k/e issue + KEP:
kubernetes/enhancements#1381

/cc @rosti

[rosti]

I was actually raising this exact question in the past few weeks with WG Component Standard, but so far no concrete decisions have been taken.
This same problem is faced, not only by kubeadm, but by other cluster provisioning tools too (like kops for instance). So I suppose that we need a broader WG Component Standard meeting with kubeadm and other cluster provisioning tools on that matter.
@mtaufen @stealthybox @sttts @justinsb WDYT?

[neolit123]

/assign @rosti
xref https://github.com/kubernetes/enhancements/pull/1256/files

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/lifecycle frozen

[xlgao-zju]

like what we discussed in #1582
seems up-convert these component configs is out of kubeadm's scope...?

[neolit123]

yes, unless the component start exposing a conversion path for their managed ComponentConfig API / types.

[wangyysde]

/cc @wangyysde

[neolit123]

the kube-proxy v1alpha2 is planned for release in 1.31.
kubernetes/kubernetes#121830
cc @SataQiu @pacoxu @carlory

kube-proxy has enabled the following as a migration plan, which is good:

https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/784-kube-proxy-component-config/README.md#upgrade--downgrade-strategy

/usr/local/bin/kube-proxy --config old-v1alpha1.yaml --write-config-to new-v1alpha2.yaml

since kubeadm runs kube-proxy as a container / DS we might have to jump a couple of additional hoops, but up-converting the already-in-cluster kube-proxy config to v1alpha2 seems doable.

• during upgrade apply if there is a kube-system/kube-proxy config CM:
  • upgrade kube-proxy to 1.31 or a version that has v1alpha2.
  • download the config and save it somewhere locally
  • start a Job with the kube-proxy container, mounting the config location
  • call kube-proxy --config old-v1alpha1.yaml --write-config-to ... from the container
  • write the resulting config to the kube-system/kube-proxy config CM

i will experiment with this and provide a PR or if there are complications this can be a google doc first.
please add any comments if needed.

[neolit123]

looks like both kube-proxy and kube-scheduler support the --write-config-to flag. the kubelet does not yet
logged this ticket for it:
kubernetes/kubernetes#125344

[neolit123]

in terms of the ticket here, IMO the upgrade path is:

• on upgrade kubeadm calls the component binary to migrate the user config on disk and in cluster config maps. this should be done after kube-proxy is upgraded to 1.31 or a version that has v1alpha2.
• on init similar is done if the user passes an old config version. tricky for kube-proxy on init but doable.
  • kubeadm continues to accept the old CC config on init for N releases (to be discussed).
  • after N releases kubeadm stops doing the migration on init, but can throw an error.