kubernetes-cni pinned to 1.1.1*, but kubeadm requires >=1.2.0 for k8s 1.24.10

[7oku]

What happened?

Upgrading kubernetes from 1.24.9 to 1.24.10 via kubeone 1.5.5 fails.

Expected behavior

kubeone should be able to upgrade kubernetes

How to reproduce the issue?

run kubeone apply -f kubeone.yml

What KubeOne version are you using?

kubeone 1.5.5

{
  "kubeone": {
    "major": "1",
    "minor": "5",
    "gitVersion": "1.5.5",
    "gitCommit": "99303c67363dc3788e24ff47b4ae0eae3ac7b8ae",
    "gitTreeState": "",
    "buildDate": "2023-01-17T10:27:32Z",
    "goVersion": "go1.19.4",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "machine_controller": {
    "major": "1",
    "minor": "54",
    "gitVersion": "v1.54.2",
    "gitCommit": "",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "",
    "compiler": "",
    "platform": "linux/amd64"
  }
}

Provide your KubeOneCluster manifest here (if applicable)

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: '1.24.10'
cloudProvider:
  hetzner: {}
  external: true
clusterNetwork:
  kubeProxy:
    skipInstallation: true
  cni:
    cilium:
      enableHubble: true
      kubeProxyReplacement: strict
apiEndpoint:
  alternativeNames: ["<domain>"]
addons:
  enable: true
  addons:
  - name: default-storage-class

What cloud provider are you running on?

Hetzner

What operating system are you running in your cluster?

Ubuntu 20.04 Focal

Additional information

We're failing to upgrade k8s to the latest 1.24.10 version, because kubeadm requires package kubernetes-cni v1.2.0. However, somehow the kubernetes-cni version is pinned to 1.1.1* :

[<ip>] + sudo DEBIAN_FRONTEND=noninteractive apt-get install --option Dpkg::Options::=--force-confold --no-install-recommends -y 'kubeadm=1.24.10*' 'kubernetes-cni=1.1.1*'
[<ip>] Reading package lists...
[<ip>] Building dependency tree...
[<ip>] Reading state information...
[<ip>] kubernetes-cni is already the newest version (1.1.1-00).
[<ip>] Some packages could not be installed. This may mean that you have
[<ip>] requested an impossible situation or if you are using the unstable
[<ip>] distribution that some required packages have not yet been created
[<ip>] or been moved out of Incoming.
[<ip>] The following information may help to resolve the situation:
[<ip>] 
[<ip>] The following packages have unmet dependencies:
[<ip>]  kubeadm : Depends: kubernetes-cni (>= 1.2.0)
[<ip>] E: Unable to correct problems, you have held broken packages.
time="11:05:54 UTC" level=warning msg="Task failed, error was: runtime: running task on \"<ip>\"\nssh: upgrading kubeadm and CNI plugins\nssh: popen\nProcess exited with status 100"

What's the reason for pinning kubernetes-cni to 1.1.1* and can it be safely bumped to 1.2.0 to allow upgrade to k8s 1.24.10? I additionally want to highlight we are using cilium as cni!

[xmudrii]

This is a known issue and we're working on fixing it (see #2606). We'll provide more updates regarding this issue next week.

[7oku]

@xmudrii awesome thanks. I didn't find this prior opening this issue, seems they were timed very close to each other. I'll keep an eye on #2606!

[wethinkagile]

Running into something similar, we are getting:

+ sudo systemctl daemon-reload
+ sudo systemctl enable containerd
+ sudo systemctl restart containerd
+ sudo DEBIAN_FRONTEND=noninteractive apt-get install --option Dpkg::Options::=--force-confold --no-install-recommends -y 'kubelet=1.24*' 'kubeadm=1.24*' 'kubectl=1.24*' 'kubernetes-cni=1.1.1*'
E: Unable to correct problems, you have held broken packages.

Now it loops. This is 1.24* on latest kubeone and Hetzner.

1.24.9 in the meanwhile worked.

[Frisch12]

Have the same problem on esxi/vsphere

+ sudo DEBIAN_FRONTEND=noninteractive apt-get install --option Dpkg::Options::=--force-confold --no-install-recommends -y 'kubeadm=1.24.10*' 'kubernetes-cni=1.1.1*'
E: Unable to correct problems, you have held broken packages.

[xmudrii]

We just released v1.5.6 which includes a fix for this issue. We're still working on fixing it for v1.4, but at this time, we strongly recommend upgrading to v1.5.6.

[7oku]

Thanks for the quick resolution @xmudrii, we've successfully upgraded our clusters to 1.24.10 using kubeone 1.5.6. As we are not using 1.4 feel free to close this issue whenever you're ready.

[xmudrii]

KubeOne 1.4 has reached end-of-life with the KubeOne 1.6 release. Therefore, we're not planning on fixing this bug for KubeOne 1.4 as it would require significant effort. We strongly recommend that everyone upgrades to KubeOne 1.5 or 1.6. I'll go ahead and close this issue.
/close

[kubermatic-bot]

@xmudrii: Closing this issue.

In response to this:

KubeOne 1.4 has reached end-of-life with the KubeOne 1.6 release. Therefore, we're not planning on fixing this bug for KubeOne 1.4 as it would require significant effort. We strongly recommend that everyone upgrades to KubeOne 1.5 or 1.6. I'll go ahead and close this issue.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.