Release Proposal v2.21

[floryut]

Deprecation / Removal

• Drop calico v3.21 support (Drop calico v3.21 support #9515, @oomichi)

Feature / Major Changes

• Add Check resolv.conf is empty to avoid CoreDNS crash (Add Check resolv.conf is empty to avoid CoreDNS crash #9502, @yankay)
• Add XDG related Helm paths to be removed from reset tasks (Add XDG related Helm paths to be removed #9561, @emiran-orange)
• Add a parameter (disable_host_nameservers) to disable host nameservers (feat: add a parameter to disable host nameservers #9357, @eminaktas)
• Add an option (populate_loadbalancer_apiserver_to_hosts_file) to skip adding load balancer name in the hosts file (Add possibility to skip adding load balancer name in the hosts file #9331, @JRaver)
• Add custom options to coredns kubernets plugin (coredns_kubernetes_extra_opts ) ([feat] Add custom options to coredns kubernets plugin #9608, @mvandergiesen)
• Add docker support for openEuler linux (Add docker support for openEuler linux #9498, @ErikJiang)
• Add support for the OpenEuler Linux (Add support for the OpenEuler Linux #9494, @ErikJiang)
• Add terraform script for Flatcar Linux on Hetzner (Added terraform script for Flatcar Linux on Hetzner #9618, @florianow)
• Add the ability to define options for DNS upstream servers (using new variable dns_upstream_forward_extra_opts) (Ability to define options for DNS upstream servers #9311, @emiran-orange)
• Add var (ingress_nginx_probe_initial_delay_seconds) for control initialDelaySeconds in ingress-nginx probes ([Ingress_nginx] Add var for control initialDelaySeconds #9405, @zvlb)
• Add variable condition snapshot in vSphere CSI (vsphere_csi_block_volume_snapshot) (add variable condition snapshot in vSphere CSI #9429, @yanggangtony)
• Add variable in metrics_server deployment (metrics_server_replicas) to enable HA mode ([metrics_server]: Enabled HA mode by adding 'metrics_server_replicas'… #9539, @ugur99)
• Change dns upstream condition for nodelocaldns when using host_resolvconf (Change dns upstream condition for nodelocaldns #9378, @unai-ttxu)
• Download coredns image to all hosts in k8s_cluster (Download coredns image to all hosts in k8s_cluster #9316, @joes)
• Enable check mode in DNS Cleanup tasks (Enable check mode in DNS Cleanup tasks #9472, @emiran-orange)
• Etcd image has the same tag accross multiple archs (Update etcd image tag with the same tag between multi archs #9516, @hangscer8)
• Fix a pre-upgrade node drain rescue task failure when kube_override_hostname is set (Fix a pre-upgrade node drain rescue task failure when kube_override_hostname is set #9556, @chadswen)
• Fix default value for kubelet_secure_addresses (Fix default value for kubelet_secure_addresses #9355, @willtrnr)
• Provides <kubeadm_init_timeout> to change the timeout of first control-plane initialization (Allow custom timeout for kubeadm init #9617, @tu1h)
• Remove PodSecurityPolicies in MetalLB for kubernetes 1.25 (Remove PodSecurityPolicies in Metallb for kubernetes 1.25 #9442, @yanggangtony)
• Support Python 3.11 - ruamel.yaml.clib need to be updated to 0.2.7 (Upgrade ruamel.yaml.clib to work with Python 3.11 #9426, @olivierlemasle)
• Support customize the additional sysctl variables using additional_sysctl (Support customize the additional sysctl variables #9351, @yankay)
• Support patches field in kubeadm v1beta3 in both InitConfiguration and JoinConfiguration (using new variable kubeadm_patches) (Features/support kubeadm patches v1beta3 #9326, @titaneric)
• Switch helm install (from synchronize to copy) to support password authentication (fix helm install with password authentication #9343, @ghostloda)
• Update api version for pdb and batch (deprecated in 1.25) (Update api version, deprecated in 1.25 #9369, @yankay)
• Update dashboard image repo to remove arch flag (update dashboard image repo to remove arch flag #9530, @tu1h)
• Update etcd log-level parameter name (new name: ETCD_LOG_LEVEL) (Update etcd log-level parameter name #9540, @ErikJiang)
• Update local-volume-provisioner to 2.5.0 + add documentation (Update local-volume-provisioner + doc #9463, @olivierlemasle)
• Update the number of nofile limits in containerd to 65535 (Update the number of nofile limits in containerd #9507, @ErikJiang)
• Upgrade metrics server to v0.6.2 ([feat] Upgrade metrics server to v0.6.2 #9554, @mzaian)
• Upgrade the load balancer ( nginx and haproxy ) image version. (Update load balancers Versions  #9506, @yankay)
• Use kube_apiserver_port variable instead of hard-coding 6443 (fix(k8s-certs-renew): Use kube_apiserver_port instead of hard-coding #9620, @huangkevin404)
• [etcd] Default version to 3.5.5 for k8s 1.25.x ([etcd] add 3.5.5 hashes, make it default for k8s 1.25 #9419, @mzaian)
• Update CoreDNS version to v1.9.3 (Update CoreDNS version to v1.9.3  #9503, @yankay)
• Add the possibility to specify extra domains for the coredns kubernets plugin (using coredns_kubernetes_extra_domains) (feat(coredns): Forward extra domains to coredns kubernetes plugin #9635, @mvandergiesen)
• Streamline ansible_default_ipv4 gathering loop (streamline ansible_default_ipv4 gathering loop  #9281, @rptaylor)
• Update kubernetes dashboard to 2.7.0 (k8s 1.25 support) (Update kubernetes dashboard to 2.7.0 (k8s 1.25 support) #9425, @mzaian)
• Skip retry operation with containerd when etcd installed on host VM (Skip retry operation with containerd when etcd installed on host VM #9560, @JRaver)
• Update pause image version to v3.8 (Update pause image version to v3.8 #9668, @mzaian)
• Enable kubelet_authorization_mode_webhook back by default and remove extra role (enable back kubelet_authorization_mode_webhook by default #9662, @MrFreezeex)
• Terraform gcp can now have extra ingress firewall rules, using new variable extra_ingress_firewalls (Terraform gcp extra ingress firewall #9658, @sathieu)
• kubeadm/etcd: use config to download certificate (kubeadm/etcd: use config to download certificate #9609, @MrFreezeex)

Applications

• [argocd] update argocd to v2.5.5 ([argocd] update argocd to v2.5.5 #9604, @mzaian)
• Upcloud: Reclaim policy for PV is now delete (upcloud: Delete default reclaim policy #9574, @robinAwallace)
• [Exoscale] Add missing zone input variable (Add missing zone input variable - Exoscale #9495, @ayoubeddafali)
• [MetalLB] Avoid MetalLB speaker image download when MetalLB speaker is disabled (Avoid MetalLB speaker image download when MetalLB speaker is disabled #9248, @unai-ttxu)
• [Openstack] Replace deprecated "template" Terraform provider with supported "cloudinit" Terraform provider ( Terraform Openstack: replace deprecated template provider with suppo… #9536, @inflatador)
• [OpenStack] Updated openstack cloud controller to version v1.25.3 ([openstack cloud controller] bump to v1.25.3 #9500, @robinAwallace)
• [Openstack] Add bastion_allowed_ports to allow custom security group rules on bastion node ([Openstack] Add bastion_allowed_ports to allow custom security group rules on bastion node #9336, @bl0m1)
• [Openstack] Upgrade 1.22.0 to 1.23.4 (cloud-provider-openstack: upgrade 1.22.0 to 1.23.4 #9332, @QcFe) (See Notes 1)
• [Openstack] Added override variable, additional server groups and cloudinit config (Added override variable, additional server groups and cloudinit config #9452, @Xartos)
• [cinder-csi-nodeplugin] Remove the pods-cloud-data volume (delete upstream) (fix(cinder-csi-nodeplugin): Remove the pods-cloud-data volume #9362, @huangkevin404)
• [vsphere-csi] Add missing defaults for external_vsphere_* variables in the csi_driver/vsphere role (Add missing defaults for external_vsphere_* variables in the csi_driver/vsphere role #9664, @rlacko58)
• [hetzner] In config, rename ansible groups to use _ instead of - (rename ansible groups to use _ instead of #9569, @ym)
• [kube-vip] Minor changes on Kube VIP configuration parameters (and fix wrong properties) (Minor changes on Kube VIP configuration parameters #9414, @woutergd)
• [cert-manager] Upgrade to v1.10.1 ([cert-manager] Upgrade to v1.10.1 #9512, @rtsp) then v1.11.0 ([cert-manager] update cert-manager to v1.11.0 #9661, @mzaian)
• [helm] upgrade to 3.10.3 ([helm] upgrade to 3.10.3 #9605, @mzaian)
• [ingress-nginx] upgrade to 1.5.1 ([ingress-nginx] upgrade to 1.5.1 #9532, @mzaian)
• [vSphere] Removing unneeded terraform dependencie & mark vsphere_password as sensitive (Terraform vsphere cleanup #9672, @sathieu)

Container-Managers

• Optimize cgroups settings for node reserved (using new kube_reserved, see docs for more information) (optimize cgroups settings for node reserved #9209, @shelmingsong)
• [Docker] Update docker package to 20.10.20 (partial fix for CVE-2022-39253) (Update docker version to 20.10.20 #9410, @floryut)
• [containerd] Add support for 1.6.11 (support containerd 1.6.11 #9544, @yanggangtony)
• [containerd] Added variables for unpriviledged ports and icmp ([containerd] Add config for unpriviledged ports and icmp #9517, @Xartos)
• [containerd] Allow containerd-common to execute multiple times per play (Allow containerd-common to execute multiple times per play #9543, @chadswen)
• [containerd] Newly started containers will be limited to 16384 open files. To change this number, set containerd_base_runtime_spec_rlimit_nofile, or remove base_runtime_spec from runc runtime to revert to previous behaviour. ([containerd] Limit number of open files per container #9319, @fungusakafungus)
• [containerd] Support v1.6.13 and v1.6.14 (Update containerd version : containerd1.6.13 and 1.6.14. #9585, @yanggangtony)
• [containerd] Add config_path var in config.toml.j2 file (add containerd config_path #9566, @lengrongfu)
• [containerd] Add hashes for containerd versions 1.5.14 , 1.5.15 , 1.5.16 ([containerd] add hashes for 1.5.x #9678, @yanggangtony)
• [cri-o] Use cri-o from upstream instead of kubic/OBS (use cri-o from upstream instead of kubic/OBS #9374, @cristicalin)
• [nerdctl] upgrade to version 1.0.0 ([nerdctl] upgrade to version 1.0.0 #9424, @mzaian)

Network

• Bump cni-plugins version to v1.2.0 (bump cni-plugins version to v1.2.0 #9671, @cyclinder)
• Fix remove Cilium CNI failed because the CNI bin dependency (Remove CNI BIN dependency for cilium #9563, @yankay)
• [Calico] Add cni bin when installing (Add cni bin when installing calico #9367, @ErikJiang)
• [Calico] Add retry for start calico kube controller (add retry for start calico kube controller #9450, @cleverhu)
• [Calico] Adjust calico-kube-controller pod to non hostNetwork pod (adjust calico-kube-controller to non-hostnetwork pod #9465, @cyclinder)
• [Calico] Adjust calico-kube-controller pod to use hostnetwork if using etcd (Calico in etcd mode requires hostNetwork #9573, @JSpon)
• [Calico] Disable 'Check that IP range is enough for the nodes' (Disable 'Check that IP range is enough for the nodes' when calico is used #9491, @mzaian)
• [Calico] Update the tag image to support multiple architectures with the same tag (Update calico image tag #9529, @ErikJiang)
• [Calico] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (Remove PodSecurityPolicies in Calico for kubernetes 1.25 #9395, @yankay)
• |Calico] Allow user to set env: FELIX_MTUIFACEPATTERN in calico-node.yml (using calico_felix_mtu_iface_pattern) (allow user to set env: FELIX_MTUIFACEPATTERN in calico-node.yml #9330, @shelmingsong)
• [Calico] Replace node-role.kubernetes.io/master with control-plane (cleanup: replace node-role.kubernetes.io/master #9627, @my-git9)
• [Calico] upgrade default calico version to v3.24.5 (Upgrade Calico to v3.24.5 #9580, @yankay)
• [Calico] Add vxlan-v6.calico to the list of NetworkManager unmanaged interfaces (calico: add vxlan-v6.calico to the list of NM unmanaged interfaces #9631, @cyclinder)
• [Calico] Add retry to avoid 'unknown' state for calicoctl (Add retry to avoid 'unknown' state for calicoctl #9633, @tu1h)
• [Calico] Update Calico VXLAN offload docs because Calico changed the default value (Update Calico VXLAN offload docs because Calico changed the default value. #9639, @yankay)
• [Calico] Add possibility to enable calico floatingIPs feature (using calico_felix_floating_ips) (feat(calico): add possibility to enable calico floatingIPs feature #9680, @MatthieuFin)
• [Cilium] Add download configuration for cilium hubble images (using cilium_enable_hubble variable) (Added download configuration for cilium hubble images #9376, @ErikJiang)
• [Cilium] Add switch cilium_enable_bandwidth_manager (Add switch cilium_enable_bandwidth_manager #9441, @dcwbq)
• [Cilium] Cleanup cilium-init image from cilium template (Clean up the cilium-init image #9508, @ErikJiang)
• [Cilium] update cilium cli offline download url example (update cilium cli offline download url example #9458, @cleverhu)
• [Cilium] Install Cilium CLI alongside Cilium (When using cilium CNI, install Cilium CLI #9436, @dcwbq)
• [flannel] Initcontainer image now correctly support architecture suffix (Flannel: fix init container image arch #9461, @rollandf)
• [flannel] Upgrade version to v0.20.1 (Upgrade flannel version to v0.20.1 #9528, @ErikJiang)
• [flannel] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (Remove PodSecurityPolicy in flannel (required for the kubernetes 1.25) #9365, @yankay)
• [flannel] Add wireguard encryption backend as option (Feature/add flannel wireguard encryption backend as option #9583, @janaurka)
• [flannel] Support dual stack IPv4 & IPv6 networking (Support dual stack IPv4 & IPv6 networking for flannel #9564, @styshoo)
• [flannel] Allow setting the DirectRouting option on VXLAN (Allow setting the DirectRouting option on flannel VXLAN #9438, @willtrnr)
• [flannel] update to v0.20.2 & make it default ([flannel] update to v0.20.2 & make it default #9675, @mzaian)
• [kube-ovn] Update version to v1.10.7 ([kube-ovn]: update version v1.10.7 #9527, @liupeng0518)
• [kube-ovn] Remove kube-ovn log directories when reseting (Reset role: Remove kube-ovn log directories #9625, @JochenFriedrich)
• [kube-ovn] Remove ovn.kubernetes.io/ovs_dp_type from nodeSelector (Remove ovn.kubernetes.io/ovs_dp_type from nodeSelector #9594, @JochenFriedrich)
• [kube-ovn] Support OVN Interconnect (Support OVN Interconnect #9599, @JochenFriedrich)
• [multus] added support for mixed type of container engine ([multus] added support for mixed type of container engine #9224, @mr-yaky)

Bug or Regression

• Change include to import_playbook in recover_control_plane playbook, to support ansible 2.12+ (Remove include task in play, deprecated in favor of import_playbook #9576, @floryut)
• Corrected vsphere directory in docs (Corrected vsphere directory #9534, @wojciehm)
• Deleting worker nodes is now skipped if there is no kube_control_plane node. (Skip removing worker nodes if kube_control_plane is not set #9430, @kerryeon)
• Etcd arch can now support arm64 and amd64 (etcd arch can support arm64 and amd64 #9421, @yanggangtony)
• Fix cert-manager deployment on hardening environments (Fix cert-manager deployment on hardening environments #9404, @oomichi)
• Fix checksum of ciliumcli v0.12.5 for arm64 (Fix checksum of ciliumcli v0.12.5 for arm64 #9614, @oomichi)
• Fix inconsistent handling of admission plugin list (kube_apiserver_enable_admission_plugins must be specified as a list of individual plugin names instead of a single item comma-separated list) (Fix inconsistent handling of admission plugin list #9407, @willtrnr)
• Fix kube token dir permissions (fix kube token dir permissions #9590, @C-Romeo)
• Fix missing control plane taint in kubeadm (Fix missing control plane taint in kubeadm #9592, @yankay)
• Fix regex for comments nameserver in resolv.conf (Fix regex for comments nameserver in resolv.conf #9523, @yankay)
• Fix reset for RedHat based distro with major version >=8 (reset: RedHat based distro with major version >=8 #9537, @dougsland)
• Fix wrong cri_socket path for containerd (fix(defaults): wrong cri_socket path for containerd #9401, @maxime1907)
• Fix wrong rbac of the ClusterRole csi-snapshotter-role (fix(apps): cinder: wrong rbac for csi-snapshotter-role #9610, @maxime1907)
• Remove coredns_server from supersede_nameserver in dhclient.conf if nodelocaldns is enabled. (Don't use coredns_server in dhclient.conf if nodelocaldns is enabled #9392, @JiffsMaverick)
• Remove immutable flag from /var/lib/kubelet subdirs (Remove immutable flag from /var/lib/kubelet subdirs #9597, @emiran-orange)
• Skip the install of ping package in Fedora CoreOS & Flatcar (Fix ensure ping package error in fedora CoreOS & Flatcar #9370, @yankay)
• Fix OL9 setup - disable Centos Extras repo creation (Fix OL9 setup - disable Centos Extras repo creation #9483, @psvmcc)
• Use hostname override in post-remove role, just as pre-remove role does (Use hostname override in post-remove role, just as pre-remove role does #9360, @JSpon)
• [Calico] Install calico-kube-controller also when using kdd datastore (Fix: install calico-kube-controller on kdd #9358, @wayfrro)
• [Cilium] Fix the Hubble certificate being faulty because the cluster name has an hard coded value (Fix: The Hubble certificate is faulty because the cluster name is hard coded #9340, @dcwbq)
• [Cilium] Fix tls settings not being properly set (fix: hubble relay tls error #9457, @charlychiu)
• [Cilium] Remove trailing backslash and fix yaml indent (fix: remove trailing backslash and yaml indent #9339, @reneluria)
• [Openstack] Fix a race condition in terraform causing ports to not get an IP ([Openstack] Fix a race condition in terraform causing ports to not get an IP #9345, @bl0m1)
• [Openstack] Fix missing permissions for Openstack cloud-controller-manager ([Openstack] Fix missing permissions for Openstack cloud-controller-manager #9335, @bl0m1)
• [gVisor] Allow installation on arm architecture systems (Update gvisor checksum #9493, @ErikJiang)
• [kube-ovn] Cluster support for ovn-central (Cluster support for ovn-central. #9596, @JochenFriedrich)
• [upcloud] Fixed issue where DNS would be blocked while using allowlist ([upcloud] Fixed issue where DNS would be blocked while using allowlist #9510, @Xartos)

Other (Cleanup or Flake)

• Use the correct api version and resource type in secrets_encryption.yaml.j2 (Use the correct api version and resourcer type. The current values wo… #9575, @LukasNajman)
• Minor cleanup of docs by rephrasing some unclear documentation (Minor cleanup of README.md and two other docs #9621, @anthonyeleven)
• Add mirror doc to support mirror usage. ([Docs] support the download mirror for China #9396, @yankay)
• [CI] Add check_typo job (and fix a bunch of typos) (Add check_typo job #9361, @oomichi)
• [CI] Stop using python 'test' internal package (Stop using python 'test' internal package #9454, @olivierlemasle)
• [CI] Update securityContext of netchecker (Update securityContext of netchecker #9398, @oomichi)
• [CI] Use agnhost instead of busybox for network test (Use agnhost instead of busybox for network test #9390, @oomichi)
• [CI] Add ubuntu20 hardening job (Add packet_ubuntu20-calico-aio-hardening #9359, @oomichi)
• [CI] Fix YAML format in hardening.md file (Fix YAML format in hardening.md #9387, @oomichi)
• [CI] Make vagrant-ubuntu20-flannel voting (by removing allow failure) (Make vagrant-ubuntu20-flannel voting #9469, @oomichi)
• [CI] Update sonobuoy version to a more recent one (Update sonobuoy version #9485, @oomichi)
• [CI] Increase the fedora memory at CI to fix the CI broken (Increase the fedora memory at CI to fix the CI broken #9640, @yankay)
• [CI] Add CI for rockylinux9 and cilium (Add CI for rockylinux9 and cilium #9562, @yankay)

---

Component versions

• Core
  • kubernetes v1.25.6
  • etcd v3.5.6
  • docker v20.10 (cri_dockerd: v0.3.0)
  • containerd v1.6.15
  • cri-o v1.24
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.24.5
  • cilium v1.12.1
  • flannel v0.20.2
  • kube-ovn v1.10.7
  • kube-router v1.5.1
  • multus v3.8
  • weave v2.8.1
  • kube-vip v0.5.5
• Application
  • cert-manager v1.11.0
  • coredns v1.9.3
  • ingress-nginx v1.5.1
  • krew v0.4.3
  • argocd v2.5.7
  • helm v3.10.3
  • metallb v0.12.1
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.22.0
  • gcp-pd-csi-plugin v1.4.0
  • local-path-provisioner v0.0.22
  • local-volume-provisioner v2.5.0

Known issues

N/A

Notes

1. As stated in cloud-provider-openstack:1.23.0: Load balancers don't relate to a dedicated Service anymore, any scripts relying on that relationship previously need to change to use the load balancer tags instead

[yankay]

The Calico VXLAN Offload Issue has been fixed at the calico 3.25 & 3.24.5 by projectcalico/calico#6842 , So I think the kubespray may not need the "note" to disable it. :-)

[cyclinder]

Can #9631 be part of this RELEASE?

[floryut]

The Calico VXLAN Offload Issue has been fixed at the calico 3.25 & 3.24.5 by projectcalico/calico#6842 , So I think the kubespray may not need the "note" to disable it. :-)

Indeed I saw you PR, update done

[floryut]

Can #9631 be part of this RELEASE?

As soon as it's rebased and go through the CI 👍

[mvandergiesen]

Hi, Can #9635 be part of the release? #9635 <#9635> Thanks, Marijn

…

On 5 Jan 2023, at 04:40, Cyclinder ***@***.***> wrote: Can #9631 <#9631> be part of this RELEASE? — Reply to this email directly, view it on GitHub <#9638 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANJIBRW7XDKUPKQB6RLQGJLWQY7ETANCNFSM6AAAAAATRHP6E4>. You are receiving this because you were mentioned.

[JRaver]

Hi, please add this PR to the release #9560

[floryut]

Hi, Can #9635 be part of the release? #9635 <#9635> Thanks, Marijn
…
On 5 Jan 2023, at 04:40, Cyclinder @.***> wrote: Can #9631 <#9631> be part of this RELEASE? — Reply to this email directly, view it on GitHub <#9638 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANJIBRW7XDKUPKQB6RLQGJLWQY7ETANCNFSM6AAAAAATRHP6E4. You are receiving this because you were mentioned.

Done

Can #9631 be part of this RELEASE?

And done

[oomichi]

Thanks for preparing the release, lgtm!

[mzaian]

Thank you, can we sdd these 2 PRs.

#9532

#9425

[floryut]

🥳 https://github.com/kubernetes-sigs/kubespray/releases/tag/v2.21.0