Chart order change results in multiple manifest files and chart failing to update

[emosbaugh]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 6.5.0-1023-gcp #25~22.04.1-Ubuntu SMP Thu Jun 13 19:41:39 UTC 2024 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version

v1.30.2+k0s.0

Sysinfo

`k0s sysinfo`

Total memory: 29.4 GiB (pass)
Disk space available for /var/lib/k0s: 190.2 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.5.0-1023-gcp (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: built-in (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: built-in (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: built-in (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: built-in (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

When the chart order is changed in the clusterconfig, multiple conflicting manifest files are created for the same chart. When the order is increased upon update, the new changes do not get applied to the chart custom resource.

Steps to reproduce

/etc/k0s/k0s.yaml

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  extensions:
    helm:
      concurrencyLevel: 5
      repositories:
      - name: stable
        url: https://charts.helm.sh/stable
      charts:
      - name: prometheus-stack
        chartname: prometheus-community/prometheus
        version: "14.6.1"
        timeout: 20m
        order: 1
        values: |
          alertmanager:
            persistentVolume:
              enabled: false
          server:
            persistentVolume:
              enabled: false
        namespace: default

/etc/k0s/k0s-update.yaml

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  extensions:
    helm:
      concurrencyLevel: 5
      repositories:
      - name: stable
        url: https://charts.helm.sh/stable
      charts:
      - name: prometheus-stack
        chartname: prometheus-community/prometheus
        version: "14.6.1"
        timeout: 20m
        order: 101
        values: |
          alertmanager:
            persistentVolume:
              enabled: false
          server:
            env:
            - name: "TEST"
              value: "test"
            persistentVolume:
              enabled: false
        namespace: default

Install k0s with chart

$ sudo k0s install controller -c /etc/k0s/k0s.yaml --single --enable-dynamic-config
$ sudo k0s start

Update k0s dynamic config

$ sudo k0s kubectl apply -n kube-system -f /etc/k0s/k0s-update.yaml

There are now two chart manifests

$ ls /var/lib/k0s/manifests/helm/
101_helm_extension_prometheus-stack.yaml  1_helm_extension_prometheus-stack.yaml  helm-crd-helm.k0sproject.io_charts.yaml

The chart does not reflect the latest updates

$ sudo k0s kubectl -n kube-system get charts k0s-addon-chart-prometheus-stack -oyaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  annotations:
    k0s.k0sproject.io/last-applied-configuration: |
      {"apiVersion":"helm.k0sproject.io/v1beta1","kind":"Chart","metadata":{"finalizers":["helm.k0sproject.io/uninstall-helm-release"],"name":"k0s-addon-chart-prometheus-stack","namespace":"kube-system"},"spec":{"chartName":"prometheus-community/prometheus","namespace":"default","releaseName":"prometheus-stack","timeout":"20m0s","values":"\nalertmanager:\n  persistentVolume:\n    enabled: false\nserver:\n  persistentVolume:\n    enabled: false\n","version":"14.6.1"}}
    k0s.k0sproject.io/stack-checksum: 9f5285ad2f19ce093bc5c8866612e841
  creationTimestamp: "2024-06-28T19:04:56Z"
  finalizers:
  - helm.k0sproject.io/uninstall-helm-release
  generation: 5
  labels:
    k0s.k0sproject.io/stack: helm
  name: k0s-addon-chart-prometheus-stack
  namespace: kube-system
  resourceVersion: "1114"
  uid: 62e66d79-ff6a-4fc4-b58e-7aadd3ef7a86
spec:
  chartName: prometheus-community/prometheus
  namespace: default
  releaseName: prometheus-stack
  timeout: 20m0s
  values: |2

    alertmanager:
      persistentVolume:
        enabled: false
    server:
      persistentVolume:
        enabled: false
  version: 14.6.1
status:
  appVersion: 2.26.0
  namespace: default
  releaseName: prometheus-stack
  revision: 4
  updated: 2024-06-28 19:10:22.025186201 +0000 UTC m=+338.646444028
  valuesHash: 615a9f1dea460d79fcde5e0d57769f7459c4e66ec6fc04970853d1bc43ee105a
  version: 14.6.1

Expected behavior

When the chart order is changed there will only be a single manifest for the chart.

Actual behavior

When the chart order is changed in the clusterconfig, multiple conflicting manifest files are created for the same chart. When the order is increased upon update, the new changes do not get applied to the chart custom resource.

Screenshots and logs

No response

Additional context

No response

[jnummelin]

When the chart order is changed in the clusterconfig, multiple conflicting manifest files are created for the same chart.

Right, k0s just generates a file per chart, using the order as prefix. And hence we end up with multiple files per chart. 🤦

What I think we'd need to do is to:

• Generate files per chart, keep track of newly generated files
• Remove all other files (so we get rid of old ones if the order changes)

[twz123]

Alright, fixed and released for 1.28-1.30. Unfortunately #4764 came too late for 1.27.16, but it will be part of 1.27.17.