[Feature] [Upgrade nodegroup] Update Bottlerocket nodes to latest AMI release version

[cloudkarthik99]

What were you trying to accomplish?

Trying to update private EKS cluster bottlerocket nodegroup with the latest bottlerocket AMI version in the same Kubernetes Version

What happened?

eksctl upgrade nodegroup is not fetching the latest Bottlerocket AMI while it successfully does for AmazonLinux2 AMI (even when not provisining releaseVersion parameter in cluster config for AL2 NG) saying:

unchanged fields for nodegroup apigw-xlarge: the following fields remain unchanged; they are not supported by `eksctl update nodegroup`: AMIFamily, InstanceType, ScalingConfig, Labels, PrivateNetworking, Tags, IAM, ReleaseVersion

How to reproduce it?

1. create a EKS managed nodegroup with the old bottlerocket AMI version using the below config
2. Upgrade using eksctl upgrade npdegroup passing new AMI version in the releaseVersion parameter
3. eksctl upgrade nodegroup is updating to latest AMI version only when nodegroup name changes or K8s version updates

eksctl cluster config:

cluster_config = <<CLUSTERCONFIG
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ${local.ressources_name}
  region: ${local.region}
  tags: ${jsonencode(local.common_tags)}
  version: "${var.version}" #1.23

vpc:
  subnets:
    private:
  %{for item in data.aws_subnet.vpc_subnets~}
    ${item.availability_zone}: { id: ${item.id} }
  %{endfor~}

managedNodeGroups:
  - name: ${var.nodegroup_name}
    labels: { 
      role: workers,
      project: ${lower(var.tag_project)},
      application: ${replace(lower(var.tag_application), " ", "_")},
      environment: ${local.environment}, 
      classification: ${lower(local.classification)}, # TEST or PROD
      region: ${lower(local.region)},
      k8s_dns_name: ${local.k8s_dns_name},}
    }
    instanceType: ${var.eks_instance_type}     #t3.xlarge
    minSize: ${var.eks_min_size}
    maxSize: ${var.eks_max_size}
    desiredCapacity: ${var.eks_desired_capacity}
    updateConfig:
      # https://eksctl.io/usage/eks-managed-nodes/#handling-parallel-upgrades-for-nodes
      maxUnavailablePercentage: 50
    amiFamily: ${var.ami_family}  # Bottlerocket or AmazonLinux2
    releaseVersion: 1.13.4-f549851d #old AMI Version 
    privateNetworking: true
    tags: ${jsonencode(local.common_tags)}
    iam:
      withAddonPolicies:
        autoScaler: true
        externalDNS: true
        certManager: true
        cloudWatch: true
        imageBuilder: true

cloudWatch:
  clusterLogging:
    enableTypes: ["audit", "authenticator"]

addons:
- name: aws-ebs-csi-driver
  
  CLUSTERCONFIG
}

terraform data module:

data "aws_ssm_parameter" "ami_version" {
  name = "/aws/service/bottlerocket/aws-k8s-${var.eks_version}/x86_64/latest/image_version"
}

eksctl commands:

eksctl upgrade nodegroup --name="${nodegroup_name}" --cluster="${cluster_name}" --region="${region}" --kubernetes-version="${cluster_version}"

eksctl update nodegroup -f "${config_file}"

Anything else we need to know?
Everytime we run the CI/CD pipeline, the eksctl commands get executed to create the cluster, nodegroup and upgrade nodegroup and update nodegroup commands and the cloudformation stack is checking for the latest AL2 AMI version if present and updates the EKS node group to the latest AMI releaseversion (when and only we run the pipeline not auto updates) which is not happening for Bottlerocket AMI

We don't want to stick to any particular AMI version but we want to have latest AMI release versions for the same k8s version when available and we dont want to use any operators like bottlerocket update operator as we dont want any auto updates happening.

Bottlerocket nodegroup is updating only when we change the nodegroup name or kubernetes version.

Versions
Bottlerocket OS 1.13.4 (aws-k8s-1.23)
eksctl version=0.141.0

[github-actions]

Hello remo1435 👋 Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[cloudkarthik99]

@cPu1 I saw that you worked on related bottlerocket issues previously. Just tagging you so that may be you could help or suggest any workaround

[cloudkarthik99]

#4666
I just wanted to mention this issue as it seems to be bit related with my issue. If not please remove it.

When I checked my CNF template of AmazonLinux2 EKS nodegroup and Bottlerocket EKS nodegroup, the default difference is ReleaseVersion parameter is absent by default in bottlerocket CNF template (I could pass the releaseVersion parameter in cluster config as I explained above but the AMI version has to be hardocoded but we want the AMI version to be latest and it should check automatically for the version (just like for AL2 )

AmazonLinux2 CNF template:

 "ManagedNodeGroup": {
      "Type": "AWS::EKS::Nodegroup",
      "Properties": {
        "AmiType": "AL2_x86_64",
        "ClusterName": "al2-cluster",
        "ForceUpdateEnabled": false,
        "InstanceTypes": [
          "t3.xlarge"
        ],
        
        "NodegroupName": "al2-nodegroup",
        "ReleaseVersion": "1.23.17-20230607",
        "ScalingConfig": {
          "DesiredSize": 3,
          "MaxSize": 6,
          "MinSize": 3
        },

Bottlerocket CNF template:

"ManagedNodeGroup": {
      "Type": "AWS::EKS::Nodegroup",
      "Properties": {
        "AmiType": "BOTTLEROCKET_x86_64",
        "ClusterName": "bottlerocket-cluster",
        "ForceUpdateEnabled": false,
        "InstanceTypes": [
          "t3.xlarge"
        ],
        
        "LaunchTemplate": {
          "Id": {
            "Ref": "LaunchTemplate"
          }
        },
        "NodegroupName": "bottlerocket-ng",
        "ScalingConfig": {
          "DesiredSize": 3,
          "MaxSize": 4,
          "MinSize": 3
        },

[TiberiuGC]

Hi @remo1435 , indeed seems like updating to latest AMI release version is currently only supported for AL2 images (see code snipped below).

https://github.com/weaveworks/eksctl/blob/0feb86cd2bd6b8872e658735b5300bbe048c439d/pkg/ami/ssm_resolver.go#L82-L93

I don't know yet whether there are any deterrents in having the same behaviour for other AMI families. I've marked this as a feature request. Will discuss it internally and come back to you.

[Himangini]

Spike: To come up with a proposal for this feature and understand any limitations
Timebox: 1-2 days
Document findings here

[TiberiuGC]

Spike: To come up with a proposal for this feature and understand any limitations Timebox: 1-2 days Document findings here

Moving forward with implementing this, as the complexity is fairly low and there don't seem to be any blockers.