EKS add-ons

[onabison]

Hi,

@bryantbiggs
@mbeacom

I'm not sure if to categorize this question as a feature request or not.

When using the eksupgrade tool, I noticed that it only looks at the 3 common add-ons (kube-proxy, vpc_cni, coredns). What if I have enabled other add-ons Amazon EBS CSI Driver in my case, would it check for update and upgrade it (or any other for that matter)?

[mbeacom]

Hello,

Which version of eksupgrade are you using?

Not sure I understand your question. Originally, the tool would only upgrade vpc-cni, kube-proxy, and coredns. In the current release (0.7.0), the tool will automatically upgrade any officially recognized add-on. We are pulling the installed add-on details from the AWS API directly. If it's installed and should be upgraded, it will be upgraded.

The pre-flight and post-flight checks will be updated to reflect the upgrade summary as defined in an upcoming release.

Thank you!

[bryantbiggs]

@mbeacom I think we need to change some of this since its only looking at the addons mentioned

eks-cluster-upgrade/eksupgrade/src/preflight_module.py

Lines 279 to 320 in 3a6fd98

	coredns_target_version: str = get_default_version("coredns", **default_version_kwargs).split("-")[0].lstrip("v")
	proxy_target_version: str = get_default_version("kube-proxy", **default_version_kwargs).split("-")[0].lstrip("v")
	cni_target_version: str = get_default_version("vpc-cni", **default_version_kwargs).split("-")[0].lstrip("v")
	# Kube Proxy config
	kube_proxy_config = get_package_dict("kube-proxy.json")
	kube_proxy_container = kube_proxy_config["spec"]["template"]["spec"]["containers"][0]
	yaml_data["kube-proxy"] = {
	"image": kube_proxy_container["image"],
	"volumeMount": kube_proxy_container["volumeMounts"],
	"env": None,
	}
	# Core DNS config
	core_dns_config = get_package_dict("coredns.json")
	coredns_container = core_dns_config["spec"]["template"]["spec"]["containers"][0]
	yaml_data["coredns"] = {
	"image": coredns_container["image"],
	"volumeMount": coredns_container["volumeMounts"],
	"env": None,
	}
	# VPC CNI config
	vpc_cni_config = get_package_dict("vpc-cni.json")
	vpc_cni_container = vpc_cni_config["spec"]["template"]["spec"]["containers"][0]
	yaml_data["vpc-cni"] = {
	"image": vpc_cni_container["image"],
	"volumeMount": vpc_cni_container["volumeMounts"],
	"env": vpc_cni_container["env"],
	}
	# Kube Proxy config map
	kube_proxy_config_map = get_package_dict("kube-proxy-configmap.json")
	config_map["certificate-authority"] = yaml.safe_load(kube_proxy_config_map["data"]["kubeconfig"])["clusters"][0][
	"cluster"
	]["certificate-authority"]
	try:
	logger.info("Check addon version compatibility .....")
	addons: List[Dict[str, Any]] = []
	report["addon_params"] = {}
	customer_report["addons"] = {"vpc-cni": {}, "kube-proxy": {}, "coredns": {}}

In the new models, we are retrieving all of the EKS addons associated with the cluster

eks-cluster-upgrade/eksupgrade/models/eks.py

Line 712 in 3a6fd98

def current_addons(self) -> List[str]:

- which would include any marketplace installed addons as well.

[mbeacom]

Tracking this as a part of: #86

[onabison]

Firstly, thank you all for chiming in and actively working on this! I am very grateful and thankful!

Secondly, I would like to clarify, although, it seems my point was made by @bryantbiggs.

Here is the current list of addons I have on my cluster:

{
    "addons": [
        "aws-ebs-csi-driver",
        "coredns",
        "kube-proxy",
        "vpc-cni"
    ]
}

And here is the output of my upgrade using eksupgrade version 0.7.0 and you can clearly see that the aws-ebs-csi-driver addon is not included/detected by the upgrade process. From your conversation it seems clear that the tool is only looking for the 3 "base" addons.

INFO:eksupgrade.src.preflight_module:Check addon version compatibility .....
INFO:eksupgrade.src.preflight_module:VPC CNI found version: ['1', '12', '5'] - version_str: 1.12.5
INFO:eksupgrade.src.preflight_module:Likely desired vpc-cni version: 1.12.2
INFO:eksupgrade.src.preflight_module:aws-node pod is running
INFO:eksupgrade.src.preflight_module:aws-node pod is running
INFO:eksupgrade.src.preflight_module:vpc-cni version up to date
INFO:eksupgrade.src.preflight_module:************* Parameter check for vpc-cni *************
INFO:eksupgrade.src.preflight_module:Image verified
INFO:eksupgrade.src.preflight_module:Mount path verification successful
INFO:eksupgrade.src.preflight_module:These user defined envs are present ANNOTATE_POD_IP CLUSTER_ENDPOINT CLUSTER_NAME DISABLE_NETWORK_RESOURCE_PROVISIONING ENABLE_IPv4 ENABLE_IPv6 ENABLE_PREFIX_DELEGATION VPC_ID WARM_PREFIX_TARGET
INFO:eksupgrade.src.preflight_module:kube-proxy pod is running
INFO:eksupgrade.src.preflight_module:kube-proxy pod is running
INFO:eksupgrade.src.preflight_module:Checking if kube-proxy target version: 1.25.6 is equal to the current version: 1.25.6
INFO:eksupgrade.src.preflight_module:kube-proxy version up to date
INFO:eksupgrade.src.preflight_module:************* Parameter check for kube-proxy *************
INFO:eksupgrade.src.preflight_module:Image verified
INFO:eksupgrade.src.preflight_module:Mount path verification successful
INFO:eksupgrade.src.preflight_module:Certificate Authority Verified in kube config
INFO:eksupgrade.src.preflight_module:Server end point verified
INFO:eksupgrade.src.preflight_module:coredns pod is running
INFO:eksupgrade.src.preflight_module:coredns pod is running
INFO:eksupgrade.src.preflight_module:Checking if coredns target version: 1.9.3 is equal to the current version: 1.9.3
INFO:eksupgrade.src.preflight_module:core-dns version up to date
INFO:eksupgrade.src.preflight_module:************* Parameter check for coredns *************
INFO:eksupgrade.src.preflight_module:Image verified
INFO:eksupgrade.src.preflight_module:Mount path verification success

on a slightly different note, and not important at all, what is the reason for the list output here? only the CNI does that and I'm not sure for what purpose. Especially since most versions have minor versions like eksbuild.1 and so on.

INFO:eksupgrade.src.preflight_module:VPC CNI found version: ['1', '12', '5'] - version_str: 1.12.5

[quixoticmonk]

Thank you for bringing this up. I did remember seeing the csi driver updated in a test I had and so ran it again. Was the screenshot from a run with just pre-flight checks ?

INFO:eksupgrade.models.eks:Fetching Cluster Addons...
INFO:eksupgrade.models.eks:Getting the list of current cluster addons for cluster: eksup-cluster-csi...
INFO:eksupgrade.models.eks:Updating addon: aws-ebs-csi-driver from original version: v1.15.1-eksbuild.1 to version: v1.16.1-eksbuild.1
INFO:eksupgrade.models.eks:Updating addon: aws-ebs-csi-driver - ID: a29e2763-fcb5-35f6-b1f5-4be28f1bcc03 - Status: InProgress
INFO:eksupgrade.models.eks:Waiting for the add-on: aws-ebs-csi-driver to become active...
INFO:eksupgrade.models.eks:Add-on: aws-ebs-csi-driver upgraded!

[onabison]

No, mine was an actual run (sorry for the late response).

[onabison]

@mbeacom

THANK YOU for the color coding! Looks Amazing!

I just created a new cluster with version 1.22 and intentionally created add-ons that were not latest versions. I then let it run with preflight flag. I still don't see it detecting my additional add-on.

eksupgrade poc-upgrade-eks-cluster 1.22 us-east-1 --force --latest-addons --preflight
Running validation checks against cluster: poc-upgrade-eks-cluster...
Verifying User IAM Role...
IAM role for user verified!
Fetching cluster details...
Cluster control plane version 1.22
Cluster already upgraded to version 1.22
Checking if customer management key exists...
No CMK Key associated with the cluster
Fetching security group details...
Security Group sg-xxx is present in VPC with ID vpc-xxx
Security Group sg-xxx is present in VPC with ID vpc-xxx
Pod Security Policies...
Pod Security Policy with eks.privileged role exists.
Fetching node group details...
Checking Available IP for subnets associated with cluster...
Subnet ID subnet-xxx has 310 available IP addresses
Subnet ID subnet-xxx has 115 available IP addresses
Subnet ID subnet-xxx has 279 available IP addresses
Subnet ID subnet-xxx has 270 available IP addresses
Subnet ID subnet-xxx has 112 available IP addresses
Subnet ID subnet-xxx has 111 available IP addresses
Available IPs for Subnet verified
Checking important cluster role are present or not...
All cluster role needed successfully verified
Check addon version compatibility...
Likely desired vpc-cni version: 1.10.1
aws-node pod is running
aws-node pod is running
vpc-cni version up to date
************* Parameter check for vpc-cni *************
Image verified
Mount path verification successful
These user defined envs are present ANNOTATE_POD_IP CLUSTER_NAME DISABLE_NETWORK_RESOURCE_PROVISIONING ENABLE_IPv4 ENABLE_IPv6 ENABLE_PREFIX_DELEGATION VPC_ID WARM_PREFIX_TARGET
kube-proxy pod is running
kube-proxy pod is running
Checking if kube-proxy target version: 1.22.6 is equal to the current version: 1.22.11
kube-proxy version not compatible
coredns pod is running
coredns pod is running
Checking if coredns target version: 1.8.7 is equal to the current version: 1.8.7
core-dns version up to date
************* Parameter check for coredns *************
Image verified
Mount path verification successful
Corefile verified successfully
Fetching Pod Disruption Budget Details...
Pod Disruption Budget exists in cluster therefore force upgrade is required to upgrade the cluster
Pod disruption budget exists with max unavailable as 1 and min available as None
Fetching Horizontal Autoscaler Details...
No Horizontal Auto Scaler exists in cluster
Fetching Cluster Auto Scaler Details...
Cluster Autoscaler doesn't exist
Pre-flight check for the cluster poc-upgrade-eks-cluster succeeded!

Also, I have a few questions about the output:

1. Why does it always think that low version is desired
   Check addon version compatibility...
Likely desired vpc-cni version: 1.10.1

2. I don't understand this output Checking if kube-proxy target version: 1.22.6 is equal to the current version: 1.22.11

[mbeacom]

@onabison The pre-flight checks are still using the old core implementation. This is being replaced as a part of an upcoming release.

[bryantbiggs]

I still don't see it detecting my additional add-on.

Which addons are you adding that it did not detect?

[onabison]

Sorry, I figured it didn't really matter, but it's the same one I reported before aws-ebs-csi-driver. Thanks!

[quixoticmonk]

@onabison Do you have an eksctl or a Terraform config you are using to create the test cluster in question that can be shared ?

[onabison]

@quixoticmonk I use terraform to build it. How would you like me to share it?

[quixoticmonk]

You can post it inline or use gist . Please ensure that any sensitive information is removed prior to sharing this.

[onabison]

@quixoticmonk It's splint into a few code blocks. I'll try to put it here and hope that it makes sense.

in the main.tf:

resource "aws_eks_addon" "addons" {
  for_each          = local.addons
  cluster_name      = aws_eks_cluster.cluster.id
  addon_name        = each.value["name"]
  addon_version     = each.value["version"]
  resolve_conflicts = "OVERWRITE"

  depends_on = [
    aws_eks_node_group.eks_ec2_node_group
  ]
}

In data.tf under the locals it does an iteration like this:
addons = { for addon in var.addons : addon["name"] => addon

Then in variables.tfvars.json I list them like this:

  "addons": [
    {
      "name": "kube-proxy",
      "version": "v1.22.11-eksbuild.2"
    },
    {
      "name": "vpc-cni",
      "version": "v1.12.2-eksbuild.1"
    },
    {
      "name": "coredns",
      "version": "v1.8.7-eksbuild.4"
    },
    {
      "name": "aws-ebs-csi-driver",
      "version": "v1.8.0-eksbuild.0"
    }
   
Let me know if that makes sense. If you'd like to get on a call to get a better understanding I'd be happy to arrange. 

[bryantbiggs]

@onabison you may want to checkout how we manage EKS addons here. We use the aws_eks_addon_version data source so that the addon version is matched to the default version for a given Kubernetes version. This also aligns with what eksupgrade is doing during an upgrade.

In theory, if you used the terraform-aws-eks module, after performing an upgrade with eksupgrade, you would only need to update the cluster_version to re-align the code with the current state (using EKS managed nodegroups or Fargate profiles, additional changes would need to be made with self-managed nodegroups or custom AMIs)

[onabison]

@bryantbiggs It does look like I could use a facelift to my EKS deployment design, but I am not sure I'm ready for that just yet.

Until I get around to that, would the current eksupgrade tool never discovery any other addon I have that is not the "basic" one?

[onabison]

on a similar, yet, different note...

Can I run this command if I wanted to just update my addons to the latest version for the current EKS version I'm running without upgrading anything else?

eksupgrade _my_cluster_name_ 1.22 us-east-1 --force --latest-addons

[mbeacom]

Yes

[onabison]

@bryantbiggs @mbeacom
I was thinking about our earlier conversation as to how to keep the state file up-to-date after using the eksupgrade tool.

What is after the tool have upgraded one will run terraform plan, anything that has changed will be listed in the plan, using that, one can do terraform import on every component that was listed in the plan and that should sync the state file with the current state of the EKS.

Maybe that can be automated in a script or maybe even in GitHub Actions. Which brings me to my next question. Are you planning (or already have) a GitHub Action for the EKS upgrade tool?

Also, I haven't used the tool in a while, but has the issue where some of the add-ons are not "picked up" by the tool and don't get upgraded by the tool, been resolved?

In my case, if you recall, the EBS CSI driver add-on was not "discoverable" as part of the upgrade process.

Thanks!
Nave

[mbeacom]

The add-ons should have been automatically upgraded, but not displayed in the pre-flight because it was wrong.

The pre-flight has since been removed. Upgrade of EBS CSI should work as expected and be displayed in the active upgrade execution.

[onabison]

I will test that today. Thanks!

[onabison]

@mbeacom @bryantbiggs
I'm trying to use GitHub Actions to upgrade an EKS cluster. Do you see any issues with that?

So, far it is only partially successful. The first time I ran it, it worked and upgraded the Control Plane and some of the add-ons. I ran it again and it completed the add-ons but not the node groups. Now, when I run it I get this error:

Getting the list of current cluster addons for cluster: lab-upgrade-eks-cluster...
Exception encountered! Error: HTTPSConnectionPool(host='c2674c52deca33ec26b20d661bac50df.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /apis/apps/v1/deployments (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f2125152430>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Has anyone tries using GitHub actions before? Maybe there's even an action for it?

Here's my yaml while removing some info:

name: EKS Cluster Upgrade

on:
  workflow_dispatch:
    inputs:
      cluster_name:
        description: 'Cluster Name'
        required: true
      cluster_version:
        description: 'Cluster Version'
        required: true
      region:
        description: 'Region'
        required: true

env:
  DEV_IAM_ROLE: "arn:aws:iam::{account-ID}:role/{some-role-name}"
  DEV_AWS_ACCOUNT: "{account-ID}"

jobs:
  upgrade_eks_cluster:
    name: Upgrade Cluster
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
        with:
          repository: {repo_name}

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: ${{env.DEV_IAM_ROLE}}
          aws-region: ${{ github.event.inputs.region }}

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.9

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install eksupgrade

      - name: Upgrade Cluster
        run: |
          echo "Upgrading cluster: ${{ github.event.inputs.cluster_name }}"
          echo "Upgrading to version: ${{ github.event.inputs.cluster_version }}" 
          # Use the cluster_name input variable in your deployment command
          # For example:
          eksupgrade ${{ github.event.inputs.cluster_name }} ${{ github.event.inputs.cluster_version }} ${{ github.event.inputs.region }} --no-interactive --force --latest-addons 

[onabison]

I have tried multiple times to re-run the eksupgrade command via GitHub Actions and I'm getting the same error. I then ran the command locally on the same cluster and it works no problem. So, my observation is that the command works the first time and only updates the Control Plane and some of the add-ons, the second time it completed the add-on update, but never the node groups, and from that point on it will just time out.

Any thoughts?

[quixoticmonk]

How long is this taking ? The reason for asking is to check your session timeout for the AWS credentials.

Is this on all versions or some specific version?

[onabison]

It seems no matter what the version is. it gets stuck at the nodegroup upgrade

Run echo "Upgrading cluster: lab-upgrade-eks-cluster"
Upgrading cluster: lab-upgrade-eks-cluster
Upgrading to version: 1.27
Upgrading cluster: lab-upgrade-eks-cluster from version: 1.26 to 1.27...
The current version of the cluster was detected as: 1.26
Upgrading cluster: lab-upgrade-eks-cluster from version: 1.26 to version: 1.27
Waiting for cluster: lab-upgrade-eks-cluster to become active...
Cluster: lab-upgrade-eks-cluster now active, control plane upgrade should be completed!
Found the following Managed Nodegroups
	* on-demand
	* spot
Getting cluster managed nodegroup details...
Managed Node Group: on-demand - Version: 1.26 - Release Version: 1.26.4-20230526 - Cluster: lab-upgrade-eks-cluster
Getting cluster managed nodegroup details...
Managed Node Group: spot - Version: 1.26 - Release Version: 1.26.4-20230526 - Cluster: lab-upgrade-eks-cluster
Getting cluster autoscaling group details...
Autoscaling Group: eks-on-demand-48c35d44-fab4-c276-44fb-806c3bd95456 - Cluster: lab-upgrade-eks-cluster
Healthy Instances:
	 * i-08647da9cf367c6fc
	 * i-0a7a176196ed25284
Getting cluster autoscaling group details...
Autoscaling Group: eks-spot-2ac35d44-faa1-ae7d-44c4-[18](https://github.com/discoveryinc-dci/gcs-eks-infra/actions/runs/5193276465/jobs/9363602738#step:6:19)baf51ee5e1 - Cluster: lab-upgrade-eks-cluster
The add-ons update has been initiated...
Fetching Cluster Addons...
Getting the list of current cluster addons for cluster: lab-upgrade-eks-cluster...
Updating addon: coredns from original version: v1.9.3-eksbuild.3 to version: v1.10.1-eksbuild.1
Updating addon: coredns - ID: 71f9888e-ee54-3cb3-ab05-5ae9be7bf84e - Status: InProgress
Waiting for the add-on: coredns to become active...
Add-on: coredns upgraded!
Updating addon: kube-proxy from original version: v1.26.4-eksbuild.1 to version: v1.27.1-eksbuild.1
Updating addon: kube-proxy - ID: 81006214-cf6a-3ff4-ba1d-5ddbeb001cde - Status: InProgress
Waiting for the add-on: kube-proxy to become active...
Add-on: kube-proxy upgraded!
Exception encountered! Error: HTTPSConnectionPool(host='c2674c52deca33ec26b[20](https://github.com/discoveryinc-dci/gcs-eks-infra/actions/runs/5193276465/jobs/9363602738#step:6:21)d661bac50df.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /apis/apps/v1/deployments (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1[31](https://github.com/discoveryinc-dci/gcs-eks-infra/actions/runs/5193276465/jobs/9363602738#step:6:32)c[35](https://github.com/discoveryinc-dci/gcs-eks-infra/actions/runs/5193276465/jobs/9363602738#step:6:36)87c0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

You can see that it completed the control place as well as the add-ons but then times out again. This step shows it took 44 min.

Thanks!

[onabison]

@quixoticmonk Or anyone can please chime in on this?

[quixoticmonk]

The tests we run currently on GitHub actions are mostly on clusters that are created with the configuration we have with eksctl in the same process. I cannot unfortunately speak to if we can attest GA based workflow upgrades to work as expected without timeouts considering the environment it runs on and the little control we may have on the actual runtime environment.

[onabison]

@mbeacom @bryantbiggs

Following up on this thread. If using GitHub Actions is not recommended for this process, please let me know so that I don't waste more time on this.

As I mention previously I think I found a process that works (at least for our requirements) which is:

1. Running eksupgrade
2. Using terraform refresh to update the state file with the actual new versions that the eksupgrade tool has created
3. Updating my Terraform Template with the matching versions for Control Plane, NodeGroups and add-ons

By following those steps I maintain the latest version of EKS and it's components and making sure Terraform is updates to the latest such that I can spin up an exact duplicate cluster if needed (which are the requirements from my manager).

I wanted to automate that process by using GHA, but as I mentioned that seems to be running into timeout issues.

Any input you have with regards to anything I mentioned above would be greatly appreciated.

Thanks,
Nave