[aws-load-balancer-controller] Pipeline fails with "Release does not exist. Installing it now" when trying to install with CDK

[briancaffey]

Describe the bug

I'm trying to install the AWS Load Balancer Controller Helm chart on an EKS cluster using CDK and I'm getting errors saying that the release and version cannot be found in the https://aws.github.io/eks-charts repository. I'm following instructions on this website: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/

Steps to reproduce

Here is the GitHub repo that I'm using to build a high-level CDK construct for deploying Python web applications: https://github.com/briancaffey/django-cdk. It currently includes a construct for ECS, and I'm trying to add a construct for EKS, but I'm stuck. To produce the error I'm seeing, you could clone the repo, and then run:

npm run build
make deploy-eks

The relevant part of the code from this file: https://github.com/briancaffey/django-cdk/blob/main/src/django-eks.ts#L156

    // AWS Load Balancer Controller Helm Chart
    // https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/#summary

    // AWS Application Load Balancer Controller Helm Chart CRDs
    // These must be installed before installing the Helm Chart because CDK
    // installs the Helm chart with `helm upgrade` which does not
    // automatically install CRDs
    // This should be equivalent to Step 2:
    // kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

    const awslbcCrdsUrl = 'https://raw.githubusercontent.com/aws/eks-charts/master/stable/aws-load-balancer-controller/crds/crds.yaml';
    const awslbcCrdsYaml = request('GET', awslbcCrdsUrl).getBody('utf8');
    const awslbcCrdsObjects = yaml.loadAll(awslbcCrdsYaml);

    new eks.KubernetesManifest(this, 'alb-custom-resource-definition', {
      cluster: this.cluster,
      manifest: awslbcCrdsObjects,
      overwrite: true,
      prune: true,
    });

    // The following is basically trying to do:
    // helm repo add eks https://aws.github.io/eks-charts
    // helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=<cluster-name> --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
    // but it is failing saying that the version and release cannot be found in
    // the repository (https://aws.github.io/eks-chart)
    // TODO: Fix this
    new eks.HelmChart(scope, 'alb-ingress-controller', {
      cluster: this.cluster,
      wait: true,
      // Not sure if this value is needed
      release: 'aws-load-balancer-controller',
      // I'm not sure if the `eks/` prefix is needed here
      chart: 'eks/aws-load-balancer-controller',
      repository: 'https://aws.github.io/eks-charts',
      version: '2.2.0',
      namespace: 'kube-system',
      values: {
        clusterName: this.cluster.clusterName,
        serviceAccount: {
          create: false,
          name: 'aws-load-balancer-controller',
        },
      },
    });

Expected outcome

I'm expecting the CDK pipeline to succeed. Instead, I'm getting a CDK pipeline failure that contains the following message about the Helm chart not being able to find a release and a version in the https://aws.github.io/eks-charts repository:

1:35:23 AM | CREATE_FAILED        | Custom::AWSCDK-EKS-HelmChart          | alb-ingress-controller/Resourc
e/Default
Received response status [FAILED] from custom resource. Message returned: Error: b'WARNING: Kubernetes con
figuration file is group-readable. This is insecure. Location: /tmp/kubeconfig\nWARNING: Kubernetes config
uration file is world-readable. This is insecure. Location: /tmp/kubeconfig\nRelease "aws-load-balancer-co
ntroller" does not exist. Installing it now.\nError: chart "eks/aws-load-balancer-controller" version "2.2
.0" not found in https://aws.github.io/eks-charts repository\n'

Here are the full logs from the kubectl lambda:

| 1622439319006 | {"RequestType": "Create", "ServiceToken": "arn:aws:lambda:us-east-1:111111111111:function:DjangoEks-awscdkawseksKub-ProviderframeworkonEvent-mTlY3DMr7nvU", "ResponseURL": "https://cloudformation-custom-resource-response-useast1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%111111111111%3Astack/DjangoEks/145896c0-c1ba-11eb-92c6-12417e4a260f%7Calbingresscontroller3EA12603%7C24a194a9-13f1-471b-9f47-720c0a4e26e5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210531T053516Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA6L7Q4OWTW7BU6FUG%2F20210531%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=5e889872d0316f36aa5bc8486c77606f8420b760c0608ed212bcb0af95ac0389", "StackId": "arn:aws:cloudformation:us-east-1:111111111111:stack/DjangoEks/145896c0-c1ba-11eb-92c6-12417e4a260f", "RequestId": "24a194a9-13f1-471b-9f47-720c0a4e26e5", "LogicalResourceId": "albingresscontroller3EA12603", "ResourceType": "Custom::AWSCDK-EKS-HelmChart", "ResourceProperties": {"ServiceToken": "arn:aws:lambda:us-east-1:111111111111:function:DjangoEks-awscdkawseksKub-ProviderframeworkonEvent-mTlY3DMr7nvU", "Repository": "https://aws.github.io/eks-charts", "Version": "2.2.0", "Values": "{\"clusterName\":\"CdkSampleLibMyEksClusterDEADB58D-44e82832326e4b30a5972f0532acfc9c\",\"serviceAccount\":{\"create\":false,\"name\":\"aws-load-balancer-controller\"}}", "ClusterName": "CdkSampleLibMyEksClusterDEADB58D-44e82832326e4b30a5972f0532acfc9c", "Wait": "true", "RoleArn": "arn:aws:iam::111111111111:role/DjangoEks-CdkSampleLibMyEksClusterCreationRole1020-O0GVMHOEQ0MZ", "Release": "aws-load-balancer-controller", "Chart": "eks/aws-load-balancer-controller", "Namespace": "kube-system", "CreateNamespace": "true"}}                                                                       |
| 1622439319006 | [INFO] 2021-05-31T05:35:19.006Z 0d6e27cc-afea-47c0-a6a5-c6a62d3969ed {"RequestType": "Create", "ServiceToken": "arn:aws:lambda:us-east-1:111111111111:function:DjangoEks-awscdkawseksKub-ProviderframeworkonEvent-mTlY3DMr7nvU", "ResponseURL": "https://cloudformation-custom-resource-response-useast1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%111111111111%3Astack/DjangoEks/145896c0-c1ba-11eb-92c6-12417e4a260f%7Calbingresscontroller3EA12603%7C24a194a9-13f1-471b-9f47-720c0a4e26e5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210531T053516Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA6L7Q4OWTW7BU6FUG%2F20210531%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=5e889872d0316f36aa5bc8486c77606f8420b760c0608ed212bcb0af95ac0389", "StackId": "arn:aws:cloudformation:us-east-1:111111111111:stack/DjangoEks/145896c0-c1ba-11eb-92c6-12417e4a260f", "RequestId": "24a194a9-13f1-471b-9f47-720c0a4e26e5", "LogicalResourceId": "albingresscontroller3EA12603", "ResourceType": "Custom::AWSCDK-EKS-HelmChart", "ResourceProperties": {"ServiceToken": "arn:aws:lambda:us-east-1:111111111111:function:DjangoEks-awscdkawseksKub-ProviderframeworkonEvent-mTlY3DMr7nvU", "Repository": "https://aws.github.io/eks-charts", "Version": "2.2.0", "Values": "{\"clusterName\":\"CdkSampleLibMyEksClusterDEADB58D-44e82832326e4b30a5972f0532acfc9c\",\"serviceAccount\":{\"create\":false,\"name\":\"aws-load-balancer-controller\"}}", "ClusterName": "CdkSampleLibMyEksClusterDEADB58D-44e82832326e4b30a5972f0532acfc9c", "Wait": "true", "RoleArn": "arn:aws:iam::111111111111:role/DjangoEks-CdkSampleLibMyEksClusterCreationRole1020-O0GVMHOEQ0MZ", "Release": "aws-load-balancer-controller", "Chart": "eks/aws-load-balancer-controller", "Namespace": "kube-system", "CreateNamespace": "true"}}  |
| 1622439320446 | Added new context arn:aws:eks:us-east-1:111111111111:cluster/CdkSampleLibMyEksClusterDEADB58D-44e82832326e4b30a5972f0532acfc9c to /tmp/kubeconfig                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 1622439322342 | [ERROR] Exception: b'WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /tmp/kubeconfig\nWARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /tmp/kubeconfig\nRelease "aws-load-balancer-controller" does not exist. Installing it now.\nError: chart "eks/aws-load-balancer-controller" version "2.2.0" not found in https://aws.github.io/eks-charts repository\n' Traceback (most recent call last):   File "/var/task/index.py", line 17, in handler     return helm_handler(event, context)   File "/var/task/helm/__init__.py", line 50, in helm_handler     helm('upgrade', release, chart, repository, values_file, namespace, version, wait, timeout, create_namespace)   File "/var/task/helm/__init__.py", line 94, in helm     raise Exception(output)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1622439322343 | END RequestId: 0d6e27cc-afea-47c0-a6a5-c6a62d3969ed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 1622439322343 | REPORT RequestId: 0d6e27cc-afea-47c0-a6a5-c6a62d3969ed Duration: 3337.59 ms Billed Duration: 3338 ms Memory Size: 1024 MB Max Memory Used: 189 MB Init Duration: 124.69 ms                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

Environment

• Chart name: aws-load-balancer-controller
• Chart version: 2.2.0
• Kubernetes version: 1.19
• Using EKS (yes/no), if so version? Yes, 1.19

Additional Context: I'm trying to automate the deployment of the AWS Load Balancer Controller as described here in the documentation (https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/) by installing it using CDK.

Some relevant issues:

• AWS LB Controller cannot be automated because CRD step #374
• [aws-eks] AWS load balancer controller support aws-cdk#8836 (someone claims to have been able to install this controller using CDK and provides a detailed code sample, but I'm unable to reproduce successfully)

I would love to get some direction on how I can install this controller in my cluster with CDK. I think I'm close, but I haven't been able to get help from the CDK slack channels or other searches, and I have tried lots of different values in my CDK script that have caused the pipeline to fail for what seems like the same reason (Release does not exist / Version not found). I would love to contribute a CDK example to the Deployment/Installation section of the docs if I can get some help sorting out how to fix the errors I'm having.

[briancaffey]

I was using the wrong version, it should be 1.2.0, not 2.2.0.

1.2.0 is the Chart version, which installs version 2.2.0 of AWS Load Balancer Controller.