Can not connect to private EKS cluster

[shrivastavshubham34]

I'm trying to access a private EKS cluster already deployed.

Following the readme instructions, I pulled the SARs, but was not sure how to use it.
So, I pulled the repo and had to build it manually based on the existing CDK application.
https://github.com/aws-samples/aws-lambda-layer-kubectl/blob/master/cdk/README.md

I made sure my handler function was using cluster_name as Environment but still I'm getting a time-out after 30 seconds and not sure how to debug this.

[pahud]

Thanks for the report. It's very important to support this scenario. I will look into it in the next 1 or 2 days.

related to aws/aws-cdk#7972

[shrivastavshubham34]

I have assigned a VPC to this lambda function, along with all subnets and a security group with all VPC access.
However, to connect to EKS I tried to add relevant security groups that I used when I tested EKS connectivity from an EC2 instance within the same VPC.
Lambda function, however, keeps timing out.

https://stackoverflow.com/questions/61789276/how-can-i-connect-to-private-eks-endpoint-from-a-lambda-function

Thanks for looking into it!

[pahud]

Yes I noticed that. I will get it sorted and provide a working sample with required fix if any.

[shrivastavshubham34]

Meanwhile, can you suggest a workaround, from a lambda function's perspective?

[pahud]

Not at this moment. But I guess the Lambda function will need to share the same security group with the eks control plane. I'd try explore this way.

[pahud]

@shrivastavshubham34

OK. It works in my environment now

If your lambda function with kubectl lambda layer need to connect to the eks private endpoint, you need:

1. enable Lambda function with vpc support and associate with the same VPC with the eks cluster
2. attach the EKS Cluster Security Group on the Lambda function

In your EKS console

In your Lambda function console

And kubectl get no shoudl return the node list successfully

Let me know if it works in your environment.

[pahud]

Just added some notes in the README FYR

https://github.com/aws-samples/aws-lambda-layer-kubectl/tree/master/cdk#amazon-eks-private-endpoint-support

[shrivastavshubham34]

Hey, thanks for the help. Yes, I was able to make it work using your guide.

Although, the Lambda Role could not connect to my cluster.
Kept getting permission denied.
error: You must be logged in to the server (Unauthorized)

To solve this:

1. I attached an Inline Policy to my role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::1234567890123:role/newrole"
        }
    ]
}

2. Then modified the configmap/aws-auth removing the lambda role with arn:aws:iam::1234567890123:role/newrole.

3. Modified the update kubeconfig with

    aws eks update-kubeconfig --name "$1" --role-arn "$2" --kubeconfig /tmp/kubeconfig

And added role arn:aws:iam::1234567890123:role/newrole.

This made me connect.

Still not sure why I had to assume a role.
But, I'm able to connect to EKS now so I'll close this issue.

Thanks again, @pahud

[Chetan496]

@shrivastavshubham34 your suggestion worked for me. In my case, the EKS worker nodes are in private subnet, the API server endpoint is public. Yet, I was also facing the same issue. Followed the suggestion mentioned by you, and it worked,