[aws-eks] ec2:DescribeVpcs permission is missing when creating clusters

[eladb]

Today I've also gotten the same error but on the ec2:DescribeVpcs permission. This was not the case on Friday because I rolled out an EKS cluster with a CDK build done from the master branch. Adding the DescribeVpcs permission solved the problem, same as the PR in #8859

Seems like AWS is changing stuff on their end that requires more permissions?

Originally posted by @jargelo in #8574 (comment)

[jargelo]

Thanks @eladb , I was about to raise a new issue for this since I wasn't sure you seen my comment.

[stefanolczak]

I've got the issue when deploying CDK app from scratch ( VPC and EKS as seperate stacks ). Deployment of EKS stack failed with the same error. During redeploy it worked fine. I am using CDK 1.60

[iliapolo]

@stefanolczak Can you share the code? Thanks

[stefanolczak]

#!/usr/bin/env python3

from aws_cdk import (
    core,
    aws_ec2,
    aws_eks
)


class VpcStack(core.Stack):

    def __init__(self, app: core.App) -> None:
        super().__init__(app, 'vpc-stack')

        self.vpc = aws_ec2.Vpc(
            scope=self,
            id='vpc'
        )


class EksStack(core.Stack):
    def __init__(self, app: core.App, vpc: aws_ec2.Vpc) -> None:
        super().__init__(app, 'eks-stack')

        aws_eks.Cluster(
            scope=self,
            id='eks',
            cluster_name='eks-cluster',
            default_capacity=0,
            version=aws_eks.KubernetesVersion.V1_17,
            vpc=vpc,
            endpoint_access=aws_eks.EndpointAccess.PRIVATE
        )


app = core.App()
vpc_stack = VpcStack(app)
EksStack(app, vpc_stack.vpc)

app.synth()

The issue reproduces only on empty AWS account or on AWS account where CDK or EKS wasn't used for a while. I'm investigating it more why it does matter.

[stefanolczak]

I'm almost sure I have once seen other issue ( also with IAM policy ) related to deploying EKS that was fixed by redeploy so I'm guessing there is some global problem in that matter.

I found the error message from mentioned issue from the past:

Error: Caller does not have permission to perform `iam:listAttachedRolePolicies`

[sandeep-balouria]

Upgraded to 1.60.0
Tried deploying to fresh aws account and failed with this

Failed to create resource. IAM role's policy must include the ec2:DescribeVpcs action

redeploy goes through fine.

[kossmoboleat]

@eladb This issue might have to be reopened. I've had the same issue with CDK 1.66.0 yesterday when trying to deploy a stack from scratch too.

The first try to deploy fails with the mentioned error message but a re-deploy works. Seems the CDK cannot clean up all created stacks/resources and then those are re-used and it succeeds.

I've also checked and the creation role now has the necessary permissions. So I'm speculating the policy is added to late with the changes in the MR when deploying from scratch.

In our stack we're using a shared VPC with another AWS account that we're importing like so:

    this.vpc = Vpc.fromLookup(this, this.vpcName + '-VPC', {
      tags: {
        Name: this.vpcName
      }
    });

And then using it like so to create the EKS cluster (which fails the first time around):

    const clusterAdmin = new Role(this, 'AdminRole', {
      assumedBy: new AccountRootPrincipal()
    });
    const clusterProps: ClusterProps = {
      version: KubernetesVersion.V1_17,
      outputClusterName: true,
      kubectlEnabled: true,
      defaultCapacity: 1,
      defaultCapacityInstance: new InstanceType("t2.small"),
      mastersRole: clusterAdmin,
      vpc: this.vpc
    };

    return new Cluster(this, name, clusterProps);

[iliapolo]

Hi @kossmoboleat - Thanks, yeah looks like there is still something here. Re-opening and we'll investigate.

[scalasm]

I've got the same issue with CDK 1.67.0 . If you need a simple test project, just use this on GitHub.

You can see the file log in aws-cdk-issue-9027.log . I can confirm what @kossmoboleat: if I run 'cdk deploy' just after this error, then the deploy is succesful :|

[PerArneng]

I also ran in to this problem when setting up a test cluster. https://github.com/PerArneng/eks-cdk-test/tree/issue_9027 . I ran it against a blank account and then it worked on the second attempt. I also got an email from AWS saying that i was granted access to AWS resources You recently requested an AWS Service that required additional validation. Your request has now been validated for AWS Europe (Stockholm) region(s)

Edit: In the branch above the instance type should be 't3.nano' because 't2.nano' is not available in that region

[aakrem]

Still happening with 1.76.0 and redeploying the stack fixed it

[iliapolo]

Managed to reproduce this on a blank account as well, and also got the email that @PerArneng mentioned.

I believe the error is coming from a validation that EKS makes on the cluster role, and not necessarily from an operation that requires this action.

What happens is that the cluster role we pass gets created by default like so:

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster.ts

Lines 911 to 916 in 8216738

	this.role = props.role || new iam.Role(this, 'Role', {
	assumedBy: new iam.ServicePrincipal('eks.amazonaws.com'),
	managedPolicies: [
	iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKSClusterPolicy'),
	],
	});

This AmazonEKSClusterPolicy does contain the ec2:DescribeVpcs action, but I suspect that on a blank account, granting usage of a managed policy is what requires this additional validation.

I'll do some internal investigation to see what the best path forward here.

[kay-is]

I got the same problem.

A deployment takes 30minutes, so re-deploying isn't a viable workaround for me.

Any other solutions?

[iliapolo]

We are still investigating the exact set of conditions where this would happen, and how one can avoid it.

[sunny0826]

I had the same problem with CDK version: 1.80.0 (build 31132ca)

[bhargav18]

Same issue with CDK 1.86 on blank account.

[fspaniol]

Same issue with 1.87.1. Re-deploy did the trick

[rafaelpereyra]

Hello,

I was able to reproduce this (several times) using a tool that vends new accounts for training. Manually adding a policy to the synthesized CloudFormation Template to the role created by CDK fixed the issue for me:

  custernameCreationRole942A9DDE:
    Type: AWS::IAM::Role
    Properties:
      Policies: 
        - PolicyName: PatchForEKSIssues
          PolicyDocument:
            Statement:
              - Action: ec2:Describe*
                Effect: Allow
                Resource: "*"
            Version: "2012-10-17"

[iliapolo]

I was able to reproduce this multiple times in a single account by manually deleting the AWSServiceRoleForAmazonEKS SLR before each deployment. In this scenario (which simulates a blank account) - EKS will create the SLR upon first cluster deployment, and use it for all subsequent deployments as well.

However, seems like there might be an issue with this first deployment flow - which I am following up on internally.
In the meantime, I can verify that the patch policy @rafaelpereyra suggested solves the issue, and I've created a PR to include it in the default policy of the creation role.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[dennisvang]

Unrelated to the original post, but it may be useful for people ending up here based on the title:

ec2:DescribeVpcs permission errors may also arise if you try to deploy a cdk-pipeline stack that calls Vpc.fromLookup(), without first running cdk synth locally and commiting the resulting cdk.context.json.

Also see context-lookups