[aws-eks] 1.45.0 introduces regression in EKS Cluster creation: Caller does not have permission to perform `iam:listAttachedRolePolicies` #8574

winjer · 2020-06-16T10:53:32Z

Reproduction Steps

new eks.Cluster() in 1.45.0.

Works ok in 1.44.0

Error Log

EKSCluster/Resource/Resource/Default (EKSClusterE11008B6) Failed to create resource. Error: Caller does not have permission to perform `iam:listAttachedRolePolicies`
    at invokeUserFunction (/var/task/framework.js:85:19)
    at process._tickCallback (internal/process/next_tick.js:68:7)
Remote function error: InvalidParameterException: Caller does not have permission to perform `iam:listAttachedRolePolicies`
    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-s
        new CustomResource (/home/doug/projects/trustsrv.io/node_modules/@aws-cdk/core/lib/custom-resource.ts:115:21)
        \_ new ClusterResource (/home/doug/projects/trustsrv.io/node_modules/@aws-cdk/aws-eks/lib/cluster-resource.ts:114:22)
        \_ new Cluster (/home/doug/projects/trustsrv.io/node_modules/@aws-cdk/aws-eks/lib/cluster.ts:406:18)

Environment

CLI Version : 1.45.0
Framework Version: 1.45.0
Node.js Version: 14.2.0
OS : debian
Language (Version): Typescript 3.7.3

Other

This is 🐛 Bug Report

The text was updated successfully, but these errors were encountered:

rameshmimit · 2020-06-17T17:29:00Z

I have spent couple of hours already to debug this issue and realized, it could be related to CDK upgrade. Getting the similar error while creating EKS fargate cluster. Below is the error

fault-profile/Resource/Default (fargateclusterfargateprofiledefaultprofile00E7FED0) Resource creation Initiated 44/49 | 3:20:33 am | CREATE_FAILED | Custom::AWSCDK-EKS-FargateProfile | fargate-cluster/fargate-profile-default-profile/Resource/Default (fargateclusterfargateprofiledefaultprofile00E7FED0) Failed to create resource. Error: User: arn:aws:sts::123456789012:assumed-role/ServerlessEksStack-fargateclusterCreationRole55403-T30WTXW6EOV4/AWSCDK.EKSCluster.Create.4db13c7b-f75a-44d1-b1d2-2a7256327c1a is not authorized to perform: iam:PassRole on resource: arn:aws:iam::123456789012:role/ServerlessEksStack-fargateprofileroleEC9BD101-F8LX3GQK3NK at invokeUserFunction (/var/task/framework.js:85:19) at process._tickCallback (internal/process/next_tick.js:68:7) Remote function error: AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/ServerlessEksStack-fargateclusterCreationRole55403-T30WTXW6EOV4/AWSCDK.EKSCluster.Create.4db13c7b-f75a-44d1-b1d2-2a7256327c1a is not authorized to perform: iam:PassRole on resource: arn:aws:iam::123456789012:role/ServerlessEksStack-fargateprofileroleEC9BD101-F8LX3GQK3NK at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27) at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)

Same code use to work on 1.44

sogos · 2020-06-20T12:22:21Z

Encountered the same problem that @rameshmimit

Failed to create resource. Error: 
User: arn:aws:sts::XXXXXXXXXXX:assumed-role/TinyPanda-MyEKSClusterNes-MyFargateClusterCreation-1LQDNGMQ71LLK/AWSCDK.EKSCluster.Create.23d963e4-fe80-4743-98b6-a4362805011e
is not authorized to perform:
iam:PassRole
 on resource: arn:aws:iam::XXXXXXXXXXX:role/TinyPanda-MyEKSClusterNes-EKSFargatePodExecutionRo-GXIXGDEUUCM5
 at invokeUserFunction (/var/task/framework.js:85:19)
 at process._tickCallback (internal/process/next_tick.js:68:7)
Remote function error: 
    AccessDeniedException: 
    User: 
        arn:aws:sts::XXXXXXXXXXX:assumed-role/TinyPanda-MyEKSClusterNes-MyFargateClusterCreation-1LQDNGMQ71LLK/AWSCDK.EKSCluster.Create.23d963e4-fe80-4743-98b6-a4362805011e
    is not authorized to perform:
    iam:PassRole on resource:
    arn:aws:iam::XXXXXXXXXXX:role/TinyPanda-MyEKSClusterNes-EKSFargatePodExecutionRo-GXIXGDEUUCM5
        at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)
        at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json

Tested in 1.46.0 and seem fixed for me

eladb · 2020-06-22T09:27:36Z

Confirming this is an issue caused by new permissions needed by EKS. Work in progress.

eladb · 2020-06-22T12:53:13Z

@pahud can you please take a look at this. I suspect it's related to #7637 - did we have to explicitly enable the service linked role here?

pahud · 2020-06-22T12:58:48Z

Looking into this.

pahud · 2020-06-22T14:00:47Z

Hi,

We should have already fixed this issue in 1.46.0. See #8548

eladb · 2020-06-22T14:03:07Z

Resolving. Please chime in if this persists >= 1.46.0

eladb · 2020-07-01T17:04:14Z

This is still an issue with the latest version 1.48.0. Reopening.

When a Fargate profile is being created, the Kubernetes API server in EKS sometimes rejects requests. This means that kubectl-related resources such as KubernetesResources Helm charts may fail during deployment. To address this, we add a "barrier resource" (in the form of an SSM parameter) which waits for all fargate profiles to be created before allowing kubectl resources to continue. This is done by the barrier taking a dependency on all FargateProfile resources and all kubectl resources taking a dependency on the barrier. Fixes #8854 This commit also fixes #8574 by adding `iam:ListAttachedRolePolicies` to the cluster's creation role IAM policy.

…8859) When a Fargate profile is being created, the Kubernetes API server in EKS sometimes rejects requests. This means that kubectl-related resources such as KubernetesResources Helm charts may fail during deployment. To address this, we add a "barrier resource" (in the form of an SSM parameter) which waits for all fargate profiles to be created before allowing kubectl resources to continue. This is done by the barrier taking a dependency on all FargateProfile resources and all kubectl resources taking a dependency on the barrier. Fixes #8854 This commit also fixes #8574 by adding `iam:ListAttachedRolePolicies` to the cluster's creation role IAM policy. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

jargelo · 2020-07-06T13:33:51Z

Today I've also gotten the same error but on the ec2:DescribeVpcs permission. This was not the case on Friday because I rolled out an EKS cluster with a CDK build done from the master branch. Adding the DescribeVpcs permission solved the problem, same as the PR in #8859

Seems like AWS is changing stuff on their end that requires more permissions?

winjer added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 16, 2020

SomayaB added the @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service label Jun 17, 2020

SomayaB assigned eladb Jun 17, 2020

eladb added the p1 label Jun 22, 2020

eladb closed this as completed Jun 22, 2020

eladb reopened this Jul 1, 2020

SomayaB removed the needs-triage This issue or PR still needs to be triaged. label Jul 2, 2020

eladb mentioned this issue Jul 2, 2020

fix(eks): kubectl resources fail before fargate profiles are created #8855

Merged

1 task

eladb mentioned this issue Jul 2, 2020

fix(eks): kubectl resources fail before fargate profiles are created #8859

Merged

mergify bot closed this as completed in #8859 Jul 2, 2020

eladb mentioned this issue Jul 12, 2020

[aws-eks] ec2:DescribeVpcs permission is missing when creating clusters #9027

Closed

github-actions bot assigned iliapolo Aug 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[aws-eks] 1.45.0 introduces regression in EKS Cluster creation: Caller does not have permission to perform `iam:listAttachedRolePolicies` #8574

[aws-eks] 1.45.0 introduces regression in EKS Cluster creation: Caller does not have permission to perform `iam:listAttachedRolePolicies` #8574

winjer commented Jun 16, 2020 •

edited

Loading

rameshmimit commented Jun 17, 2020

sogos commented Jun 20, 2020

eladb commented Jun 22, 2020

eladb commented Jun 22, 2020

pahud commented Jun 22, 2020

pahud commented Jun 22, 2020

eladb commented Jun 22, 2020

eladb commented Jul 1, 2020

jargelo commented Jul 6, 2020

[aws-eks] 1.45.0 introduces regression in EKS Cluster creation: Caller does not have permission to perform iam:listAttachedRolePolicies #8574

[aws-eks] 1.45.0 introduces regression in EKS Cluster creation: Caller does not have permission to perform iam:listAttachedRolePolicies #8574

Comments

winjer commented Jun 16, 2020 • edited Loading

Reproduction Steps

Error Log

Environment

Other

rameshmimit commented Jun 17, 2020

sogos commented Jun 20, 2020

eladb commented Jun 22, 2020

eladb commented Jun 22, 2020

pahud commented Jun 22, 2020

pahud commented Jun 22, 2020

eladb commented Jun 22, 2020

eladb commented Jul 1, 2020

jargelo commented Jul 6, 2020

[aws-eks] 1.45.0 introduces regression in EKS Cluster creation: Caller does not have permission to perform `iam:listAttachedRolePolicies` #8574

[aws-eks] 1.45.0 introduces regression in EKS Cluster creation: Caller does not have permission to perform `iam:listAttachedRolePolicies` #8574

winjer commented Jun 16, 2020 •

edited

Loading