eks: a destructive update of the eks cluster and fails to do so #24174

watany-dev · 2023-02-15T09:05:22Z

Describe the bug

When updating an eks Cluster, there is no permission to DeleteCluster from Role if there are destructive changes to the cluster.

https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-eks/lib/cluster-resource.ts#L152-L166

The following rolls failed. Probably because of the following

The "resource" in the IAM policy is changed to the new cluster.
Attempting to "eks:DeleteCluster" on the old cluster
the deletion fails because the access rights to the old cluster were revoked in 1.

Expected Behavior

Successful deletion

Current Behavior

Failure to do this.

Reproduction Steps

First deployed as follows.

    new eks.Cluster(this, 'hello-eks', {
      version: eks.KubernetesVersion.V1_24,
      kubectlLayer: new KubectlV24Layer(this, 'kubectl'),
    });

Then update by rewriting it like this (changing the cluster name is a destructive change)

    new eks.Cluster(this, 'hello-eks', {
      clusterName: "sample",
      version: eks.KubernetesVersion.V1_24,
      kubectlLayer: new KubectlV24Layer(this, 'kubectl'),
    });

Possible Solution

EKS:DeleteCluster's Statement separation, Resouce='*'

Additional Information/Context

No response

CDK CLI Version

2.62.2

Framework Version

No response

Node.js Version

any

OS

any

Language

Typescript

Language Version

No response

Other information

No response

pahud · 2023-02-16T13:37:27Z

Hi @watany-dev , what is the error message when you delete the cluster?

watany-dev · 2023-02-16T14:51:23Z

@pahud
Thanks for bringing this to my attention.

I logged it again. This is when the cluster name is changed from "sample" to "sample2”.

Received response status [FAILED] from custom resource. Message returned: User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloe
ksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66 is not authorized to perform: eks:De
leteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample

Logs: /aws/lambda/eks-dev-awscdkawseksCluster-OnEventHandler42BEBAE0-MGF8s2fAMvvq

The following are CloudWatch logs

INIT_START Runtime Version: nodejs:14.v28	Runtime Version ARN: arn:aws:lambda:ap-northeast-1::runtime:be4a32da6aeaa90583d476cb548ced79ae2c9aa7b14a45129198bbd13ea5822d
START RequestId: 510218c4-6150-47d9-9aa9-31fda0cab101 Version: $LATEST
2023-02-16T14:39:53.756Z	510218c4-6150-47d9-9aa9-31fda0cab101	INFO	
{
    "assumeRole": {
        "RoleArn": "arn:aws:iam::aaaaaaaaaaaaa:role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8",
        "RoleSessionName": "AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66"
    }
}

2023-02-16T14:39:53.847Z	510218c4-6150-47d9-9aa9-31fda0cab101	INFO	onDelete: deleting cluster sample
2023-02-16T14:39:54.428Z	510218c4-6150-47d9-9aa9-31fda0cab101	INFO	[AWS sts 200 0.539s 0 retries] assumeRole({
  RoleArn: 'arn:aws:iam::aaaaaaaaaaaaa:role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8',
  RoleSessionName: 'AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66'
})
2023-02-16T14:39:54.638Z	510218c4-6150-47d9-9aa9-31fda0cab101	INFO	[AWS eks 403 0.752s 0 retries] deleteCluster({ name: 'sample' })
2023-02-16T14:39:54.639Z	510218c4-6150-47d9-9aa9-31fda0cab101	ERROR	Invoke Error 	
{
    "errorType": "AccessDeniedException",
    "errorMessage": "User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
    "code": "AccessDeniedException",
    "message": "User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
    "time": "2023-02-16T14:39:54.637Z",
    "requestId": "4cdd26b0-1aa0-4fc2-9a10-156e806f5d75",
    "statusCode": 403,
    "retryable": false,
    "retryDelay": 25.084285757492264,
    "stack": [
        "AccessDeniedException: User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
        "    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"
    ]
}

END RequestId: 510218c4-6150-47d9-9aa9-31fda0cab101
REPORT RequestId: 510218c4-6150-47d9-9aa9-31fda0cab101	Duration: 910.85 ms	Billed Duration: 911 ms	Memory Size: 128 MB	Max Memory Used: 89 MB	Init Duration: 623.59 ms	
START RequestId: 6d5d5d74-a4be-4a03-b14b-69b0a70057c6 Version: $LATEST
2023-02-16T14:43:00.287Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	INFO	
{
    "assumeRole": {
        "RoleArn": "arn:aws:iam::aaaaaaaaaaaaa:role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8",
        "RoleSessionName": "AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f"
    }
}

2023-02-16T14:43:00.288Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	INFO	onDelete: deleting cluster sample
2023-02-16T14:43:00.777Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	INFO	[AWS sts 200 0.488s 0 retries] assumeRole({
  RoleArn: 'arn:aws:iam::aaaaaaaaaaaaa:role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8',
  RoleSessionName: 'AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f'
})
2023-02-16T14:43:00.967Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	INFO	[AWS eks 403 0.679s 0 retries] deleteCluster({ name: 'sample' })
2023-02-16T14:43:00.968Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	ERROR	Invoke Error 	
{
    "errorType": "AccessDeniedException",
    "errorMessage": "User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
    "code": "AccessDeniedException",
    "message": "User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
    "time": "2023-02-16T14:43:00.967Z",
    "requestId": "32ffabd4-e25b-4fc7-ab35-1576aa68b2db",
    "statusCode": 403,
    "retryable": false,
    "retryDelay": 45.26429123253239,
    "stack": [
        "AccessDeniedException: User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
        "    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"
    ]
}

END RequestId: 6d5d5d74-a4be-4a03-b14b-69b0a70057c6
REPORT RequestId: 6d5d5d74-a4be-4a03-b14b-69b0a70057c6	Duration: 940.47 ms	Billed Duration: 941 ms	Memory Size: 128 MB	Max Memory Used: 92 MB

pahud · 2023-02-20T17:59:46Z

@watany-dev Thank you for the details. I am making it as p2 now but making EKS:DeleteCluster on * doesn't seem to be a good idea. I am leaving this issue open and any ideas are welcome here.

maxjiang153 · 2023-11-15T03:11:39Z

@pahud Is it possible to add a specific tag to all EKS clusters created by the CDK stack? So adding EKS:DeleteCluster based on tag policy might be a good idea

pahud · 2024-02-27T17:25:02Z

related to #29282

pahud · 2024-02-27T17:54:27Z

closing with #29283

github-actions · 2024-02-27T17:54:43Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

### Issue # (if applicable) As described in #29282 , when renaming the cluster, an additional temporary IAM policy will be required. I am proposing the doc update to clarify this with this PR. Closes #29282 #24174 ### Reason for this change To address this use case. ### Description of changes ### Description of how you validated changes ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

watany-dev added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 15, 2023

github-actions bot added the @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service label Feb 15, 2023

pahud added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Feb 16, 2023

pahud self-assigned this Feb 16, 2023

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Feb 16, 2023

pahud added p2 effort/medium Medium work item – several days of effort labels Feb 20, 2023

pahud removed their assignment Feb 23, 2023

This was referenced Feb 27, 2024

eks: renaming the cluster would trigger rollback due to not authorized to delete the cluster #29282

Closed

chore(eks): improve the doc on updating clusters #29283

Merged

pahud closed this as completed Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

eks: a destructive update of the eks cluster and fails to do so #24174

eks: a destructive update of the eks cluster and fails to do so #24174

watany-dev commented Feb 15, 2023 •

edited

Loading

pahud commented Feb 16, 2023

watany-dev commented Feb 16, 2023 •

edited

Loading

pahud commented Feb 20, 2023

maxjiang153 commented Nov 15, 2023

pahud commented Feb 27, 2024

pahud commented Feb 27, 2024

github-actions bot commented Feb 27, 2024

eks: a destructive update of the eks cluster and fails to do so #24174

eks: a destructive update of the eks cluster and fails to do so #24174

Comments

watany-dev commented Feb 15, 2023 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

pahud commented Feb 16, 2023

watany-dev commented Feb 16, 2023 • edited Loading

pahud commented Feb 20, 2023

maxjiang153 commented Nov 15, 2023

pahud commented Feb 27, 2024

pahud commented Feb 27, 2024

github-actions bot commented Feb 27, 2024

⚠️COMMENT VISIBILITY WARNING⚠️

watany-dev commented Feb 15, 2023 •

edited

Loading

watany-dev commented Feb 16, 2023 •

edited

Loading