Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDK fails to delete old global table upon id change #7189

Closed
SachinShekhar opened this issue Apr 6, 2020 · 1 comment · Fixed by #8224
Closed

CDK fails to delete old global table upon id change #7189

SachinShekhar opened this issue Apr 6, 2020 · 1 comment · Fixed by #8224
Assignees
Labels
@aws-cdk/aws-dynamodb Related to Amazon DynamoDB bug This issue is a bug. p1

Comments

@SachinShekhar
Copy link
Contributor

When I change id of a Dynamodb Global Table construct with removal policy DESTROY, CDK fails to remove old Global Table.

Note: I am NOT talking about deprecated aws-dynamodb-global. I am talking about new experimental aws-dynamodb.Table.replicationRegions.

Reproduction Steps

  1. Create a new Global Table:
new dynamodb.Table(this, 'Table1', {
      ...
      ...
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      replicationRegions: ['us-west-1', 'eu-central-1']
    });
  1. Deploy the stack.
  2. Change the id of Global Table:
new dynamodb.Table(this, 'Table2', {
      ...
      ...
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      replicationRegions: ['us-west-1', 'eu-central-1']
    });
  1. Deploy the stack.

It'll create a new table with proper replicas, but fail to delete old table. The old table and its replica will remain in the account even after you destroy the stack (meaning it gets completely detached with the stack).

Error Log

The errors are visible in deployment logs.

Error 1 (appears 6 times for 2 replica regions):

5/8 | 2:43:54 PM | DELETE_FAILED        | AWS::CloudFormation::CustomResource | <Old Id>Replicaeucentral15DB8EA77 Failed to delete resource. Error: User: arn:aws:sts::<Account No.>:assumed-role/<Stack Name>-awscdkawsd-OnEventHandlerServiceRol-1WR6JLLTTGU0W/<Stack Name>-awscdkawsdyn-OnEventHandler42BEBAE0-94ML7HG65IAU is not authorized to perform:
dynamodb:UpdateTable on resource: arn:aws:dynamodb:us-east-1:<Account No.>:table/<Old Table Id>
    at invokeUserFunction (/var/task/framework.js:85:19)
    at process._tickCallback (internal/process/next_tick.js:68:7)
Remote function error: AccessDeniedException: User: arn:aws:sts::<Account No.>:assumed-role/<Stack Name>-awscdkawsd-OnEventHandlerServiceRol-1WR6JLLTTGU0W/<Stack Name>-awscdkawsdyn-OnEventHandler42BEBAE0-94ML7HG65IAU is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:us-east-1:<Account No.>:table/<Old Table Id Online>
    at Request.extractError (/tmp/node_modules/aws-sdk/lib/protocol/json.js:51:27)
    at Request.callListeners (/tmp/node_modules/aws-sdk/lib/sequential_executo

Error 2 (appears 3 times for 2 replica regions):

8/8 | 2:44:10 PM | DELETE_FAILED        | AWS::DynamoDB::Table       | <Old Construct Id><Random No.> Replica cannot be deleted because it has acted as a source region for new replica(s) being added to the table in the last 24 hours. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: <Random No.>

Note: I have replaced private info in errors with <xyz>.

Environment

  • CLI Version : 1.31.0 (build 8f3ac79)
  • Framework Version: 1.31.0
  • OS : Windows 10 Pro
  • Language : Typescript

This is 🐛 Bug Report

@SachinShekhar SachinShekhar added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Apr 6, 2020
@SachinShekhar SachinShekhar changed the title Fails to delete old global table upon id change CDK fails to delete old global table upon id change Apr 6, 2020
@SomayaB SomayaB added the @aws-cdk/aws-dynamodb Related to Amazon DynamoDB label Apr 6, 2020
@RomainMuller RomainMuller added p1 and removed needs-triage This issue or PR still needs to be triaged. labels Apr 14, 2020
@RomainMuller
Copy link
Contributor

I suppose it's because the old permissions either get removed or updated away before the delete attempt begins. Maybe we can make this work by changing how the permissions are added, or by tuning around dependencies.

RomainMuller added a commit that referenced this issue May 27, 2020
The permissions required to clean up old DynamoDB Global Tables replicas
were set up in such a way that removing a replication region, or
dropping replication entirely (or when cuasing a table replacement),
they were removed before CloudFormation gets to the `CLEAN_UP` phase,
causing a clean up failure (and old tables would remain there).

This changes the way permissions are granted to the replication handler
resource so that they are added using a separate `iam.Policy` resource,
so that deleted permissions are also removed during the `CLEAN_UP` phase
after the resources depending on them have been deleted.

The tradeoff is that two additional resources are added to the stack
that defines the DynamoDB Global Tables, where previously those
permissions were mastered in the nested stack that holds the replication
handler. Unofrtunately, the nested stack gets it's `CLEAN_UP` phase
executed as part of the nested stack resource update, not during it's
parent stack's `CLEAN_UP` phase.

Fixes #7189
@mergify mergify bot closed this as completed in #8224 Jun 8, 2020
mergify bot pushed a commit that referenced this issue Jun 8, 2020
The permissions required to clean up old DynamoDB Global Tables replicas
were set up in such a way that removing a replication region, or
dropping replication entirely (or when causing a table replacement),
they were removed before CloudFormation gets to the `CLEAN_UP` phase,
causing a clean up failure (and old tables would remain there).

This changes the way permissions are granted to the replication handler
resource so that they are added using a separate `iam.Policy` resource,
so that deleted permissions are also removed during the `CLEAN_UP` phase
after the resources depending on them have been deleted.

The tradeoff is that two additional resources are added to the stack
that defines the DynamoDB Global Tables, where previously those
permissions were mastered in the nested stack that holds the replication
handler. Unofrtunately, the nested stack gets it's `CLEAN_UP` phase
executed as part of the nested stack resource update, not during it's
parent stack's `CLEAN_UP` phase.

Fixes #7189


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-dynamodb Related to Amazon DynamoDB bug This issue is a bug. p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants