Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(aws-dynamodb): No replica created when new table created via Update event #12332

Closed
tom-noble opened this issue Jan 4, 2021 · 2 comments · Fixed by #13300
Closed

(aws-dynamodb): No replica created when new table created via Update event #12332

tom-noble opened this issue Jan 4, 2021 · 2 comments · Fixed by #13300
Assignees
@skinny85
Labels
@aws-cdk/aws-dynamodb Related to Amazon DynamoDB bug This issue is a bug. effort/medium Medium work item – several days of effort p1

Comments

@tom-noble
Copy link

tom-noble commented Jan 4, 2021 •
edited
Loading

Reproduction Steps

  1. Create a DynamoDB Table using AWS CDK with replication_regions set
aws_dynamodb.Table(
   self,
   "lcrCalculatorStatusTable",
   partition_key={"name": "RunId", "type": aws_dynamodb.AttributeType.STRING},
   sort_key=None,
   billing_mode=aws_dynamodb.BillingMode.PAY_PER_REQUEST,
   server_side_encryption=True,
   replication_regions=["eu-central-1"],
)
  1. Synth and Deploy
  2. Update the partition key of the Table using AWS CDK (RunId to RUN_ID in this case)
aws_dynamodb.Table(
   self,
   "lcrCalculatorStatusTable",
   partition_key={"name": "RUN_ID", "type": aws_dynamodb.AttributeType.STRING},
   sort_key=None,
   billing_mode=aws_dynamodb.BillingMode.PAY_PER_REQUEST,
   server_side_encryption=True,
   replication_regions=["eu-central-1"],
)
  1. Synth and Deploy (CDK attempts to update the table, and subsequently creates a new one)
5/28 | 12:29:06 PM | UPDATE_IN_PROGRESS   | AWS::DynamoDB::Table                | lcrCalculatorStatusTable (lcrCalculatorStatusTable99128E5D) Requested update requires the creation of a new physical resource; hence creating one.
5/28 | 12:29:06 PM | UPDATE_IN_PROGRESS   | AWS::DynamoDB::Table                | lcrCalculatorStatusTable (lcrCalculatorStatusTable99128E5D) Resource creation Initiated
7/28 | 12:29:37 PM | UPDATE_COMPLETE      | AWS::DynamoDB::Table                | lcrCalculatorStatusTable (lcrCalculatorStatusTable99128E5D)

What did you expect to happen?

Since a new table is created as part of the update, I expected the new table to be automatically replicated and the deployment to succeed.

What actually happened?

CDK attempts to create a new table with the new partition key but a replica is not created. This causes the DynamoDBReplica custom resource to hang in the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state.

Screenshot 2021-01-04 at 11 17 19

This eventually fails the deployment due to timeout

 15/28 | 1:00:51 PM | UPDATE_FAILED        | Custom::DynamoDBReplica             | lcrCalculatorStatusTable/Replicaeu-central-1/Default (lcrCalculatorStatusTableReplicaeucentral169ED77F2) Failed to update resource. Operation timed out
	new CustomResource (/tmp/jsii-kernel-ixys5V/node_modules/@aws-cdk/core/lib/custom-resource.js:28:25)
	\_ Table.createReplicaTables (/tmp/jsii-kernel-ixys5V/node_modules/@aws-cdk/aws-dynamodb/lib/table.js:875:35)
	\_ new Table (/tmp/jsii-kernel-ixys5V/node_modules/@aws-cdk/aws-dynamodb/lib/table.js:487:18)
	\_ /usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:3556:49
	\_ Kernel._wrapSandboxCode (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:4040:16)
	\_ Kernel._create (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:3556:26)
	\_ Kernel.create (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:3300:21)
	\_ KernelHost.processRequest (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13119:28)
	\_ KernelHost.run (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13057:14)
	\_ Immediate._onImmediate (/usr/local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13060:37)

Environment

  • CDK CLI Version :
  • Framework Version: CDK 1.74.0
  • Node.js Version: 12.20.0
  • OS : python:3.7 Docker Image (running on Bitbucket Pipelines)
  • Language (Version): Python 3.7.9

Other

I think the reason this happens is that despite creating a new table the event is marked as an Update (shown below).

Screenshot 2021-01-04 at 12 01 01

The code generated by CDK to handle DynamoDB replication only checks for Create and Delete events, so a replica isn't created (shown below).

Screenshot 2021-01-04 at 16 27 25

This is reinforced by the fact that deleting and rebuilding the stack fixes the issue. CDK attempts to Create the table from scratch, rather than Update an existing one, and a replica is created successfully. If I update the partition key of the table again, I expect to run into this issue again.

This is 🐛 Bug Report
@tom-noble tom-noble added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 4, 2021
@tom-noble tom-noble changed the title Dynamo DB Replication: No replica created when new table created via Update event (aws-dynamodb): No replica created when new table created via Update event Jan 4, 2021
@github-actions github-actions bot added the @aws-cdk/aws-dynamodb Related to Amazon DynamoDB label Jan 4, 2021
@skinny85 skinny85 added effort/medium Medium work item – several days of effort p1 and removed needs-triage This issue or PR still needs to be triaged. labels Jan 4, 2021
@tom-noble
Copy link
Author

Any updates on this?

jogold added a commit to jogold/aws-cdk that referenced this issue Feb 26, 2021
@jogold
fix(dynamodb): replicas not created on table replacement
adf6195 
Closes aws#12332
jogold added a commit to jogold/aws-cdk that referenced this issue Feb 26, 2021
@jogold
fix(dynamodb): replicas not created on table replacement
e1662ce 
Closes aws#12332
@jogold jogold mentioned this issue Feb 26, 2021
Merged
@mergify mergify bot closed this as completed in #13300 Mar 5, 2021
mergify bot pushed a commit that referenced this issue Mar 5, 2021
@jogold
fix(dynamodb): replicas not created on table replacement (#13300)
c7c424f 
Process `Update` events resulting from table replacements.

Include the table name in the physical resource id to receive a `Delete`
event when the table is replaced. This allows to clean "old" replicas.

Use a managed policy instead of an inline policy for the custom resource.
An update of the description property of a managed policy requires a
replacement. If we use the table name in the description it forces a
managed policy replacement when the table name changes. This way we preserve
permissions to delete old replicas in case of a table replacement: a new
managed policy with permissions for the new table is created during the
update phase and the old managed policy with permissions for the old table is
removed only during the update clean up phase.

The logical ID of the `SourceTableAttachedPolicy` needs to be updated because
CF doesn't allow to change a resource type.

Closes #12332


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Mar 5, 2021

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

cornerwings pushed a commit to cornerwings/aws-cdk that referenced this issue Mar 8, 2021
@jogold @cornerwings
fix(dynamodb): replicas not created on table replacement (aws#13300)
41d15e9 
Process `Update` events resulting from table replacements.

Include the table name in the physical resource id to receive a `Delete`
event when the table is replaced. This allows to clean "old" replicas.

Use a managed policy instead of an inline policy for the custom resource.
An update of the description property of a managed policy requires a
replacement. If we use the table name in the description it forces a
managed policy replacement when the table name changes. This way we preserve
permissions to delete old replicas in case of a table replacement: a new
managed policy with permissions for the new table is created during the
update phase and the old managed policy with permissions for the old table is
removed only during the update clean up phase.

The logical ID of the `SourceTableAttachedPolicy` needs to be updated because
CF doesn't allow to change a resource type.

Closes aws#12332


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@skinny85 skinny85

Labels
@aws-cdk/aws-dynamodb Related to Amazon DynamoDB bug This issue is a bug. effort/medium Medium work item – several days of effort p1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(dynamodb): replicas not created on table replacement jogold/aws-cdk
2 participants
@skinny85 @tom-noble