[aws-dynamodb] Fail to create a global table due to replication time-out

[sungbokang]

Hi,

I'm trying to add one more DynamoDB global table(replica) in us-west-2 region.

But it never succeeded in creating the replica. replica-provider currently uses a default totalTimeout(30 minutes) value of CustomResourceProvider so that it always fails if expected replication time is more than 30 minutes: https://github.com/aws/aws-cdk/blob/master/packages/@aws-cdk/aws-dynamodb/lib/replica-provider.ts#L74-L77

Reproduction Steps

First, my DynamoDB constructor is like this, which tells that it already had us-east-1 and eu-west-1 regions.

import { Construct } from '@aws-cdk/core';
import {
    AttributeType,
    BillingMode,
    Table,
    StreamViewType
} from '@aws-cdk/aws-dynamodb';

export class TestTable extends Table {

    static readonly TABLE_NAME = 'test'

    constructor(parent: Construct, currentRegion: string) {
        super(parent, 'test', {
            tableName: test.TABLE_NAME,
            partitionKey: {
                name: 'key',
                type: AttributeType.STRING
            },
            sortKey: {
                name: 'rank',
                type: AttributeType.NUMBER
            },
            timeToLiveAttribute: 'ttl',
            billingMode: BillingMode.PAY_PER_REQUEST,
            stream: StreamViewType.NEW_AND_OLD_IMAGES,
            replicationRegions: ['us-east-1', 'eu-west-1']
                .filter(region => region.toLowerCase() !== currentRegion.toLowerCase())
        });
    }
}

And then create a bunch of data enough to make a replication process takes more than 30 minutes. (For our case, we have 20Gb data in our table approximately)

After that, append us-west-2 region in order to add the global table dynamically.

replicationRegions: ['us-east-1', 'eu-west-1', `us-west-2`]

What did you expect to happen?

The global table in us-west-2 region should have been created well.

What actually happened?

While creating a global table, CloudFormation threw CREATE_FAILED with a Failed to create resource. Operation timed out message. As trying to create the global table several times, this happens in 33 minutes exactly after the time when Custom::DynamoDBReplica becomes ``CREATE_IN_PROGRESS` status.

Environment

• CLI Version : 1.32
• Framework Version:
• Node.js Version: NodeJS12.x
• OS : Amazon Linux 2012.03
• Language (Version): Typescript 3.9.2

Other

• Suggestion on how to fix : I'd suggest to fix this by allowing clients to inject customised totalTimeout value into replica-provider instead of sticking to the default 30 minutes value.
• Related issues : After the replica failure happens, Rollback procedure happens but it also fails. Because a IAM role of OnEventHandler reverts before deleting the global table. So deleting the global table fails with a DELETE_FAILED - Failed to delete resource because of lack of permission. Regarding this, I created one another issue: [aws-dynamodb] Fail to rollback if global table creation is failed #10256

---

This is 🐛 Bug Report

[skinny85]

Hey @sungbokang ,

thanks for opening the issue.

Do I understand correctly that your ask is basically to surface the totalTimeout property of the custom resource provider when creating the replication tables in DynamoDB?

Thanks,
Adam

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.