(aws-rds): Cannot add both export and import bucket policies to rds cluster

[jfrconley]

I am trying to add both import and export bucket policies to my RDS cluster. Everything compiles and builds correctly, but when I deploy I get the following error message:

The ed1tokkl9qk0il4 DB instance is not available to manage the arn:aws:iam::***********:role/*******-databaserdsinstanceS3ImportRole91666B-QS8UMWLG99XI IAM role for the feature na
me s3Export feature. Wait for a moment and try again. (Service: AmazonRDS; Status Code: 400; Error Code: InvalidDBInstanceState; Request ID: 2a40ab50-d63e-450c-aad9-472d6d100a53; Proxy:
null)

It seems like it's trying to add both roles at once and the second one is failing.

Reproduction Steps

Create any RDS instance and add both an import and export bucket policy.

db = new DatabaseInstance(this, 'rds-instance', {
        engine: DatabaseInstanceEngine.postgres({
          version: PostgresEngineVersion.VER_11_12,
        }),
        vpc: dbVpc,
        multiAz: false,
        instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.MICRO),
        publiclyAccessible: true,
        vpcSubnets: {
          subnetType: SubnetType.PUBLIC,
        },
        credentials: dbCredentials,
        s3ExportBuckets: [dataExportBucket],
        s3ImportBuckets: [dataImportBucket],
        iamAuthentication: true,
      });

What did you expect to happen?

Both policies should be added

What actually happened?

Stack deploy fails and rollsback with neither added.

Environment

• CDK CLI Version : 1.25.0
• Framework Version: 1.25.0
• Node.js Version: 14.16.0
• OS : Regolith Linux (Ubuntu Base)
• Language (Version): TS 4.4.3

---

This is 🐛 Bug Report

[skinny85]

Hey @jfrconley,

thanks for opening the issue. You're right, I can reproduce your error reliably with the following code:

        const vpc = new ec2.Vpc(this, 'VPC', {
            maxAzs: 2,
            natGateways: 1,
            subnetConfiguration: [
                {
                    name: 'rds',
                    subnetType: ec2.SubnetType.PUBLIC,
                },
            ],
        });
        const instance = new rds.DatabaseInstance(this, 'Instance', {
            vpc,
            vpcSubnets: {
                subnetType: ec2.SubnetType.PUBLIC,
            },
            engine: rds.DatabaseInstanceEngine.postgres({
                version: rds.PostgresEngineVersion.VER_11_12,
            }),
            instanceType: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MICRO),
            multiAz: false,
            publiclyAccessible: true,
            iamAuthentication: true,
            s3ExportBuckets: [new s3.Bucket(this, 'ExportBucket', { removalPolicy: cdk.RemovalPolicy.DESTROY })],
            s3ImportBuckets: [new s3.Bucket(this, 'ImportBucket', { removalPolicy: cdk.RemovalPolicy.DESTROY })],
        });

I wonder what's the problem here... could it be that Postgres version 11.12 does not support either S3 exports or S3 imports?

I don't see what can CDK be doing wrong here, and the error message is also weird, TBH...

[skinny85]

Interestingly, commenting out the s3ExportBuckets line, while leaving the s3ImportBuckets, makes the deployments succeed!

Let me try now with the reverse setup (commenting out s3ImportBuckets, but leaving s3ExportBuckets), and see if that works, or fails.

[skinny85]

Yep, that also worked!

This is crazy. So only when they're both present, does it fail!

Could this be some bug in either the RDS API, or in the RDS CloudFormation API...?

[jfrconley]

This is what I encountered as well, it only seems to fail if both are present. It seems like it's trying to apply both policies at the same time, but RDS has some kind of a processing period that it needs between them. Is there a way that policies can be added as separate resources in cdk? I bet if I added an explicit dependency between the policies it would work.

[skinny85]

Hmm, this is interesting - the CDK actually re-uses the same Role for both the import and export features!

Here's the resulting CloudFormation template after synthesizing the code in #16757 (comment):

{
  "Resources": {
    "InstanceC1063A87": {
      "Type": "AWS::RDS::DBInstance",
      "Properties": {
        "DBInstanceClass": "db.t3.micro",
        "AllocatedStorage": "100",
        "AssociatedRoles": [
          {
            "FeatureName": "s3Import",
            "RoleArn": {
              "Fn::GetAtt": [
                "InstanceS3ImportRole30959D06",
                "Arn"
              ]
            }
          },
          {
            "FeatureName": "s3Export",
            "RoleArn": {
              "Fn::GetAtt": [
                "InstanceS3ImportRole30959D06",
                "Arn"
              ]
            }
          }
        ],
        // ...
      },
    },
    "InstanceS3ImportRole30959D06": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Action": "sts:AssumeRole",
              "Effect": "Allow",
              "Principal": {
                "Service": "rds.amazonaws.com"
              }
            }
          ],
          "Version": "2012-10-17"
        }
      },
      "Metadata": {
        "aws:cdk:path": "RdsDbInstanceStack/Instance/S3ImportRole/Resource"
      }
    },
    "InstanceS3ImportRoleDefaultPolicy297F292A": {
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyDocument": {
          "Statement": [
            {
              "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*"
              ],
              "Effect": "Allow",
              "Resource": [
                {
                  "Fn::GetAtt": [
                    "ImportBucketBAF3A8E9",
                    "Arn"
                  ]
                },
                {
                  "Fn::Join": [
                    "",
                    [
                      {
                        "Fn::GetAtt": [
                          "ImportBucketBAF3A8E9",
                          "Arn"
                        ]
                      },
                      "/*"
                    ]
                  ]
                }
              ]
            },
            {
              "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:DeleteObject*",
                "s3:PutObject*",
                "s3:Abort*"
              ],
              "Effect": "Allow",
              "Resource": [
                {
                  "Fn::GetAtt": [
                    "ExportBucket4E99310E",
                    "Arn"
                  ]
                },
                {
                  "Fn::Join": [
                    "",
                    [
                      {
                        "Fn::GetAtt": [
                          "ExportBucket4E99310E",
                          "Arn"
                        ]
                      },
                      "/*"
                    ]
                  ]
                }
              ]
            }
          ],
          "Version": "2012-10-17"
        },
        "PolicyName": "InstanceS3ImportRoleDefaultPolicy297F292A",
        "Roles": [
          {
            "Ref": "InstanceS3ImportRole30959D06"
          }
        ]
      },
      "Metadata": {
        "aws:cdk:path": "RdsDbInstanceStack/Instance/S3ImportRole/DefaultPolicy/Resource"
      }
    },
  }
}

@jfrconley so do you think adding a DependsOn in the Instance resource on the Policy of that Role will solve this problem?

[jfrconley]

Based on that cloudformation, I would almost think this was a cloudformation bug. It seems like there isn't a way to add roles separately from the top level RDS resource so adding a dependency wouldn't fix it. The error we get back seems to indicate that an update is being applied during an unready state for the instance, so the actual execution of the CF would need to change.

[skinny85]

Yep, I agree.

[jfrconley]

After talking directly with AWS support about the generated cloudformation, it appears that the issue is actually that cdk uses the same policy for both import and export. Essentially, RDS does not allow attaching the same policy twice even when it is given different purposes. This is a bug in CDK and can be fixed by simply keeping the generated policies separate.

[skinny85]

Hmm, that's kind of insane - why would the number of Policies matter? 🙂

I feel like the RDS service is doing something weird here. However, I'm open to changing the implementation here to use multiple policies.

[skinny85]

Interesting... our code contains the following comment:

aws-cdk/packages/@aws-cdk/aws-rds/lib/instance.ts

Line 594 in 370cb31

* This feature is only supported by the Microsoft SQL Server and Oracle engines.

Looks like this is not actually true 🙂.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.