-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CloudFormation returns Internal Failure when changing existing RDS Instance parameters #6542
Comments
@nija-at I've tried a very minimal configuration for creating RDS using cdk, but I'm getting the same error when trying to introduce a change const vpc = ec2.Vpc.fromLookup(this, "vpc", { vpcId: "vpc-1234567890qwert" });
const dbInstance = new rds.DatabaseInstance(this, "TestingRds", {
engine: rds.DatabaseInstanceEngine.POSTGRES,
engineVersion: "11.6",
instanceClass: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MEDIUM),
masterUsername: "test",
vpc,
}); CDK Diff: ➜ test git:(master) ✗ cdk diff
Stack TestStack
Resources
[~] AWS::RDS::DBInstance TestingRds TestingRdsCD873710
└─ [~] DBInstanceClass
├─ [-] db.t3.small
└─ [+] db.t3.medium CloudFormation Template TestingRdsCD873710:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: db.t3.medium
AllocatedStorage: "100"
CopyTagsToSnapshot: true
DBSubnetGroupName:
Ref: TestingRdsSubnetGroupF14512CF
DeletionProtection: false
Engine: postgres
EngineVersion: "11.6"
MasterUsername:
Fn::Join:
- ""
- - "{{resolve:secretsmanager:"
- Ref: TestingRdsSecretDD42C1F2
- :SecretString:username::}}
MasterUserPassword:
Fn::Join:
- ""
- - "{{resolve:secretsmanager:"
- Ref: TestingRdsSecretDD42C1F2
- :SecretString:password::}}
StorageType: gp2
VPCSecurityGroups:
- Fn::GetAtt:
- TestingRdsSecurityGroup26A56BEF
- GroupId
UpdateReplacePolicy: Retain
DeletionPolicy: Retain
Metadata:
aws:cdk:path: TestStack/TestingRds/Resource Error log: ➜ test git:(master) ✗ cdk deploy '*'
TestStack: deploying...
TestStack: creating CloudFormation changeset...
0/2 | 4:19:31 PM | UPDATE_IN_PROGRESS | AWS::RDS::DBInstance | TestingRds (TestingRdsCD873710)
1/2 | 4:19:32 PM | UPDATE_FAILED | AWS::RDS::DBInstance | TestingRds (TestingRdsCD873710) Internal Failure
new DatabaseInstance (/Users/rdjurasaj/code/rds/test/node_modules/@aws-cdk/aws-rds/lib/instance.ts:795:22)
\_ new TestStack (/Users/rdjurasaj/code/rds/test/lib/test-stack.ts:11:24)
\_ Object.<anonymous> (/Users/rdjurasaj/code/rds/test/bin/test.ts:12:14)
\_ Module._compile (internal/modules/cjs/loader.js:1151:30)
\_ Module.m._compile (/Users/rdjurasaj/code/rds/test/node_modules/ts-node/src/index.ts:814:23)
\_ Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
\_ Object.require.extensions.<computed> [as .ts] (/Users/rdjurasaj/code/rds/test/node_modules/ts-node/src/index.ts:817:12)
\_ Module.load (internal/modules/cjs/loader.js:1000:32)
\_ Function.Module._load (internal/modules/cjs/loader.js:899:14)
\_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
\_ main (/Users/rdjurasaj/code/rds/test/node_modules/ts-node/src/bin.ts:226:14)
\_ Object.<anonymous> (/Users/rdjurasaj/code/rds/test/node_modules/ts-node/src/bin.ts:485:3)
\_ Module._compile (internal/modules/cjs/loader.js:1151:30)
\_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
\_ Module.load (internal/modules/cjs/loader.js:1000:32)
\_ Function.Module._load (internal/modules/cjs/loader.js:899:14)
\_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
\_ /Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/npm/node_modules/libnpx/index.js:268:14
1/2 | 4:19:33 PM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack | TestStack The following resource(s) failed to update: [TestingRdsCD873710].
2/2 | 4:19:47 PM | UPDATE_COMPLETE | AWS::RDS::DBInstance | TestingRds (TestingRdsCD873710)
2/2 | 4:19:48 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | TestStack
3/2 | 4:19:49 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | TestStack
❌ TestStack failed: Error: The stack named TestStack is in a failed state: UPDATE_ROLLBACK_COMPLETE
at /Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:165:13
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at waitFor (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:76:20)
at Object.deployStack (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:155:7)
at CdkToolkit.deploy (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:137:24)
at main (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/bin/cdk.ts:218:16)
at initCommandLine (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/bin/cdk.ts:168:9)
The stack named TestStack is in a failed state: UPDATE_ROLLBACK_COMPLETE |
I apologize but there's nothing the CDK team can do here. The error is coming from RDS' Frontend that is being invoked by CloudFormation on your behalf. One possibility is that the CDK turns on DeletionProtection on an instance by default. Modifying any property that would need the instance to be replaced would be forbidden. Maybe this is what's causing the error. Can you try setting up a brand new stack with Nevertheless, 'Internal Failure' is not an acceptable error message. Continue engaging with AWS support and notify them to reach out to the RDS team. Internal reference: tt/0308387897 |
@nija-at I understand. I'm going to leave this issue open for the AWS Support team's reference. |
@nija-at I'm getting same issue w/ |
Hey @nija-at... @MrArnoldPalmer and I were troubleshooting this on Gitter and I believe we found the possible culprit. We've found out that if I create and update stack in default account VPC then everything works fine. I was able to change instance size from t3.small to t3.medium and deletion protection parameter. ➜ test git:(master) ✗ cdk diff
Stack TestStack
Resources
[~] AWS::RDS::DBInstance TestingRds TestingRdsCD873710
├─ [~] DBInstanceClass
│ ├─ [-] db.t3.small
│ └─ [+] db.t3.medium
└─ [~] DeletionProtection
➜ test git:(master) ✗ cdk deploy
TestStack: deploying...
TestStack: creating CloudFormation changeset...
0/2 | 6:04:01 PM | UPDATE_IN_PROGRESS | AWS::RDS::DBInstance | TestingRds (TestingRdsCD873710)
0/2 Currently in progress: TestingRdsCD873710
1/2 | 6:11:53 PM | UPDATE_COMPLETE | AWS::RDS::DBInstance | TestingRds (TestingRdsCD873710)
1/2 | 6:11:56 PM | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack | TestStack
✅ TestStack
Stack ARN:
arn:aws:cloudformation:us-west-2:xxxxxxxxxxxxx:stack/TestStack/40b420d0-5db2-11ea-beef-06cac24c53a2 However, if I create the RDS stack in a shared VPC (imported from different account due to original VPC running low on IPs) then I'm not able to make any changes to that RDS stack (Internal Failure error). Hopefully this helps AWS team narrow down the issue. |
To add to this, I am also running the Instance in a CDK created VPC:
Same problems (See #6439 ), did not try with default VPC though. |
@jls-tschanzc to clarify a bit, is this VPC in the same account that the DB instances is being created in? I'd encourage you to make a case with AWS support as well if you can since this issues seems to be on the AWS service side and not with CDK. |
Hi @robertd , thanks for the detailed information in how to reproduce the issue. I can confirm there is a known issue on Cloudformation with respect to AWS::RDS::DBInstance being created in a VPC shared from a different AWS Account via RAM. Please follow the link below for updates:
|
Thanks for the update @maiconrocha. |
If it's alright, I'm going to go ahead and close this issue since this isn't related to the CDK. You may continue tracking the issue that @maiconrocha linked above for updates, or with case opened with AWS support. Let me know if I've missed anything here. |
@MrArnoldPalmer @nija-at |
@jls-tschanzc I think it would be the best if you can drop a comment w/ what you're experiencing in aws-cloudformation/cloudformation-coverage-roadmap#373. Also it would be beneficial to create an AWS Support ticket in the account you're experiencing issues in linking to both aws-cdk repo and aws-cloudformation-coverage-roadmap issues. |
@robertd I had hoped the CFN team had a way to report issues/bugs without paying for the privilege to do so; Technical Support Tickets are only available for Support Plans higher than "basic". |
Check #6439 for more details.
AWS Support Case ID: 6844571841
We are having issues applying any kind of changes to an existing RDS stack. We're using aws-cdk in Typescript. AWS Support noted that Python version worked for them running on aws-cdk v1.18.0. There is a strong possibility this may be a CloudFormation issue, but we're looking into figuring out reproduction steps.
aws-cdk team members aware of this issue: @nija-at (through #6439) @MrArnoldPalmer (through Gitter)
aws-cdk code:
Here is our redacted template:
Reproduction Steps
Internal Failure
Error Log
Example of changing
multiAz
parameter:Example of changing
instanceClass
parameter:Environment
Other
This is 🐛 Bug Report
The text was updated successfully, but these errors were encountered: