Modifying Performance Insights on an existing RDS instance doesn't work

[robertd]

We are unable to remove "PerformanceInsights" on an existing RDS instance through aws-cdk and CloudFromation. We're getting Internal Failure error. I'm opening an issue here just for visibility as this is not an aws-cdk issue, I believe.

This is somewhat similar to #6542 and I'll be opening an issue on https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap as well.

Edited:
aws-cloudformation/cloudformation-coverage-roadmap#447

Reproduction Steps

• Create an RDS instance with EnablePerformanceInsights set to true
• Try changing it to false

Error Log

Stack dlv-tnmaccess-alphaRDSSnapshotStack
Resources
[~] AWS::RDS::DBInstance dlv-tnmaccess-alpha dlvtnmaccessalpha6F7D7123 
 ├─ [-] EnablePerformanceInsights
 │   └─ true
 └─ [-] PerformanceInsightsRetentionPeriod
     └─ 7
$ cdk deploy '*' --require-approval 'never'
dlv-tnmaccess-alphaRDSSnapshotStack: deploying...
dlv-tnmaccess-alphaRDSSnapshotStack: creating CloudFormation changeset...
 1/2 | 3:47:25 PM | UPDATE_FAILED        | AWS::RDS::DBInstance    | dlv-tnmaccess-alpha (dlvtnmaccessalpha6F7D7123) Internal Failure
	new DatabaseInstanceFromSnapshot (/builds/ngtoc-devops/rds/dlv-tnmaccess-snapshots/node_modules/@aws-cdk/aws-rds/lib/instance.ts:971:22)
	\_ new NgtocRdsSnapshotStack (/builds/ngtoc-devops/rds/dlv-tnmaccess-snapshots/lib/ngtoc-rds-snapshot-stack.ts:36:24)
	\_ /builds/ngtoc-devops/rds/dlv-tnmaccess-snapshots/bin/ngtoc-rds-snapshot.ts:42:17
	\_ Array.forEach (<anonymous>)
	\_ Object.<anonymous> (/builds/ngtoc-devops/rds/dlv-tnmaccess-snapshots/bin/ngtoc-rds-snapshot.ts:25:12)
	\_ Module._compile (internal/modules/cjs/loader.js:1147:30)
	\_ Module.m._compile (/builds/ngtoc-devops/rds/dlv-tnmaccess-snapshots/node_modules/ts-node/src/index.ts:836:23)
	\_ Module._extensions..js (internal/modules/cjs/loader.js:1167:10)
	\_ Object.require.extensions.<computed> [as .ts] (/builds/ngtoc-devops/rds/dlv-tnmaccess-snapshots/node_modules/ts-node/src/index.ts:839:12)
	\_ Module.load (internal/modules/cjs/loader.js:996:32)
	\_ Function.Module._load (internal/modules/cjs/loader.js:896:14)
	\_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
	\_ main (/builds/ngtoc-devops/rds/dlv-tnmaccess-snapshots/node_modules/ts-node/src/bin.ts:226:14)
	\_ Object.<anonymous> (/builds/ngtoc-devops/rds/dlv-tnmaccess-snapshots/node_modules/ts-node/src/bin.ts:485:3)
	\_ Module._compile (internal/modules/cjs/loader.js:1147:30)
	\_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1167:10)
	\_ Module.load (internal/modules/cjs/loader.js:996:32)
	\_ Function.Module._load (internal/modules/cjs/loader.js:896:14)
	\_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
	\_ /usr/local/lib/node_modules/npm/node_modules/libnpx/index.js:268:14
 1/2 | 3:47:25 PM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack | dlv-tnmaccess-alphaRDSSnapshotStack The following resource(s) failed to update: [dlvtnmaccessalpha6F7D7123]. 
 2/2 | 3:47:30 PM | UPDATE_COMPLETE      | AWS::RDS::DBInstance    | dlv-tnmaccess-alpha (dlvtnmaccessalpha6F7D7123) 
 2/2 | 3:47:30 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | dlv-tnmaccess-alphaRDSSnapshotStack 
 3/2 | 3:47:31 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack | dlv-tnmaccess-alphaRDSSnapshotStack 
 ❌  dlv-tnmaccess-alphaRDSSnapshotStack failed: Error: The stack named dlv-tnmaccess-alphaRDSSnapshotStack is in a failed state: UPDATE_ROLLBACK_COMPLETE
    at /usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:240:13
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at waitFor (/usr/local/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:150:20)
    at Object.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:230:26)
    at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:179:24)
    at main (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:229:16)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:167:9)
The stack named dlv-tnmaccess-alphaRDSSnapshotStack is in a failed state: UPDATE_ROLLBACK_COMPLETE
ERROR: Job failed: exit code 1

Environment

• Framework Version: 1.32.2
• OS : Mac/Linux
• Language : typescript

Other

---

This is 🐛 Bug Report

[robertd]

I was able to turn off Performance Insights in two steps.

1. Removed Performance Insights by setting it explicitly to false, and then allowing aws-cdk to push the change trough CloudFormation.

Stack dlv-tnmaccess-alphaRDSSnapshotStack
 Resources
 [~] AWS::RDS::DBInstance dlv-tnmaccess-alpha dlvtnmaccessalpha6F7D7123 
  ├─ [~] EnablePerformanceInsights
  └─ [-] PerformanceInsightsRetentionPeriod
      └─ 7
 $ cdk deploy '*' --require-approval 'never'
 dlv-tnmaccess-alphaRDSSnapshotStack: deploying...
 dlv-tnmaccess-alphaRDSSnapshotStack: creating CloudFormation changeset...
  0/2 | 9:27:21 PM | UPDATE_IN_PROGRESS   | AWS::RDS::DBInstance    | dlv-tnmaccess-alpha (dlvtnmaccessalpha6F7D7123) 
 0/2 Currently in progress: dlvtnmaccessalpha6F7D7123
  ✅  dlv-tnmaccess-alphaRDSSnapshotStack
 Outputs:
 dlv-tnmaccess-alphaRDSSnapshotStack.dlvtnmaccessalphadnsendpoint = dlv-tnmaccess-alpha.cas6rxqzgkd9.us-west-2.rds.amazonaws.com
 Stack ARN:
 arn:aws:cloudformation:us-west-2:228513568230:stack/dlv-tnmaccess-alphaRDSSnapshotStack/2832c890-7e7d-11ea-99e5-06969188ef6e
 Job succeeded

2. Removed Performance Insights completely and then pushed aws-cdk change without the parameter present.

Stack dlv-tnmaccess-bravoRDSSnapshotStack
 Resources
 [~] AWS::RDS::DBInstance dlv-tnmaccess-bravo dlvtnmaccessbravo1E585176 
  └─ [-] EnablePerformanceInsights
      └─ false
 $ cdk deploy '*' --require-approval 'never'
 dlv-tnmaccess-bravoRDSSnapshotStack: deploying...
 dlv-tnmaccess-bravoRDSSnapshotStack: creating CloudFormation changeset...
  0/2 | 9:34:30 PM | UPDATE_IN_PROGRESS   | AWS::RDS::DBInstance    | dlv-tnmaccess-bravo (dlvtnmaccessbravo1E585176) 
 0/2 Currently in progress: dlvtnmaccessbravo1E585176
  1/2 | 9:35:47 PM | UPDATE_COMPLETE      | AWS::RDS::DBInstance    | dlv-tnmaccess-bravo (dlvtnmaccessbravo1E585176) 
  ✅  dlv-tnmaccess-bravoRDSSnapshotStack
 Outputs:
 dlv-tnmaccess-bravoRDSSnapshotStack.dlvtnmaccessbravodnsendpoint = dlv-tnmaccess-bravo.cbopwmis6alt.us-west-2.rds.amazonaws.com
 Stack ARN:
 arn:aws:cloudformation:us-west-2:746822052750:stack/dlv-tnmaccess-bravoRDSSnapshotStack/3ebfbbd0-7514-11ea-9c05-02f90cd818d0
 Job succeeded

I still believe this should be handled by CloudFormation so if you can pass that to the internal CFN team that'd be great. I'm still going to leave github issue open for awareness.

[nija-at]

Unfortunately, there's nothing we can do here from the CDK side.

You have done the correct thing of opening an issue with cloudformation. Feel free to reach out to AWS support as well if you're still blocked on this.

We'll have to close this issue.