aws-rds: modifying instance_size for DatabaseCluster results in significant downtime

[ekeyser]

Description

If you have defined your infrastracture using cdk and you implement an rds.DatabaseCluster with multiple members, modifying the instance_props for the members results in 5-10 minutes downtime depending on instance class/size. This defeats the purpose of multi-az (or partner promotion as it's now known I think). Typically you would see 5-10 seconds downtime max if performing this operation otherwise or using best practices.

Use Case

Modify instance_props.instance_type.instance_size to adjust infrastructure to current traffic (up or down). Or if you happen to know of high traffic periods you need to modify your instance size to adjust accordingly.

Proposed Solution

The brute force method would be to move the instance_props outside of the cluster and be handled on an instance by instance case. That way you could deploy the change on one instance, wait for CFN/CDK to finish, then perform the same on remaining instances. Promotion would automatically happen as part of the deployment. This is probably not the right solution and would be too disruptive to the current api.

An alternative would be for CFN/CDK to transparently handle the resize operation internally. I don't have enough knowledge of the inner workings at this point. I'm also not sure if this is related to #10595 (that issue seems to be talking specifically about Cfn constructs and not native CDK constructs). Either way, the solution should be to calculate the graph change or resulting changeset and then perform the intended operation on a step-wise, instance by instance manner only progressing once individual changes are complete/successful or after achieving waypoints (like partner promotion success). This step-wise heuristics could also be abstracted to a standardized class if it were to be reused in other constructs (redshift, elasticache, etc). Additionally, there may also be an addition of a DatabaseCluster property to specify the desired operation, e.g., change_behavior: aws-rds.CHANGESET_BEHAVIOR.PARTNER_PROMOTION or aws-rds.CHANGESET_BEHAVIOR.ONE_AT_A_TIME.

Other information

No response

Acknowledge

• I may be able to implement this feature request
• This feature might incur a breaking change

[ekeyser]

I suspect that this is a CloudFormation change and not something that can be achieved in CDK alone unless there's something I'm not aware of in defining the resulting dependency tree to force a dependency between individual cluster members. Just my guess.

[skinny85]

Hey @ekeyser,

I believe this is exactly what the issue you linked to, #10595, is trying to achieve too.

What do you think about consolidating the discussion there?

Thanks,
Adam

[ekeyser]

That's fine. Is there anything I need to do specifically or would you like me to copy my notes to that issue and close this one? I got the impression #10595 was abandoned. If this is a CloudFormation I suspect there's not a whole lot that can be done as it relates to cdk. Could be wrong.

[skinny85]

I think just commenting on that issue, linking to your description here, is good enough. No need to duplicate it in two places 🙂.

[peterwoodworth]

Closing this out as duplicate

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.