Pulumi tries to replace RDS instance even when no changes are made

[bothra90]

What happened?

pulumi up tried to replace our production RDS instance (Aurora MySQL) even though nothing changed on the surface. but Thankfully it failed because protect flag was set to true, but this could be a catastrophic failure.

Expected Behavior

No updates

Steps to reproduce

I'm not sure how to reproduce this apart from trying to run pulumi up on a pre-existing aurora cluster. If such a cluster exists, just running pulumi up should lead to a repro. Our conclusion is that this is happening because Amazon is associating aurora as the storageType for these clusters and this leads pulumi to believe that it needs to recreate the instance. It's possible that AWS only started doing this recently (in the last 2-3 days).

Output of pulumi about

CLI
Version      3.59.0
Go Version   go1.20.2
Go Compiler  gc

Plugins
NAME        VERSION
aws         5.13.0
command     0.0.3
confluent   0.2.2
docker      3.1.0
eks         0.40.0
kafka       3.1.2
kubernetes  3.21.2
mysql       3.1.0
nodejs      unknown
postgresql  3.4.0

Host
OS       darwin
Version  13.2.1
Arch     arm64

This project is written in nodejs: executable='/nix/store/q8wfjfd61il46wz1bzfrq20zy6shmzvl-nodejs-18.13.0/bin/node' version='v18.13.0'

Backend
Name           pulumi.com
URL            xxxxxxxx
User           xxxxxxxxxx
Organizations  xxxxxxx

Dependencies:
NAME                VERSION
typescript          4.6.2
@pulumi/command     0.0.3
@pulumi/docker      3.1.0
@pulumi/mysql       3.1.0
ts-node             10.9.1
@pulumi/confluent   0.2.2
@pulumi/postgresql  3.4.0
@pulumi/pulumi      3.28.0
uuid                8.3.2
@databases/mysql    5.2.1
@types/node         17.0.21
node-jq             2.3.3
process             0.11.10
@types/uuid         8.3.4
netmask             2.0.2
ts-md5              1.2.11
@pulumi/aws         5.13.0
@pulumi/eks         0.40.0
@pulumi/kafka       3.1.2
@pulumi/kubernetes  3.21.2

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[bothra90]

[kpitzen]

Hi @bothra90 - Thank you for opening this issue. I understand unexpected changes like this can be frustrating, and we'll be looking into this issue as soon as we can. Thank you.

[bothra90]

@kpitzen: FWIW, we see this on our cluster in ap-south-1 (when running aws rds describe), but not us-west-2. It's possible AWS is rolling out a maintenance update that's applied in some regions but not all.

[kpitzen]

Interesting - thank you for the extra context @bothra90 ! I was unable to reproduce the issue in us-west-2:

With this program:

"""An AWS Python Pulumi program"""

import pulumi
from pulumi_aws import rds

# Create an AWS resource (S3 Bucket)
cluster = rds.Cluster(
    'my-rds-cluster',
    args=rds.ClusterArgs(
        engine='aurora-mysql',
        engine_version='5.7.mysql_aurora.2.07.1',
        master_username="foo",
        master_password="barbarbar",
    )
)
# Export the name of the bucket
pulumi.export('cluster_id', cluster.id)

Let me try the same program in ap-south-1

[kpitzen]

@bothra90 Update: I've confirmed that the above program reproduces this issue in ap-south-1:

We'll be looking at this ASAP - appreciate your patience!

[bothra90]

That's awesome. Thanks, @kpitzen!

[kpitzen]

I've also confirmed that running pulumi up followed by pulumi refresh does not update Pulumi state - the subsequent pulumi up will still reproduce this issue.

[rquitales]

From my testing, it appears that this issue seems to only seems to affect the ap-south-1 region currently. I've tried to look into upstream AWS documentation and the upstream provider documentation/issues to see if this is noted, but it looks like this is a new inconsistent behaviour introduced by the AWS RDS API.

What's interesting is that storageType is not supported to be specified and should be unset when creating an Aurora instance. The following error occurs when you attempt to create an aurora-mysql cluster with a non-empty storageType field:

* creating RDS Cluster (aurora-cluster-rquitales): InvalidParameterCombination: StorageType isn't supported for DB engine aurora-mysql.
status code: 400,

Workaround Solution:

As this issue is isolated to the ap-south-1 region, and more specifically, it appears to be an inconsistent issue with the upstream API, a potential workaround for now would be to add the storageType field to the ignoreChanges field in your Pulumi code as documented here: https://www.pulumi.com/docs/intro/concepts/resources/options/ignorechanges/

An example minimal Typescript code:

const rds = new aws.rds.Cluster("default-cluster", {
    backupRetentionPeriod: 5,
    clusterIdentifier: "aurora-cluster-default",
    databaseName: "mydb",
    engine: "aurora-mysql",
    engineVersion: "5.7",
    masterPassword: "bar",
    masterUsername: "foo",
    preferredBackupWindow: "07:00-09:00",
    skipFinalSnapshot: true,
}, { ignoreChanges: ["storageType"] });

This should help unblock usage of Pulumi with aurora RDS instances in the ap-south-1 region while we investigate the next steps to improve the user experience here.

[rquitales]

Note: Downgrading from p1 since this issue only affects a single region, we have a workaround provided, and it appears to be an upstream API issue currently.

[ffMathy]

I am seeing this in eu-west-1 too, with a MariaDB instance in RDS as well.

[arpadgabor]

Seeing this exact behaviour in eu-central-1 as well with Postgres Aurora Serverless V2. Well, I'm using ion.sst.dev but I don't think this issue is on their side. I also checked the state json before/after and there are no changes in the cluster whatsoever, so ignoreChanges does nothing and it still tries to recreate the cluster.

Edit: found the problem. I was only specifying 1 AZ in the cluster definition. It needs 3.