RDS Cluster cannot be deleted

[ghost]

This issue was originally opened by @sriraja53 as hashicorp/terraform#20929. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Hi there,

I am creating RDS global cluster using Terraform aws_rds_global_cluster resource.
Reference : https://www.terraform.io/docs/providers/aws/r/rds_global_cluster.html

I am able to create RDS global cluster without any issue but while destroying the RDS global cluster I am facing below issue.

Error :
aws_rds_cluster.secondary: RDS Cluster cannot be deleted: InvalidDBClusterStateFault: This cluster is a part of a global cluster, please remove it from globalcluster first

After waiting some time (10 mins) rerunning the terraform destroy command it will delete the cluster without any issue.

Terraform Version

Terraform v0.11.10
provider.aws: version = "~> 1.54"

Configuration

provider "aws" {
  alias  = "primary"
  region = "us-east-2"
}

provider "aws" {
  alias  = "secondary"
  region = "us-west-2"
}

resource "aws_rds_global_cluster" "example" {
  provider = "aws.primary"

  global_cluster_identifier = "example"
}

resource "aws_rds_cluster" "primary" {
  provider = "aws.primary"

  # ... other configuration ...
  engine_mode               = "global"
  global_cluster_identifier = "${aws_rds_global_cluster.example.id}"
}

resource "aws_rds_cluster_instance" "primary" {
  provider = "aws.primary"

  # ... other configuration ...
  cluster_identifier = "${aws_rds_cluster.primary.id}"
}

resource "aws_rds_cluster" "secondary" {
  depends_on = ["aws_rds_cluster_instance.primary"]
  provider   = "aws.secondary"

  # ... other configuration ...
  engine_mode               = "global"
  global_cluster_identifier = "${aws_rds_global_cluster.example.id}"
}

resource "aws_rds_cluster_instance" "secondary" {
  provider = "aws.secondary"

  # ... other configuration ...
  cluster_identifier = "${aws_rds_cluster.secondary.id}"
}

[jgreat]

I'm also hitting this issue. For some reason global_cluster_identifier on my aws_rds_cluster resource ends up empty in the state after creation.

If I do a plan after my initial apply the aws_rds_cluster resource shows changed.

I'm not super familiar with creating Terraform Providers, but I think this is due to the resource assuming that the engine_mode will be "global" to save the setting here. https://github.com/terraform-providers/terraform-provider-aws/blob/94d06427f51d65d2e5776e3a26e4308fe2ad47b2/aws/resource_aws_rds_cluster.go#L1049

Just to be super confusing, "global" seems to only be a valid engine_mode on some limited "aurora" engine versions. For aurora-postgresql the only valid engine_mode is "provisioned".

Just for reference I went spelunking for valid engine types for various database engine versions with the aws cli:

# all versions for aurora-postgresql
aws describe-db-engine-versions --engine aurora-postgresql

# all versions that allow engine_mode "global"
aws describe-db-engine-versions | jq -r '.DBEngineVersions[] | select(.SupportedEngineModes[]? | contains("global"))' 

➜ terraform version
Terraform v0.12.24
+ provider.aws v2.58.0

[jgreat]

Ha, looks like this was fixed with yesterday's 2.59.0 release. #12867

Is my timing good or what 😄

[marinsalinas]

@jgreat does your secondary cluster is deleted? on the new release?

[callaingit]

Hi

I run TF 0.13.4 and the aws provider v3.11.0 and I have a similar issue. I get:

Error: error removing RDS DB Cluster (arn:aws:rds:REGION:ACCOUNT:cluster:SECONDARYCLUSTERNAME) from Global Cluster (GLOBALCLUSTERNAME): InvalidParameterValue: Cluster arn:aws:rds:REGION:ACCOUNT:cluster:SECONDARYCLUSTERNAME is not found in global cluster GLOBALCLUSTERNAME

After successfully creating a global cluster (from a snapshot, using the method found in the section "New Global Cluster from Existing DB Cluster) on https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_global_cluster, this error shows up. In the console, I see the secondary cluster has been deleted so it looks like if TF or the provider is trying to remove the secondary cluster from the global cluster after the secondary cluster has been deleted.

I have force_destroy = true and source_db_cluster_identifier set on my aws_rds_global_cluster.
I have the following on both my aws_rds_cluster (primary and secondary, both are created from a common module and lifecycle blocks cannot be added or removed dynamically as far as I know)
lifecycle {
ignore_changes = [global_cluster_identifier, replication_source_identifier, snapshot_identifier]
}

Note that global_cluster_identifier is prescribed as per https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_global_cluster when creating the global cluster from a primary cluster that is initialized from a snapshot at creation.

I added replication_source_identifier as it is subject to constant drift on the secondary cluster - see https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_cluster#replication_source_identifier for that rationale

I also added snapshot_identifier as the initial snapshot to use on creation of my primary cluster is passed only at the initial apply and subsequent apply have it blank butI don't think that matters here as we're talking destroying the global cluster.

My secondary cluster is dependent (depends_on) on the global cluster because on creation, we need the global cluster being created first before creating the secondary cluster.

My global cluster depends on the primary cluster as it is using it to set source_db_cluster_identifier (I still have a depends_on the module that creates the primary cluster as the aws_rds_cluster creation is hidden in there). Everything goes fine on creation.

Running terraform destroy a second time destroys everything, but this is a pain to check why if fails everytime and run it twice if that is the usual error (difficult to automate).

[callaingit]

When I look at the code here https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_rds_global_cluster.go#L233 it looks to me that the global_cluster_members should somehow be updated when a secondary cluster is deleted. Otherwise, I can't see how we can have aws_rds_cluster depends on aws_rds_global_cluster on creation for creation order (so the secondary cluster is created after the global cluster is created) and expect for global cluster members to hold true - in memory I mean - when the global cluster is deleted after the secondary cluster is deleted). Or else maybe the logic could be adapted so it does not try to remove secondary clusters from the global cluster, just the primary one (as I think this is why force_destroy was introduced)?

[callaingit]

@bflad , I know you helped fixing stuff and are aware of the introduction of the force_destroy flag, can you provide us with your insights regarding this one? Same for @anGie44. Thank in advance.

[callaingit]

Looks to me like the code here: https://github.com/terraform-providers/terraform-provider-aws/blob/v3.11.0/aws/resource_aws_rds_cluster.go#L1170 in the resourceAwsRDSClusterDelete function would be removing the secondary cluster from the global cluster as part of the destroy on the aws_rds_cluster.

[callaingit]

More input.

I tried not putting force_destroy at all and got this error:
"source_db_cluster_identifier": all of force_destroy,source_db_cluster_identifier must be specified

I tried putting force_destroy = false and got this error:
Error: error deleting RDS Global Cluster: InvalidGlobalClusterStateFault: Global Cluster arn:aws:rds::ACCOUNT:global-cluster:GLOBALCLUSTERNAMEis not empty

So to me, really, there seems to be a conflict in the current code base when we use the proposed setup where the primary cluster gets created, the global cluster gets created (from the primary cluster) and then the secondary cluster is created (referencing the global cluster) on a destroy - all that for the global cluster to be initialized from a primary cluster loaded with an initial snapshot based on my understanding (this is what I've been trying for 2 days now ;-))

I think that the code that deletes the cluster members based on force_destroy = true should concentrate on the primary (maybe we need an extra parameter just for that to preserve existing behavior of deleting all members from the global cluster which may be ok in other use cases)

The existing global_cluster_members attribute of the global cluster happens to have is_writer, maybe it is ok then to just delete where is_writer = true (I am not sure what would happen in case the writer changes from one cluster to another, but if that is not done via TF, I don't know how TF can handle that anyway right now)

global_cluster_members       = [
    {
        db_cluster_arn = "arn:aws:rds:REGION:ACCOUNT:cluster:PRIMARYCLUSTERNAME"
        is_writer      = true
    },
    {
        db_cluster_arn = "arn:aws:rds:REGION:ACCOUNT:cluster:SECONDARYCLUSTERNAME"
        is_writer      = false
    },
]

[callaingit]

More thoughts. In fact, we can think about the issue in terms of idempotency. If the force_destroy = true attempts to delete any cluster member and gets that error I got (which says basically that there is no such global cluster member), why just not keep going instead of failing the run as the end result of getting rid of that global cluster membership has already been achieved as a side-effect somewhere else in the code. I think that would help and would not introduce too much noise in something that is already pretty complicated.

The code could be also checking that a member in the global list is still a member in the real infrastructure by querying AWS before making the membership removal code. That would probably be more robust than using the end part of a text message in an error (if there is no easy way to pinpoint that error we get).

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.