Report: aws_rds_cluster diffs didn't match during apply

[jwatson0]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.8

• provider.aws v1.37.0

Affected Resource(s)

aws_rds_cluster
aws_rds_cluster_parameter_group

Terraform Configuration Files

Encrypted: https://gist.github.com/jwatson0/72707e13fcc1897baac5e9726c8abd18

Expected Behavior

Delete and recreate aws_rds_cluster and aws_rds_cluster_parameter_group

Actual Behavior

Error: Error applying plan:

2 error(s) occurred:

• module.jira.aws_rds_cluster_parameter_group.jira-db (destroy): 1 error(s) occurred:

• aws_rds_cluster_parameter_group.jira-db: InvalidDBParameterGroupState: One or more database instances are still members of this parameter group test-core-jira-db, so the group cannot be deleted
  status code: 400, request id: 00000000-0000-0000-0000-000000000000

• module.jira.aws_rds_cluster.jira-db: aws_rds_cluster.jira-db: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Full output encrypted: https://gist.github.com/jwatson0/e43fdede5a9b6ed1aa4ed7a2bad74bd8

Steps to Reproduce

1. Apply initial config with engine = "aurora", engine_mode = "serverless"
2. terraform apply
3. Change to engine = "aurora-postgresql", engine_version = "9.6", engine_mode = "provisioned"
4. terraform apply

[bflad]

Hi folks 👋

Sorry you ran into this confusing error and thank you for reporting it. There have been a lot of relevant changes in the Terraform Core and Terraform AWS Provider codebases since this issue was originally reported so as a first step it might be best to try updating to recent versions of both and seeing if any of those updates may have resolved this issue.

For example, I tried a full version of the original issue configuration and was not able to reproduce the original issue on Terraform CLI 0.12.13 and Terraform AWS Provider 2.34.0 following the reproduction steps.

Sanitized apply output:

$ terraform apply
...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_db_parameter_group.xxx must be replaced
-/+ resource "aws_db_parameter_group" "xxx" {
      ~ arn         = "arn:aws:rds:us-east-2:--OMITTED--:pg:bflad-testing-xxx-params" -> (known after apply)
        description = "xxx parameter group in env bflad-testing"
      ~ family      = "aurora5.6" -> "aurora-postgresql9.6" # forces replacement
      ~ id          = "bflad-testing-xxx-params" -> (known after apply)
        name        = "bflad-testing-xxx-params"
      + name_prefix = (known after apply)
      - tags        = {} -> null
    }

  # aws_rds_cluster.xxx must be replaced
-/+ resource "aws_rds_cluster" "xxx" {
        apply_immediately                   = true
      ~ arn                                 = "arn:aws:rds:us-east-2:--OMITTED--:cluster:bflad-testing-xxx" -> (known after apply)
      ~ availability_zones                  = [
          - "us-east-2a",
          - "us-east-2b",
          - "us-east-2c",
        ] -> (known after apply)
      - backtrack_window                    = 0 -> null
        backup_retention_period             = 1
        cluster_identifier                  = "bflad-testing-xxx"
      + cluster_identifier_prefix           = (known after apply)
      ~ cluster_members                     = [] -> (known after apply)
      ~ cluster_resource_id                 = "cluster-S4BPTAMNGOG5OKI3U6F6EKXJNI" -> (known after apply)
        copy_tags_to_snapshot               = false
      + database_name                       = (known after apply)
        db_cluster_parameter_group_name     = "bflad-testing-xxx-clusterparams"
        db_subnet_group_name                = "bflad-testing-xxx"
      - deletion_protection                 = false -> null
      - enabled_cloudwatch_logs_exports     = [] -> null
      ~ endpoint                            = "bflad-testing-xxx.cluster-cz7jckt4fsmo.us-east-2.rds.amazonaws.com" -> (known after apply)
      ~ engine                              = "aurora" -> "aurora-postgresql" # forces replacement
      ~ engine_mode                         = "serverless" -> "provisioned" # forces replacement
      ~ engine_version                      = "5.6.10a" -> "9.6"
        final_snapshot_identifier           = "snapshot-bflad-testing-xxx-final-q67dfd6z"
      ~ hosted_zone_id                      = "Z2XHWR1WZ565X2" -> (known after apply)
      - iam_database_authentication_enabled = false -> null
      - iam_roles                           = [] -> null
      ~ id                                  = "bflad-testing-xxx" -> (known after apply)
        kms_key_id                          = "arn:aws:kms:us-east-2:--OMITTED--:key/b159df48-fe1a-43a5-a345-49299ee40b7f"
        master_password                     = (sensitive value)
        master_username                     = "master"
      ~ port                                = 3306 -> (known after apply)
        preferred_backup_window             = "04:00-06:00"
        preferred_maintenance_window        = "thu:06:00-thu:06:30"
      + reader_endpoint                     = (known after apply)
        skip_final_snapshot                 = false
        storage_encrypted                   = true
        tags                                = {
            "Environment" = "bflad-testing"
            "Name"        = "bflad-testing-xxx"
            "Terraform"   = "true"
        }
        vpc_security_group_ids              = [
            "sg-00ba2af75f2a723d7",
        ]

      - scaling_configuration {
          - auto_pause               = true -> null
          - max_capacity             = 64 -> null
          - min_capacity             = 2 -> null
          - seconds_until_auto_pause = 300 -> null
          - timeout_action           = "RollbackCapacityChange" -> null
        }
    }

  # aws_rds_cluster_instance.cluster_instances[0] will be created
  + resource "aws_rds_cluster_instance" "cluster_instances" {
      + apply_immediately               = (known after apply)
      + arn                             = (known after apply)
      + auto_minor_version_upgrade      = true
      + availability_zone               = (known after apply)
      + cluster_identifier              = (known after apply)
      + copy_tags_to_snapshot           = false
      + db_parameter_group_name         = "bflad-testing-xxx-params"
      + db_subnet_group_name            = "bflad-testing-xxx"
      + dbi_resource_id                 = (known after apply)
      + endpoint                        = (known after apply)
      + engine                          = "aurora"
      + engine_version                  = (known after apply)
      + id                              = (known after apply)
      + identifier                      = "bflad-testing-xxx-0"
      + identifier_prefix               = (known after apply)
      + instance_class                  = "db.r4.large"
      + kms_key_id                      = (known after apply)
      + monitoring_interval             = 0
      + monitoring_role_arn             = (known after apply)
      + performance_insights_enabled    = (known after apply)
      + performance_insights_kms_key_id = (known after apply)
      + port                            = (known after apply)
      + preferred_backup_window         = "04:00-06:00"
      + preferred_maintenance_window    = "thu:06:00-thu:06:30"
      + promotion_tier                  = 0
      + publicly_accessible             = false
      + storage_encrypted               = (known after apply)
      + writer                          = (known after apply)
    }

  # aws_rds_cluster_parameter_group.xxx-cluster must be replaced
-/+ resource "aws_rds_cluster_parameter_group" "xxx-cluster" {
      ~ arn         = "arn:aws:rds:us-east-2:--OMITTED--:cluster-pg:bflad-testing-xxx-clusterparams" -> (known after apply)
        description = "xxx cluster parameter group in env bflad-testing"
      ~ family      = "aurora5.6" -> "aurora-postgresql9.6" # forces replacement
      ~ id          = "bflad-testing-xxx-clusterparams" -> (known after apply)
        name        = "bflad-testing-xxx-clusterparams"
      + name_prefix = (known after apply)
      - tags        = {} -> null
    }

Plan: 4 to add, 0 to change, 3 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_db_parameter_group.xxx: Destroying... [id=bflad-testing-xxx-params]
aws_rds_cluster.xxx: Destroying... [id=bflad-testing-xxx]
aws_db_parameter_group.xxx: Destruction complete after 0s
aws_db_parameter_group.xxx: Creating...
aws_db_parameter_group.xxx: Creation complete after 1s [id=bflad-testing-xxx-params]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 10s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 20s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 30s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 40s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 50s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 1m0s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 1m10s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 1m20s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 1m30s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 1m40s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 1m50s elapsed]
aws_rds_cluster.xxx: Still destroying... [id=bflad-testing-xxx, 2m0s elapsed]
aws_rds_cluster.xxx: Destruction complete after 2m8s
aws_rds_cluster_parameter_group.xxx-cluster: Destroying... [id=bflad-testing-xxx-clusterparams]
aws_rds_cluster_parameter_group.xxx-cluster: Destruction complete after 0s
aws_rds_cluster_parameter_group.xxx-cluster: Creating...
aws_rds_cluster_parameter_group.xxx-cluster: Creation complete after 1s [id=bflad-testing-xxx-clusterparams]
aws_rds_cluster.xxx: Creating...
aws_rds_cluster.xxx: Still creating... [10s elapsed]
aws_rds_cluster.xxx: Still creating... [20s elapsed]
aws_rds_cluster.xxx: Still creating... [30s elapsed]
aws_rds_cluster.xxx: Still creating... [40s elapsed]
aws_rds_cluster.xxx: Still creating... [50s elapsed]
aws_rds_cluster.xxx: Still creating... [1m0s elapsed]
aws_rds_cluster.xxx: Creation complete after 1m2s [id=bflad-testing-xxx]
aws_rds_cluster_instance.cluster_instances[0]: Creating...

My guess here is that the Terraform CLI 0.12 upgrade had a fix for the underlying issue.

Attempting to troubleshoot these types of errors is very difficult in Terraform 0.11 and earlier as those versions will report this problem (diffs didn't match during apply error) where a problematic configuration value exists or is referenced and not where the Terraform resource with the underlying problem actually exists. We typically will need much more information than the original error message suggests for filing the issue (e.g. the full Terraform configuration). There are also frequent issues in the Terraform 0.11 error reporting where the error message is due to behavior outside the control of the Terraform resources themselves (e.g. lifecycle configurations).

In Terraform 0.12 and later, the "value did not match" type of error can now be found with the newer Provider produced inconsistent result after apply error, which better reports the source of the problem so the provider maintainers and community can more easily work towards a fix of the issue. Other similar errors that fell previously into diffs didn't match during apply errors, but outside problems within the Terraform provider are also better diagnosed before being displayed.

Since the majority of the Terraform code path and reporting structure of these errors has changed between Terraform 0.11 and 0.12, along with the difficult triage process in Terraform 0.11 and earlier, the maintainers prefer to close these older issues to continue troubleshooting and fixing based on Terraform 0.12's error reporting instead. I apologize for this unsatisfying closure of this particular version of the issue.

That said, if the configuration does happen to reproduce the newer Provider produced inconsistent result after apply error in Terraform 0.12 (which requires Terraform AWS Provider version 2.7.0 or later), please do not hesitate to file a new issue and complete all of the information requested in the Bug Report template so we can hopefully get to the root cause for fixing the issue. Thanks again for this bug report and sorry about potential additional steps.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!