Aurora Postgres Engine Upgrade is not respecting apply_immediately = false

[forestoden]

I was trying to schedule an Aurora Postgres engine upgrade from 10.7 to 10.11 for the next maintenance window. I have explicit apply_immediately = false in the module. However when I went to apply the changes, it just immediately tried to update and actually failed.

The change I made to attempt to schedule the upgrade was engine_version = "10.7" to engine_version = "10.11"

Here's a snippet of the plan with most of the irrelevant information redacted

  # module.vault_db.aws_rds_cluster.this will be updated in-place
  ~ resource "aws_rds_cluster" "this" {
        apply_immediately                   = false
        engine                              = "aurora-postgresql"
        engine_mode                         = "provisioned"
      ~ engine_version                      = "10.7" -> "10.11"

        preferred_maintenance_window        = "sun:05:00-sun:06:00"

    }

  # module.vault_db.aws_rds_cluster_instance.this[0] must be replaced
-/+ resource "aws_rds_cluster_instance" "this" {
        apply_immediately               = false
        auto_minor_version_upgrade      = true
        engine                          = "aurora-postgresql"
      ~ engine_version                  = "10.7" -> "10.11" # forces replacement
        preferred_maintenance_window    = "sun:05:00-sun:06:00"

    }

I was hoping that the warning about the DB Instance being replaced was a false alarm, and that it would just be replaced during the maintenance window, but it immediately started to delete the instance. What's worse is that it even failed to upgrade the cluster. It deleted the two DB Instances but then couldn't upgrade the cluster itself because no instances were running.

module.vault_db.aws_rds_cluster_instance.this[1]: Destroying... [id=redacted-2]
module.vault_db.aws_rds_cluster_instance.this[0]: Destroying... [id=redacted-1]
module.vault_db.aws_rds_cluster.this: Modifying... [id=redacted]
module.vault_db.aws_rds_cluster_instance.this[1]: Still destroying... [id=redacted-2, 10s elapsed]
module.vault_db.aws_rds_cluster_instance.this[0]: Still destroying... [id=redacted-1, 10s elapsed]
module.vault_db.aws_rds_cluster.this: Still modifying... [id=redacted, 10s elapsed]
module.vault_db.aws_rds_cluster_instance.this[1]: Still destroying... [id=redacted-2, 20s elapsed]
module.vault_db.aws_rds_cluster_instance.this[0]: Still destroying... [id=redacted-1, 20s elapsed]
module.vault_db.aws_rds_cluster.this: Still modifying... [id=redacted, 20s elapsed]
module.vault_db.aws_rds_cluster_instance.this[1]: Still destroying... [id=redacted-2, 30s elapsed]
module.vault_db.aws_rds_cluster_instance.this[0]: Still destroying... [id=redacted-1, 30s elapsed]
module.vault_db.aws_rds_cluster.this: Still modifying... [id=redacted, 30s elapsed]
module.vault_db.aws_rds_cluster.this: Modifications complete after 33s [id=redacted]
...
Error: error creating RDS DB Instance: InvalidParameterCombination: The engine version that you requested for your DB instance (10.11) does not match the engine version of your DB cluster (10.7).
        status code: 400, request id: 8cce8f94-7ba3-4dc6-97b0-610d177f6e27

  on .terraform/modules/vault_db/main.tf line 81, in resource "aws_rds_cluster_instance" "this":
  81: resource "aws_rds_cluster_instance" "this" {

I'm not sure if this is user error, a bug in the module, or in the provider. I searched in both this module and the provider for anything relating to this, but couldn't find anything.

[cembas]

I'm currently seeing the exact same problem as you have in your output. I cannot upgrade from 10.7 to 10.11 as well. It seems rds_engine_version doesn't get applied to the cluster configuration.

[komljen]

@cembas You can specify apply_immediately = true if you want to upgrade right away, but keep in mind to only upgrade cluster, and not instances. You can use target option:

terraform apply -target=module.vault_db.aws_rds_cluster.this

This module should probably be fixed to ignore aws_rds_cluster_instance.this[*] on engine upgrades because there is no need to replace them. Upgrading cluster upgrades instances as well.

[komljen]

@forestoden apply_immediately = false is working, but as I noted above, the issue is that module updates instances (replaces them), and it should just ignore it and update engine version on the cluster resource.

As a workaround, you can use the above command with target option.

[antonbabenko]

This has been fixed in #134, please update your code to use v2.27.0 of this module.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.