resource aws_db_parameter_group has continuous drift since AWS provider 3.55.0

[r-kok]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.14.11
AWS provider 3.55.0

Affected Resource(s)

• aws_db_parameter_group

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Debug Output

Panic Output

Expected Behavior

since we had no drift using provider version 3.54, we expected no drift with version 3.55 either.

Actual Behavior

A terraform plan using AWS provider 3.55 generates the output shown below.
When we terraform apply these changes, and run terraform plan a second time, we get the same output again.
Thus the changes were not applied as expected.

  # module.pcs.module.oracle.aws_db_parameter_group.oracle will be updated in-place
  ~ resource "aws_db_parameter_group" "oracle" {
        id          = "kt-pcs-oracle"
        name        = "kt-pcs-oracle"
        tags        = {
            "Environment" = "kt"
        }
        # (4 unchanged attributes hidden)

      + parameter {
          + apply_method = "immediate"
          + name         = "open_cursors"
          + value        = "300"
        }
      - parameter {
          - apply_method = "pending-reboot" -> null
          - name         = "open_cursors" -> null
          - value        = "300" -> null
        }
        # (27 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Steps to Reproduce

1. terraform plan : shows terraform drift
2. terraform apply -auto-approve: apply seems succesful.
3. terraform plan : shows same terraform drift as if the apply failed (UNEXPECTED)

Important Factoids

We use the parameter group in the context of an Oracle RDS database.

As a workaround we have temporarily pinned the version of the AWS provider in a versions.tf file:

terraform {
  required_version = ">= 0.14"
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "3.54.0"
    }
  }

References

We suspect it is related to or caused by this item in the AWS provider changelog:

• resource/aws_db_parameter_group: Allow parameter values to be mixed case, prioritize certain parameters when chunking, and avoid diffs with mixed-case parameter names (r/db_parameter_group: Allow mixed case parameter values #18818)

• #0000

[sshishov]

@r-kok , you have a typo in the title. The version indicated in the title should be 3.55.0. Please correct.

[antdking]

Confirming I get this on the latest versions:

$ terraform version
Terraform v1.0.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.59.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0

[r-kok]

Today I had some time to reproduce the issue.
The following terraform file shows a regression since provider 3.55.0:

resource "aws_db_parameter_group" "attempt4" {
  name = "attempt4"
  family = "oracle-ee-12.1"

  parameter {
    apply_method = "immediate"
    name         = "open_cursors"
    value        = "300"
  }
}

Steps to reproduce

1. Plan and apply the above code using AWS provider version 3.54.0.
2. Run terraform plan once more. Note that terraform reports there are no changes.
3. Upgrade to AWS provider version 3.55.0 (or higher).
4. Run terraform plan. Note this time there are changes (!). See below for the diff reported by terraform.
5. Run terraform apply. Hopefully this resolves that changes
6. Run terraform plan. Alas, the changes are still there.
7. Downgrade to AWS provider version 3.54.0
8. Run terraform plan. There are no changes.

  # aws_db_parameter_group.attempt4 will be updated in-place
  ~ resource "aws_db_parameter_group" "attempt4" {
        id          = "attempt4"
        name        = "attempt4"
        tags        = {}
        # (4 unchanged attributes hidden)

      + parameter {
          + apply_method = "immediate"
          + name         = "open_cursors"
          + value        = "300"
        }
      - parameter {
          - apply_method = "pending-reboot" -> null
          - name         = "open_cursors" -> null
          - value        = "300" -> null
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

[r-kok]

I learned some more about the circumstances under which this error occurs.

The problem arises when you try to create a "user" parameter with a value that is identical to value on "system" or "engine-default" source level. For example, oracle parameter "open_cursors" has value 300 on "system" source level.

If you try to set "open_cursors" = 300 you have drift.
In the AWS console you can see that the parameter that gets created is a parameter with source level "system".
This is unexpected.

If instead you set "open_cursors" = 301 there is no drift.
In the AWS console you see that the parameter that gets created has source level "user", as expected.

Tested with terraform AWS provider 4.0.0

[r-kok]

The bad resource has drift, the ok resource does not.

resource "aws_db_parameter_group" "bad" {
  name = "bad"
  family = "oracle-ee-12.1"

  parameter {
    apply_method = "immediate"
    name         = "open_cursors"
    value        = "300"
  }
}

resource "aws_db_parameter_group" "ok" {
  name = "ok"
  family = "oracle-ee-12.1"

  parameter {
    apply_method = "immediate"
    name         = "open_cursors"
    value        = "301"
  }
}

[gclough]

@r-kok, we have noticed the same thing, in that when a parameter value matches the AWS default setting we get constant drift. On PostgreSQL RDS we changed our minimum max_wal_size formula to 2112 to ensure it never matches the default of 2048. Now we no longer get drift on every terraform run:

"max_wal_size" = [min(max(lookup(lookup(local.instance_parameter_map, local.instance_class), "db_instance_class_memory_gib") * 128, 2112), 65536), "immediate"]       # Max WAL before forced checkpoint RAM/10485760, min 2112MiB, max 64GiB (default: 2GiB) # WARNING: Do not set to 2048, or it triggers a terraform provider bug

[leslie-alldridge]

pls fix :)

[blowfishpro]

Is this the same issue as #22028 ?

[gclough]

Is this the same issue as #22028 ?

@blowfishpro , it appears so, yes. As discussed above, the trigger seems to be when a parameter is supplied which also matches the default setting in AWS. I would suggest that #22028 is closed as a dupe of this.

[james-valente-simplisafe]

I believe this issue can be closed now.

See the merged PR #24737 for a demonstration of why this is a behavioral artifact of the AWS API and not Terraform itself. The PR adds a NOTE to the latest aws_db_parameter_group resource documentation, explaining why this is happening.

[justinretzolk]

Hey y'all 👋 Based on the details @jvalente11 included in their PR -- which added some information around this to the resource documentation -- this looks to be an upstream issue with AWS. Since the behavior is now documented, we'll close this issue for now, but I'll also be following up with a support ticket to AWS to report the unexpected behavior.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.