aws db parameter group ... converts keys and values to lower case and fails 'apply' due to aws_db_parameter_group changes

[hashibot]

This issue was originally opened by @ebgc as hashicorp/terraform#9215. It was migrated here as part of the provider split. The original body of the issue is below.

---

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

mac os x 10.12
Terraform v0.7.4

Affected Resource(s)

• resource_aws_db_parameter_group
• aws_db_instance

Terraform Configuration Files

resource "aws_db_instance" "default" {
  allocated_storage = "${var.db_storage}"
  allow_major_version_upgrade = false
  apply_immediately = false
  auto_minor_version_upgrade = false
  backup_retention_period = "${var.backup_retention_period}"
  backup_window = "${var.backup_window}"
  copy_tags_to_snapshot = "${var.copy_tags_to_snapshot}"
  db_subnet_group_name = "${aws_db_subnet_group.rds.id}"
  engine = "${var.db_engine}"
  engine_version = "${var.db_version}"
  final_snapshot_identifier = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${var.date}"
  identifier = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${var.db_engine}"
  instance_class = "${var.db_instance}"
  maintenance_window = "${var.maintenance_window}"
  multi_az = "${var.multi_az}"
  name = "${var.db_name}"
  parameter_group_name = "${aws_db_parameter_group.default.id}"
  password = "${var.db_password}"
  publicly_accessible = "${var.db_public_accessible}"
  skip_final_snapshot = "${var.skip_final_snapshot}"
  snapshot_identifier = "${var.db_snapshot_id}"
  storage_type = "${var.db_storage_type}"
  username = "${var.db_username}"
  vpc_security_group_ids = ["${aws_security_group.rds.id}"]

  lifecycle {
    ignore_changes = ["final_snapshot_identifier"]
  }

  tags {
    Name = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${var.db_engine}"
    Description = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${var.db_engine}"
    Environment = "${var.environment}"
  }
}

resource "aws_db_parameter_group" "default" {
  name = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${data.template_file.db_family_no_dot.rendered}-pg"
  description = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${data.template_file.db_family_no_dot.rendered}-pg"
  family = "${data.template_file.db_family_dot.rendered}"

  # tf ignores them : timezone, client_encoding
  parameter {
    name = "lc_messages"
    value = "en_US.utf8"
    apply_method = "immediate"
  }

  parameter {
    name = "lc_monetary"
    value = "en_US.utf8"
    apply_method = "immediate"
  }

  parameter {
    name = "lc_numeric"
    value = "en_US.utf8"
    apply_method = "immediate"
  }

  parameter {
    name = "lc_time"
    value = "en_US.utf8"
    apply_method = "immediate"
  }

  tags {
    Name = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${data.template_file.db_family_no_dot.rendered}-pg"
    Description = "${var.env_type}-${var.env_name}-${var.env_number}-${var.service}-${data.template_file.db_family_no_dot.rendered}-pg"
    Environment = "${var.environment}"
  }
  lifecycle {
    ignore_changes = ["name", "description", "family", "tags"]
  }
}

Debug Output

1 error(s) occurred:

• aws_db_parameter_group.default: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

Terraform Version: 0.7.4
Resource ID: aws_db_parameter_group.default
Mismatch reason: attribute mismatch: parameter.1848500544.value
Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"parameter.1848500544.value":*terraform.ResourceAttrDiff{Old:"en_us.utf8", New:"en_US.utf8", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "parameter.2080321556.value":*terraform.ResourceAttrDiff{Old:"en_us.utf8", New:"en_US.utf8", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "parameter.4193353794.value":*terraform.ResourceAttrDiff{Old:"en_us.utf8", New:"en_US.utf8", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "parameter.3444491795.value":*terraform.ResourceAttrDiff{Old:"en_us.utf8", New:"en_US.utf8", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

Panic Output

n.a.

Expected Behavior

take what aws api returns without modification (dont use flattenParameters)
save what tf file defines at tfstate file without modification (dont use flattenParameters)

Actual Behavior

all keys and values will be down-cased on tfstate file even though on initial run they are set on aws as defined in tf file

Steps to Reproduce

have an aws_db_instance (postgres, 9.4.5) with resource_aws_db_parameter_group (postgres9.4) which sets 'lc_messages' to 'en_US.utf8', or 'timezone' to 'UTF'

even though we set 'en_US.utf8' in tf file (terraform during 'apply' will set 'en_US.utf8' on aws rds resource) - tfstate file will have 'en_us.utf8' set.
next run of terraform 'apply' will read resource from aws and convert it to lower case 'en_us.utf8' (see below 'flattenParameters' function) which will not fail compared to state file value (match). but because there is a diff with tf file (remember we set 'en_US.utf8') value it will try to modify the resource.
here another bug comes in play where terraform complains that db parameter group has a diff and reports: '* aws_db_parameter_group.default: diffs didn't match during apply.'

Important Factoids

---- terraform/builtin/providers/aws/resource_aws_db_parameter_group.go ----
func resourceAwsDbParameterGroupRead
...
d.Set("parameter", flattenParameters(describeParametersResp.Parameters))
...
---- terraform/builtin/providers/aws/resource_aws_db_parameter_group.go ----
func flattenParameters
...
r["name"] = strings.ToLower(_i.ParameterName)
...
r["value"] = strings.ToLower(_i.ParameterValue)
....
---- terraform/builtin/providers/aws/structure.go ----

---- terraform/builtin/providers/aws/structure.go ----

References

[mlb5000]

can confirm this is till an issue. When you run tf plan it will always want to create a new parameter group version.

[domcleal]

As a workaround I was able to set it to the lower case value, which worked OK for setting binlog_checksum:

   parameter {
     name  = "binlog_checksum"
-    value = "NONE"
+    value = "none"
   }

Aurora (MySQL 5.6) seemed to handle this being lower case, but this might not work correctly with other parameters.

The provider would then retrieve the lower case value from the API, compare it against the resource which is also specifying lower case and not try to reapply the change.

[anGie44]

Hi @ebgc, thanks again for creating this issue and apologies you came across this confusing error!

There have been a lot of relevant changes in the Terraform Core and Terraform AWS Provider codebases since this issue was originally reported so as a first step it might be best to try updating to recent versions of both and seeing if any of those updates may have resolved this issue.

Attempting to troubleshoot these types of errors is very difficult in Terraform 0.11 and earlier as those versions will report this problem (diffs didn't match during apply error) where a problematic configuration value exists or is referenced and not where the Terraform resource with the underlying problem actually exists. We typically will need much more information than the original error message suggests for filing the issue (e.g. the full Terraform configuration). There are also frequent issues in the Terraform 0.11 error reporting where the error message is due to behavior outside the control of the Terraform resources themselves (e.g. lifecycle configurations).

In Terraform 0.12 and later, the "value did not match" type of error can now be found with the newer Provider produced inconsistent result after apply error, which better reports the source of the problem so the provider maintainers and community can more easily work towards a fix of the issue. Other similar errors that fell previously into diffs didn't match during apply errors, but outside problems within the Terraform provider are also better diagnosed before being displayed.
Since the majority of the Terraform code path and reporting structure of these errors has changed between Terraform 0.11 and 0.12, along with the difficult triage process in Terraform 0.11 and earlier, the maintainers prefer to close these older issues to continue troubleshooting and fixing based on Terraform 0.12's error reporting instead. I apologize for this unsatisfying closure of this particular version of the issue.

That said, if the configuration does happen to reproduce the newer Provider produced inconsistent result after apply error in Terraform 0.12 (which requires Terraform AWS Provider version 2.7.0 or later), please do not hesitate to file a new issue and complete all of the information requested in the Bug Report template so we can hopefully get to the root cause for fixing the issue. Thanks again for this bug report and sorry about the additional steps.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!