cannot enable aws rds performance insights on existing rds/postgres instance

[ubdb]

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.14

Affected Resource(s)

• • provider.aws v2.19.0

Terraform Configuration Files

resource "aws_db_parameter_group" "wasa" {
  name        = "wasa-${var.cluster}-${terraform.workspace}"
  description = "parameter group for wasa database"
  family      = "postgres10"

  parameter = [
    { name = "log_lock_waits", value = true },
    { name = "log_min_duration_statement", value = 1000 },
    { name = "max_wal_senders", value = 15, apply_method = "pending-reboot" }, 
    { name = "log_connections", value = 0 }, 
  ]

  tags {
    ManagedBy            = "Terraform. Do not touch."
    database-component   = "wasa-backend"
    kubernetes-cluster   = "${var.cluster}"
    terraform-workspace  = "${terraform.workspace}"
    kubernetes-namespace = "${terraform.workspace}"
  }
}

resource "aws_db_instance" "wasa" {
  identifier                 = "wasa-${var.cluster}-${terraform.workspace}"
  allocated_storage          = "10"
  storage_type               = "${var.rds_storage_type[terraform.workspace]}"
  engine                     = "postgres"
  engine_version             = "${var.workspace_to_engine_version_map[terraform.workspace]}"
  instance_class             = "${var.instance_class_4_wasa_rds[terraform.workspace]}"
  name                       = "postgres"
  username                   = "postgres"
  password                   = "${var.wasa_db_password}"
  db_subnet_group_name       = "${var.workspace_to_db_subnet_group_name_map[terraform.workspace]}"
  backup_retention_period    = "${var.wasa_backup_retention_period}"
  backup_window              = "01:00-01:30"
  maintenance_window         = "wed:02:30-wed:03:30"
  performance_insights_enabled = true
  performance_insights_retention_period = 7
  skip_final_snapshot        = true                                                                # used for being able to destroy
  vpc_security_group_ids     = ["${var.rds_vpc_security_group_ids[terraform.workspace]}"]
  auto_minor_version_upgrade = true
  deletion_protection        = false
  multi_az                   = false
  parameter_group_name       = "${aws_db_parameter_group.wasa.name}"

  # only works from inside aws
  # coded for gitlab-ci
  provisioner "local-exec" {
    command = "sleep 10 && scripts/wasa.sh"

    environment = {
      PGENDPOINT = "${aws_db_instance.wasa.endpoint}"
      PGUSER     = "${aws_db_instance.wasa.username}"
      PGPASSWORD = "${aws_db_instance.wasa.password}"
    }
  }

  tags {
    ManagedBy            = "Terraform. Do not touch."
    database-component   = "wasa-backend"
    kubernetes-cluster   = "${var.cluster}"
    terraform-workspace  = "${terraform.workspace}"
    kubernetes-namespace = "${terraform.workspace}"
  }
}

Debug Output

https://gist.github.com/ubdb/6e8aa7c8e68872fb1535054062d228e6

Panic Output

None

Expected Behavior

Terraform should enable rds performance insights
Terraform should exit with an error, when something doesn't work.

Actual Behavior

Terraform does not apply the changes, as AWS returns a HTTP Status Code 500.
Terraform tries to apply the changes endless. Tt doesn't seem to handle error found incorrectly

Steps to Reproduce

create an rds instance without specifying the parameters.

  performance_insights_enabled = true
  performance_insights_retention_period = 7

1. add the two parameters to the terraform code
2. terraform apply

Important Factoids

none

References

None

[Haegin]

Does specifying the performance_insights_kms_key_id fix the issue?

[ubdb]

Hi Haegin,

actually not. But I'm pretty sure there were some permissions missing.
with my personal account I first got 500er errors on aws web console and it worked there after assigning all kms related permissions.

However - terraform tries endlessly to apply these changes and doesn't seem to care about the 500er error code. This is definitely a bug.

After generating a kms key with terraform and using the corresponding id for that key, I get another error:
Error: Error creating DB Instance: InvalidParameterCombination: To enable encryption at rest StorageEncrypted must be set to 'true'

I currently don't use encrypted storage.
In the web gui I can enable performance insight without using encrypted storage.

Do you have any ideas how to achieve this with terraform or is this another bug?

Best Regards,
Uwe

Below you find the corresponding debug log output:

2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: 2019/07/29 09:12:19 [DEBUG] DB Instance create configuration: {
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   AllocatedStorage: 10,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   AutoMinorVersionUpgrade: true,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   BackupRetentionPeriod: 7,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   CopyTagsToSnapshot: false,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   DBInstanceClass: "db.t2.micro",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   DBInstanceIdentifier: "wasa-devcluster-sandbox",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   DBName: "postgres",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   DBParameterGroupName: "wasa-devcluster-sandbox",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   DBSubnetGroupName: "default-vpc-xxxxxxxxx",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   DeletionProtection: false,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   EnablePerformanceInsights: true,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   Engine: "postgres",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   EngineVersion: "10.6",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   KmsKeyId: "arn:aws:kms:eu-central-1:xxxxxxxxxxxxxxxx:key/c878ebfb-476a-4c8f-8f4d-f8a6b72480d0",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   MasterUserPassword: "xxxxxxxxxxxxxxxxxxx",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   MasterUsername: "postgres",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   PerformanceInsightsRetentionPeriod: 7,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   PreferredBackupWindow: "01:00-01:30",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   PreferredMaintenanceWindow: "wed:02:30-wed:03:30",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   PubliclyAccessible: false,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   StorageEncrypted: false,
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   StorageType: "standard",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   Tags: [
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     {
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Key: "kubernetes-namespace",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Value: "sandbox"
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     },
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     {
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Key: "database-component",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Value: "wasa-backend"
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     },
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     {
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Key: "terraform-workspace",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Value: "sandbox"
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     },
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     {
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Key: "ManagedBy",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Value: "Terraform. Do not touch."
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     },
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     {
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Key: "kubernetes-cluster",
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:       Value: "devcluster"
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     }
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   ],
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   VpcSecurityGroupIds: ["sg-xxxxxxxx"]
2019-07-29T09:12:19.199+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: }




2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: 2019/07/29 09:12:19 [DEBUG] [aws-sdk-go] DEBUG: Response rds/CreateDBInstance Details:
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: ---[ RESPONSE ]--------------------------------------
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: HTTP/1.1 400 Bad Request
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: Connection: close
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: Content-Length: 325
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: Content-Type: text/xml
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: Date: Mon, 29 Jul 2019 07:12:19 GMT
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: X-Amzn-Requestid: b9d285e2-a164-4d83-9016-d2e872a94cbb
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: 
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: 
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: -----------------------------------------------------
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: 2019/07/29 09:12:19 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   <Error>
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     <Type>Sender</Type>
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     <Code>InvalidParameterCombination</Code>
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:     <Message>To enable encryption at rest StorageEncrypted must be set to 'true'</Message>
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   </Error>
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4:   <RequestId>b9d285e2-a164-4d83-9016-d2e872a94cbb</RequestId>
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: </ErrorResponse>
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: 2019/07/29 09:12:19 [DEBUG] [aws-sdk-go] DEBUG: Validate Response rds/CreateDBInstance failed, attempt 0/25, error InvalidParameterCombination: To enable encryption at rest StorageEncrypted must be set to 'true'
2019-07-29T09:12:19.342+0200 [DEBUG] plugin.terraform-provider-aws_v2.21.0_x4: 	status code: 400, request id: b9d285e2-a164-4d83-9016-d2e872a94cbb
2019/07/29 09:12:19 [DEBUG] aws_db_instance.wasa[0]: apply errored, but we're indicating that via the Error pointer rather than returning it: Error creating DB Instance: InvalidParameterCombination: To enable encryption at rest StorageEncrypted must be set to 'true'
	status code: 400, request id: b9d285e2-a164-4d83-9016-d2e872a94cbb

[Haegin]

Ah I was running into a similar problem but I was using encrypted storage. Terraform would fail to enable performance insights and there is no way to specify the kms key for performance insights on the Terraform AWS module I'm using but enabling it in the web console then running terraform apply updated the state and fixed the problem for me. Good luck!

[ubdb]

damn, I need to handle almost 200 database servers. But thanks for your response

[ubdb]

no luck at all.
I have to handle about 200 database servers. And most of them have instance class db.t2.micro. Here I get the error: Error creating DB Instance: InvalidParameterCombination: DB Instance class db.t2.micro does not support encryption at rest

[ryancalhoun]

I had the same issue, and had to fork the module source, and specify both performance_insights_kms_key_id and performance_insights_retention_period to get the apply to succeed. Without the retention period, I got an invalid parameter error. Without the kms key, terraform just hung without making any changes or modifying the state.

[ubdb]

cool. what did you change in the forked code?
I'm using in my request both performance_insights_kms_key_id and performance_insights_retention_period-

[ryancalhoun]

@ubdb The pull request is linked above, number 157. The files diff is here https://github.com/terraform-aws-modules/terraform-aws-rds/pull/157/files.

I had to add both kms key and retention period to get the terraform apply to work.

[bflad]

Fix submitted for the resource: #9745

[bflad]

The fix for this has been merged and will release with version 2.24.0 of the Terraform AWS Provider, tomorrow. 👍

[ghost]

This has been released in version 2.24.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!