RDSCluster Parameter Group not applied when restoring from snapshot

[hashibot]

This issue was originally opened by @luis-silva as hashicorp/terraform#12490. It was migrated here as part of the provider split. The original body of the issue is below.

---

When creating an AWS Aurora RDS cluster from an existing snapshot, the new cluster parameter group is not applied.
I believe this is caused by the fact that when restoring a cluster from a snapshot AWS API does not allow that parameter to be set, check the API, AWS just assigns the default parameter group. It is then required to issue a new API request to modify the cluster, supplying the new parameter group. I don't believe terraform is executing this second step.

If you check the output logs below you can clearly see that terraform is sending a request for "RestoreDBClusterFromSnapshot" supplying the custom parameter group and then AWS responds with <DBClusterParameterGroup>default.aurora5.6</DBClusterParameterGroup>.

The same happens with the backup retention period (and possibly other parameters that I haven't yet noticed)

Terraform Version

$ terraform -v
Terraform v0.8.7

Affected Resource(s)

• AWS_RDS_CLUSTER

Terraform Configuration Files

 resource "aws_rds_cluster" "main_aurora_cluster" {
  provider                        = "aws"
  cluster_identifier              = "main-database-cluster"
  availability_zones              = 3
  final_snapshot_identifier       = "main-database-cluster-${uuid()}"
  skip_final_snapshot             = "false"
  backup_retention_period         = "35"
  preferred_backup_window         = "03:00-04:00"
  preferred_maintenance_window    = "mon:05:00-mon:07:00"
  port                            = "3306"
  storage_encrypted               = "false"
  apply_immediately               = "true"
  snapshot_identifier             = "arn:aws:rds:eu-west-1:redacted:cluster-snapshot:aurora-base"
  db_subnet_group_name            = "${module.main_db_subnet_group.id}"
  db_cluster_parameter_group_name = "${aws_rds_cluster_parameter_group.main_aurora_cluster_parameter_group.id}"
  vpc_security_group_ids          = ["${module.security_group_main_database.id}"]
  lifecycle {
    prevent_destroy           = true
    ignore_changes            = ["final_snapshot_identifier"]
  }
  depends_on = ["aws_rds_cluster_parameter_group.main_aurora_cluster_parameter_group"]
}

Debug Output

2017/03/07 10:38:02 [DEBUG] plugin: terraform: Action=**RestoreDBClusterFromSnapshot**&AvailabilityZones.member.1=3&DBClusterIdentifier=main-database-cluster&DBSubnetGroupName=main-db-subnet-group&Engine=aurora&Port=3306&SnapshotIdentifier=arn%3Aaws%3Ards%3Aeu-west-1%3Acluster-snapshot%3Aaurora-base&Tags=&Version=2014-10-31&VpcSecurityGroupIds.member.1=redacted
2017/03/07 10:38:02 [DEBUG] plugin: terraform: -----------------------------------------------------
  cluster_members.#:                 "" => "<computed>"
  database_name:                     "" => "<computed>"
  db_cluster_parameter_group_name:   "" => "main-aurora-cluster-parameter-group"
  db_subnet_group_name:              "" => "main-db-subnet-group"
  endpoint:                          "" => "<computed>"
  engine:                            "" => "<computed>"
  final_snapshot_identifier:         "" => "main-database-cluster-6b64173f-35e1-ae70-9182-decf5e3a48c2"
  kms_key_id:                        "" => "<computed>"
  master_username:                   "" => "<computed>"
  port:                              "" => "3306"
  preferred_backup_window:           "" => "03:00-04:00"
  preferred_maintenance_window:      "" => "mon:05:00-mon:07:00"
  reader_endpoint:                   "" => "<computed>"
  skip_final_snapshot:               "" => "false"
  snapshot_identifier:               "" => "arn:aws:rds:eu-west-1:redacted:cluster-snapshot:aurora-base"
  storage_encrypted:                 "" => "false"
  vpc_security_group_ids.#:          "" => "1"
  vpc_security_group_ids.3685952533: "" => "redacted"
2017/03/07 10:38:03 [DEBUG] plugin: terraform: aws-provider (internal) 2017/03/07 10:38:03 [DEBUG] [aws-sdk-go] DEBUG: Response rds/RestoreDBClusterFromSnapshot Details:
2017/03/07 10:38:03 [DEBUG] plugin: terraform: --- **[ RESPONSE ]**--------------------------------------
2017/03/07 10:38:03 [DEBUG] plugin: terraform: HTTP/1.1 200 OK
2017/03/07 10:38:03 [DEBUG] plugin: terraform: Connection: close
2017/03/07 10:38:03 [DEBUG] plugin: terraform: Transfer-Encoding: chunked
2017/03/07 10:38:03 [DEBUG] plugin: terraform: Content-Type: text/xml
2017/03/07 10:38:03 [DEBUG] plugin: terraform: Date: Tue, 07 Mar 2017 10:38:04 GMT
2017/03/07 10:38:03 [DEBUG] plugin: terraform: Vary: Accept-Encoding
2017/03/07 10:38:03 [DEBUG] plugin: terraform: X-Amzn-Requestid: 24a34294-0322-11e7-8295-81f721831501
2017/03/07 10:38:03 [DEBUG] plugin: terraform:
2017/03/07 10:38:03 [DEBUG] plugin: terraform: 8ac
2017/03/07 10:38:03 [DEBUG] plugin: terraform: <RestoreDBClusterFromSnapshotResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
2017/03/07 10:38:03 [DEBUG] plugin: terraform:   <RestoreDBClusterFromSnapshotResult>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:     <DBCluster>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <AllocatedStorage>1</AllocatedStorage>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <DatabaseName>main</DatabaseName>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <AssociatedRoles/>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <AvailabilityZones>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:         <AvailabilityZone>eu-west-1a</AvailabilityZone>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:         <AvailabilityZone>eu-west-1b</AvailabilityZone>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:         <AvailabilityZone>eu-west-1c</AvailabilityZone>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       </AvailabilityZones>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <PreferredBackupWindow>23:37-00:07</PreferredBackupWindow>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <Endpoint>main-database-cluster.cluster-casv1putpmcz.eu-west-1.rds.amazonaws.com</Endpoint>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <Engine>aurora</Engine>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <ReaderEndpoint>main-database-cluster.cluster-ro-casv1putpmcz.eu-west-1.rds.amazonaws.com</ReaderEndpoint>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <ReadReplicaIdentifiers/>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <IAMDatabaseAuthenticationEnabled>false</IAMDatabaseAuthenticationEnabled>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <ClusterCreateTime>2017-03-07T10:38:04.320Z</ClusterCreateTime>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <EngineVersion>5.6.10a</EngineVersion>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <MasterUsername>admin</MasterUsername>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <DBClusterMembers/>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <MultiAZ>false</MultiAZ>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <StorageEncrypted>false</StorageEncrypted>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <DBSubnetGroup>main-db-subnet-group</DBSubnetGroup>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <HostedZoneId>redacted</HostedZoneId>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <VpcSecurityGroups>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:         <VpcSecurityGroupMembership>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:           <VpcSecurityGroupId>sg-4c0ae535</VpcSecurityGroupId>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:           <Status>active</Status>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:         </VpcSecurityGroupMembership>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       </VpcSecurityGroups>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <Port>3306</Port>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <PreferredMaintenanceWindow>tue:00:09-tue:00:39</PreferredMaintenanceWindow>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       **<DBClusterParameterGroup>default.aurora5.6</DBClusterParameterGroup>**
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <BackupRetentionPeriod>1</BackupRetentionPeriod>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <DBClusterIdentifier>main-database-cluster</DBClusterIdentifier>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <DbClusterResourceId>cluster-YRHG7STBBUX6STCNHMIHIAOBXM</DbClusterResourceId>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <DBClusterArn>arn:aws:rds:eu-west-1:redacted:cluster:main-database-cluster</DBClusterArn>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:       <Status>creating</Status>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:     </DBCluster>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:   </RestoreDBClusterFromSnapshotResult>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:   <ResponseMetadata>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:     <RequestId>redacted</RequestId>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:   </ResponseMetadata>
2017/03/07 10:38:03 [DEBUG] plugin: terraform: </RestoreDBClusterFromSnapshotResponse>
2017/03/07 10:38:03 [DEBUG] plugin: terraform:
2017/03/07 10:38:03 [DEBUG] plugin: terraform: 0
2017/03/07 10:38:03 [DEBUG] plugin: terraform:
2017/03/07 10:38:03 [DEBUG] plugin: terraform:
2017/03/07 10:38:03 [DEBUG] plugin: terraform: -----------------------------------------------------
2017/03/07 10:38:03 [DEBUG] plugin: terraform: aws-provider (internal) 2017/03/07 10:38:03 [INFO] RDS Cluster is restoring from snapshot with default security, but custom security should be set, will now update after snapshot is restored!
2017/03/07 10:38:03 [DEBUG] plugin: terraform: aws-provider (internal) 2017/03/07 10:38:03 [INFO] RDS Cluster Instance ID: main-database-cluster

Expected Behavior

The cluster is created from the snapshot and then a modify cluster request is sent to apply the new parameters

Actual Behavior

The cluster is created from the snapshot but parameters like the cluster parameter group are not correctly set (fallbacks to default.aurora5.6), since they can only be applied after the cluster is created by a modify cluster api call, and not during the initial creation from snapshot.

Steps to Reproduce

Create an aurora rds cluster from an existing cluster snapshot and specify a different parameter group. Notice that it doesn't get applied.

[luis-silva]

Just to mention that this issue is still present on 0.9.8.

[catsby]

I merged #926 which should address this, thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!