aws_db_instance doesn't properly handle monitoring_role_arn

[iancward]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7
+ provider.aws v1.31.0

Affected Resource(s)

• aws_db_instance

Terraform Configuration Files

resource "aws_db_instance" "main" {
  identifier = "test_db"
  engine = "postgres"
  engine_version = "9.6.6"
  instance_class = "db,t2.large"
  monitoring_interval = "60"
  monitoring_role_arn = "${ aws_iam_role.monitoring_role.arn }"
}

Expected Behavior

1. When I add monitoring_role_arn and monitoring_interval to a resource (that had neither previously), terraform should add both to the DB in AWS
2. If I add a monitoring interval and monitoring role to an RDS instance via the AWS console, and go back and run terraform against a resource with neither, it should want to remove both; currently, it only attempts to set the monitoring_interval to 0. (unless AWS automatically does this..)
3. If I add a monitoring interval and monitoring role via the AWS console and then add only the monitoring interval to terraform, then when I run terraform, it should do something (currently it ignores monitoring_role_arn completely); it should either attempt to remove the monitoring_role_arn from the DB instance (which would probably cause an AWS API error) or it should throw an error stating that when monitoring_interval is used, monitoring_role_arn should also be required.

Actual Behavior

Unless terraform creates a db instance with monitoring_role_arn, it seems pretty dumb to whatever the DB instance actually has on it.

Steps to Reproduce

1. code a terraform resource for aws_db_instance, omitting both monitoring_interval and monitoring_role_arn
2. terraform apply
3. update terraform resource to add monitoring_interval and monitoring_role_arn
4. terraform plan
5. See that terraform wants to apply monitoring_interval but not monitoring_role_arn
6. terraform apply
7. Receive error from AWS: `A MonitoringRoleARN value is required if you specify a MonitoringInterval value other than 0

Important Factoids

It appears that, when a DB instance has a MonitoringInterval of 0, the API omits MonitoringRoleArn from the response. This might be why terraform is having a hard time with this field (since it's not in on the resource at the start).

References

N/A

[iancward]

Well, this might be more nuanced than this.
On the terraform apply to add enhanced monitoring (step 5 in "Steps to Reproduct" above), I get this error from the AWS API via terraform:

* aws_db_instance.main: Error modifying DB Instance quay-registry-8f71cc36-qa: InvalidParameterValue: IAM role ARN value is invalid or does not include the required permissions for: ENHANCED_MONITORING

The following terraform plan shows that terraform wants to only change the monitoring_interval. And then a subsequent terraform apply gives the error from step 7 of "steps to reproduce" above. Debug output from terraform apply shows that it's saved both monitoring_interval and monitoring_role_arn to the remote state, even though the apply failed (and both show up in terraform state show aws_db_instance.main).
So maybe because the API doesn't show the MonitoringRoleArn in the response for describe-db-instances, terraform doesn't think it needs to be applied?

I'm not sure why I received the error from the AWS API about not having ENHANCED_MONITORING privileges. I created a role with the same assume role policy and same policy attachment to AmazonRDSEnhancedMonitoringRole. I can also use the console to manually modify the DB instance to enable enhanced monitoring with that role I created.

[iancward]

I ran a terraform state pull, removed the monitoring_role_arn and monitoring_interval properties and pushed it back with terraform state push. It looks like the terraform apply is succeeding, so this must be that terraform attempted to add the role prior to the role policy attachment for AmazonRDSEnhancedMonitoringRole.

I'll add a depends_on statement to get around this; but terraform probably needs to be made to understand that a lack of a property in the response from a describe API call means that property doesn't exist and needs to be modified.

[iancward]

Well, it looks like even with aws_iam_role_policy_attachment.monitoring_policy_attach in a depends_on property for the db_instance (and I can confirm from the terraform output that the attachment preceded the db modify), I still got the "does not include the required permissions" error. So maybe there's also an issue with the AWS API...

[bflad]

@iancward you may be interested in: https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html#troubleshoot_general_eventual-consistency

We generally need to handle this problem in downstream resources by adding retry conditions for specific error messages. It looks like the create function handles errors like these but the ModifyDbInstance call does not.

My recommendation here would be to add logic around that call, e.g.

// Handle IAM eventual consistency
err = resource.Retry(1*time.Minute, func() *resource.RetryError {
  _, err := conn.ModifyDBInstance(req)
  // InvalidParameterValue: IAM role ARN value is invalid or does not include the required permissions for: ENHANCED_MONITORING
  if isAWSErr(err, "InvalidParameterValue", "IAM role ARN value is invalid or does not include the required permissions") {
    return resource.RetryableError(err)
  }
  if err != nil {
    return resource.NonRetryableError(err)
  }
  return nil
})
if err != nil {
  return fmt.Errorf("Error modifying DB Instance %s: %s", d.Id(), err)
}

[bflad]

Fix submitted: #9747

[bflad]

The fix for this has been merged and will release in version 2.25.0 of the Terraform AWS Provider, in about a week.

[ghost]

This has been released in version 2.25.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!