Pathway to resolve externalId drift?

[TaxBusby]

Problem

I am using the Terraform module datadog_integration_aws to set up the DD->AWS integration. This creates an IAM role with a trust policy using a specific externalId sourced from the Datadog API.

This same externalId can also be rotated manually via the DataDog UI:

I am facing a scenario where this externalId has been rotated manually outside of the terraform code (most likely by accident?), and I need to get the terraform code in sync.

The Documentation for datadog_integration_aws states that outside changes to externalId will not be detected. But no guidance is provided for how to work around with this drift.

Is there any path I can take to resolve this drift? Do I have to delete and re-create the terraform resource? Should I manually edit the IAM Role? Is my integration going to have downtime if I take any of these steps?

I have tried to search the web & other issues for a resolution but have been unsuccessful at finding other references to this particular issue.

Terraform Version

1.2.2

Affected Resource(s)

• datadog_integration_aws

Terraform Configuration Files

# Create a new Datadog - Amazon Web Services integration
resource "datadog_integration_aws" "datadog_aws" {
  account_id = var.target_aws_account
  role_name  = "DatadogIntegrationRole"
  host_tags  = ["REDACTED"]
}

Expected Behavior

Module or documentation should provide a path to deal with unexpected externalId drift.

Actual Behavior

ExternalId drift cannot be resolved from the module interface or following documentation steps

Steps to Reproduce

1. Set up a datadog_integration_aws resource in terraform and apply it to an AWS account.
2. Navigate to Datadog's AWS Integration Page and select your AWS account.
3. Navigate to the "Account Details" tab
4. Click "Generate New Id" next to the "AWS External ID" field

Result: Datadog AWS Integration now has "Issues:"

Errors last seen 19 hours ago
Datadog is not authorized to perform action sts:AssumeRole
Regions: Every region

[TaxBusby]

I found #335 which suggests others noticed some issues with externalId management though it's not clear this provides a pathway to resolve my scenario.

I am likely to resolve this by manually editing the IAM Role Trust Relationship policy document. However, I don't know if this is the only place this externalId is referenced by the TF module so I am a little uncomfortable making this change manually.

[nkzou]

To solve the drift in terraform configuration, I think the best way would be to remove the resource from the local state with terraform state rm datadog_integration_aws.<resource id> to clear the dirty local state, then import the integration and manually set the external id via environment variable:
EXTERNAL_ID=${external_id} terraform import datadog_integration_aws.<resource id> ${account_id}:${role_name} (import docs here)

I'm unsure if this alone will fix the integration itself though. I believe it should be sufficient if you have an iam policy defined in your terraform config that references the aws integration's external_id field, but if that policy is configured somewhere else you'd need to track down where that is. Looking through the setup guide for each setup method should help for figuring out where the external id is being configured, though this depends on your own environment.

Hopefully that helped, please reach out with more details if there's still confusion.

[berniedurfee-renaissance]

@nkzou why doesn't the provider return the latest external ID when it checks the state of the integration resource?

Shouldn't I be able to do a --refresh-only run to get the new external_id updated in the state?

I have an issue now where the state thinks the external_id is still the old one, so it keeps trying to update the role trust policy to the old external_id.

[alyssa-dahlberg]

I have also run in to the behaviour @berniedurfee-renaissance describes. I did look to see if there is a data provider that could give the current external ID but I can't see one. There is this note in the documentation:

Note: The External ID remains available and is not regenerated for 48 hours, unless explicitly changed by a user or another AWS account is added to Datadog during this period.

Does this imply that the ID might change after 48 hours?