[resource datadog_integration_aws] External ID Lifecycle

[mlclmj]

I've been using the AWS integration resource recently and it seems inefficient to regenerate a new external_id for each AWS account every time a change is made to an AWS account integration.

As a solution to this, I'd like to propose either: transparently replacing the AWS integration until a proper update is supported, or adding a separate resource to manage the external_id of AWS account integrations.

Approach 1 - Support Transparent Replacement

To support a transparent replacement of the integration using the update API,
the resource schema could be changed to the following:

Schema: map[string]*schema.Schema{
			"account_id": {
				Type:     schema.TypeString,
				Required: true,
				ForceNew: true,
			},
			"role_name": {
				Type:     schema.TypeString,
				Required: true,
			},
			"filter_tags": {
				Type:     schema.TypeList,
				Optional: true,
				Elem:     &schema.Schema{Type: schema.TypeString},
			},
			"host_tags": {
				Type:     schema.TypeList,
				Optional: true,
				Elem:     &schema.Schema{Type: schema.TypeString},
			},
			"account_specific_namespace_rules": {
				Type:     schema.TypeMap,
				Optional: true,
				Elem:     schema.TypeBool,
			},
			"external_id": {
				Type:     schema.TypeString,
				Computed: true,
			},
		},

This approach may be favorable, since it wouldn't require changes to be made once the Datadog API supports a proper update method.

Approach 2 - Add a New Resource

Add a generate_external_id attribute to the datadog_integration_aws resource.

This value would default to true, preserving current behavior of this resource and generating a new external_id each time Terraform triggers a create or update for this resource. Setting this attribute to false would prevent this resource from (re)generating external_id values on create or update and allow it to be managed separately.

Add a new datadog_integration_aws_external_id resource.

Add a separate resource that would be dedicated to managing the lifecycle of the external_id, and would output the external_id when it is created. The schema of this new resource would look something like:

Schema: map[string]*schema.Schema{
			"account_id": {
				Type:     schema.TypeString,
				Required: true,
			},
			"external_id": {
				Type:     schema.TypeString,
				Computed: true,
			},
		},

An example of how this might look in an actual Terraform file:

resource "datadog_integration_aws" "example" {
    account_id = "1234567890"
    role_name = "DatadogAWSIntegrationRole"
    filter_tags = ["key:value"]
    host_tags = ["key:value", "key2:value2"]
    account_specific_namespace_rules = {
        auto_scaling = false
        opsworks = false
    }
    generate_external_id = false
}

resource "datadog_integration_aws_external_id" "example_id" {
    depends_on = [ resource.datadog_integration_aws.example" ]

    account_id = resource.datadog_integration_aws.example.account_id
}

Affected Resource

• resource_datadog_integration_aws

[nmuesch]

Hey @mlclmj Thanks for the detailed issue report and suggestions! We actually now have support for a PUT method in the Datadog AWS Integration API - https://docs.datadoghq.com/api/?lang=bash#update-an-aws-integration that allows you to update a single AWS Integration (account_id&role_name pair) at a time

Though its not yet included in this provider, and I don't have a timeframe for this, I believe implementing this in the provider (and the underlying zorkian golang client) would resolve this issue of external ID regeneration on each update.

Approach 1 seems favorable in that we can remove the ForceNew from these attributes and properly create the Update resource, removing a lot of the comments 😄 What do you think?

[iniinikoski]

Sorry @mlclmj for stealing the issue a bit, but did you bump into any cycle errors on your Terraform code? I keep getting this:

Error: Cycle: aws_iam_role.datadog_role, datadog_integration_aws.account

...and for a reason, as "datadog_integration_aws" requires the AWS role name - and "aws_iam_role" requires the external ID :) Chicken-egg - which came first? :D

PS. It's been quite nasty in Datadog UI as well as it keeps creating a new ExternalId if you do any misclicks... Great to hear these issues being solved...

[nmuesch]

Hey @mlclmj We just merged in a PR - #521 that introduces the update method into the AWS resource. This will be available in the next release of the provider.

Since this will allow for updates to occur without the need to delete and recreate, you shouldn't see the external_id be changed on each modification once this is released.

I'll go ahead and close out this issue for now but let us know if this doesn't address your use case. Thanks for the detailed issue report and suggested approaches!