[Bug]: Org Trail resource creation failed for CloudTrail Delegated Administrator

[mhgeay]

Terraform Core Version

1.3.6

AWS Provider Version

4.47.0

Affected Resource(s)

aws_cloudtrail

Expected Behavior

1. We choose a Delegated Administrator for the CloudTrail Service (see AWS doc). All CloudTrail resources remain in the Management Account, even if they are created by the Delegated Account. (the ARN show the Account ID of the Management Account).
2. We deploy a new organization trail on this Delegated Account with Terraform.
3. The organization trail is fully created and enabled in the Management Account.

Actual Behavior

When Terraform applies, it's failed when it try to enable logging it in the Delegated Account because it's did not find the trail.

After successfully creation, The AWS provider seems try to enable the trail locally (in the Delegated account) instead to referer to the trail 'stored' in the Management Account. The ARN in the below log explains the bug.

Relevant Error/Panic Output Snippet

'111111111111' is the Delegated Account ID.

aws_cloudtrail.org_main: Creating...
╷
│ Error: Error starting logging on CloudTrail (org-cloudtrail): TrailNotFoundException: Unknown trail: arn:aws:cloudtrail:us-east-1:111111111111:trail/org-cloudtrail for the user: 111111111111
│ 
│   with aws_cloudtrail.org_main,
│   on 01-cloudtrail.tf line 4, in resource "aws_cloudtrail" "org_main":
│    4: resource "aws_cloudtrail" "org_main" {
│ 
╵
Operation failed: failed running terraform apply (exit 1)�

Terraform Configuration Files

resource "aws_cloudtrail" "org_main" {
  name                          = "org-cloudtrail"
  enable_logging                = true
  s3_bucket_name                = aws_s3_bucket.cloudtrail.id
  s3_key_prefix                 = ""
  include_global_service_events = true
  enable_log_file_validation    = false
  is_multi_region_trail         = true
  is_organization_trail         = true

  event_selector {
    read_write_type                  = "WriteOnly"
    include_management_events        = true
    exclude_management_event_sources = ["kms.amazonaws.com", "rdsdata.amazonaws.com"]
  }
}

Steps to Reproduce

1. TF plan (ok)
2. TF apply (failed)

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-delegated-administrator.html

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[F-Xu]

I have got the same issue.
The following error occur when "terraform apply" is run again:
"Error creating CloudTrail: TrailAlreadyExistsException: Trail [trailname] already exists for customer: 111111111 (Service: AWSCloudTrail; Status Code: 400; Error Code: TrailAlreadyExistsException;...".

[dave-dotnet-overall]

Have the exact same issue occurring as the original poster, when i go into the AWS console i can see the trail has been created but has not been activated. If i click start logging it starts up fine, so could be the issue be a timing related one.

Also if i run the script a second i get the exact same output as F-Xu above, Terraform doesn't think it has already created the trail so its attempting to create it again.

[andrewnicolalde]

I think I might have found a solution (more of a workaround really) to this issue in the interim, at least until the AWS Terraform provider can be updated to properly manage resources created from delegated administrator accounts. This was inspired by a bit of AWS sample code for using delegated administration with GuardDuty.

You can define a separate terraform provider for your management account

provider "aws" {
  region  = "us-east-1"
  profile = "security-acct"
}

provider "aws" {
  alias   = "mgmt"
  profile = "management-acct"
  region  = "us-east-1"
}

and use it to directly manage just the aws_cloudtrail resource.

resource "aws_cloudtrail" "org_trail" {
  provider = aws.mgmt
  name  = "org_trail"
  ...
}

I believe there is an underlying issue with the way AWS has implemented the GetTrail action that prevents the AWS terraform provider from implementing this properly though. It seems that Cloudtrail delegated administrator accounts don't have the ability to perform GetTrail against trails they created as delegated administrators, which are owned by the management account. This is especially weird given what the AWS docs say about the Cloudtrail delegated administration feature:

If you choose a delegated administrator, this member account has administrative permissions on all trails and event data stores in the organization. Adding a delegated administrator does not disrupt the management or operation of the organization's trails or event data stores.

If you run aws --profile security-acct --region us-east-1 cloudtrail describe-trails, you get

{
    "trailList": [
        {
            "Name": "regular-trail-1",
            ...
            "TrailARN": "arn:aws:cloudtrail:us-east-1:<SECURITY ACCOUNT ID>:trail/regular-trail-1",
            ...
        },
        {
            "Name": "regular-trail-2",
            ...
            "TrailARN": "arn:aws:cloudtrail:us-east-1:<SECURITY ACCOUNT ID>:trail/regular-trail-2",
            ...
        },
        {
            "Name": "org_trail",
            ...
            "TrailARN": "arn:aws:cloudtrail:us-east-1:<MANAGEMENT ACCOUNT ID>:trail/org_trail",
            ...
        }
    ]
}

but when you run aws --profile security-acct --region us-east-1 cloudtrail get-trail --name org_trail, you get back

An error occurred (TrailNotFoundException) when calling the GetTrail operation: Unknown trail: org_trail for the user: <SECURITY ACCOUNT ID>

When you try to run the same command but from your management account (i.e. aws --profile management-acct --region us-east-1 cloudtrail get-trail --name org_trail) , it works as expected.

[zampettim]

As a follow up to @andrewnicolalde, if you run aws --profile security-acct --region us-east-1 cloudtrail get-trail --name <arn of org trail> instead of the trail name in the Delegated Admin account, you DO get the details back.

So it would seem a simple fix would be to update the terraform aws provider to use the ARN of the trail instead of the name. The other option would be to use the describe-trails without any option, which DOES return the org trail, and then iterate over the values find the one that matches the Name that is being requested. Not as efficient, but given that the vast majority of users are not going to have more than a small number of trails defined (there isn't much of a reasons to do so) it shouldn't be an issue.

[aavandry]

Could you merge the PR please? Have the same issue

[ssh2003]

and we too, so just waiting (

[popsicleslayer]

I tried the workaround suggested by @andrewnicolalde and it didn't work. Could you please merge the PR- would make it amazing if this got fixed. 🥺

[schroehe]

Having the same issue. Upvoting for fix/PR

[Mummycutie]

Having same issue. Upvoting

[pstsef-blackberry]

Having the same issue. Upvoting.

[waltzbucks]

Upvoting. 🔥

[sidekick-eimantas]

@bflad @ewbankkit This is impacting quite a few people and is critical functionality for cloudtrail management
Could someone have a look at the PR above please?

Thank you

[rapides]

I have the same issue. Upvoting 👍🏻

[taylorwilliams]

Upvoting

[github-actions]

This functionality has been released in v5.25.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.