Stratus Does Not Use Instance Profile On EC2

[mrugank-canva]

What is not working?

When trying to run Stratus Red Team in an EC2 Instance, we found that it wasn’t able to leverage the instance credentials allocated to the instance. It gives us the following error:

Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

Please see https://registry.terraform.io/providers/hashicorp/aws
for more information about providing credentials.

Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestCanceled: EC2 IMDS access disabled via AWS_EC2_METADATA_DISABLED env var

We are calling Stratus programmatically as follows:

import (
...
stratusrunner "github.com/datadog/stratus-red-team/v2/pkg/stratus/runner"
...

)
stratusRunner := stratusrunner.NewRunner(ttp,stratusrunner.StratusRunnerNoForce)
...
stratusRunner.Detonate()

When we try to run Stratus as a binary on the EC2 instance to execute the same technique, we run into the same error message:

root@ip-10-128-184-185:~# ./stratus detonate aws.defense-evasion.cloudtrail-stop
2023/06/05 04:11:07 Checking your authentication against AWS
2023/06/05 04:11:07 Warming up aws.defense-evasion.cloudtrail-stop
2023/06/05 04:11:07 Initializing Terraform to spin up technique prerequisites
2023/06/05 04:11:09 Applying Terraform to spin up technique prerequisites
2023/06/05 04:11:10 Error during warm up. Cleaning up technique prerequisites with terraform destroy
2023/06/05 04:11:11 unable to run terraform apply on prerequisite: unable to apply Terraform: exit status 1

Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

Please see https://registry.terraform.io/providers/hashicorp/aws
for more information about providing credentials.

Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestCanceled: EC2 IMDS access disabled via AWS_EC2_METADATA_DISABLED env var


  with provider["registry.terraform.io/hashicorp/aws"],
  on main.tf line 9, in provider "aws":
   9: provider "aws" {

This is because both the binary and the SDK are leveraging the same terraform configuration for the AWS provider:

provider aws {
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_get_ec2_platforms      = true
    skip_metadata_api_check     = true
}

When we unset the skip_metadata_api_check above, Terraform was able to run just fine using the assigned instance profile.

We noticed that the change was introduced very early here.

These changes specifically stop terraform (and thereby Stratus) from being able to discover the instance profile and use it on an EC2 instance.

Our questions for you:

• Were these flags introduced into the terraform by design?
• Would you be okay with us providing a PR to remove these checks if they weren’t intentional?

What OS are you using?
20.04.1-Ubuntu

What is your Stratus Red Team version?
stratus version
2.5.6

Files in $HOME/.stratus-red-team?
ls -lahR

root@ip-10-128-184-185:~/.stratus-red-team# ls -lahR
.:
total 59M
drwxr--r--  4 root root 4.0K Jun  5 04:11 .
drwx------ 12 root root 4.0K Jun  5 03:44 ..
drwxr--r--  3 root root 4.0K Jun  5 01:43 aws.credential-access.secretsmanager-retrieve-secrets
drwxr--r--  3 root root 4.0K Jun  5 04:12 aws.defense-evasion.cloudtrail-stop
-rwx------  1 root root  59M Jun  2 22:42 terraform

./aws.credential-access.secretsmanager-retrieve-secrets:
total 20K
drwxr--r-- 3 root root 4.0K Jun  5 01:43 .
drwxr--r-- 4 root root 4.0K Jun  5 04:11 ..
-rw-r--r-- 1 root root  997 Jun  5 01:43 main.tf
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 .terraform
-rw-r--r-- 1 root root    0 Jun  5 01:35 .terraform-initialized
-rw-r--r-- 1 root root 2.4K Jun  5 01:35 .terraform.lock.hcl

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 .
drwxr--r-- 3 root root 4.0K Jun  5 01:43 ..
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 providers

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 ..
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 registry.terraform.io

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers/registry.terraform.io:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 ..
drwxr-xr-x 4 root root 4.0K Jun  5 01:35 hashicorp

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers/registry.terraform.io/hashicorp:
total 16K
drwxr-xr-x 4 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 ..
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 aws
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 random

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers/registry.terraform.io/hashicorp/aws:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 4 root root 4.0K Jun  5 01:35 ..
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 3.76.1

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers/registry.terraform.io/hashicorp/aws/3.76.1:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 ..
drwxr-xr-x 2 root root 4.0K Jun  5 01:35 linux_arm64

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers/registry.terraform.io/hashicorp/aws/3.76.1/linux_arm64:
total 238M
drwxr-xr-x 2 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 ..
-rwxr-xr-x 1 root root 238M Jun  5 01:35 terraform-provider-aws_v3.76.1_x5

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers/registry.terraform.io/hashicorp/random:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 4 root root 4.0K Jun  5 01:35 ..
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 3.5.1

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers/registry.terraform.io/hashicorp/random/3.5.1:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 ..
drwxr-xr-x 2 root root 4.0K Jun  5 01:35 linux_arm64

./aws.credential-access.secretsmanager-retrieve-secrets/.terraform/providers/registry.terraform.io/hashicorp/random/3.5.1/linux_arm64:
total 13M
drwxr-xr-x 2 root root 4.0K Jun  5 01:35 .
drwxr-xr-x 3 root root 4.0K Jun  5 01:35 ..
-rwxr-xr-x 1 root root  13M Jun  5 01:35 terraform-provider-random_v3.5.1_x5

./aws.defense-evasion.cloudtrail-stop:
total 20K
drwxr--r-- 3 root root 4.0K Jun  5 04:12 .
drwxr--r-- 4 root root 4.0K Jun  5 04:11 ..
-rw-r--r-- 1 root root 1.9K Jun  5 04:12 main.tf
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 .terraform
-rw-r--r-- 1 root root    0 Jun  5 04:11 .terraform-initialized
-rw-r--r-- 1 root root 2.4K Jun  5 04:11 .terraform.lock.hcl

./aws.defense-evasion.cloudtrail-stop/.terraform:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 .
drwxr--r-- 3 root root 4.0K Jun  5 04:12 ..
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 providers

./aws.defense-evasion.cloudtrail-stop/.terraform/providers:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 ..
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 registry.terraform.io

./aws.defense-evasion.cloudtrail-stop/.terraform/providers/registry.terraform.io:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 ..
drwxr-xr-x 4 root root 4.0K Jun  5 04:11 hashicorp

./aws.defense-evasion.cloudtrail-stop/.terraform/providers/registry.terraform.io/hashicorp:
total 16K
drwxr-xr-x 4 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 ..
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 aws
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 random

./aws.defense-evasion.cloudtrail-stop/.terraform/providers/registry.terraform.io/hashicorp/aws:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 4 root root 4.0K Jun  5 04:11 ..
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 3.76.1

./aws.defense-evasion.cloudtrail-stop/.terraform/providers/registry.terraform.io/hashicorp/aws/3.76.1:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 ..
drwxr-xr-x 2 root root 4.0K Jun  5 04:11 linux_arm64

./aws.defense-evasion.cloudtrail-stop/.terraform/providers/registry.terraform.io/hashicorp/aws/3.76.1/linux_arm64:
total 238M
drwxr-xr-x 2 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 ..
-rwxr-xr-x 1 root root 238M Jun  5 04:11 terraform-provider-aws_v3.76.1_x5

./aws.defense-evasion.cloudtrail-stop/.terraform/providers/registry.terraform.io/hashicorp/random:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 4 root root 4.0K Jun  5 04:11 ..
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 3.5.1

./aws.defense-evasion.cloudtrail-stop/.terraform/providers/registry.terraform.io/hashicorp/random/3.5.1:
total 12K
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 ..
drwxr-xr-x 2 root root 4.0K Jun  5 04:11 linux_arm64

./aws.defense-evasion.cloudtrail-stop/.terraform/providers/registry.terraform.io/hashicorp/random/3.5.1/linux_arm64:
total 13M
drwxr-xr-x 2 root root 4.0K Jun  5 04:11 .
drwxr-xr-x 3 root root 4.0K Jun  5 04:11 ..
-rwxr-xr-x 1 root root  13M Jun  5 04:11 terraform-provider-random_v3.5.1_x5

[christophetd]

First of all, great to hear you're using the programmatic interface - and thank you for the very clear report!

From the docs, it looks like provider configuration takes precedence over environment variables, so setting AWS_EC2_METADATA_DISABLED before running your code would not help - can you confirm?

$ export AWS_EC2_METADATA_DISABLED=false
$ go run your-code.go

[christophetd]

For background, these flags were introduced to speed up the Terraform provisioning. But I 100% agree that there should be a way to have Stratus work when using instance roles

[mrugank-canva]

Hey @christophetd thanks so much for the swift reply.

We tried setting the variable AWS_EC2_METADATA_DISABLED=false and then running it, it doesn't help (IMDS is still unaccessible).

How best do you suggest we proceed with getting Stratus to work with instance roles?

[christophetd]

I ran a few tests, and setting skip_metadata_api_check = false (default value) in the provider configuration seems to have little to no impact on the warmup time. I tracked down this parameter in the Terraform provider Go code, and it seems to only end up controlling whether the IMDS endpoint is used (at all) for credentials when nothing else is available.

See

• https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/provider/provider.go#L493-L499
• https://github.com/hashicorp/aws-sdk-go-base/blob/main/aws_config.go#L353-L358
• https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list:~:text=AWS_EC2_METADATA_DISABLED

Based on this, I believe it would take more time only when no credentials are configured at all when Terraform is run (i.e., trying to fall back to IMDS and timing out). But since we perform an authentication check before running Terraform, this should literally have 0 impact.

So we're good to go to remove this flag! Great catch. PR appreciated, otherwise I will look into fixing it in the coming weeks!

[mrugank-canva]

Great, thank you! I'll raise one for you today :)

[mrugank-canva]

I've raised #368 🙂

[christophetd]

Will be released shortly as part of v2.5.7. Thanks for the contribution!