Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Download EC2 Instance User Data technique does request userdata properly #353

Closed
agroyz opened this issue May 3, 2023 · 22 comments
Closed

Download EC2 Instance User Data technique does request userdata properly #353

agroyz opened this issue May 3, 2023 · 22 comments
Assignees
@christophetd
Labels
kind/bug Something isn't working platform/aws status/confirmed

Comments

@agroyz
Copy link

agroyz commented May 3, 2023

What is not working?
Download EC2 Instance User Data technique does send userdata properly.

  • generates only one DescribeAccountAttributes CloudTrail event instead of 15
  • requestParameters is empty

"requestParameters": {
"accountAttributeNameSet": {},
"filterSet": {}
},
What OS are you using?
Ubuntu

What is your Stratus Red Team version?
2.5.3
@christophetd
Copy link
Contributor

Related code: https://github.com/DataDog/stratus-red-team/blob/main/v2/internal/attacktechniques/aws/discovery/ec2-get-user-data/main.go

@christophetd
Copy link
Contributor

christophetd commented May 4, 2023 •
edited
Loading

Tested it and it seems to be working properly:

image

image

And the requestParameters looks fine to me:

image

Can you share the full output and a full CloudTrail log sample?

@agroyz
Copy link
Author

agroyz commented May 4, 2023 via email •
edited
Loading

@christophetd
Copy link
Contributor

Thanks, can you share your full Stratus Red Team output? Make sure the technique is clean first:

stratus cleanup --force aws.discovery.ec2-download-user-data

@agroyz
Copy link
Author

agroyz commented May 4, 2023

./stratus detonate aws.discovery.ec2-download-user-data
2023/05/04 21:55:48 Checking your authentication against AWS
2023/05/04 21:55:49 Warming up aws.discovery.ec2-download-user-data
2023/05/04 21:55:49 Initializing Terraform to spin up technique prerequisites
2023/05/04 21:55:51 Applying Terraform to spin up technique prerequisites
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-a5af9dd3
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-4fcc7797
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-d53553b6
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-70167988
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-f6ed2bf8
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-4bd0bfe3
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-9b3dfb7a
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-bdd25338
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-e17fc89a
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-e88e6bd3
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-12c2d31b
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-688eb774
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-a5a278e4
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-6afbf690
2023/05/04 21:55:56 Running ec2:DescribeInstanceAttribute to retrieve userData on i-14ab6de4

@christophetd
Copy link
Contributor

This looks correct, what makes you think that "It looks like IAM user fails to assume the role"?

@agroyz
Copy link
Author

agroyz commented May 5, 2023 •
edited
Loading

From my understanding, the Download EC2 Instance User Data technique uses the initial IAM user, Download-EC2-Instance-User-Data-0504, to execute the technique to create a 'stratus-red-team-get-usr-data-role' role and then assume it. Then makes, 15 DescribeInstanceAttribute API calls.

This is what I see when I look for the events in CloudTrail event history for Download-EC2-Instance-User-Data-0504 user.
image

So when I look though the events, I see access denied for all AssumeRole events. The one successful call, up top, is when I assumed the role manually to test.

And when I run query on event name DescribeInstanceAttribute in CloudTrail event history I get no results back.

Perhaps I'm missing something.

@christophetd
Copy link
Contributor

Can you paste here the whole CSV, or show the whole user agent field? Thanks

@agroyz
Copy link
Author

agroyz commented May 8, 2023

Here is screenshot of the user agent field.
image

@agroyz
Copy link
Author

agroyz commented May 15, 2023

Hi, I wanted to follow up on this ticket. Are there any local logs I can check? We have a detection that this technique should trigger. So it would be great if I could get this to work. Thanks

@christophetd
Copy link
Contributor

Hello,

I'm not sure why this fails in your account, to be frank! Can you share the IAM policy attached to the identity you're using to run Stratus Red Team?

@agroyz
Copy link
Author

agroyz commented May 18, 2023 •
edited
Loading

I understand what the issue is. It looks like the code, in general, does not assume roles correctly and always gets Access Denied.
I found two techniques that assume roles—this one and aws.credential-access.ec2-get-password-data.
When I run the aws.credential-access.ec2-get-password-data technique, I get the same Access Denied error when trying to assume stratus-red-team-ec2-get-password-data-role
I have run these techniques on Ubuntu, windows, and my college ran it on a Mac. And all have the same issue.

aws.discovery.ec2-download-user-data
image

aws.credential-access.ec2-get-password-data
image

We use Full Administrative policy for users used to execute these modules. The policies work because I can manually assume these roles through cli.

@bmp-dot
Copy link

bmp-dot commented May 18, 2023

Hello,

I have the same issue. The role is created correctly "stratus-red-team-get-usr-data-role". However, the role cannot be assumed by Stratus and in the CloudTrail logs you get "AccessDenied" on the AssumeRole events. I can assume the role manually with aws cli using the same IAM user that Stratus is using "aws sts assume-role --role-arn arn:aws:iam::123456789101:role/stratus-red-team-get-usr-data-role --role-session-name stratus --profile stratus"

I am using an Intel Mac and used brew to install Stratus.

@christophetd
Copy link
Contributor

Thanks for the detail, I was able to reproduce this. Will investigate ASAP

@christophetd
Copy link
Contributor

Looks like this is due to the eventual consistency of AWS. When the role is created, it cannot be assumed right away

@christophetd
Copy link
Contributor

PR coming with initial retries to assume the AWS role. Will create a few access denied events on the role created, does that seem acceptable? Given eventual consistency of AWS IAM, I don't think we can do much better

image

christophetd added a commit that referenced this issue May 22, 2023
@christophetd
Implement retry mechanism when creating and assuming an IAM role (closes
3717106 
 #353)
@christophetd christophetd mentioned this issue May 22, 2023
Merged
@christophetd
Copy link
Contributor

@bmp-dot @agroyz Can you give #358 a try? You can use make to create the Stratus binary in bin/

@agroyz
Copy link
Author

agroyz commented May 22, 2023

Ok. Now that I understand the issue, I have a workaround for my tests. Up to now, I've scripted to trigger all the techniques using detonate without warmup. But now I've changed it to warm up first, wait n seconds, and then detonate. This approach solved the issue. Thanks for your help.

@christophetd
Copy link
Contributor

Good to know! Are you able to give #358 a try to confirm it fixes the issue, even without waiting manually?

@agroyz
Copy link
Author

agroyz commented May 24, 2023

I tried this branch and unfortunately same issue. Although I when broke down the attack into warmup and then detonate with 30 wait in between it worked using the released version.

image

@christophetd
Copy link
Contributor

christophetd commented May 25, 2023 •
edited
Loading

Thanks for testing! Did you have any errors? I just added debug output to the PR, could you try compiling and giving another try? Appreciate your help!

FWIW this is what I see:

image

@christophetd
Copy link
Contributor

I will assume that #358 fixes your issue - please let me know otherwise and I'll be happy to revisit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@christophetd christophetd

Labels
kind/bug Something isn't working platform/aws status/confirmed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@christophetd @bmp-dot @agroyz