Please support AWS IAM Instance profiles! #12464

davidmco65 · 2019-06-06T15:28:09Z

I can't use the Functionbeat on my EC2 server because it doesn't appear to use the default credential provider chain. Meaning that I have a Role assigned to me EC2 instance that gives it permission to access the bucket specified in the configuration, but the deploy function doesn't see use it and therefore has no permissions to do anything with S3.

I'm trying to deploy the functionbeat from our EC2 server and it's not possible given this missing functionality. I've got my entire ELK stack deployment scripted except for this functionbeat.

exekias · 2019-07-16T10:14:33Z

I think #12727 would fix this if used in Functionbeat? cc @kvch

kvch · 2019-07-16T10:16:45Z

Yes, I think so. But still, it needs a bit of work in Functionbeat. :)

kaiyan-sheng · 2019-07-16T13:28:05Z

@davidmco65 Thanks for creating this issue. When you mentioned default credential provider chain, do you mean the aws credential file under ~/.aws/credentials?

davidmco65 · 2019-07-17T03:37:42Z

@kaiyan-sheng No, the default credential provider chain runs through a set of potential credential providers. One of which is the ~/.aws/credentials file, but others include the EC2 Instance Profile - a role that's assigned to the EC2 instance running the beat and would give the beat the permissions it needs to access the resources it needs, i.e., S3 bucket. In our environment, we literally cannot use access key id/secret key, so there is no way to configure the function beat that will allow it to install properly.

Thanks!

jderose9 · 2019-10-31T21:16:33Z

I need to be able to specify a role to assume in a child account via a named profile present in the ~/.aws/config file. This is typically passed as --profile parameter on aws CLI commands and many other tools that interact with AWS.

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

Thiago-Dantas · 2020-01-06T14:40:49Z

I also need this but in Metricbeat, should I open a new issue?

kaiyan-sheng · 2020-01-06T15:25:43Z

@Thiago-Dantas I'm planning to use this same issue to track for Metricbeat as well! Thanks!

jeffvestal · 2020-04-09T14:30:18Z

I have a user who is interested in this feature so they don't have to store creds locally on the ec2 instance.
@kaiyan-sheng ping me if you need more details.
Thanks!

kvch · 2020-04-15T14:32:24Z

@kaiyan-sheng Are you sure this issue can be closed? The original author requested the feature for Functionbeat. However, your PR only adds support in metrcibeat and filebeat.

kvch · 2020-04-16T06:46:37Z

No, it does not. Functionbeat depends on the AWS SDK to handle credentials. Thus, it only supports passing those via environment variables. Thank you for reopening.

prabaharang · 2020-04-23T01:30:22Z

If Functionbeat uses AWS SDK, what is the credential provider chain is being used. Wouldn't AWS SDK allow to use Instance profile credentials by default (in case of Default Credential Provider Chain)?

kvch · 2020-04-27T16:50:46Z

I guess yes, but we haven't tested it yet. Also, all of the information I was able to found on the subject is for the v1 of aws-sdk-go. However, in Beats we use aws-sdk-go-v2, so there migth be differences between the docs of v1 and reality of v2.

kaiyan-sheng · 2020-06-16T17:16:03Z

Hi @kvch, since this issue is related to Functionbeat, is it ok if I assign it to you instead?

kvch · 2020-06-24T06:53:30Z

@kaiyan-sheng I've just assinged myself. Thanks for the ping.

s-t-ev · 2020-08-18T20:12:00Z

@kvch Any luck with implementing this for functionbeat? Thanks!

BrunoMCBraga · 2020-12-28T12:41:18Z

Fellas, i come from the future and this problem still persists. Functionbeat seems to still be using AWS env variable credentials. I am currently playing with Elastic stack using one single node on as private network and i cannot afford to put my credentials there, specially when the list of permissions necessary: https://www.elastic.co/guide/en/beats/functionbeat/current/iam-permissions.html#iam-permissions-cloudwatch

Is quite big. Looks like a disaster ready to happen.

## What does this PR do? This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced: 1. `access_key_id`, `secret_access_key` and/or `session_token` for tokens ```yaml functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}' functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}' functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}' ``` 2. `role_arn` for assuming IAM roles ```yaml functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb ``` 3. `credential_profile_name` and/or `shared_credential_file` for credential files ```yaml functionbeat.provider.aws.credential_profile_name: fnb-aws functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials ``` ## Why is it important? Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat. ## Related issues Based on #17658 Closes #12464 Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>

This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced: 1. `access_key_id`, `secret_access_key` and/or `session_token` for tokens ```yaml functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}' functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}' functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}' ``` 2. `role_arn` for assuming IAM roles ```yaml functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb ``` 3. `credential_profile_name` and/or `shared_credential_file` for credential files ```yaml functionbeat.provider.aws.credential_profile_name: fnb-aws functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials ``` Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat. Based on elastic#17658 Closes elastic#12464 Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co> (cherry picked from commit 5e6558b)

…3386) This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced: 1. `access_key_id`, `secret_access_key` and/or `session_token` for tokens ```yaml functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}' functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}' functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}' ``` 2. `role_arn` for assuming IAM roles ```yaml functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb ``` 3. `credential_profile_name` and/or `shared_credential_file` for credential files ```yaml functionbeat.provider.aws.credential_profile_name: fnb-aws functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials ``` Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat. Based on #17658 Closes #12464 Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co> (cherry picked from commit 5e6558b)

ruflin added the Team:Integrations Label for the Integrations team label Jun 7, 2019

kaiyan-sheng added the Functionbeat label Jun 10, 2019

andresrc added the [zube]: Investigate label Jun 24, 2019

andresrc assigned kaiyan-sheng Jun 24, 2019

exekias added the enhancement label Jul 16, 2019

andresrc added Team:Platforms Label for the Integrations - Platforms team [zube]: Inbox [zube]: Ready and removed [zube]: Investigate [zube]: Inbox labels Jan 27, 2020

zube bot added [zube]: Inbox [zube]: Ready and removed [zube]: Ready [zube]: Inbox labels Jan 27, 2020

kaiyan-sheng mentioned this issue Feb 14, 2020

Add more keystore backends to support Functionbeat cloud providers #15879

Closed

kaiyan-sheng changed the title ~~Please support AWS Instance profiles!~~ Please support AWS IAM Instance profiles! Apr 9, 2020

kaiyan-sheng mentioned this issue Apr 10, 2020

Add support for IAM role arn in aws config #17658

Merged

6 tasks

kaiyan-sheng closed this as completed in #17658 Apr 15, 2020

zube bot added [zube]: Done and removed [zube]: Ready labels Apr 15, 2020

kaiyan-sheng mentioned this issue Apr 15, 2020

Cherry-pick #17658 to 7.x: Add support for IAM role arn in aws config #17726

Merged

6 tasks

zube bot assigned kvch Apr 15, 2020

zube bot closed this as completed Apr 15, 2020

zube bot added [zube]: Done and removed [zube]: Inbox labels Apr 15, 2020

zube bot unassigned kvch Apr 15, 2020

zube bot reopened this Apr 15, 2020

zube bot added [zube]: Inbox and removed [zube]: Done labels Apr 15, 2020

andresrc added [zube]: Backlog and removed Team:Platforms Label for the Integrations - Platforms team [zube]: Inbox labels Apr 19, 2020

kvch assigned kvch and unassigned kaiyan-sheng Jun 24, 2020

kvch mentioned this issue Jan 4, 2021

Provide more ways to set AWS credentials in Functionbeat #23344

Merged

4 tasks

kvch closed this as completed in #23344 Jan 7, 2021

zube bot added [zube]: Done and removed [zube]: Backlog labels Jan 7, 2021

kvch mentioned this issue Jan 7, 2021

Cherry-pick #23344 to 7.x: Provide more ways to set AWS credentials in Functionbeat #23386

Merged

4 tasks

zube bot removed the [zube]: Done label Apr 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Please support AWS IAM Instance profiles! #12464

Please support AWS IAM Instance profiles! #12464

davidmco65 commented Jun 6, 2019

exekias commented Jul 16, 2019

kvch commented Jul 16, 2019

kaiyan-sheng commented Jul 16, 2019

davidmco65 commented Jul 17, 2019

jderose9 commented Oct 31, 2019 •

edited

Loading

Thiago-Dantas commented Jan 6, 2020

kaiyan-sheng commented Jan 6, 2020

jeffvestal commented Apr 9, 2020 •

edited by kvch

Loading

kvch commented Apr 15, 2020

kvch commented Apr 16, 2020 •

edited

Loading

prabaharang commented Apr 23, 2020

kvch commented Apr 27, 2020

kaiyan-sheng commented Jun 16, 2020

kvch commented Jun 24, 2020

s-t-ev commented Aug 18, 2020

BrunoMCBraga commented Dec 28, 2020

Please support AWS IAM Instance profiles! #12464

Please support AWS IAM Instance profiles! #12464

Comments

davidmco65 commented Jun 6, 2019

exekias commented Jul 16, 2019

kvch commented Jul 16, 2019

kaiyan-sheng commented Jul 16, 2019

davidmco65 commented Jul 17, 2019

jderose9 commented Oct 31, 2019 • edited Loading

Thiago-Dantas commented Jan 6, 2020

kaiyan-sheng commented Jan 6, 2020

jeffvestal commented Apr 9, 2020 • edited by kvch Loading

kvch commented Apr 15, 2020

kvch commented Apr 16, 2020 • edited Loading

prabaharang commented Apr 23, 2020

kvch commented Apr 27, 2020

kaiyan-sheng commented Jun 16, 2020

kvch commented Jun 24, 2020

s-t-ev commented Aug 18, 2020

BrunoMCBraga commented Dec 28, 2020

jderose9 commented Oct 31, 2019 •

edited

Loading

jeffvestal commented Apr 9, 2020 •

edited by kvch

Loading

kvch commented Apr 16, 2020 •

edited

Loading