Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No credential caching when using AWS AssumeRole #37787

Closed
andrewkroh opened this issue Jan 30, 2024 · 3 comments · Fixed by #37788
Closed

No credential caching when using AWS AssumeRole #37787

andrewkroh opened this issue Jan 30, 2024 · 3 comments · Fixed by #37788
Assignees
@andrewkroh
Labels
bug Filebeat Filebeat libbeat Metricbeat Metricbeat needs_team Indicates that the issue/PR needs a Team:* label

Comments

@andrewkroh
Copy link
Member

When using the AssumeRole authentication feature of the AWS SDK, libbeat's helper does not add a credential caching layer. The impact is that every operation that needs to retrieve credentials (i.e. every signed API request) must issue a request to AWS to get a new session token for that one request.

This has multiple impacts:

  • it slows down all requests because there is an added overhead of fetching a new token
  • rate limits can easiliy be reached if lots of requests are being made

AssumeRole session tokens normally having a minimum TTL of 15 minutes (and up to 12 hour max). So the expected behavior is for one request to be made every 15 minutes. Another issue is that the TTL duration is not configurable so it will always be 15m.

Example error message:

failed to sign request: failed to retrieve credentials: operation error STS: AssumeRole, failed to get rate limit token, retry quota exceeded, 2 available, 5 requested

For confirmed bugs, please report:

  • Version: 8.12.0
  • Affected features: Filebeat aws-s3, aws-cloudwatch, auto-discover, Metricbeat aws module
  • Operating System: all
  • Affected configurations:
    • any aws configuration using role_arn
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label labels Jan 30, 2024
@botelastic
Copy link

botelastic bot commented Jan 30, 2024

This issue doesn't have a Team:<team> label.

1 similar comment
@botelastic
Copy link

botelastic bot commented Jan 30, 2024

This issue doesn't have a Team:<team> label.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Jan 30, 2024
@andrewkroh
aws - add credential caching for aws assume role sessions
cdf1040 
Add caching for AWS AssumeRole session credentials are not requested
for every single request. Sessions are valid for 15m by default but without
caching that did not matter. This will speed up requests and stop users from
hitting rate-limiting issues with the STS (session token service).

Fixes elastic#37787
@andrewkroh andrewkroh mentioned this issue Jan 30, 2024
Merged
8 tasks
@andrewkroh andrewkroh self-assigned this Jan 30, 2024
@andrewkroh
Copy link
Member Author

Proposed change: #37788

andrewkroh added a commit that referenced this issue Jan 31, 2024
@andrewkroh
aws - add credential caching for aws assume role sessions (#37788)
a6e5b04 
Add caching so that AWS `AssumeRole` session credentials are not requested for every single request. Sessions are valid for 15m by default but without caching that does not matter. This will speed up requests for users of `role_arn` by removing the overhead of most STS (session token service) calls and stop users from hitting rate-limiting issues with the STS.

Fixes #37787
mergify bot pushed a commit that referenced this issue Jan 31, 2024
@andrewkroh @mergify
aws - add credential caching for aws assume role sessions (#37788)
17958ee 
Add caching so that AWS `AssumeRole` session credentials are not requested for every single request. Sessions are valid for 15m by default but without caching that does not matter. This will speed up requests for users of `role_arn` by removing the overhead of most STS (session token service) calls and stop users from hitting rate-limiting issues with the STS.

Fixes #37787

(cherry picked from commit a6e5b04)
mergify bot pushed a commit that referenced this issue Jan 31, 2024
@andrewkroh @mergify
aws - add credential caching for aws assume role sessions (#37788)
4eec890 
Add caching so that AWS `AssumeRole` session credentials are not requested for every single request. Sessions are valid for 15m by default but without caching that does not matter. This will speed up requests for users of `role_arn` by removing the overhead of most STS (session token service) calls and stop users from hitting rate-limiting issues with the STS.

Fixes #37787

(cherry picked from commit a6e5b04)

# Conflicts:
#	x-pack/libbeat/common/aws/credentials.go
#	x-pack/libbeat/docs/aws-credentials-config.asciidoc
andrewkroh added a commit that referenced this issue Jan 31, 2024
@andrewkroh
aws - add credential caching for aws assume role sessions (#37788)
b390f0b 
Add caching so that AWS `AssumeRole` session credentials are not requested for every single request. Sessions are valid for 15m by default but without caching that does not matter. This will speed up requests for users of `role_arn` by removing the overhead of most STS (session token service) calls and stop users from hitting rate-limiting issues with the STS.

Fixes #37787
andrewkroh added a commit that referenced this issue Jan 31, 2024
@mergify @andrewkroh
[8.12](backport #37788) aws - add credential caching for aws assume r…
b32a860 
…ole sessions (#37808)

Add caching so that AWS `AssumeRole` session credentials are not requested for every single request. Sessions are valid for 15m by default but without caching that does not matter. This will speed up requests for users of `role_arn` by removing the overhead of most STS (session token service) calls and stop users from hitting rate-limiting issues with the STS.

Fixes #37787

(cherry picked from commit a6e5b04)

---------

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Scholar-Li pushed a commit to Scholar-Li/beats that referenced this issue Feb 5, 2024
@andrewkroh @Scholar-Li
aws - add credential caching for aws assume role sessions (elastic#37788
6f4fdd5 
)

Add caching so that AWS `AssumeRole` session credentials are not requested for every single request. Sessions are valid for 15m by default but without caching that does not matter. This will speed up requests for users of `role_arn` by removing the overhead of most STS (session token service) calls and stop users from hitting rate-limiting issues with the STS.

Fixes elastic#37787
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrewkroh andrewkroh

Labels
bug Filebeat Filebeat libbeat Metricbeat Metricbeat needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

aws - add credential caching for aws assume role sessions andrewkroh/beats
1 participant
@andrewkroh