[Filebeat] [AWS] Add support to source AWS cloudwatch logs from linked accounts #41188

Kavindu-Dodan · 2024-10-09T19:03:05Z

Proposed commit message

PR adds support to Cloudwatch logs from source linked accounts. This is implemented by using existing configuration log_group_arn and mapping it to LogGroupIdentifier of FilterLogEvents API [1]

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Note for reviewers

You could review commit by commit for better understanding of the changes

How to test this PR locally

This require a linked cloudwatch account. If already has one, then,

Push logs to a newly created log group OR use an already existing log group in a source account
- Note - you may use data-gen Go program to generate and push logs to your log group (using output CLOUDWATCH_LOG) [2]
Configure filebeat cloudwatch input with log group ARN to log_group_arn
Run filebeat and observe filebeat logs in Kibana discover

Related issues

Addresses: #36642
And makes #36645 PR obsolete
closes #37681

Next step

Utilize includeLinkedAccounts when dealing with prefixes. To be done in a dedicated PR.

[1] - https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_FilterLogEvents.html
[1] - https://github.com/Kavindu-Dodan/data-gen

elasticmachine · 2024-10-09T19:03:30Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

mergify · 2024-10-09T19:03:58Z

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

CHANGELOG.next.asciidoc

x-pack/filebeat/docs/inputs/input-aws-cloudwatch.asciidoc

elasticmachine · 2024-10-10T07:18:09Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

belimawr

There are just a couple of small things to fix:

The changelog entry
Addressing the comment from @kaiyan-sheng.

CHANGELOG.next.asciidoc

…meter, field renaming Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan · 2024-10-10T14:31:59Z

@kaiyan-sheng @belimawr thanks for the reviews, I added proposed changes with my latest commit :) appreciate another look

x-pack/filebeat/input/awscloudwatch/input.go

x-pack/filebeat/input/awscloudwatch/cloudwatch.go

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

x-pack/filebeat/input/awscloudwatch/input.go

Kavindu-Dodan · 2024-10-10T22:32:56Z

@belimawr appreciate another review from you :)

mergify · 2024-10-14T16:28:57Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/filebeat-support-linked-accounts upstream/feat/filebeat-support-linked-accounts
git merge upstream/main
git push upstream feat/filebeat-support-linked-accounts

…d accounts (#41188) * use LogGroupIdentifier fiter instead of LogGroupName and related parameter, field renaming Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * configuration parsing to support arn & linked accounts Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * document the ARN usage Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * code review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * code review change - fix typo Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> (cherry picked from commit 42f2d41)

…udwatch logs from linked accounts (#41240) * [Filebeat] [AWS] Add support to source AWS cloudwatch logs from linked accounts (#41188) * use LogGroupIdentifier fiter instead of LogGroupName and related parameter, field renaming Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * configuration parsing to support arn & linked accounts Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * document the ARN usage Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog entry Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * code review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * code review change - fix typo Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> (cherry picked from commit 42f2d41) * fix backport commit Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com> Co-authored-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan added the enhancement label Oct 9, 2024

Kavindu-Dodan requested review from a team as code owners October 9, 2024 19:03

Kavindu-Dodan requested review from belimawr and leehinman October 9, 2024 19:03

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 9, 2024

Kavindu-Dodan added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Oct 9, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 9, 2024

mergify bot assigned Kavindu-Dodan Oct 9, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 9, 2024

elastic deleted a comment from mergify bot Oct 9, 2024

kaiyan-sheng reviewed Oct 9, 2024

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

kaiyan-sheng reviewed Oct 10, 2024

View reviewed changes

x-pack/filebeat/docs/inputs/input-aws-cloudwatch.asciidoc Show resolved Hide resolved

pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Oct 10, 2024

pierrehilbert requested a review from faec October 10, 2024 07:18

belimawr reviewed Oct 10, 2024

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

Kavindu-Dodan added 5 commits October 10, 2024 07:19

use LogGroupIdentifier fiter instead of LogGroupName and related para…

dfd2e81

…meter, field renaming Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

configuration parsing to support arn & linked accounts

15d3e9d

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

document the ARN usage

34325bb

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

add changelog entry

3d22e98

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

code review changes

e280d23

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan force-pushed the feat/filebeat-support-linked-accounts branch from 90fb3d5 to e280d23 Compare October 10, 2024 14:29

Kavindu-Dodan requested review from belimawr and kaiyan-sheng October 10, 2024 14:30

kaiyan-sheng reviewed Oct 10, 2024

View reviewed changes

x-pack/filebeat/input/awscloudwatch/input.go Outdated Show resolved Hide resolved

kaiyan-sheng reviewed Oct 10, 2024

View reviewed changes

x-pack/filebeat/input/awscloudwatch/cloudwatch.go Show resolved Hide resolved

code review change - fix typo

e8f2ef9

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

stefanhettich reviewed Oct 10, 2024

View reviewed changes

x-pack/filebeat/input/awscloudwatch/input.go Show resolved Hide resolved

Kavindu-Dodan mentioned this pull request Oct 10, 2024

[Filebeat] [AWS] Support getting cloudwatch logs from linked cross-account monitoring source accounts #36642

Closed

kaiyan-sheng approved these changes Oct 10, 2024

View reviewed changes

Kavindu-Dodan mentioned this pull request Oct 11, 2024

[Filebeat] [AWS] add support to source logs from AWS linked source accounts when using log_group_name_prefix #41206

Merged

6 tasks

belimawr approved these changes Oct 14, 2024

View reviewed changes

Merge branch 'main' into feat/filebeat-support-linked-accounts

3142139

Kavindu-Dodan merged commit 42f2d41 into elastic:main Oct 15, 2024
22 checks passed

mergify bot mentioned this pull request Oct 15, 2024

[8.x](backport #41188) [Filebeat] [AWS] Add support to source AWS cloudwatch logs from linked accounts #41240

Merged

6 tasks

mergify bot mentioned this pull request Oct 15, 2024

[8.x](backport #41206) [Filebeat] [AWS] add support to source logs from AWS linked source accounts when using log_group_name_prefix #41247

Merged

6 tasks

This was referenced Oct 15, 2024

[AWS] Requests include loggroups from linked accounts #37681

Closed

[AWS] Support linked accounts when using log_group_name_prefix to select log groups elastic/integrations#11457

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] [AWS] Add support to source AWS cloudwatch logs from linked accounts #41188

[Filebeat] [AWS] Add support to source AWS cloudwatch logs from linked accounts #41188

Kavindu-Dodan commented Oct 9, 2024 •

edited by kaiyan-sheng

Loading

elasticmachine commented Oct 9, 2024

mergify bot commented Oct 9, 2024

elasticmachine commented Oct 10, 2024

belimawr left a comment

Kavindu-Dodan commented Oct 10, 2024

Kavindu-Dodan commented Oct 10, 2024

mergify bot commented Oct 14, 2024

[Filebeat] [AWS] Add support to source AWS cloudwatch logs from linked accounts #41188

[Filebeat] [AWS] Add support to source AWS cloudwatch logs from linked accounts #41188

Conversation

Kavindu-Dodan commented Oct 9, 2024 • edited by kaiyan-sheng Loading

Proposed commit message

Checklist

Note for reviewers

How to test this PR locally

Related issues

Next step

elasticmachine commented Oct 9, 2024

mergify bot commented Oct 9, 2024

elasticmachine commented Oct 10, 2024

belimawr left a comment

Choose a reason for hiding this comment

Kavindu-Dodan commented Oct 10, 2024

Kavindu-Dodan commented Oct 10, 2024

mergify bot commented Oct 14, 2024

Kavindu-Dodan commented Oct 9, 2024 •

edited by kaiyan-sheng

Loading