add support for kinesis record deaggregation in functionbeat #28241

jpaskhay · 2021-10-04T21:50:43Z

What does this PR do?

This PR adds support for properly deaggregating Kinesis records which are in the aggregated record format.

Why is it important?

This change allows FunctionBeat to properly consume records being sent to Kinesis using the aggregated record format. Users often do this for throughput and cost optimization purposes. The current implementation will instead show single records that have the embedded Protobuf serialized format.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Notes:

Had to add AWS SDK v1 due to current kinesis-aggregation support. There is an open PR that adds v2 support. Whenever it is reviewed and merged, there can be a followup change to remove the v1 dependency.
Performed end to end test to verify functionality using a local functionbeat build and test AWS account. Thanks to @ravinaik1312 for his help on testing this!

How to test this PR locally

Build local version of functionbeat
Deploy to AWS Lambda function
Send aggregated records to Kinesis stream (e.g. using KPL, fluentd, fluent-bit, etc. with the aggregated records option enabled)
Verify records are properly deaggregated in Elastic

Related issues

Closes [Functionbeat] Add support for AWS Kinesis record deaggregation #28240

Use cases

Screenshots

Logs

- tests are heavily based on record generation in https://github.com/awslabs/kinesis-aggregation/blob/master/go/deaggregator/deaggregator_test.go - had to add AWS SDK v1 due to current kinesis-aggregation support (there is an open PR that adds v2 support)

mergify · 2021-10-04T21:51:08Z

This pull request does not have a backport label. Could you fix it @jpaskhay? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-10-04T21:55:36Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Reason: The PR is not allowed to run in the CI yet
Start Time: 2021-10-25T15:47:31.149+0000
Duration: 10 min 42 sec
Commit: 11db8de

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

mergify · 2021-10-07T15:22:51Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b functionbeat-kinesis-deaggregation upstream/functionbeat-kinesis-deaggregation
git merge upstream/master
git push upstream functionbeat-kinesis-deaggregation

…sis-deaggregation

elasticmachine · 2021-10-14T08:08:23Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

kvch · 2021-10-19T07:29:00Z

go.mod

@@ -31,8 +31,10 @@ require (
 	github.com/apoydence/eachers v0.0.0-20181020210610-23942921fe77 // indirect
 	github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5
 	github.com/aws/aws-lambda-go v1.13.3
+	github.com/aws/aws-sdk-go v1.19.48


Is there any reason why you need this version? In beats we use github.com/aws/aws-sdk-go-v2. I would rather not add a new dependency for the same service, if it is not necessary.

Yes, I actually called that out in the Author's Checklist section (felt like the most appropriate place for the note). Unfortunately, the AWS deaggregation library only supports v1 of the SDK. There is an open PR that adds support for v2, but we're still waiting on the repo maintainer to review and merge it.

kvch

Could you please cleanup the depencency? Otherwise great feature and the code impeccable!

kvch · 2021-10-19T07:32:19Z

jenkins run tests

jpaskhay · 2021-10-19T18:59:01Z

Could you please cleanup the depencency? Otherwise great feature and the code impeccable!

Thank you! 😄 Replied inline to your dependency comment.

I'll try to resolve the license linting build failure in the meantime.

- the go mod is embedded in the repo since it is a monorepo, so couldn't specify the licenseFile. linked to the license file in the url field instead ("make notice" succeeded locally)

mergify · 2021-10-20T01:50:03Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b functionbeat-kinesis-deaggregation upstream/functionbeat-kinesis-deaggregation
git merge upstream/master
git push upstream functionbeat-kinesis-deaggregation

…sis-deaggregation

kvch · 2021-10-25T10:16:47Z

jenkins run tests

kvch · 2021-10-25T10:56:55Z

Could you please run make notice and push the updated NOTICE.txt file?

jpaskhay · 2021-10-25T15:48:39Z

Could you please run make notice and push the updated NOTICE.txt file?

Done. Sorry, I should've realized I needed to push that after testing the overrides locally.

kvch · 2021-10-27T13:56:28Z

jenkins run tests

kvch · 2021-10-27T15:48:56Z

Thank you for your contribution!

elasticmachine · 2021-10-27T16:07:57Z

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

(cherry picked from commit 5ac77d3)

…#28681) (cherry picked from commit 5ac77d3) Co-authored-by: Joey Paskhay <joey.paskhay@gmail.com>

…#28241)

(cherry picked from commit 5ac77d3)

…#29340) (cherry picked from commit 5ac77d3) Co-authored-by: Joey Paskhay <joey.paskhay@gmail.com>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 4, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 4, 2021

update entry in CHANGELOG.next.asciidoc

e5edebb

Merge remote-tracking branch 'upstream/master' into functionbeat-kine…

efa62eb

…sis-deaggregation

bvader added Functionbeat and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 13, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 13, 2021

bvader requested a review from jsoriano October 13, 2021 17:47

jsoriano removed their request for review October 14, 2021 08:01

andresrc added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Oct 14, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 14, 2021

kvch self-assigned this Oct 14, 2021

kvch reviewed Oct 19, 2021

View reviewed changes

kvch suggested changes Oct 19, 2021

View reviewed changes

add kinesis-aggregation module to license overrides.json

ad77e36

- the go mod is embedded in the repo since it is a monorepo, so couldn't specify the licenseFile. linked to the license file in the url field instead ("make notice" succeeded locally)

Merge remote-tracking branch 'upstream/master' into functionbeat-kine…

ed095fa

…sis-deaggregation

update NOTICE.txt file w/ updated license info

11db8de

kvch approved these changes Oct 27, 2021

View reviewed changes

kvch merged commit 5ac77d3 into elastic:master Oct 27, 2021

kvch added the backport-v7.16.0 Automated backport with mergify label Oct 27, 2021

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Oct 27, 2021

mergify bot pushed a commit that referenced this pull request Oct 27, 2021

add support for kinesis record deaggregation in functionbeat (#28241)

fc4e8b7

(cherry picked from commit 5ac77d3)

mergify bot mentioned this pull request Oct 27, 2021

[7.16](backport #28241) add support for kinesis record deaggregation in functionbeat #28681

Merged

jpaskhay deleted the functionbeat-kinesis-deaggregation branch October 27, 2021 16:44

kvch pushed a commit that referenced this pull request Oct 28, 2021

add support for kinesis record deaggregation in functionbeat (#28241) (…

558ffdf

…#28681) (cherry picked from commit 5ac77d3) Co-authored-by: Joey Paskhay <joey.paskhay@gmail.com>

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

add support for kinesis record deaggregation in functionbeat (elastic…

910e792

…#28241)

ruflin added the backport-v8.0.0 Automated backport with mergify label Dec 8, 2021

mergify bot mentioned this pull request Dec 8, 2021

[8.0](backport #28241) add support for kinesis record deaggregation in functionbeat #29340

Merged

kvch pushed a commit that referenced this pull request Dec 8, 2021

add support for kinesis record deaggregation in functionbeat (#28241)

7b5ef8c

(cherry picked from commit 5ac77d3)

kvch pushed a commit that referenced this pull request Dec 8, 2021

add support for kinesis record deaggregation in functionbeat (#28241) (…

cd927a6

…#29340) (cherry picked from commit 5ac77d3) Co-authored-by: Joey Paskhay <joey.paskhay@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for kinesis record deaggregation in functionbeat #28241

add support for kinesis record deaggregation in functionbeat #28241

jpaskhay commented Oct 4, 2021 •

edited

Loading

mergify bot commented Oct 4, 2021

elasticmachine commented Oct 4, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

mergify bot commented Oct 7, 2021

elasticmachine commented Oct 14, 2021

kvch Oct 19, 2021

jpaskhay Oct 19, 2021

kvch left a comment

kvch commented Oct 19, 2021

jpaskhay commented Oct 19, 2021

mergify bot commented Oct 20, 2021

kvch commented Oct 25, 2021

kvch commented Oct 25, 2021

jpaskhay commented Oct 25, 2021

kvch commented Oct 27, 2021

kvch commented Oct 27, 2021

elasticmachine commented Oct 27, 2021

add support for kinesis record deaggregation in functionbeat #28241

add support for kinesis record deaggregation in functionbeat #28241

Conversation

jpaskhay commented Oct 4, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Oct 4, 2021

elasticmachine commented Oct 4, 2021 • edited by jenkins-beats-ci bot Loading

❕ Build Aborted

Build stats

🤖 GitHub comments

mergify bot commented Oct 7, 2021

elasticmachine commented Oct 14, 2021

kvch Oct 19, 2021

Choose a reason for hiding this comment

jpaskhay Oct 19, 2021

Choose a reason for hiding this comment

kvch left a comment

Choose a reason for hiding this comment

kvch commented Oct 19, 2021

jpaskhay commented Oct 19, 2021

mergify bot commented Oct 20, 2021

kvch commented Oct 25, 2021

kvch commented Oct 25, 2021

jpaskhay commented Oct 25, 2021

kvch commented Oct 27, 2021

kvch commented Oct 27, 2021

elasticmachine commented Oct 27, 2021

💚 Flaky test report

🤖 GitHub comments

jpaskhay commented Oct 4, 2021 •

edited

Loading

elasticmachine commented Oct 4, 2021 •

edited by jenkins-beats-ci bot

Loading