Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Functionbeat] Add support for AWS Kinesis record deaggregation #28240

Closed
jpaskhay opened this issue Oct 4, 2021 · 1 comment · Fixed by #28241
Closed

[Functionbeat] Add support for AWS Kinesis record deaggregation #28240

jpaskhay opened this issue Oct 4, 2021 · 1 comment · Fixed by #28241
Labels
Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Comments

@jpaskhay
Copy link
Contributor

jpaskhay commented Oct 4, 2021

Describe the enhancement:
FunctionBeat does not currently handle the Kinesis aggregated record format, which is commonly used to improve throughput and cost savings with AWS Kinesis. It should be updated to handle consumer deaggregation appropriately so that users can take advantage of the optimization.

Describe a specific use case for the enhancement or feature:
Producers sending in high throughput of small messages (< 1KiB) are likely to use aggregated records to help improve throughput and cost savings. Potential cost savings are largest if they happen to use Kinesis Data Firehose for other Kinesis sinks (e.g. S3) due to the "rounding up" of costs to nearest 5KiB (at the time of this writing).

I'll be submitting a PR for this feature, which has also been built and tested/verified in a local AWS account end to end.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 4, 2021
@andresrc andresrc added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Oct 14, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants