Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint request exceeding memory limits #41765

Merged
merged 1 commit into from
Nov 28, 2024
Merged

x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint request exceeding memory limits #41765

merged 1 commit into from
Nov 28, 2024

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Nov 24, 2024 •
edited
Loading

Proposed commit message

The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

  • There is also the possibility of using the metrics to retain the in-flight bytes count. This may be worth doing if we actually want to use that gauge metric. If we do that, I'd also think that having a overage count to distinguish failure due to this from all others. What is here is the minimal change required to get the feature.

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@efd6 efd6 added Filebeat Filebeat bugfix Team:Security-Service Integrations Security Service Integrations Team 8.16-candidate backport-8.16 Automated backport with mergify labels Nov 24, 2024
@efd6 efd6 self-assigned this Nov 24, 2024
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 24, 2024
@efd6 efd6 force-pushed the 41764-http_endpoint branch from 189160d to f6527d5 Compare November 24, 2024 21:06
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 24, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Nov 24, 2024
@efd6 efd6 force-pushed the 41764-http_endpoint branch from f6527d5 to f24169b Compare November 24, 2024 22:08
@efd6 efd6 marked this pull request as ready for review November 24, 2024 23:46
@efd6 efd6 requested a review from a team as a code owner November 24, 2024 23:46
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@efd6 efd6 force-pushed the 41764-http_endpoint branch 2 times, most recently from ac98f9f to 7c7251a Compare November 25, 2024 10:25
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 25, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 41764-http_endpoint upstream/41764-http_endpoint
git merge upstream/main
git push upstream 41764-http_endpoint

@efd6 efd6 force-pushed the 41764-http_endpoint branch from 7c7251a to 6f1cd44 Compare November 25, 2024 19:31
@efd6 efd6 added the backport-8.17 Automated backport with mergify label Nov 25, 2024
@efd6 efd6 force-pushed the 41764-http_endpoint branch 4 times, most recently from e9148a8 to 76c7c77 Compare November 25, 2024 21:39
@efd6
x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint re…
a7bf1ce 
…quest exceeding memory limits

The input does not have a way to communicate back-pressure to clients,
potentially leading to unconstrained growth in the publisher event queue
and an OoM event. This change adds a mechanism to keep track of the
total sum of in-flight message bytes from the client in order to allow
the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that
would require a complete understanding of the allocations in the
preparation of event values to be sent to the publisher, but rather uses
the message length as a reasonable proxy.
@efd6 efd6 force-pushed the 41764-http_endpoint branch from 76c7c77 to a7bf1ce Compare November 25, 2024 21:58
kcreddy
kcreddy approved these changes Nov 28, 2024
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a clarification. LGTM otherwise.

x-pack/filebeat/input/http_endpoint/handler.go Show resolved Hide resolved
@efd6 efd6 merged commit 2ad3922 into elastic:main Nov 28, 2024
19 of 22 checks passed
mergify bot pushed a commit that referenced this pull request Nov 28, 2024
@efd6 @mergify
x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint re…
821c15b 
…quest exceeding memory limits (#41765)

The input does not have a way to communicate back-pressure to clients,
potentially leading to unconstrained growth in the publisher event queue
and an OoM event. This change adds a mechanism to keep track of the
total sum of in-flight message bytes from the client in order to allow
the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that
would require a complete understanding of the allocations in the
preparation of event values to be sent to the publisher, but rather uses
the message length as a reasonable proxy.

(cherry picked from commit 2ad3922)
@mergify mergify bot mentioned this pull request Nov 28, 2024
Merged
7 tasks
mergify bot pushed a commit that referenced this pull request Nov 28, 2024
@efd6 @mergify
x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint re…
483bc38 
…quest exceeding memory limits (#41765)

The input does not have a way to communicate back-pressure to clients,
potentially leading to unconstrained growth in the publisher event queue
and an OoM event. This change adds a mechanism to keep track of the
total sum of in-flight message bytes from the client in order to allow
the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that
would require a complete understanding of the allocations in the
preparation of event values to be sent to the publisher, but rather uses
the message length as a reasonable proxy.

(cherry picked from commit 2ad3922)
@mergify mergify bot mentioned this pull request Nov 28, 2024
Merged
7 tasks
mergify bot pushed a commit that referenced this pull request Nov 28, 2024
@efd6 @mergify
x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint re…
800ed8f 
…quest exceeding memory limits (#41765)

The input does not have a way to communicate back-pressure to clients,
potentially leading to unconstrained growth in the publisher event queue
and an OoM event. This change adds a mechanism to keep track of the
total sum of in-flight message bytes from the client in order to allow
the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that
would require a complete understanding of the allocations in the
preparation of event values to be sent to the publisher, but rather uses
the message length as a reasonable proxy.

(cherry picked from commit 2ad3922)
@mergify mergify bot mentioned this pull request Nov 28, 2024
Merged
7 tasks
efd6 added a commit that referenced this pull request Nov 28, 2024
@mergify @efd6
[8.17](backport #41765) x-pack/filebeat/input/http_endpoint: fix hand…
183ce45 
…ling of http_endpoint request exceeding memory limits (#41820)

* x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint request exceeding memory limits (#41765)

The input does not have a way to communicate back-pressure to clients,
potentially leading to unconstrained growth in the publisher event queue
and an OoM event. This change adds a mechanism to keep track of the
total sum of in-flight message bytes from the client in order to allow
the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that
would require a complete understanding of the allocations in the
preparation of event values to be sent to the publisher, but rather uses
the message length as a reasonable proxy.

(cherry picked from commit 2ad3922)

* remove irrelevant changelog entries

---------

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
efd6 added a commit that referenced this pull request Nov 28, 2024
@mergify @efd6
x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint re…
c8212ce 
…quest exceeding memory limits (#41765) (#41819)

The input does not have a way to communicate back-pressure to clients,
potentially leading to unconstrained growth in the publisher event queue
and an OoM event. This change adds a mechanism to keep track of the
total sum of in-flight message bytes from the client in order to allow
the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that
would require a complete understanding of the allocations in the
preparation of event values to be sent to the publisher, but rather uses
the message length as a reasonable proxy.

(cherry picked from commit 2ad3922)

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
efd6 added a commit that referenced this pull request Nov 28, 2024
@mergify @efd6
[8.16](backport #41765) x-pack/filebeat/input/http_endpoint: fix hand…
15ec76a 
…ling of http_endpoint request exceeding memory limits (#41818)

* x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint request exceeding memory limits (#41765)

The input does not have a way to communicate back-pressure to clients,
potentially leading to unconstrained growth in the publisher event queue
and an OoM event. This change adds a mechanism to keep track of the
total sum of in-flight message bytes from the client in order to allow
the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that
would require a complete understanding of the allocations in the
preparation of event values to be sent to the publisher, but rather uses
the message length as a reasonable proxy.

(cherry picked from commit 2ad3922)

* remove irrelevant changelog entries

---------

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kcreddy kcreddy kcreddy approved these changes

Assignees

@efd6 efd6

Labels
8.16-candidate backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify backport-8.17 Automated backport with mergify bugfix Filebeat Filebeat Team:Security-Service Integrations Security Service Integrations Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

x-pack/filebeat/input/http_endpoint: add back pressure mechanism
3 participants
@efd6 @elasticmachine @kcreddy