[Security Solution] Dedupe alerts by querying _id before creation #119045
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…into alert-deduplication
marshallmain
added
v8.0.0
release_note:skip
madirey
approved these changes
Nov 18, 2021
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
History
To update your PR or re-run it, just comment with: |
kibanamachine
added a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 18, 2021
…astic#119045) * Dedupe alerts by querying _id before creation * Update alert chunk size * Use aggregations to find existing alert _ids * Remove tightly coupled tests * Add api integration test for alert deduplication * Remove unused import * Cleaner util implementation * Skip flaky test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
💚 Backport successful
This backport PR will be merged automatically after passing CI. |
kibanamachine
added a commit
that referenced
this pull request
Nov 19, 2021
…19045) (#119122) * Dedupe alerts by querying _id before creation * Update alert chunk size * Use aggregations to find existing alert _ids * Remove tightly coupled tests * Add api integration test for alert deduplication * Remove unused import * Cleaner util implementation * Skip flaky test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>
TinLe
pushed a commit
to TinLe/kibana
that referenced
this pull request
Nov 20, 2021
…astic#119045) * Dedupe alerts by querying _id before creation * Update alert chunk size * Use aggregations to find existing alert _ids * Remove tightly coupled tests * Add api integration test for alert deduplication * Remove unused import * Cleaner util implementation * Skip flaky test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
dmlemeshko
pushed a commit
that referenced
this pull request
Nov 29, 2021
…19045) * Dedupe alerts by querying _id before creation * Update alert chunk size * Use aggregations to find existing alert _ids * Remove tightly coupled tests * Add api integration test for alert deduplication * Remove unused import * Cleaner util implementation * Skip flaky test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
New PR because the last one (#118261) was failing to report the status from CI at all, even after closing/reopening and triggering CI re-runs. Not sure what was going wrong.
This PR adds logic to query for candidate alerts by
_idbefore attempting to create them. This fixes 2 bugs.First bug: when a rule runs with a significant "additional lookback", it can find the same alert on multiple consecutive executions. The alerts generated on the later executions were overwriting the existing alert from the previous execution, updating the
@timestampfield but leaving all other fields the same. With this change, the later executions see that the alert already exists and don't modify it.Second bug: for a long time, it has been possible for an alert with the same
_idto exist in multiple concrete indices. As a result, if a rule finds the same alert on multiple runs but the alerts index rolls over between rule runs, then the alert will be duplicated in both the old and new index. Since this new logic queries the alerts index alias for the_id, when an index rollover happens we will still detect that an alert has already been written and won't write a duplicate.Organization
This PR also moves more of the document creation logic into the Persistence Rule type. The
bulkCreateFactorywas responsible for parsing the bulk response before, but the persistence rule type was defining the request and response formats. The reorganization groups logic inalertWithPersistenceso that alerts to index are passed into the function and the alerts that were actually indexed are returned from the function.