Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Improve time filter efficiency for Detection Rules #86874

Closed
kostasb opened this issue Dec 23, 2020 · 0 comments · Fixed by #86921
Closed

[Security Solution] Improve time filter efficiency for Detection Rules #86874

kostasb opened this issue Dec 23, 2020 · 0 comments · Fixed by #86921
Assignees
Labels
bug Fixes for quality problems that affect the customer experience Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@kostasb
Copy link

kostasb commented Dec 23, 2020

Description:
The detection engine's time filter should narrow down the search request's timerange as much as possible, in order to keep scope to just the relevant shards. There can be performance impact when timerange is unnecessarily wide, particularly when involving nodes with slower response profile (warm/cold).

Current behavior:
Detection rules may use open-ended time ranges (relevant section here)
This allows for filters like the following:


"filter": [
            {
              "bool": {
                "should": [
                  {
                    "range": {
                      "@timestamp": {
                        "from": "2020-12-04T15:19:07.456Z",
                        "to": null,
                        "include_lower": true,
                        "include_upper": true,
                        "boost": 1
                      }
                    }
                  }
                ],
                "adjust_pure_negative": true,
                "minimum_should_match": "1",
                "boost": 1
              }
            },
            {
              "bool": {
                "should": [
                  {
                    "range": {
                      "@timestamp": {
                        "from": null,
                        "to": "2020-12-04T15:25:07.456Z",
                        "include_lower": true,
                        "include_upper": true,
                        "boost": 1
                      }
                    }
                  }
                ],
                "adjust_pure_negative": true,
                "minimum_should_match": "1",
                "boost": 1
              }
            }
          ]

Expected behavior:
Apply a more precise time filter, binding from and to based on the configured interval/lookback period, in order to avoid searching open-ended time ranges.

@kostasb kostasb added bug Fixes for quality problems that affect the customer experience Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Dec 23, 2020
@dhurley14 dhurley14 added the Team:Detections and Resp Security Detection Response Team label Dec 23, 2020
spong pushed a commit that referenced this issue Jan 6, 2021
…e date range filter (#86921)

## Summary

Ref: #86874



replaces `should` with a single lower bounded and upper bounded date range filter.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
spong pushed a commit to spong/kibana that referenced this issue Jan 6, 2021
…e date range filter (elastic#86921)

## Summary

Ref: elastic#86874



replaces `should` with a single lower bounded and upper bounded date range filter.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
spong pushed a commit to spong/kibana that referenced this issue Jan 6, 2021
…e date range filter (elastic#86921)

## Summary

Ref: elastic#86874



replaces `should` with a single lower bounded and upper bounded date range filter.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
peluja1012 pushed a commit that referenced this issue Jan 6, 2021
…e date range filter (#86921) (#87428)

## Summary

Ref: #86874



replaces `should` with a single lower bounded and upper bounded date range filter.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

Co-authored-by: Devin W. Hurley <devin.hurley@elastic.co>
peluja1012 pushed a commit that referenced this issue Jan 6, 2021
…e date range filter (#86921) (#87429)

## Summary

Ref: #86874



replaces `should` with a single lower bounded and upper bounded date range filter.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

Co-authored-by: Devin W. Hurley <devin.hurley@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
2 participants