[Security Solution][Detection Engine] - Update exceptions logic #71512

yctercero · 2020-07-13T19:08:52Z

Summary

This PR addresses a few different issues with how exceptions are handled in the detections engine. We previously were using the following logic:

(rule_query and not exception1) OR (rule_query and not exception2)

This resulted in some confusion in with the logic and expected results. The logic that creates the KQL from the exceptions was updated so that the logic now resembles (visually looks) closer to how the user thinks of it. So given the following:

query & !((item1.entry1 & item1.entry2) || (item2.entry1 & item2.entry2))

...is now being broken down into the following....

query & !((item1.entry1 & item1.entry2) || (item2.entry1 & item2.entry2))

Due to a few limitations, primarily the lack of subquery functionality, we want to find an optimal way of querying the raw events. We've found that we can leverage KQL/ES with exceptions that do not include large value lists as shown above. Any exception items that include large value lists will get dealt with in filter_with_lists.ts.

For context, the exception items are formed using KQL to allow a user to still add a nested exception even when the rule language is lucene. Lucene does not support nested queries, but we are able to send a mix of kql and lucene queries to be built into the proper ES dsl.

Single exception item w/ single entry

Rule Query: host.name: *
KQL: [{"query":"host.name:*", "language":"kuery"}, {"query":"not (event.module:\"traefik\")", "language":"kuery"}]
Exception per UI: event.module - is - traefik
ES query:

{
          query: {
            bool: {
              filter: [
                {
                  bool: {
                    must: [],
                    filter: [
                      {
                        bool: {
                          should: [{ exists: { field: 'host.name' } }],
                          minimum_should_match: 1,
                        },
                      },
                      {
                        bool: {
                          must_not: {
                            bool: {
                              should: [{ match_phrase: { 'event.module': 'traefik' } }],
                              minimum_should_match: 1,
                            },
                          },
                        },
                      },
                    ],
                    should: [],
                    must_not: [],
                  },
                },
                {
                  bool: {
                    filter: [
                      {
                        bool: {
                          should: [
                            { range: { '@timestamp': { gte: '2020-07-14T23:14:51.398Z' } } },
                          ],
                          minimum_should_match: 1,
                        },
                      },
                      {
                        bool: {
                          should: [
                            { range: { '@timestamp': { lte: '2020-07-14T23:16:21.398Z' } } },
                          ],
                          minimum_should_match: 1,
                        },
                      },
                    ],
                  },
                },
                { match_all: {} },
              ],
            },
          },
          sort: [{ '@timestamp': { order: 'asc' } }],
       }

Multiple exception items w/ single entry

Rule Query: host.name: *
KQL: [{"query":"host.name:*","language":"kuery"},{"query":"not ((event.module:\"traefik\") or (source.port:*))","language":"kuery"}]
Exception 1 per UI: event.module - is - traefik
Exception 2 per UI: source.port - exists
ES query:

{
          query: {
            bool: {
              filter: [
                {
                  bool: {
                    must: [],
                    filter: [
                      {
                        bool: {
                          should: [{ exists: { field: 'host.name' } }],
                          minimum_should_match: 1,
                        },
                      },
                      {
                        bool: {
                          must_not: {
                            bool: {
                              should: [{ match_phrase: { 'event.module': 'traefik' } }],
                              minimum_should_match: 1,
                            },
                          },
                        },
                      },
                      {
                        bool: {
                          must_not: {
                            bool: {
                              should: [{ exists: { field: 'source.port' } }],
                              minimum_should_match: 1,
                            },
                          },
                        },
                      },
                    ],
                    should: [],
                    must_not: [],
                  },
                },
                {
                  bool: {
                    filter: [
                      {
                        bool: {
                          should: [
                            { range: { '@timestamp': { gte: '2020-07-14T23:20:18.206Z' } } },
                          ],
                          minimum_should_match: 1,
                        },
                      },
                      {
                        bool: {
                          should: [
                            { range: { '@timestamp': { lte: '2020-07-14T23:21:48.206Z' } } },
                          ],
                          minimum_should_match: 1,
                        },
                      },
                    ],
                  },
                },
                { match_all: {} },
              ],
            },
          },
          sort: [{ '@timestamp': { order: 'asc' } }],
        }

Single exception item w/ multiple entries

Rule Query: host.name: *
KQL: [{"query":"host.name:*","language":"kuery"},{"query":"not (event.module:\"suricata\" and destination.ip:\"10.128.0.33\")","language":"kuery"}]
Exception per UI: event.module - is - suricata and destination.ip - is - 10.128.0.33
ES query:

{
          query: {
            bool: {
              filter: [
                {
                  bool: {
                    must: [],
                    filter: [
                      {
                        bool: {
                          should: [{ exists: { field: 'host.name' } }],
                          minimum_should_match: 1,
                        },
                      },
                      {
                        bool: {
                          must_not: {
                            bool: {
                              filter: [
                                {
                                  bool: {
                                    should: [{ match_phrase: { 'event.module': 'suricata' } }],
                                    minimum_should_match: 1,
                                  },
                                },
                                {
                                  bool: {
                                    should: [{ match_phrase: { 'destination.ip': '10.128.0.33' } }],
                                    minimum_should_match: 1,
                                  },
                                },
                              ],
                            },
                          },
                        },
                      },
                    ],
                    should: [],
                    must_not: [],
                  },
                },
                {
                  bool: {
                    filter: [
                      {
                        bool: {
                          should: [
                            { range: { '@timestamp': { gte: '2020-07-14T20:38:54.668Z' } } },
                          ],
                          minimum_should_match: 1,
                        },
                      },
                      {
                        bool: {
                          should: [
                            { range: { '@timestamp': { lte: '2020-07-14T20:40:24.668Z' } } },
                          ],
                          minimum_should_match: 1,
                        },
                      },
                    ],
                  },
                },
                { match_all: {} },
              ],
            },
          },
          sort: [{ '@timestamp': { order: 'asc' } }],
        }

Single exception item w/ large value list entry

Rule Query: host.name: *
KQL: [{"query":"host.name: * ","language":"kuery"}
Post initial KQL search filters candidate signals against large value list
Exception per UI: source.ip - is not in list -[LIST W/ ONE IP]
ES query:

{"query":{"bool":{"filter":[{"bool":{"must":[],"filter":[{"bool":{"should":[{"exists":{"field":"host.name"}}],"minimum_should_match":1}},{"bool":{"should":[{"match_phrase":{"url.path":"/.reporting-*/_search"}}],"minimum_should_match":1}}],"should":[],"must_not":[]}},{"bool":{"filter":[{"bool":{"should":[{"range":{"@timestamp":{"gte":"2020-07-14T06:07:01.505Z"}}}],"minimum_should_match":1}},{"bool":{"should":[{"range":{"@timestamp":{"lte":"2020-07-14T06:22:31.505Z"}}}],"minimum_should_match":1}}]}},{"match_all":{}}]}},"sort":[{"@timestamp":{"order":"asc"}}]}

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

dhurley14 · 2020-07-14T03:11:04Z

Commenting here but testing this out with the below exceptions yieleded this query (excluding rule query)..

event.module:("traefik") and not source.ip:"69.250.173.110"

Looks good!

…ge value list was blocking signal creation

yctercero · 2020-07-14T06:40:06Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/filter_events_with_list.ts

@@ -37,11 +38,28 @@ export const filterEventsAgainstList = async ({
 return eventSearchResult;
 }

+ const exceptionItemsWithLargeValueLists = exceptionsList.reduce<ExceptionListItemSchema[]>(


NOTE: I found that when an exception item did not include a large value list, it still entered the logic of mapping through the exceptions to find large value lists and resulted in filteredHitsPromises being [[]]. Since it read it as not null, and having a length, we would pass back a SignalSearchResponse with 0 hits and this was passed to the function creating the signals.

tsg · 2020-07-14T12:42:45Z

I have reviewed the DSL from the PR description and they look good.

I have one general comment: the examples here tend to use exceptions that ar in the form of source.ip - is not - 67.173.227.94. While that's possible, it's going to be more likely that users use exceptions that are in the form of source.ip - is - 67.173.227.94. In other words, I expect that the exceptions will be specific in what they exclude. So I think it would be good to have some examples like that.

tsg · 2020-07-14T12:59:07Z

Single exception item w/ simple and large value list entries
Rule Query: host.name: *
KQL: [{"query":"host.name: * ","language":"kuery"},{"query":"url.path:"/.reporting-/_search"","language":"kuery"}]
Exception per UI: url.path - is not - "/.reporting-/_search" and source.ip - is not in list -[LIST W/ ONE IP]
ES query:

Ah, I'm not sure about this one.

The logic that we need to implement, on a signal per signal basis, is:

if (url.path - is not - "/.reporting-*/_search" and source.ip - is not in list -[LIST W/ ONE IP]) then ignore

If we put the url.path: "/.reporting-*/_search" part already in the KQL, then it will ignore all signals with that path, regardless of whether the match the second condition or not. So you might end up implementing:

if (url.path - is not - "/.reporting-*/_search" OR source.ip - is not in list -[LIST W/ ONE IP]) then ignore

It could be that if the exception contains a list, then the whole exception needs to be implemented in code, rather than KQL. Not sure if we can get around that.

x-pack/plugins/security_solution/common/detection_engine/build_exceptions_query.test.ts

x-pack/plugins/security_solution/common/detection_engine/build_exceptions_query.ts

elasticmachine · 2020-07-14T23:07:52Z

Pinging @elastic/siem (Team:SIEM)

…sure better, unit tests need updating

marshallmain

LGTM pending test updates. I'd like to take a closer look at the buildNested function, but that's for another PR.

yctercero · 2020-07-15T05:48:30Z

@elasticmachine merge upstream

patrykkopycinski · 2020-07-15T07:07:37Z

@elasticmachine merge upstream

kibanamachine · 2020-07-15T11:23:24Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d4d8b6a

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
securitySolution	765	+1	764

async chunks size

id	value	diff	baseline
securitySolution	9.0MB	+1.8KB	9.0MB

miscellaneous assets size

id	value	diff	baseline
securitySolution	654.2KB	-236.0B	654.4KB
upgradeAssistant	22.6KB	-5.0B	22.6KB
total	-	-241.0B	-

History

💔 Build #61826 failed b9ec245
💔 Build #61811 failed 974cd88
💔 Build #61795 failed ca674af
💔 Build #61764 failed 6ae3e49
💔 Build #61744 failed 4c08108

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…tic#71512) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Yara Tercero <yara.tercero@elastic.co>

* master: [APM] Add error rates to Service Map popovers (elastic#69520) [Security Solution][Detection Engine] - Update exceptions logic (elastic#71512) [Security Solution] Full screen timeline, Collapse event (elastic#71786) [Security Solution][Exception Modal] Create endpoint exception list if it doesn't already exist (elastic#71807) [Detection Rules] Add 7.9 rules (elastic#71808) [Search] Add telemetry for data plugin search service (elastic#70677) Add @elastic/safer-lodash-set as an alternative to lodash.set (elastic#67452) [tests] Temporarily skipped to promote snapshot

…) (#71847) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Yara Tercero <yara.tercero@elastic.co> Co-authored-by: Yara Tercero <yctercero@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Yara Tercero <yara.tercero@elastic.co>

…tic#71512) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Yara Tercero <yara.tercero@elastic.co>

…) (#72106) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Yara Tercero <yara.tercero@elastic.co> Co-authored-by: Yara Tercero <yctercero@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Yara Tercero <yara.tercero@elastic.co>

elasticmachine · 2021-09-23T14:31:24Z

Pinging @elastic/security-solution (Team: SecuritySolution)

yctercero added 2 commits July 13, 2020 15:07

wip - de exceptions logic

ec08447

Merge branch 'master' of github.com:yctercero/kibana into de_fixes

72d4d91

wip - updated filter events with lists, when item did not include lar…

54f4171

…ge value list was blocking signal creation

yctercero commented Jul 14, 2020

View reviewed changes

yctercero self-assigned this Jul 14, 2020

yctercero added release_note:enhancement Team:SIEM v7.9.0 v8.0.0 labels Jul 14, 2020

Merge branch 'master' of github.com:yctercero/kibana into de_fixes

0c14b62

yctercero added 2 commits July 14, 2020 08:44

updated unit tests post merge conflicts

229bccf

pin exceptions language to kuery

6f941a2

marshallmain reviewed Jul 14, 2020

View reviewed changes

x-pack/plugins/security_solution/common/detection_engine/build_exceptions_query.test.ts Outdated Show resolved Hide resolved

x-pack/plugins/security_solution/common/detection_engine/build_exceptions_query.ts Outdated Show resolved Hide resolved

yctercero marked this pull request as ready for review July 14, 2020 23:07

yctercero requested review from a team as code owners July 14, 2020 23:07

re-re-revert? - went back to older commit that accomadated signal clo…

89b26ab

…sure better, unit tests need updating

yctercero force-pushed the de_fixes branch from 4c08108 to 89b26ab Compare July 15, 2020 02:01

yctercero added 2 commits July 14, 2020 22:12

fix error on import

6ae3e49

updates tests

ca674af

marshallmain approved these changes Jul 15, 2020

View reviewed changes

Merge branch 'master' into de_fixes

974cd88

elasticmachine and others added 2 commits July 15, 2020 01:07

Merge branch 'master' into de_fixes

b9ec245

Fix test

d4d8b6a

cnasikas merged commit 3c9fa99 into elastic:master Jul 15, 2020

cnasikas mentioned this pull request Jul 15, 2020

[7.x] [Security Solution][Detection Engine] - Update exceptions logic (#71512) #71847

Merged

marshallmain mentioned this pull request Jul 16, 2020

[7.9] [Security Solution][Detection Engine] - Update exceptions logic (#71512) #72106

Merged

yctercero deleted the de_fixes branch October 14, 2020 12:01

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detection Engine] - Update exceptions logic #71512

[Security Solution][Detection Engine] - Update exceptions logic #71512

yctercero commented Jul 13, 2020 •

edited

Loading

dhurley14 commented Jul 14, 2020

yctercero Jul 14, 2020

tsg commented Jul 14, 2020

tsg commented Jul 14, 2020 •

edited

Loading

elasticmachine commented Jul 14, 2020

marshallmain left a comment

yctercero commented Jul 15, 2020

patrykkopycinski commented Jul 15, 2020

kibanamachine commented Jul 15, 2020

elasticmachine commented Sep 23, 2021

[Security Solution][Detection Engine] - Update exceptions logic #71512

[Security Solution][Detection Engine] - Update exceptions logic #71512

Conversation

yctercero commented Jul 13, 2020 • edited Loading

Summary

Single exception item w/ single entry

Multiple exception items w/ single entry

Single exception item w/ multiple entries

Single exception item w/ large value list entry

Checklist

For maintainers

dhurley14 commented Jul 14, 2020

yctercero Jul 14, 2020

Choose a reason for hiding this comment

tsg commented Jul 14, 2020

tsg commented Jul 14, 2020 • edited Loading

elasticmachine commented Jul 14, 2020

marshallmain left a comment

Choose a reason for hiding this comment

yctercero commented Jul 15, 2020

patrykkopycinski commented Jul 15, 2020

kibanamachine commented Jul 15, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

async chunks size

miscellaneous assets size

History

elasticmachine commented Sep 23, 2021

yctercero commented Jul 13, 2020 •

edited

Loading

tsg commented Jul 14, 2020 •

edited

Loading