[Security Solutions][Detection Engine] Adds threat matching API and rule type

[FrankHassanabad]

Summary

This is the backend, first iteration of threat matching API and rule type. You see elements using the backend API on the front end but cannot use the UI to add or edit a threshold rule with this PR.

Screen shots of it running in the UI elements that do work:

Usage
Since this is only backend API work and does not have the front end add/edit at the moment, you can use the existing UI's (for the most part) to validate the work here through CURL scripts below:

Go to the folder:

/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts

And post a small ECS threat mapping to the index called mock-threat-list:

./create_threat_mapping.sh

Then to post a small number of threats that represent simple port numbers you can run:

./create_threat_data.sh

However, feel free to also manually create them directly in your dev tools like so:

# Posts a threat list item called some-name with an IP but change these out for valid data in your system
PUT mock-threat-list-1/_doc/9999
{
  "@timestamp": "2020-09-09T20:30:45.725Z",
  "host": {
    "name": "some-name",
    "ip": "127.0.0.1"
  }
}

# Posts a destination port number to watch
PUT mock-threat-list-1/_doc/10000
{
  "@timestamp": "2020-09-08T20:30:45.725Z",
  "destination": {
    "port": "443"
  }
}

# Posts a source port number to watch
PUT mock-threat-list-1/_doc/10001
{
  "@timestamp": "2020-09-08T20:30:45.725Z",
  "source": {
    "port": "443"
  }
}

Then you can post a threat match rule:

./post_rule.sh ./rules/queries/query_with_threat_mapping.json

Click here to see Response

{
  "actions": [],
  "author": [],
  "created_at": "2020-09-16T04:25:58.041Z",
  "created_by": "yo",
  "description": "Query with a threat mapping",
  "enabled": true,
  "exceptions_list": [],
  "false_positives": [],
  "from": "now-6m",
  "id": "f4226ab0-6f88-49c3-8f09-84cf5946ee7a",
  "immutable": false,
  "interval": "5m",
  "language": "kuery",
  "max_signals": 100,
  "name": "Query with a threat mapping",
  "output_index": ".siem-signals-hassanabad3-default",
  "query": "*:*",
  "references": [],
  "risk_score": 1,
  "risk_score_mapping": [],
  "rule_id": "threat-mapping",
  "severity": "high",
  "severity_mapping": [],
  "tags": [
    "tag_1",
    "tag_2"
  ],
  "threat": [],
  "threat_index": "mock-threat-list-1",
  "threat_mapping": [
    {
      "entries": [
        {
          "field": "host.name",
          "type": "mapping",
          "value": "host.name"
        },
        {
          "field": "host.ip",
          "type": "mapping",
          "value": "host.ip"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "destination.ip",
          "type": "mapping",
          "value": "destination.ip"
        },
        {
          "field": "destination.port",
          "type": "mapping",
          "value": "destination.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.port",
          "type": "mapping",
          "value": "source.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.ip",
          "type": "mapping",
          "value": "source.ip"
        }
      ]
    }
  ],
  "threat_query": "*:*",
  "throttle": "no_actions",
  "to": "now",
  "type": "threat_match",
  "updated_at": "2020-09-16T04:25:58.051Z",
  "updated_by": "yo",
  "version": 1
}

Structure

You can see the rule structure in the file:

x-pack/plugins/security_solution/server/lib/detection_engine/scripts/rules/queries/query_with_threat_mapping.json

Click here to see JSON

{
  "name": "Query with a threat mapping",
  "description": "Query with a threat mapping",
  "rule_id": "threat-mapping",
  "risk_score": 1,
  "severity": "high",
  "type": "threat_match",
  "query": "*:*",
  "tags": ["tag_1", "tag_2"],
  "threat_index": "mock-threat-list",
  "threat_query": "*:*",
  "threat_mapping": [
    {
      "entries": [
        {
          "field": "host.name",
          "type": "mapping",
          "value": "host.name"
        },
        {
          "field": "host.ip",
          "type": "mapping",
          "value": "host.ip"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "destination.ip",
          "type": "mapping",
          "value": "destination.ip"
        },
        {
          "field": "destination.port",
          "type": "mapping",
          "value": "destination.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.port",
          "type": "mapping",
          "value": "source.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.ip",
          "type": "mapping",
          "value": "source.ip"
        }
      ]
    }
  ]
}

Structural elements that are new:

New type enum called "threat_match"

"type": "threat_match",

New threat_index string which can be set to a single threat index (This might change to an array in the near future before release):

"threat_index": "mock-threat-list"

New threat_query string which can be set any valid query to filter the threat list before executing the rule. This can be undefined, if you are only pushing in filters from the API.

"threat_query": "*:*",

New threat_filters array which can be set to any valid filter like filters. This can be undefined if you are only using the query from the API.

threat_filter": []

New threat_mapping array which can be set to a valid mapping between the threat list and the ECS list. This structure has an inner array called entries which represent a 2 level tree of 1st level OR elements followed by 2nd level AND elements.

For example, if you want to find all threat matches where ECS documents will match against some ${threatList} index where it would be like so:

Click here to see array from the boolean

"threat_mapping": [
    {
      "entries": [
        {
          "field": "host.name",
          "type": "mapping",
          "value": "host.name"
        },
        {
          "field": "host.ip",
          "type": "mapping",
          "value": "host.ip"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "destination.ip",
          "type": "mapping",
          "value": "destination.ip"
        },
        {
          "field": "destination.port",
          "type": "mapping",
          "value": "destination.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.port",
          "type": "mapping",
          "value": "source.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.ip",
          "type": "mapping",
          "value": "source.ip"
        }
      ]
    }
  ]

What that array represents in pseudo boolean logic is:

Click here to see pseduo logic

(host.name: ${threatList.host.name} AND host.ip: ${threatList.host.name}) OR
(destination.ip: ${threatList.destination.ip} AND destination.port: ${threatList.destination.port}) OR
(source.port ${threatList.source.port}) OR
(source.ip ${threatList.source.ip})

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[yctercero]

Q: For fields that are arrays, do we want to default them to empty arrays? Or did you choose to default them to "undefined" to be more explicit about like if a rule is not of type "threat_match" these fields should not be there (as opposed to them being there and being [])?

[FrankHassanabad]

Yeah you answered the question below so I think we're good here 👍

[yctercero]

Nit: It might be useful to make a getThreatMatchAddOnMock that includes these threat match values to throw in throughout.

[FrankHassanabad]

Added both a getAddPrepackagedThreatRulesSchemaMock and a getAddPrepackagedThreatRulesSchemaDecodedMock here as well as the other schema based tests to keep the boiler platting down as suggested. 👍

So now it looks like all the rest. Thanks for the good advice

[yctercero]

I think it should be:

Suggested change

	test('You can set a threat query, index, mapping, filters on a pre-packaged rule', () => {
	test('You can set a threat query, index, mapping, filters when creating rule', () => {

[FrankHassanabad]

Yep, good catch, fixed for the next commit

[yctercero]

Ahhh, I see now here why this value defaults to undefined as opposed to empty array given this requirement here.

[yctercero]

Nit: saw that I think one of the more recent PRs added the pattern of using a helper function to check rule type (x-pack/plugins/security_solution/common/detection_engine/utils.ts) like on line 98 above - maybe a threat match one can be added.

[FrankHassanabad]

Actually yeah I had one in the backend on the server side but it looks like these have moved to the common section so totally agree. I am moving the server side one to the common section and then collapsing code and cleaning this one up. Thanks for the 🦅 👀 on these.

[yctercero]

Nit: quick wording fix:

Suggested change

	test('You can set a threat query, index, mapping, filters on a pre-packaged rule', () => {
	test('You can set a threat query, index, mapping, filters on an imported rule', () => {

[yctercero]

Can case threat_match be added to the first statement on line 93?

[FrankHassanabad]

Yes! That is a better spot for it.

[yctercero]

What do you think about making this the non-empty string type? Just remembering with exceptions, we had to update it to that after the fact to avoid some bugs that were happening. Not sure if that's of concern here.

[FrankHassanabad]

Agreed, I think that is a good idea. Changed the code now to reflect this. You will see it on the next commit.

[yctercero]

Nit:

Suggested change

	* @returns String of the new maximum between the two timers
	* @returns String array of the new maximum between the two timers

[yctercero]

Not sure if this comment is at all relevant - will threat list rules have timestamp overide? If so, would that matter at all here? Like wanting to use that field over @timestamp?

[FrankHassanabad]

It can have an override of a timestamp. That is a good point. I will add this to a small TODO list I am tracking. I will more than likely have to revisit this item in a follow up PR. Appreciate you finding this.

[yctercero]

This is really awesome! There might be some more discussion that can happen as to optimizing the DSL structure - that's not an area I'm well versed on, but I don't think that's anything that should hold up the PR going in. The code itself was really easy and nice to read through, appreciate the use of reduce to help with the readability of the various loops.

Thanks so much for taking the time to talk through the code with me. I pulled it down and played around with it some more after our call. Some things that I encountered during and post call testing:

• when using a slower network, I received the following errors running the threat list rule:

log   [15:35:00.087] [error][plugins][plugins][securitySolution][securitySolution] [-] nextSearchAfter threw an error Error: Request Timeout after 30000ms
server    log   [15:35:00.088] [error][plugins][plugins][securitySolution][securitySolution] [-] search_after and bulk threw an error Error: Request Timeout after 30000ms name: "Query with a threat mapping" id: "21fb7bb1-378d-4f28-acda-b61dcf046899" rule id: "threat-mapping" signals index: ".siem-signals-yt"
server    log   [15:35:00.187] [error][plugins][plugins][securitySolution][securitySolution] Bulk Indexing of signals failed. Check logs for further details. name: "Query with a threat mapping" id: "21fb7bb1-378d-4f28-acda-b61dcf046899" rule id: "threat-mapping" signals index: ".siem-signals-yt

• We played around with this one together and found that decreasing the threat list MAX_PER_PAGE constant to 1000 allowed the rule to run without errors
• I also tested that same threat list but with an exceptions list that included 800 exceptions and hit the same timeout error (on regular home network)

I think the above is a matter of tuning, finding out the ideal settings for the user and like you mentioned, maybe allowing the user access to some of the configs to tune themselves if they like. I think your other PR that improves the error handling/bubbling of the detection engine will also improve the user experience if they're to hit such timeouts to guide them through a solution.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 7bfb649

Build metrics

async chunks size

id	value	diff	baseline
securitySolution	10.1MB	+508.0B	10.1MB

page load bundle size

id	value	diff	baseline
securitySolution	793.4KB	+18.0B	793.3KB

distributable file count

id	value	diff	baseline
default	45950	+11	45939

History

• 💚 Build #75321 succeeded cf9502e
• 💔 Build #75302 failed 3a97f89
• 💛 Build #75058 was flaky 404fee3
• 💔 Build #74958 failed f3eaff0
• 💚 Build #74678 succeeded 68599a9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)