[Security][Detections] Create Threshold-based Rule type

[spong]

This issue is for creating a new rule type based on thresholds/aggregations, and can appear as a separate card for selection within the Define Rule section of the Create Rule flow.

There are actually several kinds of aggregation-based rules that could fall in here: # hits, sum, terms, significant terms, etc.

Latest mocks:

cc @marrasherrier

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[nachiket-lab]

Hi @spong ,

This is really encouraging and definitely a required feature. I just had a question on how do we handle data that is coming with a delay?

I agree that this is a broad question, and not really something that should concern the log aggregator. But in reality, getting logs in realtime is not always feasible. An obvious workaround would be to increase the timeframe, but then you have to accept a delay in terms of rule triggers!

[spong]

Hey there @NerdSec -- thanks for commenting! 🙂

To better handle delayed events we'll be adding the ability to specify a Default Timestamp setting as part of the rule (#65941) so users can specify which timestamp field the rule will use during it's execution phase. This'll allow us to specify which timestamp is closest to ingest time, so if any data is delayed in making it into the system it'll still be processed as part of the next rule run. This'll be especially helpful for endpoints or other agents that are offline for a few days and then come back online and push up all their data. Tangentially, we also have some work with regards to mitigating gaps in rule runs -- you can see more about that here: #63290.

Hope this helps!

[nachiket-lab]

One small input here. In most of the use cases we have implemented, we ended up needing the terms, date_histogram, cardinality, and bucket_selector aggs. We did not the ESQL implementation, rather built the agg queries in the console, which wasn't quite user friendly.

Some use cases:

• Multiple logon failures from single source IP
• Concurrent user logins from multiple regions/geo locations
• Single source ip targeting multiple websites
• Multiple occurrences of an unique infection from the same machine

I am not sure if it was the correct route, but we implemented a terms agg/cardinality with a bucket selector to choose which events matched our condition!

[dontcallmesherryli]

Had discussion with @spong @MikePaquette @marrasherrier to answer questions and decrease scope.

• Product is comfortable with just using default fields of Detection Alert List and Timeline for threshold rule based alerts. Where a field would have multiple values, the alert list can show "Multiple Values" on field value. User should still be able to use the right click function to get top values and filter on values.

• Scope will be cut to only have a query field, a group by field, and threshold count field for user to create a threshold rule.

• Backend need is adding field value of "Threshold Rule" for rule kind and "threshold count".

[marrasherrier]

Mocks: https://www.figma.com/file/yNMzeaGvLkyRvra8Bu2fD2/Threshold-Based-Rules?node-id=140%3A31

[FrankHassanabad]

This is completed and shipping, so closing the ticket now.

[spong]

Resolved by #71371! Thanks again @patrykkopycinski