Skip to content

Commit

Permalink
Adds new Advanced Behavioral Detections section (#4296)
Browse files Browse the repository at this point in the history
* Adds new Advanced Behavioral Detections section

* Moves L4 pages to L3

* Moves ml-integrations under Behavioral detection use cases

* Adds Advanced Behavioral Detections intro section

* Adds links to AEA page

* Uncomments reference that previously broke the build for no reason

* Replaces verbal reference to ml-integrations

* Removes frontmatter

* Behavioral detection updates

* Apply suggestions from code review

Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com>

* Apply suggestion from TW review

* Applies review feedback

* Lowercase advanced behavioral detections

* Lowercase entity risk scoring

---------

Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com>
  • Loading branch information
natasha-moore-elastic and benironside authored Dec 5, 2023
1 parent 86523b2 commit e700a62
Show file tree
Hide file tree
Showing 19 changed files with 91 additions and 68 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[[advanced-behavioral-detections]]
= Advanced behavioral detections

Elastic's {ml} capabilities and advanced correlation, scoring, and visualization techniques can help you identify potential behavioral threats that may be associated with security incidents.

Advanced behavioral detections includes two key capabilities:

* <<machine-learning, Anomaly detection>>
* <<behavioral-detection-use-cases, Behavioral detection use cases>>
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,16 @@

Advanced Entity Analytics generates a set of threat detection and risk analytics that allows you to expedite alert triage and hunt for new threats from within an entity's environment. This feature combines the power of the SIEM detection engine and Elastic's {ml} capabilities to identify unusual user behaviors and generate comprehensive risk analytics for hosts and users.

[discrete]
[[entity-risk-scoring]]
== Entity Risk Scoring

beta::[]

Entity Risk Scoring is an advanced {elastic-sec} analytics feature that helps security analysts detect changes in an entity's risk posture, hunt for new threats, and prioritize incident response.

Entity Risk Scoring allows you to monitor the change in the risk posture of hosts and users from your environment. The risk scoring engine generates these advanced scoring analytics by factoring threats from its end-to-end XDR use cases, such as SIEM, cloud, and endpoint.

The next-generation risk scoring engine provides greater scalability and performance. It leverages the Elastic SIEM detection engine to generate host and user risk scores from the last 30 days.

It also generates risk scores on a recurring interval, and allows for easy onboarding and management. The engine is built to factor in risks from all {elastic-sec} use cases, and allows you to customize and control how and when risk is calculated.

Learn how to <<turn-on-risk-engine, turn on the latest risk scoring engine>>.

include::turn-on-risk-engine.asciidoc[]
include::analyze-risk-score-data.asciidoc[]
Advanced Entity Analytics provides two key capabilities:

* <<entity-risk-scoring, Entity risk scoring>>
* <<advanced-behavioral-detections, Advanced behavioral detections>>

include::entity-risk-scoring.asciidoc[leveloffset=+1]
include::turn-on-risk-engine.asciidoc[leveloffset=+2]
include::analyze-risk-score-data.asciidoc[leveloffset=+2]
include::advanced-behavioral-detections.asciidoc[leveloffset=+1]
include::machine-learning.asciidoc[leveloffset=+2]
include::tune-anomaly-results.asciidoc[leveloffset=+2]
include::behavioral-detection-use-cases.asciidoc[leveloffset=+2]
include::prebuilt-ml-jobs.asciidoc[leveloffset=+2]
14 changes: 7 additions & 7 deletions docs/advanced-entity-analytics/analyze-risk-score-data.asciidoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[[analyze-risk-score-data]]
== View and analyze risk score data
= View and analyze risk score data

The {security-app} provides several options to monitor the change in the risk posture of hosts and users from your environment. Use the following places in the {security-app} to view and analyze risk score data:

Expand All @@ -13,7 +13,7 @@ TIP: We recommend that you prioritize <<alert-triaging, alert triaging>> to iden

[discrete]
[[entity-analytics-dashboard]]
=== Entity Analytics dashboard
== Entity Analytics dashboard

From the Entity Analytics dashboard, you can access entity key performance indicators (KPIs), risk scores, and levels. You can also click the number link in the **Alerts** column to investigate and analyze the alerts on the Alerts page.

Expand All @@ -22,12 +22,12 @@ image::dashboards/images/entity-dashboard.png[Entity Analytics dashboard]

[discrete]
[[alert-triaging]]
=== Alert triaging
== Alert triaging
You can prioritize alert triaging to analyze alerts associated with risky entities using the following features in the {security-app}.

[discrete]
[[alerts-page]]
==== Alerts page
=== Alerts page

Use the Alerts table to investigate and analyze host and user risk levels and scores. We recommend adding the `user.risk.calculated_level` and `host.risk.calculated_level` columns to the Alerts table to easily display this data. To do this, select **Fields**, search for `user.risk` and `host.risk`, then select the appropriate fields from the list. Learn more about <<customize-the-alerts-table, customizing the Alerts table>>.

Expand All @@ -41,7 +41,7 @@ image::images/filter-by-host-risk-level.png[Alerts filtered by high host risk le

[discrete]
[[alert-details-flyout]]
==== Alert details flyout
=== Alert details flyout

To access risk score data in the alert details flyout, select **Insights** -> **Entities** on the **Overview** tab:

Expand All @@ -50,7 +50,7 @@ image::images/alerts-flyout-rs.png[Risk scores in the Alerts flyout]

[discrete]
[[hosts-users-pages]]
==== Hosts and Users pages
=== Hosts and Users pages

On the Hosts and Users pages, you can access the risk score data:

Expand All @@ -66,7 +66,7 @@ image::images/hosts-hr-data.png[Host risk data on the Host risk tab of the Hosts

[discrete]
[[host-user-details-pages]]
==== Host and user details pages
=== Host and user details pages

On the host details and user details pages, you can access the risk score data:

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
[[behavioral-detection-use-cases]]
= Behavioral detection use cases

Behavioral detection identifies potential internal and external threats based on user and host activity. It employs a threat-centric approach to flag suspicious activity by analyzing patterns, anomalies, and context enrichment.

{elastic-sec} builds the behavioral detection feature on its foundational SIEM detection capabilities, leveraging {ml} algorithms to enable proactive threat detection and hunting.

[float]
[[ml-integrations]]
=== Elastic {integrations} for behavioral detection use cases

Behavioral detection integrations provide a convenient way to enable behavioral detection capabilities. They streamline the deployment of components that implement behavioral detection, such as data ingestion, transforms, rules, {ml} jobs, and scripts.

.Requirements
[sidebar]
--
* Elastic integrations require a https://www.elastic.co/pricing[Platinum subscription] or higher.
* To learn more about the requirements for using {ml} jobs, refer to <<ml-requirements, Machine learning job and rule requirements>>.
--

Here's a list of integrations for various behavioral detection use cases:

* {integrations-docs}/ded[Data Exfiltration Detection]
* {integrations-docs}/dga[Domain Generation Algorithm Detection]
* {integrations-docs}/lmd[Lateral Movement Detection]
* {integrations-docs}/problemchild[Living off the Land Attack Detection]
* {integrations-docs}/beaconing[Network Beaconing Identification]

To learn more about {ml} jobs enabled by these integrations, refer to the https://www.elastic.co/guide/en/security/current/prebuilt-ml-jobs.html[Prebuilt jobs page].
13 changes: 13 additions & 0 deletions docs/advanced-entity-analytics/entity-risk-scoring.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[[entity-risk-scoring]]
= Entity risk scoring

beta::[]

Entity risk scoring is an advanced {elastic-sec} analytics feature that helps security analysts detect changes in an entity's risk posture, hunt for new threats, and prioritize incident response.

Entity risk scoring allows you to monitor risk score changes of hosts and users in your environment. When generating advanced scoring analytics, the risk scoring engine utilizes threats from its end-to-end XDR use cases, such as SIEM, cloud, and endpoint. It leverages the Elastic SIEM detection engine to generate host and user risk scores from the last 30 days.

It also generates risk scores on a recurring interval, and allows for easy onboarding and management. The engine is built to factor in risks from all {elastic-sec} use cases, and allows you to customize and control how and when risk is calculated.

Learn how to <<turn-on-risk-engine, turn on the latest risk scoring engine>>.

Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
[[machine-learning]]
[role="xpack"]
= Anomaly detection with {ml}

:frontmatter-description: Use the power of machine learning to detect outliers and suspicious events.
:frontmatter-tags-products: [security]
:frontmatter-tags-content-type: [overview]
:frontmatter-tags-user-goals: [manage]
= Anomaly detection

{ml-docs}/ml-ad-overview.html[{ml-cap}] functionality is available when
you have the appropriate subscription, are using a *{ess-trial}[cloud deployment]*,
Expand Down Expand Up @@ -68,7 +63,7 @@ data's index patterns in *{kib}* -> *{stack-manage-app}* -> *Data Views*.

Or

* You install one or more of the Advanced Analytics integrations (refer to the following section).
* You install one or more of the <<ml-integrations, Advanced Analytics integrations>>.

<<prebuilt-ml-jobs>> describes all available {ml} jobs and lists which ECS
fields are required on your hosts when you are not using {beats} or the {agent}
Expand All @@ -80,20 +75,6 @@ prior to the time they are enabled. After jobs are enabled, they continuously
analyze incoming data. When jobs are stopped and restarted within the two-week
time frame, previously analyzed data is not processed again.

[float]
[[ml-integrations]]
=== Jobs in Advanced Analytics (UEBA) Elastic integrations

You can also install {ml} jobs using https://docs.elastic.co/integrations[Elastic integrations]. Here are the Advanced Analytics integrations available for Security:

* {integrations-docs}/ded[Data Exfiltration Detection]
* {integrations-docs}/dga[Domain Generation Algorithm Detection]
* {integrations-docs}/lmd[Lateral Movement Detection]
* {integrations-docs}/problemchild[Living off the Land Attack Detection]
* {integrations-docs}/beaconing[Network Beaconing Identification]

To learn more about {ml} jobs enabled by these integrations, refer to the https://www.elastic.co/guide/en/security/current/prebuilt-ml-jobs.html[Prebuilt jobs page].

[float]
[[view-anomalies]]
== View detected anomalies
Expand All @@ -104,7 +85,3 @@ NOTE: To adjust the `score` threshold that determines which anomalies are shown,
you can modify
*{kib}* -> *{stack-manage-app}* -> *Advanced Settings* -> *`securitySolution:defaultAnomalyScore`*.

[[prebuilt-ml-jobs]]
== Prebuilt job reference

include::{stack-docs-root}/docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc[tag=siem-jobs]
4 changes: 4 additions & 0 deletions docs/advanced-entity-analytics/prebuilt-ml-jobs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[[prebuilt-ml-jobs]]
= Prebuilt job reference

include::{stack-docs-root}/docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc[tag=siem-jobs]
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[[tuning-anomaly-results]]
== Optimizing anomaly results
= Optimizing anomaly results

To gain clearer insights into real threats, you can tune the anomaly results. The following procedures help to reduce the number of false positives:

Expand All @@ -8,7 +8,7 @@ To gain clearer insights into real threats, you can tune the anomaly results. Th

[float]
[[rarely-used-processes]]
=== Filter out anomalies from rarely used applications and processes
== Filter out anomalies from rarely used applications and processes

When anomalies include results from a known process that only runs occasionally,
you can filter out the unwanted results.
Expand All @@ -22,7 +22,7 @@ For example, to filter out results from a housekeeping process, named

[float]
[[create-fiter-list]]
==== Create a filter list
=== Create a filter list

. Go to *Machine Learning* -> *Anomaly Detection* -> *Settings*.
. Click *Filter Lists* and then *New*.
Expand All @@ -42,7 +42,7 @@ The new filter appears in the Filter List and can be added to relevant jobs.

[float]
[[add-job-filter]]
==== Add the filter to the relevant job
=== Add the filter to the relevant job

. Go to *Machine Learning* -> *Anomaly Detection* -> *Anomaly Explorer*.
. Navigate to the job results for which the filter is required. If the job results
Expand Down Expand Up @@ -70,7 +70,7 @@ before the filter was added are still displayed.

[float]
[[clone-job]]
==== Clone and rerun the job
=== Clone and rerun the job

If you want to remove all the previously detected results for the process, you
must clone and run the cloned job.
Expand Down Expand Up @@ -108,7 +108,7 @@ After a while, results will start to appear on the *Anomaly Explorer* page.

[float]
[[define-rule-threshold]]
=== Define an anomaly threshold for a job
== Define an anomaly threshold for a job

Certain jobs use a high-count function to look for unusual spikes in
process events. For some processes, a burst of activity is a normal, such as
Expand Down
10 changes: 5 additions & 5 deletions docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[[turn-on-risk-engine]]
== Turn on the risk scoring engine
= Turn on the risk scoring engine

beta[]

IMPORTANT: To use Entity Risk Scoring, your role must have the appropriate privileges. For more information, refer to <<ers-requirements, Entity Risk Scoring prerequisites>>.
IMPORTANT: To use entity risk scoring, your role must have the appropriate privileges. For more information, refer to <<ers-requirements, Entity risk scoring prerequisites>>.

The latest risk scoring engine runs hourly to aggregate `Open` and `Acknowledged` <<alerts-ui-manage, alerts>> from the last 30 days, and assigns risk score to the host or user. It then aggregates the individual risk scores and normalizes them to a 0-100 range. The engine assigns a risk level by mapping the normalized risk score to one of these levels:

Expand All @@ -20,7 +20,7 @@ The latest risk scoring engine runs hourly to aggregate `Open` and `Acknowledged
|==============================================

[discrete]
=== Preview risky entities
== Preview risky entities

You can preview risky entities before installing the latest risk engine. The preview shows the riskiest hosts and users found in the 1000 sampled entities during the time frame selected in the date picker.

Expand All @@ -32,7 +32,7 @@ To preview risky entities, go to **Manage** -> **Entity Risk Score**:
image::images/preview-risky-entities.png[Preview of risky entities]

[discrete]
=== Turn on the latest risk engine
== Turn on the latest risk engine

[NOTE]
======
Expand All @@ -50,7 +50,7 @@ image::images/turn-on-risk-engine.png[Turn on entity risk scoring]

[discrete]
[[upgrade-risk-engine]]
=== Upgrade to the latest risk engine
== Upgrade to the latest risk engine

If you upgraded to 8.11 from an earlier {stack} version, and you have the original risk engine installed, you can upgrade to the latest risk engine. You will be prompted to upgrade in places where risk score data exists, such as:

Expand Down
4 changes: 0 additions & 4 deletions docs/detections/detections-index.asciidoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
include::machine-learning/machine-learning.asciidoc[]

include::machine-learning/tune-anomaly-results.asciidoc[]

include::detection-engine-intro.asciidoc[]

include::about-rules.asciidoc[]
Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started/ers-req.asciidoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[[ers-requirements]]
= Entity Risk Scoring prerequisites
= Entity risk scoring prerequisites

To use Entity Risk Scoring, your role must have certain cluster, index, and {kib} privileges. This feature requires a https://www.elastic.co/pricing[Platinum subscription] or higher.
To use entity risk scoring, your role must have certain cluster, index, and {kib} privileges. This feature requires a https://www.elastic.co/pricing[Platinum subscription] or higher.

This page covers the requirements and guidelines for using the Entity Risk Scoring feature, as well as its known limitations.
This page covers the requirements and guidelines for using the entity risk scoring feature, as well as its known limitations.

[discrete]
== Privileges
Expand Down

0 comments on commit e700a62

Please sign in to comment.