From e700a62621238ab59cbb6f60618d0ea302439659 Mon Sep 17 00:00:00 2001 From: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> Date: Tue, 5 Dec 2023 16:43:45 +0000 Subject: [PATCH] Adds new Advanced Behavioral Detections section (#4296) * Adds new Advanced Behavioral Detections section * Moves L4 pages to L3 * Moves ml-integrations under Behavioral detection use cases * Adds Advanced Behavioral Detections intro section * Adds links to AEA page * Uncomments reference that previously broke the build for no reason * Replaces verbal reference to ml-integrations * Removes frontmatter * Behavioral detection updates * Apply suggestions from code review Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> * Apply suggestion from TW review * Applies review feedback * Lowercase advanced behavioral detections * Lowercase entity risk scoring --------- Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> --- .../advanced-behavioral-detections.asciidoc | 9 +++++ ...dvanced-entity-analytics-overview.asciidoc | 31 ++++++++---------- .../analyze-risk-score-data.asciidoc | 14 ++++---- .../behavioral-detection-use-cases.asciidoc | 29 ++++++++++++++++ .../entity-risk-scoring.asciidoc | 13 ++++++++ .../images/cloned-job-details.png | Bin .../images/filter-add-item.png | Bin .../images/ml-rule-threshold.png | Bin .../images/ml-ui.png | Bin .../images/rule-scope.png | Bin .../images/rules-table-error-icon.png | Bin .../images/rules-table-ml-job-error.png | Bin .../images/start-job-window.png | Bin .../machine-learning.asciidoc | 27 ++------------- .../prebuilt-ml-jobs.asciidoc | 4 +++ .../tune-anomaly-results.asciidoc | 12 +++---- .../turn-on-risk-engine.asciidoc | 10 +++--- docs/detections/detections-index.asciidoc | 4 --- docs/getting-started/ers-req.asciidoc | 6 ++-- 19 files changed, 91 insertions(+), 68 deletions(-) create mode 100644 docs/advanced-entity-analytics/advanced-behavioral-detections.asciidoc create mode 100644 docs/advanced-entity-analytics/behavioral-detection-use-cases.asciidoc create mode 100644 docs/advanced-entity-analytics/entity-risk-scoring.asciidoc rename docs/{detections/machine-learning => advanced-entity-analytics}/images/cloned-job-details.png (100%) rename docs/{detections/machine-learning => advanced-entity-analytics}/images/filter-add-item.png (100%) rename docs/{detections/machine-learning => advanced-entity-analytics}/images/ml-rule-threshold.png (100%) rename docs/{detections/machine-learning => advanced-entity-analytics}/images/ml-ui.png (100%) rename docs/{detections/machine-learning => advanced-entity-analytics}/images/rule-scope.png (100%) rename docs/{detections/machine-learning => advanced-entity-analytics}/images/rules-table-error-icon.png (100%) rename docs/{detections/machine-learning => advanced-entity-analytics}/images/rules-table-ml-job-error.png (100%) rename docs/{detections/machine-learning => advanced-entity-analytics}/images/start-job-window.png (100%) rename docs/{detections/machine-learning => advanced-entity-analytics}/machine-learning.asciidoc (75%) create mode 100644 docs/advanced-entity-analytics/prebuilt-ml-jobs.asciidoc rename docs/{detections/machine-learning => advanced-entity-analytics}/tune-anomaly-results.asciidoc (95%) diff --git a/docs/advanced-entity-analytics/advanced-behavioral-detections.asciidoc b/docs/advanced-entity-analytics/advanced-behavioral-detections.asciidoc new file mode 100644 index 0000000000..eb17be7fde --- /dev/null +++ b/docs/advanced-entity-analytics/advanced-behavioral-detections.asciidoc @@ -0,0 +1,9 @@ +[[advanced-behavioral-detections]] += Advanced behavioral detections + +Elastic's {ml} capabilities and advanced correlation, scoring, and visualization techniques can help you identify potential behavioral threats that may be associated with security incidents. + +Advanced behavioral detections includes two key capabilities: + +* <> +* <> diff --git a/docs/advanced-entity-analytics/advanced-entity-analytics-overview.asciidoc b/docs/advanced-entity-analytics/advanced-entity-analytics-overview.asciidoc index 6ab4c4ba2b..2a035c4a2a 100644 --- a/docs/advanced-entity-analytics/advanced-entity-analytics-overview.asciidoc +++ b/docs/advanced-entity-analytics/advanced-entity-analytics-overview.asciidoc @@ -3,21 +3,16 @@ Advanced Entity Analytics generates a set of threat detection and risk analytics that allows you to expedite alert triage and hunt for new threats from within an entity's environment. This feature combines the power of the SIEM detection engine and Elastic's {ml} capabilities to identify unusual user behaviors and generate comprehensive risk analytics for hosts and users. -[discrete] -[[entity-risk-scoring]] -== Entity Risk Scoring - -beta::[] - -Entity Risk Scoring is an advanced {elastic-sec} analytics feature that helps security analysts detect changes in an entity's risk posture, hunt for new threats, and prioritize incident response. - -Entity Risk Scoring allows you to monitor the change in the risk posture of hosts and users from your environment. The risk scoring engine generates these advanced scoring analytics by factoring threats from its end-to-end XDR use cases, such as SIEM, cloud, and endpoint. - -The next-generation risk scoring engine provides greater scalability and performance. It leverages the Elastic SIEM detection engine to generate host and user risk scores from the last 30 days. - -It also generates risk scores on a recurring interval, and allows for easy onboarding and management. The engine is built to factor in risks from all {elastic-sec} use cases, and allows you to customize and control how and when risk is calculated. - -Learn how to <>. - -include::turn-on-risk-engine.asciidoc[] -include::analyze-risk-score-data.asciidoc[] \ No newline at end of file +Advanced Entity Analytics provides two key capabilities: + +* <> +* <> + +include::entity-risk-scoring.asciidoc[leveloffset=+1] +include::turn-on-risk-engine.asciidoc[leveloffset=+2] +include::analyze-risk-score-data.asciidoc[leveloffset=+2] +include::advanced-behavioral-detections.asciidoc[leveloffset=+1] +include::machine-learning.asciidoc[leveloffset=+2] +include::tune-anomaly-results.asciidoc[leveloffset=+2] +include::behavioral-detection-use-cases.asciidoc[leveloffset=+2] +include::prebuilt-ml-jobs.asciidoc[leveloffset=+2] diff --git a/docs/advanced-entity-analytics/analyze-risk-score-data.asciidoc b/docs/advanced-entity-analytics/analyze-risk-score-data.asciidoc index 440ab5d164..d4570835ca 100644 --- a/docs/advanced-entity-analytics/analyze-risk-score-data.asciidoc +++ b/docs/advanced-entity-analytics/analyze-risk-score-data.asciidoc @@ -1,5 +1,5 @@ [[analyze-risk-score-data]] -== View and analyze risk score data += View and analyze risk score data The {security-app} provides several options to monitor the change in the risk posture of hosts and users from your environment. Use the following places in the {security-app} to view and analyze risk score data: @@ -13,7 +13,7 @@ TIP: We recommend that you prioritize <> to iden [discrete] [[entity-analytics-dashboard]] -=== Entity Analytics dashboard +== Entity Analytics dashboard From the Entity Analytics dashboard, you can access entity key performance indicators (KPIs), risk scores, and levels. You can also click the number link in the **Alerts** column to investigate and analyze the alerts on the Alerts page. @@ -22,12 +22,12 @@ image::dashboards/images/entity-dashboard.png[Entity Analytics dashboard] [discrete] [[alert-triaging]] -=== Alert triaging +== Alert triaging You can prioritize alert triaging to analyze alerts associated with risky entities using the following features in the {security-app}. [discrete] [[alerts-page]] -==== Alerts page +=== Alerts page Use the Alerts table to investigate and analyze host and user risk levels and scores. We recommend adding the `user.risk.calculated_level` and `host.risk.calculated_level` columns to the Alerts table to easily display this data. To do this, select **Fields**, search for `user.risk` and `host.risk`, then select the appropriate fields from the list. Learn more about <>. @@ -41,7 +41,7 @@ image::images/filter-by-host-risk-level.png[Alerts filtered by high host risk le [discrete] [[alert-details-flyout]] -==== Alert details flyout +=== Alert details flyout To access risk score data in the alert details flyout, select **Insights** -> **Entities** on the **Overview** tab: @@ -50,7 +50,7 @@ image::images/alerts-flyout-rs.png[Risk scores in the Alerts flyout] [discrete] [[hosts-users-pages]] -==== Hosts and Users pages +=== Hosts and Users pages On the Hosts and Users pages, you can access the risk score data: @@ -66,7 +66,7 @@ image::images/hosts-hr-data.png[Host risk data on the Host risk tab of the Hosts [discrete] [[host-user-details-pages]] -==== Host and user details pages +=== Host and user details pages On the host details and user details pages, you can access the risk score data: diff --git a/docs/advanced-entity-analytics/behavioral-detection-use-cases.asciidoc b/docs/advanced-entity-analytics/behavioral-detection-use-cases.asciidoc new file mode 100644 index 0000000000..d11b02a114 --- /dev/null +++ b/docs/advanced-entity-analytics/behavioral-detection-use-cases.asciidoc @@ -0,0 +1,29 @@ +[[behavioral-detection-use-cases]] += Behavioral detection use cases + +Behavioral detection identifies potential internal and external threats based on user and host activity. It employs a threat-centric approach to flag suspicious activity by analyzing patterns, anomalies, and context enrichment. + +{elastic-sec} builds the behavioral detection feature on its foundational SIEM detection capabilities, leveraging {ml} algorithms to enable proactive threat detection and hunting. + +[float] +[[ml-integrations]] +=== Elastic {integrations} for behavioral detection use cases + +Behavioral detection integrations provide a convenient way to enable behavioral detection capabilities. They streamline the deployment of components that implement behavioral detection, such as data ingestion, transforms, rules, {ml} jobs, and scripts. + +.Requirements +[sidebar] +-- +* Elastic integrations require a https://www.elastic.co/pricing[Platinum subscription] or higher. +* To learn more about the requirements for using {ml} jobs, refer to <>. +-- + +Here's a list of integrations for various behavioral detection use cases: + +* {integrations-docs}/ded[Data Exfiltration Detection] +* {integrations-docs}/dga[Domain Generation Algorithm Detection] +* {integrations-docs}/lmd[Lateral Movement Detection] +* {integrations-docs}/problemchild[Living off the Land Attack Detection] +* {integrations-docs}/beaconing[Network Beaconing Identification] + +To learn more about {ml} jobs enabled by these integrations, refer to the https://www.elastic.co/guide/en/security/current/prebuilt-ml-jobs.html[Prebuilt jobs page]. diff --git a/docs/advanced-entity-analytics/entity-risk-scoring.asciidoc b/docs/advanced-entity-analytics/entity-risk-scoring.asciidoc new file mode 100644 index 0000000000..bbfc7ea1a0 --- /dev/null +++ b/docs/advanced-entity-analytics/entity-risk-scoring.asciidoc @@ -0,0 +1,13 @@ +[[entity-risk-scoring]] += Entity risk scoring + +beta::[] + +Entity risk scoring is an advanced {elastic-sec} analytics feature that helps security analysts detect changes in an entity's risk posture, hunt for new threats, and prioritize incident response. + +Entity risk scoring allows you to monitor risk score changes of hosts and users in your environment. When generating advanced scoring analytics, the risk scoring engine utilizes threats from its end-to-end XDR use cases, such as SIEM, cloud, and endpoint. It leverages the Elastic SIEM detection engine to generate host and user risk scores from the last 30 days. + +It also generates risk scores on a recurring interval, and allows for easy onboarding and management. The engine is built to factor in risks from all {elastic-sec} use cases, and allows you to customize and control how and when risk is calculated. + +Learn how to <>. + diff --git a/docs/detections/machine-learning/images/cloned-job-details.png b/docs/advanced-entity-analytics/images/cloned-job-details.png similarity index 100% rename from docs/detections/machine-learning/images/cloned-job-details.png rename to docs/advanced-entity-analytics/images/cloned-job-details.png diff --git a/docs/detections/machine-learning/images/filter-add-item.png b/docs/advanced-entity-analytics/images/filter-add-item.png similarity index 100% rename from docs/detections/machine-learning/images/filter-add-item.png rename to docs/advanced-entity-analytics/images/filter-add-item.png diff --git a/docs/detections/machine-learning/images/ml-rule-threshold.png b/docs/advanced-entity-analytics/images/ml-rule-threshold.png similarity index 100% rename from docs/detections/machine-learning/images/ml-rule-threshold.png rename to docs/advanced-entity-analytics/images/ml-rule-threshold.png diff --git a/docs/detections/machine-learning/images/ml-ui.png b/docs/advanced-entity-analytics/images/ml-ui.png similarity index 100% rename from docs/detections/machine-learning/images/ml-ui.png rename to docs/advanced-entity-analytics/images/ml-ui.png diff --git a/docs/detections/machine-learning/images/rule-scope.png b/docs/advanced-entity-analytics/images/rule-scope.png similarity index 100% rename from docs/detections/machine-learning/images/rule-scope.png rename to docs/advanced-entity-analytics/images/rule-scope.png diff --git a/docs/detections/machine-learning/images/rules-table-error-icon.png b/docs/advanced-entity-analytics/images/rules-table-error-icon.png similarity index 100% rename from docs/detections/machine-learning/images/rules-table-error-icon.png rename to docs/advanced-entity-analytics/images/rules-table-error-icon.png diff --git a/docs/detections/machine-learning/images/rules-table-ml-job-error.png b/docs/advanced-entity-analytics/images/rules-table-ml-job-error.png similarity index 100% rename from docs/detections/machine-learning/images/rules-table-ml-job-error.png rename to docs/advanced-entity-analytics/images/rules-table-ml-job-error.png diff --git a/docs/detections/machine-learning/images/start-job-window.png b/docs/advanced-entity-analytics/images/start-job-window.png similarity index 100% rename from docs/detections/machine-learning/images/start-job-window.png rename to docs/advanced-entity-analytics/images/start-job-window.png diff --git a/docs/detections/machine-learning/machine-learning.asciidoc b/docs/advanced-entity-analytics/machine-learning.asciidoc similarity index 75% rename from docs/detections/machine-learning/machine-learning.asciidoc rename to docs/advanced-entity-analytics/machine-learning.asciidoc index d44c068357..7a3cb11fb1 100644 --- a/docs/detections/machine-learning/machine-learning.asciidoc +++ b/docs/advanced-entity-analytics/machine-learning.asciidoc @@ -1,11 +1,6 @@ [[machine-learning]] [role="xpack"] -= Anomaly detection with {ml} - -:frontmatter-description: Use the power of machine learning to detect outliers and suspicious events. -:frontmatter-tags-products: [security] -:frontmatter-tags-content-type: [overview] -:frontmatter-tags-user-goals: [manage] += Anomaly detection {ml-docs}/ml-ad-overview.html[{ml-cap}] functionality is available when you have the appropriate subscription, are using a *{ess-trial}[cloud deployment]*, @@ -68,7 +63,7 @@ data's index patterns in *{kib}* -> *{stack-manage-app}* -> *Data Views*. Or -* You install one or more of the Advanced Analytics integrations (refer to the following section). +* You install one or more of the <>. <> describes all available {ml} jobs and lists which ECS fields are required on your hosts when you are not using {beats} or the {agent} @@ -80,20 +75,6 @@ prior to the time they are enabled. After jobs are enabled, they continuously analyze incoming data. When jobs are stopped and restarted within the two-week time frame, previously analyzed data is not processed again. -[float] -[[ml-integrations]] -=== Jobs in Advanced Analytics (UEBA) Elastic integrations - -You can also install {ml} jobs using https://docs.elastic.co/integrations[Elastic integrations]. Here are the Advanced Analytics integrations available for Security: - -* {integrations-docs}/ded[Data Exfiltration Detection] -* {integrations-docs}/dga[Domain Generation Algorithm Detection] -* {integrations-docs}/lmd[Lateral Movement Detection] -* {integrations-docs}/problemchild[Living off the Land Attack Detection] -* {integrations-docs}/beaconing[Network Beaconing Identification] - -To learn more about {ml} jobs enabled by these integrations, refer to the https://www.elastic.co/guide/en/security/current/prebuilt-ml-jobs.html[Prebuilt jobs page]. - [float] [[view-anomalies]] == View detected anomalies @@ -104,7 +85,3 @@ NOTE: To adjust the `score` threshold that determines which anomalies are shown, you can modify *{kib}* -> *{stack-manage-app}* -> *Advanced Settings* -> *`securitySolution:defaultAnomalyScore`*. -[[prebuilt-ml-jobs]] -== Prebuilt job reference - -include::{stack-docs-root}/docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc[tag=siem-jobs] diff --git a/docs/advanced-entity-analytics/prebuilt-ml-jobs.asciidoc b/docs/advanced-entity-analytics/prebuilt-ml-jobs.asciidoc new file mode 100644 index 0000000000..f92d8e6611 --- /dev/null +++ b/docs/advanced-entity-analytics/prebuilt-ml-jobs.asciidoc @@ -0,0 +1,4 @@ +[[prebuilt-ml-jobs]] += Prebuilt job reference + +include::{stack-docs-root}/docs/en/stack/ml/anomaly-detection/ootb-ml-jobs-siem.asciidoc[tag=siem-jobs] \ No newline at end of file diff --git a/docs/detections/machine-learning/tune-anomaly-results.asciidoc b/docs/advanced-entity-analytics/tune-anomaly-results.asciidoc similarity index 95% rename from docs/detections/machine-learning/tune-anomaly-results.asciidoc rename to docs/advanced-entity-analytics/tune-anomaly-results.asciidoc index b922f8ef16..1c0d64a399 100644 --- a/docs/detections/machine-learning/tune-anomaly-results.asciidoc +++ b/docs/advanced-entity-analytics/tune-anomaly-results.asciidoc @@ -1,5 +1,5 @@ [[tuning-anomaly-results]] -== Optimizing anomaly results += Optimizing anomaly results To gain clearer insights into real threats, you can tune the anomaly results. The following procedures help to reduce the number of false positives: @@ -8,7 +8,7 @@ To gain clearer insights into real threats, you can tune the anomaly results. Th [float] [[rarely-used-processes]] -=== Filter out anomalies from rarely used applications and processes +== Filter out anomalies from rarely used applications and processes When anomalies include results from a known process that only runs occasionally, you can filter out the unwanted results. @@ -22,7 +22,7 @@ For example, to filter out results from a housekeeping process, named [float] [[create-fiter-list]] -==== Create a filter list +=== Create a filter list . Go to *Machine Learning* -> *Anomaly Detection* -> *Settings*. . Click *Filter Lists* and then *New*. @@ -42,7 +42,7 @@ The new filter appears in the Filter List and can be added to relevant jobs. [float] [[add-job-filter]] -==== Add the filter to the relevant job +=== Add the filter to the relevant job . Go to *Machine Learning* -> *Anomaly Detection* -> *Anomaly Explorer*. . Navigate to the job results for which the filter is required. If the job results @@ -70,7 +70,7 @@ before the filter was added are still displayed. [float] [[clone-job]] -==== Clone and rerun the job +=== Clone and rerun the job If you want to remove all the previously detected results for the process, you must clone and run the cloned job. @@ -108,7 +108,7 @@ After a while, results will start to appear on the *Anomaly Explorer* page. [float] [[define-rule-threshold]] -=== Define an anomaly threshold for a job +== Define an anomaly threshold for a job Certain jobs use a high-count function to look for unusual spikes in process events. For some processes, a burst of activity is a normal, such as diff --git a/docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc b/docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc index 5c098ccd16..c90f984d8f 100644 --- a/docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc +++ b/docs/advanced-entity-analytics/turn-on-risk-engine.asciidoc @@ -1,9 +1,9 @@ [[turn-on-risk-engine]] -== Turn on the risk scoring engine += Turn on the risk scoring engine beta[] -IMPORTANT: To use Entity Risk Scoring, your role must have the appropriate privileges. For more information, refer to <>. +IMPORTANT: To use entity risk scoring, your role must have the appropriate privileges. For more information, refer to <>. The latest risk scoring engine runs hourly to aggregate `Open` and `Acknowledged` <> from the last 30 days, and assigns risk score to the host or user. It then aggregates the individual risk scores and normalizes them to a 0-100 range. The engine assigns a risk level by mapping the normalized risk score to one of these levels: @@ -20,7 +20,7 @@ The latest risk scoring engine runs hourly to aggregate `Open` and `Acknowledged |============================================== [discrete] -=== Preview risky entities +== Preview risky entities You can preview risky entities before installing the latest risk engine. The preview shows the riskiest hosts and users found in the 1000 sampled entities during the time frame selected in the date picker. @@ -32,7 +32,7 @@ To preview risky entities, go to **Manage** -> **Entity Risk Score**: image::images/preview-risky-entities.png[Preview of risky entities] [discrete] -=== Turn on the latest risk engine +== Turn on the latest risk engine [NOTE] ====== @@ -50,7 +50,7 @@ image::images/turn-on-risk-engine.png[Turn on entity risk scoring] [discrete] [[upgrade-risk-engine]] -=== Upgrade to the latest risk engine +== Upgrade to the latest risk engine If you upgraded to 8.11 from an earlier {stack} version, and you have the original risk engine installed, you can upgrade to the latest risk engine. You will be prompted to upgrade in places where risk score data exists, such as: diff --git a/docs/detections/detections-index.asciidoc b/docs/detections/detections-index.asciidoc index d853ca4df8..557b99198c 100644 --- a/docs/detections/detections-index.asciidoc +++ b/docs/detections/detections-index.asciidoc @@ -1,7 +1,3 @@ -include::machine-learning/machine-learning.asciidoc[] - -include::machine-learning/tune-anomaly-results.asciidoc[] - include::detection-engine-intro.asciidoc[] include::about-rules.asciidoc[] diff --git a/docs/getting-started/ers-req.asciidoc b/docs/getting-started/ers-req.asciidoc index 2f82a77976..ddfc1e8e44 100644 --- a/docs/getting-started/ers-req.asciidoc +++ b/docs/getting-started/ers-req.asciidoc @@ -1,9 +1,9 @@ [[ers-requirements]] -= Entity Risk Scoring prerequisites += Entity risk scoring prerequisites -To use Entity Risk Scoring, your role must have certain cluster, index, and {kib} privileges. This feature requires a https://www.elastic.co/pricing[Platinum subscription] or higher. +To use entity risk scoring, your role must have certain cluster, index, and {kib} privileges. This feature requires a https://www.elastic.co/pricing[Platinum subscription] or higher. -This page covers the requirements and guidelines for using the Entity Risk Scoring feature, as well as its known limitations. +This page covers the requirements and guidelines for using the entity risk scoring feature, as well as its known limitations. [discrete] == Privileges