[Docs]Detections and Alerts UI (#73) (#92)

* dtections-ui-overview

* initial overview draft

* typo

* restructuring for dedicated alerts section

* rewording

* exceptions from alerts

* adds new rule options

* adds new action text placeholder

* restructer

* structure, exceptions and building-blocks

* minor edits

* adds exceptions

* exceptions cont

* exceptions correction

* more stuff

* proofing and whatnot

* terminology

* nested exception conditions

* typo

* typo - thanks Nate

* corrections - round 1

* add nested conditions example

* typo

* editing

* more proofing

* updates ex example

* adds promoted endpoint events

* typo

* corrections after review

* corrections

docs/siem/detections/alerts-ui-manage.asciidoc

@@ -0,0 +1,106 @@
+[[alerts-ui-manage]]
+[role="xpack"]
+== Managing detection alerts
+
+The Detections page displays all <<detection-alert-def, detection alerts>>.
+From the Alerts table, you can change an alert's status, and start
+investigating and analyzing alerts in Timeline.
+
+TIP: From Timeline, you can <<cases-ui-open, create cases>> to track issues and
+share information with colleagues.
+
+To view detection alerts created by a specific rule, you can:
+
+* Filter for a specific rule in the KQL bar (for example,
+`signal.rule.name :"SSH (Secure Shell) from the Internet"`).
+* View detection alerts in the *Rule details* page (click
+*Manage detection rules* -> rule name in the *All rules* table).
+
+NOTE: KQL autocomplete for `.siem-signals-*` indices is available on the 
+*Detections* and *Rule details* pages, and in Timeline when either `All` or
+`Detection alerts` is selected. 
+
+TIP: Use the icons in the upper left corner of the Alerts table to customize
+displayed columns and row renderers, and view the table in full screen mode.
+
+[float]
+[[detection-alert-status]]
+=== Change alert statuses
+
+You can set an alert's status to indicate whether it needs to be investigated
+(`Open`), is under active investigation (`In progress`), or resolved
+(`Closed`). By default, the Alerts table displays open alerts. To view alerts
+with other statuses, click *In progress* or *Closed*.
+
+To change alert statuses, either:
+
+* In the alert's row, click the *more options* icon, and then select the
+required status (*Mark in progress*, *Close alert*, or *Open alert*).
+* In the Alerts table, select all the alerts you want to change, and then select
+*Take action* -> *Close selected*, *Open selected*, or *Mark in progress*.
+
+[float]
+[[signals-to-timelines]]
+=== Send alerts to Timeline
+
+To view an alert in Timeline, click the *Investigate in timeline* icon.
+
+TIP: When you send an alert generated by a
+<<rules-ui-create, threshold rule>> to Timeline, all matching events are
+listed in the Timeline, even ones that did not reach the threshold value. For
+example, if you have an alert generated by a threshold rule that detects 10
+failed login attempts, when you send that alert to Timeline all failed login
+attempts detected by the rule are listed.
+
+If the rule that generated the alert uses a Timeline template, when you
+investigate the alert in Timeline, the dropzone query values defined in the
+template are replaced with their corresponding alert values.
+
+// * `host.name`
+// * `host.hostname`
+// * `host.domain`
+// * `host.id`
+// * `host.ip`
+// * `client.ip`
+// * `destination.ip`
+// * `server.ip`
+// * `source.ip`
+// * `network.community_id`
+// * `user.name`
+// * `process.name`
+
+*Example*
+
+This Timeline template uses the `host.name: "{host.name}"` dropzone filter in
+the rule. When alerts generated by the rule are investigated in Timeline, the
+`{host.name}` value is replaced with the alert's `host.name` value. If the
+alerts's `host.name` value is `Windows-ArsenalFC`, the Timeline dropzone query
+is `host.name: "Windows-ArsenalFC"`.
+
+NOTE: See <<timelines-ui>> for information on creating Timelines and Timeline
+templates. For information on how to add Timeline templates to rules, see
+<<rules-ui-create>>.
+
+[float]
+[[add-exception-from-alerts]]
+=== Add rule exceptions
+
+You can add exceptions to the rule that generated the alert directly from the
+Alerts table. Exceptions prevent a rule from generating alerts even when its
+criteria are met.
+
+To add an exception, click the actions icon (three dots) and then select
+_Add exception_. 
+
+For information about exceptions and how to use them, see
+<<detections-ui-exceptions>>.
+
+[float]
+[[alerts-to-resolver]]
+=== Visually analyze process relationships.
+
+For process events received from the Elastic Endpoint agent, you can open a
+visual mapping of the relationships and hierarchy connecting related processes.
+
+To visualize process relationships, click the *Analyze event* icon. For more
+information, see Ben xref.

docs/siem/detections/api/rules-api-create.asciidoc

@@ -377,7 +377,7 @@ used as an identifier across systems
 * `{{context.rule.saved_id}}`: Saved search ID
 * `{{context.rule.severity}}`: Rule severity
 * `{{context.rule.threat}}`: Rule threat framework
-* `{{context.rule.threshold}}`: Rule threshold (threshold rules only)
+* `{{context.rule.threshold}}`: Rule threshold values (threshold rules only)
 * `{{context.rule.timeline_id}}`: Associated timeline ID
 * `{{context.rule.timeline_title}}`: Associated timeline name
 * `{{context.rule.type}}`: Rule type

docs/siem/detections/building-block-rule.asciidoc

@@ -0,0 +1,31 @@
+[[building-block-rule]]
+[role="xpack"]
+== About building-block rules
+
+Create building-block rules when you do not want to see their generated alerts
+in the UI. This is useful when you want:
+
+* A record of low-risk alerts without producing noise in the Alerts table.
+* Rules that execute on the alert indices (`.siem-signals-<kibana space>-*`).
+You can then use building-block rules to create hidden alerts that act as a
+basis for an 'ordinary' rule to generate visible alerts.
+
+[float]
+=== Set up rules that run on alert indices
+
+To create a rule that searches alert indices, in the *Index patterns* field,
+add the index pattern for alert indices:
+
+[role="screenshot"]
+image::images/alert-indices-ui.png[]
+
+[float]
+
+=== View building-block alerts in the UI
+
+. Go to *Security* -> *Detections*
+. In the Alert table, select _Additional filters_ ->
+_Include building-block alerts_.
+
+NOTE: On a building-block Rule details page, the rule's alerts are displayed (by
+default, _Include building-block alerts_ is selected).

docs/siem/detections/detection-engine-intro.asciidoc

@@ -1,26 +1,60 @@
 [[detection-engine-overview]]
 [role="xpack"]
 
-= Detections (beta)
+= Detections and Alerts (beta)
 
 beta[]
 
-The SIEM Detections feature automatically searches for threats and creates 
-signals when they are detected. Signal detection rules define the conditions 
-for creating signals. Additionally, you can use the {kib}
-{kibana-ref}/alerting-getting-started.html[Alerting and Actions]
-framework to send notifications via other systems, such as email and Slack,
-when signals are generated.
+Use the Detections feature to create and manage rules, and view the alerts
+these rules create. Rules periodically search indices (such as `endgame-*` and
+`filebeat-*`) for suspicious source events, and create alerts when a rule's
+conditions are met. When an alert is created, its status is `Open`. To help
+track investigations, an alert's status can be set as `Open`, `In progress`, or
+`Closed` (see <<detection-alert-status>>).
 
-NOTE: To use {kib} Alerting for signal notifications, you need the
+[role="screenshot"]
+image::images/detections-ui.png[]
+
+In addition to creating <<rules-ui-create, your own rules>>, enable
+<<load-prebuilt-rules, Elastic prebuilt rules>> to immediately start detecting
+suspicious activity. For detailed information on all the prebuilt rules, see the
+<<prebuilt-rules>> section. Once the prebuilt rules are loaded and
+running, <<tuning-detection-signals>> and <<detections-ui-exceptions>> explain
+how to modify the rules to reduce false positives and get a better set of
+actionable alerts. You can also use exceptions and value lists when creating or
+modifying your own rules.
+
+There are two special prebuilt rules you need to know about:
+
+* <<elastic-endpoint-prebuilt-rule, *Elastic Endpoint Security*>>:
+Automatically creates an alert from all incoming Elastic Endpoint alerts. To
+receive Elastic Endpoint alerts, you must install the Endpoint agent on your
+hosts (BEN: see xref).
++
+When this rule is enabled, the following Endpoint events are displayed as
+detection alerts:
++
+** Malware Prevention Alert
+** Malware Detection Alert
++
+NOTE: When you load the prebuilt rules, this is the only rule that is enabled
+by default.
+
+* <<external-alerts, *External Alerts*>>: Automatically creates an alert for
+all incoming third-party system alerts (for example, Suricata alerts).
+
+If you want to receive notifications via external systems, such as Slack or
+email, when alerts are created, use the {kib}
+{kibana-ref}/alerting-getting-started.html[Alerting and Actions] framework.
+
+NOTE: To use {kib} Alerting for detection alert notifications, you need the
 https://www.elastic.co/subscriptions[appropriate license].
 
-The {siem-app} comes with <<prebuilt-rules, prebuilt rules>> that search for
-suspicious activity on your network and hosts. For information on how to
-optimize the prebuilt rules, see <<tuning-detection-signals>>. You can also
-<<rules-ui-create, create your own rules>>.
+After rules have started running, you can monitor their executions to verify
+they are functioning correctly, as well as view, manage, and troubleshoot
+alerts (see <<alerts-ui-manage>> and <<alerts-ui-monitor>>).
 
-You can manage detection rules and signals via the UI or the
+You can create and manage rules and alerts via the UI or the
 <<rule-api-overview, Detections API>>.
 
 [IMPORTANT]
@@ -29,106 +63,45 @@ To make sure you can access Detections and manage rules, see
 <<detections-permissions>>.
 ==============
 
-[role="screenshot"]
-image::detections-ui.png[]
-
 [float]
 [[det-engine-terminology]]
 == Terminology
 
-Rules::
-Perform background tasks to detect suspicious activity. There are two types of
-rules:
-
-* Query-based rules search indices for documents that match their queries at
-scheduled intervals. When a document matches, a signal is created.
-* {ml-cap} rules create a signal when a {ml} job discovers an anomaly score
-above a rule's defined threshold.
-
-Signals::
-Always refer to {siem-soln} produced detections. Signals are never received 
-from external systems. When a rule's conditions are met, the {siem-app} 
-writes one or more signals to an Elasticsearch `signals` index.
+Actions::
+Sends notifications via other systems when a detection alert is created, such
+as email, Slack, PagerDuty, and Webhook.
+
+[[detection-alert-def]]
+Detection alerts::
+{es-sec} produced alerts. Detection alerts are never received from external
+systems. When a rule's conditions are met, {es-sec} writes a detection alert to
+an Elasticsearch `.signals` index.
 +
 [NOTE]
 ==============
-Signal indices are created for each {kib} space. The naming convention is:
-`.siem-signals-<space name>`. For the default space, the signals index is named 
-`.siem-signals-default`.
+Detection alert indices are created for each {kib} space. The naming convention
+is: `.siem-signals-<space name>`. For the default space, the alerts index is
+named `.siem-signals-default`.
 ==============
 
-Alerts and events::
-Always refer to data the {siem-app} receives from external systems, such as 
-Elastic Endpoint and Suricata.
+Detection rules::
+Background tasks that run periodically and produce alerts when suspicious
+activity is detected.
 
-Actions::
-Used to send notifications via other systems when a signal is created, such as
-email, Slack, PagerDuty, and Webhook.
-
-[float]
-== Signals and external alerts
+Endpoint exceptions::
+<<term-exceptions, Exceptions>> added to both rules and Endpoint agents on
+hosts. Endpoint exceptions can only be added when:
 
-The Detections page displays all signals and alerts. To view signals created 
-by a rule, you can:
+* Endpoint agents are installed on the hosts.
+* The Elastic Endpoint Security rule is activated.
 
-* Filter for a specific rule in the KQL bar (for example,
-`signal.rule.name :"SSH (Secure Shell) from the Internet"`).
-* View signals in the *Rule details* page (click
-*Manage signal detection rules* -> rule name in the *All rules* table).
+[[term-exceptions]]
+Exceptions::
+Added to rules to prevent specific source event field values from generating
+alerts.
 
-NOTE: KQL autocomplete for `.siem-signals-*` indices is available on the 
-*Detections* and *Rule details* pages, and in Timeline when either `All events`
-or `Signal events` is selected. 
-
-To view alerts from external data shippers, click *External alerts*.
-
-[float]
-=== Open and close signals
-
-You can close signals to indicate they do not need any further investigation. 
-By default, the All signals table displays open signals. To view closed 
-signals, click *Closed signals*.
-
-To open and close signals, either:
-
-* Click the *Close/Open signal* icon.
-* Select the signals you want to open or close and then click 
-*Close/Open selected*.
-
-[float]
-[[signals-to-timelines]]
-=== Investigate signals in Timeline
-
-To investigate a signal in Timeline, click the *Investigate in timeline*
-icon.
-
-If the rule that generated the signal uses a timeline template, when you
-investigate the signal in Timeline, the following dropzone query values are
-replaced with their corresponding signal values:
-
-* `host.name`
-* `host.hostname`
-* `host.domain`
-* `host.id`
-* `host.ip`
-* `client.ip`
-* `destination.ip`
-* `server.ip`
-* `source.ip`
-* `network.community_id`
-* `user.name`
-* `process.name`
-
-*Example*
-
-The timeline template used in the rule has this dropzone query:
-`host.name: "Linux-LiverpoolFC"`. When signals generated by the rule are
-investigated in Timeline, the `host.name` value is replaced with the signal's
-`host.name` value. If the signal's `host.name` value is `Windows-ArsenalFC`,
-the timeline dropzone query is `host.name: "Windows-ArsenalFC"`.
-
-NOTE: For information on how to add timeline templates to rules, see
-<<create-rule-ui>>.
+External alerts::
+Alerts {es-sec} receives from external systems, such as Suricata.
 
 [float]
 [[detections-permissions]]
@@ -150,14 +123,14 @@ of at least 32 characters. For example:
 
 For *all* deployments (on-premises and hosted):
 
-* To view signals and detection rules, you must have at least:
+* To view detection rules and alerts, you must have at least:
 ** `read` permissions for the `.siem-signals-<space name>` index, where
 `<space name>` is the name of the {kib} space you are using to view Detections
 (see {ref}/security-privileges.html#privileges-list-indices[Indices privileges]).
-** {kib} space `Read` privileges for the `SIEM` feature (see
+** {kib} space `Read` privileges for the `Security` feature (see
 {kibana-ref}/xpack-spaces.html#spaces-control-user-access[Feature access based on user privileges]).
 * To create and modify detection rules, you must have:
-** {kib} space `All` privileges for the `SIEM` feature (see
+** {kib} space `All` privileges for the `Security` feature (see
 {kibana-ref}/xpack-spaces.html#spaces-control-user-access[Feature access based on user privileges]).
 ** Write permissions for the `.siem-signals-<space name>` index, such as 
 `create` `create_doc`, `write`, `index`, and `all`
@@ -173,11 +146,11 @@ when you try to open the *Detections* page.
 *`Let’s set up your detection engine`*
 
 If you see this message, a user with these privileges must visit the 
-*Detections* page before you can view signals and rules:
+*Detections* page before you can view detection rules and alerts:
 
 * The `manage` cluster privilege (see {ref}/security-privileges.html[{es} security privileges]).
 * The `create_index` index privilege (see {ref}/security-privileges.html[{es} security privileges]).
-* {kib} space `All` privileges for the `SIEM` feature (see
+* {kib} space `All` privileges for the `Security` feature (see
 {kibana-ref}/xpack-spaces.html#spaces-control-user-access[Feature access based on user privileges]).
 
 NOTE: For *on-premises* {stack} deployments only, this message may be displayed