diff --git a/docs/en/siem/images/add-data.png b/docs/en/siem/images/add-data.png new file mode 100644 index 000000000..163b66b4d Binary files /dev/null and b/docs/en/siem/images/add-data.png differ diff --git a/docs/en/siem/images/hosts-ui.png b/docs/en/siem/images/hosts-ui.png new file mode 100644 index 000000000..2569df8f4 Binary files /dev/null and b/docs/en/siem/images/hosts-ui.png differ diff --git a/docs/en/siem/images/network-ui.png b/docs/en/siem/images/network-ui.png new file mode 100644 index 000000000..34cde749c Binary files /dev/null and b/docs/en/siem/images/network-ui.png differ diff --git a/docs/en/siem/images/overview-ui.png b/docs/en/siem/images/overview-ui.png new file mode 100644 index 000000000..a34b2fea0 Binary files /dev/null and b/docs/en/siem/images/overview-ui.png differ diff --git a/docs/en/siem/images/siem-architecture.png b/docs/en/siem/images/siem-architecture.png index b051cb379..0ab2915d3 100644 Binary files a/docs/en/siem/images/siem-architecture.png and b/docs/en/siem/images/siem-architecture.png differ diff --git a/docs/en/siem/images/timeline-ui.png b/docs/en/siem/images/timeline-ui.png new file mode 100644 index 000000000..9a9c1d9c7 Binary files /dev/null and b/docs/en/siem/images/timeline-ui.png differ diff --git a/docs/en/siem/index.asciidoc b/docs/en/siem/index.asciidoc index 2d866b6af..a25e854c1 100644 --- a/docs/en/siem/index.asciidoc +++ b/docs/en/siem/index.asciidoc @@ -1,17 +1,18 @@ :doctype: book -:siem-soln-cap: SIEM Monitoring -:siem-soln: SIEM monitoring +:siem-soln: SIEM +:siem-app: SIEM app :siem-ui: SIEM UI -= SIEM Solution Guide -//include::{asciidoc-dir}/../../shared/versions.asciidoc[] += SIEM Guide (Beta) + +include::{asciidoc-dir}/../../shared/versions.asciidoc[] include::{asciidoc-dir}/../../shared/attributes.asciidoc[] include::overview.asciidoc[] -//include::installation.asciidoc[] +include::installation.asciidoc[] -//include::siem-ui.asciidoc[] +include::siem-ui.asciidoc[] diff --git a/docs/en/siem/installation.asciidoc b/docs/en/siem/installation.asciidoc index 036a07ea0..71a635f4f 100644 --- a/docs/en/siem/installation.asciidoc +++ b/docs/en/siem/installation.asciidoc @@ -4,11 +4,10 @@ beta[] -To get up and running with security monitoring, you need: +You need: -* An Elasticsearch cluster and Kibana (version 6.x or later) with a basic -license. To learn how to get started quickly, see -{stack-gs}/get-started-elastic-stack.html[Getting started with the {stack}]. +* An *Elasticsearch* cluster and *Kibana* (version 7.2 or later) with a basic +license. See {stack-gs}/get-started-elastic-stack.html[Getting started with the {stack}]. + [TIP] ============== @@ -19,24 +18,82 @@ https://www.elastic.co/cloud/elasticsearch-service/signup[Try the {es} Service for free]. ============== -* {beats} shippers (version 6.x or later) installed on each system you want to +* *{beats}* shippers (version 7.x or later) installed for each system you want to monitor You might need to modify UI settings in {kib} to change default behaviors, -such as the index pattern used to query the data, and the timestamp field used -for sorting. For more information, see {kib}. +such as the index pattern used to query the data. For more information, see {kib}. [float] [[install-beats]] === Install {beats} shippers -To populate the security UI with metrics and -log data, you need to install and configure the following shippers: +To populate the {siem-app} with hosts and network security events, you need to install and +configure Beats on the systems from which you want to ingest security events: -* https://www.elastic.co/products/beats/packetbeat[{packetbeat}] for analyzing -network packets * https://www.elastic.co/products/beats/filebeat[{filebeat}] for forwarding and centralizing logs and files -* https://www.elastic.co/products/beats/auditbeat[{auditbeat}] for monitoring -directories for file changes +* https://www.elastic.co/products/beats/auditbeat[{auditbeat}] for collecting security events +* https://www.elastic.co/products/beats/winlogbeat[{winlogbeat}] for centralizing +Windows event logs +* https://www.elastic.co/products/beats/packetbeat[{packetbeat}] for analyzing +network packets + +NOTE: {siem-soln} also works with custom and third-party data sources in addition to +those supported by Beats. {ecs-ref}[Elastic Common Schema (ECS)] makes this +possible. + +You can install {beats} using a {kib}-based guide or directly from the command line. + +[float] +==== Install {beats} using the {kib}-based guide + +Follow the instructions in the Add Data section of the {kib} home page. Click +*Add log data* or *Add metrics*, and follow the links for the types of data you +want to collect. + +[role="screenshot"] +image::add-data.png[] + +[float] +==== Download and install {beats} from the command line + +If your data source isn't in the list, or you want to install {beats} the old +fashioned way: + +* *{filebeat} and {filebeat} modules.* See the +{filebeat-ref}/filebeat-modules-quickstart.html[{filebeat} modules quick start] +and enable modules for the events you want to collect. If there is no module +for the events you want to collect, see the +{filebeat-ref}/filebeat-getting-started.html[{filebeat} getting started] to +learn how to configure inputs. + +* *Auditbeat.* See {auditbeat-ref}/auditbeat-getting-started.html[{auditbeat} getting started]. + +* *Winlogbeat.* See {winlogbeat-ref}/winlogbeat-getting-started.html[{winlogbeat} getting started]. + +* *Packetbeat.* See {packetbeat-ref}/packetbeat-getting-started.html[{packetbeat} getting started]. + +[float] +=== Enable modules and configuration options + +For either approach, you need to enable modules in {auditbeat} and {filebeat} +to populate the {SIEM-app} with data. + +To populate *Hosts* data, enable these {auditbeat} modules: + +* {auditbeat-ref}/auditbeat-module-system.html[System module - Linux, macOS, Win] +* {auditbeat-ref}/auditbeat-module-auditd.html[Auditd module (Linux Kernel Audit info)] +* {auditbeat-ref}/auditbeat-module-file_integrity.html[File integrity module (FIM) - Linux, macOS, Win] + + +To populate *Network* data, enable these {filebeat} modules: +* https://www.elastic.co/products/beats/filebeat[{filebeat}] +* {filebeat-ref}/filebeat-module-zeek.html[Zeek NMS module] +* {filebeat-ref}/filebeat-module-suricata.html[Suricata IDS module] +* {filebeat-ref}/filebeat-module-iptables.html[Iptables/Ubiquiti module] +* {filebeat-ref}/filebeat-module-coredns.html[CoreDNS module] +* {filebeat-ref}/filebeat-module-envoyproxy.html[Envoy proxy module (Kubernetes)] +* {filebeat-ref}/filebeat-module-panw.html[Palo Alto Networks firewall module] +* {filebeat-ref}//filebeat-module-cisco.html[Cisco ASA firewall module] diff --git a/docs/en/siem/overview.asciidoc b/docs/en/siem/overview.asciidoc index 891c5e662..e20f0a442 100644 --- a/docs/en/siem/overview.asciidoc +++ b/docs/en/siem/overview.asciidoc @@ -1,35 +1,29 @@ [[siem-overview]] -//[role="xpack"] -//== Overview +[role="xpack"] +== Overview - -== Coming soon - -Won't be long now! - -//// beta[] -{siem-soln-cap} gives you a comprehensive view into your security operations, -and helps make those insights actionable. +{siem-soln} enables analysis of host-related and network-related security events +as part of alert investigations or interactive threat hunting. -The UI in {kib} brings together data from a variety of sources, making it easier -for you to identify and resolve security issues. +The {siem-app} in {kib} provides an interactive workspace for security teams to +triage events and perform initial investigations. [float] [[siem-components]] -=== SIEM monitoring components +=== SIEM components -Security monitoring requires the following {stack} components. +SIEM requires the following {stack} components. image::images/siem-architecture.png[] -*https://www.elastic.co/products/beats[{beats}]* are open source data -shippers that you install as agents on your servers to send operational data to -{es}. +*https://www.elastic.co/products/beats[{beats}]* are open source data shippers +that you install as agents on your systems. {beats} send security events and other +data to {es}. *https://www.elastic.co/products/elasticsearch[{es}]* is a real-time, -distributed storage, search, and analytics engine. {es} excels is indexing +distributed storage, search, and analytics engine. {es} excels at indexing streams of semi-structured data, such as logs or metrics. *https://www.elastic.co/products/kibana[{kib}]* is an open source analytics and @@ -38,5 +32,79 @@ view, and interact with data stored in {es} indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps. -{kib} {siem-ui} provides a dedicated user interface for visualizing host security. -//// +The {siem-app} in {kib} provides a dedicated user interface for analyzing and +investigating host and network security events. + +[float] +[[siem-integration]] +==== Additional Elastic components + +You can use {siem-soln} with other Elastic products and features to help you +identify and investigate suspicious activity: + +* https://www.elastic.co/products/stack/machine-learning[{ml-cap}] +* https://www.elastic.co/products/stack/alerting[Alerting] +* https://www.elastic.co/products/stack/canvas[Canvas] + +[float] +[[data-sources]] +=== Data sources + +SIEM can ingest and analyze data from a variety of sources, including Beats +and Beats modules, and third-party collectors mapped to the {ecs-ref}[Elastic +Common Schema (ECS)]. + +[float] +[[hosts-data-sources]] +==== Hosts data sources + +* https://www.elastic.co/products/beats/auditbeat[{auditbeat}] +** {auditbeat-ref}/auditbeat-module-system.html[System module - Linux, macOS, Win] +*** packages +*** processes +*** logins +*** sockets +*** users and groups +** {auditbeat-ref}/auditbeat-module-auditd.html[Auditd module (Linux Kernel Audit info)] +** {auditbeat-ref}/auditbeat-module-file_integrity.html[File integrity module (FIM) - Linux, macOS, Win] +* https://www.elastic.co/products/beats/filebeat[{filebeat}] +** system logs (auth logs) - Linux +** Santa - macOS +* https://www.elastic.co/products/beats/winlogbeat[{winlogbeat}] +** Windows event logs - Windows + +[float] +[[network-data-sources]] +==== Network data sources + +* https://www.elastic.co/products/beats/packetbeat[{packetbeat}] +** Flows +** DNS +** other protocols +* https://www.elastic.co/products/beats/filebeat[{filebeat}] +** {filebeat-ref}/filebeat-module-zeek.html[Zeek NMS module] +** {filebeat-ref}/filebeat-module-suricata.html[Suricata IDS module] +** {filebeat-ref}/filebeat-module-iptables.html[Iptables/Ubiquiti module] +** {filebeat-ref}/filebeat-module-coredns.html[CoreDNS module] +** {filebeat-ref}/filebeat-module-envoyproxy.html[Envoy proxy module (Kubernetes)] +** {filebeat-ref}/filebeat-module-panw.html[Palo Alto Networks firewall module] +** {filebeat-ref}//filebeat-module-cisco.html[Cisco ASA firewall module] + +[float] +[[ecs]] +==== Elastic Common Schema (ECS) for normalizing data + +The {ecs-ref}[Elastic Common Schema (ECS)] defines a common set of fields to be used for +storing event data in Elasticsearch. ECS helps users normalize their event data +to better analyze, visualize, and correlate the data represented in their +events. + +{siem-soln} can ingest and normalize events from ECS-compatible data sources. + +[float] +[[host_id]] +===== Host identification +All Beats use the `add_host_metadata` processor to add the `host.name` field to +events. The default value is `host.name`, but you can change it in Beats +processor settings. + diff --git a/docs/en/siem/siem-ui.asciidoc b/docs/en/siem/siem-ui.asciidoc index f473227b4..0c7b638bc 100644 --- a/docs/en/siem/siem-ui.asciidoc +++ b/docs/en/siem/siem-ui.asciidoc @@ -4,11 +4,160 @@ beta[] -After you have security monitoring <> and data is streaming to {es}, use the {siem-ui} in {kib} to monitor -and identify security problems in real time. +After you have your {siem-soln} data sources <> and +security events are streaming to {es}, check out the {siem-app} in {kib}. You +can view and analyze security events, investigate previous threat detections, or +hunt for new potential security issues. -For more information about working with the {siem-ui}, see the -{kib} documentation. +The {siem-app} is a highly interactive workspace for security analysts. It is +designed to be discoverable, clickable, draggable and droppable, expandable and +collapsible, resizable, moveable, and so forth. +The *{kibana-ref}/kuery-query.html[{kib} Query Language (KQL)]* bar is available +throughout the {siem-app} for searching and filtering. + +NOTE: The default index patterns for {siem-soln} events are `auditbeat-*`, `winlogbeat-*`, +`filebeat-*`, and `packetbeat-*`. You can change the default index patterns in +{kib} -> Management -> Advanced Settings -> `siem:defaultIndex`. + + +[float] +[[siem-overview-ui]] +=== Overview + +The Overview page provides a high-level view into security events available +for analysis, and can help surface problems with data ingestion. + +Notice the *Timeline* on the right of the {siem-ui}. It is always available for +starting an investigation, whether in its collapsed or expanded state. Identify +items you want to dig into, and drag and drop them to the Timeline. Those items +are waiting for you when you are ready to start your investigation. + +[role="screenshot"] +image::overview-ui.png[] + +[float] +[[hosts-ui]] +=== Hosts + +The Hosts view provides key metrics regarding host-related security +events, and a set of data tables that let you interact with the Timeline Event +Viewer. You can drag and drop items of interest from the Hosts view tables to +Timeline for further investigation. + +[role="screenshot"] +image::hosts-ui.png[] + +Interactive table widgets let you drill down for deeper insights: + +* Hosts +* Unique IPs +* User Authentications (success and failures) +* Uncommon Processes +* Events + + +*Hosts Detail* shows information for a selected host, including +Host ID, First Seen timestamp, Last Seen timestamp, IP and MAC addresses, OS, +versions, machine type, and so forth. + +[float] +[[network-ui]] +=== Network + +The Network view provides key network activity metrics, facilitates +investigation time enrichment, and provides network event tables that enable +interaction with the Timeline. You can drag and drop items of interest from the +Network view to Timeline for further investigation. + +[role="screenshot"] +image::network-ui.png[] + +Interactive table widgets let you drill down for deeper insights: + +* Top Talkers +* Top DNS Domains +* IP Details +* Domains +* Users +* Transport Layer Security certs + + +[float] +[[timelines-ui]] +=== Timelines + +Use timelines as your workspace for alert investigations or threat hunting. + +You can drag objects of interest into the Timeline Event Viewer to create +exactly the query filter you need to get to the bottom of an alert. You can drag +items from table widgets within Hosts and Network pages, or even from within +Timeline itself. + +A timeline is responsive and persists as you move through the {siem-app} +collecting data. Auto-saving ensures that the results of your investigation are +available for review by other analysts and incident response teams. + + +[role="screenshot"] +image::timeline-ui.png[] + +Add notes for your own use and to communicate your workflow and findings to +others. You can share a timeline, or pass it off to another person or team. You +can link to timelines from a ticketing system. + + +[float] +[[raw]] +==== See raw event data + +Many security events in Timeline are presented in an easy-to-follow rendered +view, which enables quicker analyst understanding. You can click and expand +events from within Timeline to see the underlying event data, either in tabular +form, as as {es} JSON. + +[float] +[[narrow-expand]] +==== Narrow or expand your query + +You can specify logical `AND` and `OR` operations with an item's placement in +the drop area. Horizontal filters are `AND`-ed together. Vertical filters or +sets are `OR`-ed together. As you hover the item over the drop area, you can see +whether your placement is on target to create an `AND` or `OR` filters. + +[float] +[[pivot]] +==== Pivot your data + +Click a filter to access additional operations such as exclude, temporarily +disable, or delete items from the query. For example, you can change an included +item so that it is excluded. + +[float] +[[row-renderer]] +==== Get more context for each event +As you build and modify your queries, you can see the results of your +interactions in the details pane below. + +As your query takes shape, an easy-to-follow rendered view appears for events. It +shows relevant contextual information that helps tell the backstory of the +event. If you see a particular item that interests you, you can drag it to the +drop area for further introspection. + +[float] +[[other]] +==== Other actions + +The Timeline is flexible and highly interactive. As you would expect, the +{siem-app} lets you: + +* add, remove, reorder, or resize Timeline columns. +* save, open, and list Timelines +* add notes to individual events +* add investigation notes for the whole Timeline +* pin events to the Timeline for persistence + +Try clicking to expand or collapse items, or dragging and dropping them to other +areas to see what happens. Are there interactions that you would expect to see +that aren't present? Let us know. We welcome your input.