Cherry-pick #19197 to 7.x: [Filebeat][New Module] Adding support for …

…Microsoft Defender ATP (#19907)

* [Filebeat][New Module] Adding support for Microsoft Defender ATP (#19197)

What does this PR do?

This PR adds the initial beta support for Microsoft Defender ATP.
Why is it important?

Adds new products to the supported list for filebeat

Co-authored-by: Lee E. Hinman <lee.e.hinman@elastic.co>
Co-authored-by: Marius Iversen <pillus@chasenet.org>

(cherry picked from commit fead071)

CHANGELOG.next.asciidoc

@@ -520,6 +520,7 @@ field. You can revert this change by configuring tags for the module and omittin
 - Improve ECS categorization field mappings in azure module. {issue}16155[16155] {pull}19376[19376]
 - Add automatic retries and exponential backoff to httpjson input. {pull}18956[18956]
 - Add text & flattened versions of fields with unknown subfields in aws cloudtrail fileset. {issue}18866[18866] {pull}19121[19121]
+- Added Microsoft Defender ATP Module. {issue}17997[17997] {pull}19197[19197]
 - Add initial support for configurable file identity tracking. {pull}18748[18748]
 
 *Heartbeat*

filebeat/docs/fields.asciidoc

@@ -44,6 +44,7 @@ grouped in the following categories:
 * <<exported-fields-kubernetes-processor>>
 * <<exported-fields-log>>
 * <<exported-fields-logstash>>
+* <<exported-fields-microsoft>>
 * <<exported-fields-misp>>
 * <<exported-fields-mongodb>>
 * <<exported-fields-mssql>>
@@ -25739,6 +25740,197 @@ alias to: event.duration
 
 --
 
+[[exported-fields-microsoft]]
+== Microsoft fields
+
+Microsoft Module
+
+
+
+[float]
+=== microsoft
+
+Fields from Microsoft ATP
+
+
+
+[float]
+=== defender_atp
+
+Module for ingesting Microsoft Defender ATP.
+
+
+
+*`microsoft.defender_atp.lastUpdateTime`*::
++
+--
+The date and time (in UTC) the alert was last updated.
+
+
+type: date
+
+--
+
+*`microsoft.defender_atp.resolvedTime`*::
++
+--
+The date and time in which the status of the alert was changed to 'Resolved'.
+
+
+type: date
+
+--
+
+*`microsoft.defender_atp.incidentId`*::
++
+--
+The Incident ID of the Alert.
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.investigationId`*::
++
+--
+The Investigation ID related to the Alert.
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.investigationState`*::
++
+--
+The current state of the Investigation.
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.assignedTo`*::
++
+--
+Owner of the alert.
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.status`*::
++
+--
+Specifies the current status of the alert. Possible values are: 'Unknown', 'New', 'InProgress' and 'Resolved'.
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.classification`*::
++
+--
+Specification of the alert. Possible values are: 'Unknown', 'FalsePositive', 'TruePositive'.
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.determination`*::
++
+--
+Specifies the determination of the alert. Possible values are: 'NotAvailable', 'Apt', 'Malware', 'SecurityPersonnel', 'SecurityTesting', 'UnwantedSoftware', 'Other'.
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.threatFamilyName`*::
++
+--
+Threat family.
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.rbacGroupName`*::
++
+--
+User group related to the alert
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.evidence.domainName`*::
++
+--
+Domain name related to the alert
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.evidence.ipAddress`*::
++
+--
+IP address involved in the alert
+
+
+type: ip
+
+--
+
+*`microsoft.defender_atp.evidence.aadUserId`*::
++
+--
+ID of the user involved in the alert
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.evidence.accountName`*::
++
+--
+Username of the user involved in the alert
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.evidence.entityType`*::
++
+--
+The type of evidence
+
+
+type: keyword
+
+--
+
+*`microsoft.defender_atp.evidence.userPrincipalName`*::
++
+--
+Principal name of the user involved in the alert
+
+
+type: keyword
+
+--
+
 [[exported-fields-misp]]
 == MISP fields
 

filebeat/docs/modules/microsoft.asciidoc

@@ -0,0 +1,167 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-microsoft]]
+[role="xpack"]
+
+:modulename: microsoft
+
+== Microsoft module
+
+This is a module for ingesting data from the different Microsoft Products. Currently supports these filesets:
+
+- `defender_atp` fileset: Supports Microsoft Defender ATP
+- `dhcp` fileset: Supports Microsoft DHCP logs
+
+include::../include/gs-link.asciidoc[]
+
+[float]
+=== Compatibility
+
+Currently this module supports Microsoft Defender ATP.
+
+include::../include/configuring-intro.asciidoc[]
+
+:fileset_ex: defender_atp
+
+include::../include/config-option-intro.asciidoc[]
+
+To allow the filebeat module to ingest data from the Microsoft Defender API, you would need to create a new application on your Azure domain.
+
+The procedure to create an application is found on the below link:
+
+https://docs.microsoft.com/en-us/windows/security/threat-protection/microsoft-defender-atp/exposed-apis-create-app-webapp[Create a new Azure Application]
+
+When giving the application the API permissions described in the documentation (Windows Defender ATP Alert.Read.All) it will only grant access to read alerts from ATP and nothing else in the Azure Domain.
+
+After the application has been created, it should contain 3 values that you need to apply to the module configuration.
+
+These values are:
+
+Client ID
+Client Secret
+Tenant ID
+
+[float]
+==== `defender_atp` fileset settings
+
+[source,yaml]
+----
+- module: microsoft
+  defender_atp:
+    enabled: true
+    var.oauth2.client.id: "123abc-879546asd-349587-ad64508"
+    var.oauth2.client.secret: "980453~-Sg99gedf"
+    var.oauth2.token_url: "https://login.microsoftonline.com/INSERT-TENANT-ID/oauth2/token"
+----
+
+include::../include/var-paths.asciidoc[]
+
+*`var.oauth2.client.id`*::
+
+This is the client ID related to creating a new application on Azure.
+
+*`var.oauth2.client.secret`*::
+
+The secret related to the client ID.
+
+*`var.oauth2.token_url`*::
+
+A predefined URL towards the Oauth2 service for Microsoft. The URL should always be the same with the exception of the Tenant ID that needs to be added to the full URL.
+
+[float]
+==== Defender ATP ECS fields
+
+This is a list of Defender ATP fields that are mapped to ECS.
+
+[options="header"]
+|======================================================================|
+| Defender ATP Fields                 | ECS Fields                     |
+| alertCreationTime                   | @timestamp                     |
+| aadTenantId                         | cloud.account.id               |
+| category                            | threat.technique.name          |
+| computerDnsName                     | host.hostname                  |
+| description                         | rule.description               |
+| detectionSource                     | observer.name                  |
+| evidence.fileName                   | file.name                      |
+| evidence.filePath                   | file.path                      |
+| evidence.processId                  | process.pid                    |
+| evidence.processCommandLine         | process.command_line           |
+| evidence.processCreationTime        | process.start                  |
+| evidence.parentProcessId            | process.parent.pid             |
+| evidence.parentProcessCreationTime  | process.parent.start           |
+| evidence.sha1                       | file.hash.sha1                 |
+| evidence.sha256                     | file.hash.sha256               |
+| evidence.url                        | url.full                       |
+| firstEventTime                      | event.start                    |
+| id                                  | event.id                       |
+| lastEventTime                       | event.end                      |
+| machineId                           | cloud.instance.id              |
+| relatedUser.userName                | host.user.name                 |
+| relatedUser.domainName              | host.user.domain               |
+| title                               | message                        |
+| severity                            | event.severity                 |
+|======================================================================|
+
+== Microsoft module
+
+experimental[]
+
+:fileset_ex: dhcp
+
+[float]
+==== `dhcp` fileset settings
+
+experimental[]
+
+NOTE: This was converted from RSA NetWitness log parser XML "msdhcp" device revision 99.
+
+*`var.input`*::
+
+The input from which messages are read. One of `file`, `tcp` or `udp`.
+
+*`var.syslog_host`*::
+
+The address to listen to UDP or TCP based syslog traffic.
+Defaults to `localhost`.
+Set to `0.0.0.0` to bind to all available interfaces.
+
+*`var.syslog_port`*::
+
+The port to listen for syslog traffic. Defaults to `9515`
+
+NOTE: Ports below 1024 require Filebeat to run as root.
+
+*`var.tz_offset`*::
+
+By default, datetimes in the logs will be interpreted as relative to
+the timezone configured in the host where {beatname_uc} is running. If ingesting
+logs from a host on a different timezone, use this field to set the timezone
+offset so that datetimes are correctly parsed. Valid values are in the form
+±HH:mm, for example, `-07:00` for `UTC-7`.
+
+*`var.rsa_fields`*::
+
+Flag to control the addition of non-ECS fields to the event. Defaults to true,
+which causes both ECS and custom fields under `rsa` to be are added.
+
+*`var.keep_raw_fields`*::
+
+Flag to control the addition of the raw parser fields to the event. This fields
+will be found under `rsa.raw`. The default is false.
+
+:has-dashboards!:
+
+:fileset_ex!:
+
+:modulename!:
+
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-microsoft,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -25,6 +25,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-kafka>>
   * <<filebeat-module-kibana>>
   * <<filebeat-module-logstash>>
+  * <<filebeat-module-microsoft>>
   * <<filebeat-module-misp>>
   * <<filebeat-module-mongodb>>
   * <<filebeat-module-mssql>>
@@ -71,6 +72,7 @@ include::modules/iptables.asciidoc[]
 include::modules/kafka.asciidoc[]
 include::modules/kibana.asciidoc[]
 include::modules/logstash.asciidoc[]
+include::modules/microsoft.asciidoc[]
 include::modules/misp.asciidoc[]
 include::modules/mongodb.asciidoc[]
 include::modules/mssql.asciidoc[]