Add Anomali ThreatStream support to threatintel module (#26350)

* Add Anomali ThreatStream support to threatintel module

This adds a new dataset, `anomalithreatstream` to the threatintel
module. It allows to ingest indicators from Anomali ThreatStream
Integrator via a custom SDK output.

(cherry picked from commit a6d8cdb)

CHANGELOG.next.asciidoc

@@ -586,6 +586,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add `include_s3_metadata` config option to the `aws-s3` input for including object metadata in events. {pull}26267[26267]
 - RFC 5424 and UNIX socket support in the Syslog input are now GA {pull}26293[26293]
 - Update grok patterns for HA Proxy module {issue}25827[25827] {pull}25835[25835]
+- Added dataset `anomalithreatstream` to the `threatintel` module to ingest indicators from Anomali ThreatStream {pull}26350[26350]
 
 *Heartbeat*
 

filebeat/docs/fields.asciidoc

@@ -146191,6 +146191,191 @@ type: keyword
 The STIX reference object.
 
 
+type: keyword
+
+--
+
+[float]
+=== anomalithreatstream
+
+Fields for Anomali ThreatStream
+
+
+
+*`threatintel.anomalithreatstream.classification`*::
++
+--
+Indicates whether an indicator is private or from a public feed and available publicly. Possible values: private, public.
+
+
+type: keyword
+
+example: private
+
+--
+
+*`threatintel.anomalithreatstream.confidence`*::
++
+--
+The measure of the accuracy (from 0 to 100) assigned by ThreatStream's predictive analytics technology to indicators.
+
+
+type: short
+
+--
+
+*`threatintel.anomalithreatstream.detail2`*::
++
+--
+Detail text for indicator.
+
+
+type: text
+
+example: Imported by user 42.
+
+--
+
+*`threatintel.anomalithreatstream.id`*::
++
+--
+The ID of the indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.import_session_id`*::
++
+--
+ID of the import session that created the indicator on ThreatStream.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.itype`*::
++
+--
+Indicator type. Possible values: "apt_domain", "apt_email", "apt_ip", "apt_url", "bot_ip", "c2_domain", "c2_ip", "c2_url", "i2p_ip", "mal_domain", "mal_email", "mal_ip", "mal_md5", "mal_url", "parked_ip", "phish_email", "phish_ip", "phish_url", "scan_ip", "spam_domain", "ssh_ip", "suspicious_domain", "tor_ip" and "torrent_tracker_url".
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.maltype`*::
++
+--
+Information regarding a malware family, a CVE ID, or another attack or threat, associated with the indicator.
+
+
+type: wildcard
+
+--
+
+*`threatintel.anomalithreatstream.md5`*::
++
+--
+Hash for the indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.resource_uri`*::
++
+--
+Relative URI for the indicator details.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.severity`*::
++
+--
+Criticality associated with the threat feed that supplied the indicator. Possible values: low, medium, high, very-high.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.source`*::
++
+--
+Source for the indicator.
+
+
+type: keyword
+
+example: Analyst
+
+--
+
+*`threatintel.anomalithreatstream.source_feed_id`*::
++
+--
+ID for the integrator source.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.state`*::
++
+--
+State for this indicator.
+
+
+type: keyword
+
+example: active
+
+--
+
+*`threatintel.anomalithreatstream.trusted_circle_ids`*::
++
+--
+ID of the trusted circle that imported the indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.update_id`*::
++
+--
+Update ID.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.url`*::
++
+--
+URL for the indicator.
+
+
+type: keyword
+
+--
+
+*`threatintel.anomalithreatstream.value_type`*::
++
+--
+Data type of the indicator. Possible values: ip, domain, url, email, md5.
+
+
 type: keyword
 
 --

filebeat/docs/modules/threatintel.asciidoc

@@ -23,15 +23,17 @@ fields.
 [float]
 === The available filesets are:
 
-* `abuseurl`: Supports gathering URL entities from Abuse.ch.
-* `abusemalware`: Supports gathering Malware/Payload entities from Abuse.ch.
-* `misp`: Supports gathering threat intel attributes from MISP (replaces MISP module).
-* `malwarebazaar`: Supports gathering Malware/Payload entities from Malware Bazaar.
-* `otx`: Supports gathering threat intel attributes from AlientVault OTX.
-* `anomali`: Supports gathering threat intel attributes from Anomali.
+* <<abuseurl,abuseurl>>: Supports gathering URL entities from Abuse.ch.
+* <<abusemalware,abusemalware>>: Supports gathering Malware/Payload entities from Abuse.ch.
+* <<misp,misp>>: Supports gathering threat intel attributes from MISP (replaces MISP module).
+* <<malwarebazaar,malwarebazaar>>: Supports gathering Malware/Payload entities from Malware Bazaar.
+* <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
+* <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
+* <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
 
 include::../include/gs-link.asciidoc[]
 
+[[abuseurl]]
 [float]
 ==== `abuseurl` fileset settings
 
@@ -71,6 +73,7 @@ Abuse.ch URL Threat Intel is mapped to the following ECS fields.
 | host                       | threatintel.indicator.ip/domain
 |==============================================================
 
+[[abusemalware]]
 [float]
 ==== `abusemalware` fileset settings
 
@@ -110,6 +113,7 @@ Abuse.ch Malware Threat Intel is mapped to the following ECS fields.
 | file_size                  | threatintel.indicator.file.size
 |================================================================
 
+[[malwarebazaar]]
 [float]
 ==== `malwarebazaar` fileset settings
 
@@ -164,6 +168,7 @@ Malware Bazaar Threat Intel is mapped to the following ECS fields.
 | code_sign.serial_number    | threatintel.indicator.file.x509.serial_number
 |================================================================
 
+[[misp]]
 [float]
 ==== `misp` fileset settings
 
@@ -241,6 +246,7 @@ MISP Threat Intel is mapped to the following ECS fields.
 
 `misp.value` is mapped to the appropriate field dependent on attribute type.
 
+[[otx]]
 [float]
 ==== `otx` fileset settings
 
@@ -316,6 +322,7 @@ OTX Threat Intel is mapped to the following ECS fields.
 
 `otx.indicator` is mapped to the appropriate field dependent on attribute type.
 
+[[anomali]]
 [float]
 ==== `anomali` fileset settings
 
@@ -397,6 +404,91 @@ Anomali Threat Intel is mapped to the following ECS fields.
 
 `anomali.pattern` is mapped to the appropriate field dependent on attribute type.
 
+[[anomalithreatstream]]
+[float]
+==== `anomalithreatstream` fileset settings
+
+To configure the ThreatStream integration you first need to define an output
+in the Anomali ThreatStream Integrator using the Elastic SDK provided by Anomali.
+It will deliver indicators via HTTP or HTTPS to a Filebeat instance running as
+a server.
+
+Configure an Integrator output with the following settings:
+
+* Indicator Filter: `*` (or use any desired filter).
+* SDK Executable Command: `/path/to/python /path/to/anomali-sdk/main.py`.
+  Adjust the paths to the python executable and the directory where the Elastic SDK
+  has been unpacked.
+* Metadata in JSON Format: `{"url": "https://filebeat:8080/", "server_certificate": "/path/to/cert.pem", "secret": "my secret"}`.
+    - `url`: Use the host and port where Filebeat will be running, and `http` or `https` accordingly.
+    - `server_certificate`: If using HTTPS, absolute path to the server certificate. Otherwise don't set
+        this field.
+    - `secret`: A shared secret string to authenticate messages between the SDK and Filebeat.
+
+Then configure the `anomalithreatstream` fileset in Filebeat accordingly:
+[source,yaml]
+----
+- module: threatintel
+  anomalithreatstream:
+    enabled: true
+    var.input: http_endpoint
+    var.listen_address: 0.0.0.0 # Listen on all interfaces.
+    var.listen_port: 8080
+    var.secret: 'my secret'
+    var.ssl_certificate: path/to/server_ssl_cert.pem
+    var.ssl_key: path/to/ssl_key.pem
+----
+
+*`var.listen_address`*::
+
+Local address to bind the HTTP server to. Use `0.0.0.0` to accept connections
+from all interfaces.
+
+*`var.listen_port`*::
+
+Port number to use for the HTTP server.
+
+*`var.secret`*::
+
+Shared secret between the SDK and Filebeat, used to authenticate messages.
+
+*`var.ssl_certificate`*::
+
+Path to the public SSL certificate for the HTTPS server. If unset, Filebeat
+will use unsecure HTTP connections.
+
+*`var.ssl_key`*::
+
+Path to the certificate's private key.
+
+Anomali ThreatStream fields are mapped to the following ECS fields:
+
+[options="header"]
+|=============================================================
+| ThreatStream fields | ECS Fields
+| asn         | threatintel.indicator.as.number
+| classification<<a,[1]>> | threatintel.indicator.marking.tlp
+| confidence<<a,[1]>> | threatintel.indicator.confidence
+| country     | threatintel.indicator.geo.country_iso_code
+| date_first  | threatintel.indicator.first_seen
+| date_last   | threatintel.indicator.last_seen
+| detail      | tags
+| domain      | threatintel.indicator.domain
+| email       | threatintel.indicator.email.address
+| itype<<a,[1]>> | threatintel.indicator.type
+| lat         | threatintel.indicator.geo.location.lat
+| lon         | threatintel.indicator.geo.location.lon
+| md5         | threatintel.indicator.file.hash
+| org         | threatintel.indicator.as.organization.name
+| severity<<a,[1]>> | event.severity
+| source      | threatintel.indicator.provider
+| srcip       | threatintel.indicator.ip
+| url         | threatintel.indicator.url.original
+|=============================================================
+
+[[a]]
+[small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomalithreatstream`.#
+
 :has-dashboards!:
 
 [float]

filebeat/tests/system/test_modules.py

@@ -285,6 +285,7 @@ def clean_keys(obj):
         "threatintel.abuseurl",
         "threatintel.abusemalware",
         "threatintel.anomali",
+        "threatintel.anomalithreatstream",
         "threatintel.malwarebazaar",
         "snyk.vulnerabilities",
         "snyk.audit",

x-pack/filebeat/filebeat.reference.yml

@@ -2249,6 +2249,31 @@ filebeat.modules:
     # The interval to poll the API for updates
     var.interval: 5m
 
+  anomalithreatstream:
+    enabled: true
+
+    # Input used for ingesting threat intel data
+    var.input: http_endpoint
+
+    # Address to bind to in order to receive HTTP requests
+    # from the Integrator SDK. Use 0.0.0.0 to bind to all
+    # existing interfaces.
+    var.listen_address: localhost
+
+    # Port to use to receive HTTP requests from the
+    # Integrator SDK.
+    var.listen_port: 8080
+
+    # Secret key to authenticate requests from the SDK.
+    var.secret: '<Add your secret here>'
+
+    # Uncomment the following and set the absolute paths
+    # to the server SSL certificate and private key to
+    # enable HTTPS secure connections.
+    #
+    # var.ssl_certificate: path/to/server_ssl_cert.pem
+    # var.ssl_key: path/to/ssl_key.pem
+
 #---------------------------- Apache Tomcat Module ----------------------------
 - module: tomcat
   log: