[Filebeat] Remove RecordedFuture dataset from Threat Intel module (#3…

…0564) (#30568) It's been decided to make this integration available only as a package. Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
elastic · Feb 24, 2022 · 4eed120 · 4eed120
1 parent 7b50258
commit 4eed120
Show file tree

Hide file tree

Showing 46 changed files with 2 additions and 11,217 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -18,6 +18,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Filebeat*
 
+- Remove Recorded Future integration from threatintel module. {pull}30564[30564]
 
 *Heartbeat*
 
@@ -108,7 +109,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add support in httpjson input for oAuth2ProviderDefault of password grant_type. {pull}29087[29087]
 - Add support for filtering in journald input with `unit`, `kernel`, `identifiers` and `include_matches`. {pull}29294[29294]
 - Add new `userAgent` and `beatInfo` template functions for httpjson input {pull}29528[29528]
-- threatintel module: Add new Recorded Future integration. {pull}30030[30030]
 - Add pipeline in FB's supported hints. {pull}30212[30212]
 
 *Auditbeat*

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -151909,43 +151909,6 @@ type: keyword
 The indicator type, can for example be "domain, email, FileHash-SHA256".
 
 
-type: keyword
-
---
-
-[float]
-=== recordedfuture
-
-Fields for Recorded Future Threat Intel
-
-
-
-*`recordedfuture.evidence_details`*::
-+
---
-List of sightings used as evidence for this indicator.
-
-
-type: flattened
-
---
-
-*`recordedfuture.name`*::
-+
---
-Indicator value.
-
-
-type: keyword
-
---
-
-*`recordedfuture.risk_string`*::
-+
---
-Details of risk rules observed.
-
-
 type: keyword
 
 --

diff --git a/filebeat/docs/images/filebeat-threatintel-recordedfuture.png b/filebeat/docs/images/filebeat-threatintel-recordedfuture.png
diff --git a/filebeat/docs/modules/threatintel.asciidoc b/filebeat/docs/modules/threatintel.asciidoc
@@ -29,7 +29,6 @@ The available filesets are:
 * <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
 * <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
 * <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
-* <<recordedfuture,recordedfuture>>: Supports gathering threat intel attributes from Recorded Future.
 * <<threatq,threatq>>: Supports gathering threat intel attributes from ThreatQuotient.
 
 include::../include/gs-link.asciidoc[]
@@ -515,124 +514,6 @@ Anomali ThreatStream fields are mapped to the following ECS fields:
 [[a]]
 [small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomalithreatstream`.#
 
-[[recordedfuture]]
-[float]
-==== `recordedfuture` fileset settings
-
-The `recordedfuture` fileset fetches risklists from the Recorded Future Connect API.
-It supports `domain`, `hash`, `ip` and `url` entities.
-
-In order to use it you need to define the `entity` and `list` to fetch. Check the
-https://api.recordedfuture.com/index.html[Recorded Future API Explorer] for the
-available lists for each entity.
-
-Sample configuration:
-[source,yaml]
-----
-- module: threatintel
-  recordedfuture:
-    enabled: true
-    var.input: httpjson
-    var.interval: 1h
-    var.api_token: "<RF_TOKEN>"
-    var.list: default
-    var.entity: domain
-----
-
-To fetch threat intelligence from multiple entities and/or lists, you must define more
-than one instance of the module. The following configuration fetches the default list
-for domains every hour and the rfTrending list for hashes every 12 hours:
-
-[source,yaml]
-----
-- module: threatintel
-  recordedfuture:
-    enabled: true
-    var.input: httpjson
-    var.interval: 1h
-    var.api_token: "<RF_TOKEN>"
-    var.list: default
-    var.entity: domain
-- module: threatintel
-  recordedfuture:
-    enabled: true
-    var.input: httpjson
-    var.interval: 12h
-    var.api_token: "<RF_TOKEN>"
-    var.entity: hash
-    var.list: rfTrending
-----
-
-Alternatively, you can use the module to fetch custom Fusion files by setting
-`var.custom_url` to the URL of the Fusion File:
-
-[source,yaml]
-----
-- module: threatintel
-  recordedfuture:
-    enabled: true
-    var.input: httpjson
-    var.interval: 1h
-    var.api_token: "<RF_TOKEN>"
-    var.custom_url: 'https://api.recordedfuture.com/v2/fusion/files/?path=%2Fpublic%2Frisklists%2Fdefault_domain_risklist.csv'
-----
-
-It's also possible to load CSV risklists from a file:
-
-[source,yaml]
-----
-- module: threatintel
-  recordedfuture:
-    enabled: true
-    var.input: file
-    var.paths:
-        - /path/to/risklist.csv
-----
-
-*`var.input`*::
-
-The input to use to fetch indicators. Use `httpjson` to query
-Recorded Future API or `file` to load the indicators from a file.
-
-*`var.api_token`*::
-
-The API token used to access Recorded Future API (RF-Token).
-
-*`var.interval`*::
-
-How often the API is polled for updated information. It is recommended to set this
-to `1h`. For `hash` entities, it's recommended to set this to `12h`.
-
-*`var.entity`*::
-
-The type of entity to fetch. One of `domain`, `hash`, `ip` or `url`.
-
-*`var.list`*::
-
-The indicator list to fetch.
-
-*`var.proxy_url`*::
-
-Optional URL to use as HTTP proxy.
-
-*`var.custom_url`*::
-
-An alternative URL pointing to a CSV risklist. Use this option
-to fetch custom Fusion Files.
-
-Recorded Future fields are mapped to the following ECS fields:
-
-[options="header"]
-|=============================================================
-| Recorded Future fields | ECS Fields
-| entity.name            | threat.indicator.{url,ip,domain,file.hash}
-| entity.type            | threat.indicator.type
-| fileHashes             | threat.indicator.file.hash
-| risk.score             | event.risk_score
-|=============================================================
-
-:has-dashboards!:
-
 [float]
 === Dashboards
 
@@ -675,12 +556,6 @@ image::./images/filebeat-threatintel-misp.png[]
 [float]
 Overview of the information provided by the MISP feed.
 
-[role="screenshot"]
-image::./images/filebeat-threatintel-recordedfuture.png[]
-
-[float]
-Overview of the information provided by the Recorded Future feed.
-
 [[threatq]]
 [float]
 ==== `threatq` fileset settings
@@ -787,12 +662,6 @@ image::./images/filebeat-threatintel-misp.png[]
 [float]
 Overview of the information provided by the MISP feed.
 
-[role="screenshot"]
-image::./images/filebeat-threatintel-recordedfuture.png[]
-
-[float]
-Overview of the information provided by the Recorded Future feed.
-
 [role="screenshot"]
 image::./images/filebeat-threatintel-threatq.png[]
 

diff --git a/x-pack/filebeat/filebeat.reference.yml b/x-pack/filebeat/filebeat.reference.yml
@@ -2225,32 +2225,6 @@ filebeat.modules:
     # var.ssl_certificate: path/to/server_ssl_cert.pem
     # var.ssl_key: path/to/ssl_key.pem
 
-  recordedfuture:
-    enabled: false
-
-    # Input used for ingesting threat intel data
-    var.input: httpjson
-
-    # Set your API Token.
-    var.api_token: "<RF_TOKEN>"
-
-    # The interval to poll the API for updates
-    var.interval: 1h
-
-    # The kind of entity to fetch. One of domain, hash, ip or url.
-    var.entity: domain
-
-    # The list to fetch. See the Recorded Future API Explorer for
-    # valid lists for each kind of entity.
-    var.list: default
-
-    # Uncomment to use a different API endpoint.
-    # The API endpoint used for Recorded Future API calls.
-    # var.endpoint: "https://api.recordedfuture.com/v2"
-
-    # Uncomment to fetch a custom CSV file via URL. Useful for custom Fusion Files.
-    # var.custom_url: "https://api.recordedfuture.com/v2/fusion/files/?path=%2Fhome"
-
   threatq:
     enabled: false
 

diff --git a/x-pack/filebeat/module/threatintel/_meta/config.yml b/x-pack/filebeat/module/threatintel/_meta/config.yml
@@ -138,32 +138,6 @@
     # var.ssl_certificate: path/to/server_ssl_cert.pem
     # var.ssl_key: path/to/ssl_key.pem
 
-  recordedfuture:
-    enabled: false
-
-    # Input used for ingesting threat intel data
-    var.input: httpjson
-
-    # Set your API Token.
-    var.api_token: "<RF_TOKEN>"
-
-    # The interval to poll the API for updates
-    var.interval: 1h
-
-    # The kind of entity to fetch. One of domain, hash, ip or url.
-    var.entity: domain
-
-    # The list to fetch. See the Recorded Future API Explorer for
-    # valid lists for each kind of entity.
-    var.list: default
-
-    # Uncomment to use a different API endpoint.
-    # The API endpoint used for Recorded Future API calls.
-    # var.endpoint: "https://api.recordedfuture.com/v2"
-
-    # Uncomment to fetch a custom CSV file via URL. Useful for custom Fusion Files.
-    # var.custom_url: "https://api.recordedfuture.com/v2/fusion/files/?path=%2Fhome"
-
   threatq:
     enabled: false