Add integration for Recorded Future threat intel #2757

adriansr · 2022-03-01T20:36:48Z

What does this PR do?

Adds a new integration to collect threat indicators from Recorded Future.

This supersedes the existing (recalled) Filebeat integration.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Review version number (0.1.0) and beta tag.
Review current approach as to data_streams.
Missing dashboard.

See comment below.

Related issues

Screenshots

adriansr · 2022-03-01T20:52:17Z

Note to reviewers.

I have a few doubts:

We're making this beta to start with, which makes sense. I don't think it will need too many changes before becoming ga. I was unusure what the right first version number will be in this case. 0.1.0 looks more suited for an experimental package. 1.0.0 is better reserved for when it goes ga. How does 0.9.0 sound?
The functionality of this package is a bit different. Due to single input limitations and Recorded Future separating the entities to ingest in different API endpoints (url, ip, domain and hash), the (common?) use case of ingesting more than one kind of data will require setting up multiple integrations. I've considered separate datastreams for each kind, but it will result in a lot of repetition (sample pipelines, same fields, same tests...). WDYT about this choice?
After discussion with @P1llus, we want to take our time revamping the original Filebeat dashboard to align with the other ti_* packages. For now I think it's better to merge the initial version without a dashboard.

elasticmachine · 2022-03-01T20:52:34Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-03-01T20:52:58Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-16T08:05:04.523+0000
Duration: 15 min 23 sec

Test stats 🧪

Test	Results
Failed	0
Passed	11
Skipped	0
Total	11

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

efd6

Nice.

efd6 · 2022-03-10T01:39:55Z

packages/ti_recordedfuture/manifest.yml

@@ -0,0 +1,29 @@
+name: ti_recordedfuture
+title: Recorded Future
+version: 0.1.0


I think this is the right choice FWIW. If it were experimental I'd say 0.0.1, though with semver<1.0.0 the semantics are essentially free.

efd6 · 2022-03-10T01:48:24Z

...i_recordedfuture/data_stream/threat/_dev/test/pipeline/test-domain-default.log-expected.json

@@ -0,0 +1,852 @@
+{
+    "expected": [
+        null,


Is this null expected? (also in some other inputs below: test-hash-default.log-expected.json, test-ip-default.log-expected.json and test-url-default.log-expected.json).

I assume the null is for the header line and that line is dropped.

That's right, is the header line. This only happens with CSV input from file.

andrewkroh

As always, a well documented pipeline 👍

andrewkroh · 2022-03-10T01:59:12Z

packages/ti_recordedfuture/_dev/build/docs/README.md

+
+Alternatively, it's also possible to use the integration to fetch custom Fusion files
+by supplying the URL to the CSV file as the _Custom_ _URL_ configuration option.
+


Are there any links you can add to publicly available Recorded Future docs about the API or schema?

Unfortunately the API docs now require signing up to RecordedFuture's support portal.

https://support.recordedfuture.com/hc/en-us/articles/115000897248

andrewkroh · 2022-03-10T02:00:40Z

...i_recordedfuture/data_stream/threat/_dev/test/pipeline/test-domain-default.log-expected.json

@@ -0,0 +1,852 @@
+{
+    "expected": [
+        null,


I assume the null is for the header line and that line is dropped.

andrewkroh · 2022-03-10T02:05:03Z

packages/ti_recordedfuture/data_stream/threat/elasticsearch/ingest_pipeline/default.yml

+      field: threat.feed.name
+      value: "Recorded Future"
+#
+# TODO: Add dashboard


What's the plan for addressing this TODO? Separate PR?

And given that there is a commented out constant_keyword version with a static value in the fields.yml, do we need this one at all?

The commented values could be moved to base-fields.yml, similar to how the other TI packages does it, but they should not be added before the dashboards as they are required, else it will break UI components

So for now we can leave as is and then add the dashboard in a follow up PR, is that ok?

andrewkroh · 2022-03-10T02:20:01Z

packages/ti_recordedfuture/data_stream/threat/elasticsearch/ingest_pipeline/default.yml

+        - event.dataset
+        - threat.indicator.type
+        - json.Name
+      target_field: "_id"


Do we expect to receive updates for the data and discard it due to the append only nature of data streams? I just want to understand the expected behavior and maybe document it in the pipeline comments.

I think mostly all the TI sources has it, before it was for appending indeed, but there are scenarios which could result in duplicates, so it's just to make sure values are not stored twice.

Yeah this seems useless in data streams, unless the Kibana alerting side somehow uses the ID to deduplicate.

P1llus

I don't have anything extra to add compared to the other reviewers. I added some comments into a few review notes by Andrew

Otherwise the fingerprint processor calculating `id` field doesn't have access to this constant_keyword field as it is not yet set in the document.

adriansr added Integration:ti_util Threat Intelligence Utilities Team:Security-External Integrations labels Mar 1, 2022

adriansr added 8.1 candidate 8.1-candidate enhancement New feature or request labels Mar 1, 2022

adriansr requested review from P1llus and a team March 1, 2022 20:53

efd6 reviewed Mar 10, 2022

View reviewed changes

andrewkroh reviewed Mar 10, 2022

View reviewed changes

P1llus reviewed Mar 10, 2022

View reviewed changes

adriansr requested a review from a team as a code owner March 14, 2022 16:46

adriansr added 7 commits March 14, 2022 17:59

Add Threat Intel integration for Recorded Future

8d9dae2

Correct version and changelog

9d3f0ee

Lint

f7cd424

Re-generate pipeline tests

b294cf7

Set event.dataset explicitly in pipeline

8fb4595

Otherwise the fingerprint processor calculating `id` field doesn't have access to this constant_keyword field as it is not yet set in the document.

Codeowners

5f4fbb4

Re-generate

71683f7

adriansr requested review from P1llus, andrewkroh and efd6 March 15, 2022 13:45

andrewkroh approved these changes Mar 15, 2022

View reviewed changes

P1llus approved these changes Mar 16, 2022

View reviewed changes

jsoriano approved these changes Mar 16, 2022

View reviewed changes

adriansr merged commit 28fd9fa into elastic:main Mar 16, 2022

adriansr deleted the add_recordedfuture_ti branch March 16, 2022 10:30

This was referenced May 11, 2022

Add docs for the Recorded Future fileset elastic/beats#31596

Closed

[BUG] Recorded Future integration can be added to Threat Intelligence module elastic/security-docs#1988

Closed

jamiehynds mentioned this pull request Feb 20, 2023

Deprecate Recorded Future Fileset elastic/beats#28029

Open

andrewkroh added Integration:ti_recordedfuture Recorded Future New Integration and removed Integration:ti_util Threat Intelligence Utilities labels Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add integration for Recorded Future threat intel #2757

Add integration for Recorded Future threat intel #2757

adriansr commented Mar 1, 2022 •

edited

Loading

adriansr commented Mar 1, 2022

elasticmachine commented Mar 1, 2022

elasticmachine commented Mar 1, 2022 •

edited

Loading

Build stats

Test stats 🧪

efd6 left a comment

efd6 Mar 10, 2022

efd6 Mar 10, 2022

andrewkroh Mar 10, 2022

adriansr Mar 14, 2022

andrewkroh left a comment

andrewkroh Mar 10, 2022

adriansr Mar 14, 2022

andrewkroh Mar 10, 2022

andrewkroh Mar 10, 2022

P1llus Mar 10, 2022

adriansr Mar 15, 2022

andrewkroh Mar 10, 2022 •

edited

Loading

P1llus Mar 10, 2022

adriansr Mar 15, 2022

P1llus left a comment


		Alternatively, it's also possible to use the integration to fetch custom Fusion files
		by supplying the URL to the CSV file as the _Custom_ _URL_ configuration option.

Add integration for Recorded Future threat intel #2757

Add integration for Recorded Future threat intel #2757

Conversation

adriansr commented Mar 1, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

Related issues

Screenshots

adriansr commented Mar 1, 2022

elasticmachine commented Mar 1, 2022

elasticmachine commented Mar 1, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

efd6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh Mar 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

P1llus left a comment

Choose a reason for hiding this comment

adriansr commented Mar 1, 2022 •

edited

Loading

elasticmachine commented Mar 1, 2022 •

edited

Loading

andrewkroh Mar 10, 2022 •

edited

Loading