[TI_RecordedFuture] Support IoC expiration #5460

kcreddy · 2023-03-07T09:25:32Z

What does this PR do?

Supports IoC (Indicators of Compromise) expiration by:

Removing fingerprint processor from ti_recordedfuture.threat* source indices to allow duplicate documents.
Adding latest transform to create destination indices containing only unexpired/latest IoCs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Related issues

Closes [TI_RecordedFuture] Support IoC expiration #5459

Screenshots

elasticmachine · 2023-03-07T09:36:53Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-05-12T05:55:35.247+0000
Duration: 15 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	11
Skipped	0
Total	11

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-03-07T11:55:20Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`2/2`)	💚 2.983
Classes	100.0% (`2/2`)	💚 2.983
Methods	100.0% (`16/16`)	💚 8.312
Lines	97.549% (`199/204`)	👍 5.505
Conditionals	100.0% (`0/0`)	💚

kcreddy · 2023-03-08T07:31:45Z

/test

elasticmachine · 2023-03-20T13:13:55Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2023-03-20T17:09:29Z

packages/ti_recordedfuture/data_stream/threat/elasticsearch/ingest_pipeline/default.yml

-#
-# Fingerprint event: _id = hash(dataset + indicator type + indicator value)
-#
-  - fingerprint:


Every hour (that's the default interval) this integration downloads the full CSV file. If a given CSV line has already been indexed, is there a need to index it again? Could we fingerprint the raw CSV to avoid duplicating data? (update: It looks like retention is implemented based on event.ingested so this does depend on the data being continuously reindexed again in order to keep a full threat picture in the "latest" index.)

Another step we could take to minimize retaining a bunch of duplicate data is to institute a custom ILM policy for this data such that it keeps a limited amount of the raw data (like a few hours worth). After the data has been read by the latest transforms there is not much use for it. In other words, as long as we have the output of the transform then the original data stream does not need much retention.

Got it, yeah it does make sense. I see there is a way to specify ILM policy in package-spec I can work on it. In the documentation for the integration, we probably need to mention users about how the source polling interval, transform retention_policy, and ILM delete.min_age need to be configured.

Added ILM policy to rollover after 2 days and delete 3 days after rollover. Added README explaining the same.

andrewkroh · 2023-03-20T17:30:16Z

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml

+retention_policy:
+  time:
+    field: event.ingested
+    max_age: 15m


How was this value determined? Does this need to be longer than interval at which data is fetched from the API? What happens if we fetch data every hour, but only retain the last 15 minutes of latest data for an IOC? Does that mean we have 45 minutes where this logs-ti_recordedfuture_latest.threat is empty?

My thinking is that this value needs to be at least as long as the fetch interval plus the sync time delay. Then the

This value can be anything less than the source interval through which the API gets polled.
The reason is:

If it is greater than the interval, expired IOC still remain in destination indices.

If it is less, based on my tests, it wouldn't delete the documents unless the transform runs again, i.e., same interval when documents gets ingested via source polling. In other words, deletion via this retention_policy.time.max_age only happens when the transform is triggered. The fields that determine whether to trigger transform are: frequency and latest.sort.

Changed max_age to 24h after discussion with Jamie as it seems like a reasonable default since this is not a user configurable parameter.

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml

andrewkroh · 2023-03-20T18:07:13Z

packages/ti_recordedfuture/docs/README.md

@@ -16,13 +16,13 @@ An example event for `threat` looks as following:



The docs are going to need to be updated to explain how IoC expiration works and what index to read from when building indicator match rules.

Added README explaining IOC expiration and ILM policy on source indices.

andrewkroh · 2023-03-20T18:09:22Z

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml

@@ -0,0 +1,21 @@
+source:
+  index: 
+  - ".ds-logs-ti_recordedfuture.threat-default*"


What are the downsides to sourcing data from all logs-ti_recordedfuture.threat namespaces?

My understanding was that if multiple namespaces with same dataset is added by the user when setting up the integration, then transform could read from all namespaces/datastreams and deduce the latest.

But also hardcoding to default isn't good either. Also the transform itself is named like this as per spec.. So, in case of default namespace, our transform is being named as: logs-ti_recordedfuture.latest_ioc-default-0.1.0

Updated the index to ".ds-logs-ti_recordedfuture.threat-*. So it assumes only 1 namespace exists for this integration, whatever is its value.

andrewkroh · 2023-04-03T22:51:25Z

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/manifest.yml

+          mapping:
+            ignore_above: 1024
+            type: keyword
+    date_detection: true


Why did you enable date_detection? Do we have dynamic fields that contain date values that are not mapped as date?

Actually, there aren't. Set it to false

kcreddy · 2023-04-21T05:47:40Z

/test

kcreddy · 2023-04-24T09:29:40Z

/test

kcreddy · 2023-04-25T14:47:39Z

Testing with package upgrade:

The transform isn't getting updated, when package is upgraded with elastic-package build && elastic-package stack up --version=8.8.0-SNAPSHOT -d -v -s package-registry.

Test 1: No code change, only modified manifest and changelog

- version: "1.8.0"
  changes:
    - description: IOC upgrade test
      type: enhancement
      link: https://github.com/elastic/integrations/issues/1111

Test 2: Modified transform with max_age: 12h + modified manifest and changelog

- version: "1.9.0"
  changes:
    - description: IOC upgrade test with transform
      type: enhancement
      link: https://github.com/elastic/integrations/issues/2222

Results:

andrewkroh · 2023-04-26T13:26:58Z

@kcreddy Thanks for testing this manually. Unless my expectations are wrong, I assume this is a bug in Fleet. Can you create a new elastic/kibana Fleet issue to get some clarity on on what to expect from transform upgrades in Fleet packages. If this is the expected behavior for transforms then what are the steps required by users to upgrade. We need to know this so we can document it and support users.

kcreddy · 2023-04-27T09:42:39Z

Hey @andrewkroh
There are certain changes to how the transform versioning is done in: elastic/kibana#155453
With this latest SNAPSHOT, I ran few tests again to simulate the issue.

Interestingly, the new SNAPSHOT version doesn’t seem to create destination index with package version suffix, as now the destination index name is exactly as given in the transform.yml:

"dest": {
    "index": "logs-ti_recordedfuture_latest.threat",    (# --- > no version suffix)
    "aliases": []
  },

Also these indices are not deleted. This should work for us now.

However, the transform update issue/bug during the package upgrade is still present. I have added the issue with details here: elastic/kibana#155982 and requested Kyle Pollich to take a look.

kcreddy · 2023-05-03T17:44:46Z

Starting 8.8, when upgrading the transform, the fleet_transform_version needs to be bumped up.
The transform update is now working fine as mentioned in elastic/kibana#155982 (comment)

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml

andrewkroh · 2023-05-09T11:22:48Z

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml

+  index:
+    - "logs-ti_recordedfuture.threat-*"
+dest:
+  index: "logs-ti_recordedfuture_latest.threat-1.0.0"


When would this version be changed? I think we should have a comment here to help maintainers understand when this should happen and what, if any, is the relationship to the transform version.

I have been thinking about whether we should have some versioning convention here. Like perhaps we use a simple one up counter that's not semver and not tied in with the transform version. This value gets incremented only if there is some incompatible (breaking) schema change.

If we did a version change, we would probably need a means to automatically purge the old destination index so that readers using wildcards (logs-*_latest-*) are not getting results from the old index.

Added comments for dest index versioning and a NOTE for future contributors.

That comment is missing when and why the transform index version should be changed

IIUC if we ever need some kind of breaking schema then we would 1) increment dest index version and 2) increment the transform version. Without (2) the change to the destination index would have no effect.

Is there any issue that you know of is tracking the ability to cleanup an orphaned destination index? We should register our requirement in elastic/package-spec and link to the issue in the comment. If we can't have that feature, then next time we need to make a change we would need to pursue some other approach like documenting a manual cleanup.

Like perhaps we use a simple one up counter that's not semver and not tied in with the transform version

What do you think about that? In addition I don't want to it to appear that the transform's dest.index version is related to the package version. That might lead to confusion about why the dest.index did not change after an upgrade. Using a simple -1, -2, ... versioning approach there could accomplish that.

That sounds good. I also agree that transform version has to be different from the dest.index version. Simple versions should also do the trick.

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml

andrewkroh · 2023-05-11T19:01:00Z

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml

@@ -1,6 +1,9 @@
+# Use of "*" to use any namespace defined.


s/any namespace/all namespaces/

andrewkroh · 2023-05-11T19:16:42Z

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml

+# The version suffix on destination index must be explicitly set.
+# NOTE: During version change, ensure through some mechanism old destination index automatically purges so queries on wild card such as logs-*_latest-* doesn't result in duplicates
+dest:
+  index: "logs-ti_recordedfuture_latest.threat-1.0.0"


Here's what I am thinking...

Suggested change

# The version suffix on destination index must be explicitly set.

# NOTE: During version change, ensure through some mechanism old destination index automatically purges so queries on wild card such as logs-*_latest-* doesn't result in duplicates

dest:

index: "logs-ti_recordedfuture_latest.threat-1.0.0"

# The version suffix on the dest.index should be incremented if a breaking change

# is made to the index mapping. You must also bump the fleet_transform_version

# for any change to this transform configuration to take effect. The old destination

# index is not automatically removed. We are dependent on {issue} to give

# us that ability in order to prevent having duplicate IoC data and prevent query

# time field type conflicts.

dest:

index: "logs-ti_recordedfuture_latest.threat-1"

Thank you! Updated as suggested. Linked the underlying issue.

andrewkroh

👍

kcreddy added 2 commits March 6, 2023 14:59

add changelog; remove fingerprint

961eff2

Support IoC expiration

475d569

kcreddy added enhancement New feature or request Integration:ti_recordedfuture Recorded Future labels Mar 7, 2023

kcreddy self-assigned this Mar 7, 2023

Add transform

05347bb

Change transform dir

4792484

kcreddy mentioned this pull request Mar 14, 2023

Need kibana_system privileges for Threat Intel (TI) package transforms to support IOC expiration elastic/elasticsearch#94505

Closed

kcreddy marked this pull request as ready for review March 20, 2023 13:13

kcreddy requested a review from a team as a code owner March 20, 2023 13:13

kcreddy added the Team:Security-External Integrations label Mar 20, 2023

andrewkroh reviewed Mar 20, 2023

View reviewed changes

kcreddy added 8 commits March 22, 2023 11:28

dont start transform

93a5146

updated max_age to 24h

46c49ca

Add ILM policy to source indices

6f0f785

Removed namespace and added README

3513663

Add dot to destination indices

16c71c5

Sample Event

c9e1b6b

update docs

628b002

update system tests

5123d03

andrewkroh reviewed Apr 3, 2023

View reviewed changes

kcreddy added 5 commits April 6, 2023 19:06

Set date_detection to false

90b537f

Merge remote-tracking branch 'upstream/main' into ioc_expire_rec_future

458b219

update manifest

9064575

Remove dot prefix

191716a

Change min version

ea7dbfd

Revert akamai

f0c5d3b

kcreddy added 2 commits April 21, 2023 13:54

test system

17b798a

update readme

c332355

kcreddy mentioned this pull request Apr 21, 2023

Add backing indices to kibana_system permissions for TI packages supporting IOC expiration elastic/elasticsearch#95449

Closed

return system test to check flakiness

5a3fcfd

kcreddy mentioned this pull request Apr 24, 2023

Add IOC expiration support to all TI providers #5369

Closed

10 tasks

Change source index pattern

dc0caa1

kcreddy mentioned this pull request Apr 27, 2023

[Transform] Package upgrade does not update the transform elastic/kibana#155982

Closed

kcreddy added 2 commits May 3, 2023 23:23

Merge remote-tracking branch 'upstream/main' into ioc_expire_rec_future

1f88d57

add version to dest index

ba3a101

andrewkroh reviewed May 9, 2023

View reviewed changes

packages/ti_recordedfuture/elasticsearch/transform/latest_ioc/transform.yml Show resolved Hide resolved

kcreddy added 2 commits May 9, 2023 13:40

explicit add version

db91b07

update wording

d8ed345

andrewkroh reviewed May 9, 2023

View reviewed changes

kcreddy added 2 commits May 11, 2023 11:38

Added comments on dest index

6c353bd

Merge remote-tracking branch 'upstream/main' into ioc_expire_rec_future

dc9325e

andrewkroh reviewed May 11, 2023

View reviewed changes

Address PR comments

d6e0591

andrewkroh approved these changes May 13, 2023

View reviewed changes

kcreddy merged commit 58eda22 into elastic:main May 15, 2023

kcreddy deleted the ioc_expire_rec_future branch February 7, 2025 09:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TI_RecordedFuture] Support IoC expiration #5460

[TI_RecordedFuture] Support IoC expiration #5460

kcreddy commented Mar 7, 2023 •

edited

Loading

elasticmachine commented Mar 7, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Mar 7, 2023 •

edited

Loading

kcreddy commented Mar 8, 2023

elasticmachine commented Mar 20, 2023

andrewkroh Mar 20, 2023

andrewkroh Mar 20, 2023

kcreddy Mar 22, 2023

kcreddy Mar 31, 2023

andrewkroh Mar 20, 2023

kcreddy Mar 22, 2023

kcreddy Mar 31, 2023

andrewkroh Mar 20, 2023

kcreddy Mar 31, 2023

andrewkroh Mar 20, 2023

kcreddy Mar 22, 2023

kcreddy Mar 31, 2023

andrewkroh Apr 3, 2023

kcreddy Apr 6, 2023 •

edited

Loading

kcreddy commented Apr 21, 2023

kcreddy commented Apr 24, 2023

kcreddy commented Apr 25, 2023

andrewkroh commented Apr 26, 2023 •

edited

Loading

kcreddy commented Apr 27, 2023

kcreddy commented May 3, 2023 •

edited

Loading

andrewkroh May 9, 2023 •

edited

Loading

kcreddy May 11, 2023

andrewkroh May 11, 2023

kcreddy May 12, 2023

andrewkroh May 11, 2023

andrewkroh May 11, 2023

kcreddy May 12, 2023

andrewkroh left a comment

		@@ -16,13 +16,13 @@ An example event for `threat` looks as following:

-# The version suffix on destination index must be explicitly set.
-# NOTE: During version change, ensure through some mechanism old destination index automatically purges so queries on wild card such as logs-*_latest-* doesn't result in duplicates
-dest:
-  index: "logs-ti_recordedfuture_latest.threat-1.0.0"
+# The version suffix on the dest.index should be incremented if a breaking change
+# is made to the index mapping. You must also bump the fleet_transform_version
+# for any change to this transform configuration to take effect. The old destination
+# index is not automatically removed. We are dependent on {issue} to give
+# us that ability in order to prevent having duplicate IoC data and prevent query
+# time field type conflicts.
+dest:
+  index: "logs-ti_recordedfuture_latest.threat-1"

[TI_RecordedFuture] Support IoC expiration #5460

[TI_RecordedFuture] Support IoC expiration #5460

Conversation

kcreddy commented Mar 7, 2023 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Mar 7, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Mar 7, 2023 • edited Loading

🌐 Coverage report

kcreddy commented Mar 8, 2023

elasticmachine commented Mar 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcreddy Apr 6, 2023 • edited Loading

Choose a reason for hiding this comment

kcreddy commented Apr 21, 2023

kcreddy commented Apr 24, 2023

kcreddy commented Apr 25, 2023

andrewkroh commented Apr 26, 2023 • edited Loading

kcreddy commented Apr 27, 2023

kcreddy commented May 3, 2023 • edited Loading

andrewkroh May 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

kcreddy commented Mar 7, 2023 •

edited

Loading

elasticmachine commented Mar 7, 2023 •

edited

Loading

elasticmachine commented Mar 7, 2023 •

edited

Loading

kcreddy Apr 6, 2023 •

edited

Loading

andrewkroh commented Apr 26, 2023 •

edited

Loading

kcreddy commented May 3, 2023 •

edited

Loading

andrewkroh May 9, 2023 •

edited

Loading