[TI_RecordedFuture] Support IoC expiration

packages/ti_recordedfuture/_dev/build/docs/README.md

@@ -10,6 +10,13 @@ from multiple entities, it's necessary to define one integration for each.
 Alternatively, it's also possible to use the integration to fetch custom Fusion files
 by supplying the URL to the CSV file as the _Custom_ _URL_ configuration option.
 
+### Expiration of Indicators of Compromise (IOCs)
+The ingested IOCs expire after certain duration. An [Elastic Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) is created to faciliate only active IOCs be available to the end users. This transform creates a destination index named `logs-ti_recordedfuture_latest.threat` which only contains active and unexpired IOCs. When setting up indicator match rules, use this latest destination index to avoid false positives from expired IOCs. Please read [ILM Policy](#ilm-policy) below which is added to avoid unbounded growth on source `.ds-logs-ti_recordedfuture.threat-*` indices.
+
+### ILM Policy
+To facilitate IOC expiration, source datastream-backed indices `.ds-logs-ti_recordedfuture.threat-*` are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after `5 days` from ingested date. 
+
+
 **NOTE:** For large risklist downloads, adjust the timeout setting so that the Agent has enough time to download and process the risklist.
 
 {{event "threat"}}

packages/ti_recordedfuture/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.8.0"
+  changes:
+    - description: Support for IoC Expiration
+      type: enhancement
+      link: https://github.com/elastic/integrations/issues/5459
 - version: "1.7.0"
   changes:
     - description: Scrape provider details from evidence field.

packages/ti_recordedfuture/data_stream/threat/_dev/test/pipeline/test-domain-default.log-expected.json

@@ -80,6 +80,7 @@
                         "Timestamp": "2021-12-29T07:12:02.455Z"
                     }
                 ],
+                "name": "xohrikvjhiu.eu",
                 "risk_string": "5/45"
             },
             "tags": [
@@ -193,6 +194,7 @@
                         "Timestamp": "2021-12-29T07:21:52.303Z"
                     }
                 ],
+                "name": "wgwuhauaqcrx.com",
                 "risk_string": "6/45"
             },
             "tags": [
@@ -305,6 +307,7 @@
                         "Timestamp": "2021-12-29T07:16:05.007Z"
                     }
                 ],
+                "name": "wbmpvebw.com",
                 "risk_string": "6/45"
             },
             "tags": [
@@ -405,6 +408,7 @@
                         "Timestamp": "2021-12-29T06:40:44.358Z"
                     }
                 ],
+                "name": "ckgryagcibbcf.com",
                 "risk_string": "5/45"
             },
             "tags": [
@@ -504,6 +508,7 @@
                         "Timestamp": "2021-12-29T06:46:28.155Z"
                     }
                 ],
+                "name": "jpuityvakjgg.com",
                 "risk_string": "5/45"
             },
             "tags": [
@@ -603,6 +608,7 @@
                         "Timestamp": "2021-12-29T06:40:30.778Z"
                     }
                 ],
+                "name": "jexgpprgph.com",
                 "risk_string": "5/45"
             },
             "tags": [
@@ -702,6 +708,7 @@
                         "Timestamp": "2021-12-29T06:34:06.062Z"
                     }
                 ],
+                "name": "cascotqhij.com",
                 "risk_string": "5/45"
             },
             "tags": [
@@ -801,6 +808,7 @@
                         "Timestamp": "2021-12-29T06:45:21.381Z"
                     }
                 ],
+                "name": "npcvnorvyhelagx.com",
                 "risk_string": "5/45"
             },
             "tags": [
@@ -900,6 +908,7 @@
                         "Timestamp": "2021-12-29T06:35:26.677Z"
                     }
                 ],
+                "name": "uxlyihgvfnqcrfcf.com",
                 "risk_string": "5/45"
             },
             "tags": [

packages/ti_recordedfuture/data_stream/threat/_dev/test/pipeline/test-hash-default.log-expected.json

@@ -171,6 +171,7 @@
                         "Timestamp": "2020-07-11T09:55:23.000Z"
                     }
                 ],
+                "name": "38e992eb852ab0c4ac03955fb0dc9bb38e64010fdf9c05331d2b02b6e05689c2",
                 "risk_string": "6/14"
             },
             "tags": [
@@ -370,6 +371,7 @@
                         "Timestamp": "2021-03-08T00:00:00.000Z"
                     }
                 ],
+                "name": "c15abaf51e78ca56c0376522d699c978217bf041a3bd3c71d09193efa5717c71",
                 "risk_string": "7/14"
             },
             "tags": [
@@ -573,6 +575,7 @@
                         "Timestamp": "2021-12-18T00:20:04.000Z"
                     }
                 ],
+                "name": "b66db3a06c2955a9cb71a8718970c592",
                 "risk_string": "5/14"
             },
             "tags": [
@@ -951,6 +954,7 @@
                         "Timestamp": "2020-12-17T22:59:03.000Z"
                     }
                 ],
+                "name": "027cc450ef5f8c5f653329641ec1fed91f694e0d229928963b30f6b0d7d3a745",
                 "risk_string": "8/14"
             },
             "tags": [
@@ -1090,6 +1094,7 @@
                         "Timestamp": "2019-07-01T00:00:00.000Z"
                     }
                 ],
+                "name": "ad2ad0249fafe85877bc79a01e1afd1a44d983c064ad8cb5bc694d29d166217b",
                 "risk_string": "5/14"
             },
             "tags": [
@@ -1221,6 +1226,7 @@
                         "Timestamp": "2021-04-04T07:46:20.000Z"
                     }
                 ],
+                "name": "01ba1fb41632594997a41d0c3a911ae5b3034d566ebb991ef76ad76e6f9e283a",
                 "risk_string": "5/14"
             },
             "tags": [
@@ -1422,6 +1428,7 @@
                         "Timestamp": "2021-02-10T09:10:10.000Z"
                     }
                 ],
+                "name": "fecddb7f3fa478be4687ca542c0ecf232ec35a0c2418c8bfe4875686ec373c1e",
                 "risk_string": "6/14"
             },
             "tags": [
@@ -1546,6 +1553,7 @@
                         "Timestamp": "2020-10-13T10:46:31.000Z"
                     }
                 ],
+                "name": "a1d9cd6f189beff28a0a49b10f8fe4510128471f004b3e4283ddc7f78594906b",
                 "risk_string": "3/14"
             },
             "tags": [
@@ -1674,6 +1682,7 @@
                         "Timestamp": "2021-03-08T13:00:15.000Z"
                     }
                 ],
+                "name": "85aba198a0ba204e8549ea0c8980447249d30dece0d430e3f517315ad10f32ce",
                 "risk_string": "5/14"
             },
             "tags": [