Add IOC expiration support to all TI providers

[narph]

Meta issue https://github.com/elastic/security-team/issues/5868 for detailed info.

Once support for IOC expiration has been implemented in https://github.com/elastic/security-team/issues/6114 then this should be applied (if necessary) in all the ti_* packages. The following list of packages are in priority order:

• ti_abusech (Free)
• ti_otx (Free)
• ti_misp (Free)
• ti_cybersixgill (Commercial)
• ti_threatq (Commercial)
• ti_cif3 (Free)
• ti_rapid7_threat_command (Commercial)
• ti_anomali (Commercial)
• ti_recordedfuture (Commercial)
• ti_opencti (Free)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[narph]

@P1llus , @andrewkroh can we get a confirmation here on the list?
@P1llus mentioned we are targeting anomali and recordedfuture packages.
Is the list correct? Any direction/timeline on the other ti_packages?

[P1llus]

Not all of these packages are applicable, and the "solution" is different for each TI package, for example Recorded Future does not have any expiry dates, and the issue there is much different than with Anomali.

While maybe one or two more of the TI packages might be applicable, the focus should really be on the ones that customers have been asking for (Anomali and RF), and we should take a look at some of the others individually once the first 2 are done.

[LaZyDK]

And MISP.

[P1llus]

And MISP.

The issue with MISP is going to be the fact that every source would be different, there is no specific format we can follow

[kcreddy]

PR for RecordedFuture #5460
PR for Anomali #5582

[kcreddy]

ti_abusech: I couldn't find documentation about indicator expiration in AbuseCH docs. Since they didn't have any communication channel, I filled up the form requesting information on the same: https://www.spamhaus.com/contact-us-abuse-ch/

ti_otx: The API thats currently being used is no longer supported as per API documentation: https://otx.alienvault.com/assets/static/external_api.html#Home. Created an issue: AlienVault-OTX/OTX-Python-SDK#70 to get more info. Also found that their python SDK is using /pulses/subscribed API to get indicators that users are subscribed to: https://github.com/KarmaIncarnate/OTX_Siphon/blob/master/OTX_Siphon.py#L138`

[chrisberkhout]

@kcreddy I've got most a solution for OpenCTI done. I just have some field type issues and dashboard filtering to adjust and then I'll open a PR.