[TI_CIF3] Add support for IOC expiration

[kcreddy]

Individual tracking issue for ti_cif3 package for adding IOC expiration support.
Meta Issue - #5369

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[kcreddy]

We currently have 1 datastream feed which use the CIFv3 API /feed documented here with parameters such as itype, confidence, tags, hours, reporttime to fetch only relevant filtered indicators since last reported time.

The repository which supports the v3 API is now archived. There is also a CIFv5 that is available at https://github.com/csirtgadgets/cif-v5
None of the v3 API or v5 API have documentation which provides information on indicator expiration. I created an issue on their repository to confirm it.

Issue - csirtgadgets/cif-v5#43

[jamiehynds]

@mdavis332 I know it's been awhile since you built the integration, but do you have any familiarity with the CIFv5 API and how it handles IoC expiration?

[mdavis332]

@mdavis332 I know it's been awhile since you built the integration, but do you have any familiarity with the CIFv5 API and how it handles IoC expiration?

Hi @jamiehynds, I see that Wes already replied to @kcreddy's issue at csirtgadgets/cif-v5#43. Although his answer his for CIFv5, it's mostly the same for CIFv3. The /feed endpoint handles looking back a certain number of days automatically based on itype if a reporttime field isn't specified during the indicator feed pull. Therefore, each pull will only ever yield indicators from that reporttime or forward and there's no explicit expiry field in the schema (the clients get to handle how far back they want to pull indicators for).

Hope that helps!

[kcreddy]

Thanks @mdavis332 for the info.
Yes we are using reporttime in the url parameter and fetching the latest indicators since previous reporttime.
Since there is no explicit expiry field here, we will go with Wes suggestion on the defaults provided: csirtgadgets/cif-v5#43 (comment)