Convert First EPPS Internal Enrichment Connector to External Import due to Performance Issues

[animedbz16]

Use case

As currently implemented, the First EPSS connector is designed as an internal enrichment connector, which is not ideal for anyone who wants to ensure that all vulnerabilities that are contained with OpenCTI are updated on a daily basis.

Within our OpenCTI instance it contains over 221k vulnerabilities and the only way to update every one of these is to create a Task to bulk enrich all vulnerabilities (e.g. Entity type = Vulnerability), which takes over 24 hours to complete as each entity's EPSS score is fetched individually, which can run into API rate limiting and other issues that cause this process to be extremely slow.

Current Workaround

N/A

Proposed Solution

The solution to this is to convert this connector from an Internal Enrichment Connector to be an External Import connector so that instead of being designed to enrich a single vulnerability at a time, it would instead just download a single CSV file directly that contains the EPSS scores for ALL CVE's.

See the following from EPSS API documentation (https://www.first.org/epss/api)

ALL CVEs Show EPSS scores for all CVEs for a particular date (yyyy-mm-dd). For this request, simply request the full csv directly as, https://epss.cyentia.com/epss_scores-YYYY-MM-DD.csv.gz

Which as of today (2024-12-18), would make a single HTTP request and download a 1.56 MB gzipped CSV file that extracts out to be 7.86 MB CSV file that contains the EPSS scores for 270,653 CVEs (with 2 rows for headers):

Using this method is much faster than querying for each Vulnerability directly and the steps for this connector would be:

• Configure Connector to run daily
• Connector runs and starts with downloading the gzipped CSV file and uncompress it
• Connector queries OpenCTI for a list of all Vulnerabilities to obtain the only the CVE Name and internal ID for the CVE in OpenCTI (and possibly the current EPSS score information for the CVE, but likely not necessary)
• Connector loads CSV file and loops through the contents checking to see if the CVE in the EPSS CSV file is an CVE that is in OpenCTI and if the scores have changed, which essentially they always will so this may not be necessary to check this, to put together a list of CVEs and the new EPSS score information and queue them up to be updated.

When doing this method, it is possible to update every Vulnerability in a fraction of the time and have it configured to just update this information every day.