feat(fetch/nvd): use NVD API feed #348

MaineK00n · 2023-11-17T08:01:55Z

What did you implement:

Fixes #272

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

$ go-cve-dictionary fetch nvd
INFO[11-17|15:56:23] Inserting NVD into DB (sqlite3). 
INFO[11-17|15:56:23] Deleting NVD tables... 
INFO[11-17|15:56:23] Fetching CVE information from NVD. 
INFO[11-17|15:56:23] Fetching... https://github.com/vulsio/vuls-data-raw-nvd-api-cve/archive/refs/heads/main.tar.gz 
INFO[11-17|15:56:34] Fetching CVE information from NVD(2002). 
INFO[11-17|15:56:35] Inserting fetched CVEs(2002)... 
2392 / 2392 [---------------------------------------------------------------------------------------] 100.00% 4053 p/s
INFO[11-17|15:56:35] Refreshed 2392 CVEs. 
INFO[11-17|15:56:35] Fetching CVE information from NVD(2003). 
INFO[11-17|15:56:36] Inserting fetched CVEs(2003)... 
1553 / 1553 [---------------------------------------------------------------------------------------] 100.00% 2912 p/s
INFO[11-17|15:56:36] Refreshed 1553 CVEs. 
INFO[11-17|15:56:36] Fetching CVE information from NVD(2004). 
INFO[11-17|15:56:37] Inserting fetched CVEs(2004)... 
2707 / 2707 [---------------------------------------------------------------------------------------] 100.00% 1402 p/s
INFO[11-17|15:56:39] Refreshed 2707 CVEs. 
INFO[11-17|15:56:39] Fetching CVE information from NVD(2005). 
INFO[11-17|15:56:39] Inserting fetched CVEs(2005)... 
4766 / 4766 [---------------------------------------------------------------------------------------] 100.00% 1759 p/s
INFO[11-17|15:56:42] Refreshed 4766 CVEs. 
INFO[11-17|15:56:42] Fetching CVE information from NVD(2006). 
INFO[11-17|15:56:43] Inserting fetched CVEs(2006)... 
7142 / 7142 [---------------------------------------------------------------------------------------] 100.00% 1901 p/s
INFO[11-17|15:56:47] Refreshed 7142 CVEs. 
INFO[11-17|15:56:47] Fetching CVE information from NVD(2007). 
INFO[11-17|15:56:47] Inserting fetched CVEs(2007)... 
6580 / 6580 [---------------------------------------------------------------------------------------] 100.00% 1889 p/s
INFO[11-17|15:56:51] Refreshed 6580 CVEs. 
INFO[11-17|15:56:51] Fetching CVE information from NVD(2008). 
INFO[11-17|15:56:51] Inserting fetched CVEs(2008)... 
7176 / 7176 [---------------------------------------------------------------------------------------] 100.00% 1569 p/s
INFO[11-17|15:56:56] Refreshed 7176 CVEs. 
INFO[11-17|15:56:56] Fetching CVE information from NVD(2009). 
INFO[11-17|15:56:57] Inserting fetched CVEs(2009)... 
5039 / 5039 [----------------------------------------------------------------------------------------] 100.00% 505 p/s
INFO[11-17|15:57:07] Refreshed 5039 CVEs. 
INFO[11-17|15:57:07] Fetching CVE information from NVD(2010). 
INFO[11-17|15:57:08] Inserting fetched CVEs(2010)... 
5216 / 5216 [----------------------------------------------------------------------------------------] 100.00% 828 p/s
INFO[11-17|15:57:15] Refreshed 5216 CVEs. 
INFO[11-17|15:57:15] Fetching CVE information from NVD(2011). 
INFO[11-17|15:57:15] Inserting fetched CVEs(2011)... 
4859 / 4859 [----------------------------------------------------------------------------------------] 100.00% 617 p/s
INFO[11-17|15:57:24] Refreshed 4859 CVEs. 
INFO[11-17|15:57:24] Fetching CVE information from NVD(2012). 
INFO[11-17|15:57:25] Inserting fetched CVEs(2012)... 
5890 / 5890 [----------------------------------------------------------------------------------------] 100.00% 612 p/s
INFO[11-17|15:57:35] Refreshed 5890 CVEs. 
INFO[11-17|15:57:35] Fetching CVE information from NVD(2013). 
INFO[11-17|15:57:36] Inserting fetched CVEs(2013)... 
6779 / 6779 [----------------------------------------------------------------------------------------] 100.00% 729 p/s
INFO[11-17|15:57:45] Refreshed 6779 CVEs. 
INFO[11-17|15:57:45] Fetching CVE information from NVD(2014). 
INFO[11-17|15:57:46] Inserting fetched CVEs(2014)... 
8976 / 8976 [---------------------------------------------------------------------------------------] 100.00% 1482 p/s
INFO[11-17|15:57:52] Refreshed 8976 CVEs. 
INFO[11-17|15:57:52] Fetching CVE information from NVD(2015). 
INFO[11-17|15:57:53] Inserting fetched CVEs(2015)... 
8738 / 8738 [---------------------------------------------------------------------------------------] 100.00% 1679 p/s
INFO[11-17|15:57:58] Refreshed 8738 CVEs. 
INFO[11-17|15:57:58] Fetching CVE information from NVD(2016). 
INFO[11-17|15:57:59] Inserting fetched CVEs(2016)... 
10544 / 10544 [-------------------------------------------------------------------------------------] 100.00% 1629 p/s
INFO[11-17|15:58:06] Refreshed 10544 CVEs. 
INFO[11-17|15:58:06] Fetching CVE information from NVD(2017). 
INFO[11-17|15:58:07] Inserting fetched CVEs(2017)... 
16977 / 16977 [-------------------------------------------------------------------------------------] 100.00% 1587 p/s
INFO[11-17|15:58:17] Refreshed 16977 CVEs. 
INFO[11-17|15:58:17] Fetching CVE information from NVD(2018). 
INFO[11-17|15:58:18] Inserting fetched CVEs(2018)... 
17341 / 17341 [-------------------------------------------------------------------------------------] 100.00% 2046 p/s
INFO[11-17|15:58:27] Refreshed 17341 CVEs. 
INFO[11-17|15:58:27] Fetching CVE information from NVD(2019). 
INFO[11-17|15:58:28] Inserting fetched CVEs(2019)... 
16968 / 16968 [-------------------------------------------------------------------------------------] 100.00% 1868 p/s
INFO[11-17|15:58:37] Refreshed 16968 CVEs. 
INFO[11-17|15:58:37] Fetching CVE information from NVD(2020). 
INFO[11-17|15:58:39] Inserting fetched CVEs(2020)... 
20406 / 20406 [-------------------------------------------------------------------------------------] 100.00% 1562 p/s
INFO[11-17|15:58:52] Refreshed 20406 CVEs. 
INFO[11-17|15:58:52] Fetching CVE information from NVD(2021). 
INFO[11-17|15:58:54] Inserting fetched CVEs(2021)... 
21973 / 21973 [-------------------------------------------------------------------------------------] 100.00% 1536 p/s
INFO[11-17|15:59:08] Refreshed 21973 CVEs. 
INFO[11-17|15:59:08] Fetching CVE information from NVD(2022). 
INFO[11-17|15:59:10] Inserting fetched CVEs(2022)... 
24397 / 24397 [-------------------------------------------------------------------------------------] 100.00% 1732 p/s
INFO[11-17|15:59:24] Refreshed 24397 CVEs. 
INFO[11-17|15:59:24] Fetching CVE information from NVD(2023). 
INFO[11-17|15:59:25] Inserting fetched CVEs(2023)... 
20194 / 20194 [-------------------------------------------------------------------------------------] 100.00% 1866 p/s
INFO[11-17|15:59:36] Refreshed 20194 CVEs. 
INFO[11-17|15:59:39] Finished fetching NVD.

Checklist:

You don't have to satisfy all of the following.

Write tests
Write documentation
Check that there aren't other open pull requests for the same issue/feature
Format your source code by make fmt
Pass the test by make test
Provide verification config / commands
Enable "Allow edits from maintainers" for this PR
Update the messages below

Is this ready for review?: YES

Reference

jbmaillet · 2023-11-20T14:59:40Z

Thanks a lot. Tested OK with some basic tests. 👍

One question though: shouldn't there be some API key parameter somewhere on the CLI?

Beginning six months after the release of the API keys, users transmitting requests without a key will see
a reduction in the number of requests they can make in a rolling 60 second window. Users transmitting
requests that include their API key will see no change in service and may continue to make requests
at the current rate.

From API-Key-Announcement
Keys can be obtained here: Request an API Key

I have such a key ready to be used (a bit more difficult to actually test).

I'll do my best to do further tests this week, but in the course of these tests and deployment I'll need to switch from v0.5.6 from 2020 I currently use to up to date v0.9.0, and probably re-initiate the MySQL DB is use for custom queries.

jbmaillet · 2023-11-20T17:02:49Z

(Note that of course, the API key support could as well be added later on, as another enhancement. As for me, I already have a lot to test here... Again, thanks for your work last weekend. 💯 )

MaineK00n · 2023-11-21T07:44:45Z

Since we are not fetching directly from the NVD API, an API KEY is not required. This time, the results of the NVD API are saved in this repository(https://github.com/vulsio/vuls-data-raw-nvd-api-cve), and go-cve-dictionary uses that repository.

jbmaillet · 2023-11-24T15:58:36Z

I tested this successfully in a MySQL setup and my custom client application. 👍

I don't know Go, so this is a functional test only, not a code review, sorry for that.

The next step for me will be to update my production server to v0.9.0, to be ready for the next version with this new development. Thanks again!

fredericg78 · 2023-11-28T17:05:03Z

Hi, is it related to https://nvd.nist.gov/vuln/data-feeds ?
"On December 15th, 2023, the NVD plans to retire all legacy data feeds while guiding any remaining data feed users to updated application-programming interfaces (APIs)."
https://nvd.nist.gov/General/News/change-timeline: The NVD plans to retire the remaining legacy data feeds as well as all 1.0 APIs on December 15th.
Which is the go-cve-dictionary version which starts to be compliant with this breaking change for fetching datas ?

Best regards

MaineK00n · 2023-11-28T17:36:47Z

@fredericg78

Although the version has not been clearly determined, go-cve-dictionary built after this PR was merged will use the results of the NVD API.

witchcraze · 2023-12-07T10:29:04Z

Let me report v0.10.0 roughly.
I use server-mode with php. In my code, two update was required, but almost no problem.

Under "Cvss2", "Cvss3" is empty
["Cvss2"]["BaseScore"] → ["Cvss2"][0]["BaseScore"] (layer was changed ?)

MaineK00n · 2023-12-07T11:46:15Z

@witchcraze
This was changed because there may be multiple CVSSv2 and CVSSv3.

witchcraze · 2023-12-07T12:25:54Z

Ah, OK.
In this case, two CVSS 3.x was registered. (But we can not judje which is NVD's one)
https://nvd.nist.gov/vuln/detail/CVE-2023-20254

$ curl -s http://127.0.0.1:1323/cves/CVE-2023-20254 | jq ".Nvds[].Cvss3"
[
  {
    "VectorString": "CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
    "AttackVector": "NETWORK",
    "AttackComplexity": "LOW",
    "PrivilegesRequired": "LOW",
    "UserInteraction": "NONE",
    "Scope": "UNCHANGED",
    "ConfidentialityImpact": "HIGH",
    "IntegrityImpact": "HIGH",
    "AvailabilityImpact": "HIGH",
    "BaseScore": 8.8,
    "BaseSeverity": "HIGH",
    "ExploitabilityScore": 2.8,
    "ImpactScore": 5.9
  },
  {
    "VectorString": "CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:H",
    "AttackVector": "NETWORK",
    "AttackComplexity": "LOW",
    "PrivilegesRequired": "HIGH",
    "UserInteraction": "NONE",
    "Scope": "UNCHANGED",
    "ConfidentialityImpact": "HIGH",
    "IntegrityImpact": "HIGH",
    "AvailabilityImpact": "HIGH",
    "BaseScore": 7.2,
    "BaseSeverity": "HIGH",
    "ExploitabilityScore": 1.2,
    "ImpactScore": 5.9
  }
]

MaineK00n · 2023-12-07T14:49:13Z

As for the original data, depending on the source, you can tell whether it is evaluated by NVD or another vendor......
https://github.com/vulsio/vuls-data-raw-nvd-api-cve/blob/main/2023/CVE-2023-20254.json

MaineK00n self-assigned this Nov 17, 2023

feat(fetch/nvd): use NVD API feed

0289474

MaineK00n force-pushed the MaineK00n/nvd-api branch from 7839747 to 0289474 Compare November 17, 2023 08:16

MaineK00n marked this pull request as ready for review November 18, 2023 05:51

kotakanbe merged commit 4ffa5ad into master Dec 6, 2023

kotakanbe deleted the MaineK00n/nvd-api branch December 6, 2023 05:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(fetch/nvd): use NVD API feed #348

feat(fetch/nvd): use NVD API feed #348

MaineK00n commented Nov 17, 2023

jbmaillet commented Nov 20, 2023

jbmaillet commented Nov 20, 2023 •

edited

Loading

MaineK00n commented Nov 21, 2023

jbmaillet commented Nov 24, 2023 •

edited

Loading

fredericg78 commented Nov 28, 2023

MaineK00n commented Nov 28, 2023

witchcraze commented Dec 7, 2023

MaineK00n commented Dec 7, 2023

witchcraze commented Dec 7, 2023

MaineK00n commented Dec 7, 2023

feat(fetch/nvd): use NVD API feed #348

feat(fetch/nvd): use NVD API feed #348

Conversation

MaineK00n commented Nov 17, 2023

What did you implement:

Type of change

How Has This Been Tested?

Checklist:

Reference

jbmaillet commented Nov 20, 2023

jbmaillet commented Nov 20, 2023 • edited Loading

MaineK00n commented Nov 21, 2023

jbmaillet commented Nov 24, 2023 • edited Loading

fredericg78 commented Nov 28, 2023

MaineK00n commented Nov 28, 2023

witchcraze commented Dec 7, 2023

MaineK00n commented Dec 7, 2023

witchcraze commented Dec 7, 2023

MaineK00n commented Dec 7, 2023

jbmaillet commented Nov 20, 2023 •

edited

Loading

jbmaillet commented Nov 24, 2023 •

edited

Loading