Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(fetch): fetch by year #236

Merged
merged 2 commits into from
Jan 6, 2022
Merged

feat(fetch): fetch by year #236

merged 2 commits into from
Jan 6, 2022

Conversation

MaineK00n
Copy link
Collaborator

@MaineK00n MaineK00n commented Dec 23, 2021 •
edited
Loading

What did you implement:

Due to a change at #214, go-cve-dictionary could no longer select a year to fetch.
However, if all the years were to be fetched, the DB size would become too large, and there would be a demand for only certain years, etc., so I added the ability to select and fetch years.

Fixes #231

Type of change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Difference in fetch behavior

If you selected years in the previous fetch command, they were added to the DB, but the current fetch command will recreate a new DB with only the selected years.

v0.7.1 and earlier

$ go-cve-dictionary fetch nvd --years 2020
$ sqlite3 cve.sqlite3
sqlite> SELECT * FROM nvds LIMIT 3;
id|feed_meta_id|cve_id|published_date|last_modified_date
1|1|CVE-2020-0003|2020-01-08 19:15:00+00:00|2020-01-29 21:15:00+00:00
2|1|CVE-2020-0002|2020-01-08 19:15:00+00:00|2021-07-21 11:39:00+00:00
3|1|CVE-2020-0001|2020-01-08 19:15:00+00:00|2021-07-21 11:39:00+00:00

$ go-cve-dictionary fetch nvd --years 2021
$ sqlite3 cve.sqlite3
sqlite> SELECT * FROM nvds WHERE cve_id LIKE "CVE-2020-%" LIMIT 3;
id|feed_meta_id|cve_id|published_date|last_modified_date
1|1|CVE-2020-0003|2020-01-08 19:15:00+00:00|2020-01-29 21:15:00+00:00
2|1|CVE-2020-0002|2020-01-08 19:15:00+00:00|2021-07-21 11:39:00+00:00
3|1|CVE-2020-0001|2020-01-08 19:15:00+00:00|2021-07-21 11:39:00+00:00
sqlite> SELECT * FROM nvds WHERE cve_id LIKE "CVE-2021-%" LIMIT 3;
id|feed_meta_id|cve_id|published_date|last_modified_date
17855|2|CVE-2021-45470|2021-12-23 21:15:00+00:00|2021-12-23 21:15:00+00:00
17856|2|CVE-2021-3622|2021-12-23 21:15:00+00:00|2021-12-23 21:15:00+00:00
17857|2|CVE-2021-3621|2021-12-23 21:15:00+00:00|2021-12-23 21:15:00+00:00

MaineK00n/fetch-by-year

$ go-cve-dictionary fetch nvd 2020
$ sqlite3 cve.sqlite3
sqlite> SELECT * FROM nvds LIMIT 3;
id|cve_id|published_date|last_modified_date
1|CVE-2020-17473|2020-08-14 20:15:00+00:00|2020-08-21 15:00:00+00:00
2|CVE-2020-25990|2020-10-01 14:15:00+00:00|2020-10-05 15:58:00+00:00
3|CVE-2020-35592|2021-02-18 20:15:00+00:00|2021-02-24 15:42:00+00:00

$ go-cve-dictionary fetch nvd 2021
$ sqlite3 cve.sqlite3
sqlite> SELECT * FROM nvds LIMIT 3;
id|cve_id|published_date|last_modified_date
1|CVE-2021-38947|2021-12-13 18:15:00+00:00|2021-12-15 16:56:00+00:00
2|CVE-2021-34612|2021-07-08 20:15:00+00:00|2021-07-12 17:42:00+00:00
3|CVE-2021-23840|2021-02-16 17:15:00+00:00|2021-12-10 18:12:00+00:00
sqlite> SELECT * FROM nvds WHERE cve_id LIKE "CVE-2020-%";

$ go-cve-dictionary fetch nvd 2020 2021 // If you want to fetch both 2020 and 2021
$ sqlite3 cve.sqlite3
sqlite> SELECT * FROM nvds WHERE cve_id LIKE "CVE-2020-%" LIMIT 3;
id|cve_id|published_date|last_modified_date
1|CVE-2020-0046|2020-03-10 21:15:00+00:00|2020-03-11 18:13:00+00:00
2|CVE-2020-4375|2020-07-28 12:15:00+00:00|2021-07-21 11:39:00+00:00
3|CVE-2020-5509|2020-01-14 19:15:00+00:00|2020-01-21 19:14:00+00:00
sqlite> SELECT * FROM nvds WHERE cve_id LIKE "CVE-2021-%" LIMIT 3;
id|cve_id|published_date|last_modified_date
17857|CVE-2021-38006|2021-12-23 01:15:00+00:00|2021-12-23 15:25:00+00:00
17858|CVE-2021-39931|2021-12-13 16:15:00+00:00|2021-12-16 03:09:00+00:00
17859|CVE-2021-39863|2021-09-29 16:15:00+00:00|2021-10-06 16:45:00+00:00

Behavior of the fetch command when no year is entered

Fetch all the years as before.

$ go-cve-dictionary fetch nvd
INFO[12-24|08:21:40] Inserting NVD into DB (sqlite3). 
INFO[12-24|08:21:40] Deleting NVD tables... 
INFO[12-24|08:21:40] Fetching CVE information from NVD(recent, modified). 
INFO[12-24|08:21:40] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.json.gz 
INFO[12-24|08:21:41] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.json.gz 
INFO[12-24|08:21:42] Fetching CVE information from NVD(2002). 
INFO[12-24|08:21:42] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.json.gz 
INFO[12-24|08:21:45] Inserting fetched CVEs(2002)... 
2356 / 2356 [---------------------------------------------------------------------------------------------------------] 100.00% 4018 p/s
INFO[12-24|08:21:46] Refreshed 2356 CVEs. 
INFO[12-24|08:21:46] Fetching CVE information from NVD(2003). 
INFO[12-24|08:21:46] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2003.json.gz 
INFO[12-24|08:21:47] Inserting fetched CVEs(2003)... 
1500 / 1500 [---------------------------------------------------------------------------------------------------------] 100.00% 3843 p/s
INFO[12-24|08:21:48] Refreshed 1500 CVEs. 
INFO[12-24|08:21:48] Fetching CVE information from NVD(2004). 
INFO[12-24|08:21:48] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.json.gz 
INFO[12-24|08:21:50] Inserting fetched CVEs(2004)... 
2644 / 2644 [---------------------------------------------------------------------------------------------------------] 100.00% 2227 p/s
INFO[12-24|08:21:51] Refreshed 2644 CVEs. 
INFO[12-24|08:21:51] Fetching CVE information from NVD(2005). 
INFO[12-24|08:21:51] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.json.gz 
INFO[12-24|08:21:53] Inserting fetched CVEs(2005)... 
4623 / 4623 [---------------------------------------------------------------------------------------------------------] 100.00% 2488 p/s
INFO[12-24|08:21:56] Refreshed 4623 CVEs. 
INFO[12-24|08:21:56] Fetching CVE information from NVD(2006). 
INFO[12-24|08:21:56] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.json.gz 
INFO[12-24|08:21:58] Inserting fetched CVEs(2006)... 
6991 / 6991 [---------------------------------------------------------------------------------------------------------] 100.00% 2566 p/s
INFO[12-24|08:22:01] Refreshed 6991 CVEs. 
INFO[12-24|08:22:01] Fetching CVE information from NVD(2007). 
INFO[12-24|08:22:01] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2007.json.gz 
INFO[12-24|08:22:05] Inserting fetched CVEs(2007)... 
6454 / 6454 [---------------------------------------------------------------------------------------------------------] 100.00% 2563 p/s
INFO[12-24|08:22:07] Refreshed 6454 CVEs. 
INFO[12-24|08:22:07] Fetching CVE information from NVD(2008). 
INFO[12-24|08:22:07] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2008.json.gz 
INFO[12-24|08:22:10] Inserting fetched CVEs(2008)... 
7000 / 7000 [---------------------------------------------------------------------------------------------------------] 100.00% 2039 p/s
INFO[12-24|08:22:14] Refreshed 7000 CVEs. 
INFO[12-24|08:22:14] Fetching CVE information from NVD(2009). 
INFO[12-24|08:22:14] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2009.json.gz 
INFO[12-24|08:22:17] Inserting fetched CVEs(2009)... 
4902 / 4902 [---------------------------------------------------------------------------------------------------------] 100.00% 1181 p/s
INFO[12-24|08:22:22] Refreshed 4902 CVEs. 
INFO[12-24|08:22:22] Fetching CVE information from NVD(2010). 
INFO[12-24|08:22:22] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2010.json.gz 
INFO[12-24|08:22:25] Inserting fetched CVEs(2010)... 
5037 / 5037 [---------------------------------------------------------------------------------------------------------] 100.00% 1143 p/s
INFO[12-24|08:22:30] Refreshed 5037 CVEs. 
INFO[12-24|08:22:30] Fetching CVE information from NVD(2011). 
INFO[12-24|08:22:30] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.json.gz 
INFO[12-24|08:22:33] Inserting fetched CVEs(2011)... 
4599 / 4599 [----------------------------------------------------------------------------------------------------------] 100.00% 705 p/s
INFO[12-24|08:22:40] Refreshed 4599 CVEs. 
INFO[12-24|08:22:40] Fetching CVE information from NVD(2012). 
INFO[12-24|08:22:40] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.json.gz 
INFO[12-24|08:22:44] Inserting fetched CVEs(2012)... 
5419 / 5419 [----------------------------------------------------------------------------------------------------------] 100.00% 877 p/s
INFO[12-24|08:22:50] Refreshed 5419 CVEs. 
INFO[12-24|08:22:50] Fetching CVE information from NVD(2013). 
INFO[12-24|08:22:50] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz 
INFO[12-24|08:22:55] Inserting fetched CVEs(2013)... 
6135 / 6135 [----------------------------------------------------------------------------------------------------------] 100.00% 900 p/s
INFO[12-24|08:23:02] Refreshed 6135 CVEs. 
INFO[12-24|08:23:02] Fetching CVE information from NVD(2014). 
INFO[12-24|08:23:02] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2014.json.gz 
INFO[12-24|08:23:05] Inserting fetched CVEs(2014)... 
8289 / 8289 [---------------------------------------------------------------------------------------------------------] 100.00% 1857 p/s
INFO[12-24|08:23:10] Refreshed 8289 CVEs. 
INFO[12-24|08:23:10] Fetching CVE information from NVD(2015). 
INFO[12-24|08:23:10] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2015.json.gz 
INFO[12-24|08:23:13] Inserting fetched CVEs(2015)... 
7923 / 7923 [---------------------------------------------------------------------------------------------------------] 100.00% 1760 p/s
INFO[12-24|08:23:18] Refreshed 7923 CVEs. 
INFO[12-24|08:23:18] Fetching CVE information from NVD(2016). 
INFO[12-24|08:23:18] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2016.json.gz 
INFO[12-24|08:23:21] Inserting fetched CVEs(2016)... 
9213 / 9213 [---------------------------------------------------------------------------------------------------------] 100.00% 1782 p/s
INFO[12-24|08:23:27] Refreshed 9213 CVEs. 
INFO[12-24|08:23:27] Fetching CVE information from NVD(2017). 
INFO[12-24|08:23:27] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2017.json.gz 
INFO[12-24|08:23:31] Inserting fetched CVEs(2017)... 
14383 / 14383 [-------------------------------------------------------------------------------------------------------] 100.00% 1935 p/s
INFO[12-24|08:23:39] Refreshed 14383 CVEs. 
INFO[12-24|08:23:39] Fetching CVE information from NVD(2018). 
INFO[12-24|08:23:39] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2018.json.gz 
INFO[12-24|08:23:43] Inserting fetched CVEs(2018)... 
15629 / 15629 [-------------------------------------------------------------------------------------------------------] 100.00% 2221 p/s
INFO[12-24|08:23:50] Refreshed 15629 CVEs. 
INFO[12-24|08:23:50] Fetching CVE information from NVD(2019). 
INFO[12-24|08:23:50] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz 
INFO[12-24|08:23:55] Inserting fetched CVEs(2019)... 
15385 / 15385 [-------------------------------------------------------------------------------------------------------] 100.00% 2074 p/s
INFO[12-24|08:24:02] Refreshed 15385 CVEs. 
INFO[12-24|08:24:02] Fetching CVE information from NVD(2020). 
INFO[12-24|08:24:02] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2020.json.gz 
INFO[12-24|08:24:09] Inserting fetched CVEs(2020)... 
17856 / 17856 [-------------------------------------------------------------------------------------------------------] 100.00% 1784 p/s
INFO[12-24|08:24:19] Refreshed 17856 CVEs. 
INFO[12-24|08:24:19] Fetching CVE information from NVD(2021). 
INFO[12-24|08:24:19] Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2021.json.gz 
INFO[12-24|08:24:24] Inserting fetched CVEs(2021)... 
15775 / 15775 [-------------------------------------------------------------------------------------------------------] 100.00% 1955 p/s
INFO[12-24|08:24:32] Refreshed 15775 CVEs. 
INFO[12-24|08:24:32] Finished fetching NVD.

Checklist:

You don't have to satisfy all of the following.

  • Write tests
  • Write documentation
  • Check that there aren't other open pull requests for the same issue/feature
  • Format your source code by make fmt
  • Pass the test by make test
  • Provide verification config / commands
  • Enable "Allow edits from maintainers" for this PR
  • Update the messages below

Is this ready for review?: YES

Reference

@MaineK00n MaineK00n self-assigned this Dec 23, 2021
@MaineK00n MaineK00n mentioned this pull request Dec 23, 2021
Closed
@MaineK00n MaineK00n marked this pull request as ready for review December 23, 2021 23:25
@MaineK00n MaineK00n requested a review from kotakanbe December 23, 2021 23:25
@MaineK00n MaineK00n force-pushed the MaineK00n/fetch-by-year branch from 3b4b490 to aa06c41 Compare December 28, 2021 00:53
@MaineK00n MaineK00n force-pushed the MaineK00n/fetch-by-year branch from aa06c41 to 1d70569 Compare December 28, 2021 00:55
@kotakanbe kotakanbe merged commit aa64b3b into master Jan 6, 2022
@kotakanbe kotakanbe deleted the MaineK00n/fetch-by-year branch January 6, 2022 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kotakanbe kotakanbe kotakanbe approved these changes

Assignees

@MaineK00n MaineK00n

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

go-cve-dictionary does not support fetch CVE by year
2 participants
@MaineK00n @kotakanbe