fix(fetch): not to use htcat, use backoff retry, set user-agent as curl

[kotakanbe]

What did you implement:

In some environments, it seems that the following error occurs and the NVD cannot be fetched.
So, Implemented retry on error.

Failed to FetchConvert. err: Failed to fetch. err: Failed to fetch. err: Failed to fetch file. err: aborting: could not write to output stream: Get "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.json.gz": dial tcp 54.85.30.225:443: connect: operation timed out

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Complete fetches while retrying on error.

08:13 ≡ go-cve-dictionary fetch nvd  --log-json --dbtype redis --dbpath redis://localhost/0
{"lvl":"info","msg":"Inserting NVD into DB (redis).","t":"2021-09-24T08:14:21.179141+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(recent, modified).","t":"2021-09-24T08:14:21.17926+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.json.gz","t":"2021-09-24T08:14:21.179285+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.json.gz","t":"2021-09-24T08:14:22.42739+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2002).","t":"2021-09-24T08:14:24.323485+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.json.gz","t":"2021-09-24T08:14:24.323586+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2002)...","t":"2021-09-24T08:14:26.906518+09:00"}
2355 / 2355 [-----------------------------------------------------------------------------] 100.00% 2852 p/s
{"lvl":"info","msg":"Refreshed 2355 CVEs.","t":"2021-09-24T08:14:27.932921+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2003).","t":"2021-09-24T08:14:27.932954+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2003.json.gz","t":"2021-09-24T08:14:27.932964+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2003)...","t":"2021-09-24T08:14:29.724067+09:00"}
1500 / 1500 [-----------------------------------------------------------------------------] 100.00% 3417 p/s
{"lvl":"info","msg":"Refreshed 1500 CVEs.","t":"2021-09-24T08:14:30.363435+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2004).","t":"2021-09-24T08:14:30.363479+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.json.gz","t":"2021-09-24T08:14:30.363491+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2004)...","t":"2021-09-24T08:14:32.467547+09:00"}
2644 / 2644 [-----------------------------------------------------------------------------] 100.00% 1340 p/s
{"lvl":"info","msg":"Refreshed 2644 CVEs.","t":"2021-09-24T08:14:34.640692+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2005).","t":"2021-09-24T08:14:34.640716+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.json.gz","t":"2021-09-24T08:14:34.640726+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2005)...","t":"2021-09-24T08:14:37.911279+09:00"}
4623 / 4623 [-----------------------------------------------------------------------------] 100.00% 1753 p/s
{"lvl":"info","msg":"Refreshed 4623 CVEs.","t":"2021-09-24T08:14:40.749182+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2006).","t":"2021-09-24T08:14:40.749218+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.json.gz","t":"2021-09-24T08:14:40.749228+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2006)...","t":"2021-09-24T08:14:56.946434+09:00"}
6991 / 6991 [-----------------------------------------------------------------------------] 100.00% 2358 p/s
{"lvl":"info","msg":"Refreshed 6991 CVEs.","t":"2021-09-24T08:15:00.111634+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2007).","t":"2021-09-24T08:15:00.111665+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2007.json.gz","t":"2021-09-24T08:15:00.111681+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2007)...","t":"2021-09-24T08:15:02.800024+09:00"}
6454 / 6454 [-----------------------------------------------------------------------------] 100.00% 3170 p/s
{"lvl":"info","msg":"Refreshed 6454 CVEs.","t":"2021-09-24T08:15:05.036118+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2008).","t":"2021-09-24T08:15:05.036142+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2008.json.gz","t":"2021-09-24T08:15:05.036152+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2008)...","t":"2021-09-24T08:15:08.016479+09:00"}
7000 / 7000 [-----------------------------------------------------------------------------] 100.00% 2401 p/s
{"lvl":"info","msg":"Refreshed 7000 CVEs.","t":"2021-09-24T08:15:11.132543+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2009).","t":"2021-09-24T08:15:11.132575+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2009.json.gz","t":"2021-09-24T08:15:11.132587+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2009)...","t":"2021-09-24T08:15:14.157729+09:00"}
4902 / 4902 [-----------------------------------------------------------------------------] 100.00% 1561 p/s
{"lvl":"info","msg":"Refreshed 4902 CVEs.","t":"2021-09-24T08:15:17.497675+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2010).","t":"2021-09-24T08:15:17.4977+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2010.json.gz","t":"2021-09-24T08:15:17.49771+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2010)...","t":"2021-09-24T08:15:20.552691+09:00"}
5036 / 5036 [-----------------------------------------------------------------------------] 100.00% 1571 p/s
{"lvl":"info","msg":"Refreshed 5036 CVEs.","t":"2021-09-24T08:15:23.957421+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2011).","t":"2021-09-24T08:15:23.957455+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.json.gz","t":"2021-09-24T08:15:23.957468+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2011)...","t":"2021-09-24T08:15:32.704571+09:00"}
4591 / 4591 [-----------------------------------------------------------------------------] 100.00% 1312 p/s
{"lvl":"info","msg":"Refreshed 4591 CVEs.","t":"2021-09-24T08:15:36.403319+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2012).","t":"2021-09-24T08:15:36.403358+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.json.gz","t":"2021-09-24T08:15:36.403369+09:00"}
{"lvl":"warn","msg":"Failed to HTTP GET. retrying in 486.818453ms seconds. err: HTTP GET error, url: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.json.gz, err:\n    github.com/vulsio/go-cve-dictionary/fetcher.FetchFeedFile.func1\n        /Users/ma2no/go/src/github.com/vulsio/go-cve-dictionary/fetcher/fetcher.go:29\n  - Failed to GET. url: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.json.gz, err:\n    github.com/vulsio/go-cve-dictionary/fetcher.fetchFile\n        /Users/ma2no/go/src/github.com/vulsio/go-cve-dictionary/f
etcher/fetcher.go:74\n  - Get \"https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.json.gz\": dial tcp 54.85.30.225:443: connect: operation timed out","t":"2021-09-24T08:16:02.443445+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2012)...","t":"2021-09-24T08:16:06.330743+09:00"}
5418 / 5418 [-----------------------------------------------------------------------------] 100.00% 1600 p/s
{"lvl":"info","msg":"Refreshed 5418 CVEs.","t":"2021-09-24T08:16:09.916593+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2013).","t":"2021-09-24T08:16:09.916617+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz","t":"2021-09-24T08:16:09.916626+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2013)...","t":"2021-09-24T08:16:14.983061+09:00"}
6136 / 6136 [------------------------------------------------------------------------------] 100.00% 911 p/s
{"lvl":"info","msg":"Refreshed 6136 CVEs.","t":"2021-09-24T08:16:21.918999+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2014).","t":"2021-09-24T08:16:21.919037+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2014.json.gz","t":"2021-09-24T08:16:21.919053+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2014)...","t":"2021-09-24T08:16:26.311249+09:00"}
8289 / 8289 [-----------------------------------------------------------------------------] 100.00% 3006 p/s
{"lvl":"info","msg":"Refreshed 8289 CVEs.","t":"2021-09-24T08:16:29.268384+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2015).","t":"2021-09-24T08:16:29.268405+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2015.json.gz","t":"2021-09-24T08:16:29.268415+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2015)...","t":"2021-09-24T08:16:32.228407+09:00"}
7919 / 7919 [-----------------------------------------------------------------------------] 100.00% 2939 p/s
{"lvl":"info","msg":"Refreshed 7919 CVEs.","t":"2021-09-24T08:16:35.12287+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2016).","t":"2021-09-24T08:16:35.1229+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2016.json.gz","t":"2021-09-24T08:16:35.122913+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2016)...","t":"2021-09-24T08:16:38.245556+09:00"}
9211 / 9211 [-----------------------------------------------------------------------------] 100.00% 2222 p/s
{"lvl":"info","msg":"Refreshed 9211 CVEs.","t":"2021-09-24T08:16:42.591647+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2017).","t":"2021-09-24T08:16:42.591667+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2017.json.gz","t":"2021-09-24T08:16:42.591676+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2017)...","t":"2021-09-24T08:16:46.756969+09:00"}
14369 / 14369 [---------------------------------------------------------------------------] 100.00% 3132 p/s
{"lvl":"info","msg":"Refreshed 14369 CVEs.","t":"2021-09-24T08:16:51.544099+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2018).","t":"2021-09-24T08:16:51.544121+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2018.json.gz","t":"2021-09-24T08:16:51.54413+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2018)...","t":"2021-09-24T08:16:56.714512+09:00"}
15618 / 15618 [---------------------------------------------------------------------------] 100.00% 3150 p/s
{"lvl":"info","msg":"Refreshed 15618 CVEs.","t":"2021-09-24T08:17:01.872583+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2019).","t":"2021-09-24T08:17:01.872608+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz","t":"2021-09-24T08:17:01.872618+09:00"}
{"lvl":"warn","msg":"Failed to HTTP GET. retrying in 723.68858ms seconds. err: HTTP GET error, url: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz, err:\n    github.com/vulsio/go-cve-dictionary/fetcher.FetchFeedFile.func1\n        /Users/ma2no/go/src/github.com/vulsio/go-cve-dictionary/fetcher/fetcher.go:29\n  - Failed to GET. url: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz, err:\n    github.com/vulsio/go-cve-dictionary/fetcher.fetchFile\n        /Users/ma2no/go/src/github.com/vulsio/go-cve-dictionary/fe
tcher/fetcher.go:74\n  - Get \"https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz\": dial tcp 54.85.30.225:443: connect: operation timed out","t":"2021-09-24T08:17:20.635214+09:00"}
{"lvl":"warn","msg":"Failed to HTTP GET. retrying in 710.22612ms seconds. err: HTTP GET error, url: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz, err:\n    github.com/vulsio/go-cve-dictionary/fetcher.FetchFeedFile.func1\n        /Users/ma2no/go/src/github.com/vulsio/go-cve-dictionary/fetcher/fetcher.go:29\n  - Failed to GET. url: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz, err:\n    github.com/vulsio/go-cve-dictionary/fetcher.fetchFile\n        /Users/ma2no/go/src/github.com/vulsio/go-cve-dictionary/fe
tcher/fetcher.go:74\n  - Get \"https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz\": dial tcp 54.85.30.225:443: connect: operation timed out","t":"2021-09-24T08:17:39.810403+09:00"}
{"lvl":"warn","msg":"Failed to HTTP GET. retrying in 1.476809348s seconds. err: HTTP GET error, url: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz, err:\n    github.com/vulsio/go-cve-dictionary/fetcher.FetchFeedFile.func1\n        /Users/ma2no/go/src/github.com/vulsio/go-cve-dictionary/fetcher/fetcher.go:29\n  - Failed to GET. url: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz, err:\n    github.com/vulsio/go-cve-dictionary/fetcher.fetchFile\n        /Users/ma2no/go/src/github.com/vulsio/go-cve-dictionary/f
etcher/fetcher.go:74\n  - Get \"https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz\": dial tcp 54.85.30.225:443: connect: operation timed out","t":"2021-09-24T08:17:59.161741+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2019)...","t":"2021-09-24T08:18:25.07969+09:00"}
15372 / 15372 [---------------------------------------------------------------------------] 100.00% 3125 p/s
{"lvl":"info","msg":"Refreshed 15372 CVEs.","t":"2021-09-24T08:18:30.199457+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2020).","t":"2021-09-24T08:18:30.199478+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2020.json.gz","t":"2021-09-24T08:18:30.199487+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2020)...","t":"2021-09-24T08:18:35.472962+09:00"}
17521 / 17521 [---------------------------------------------------------------------------] 100.00% 2575 p/s
{"lvl":"info","msg":"Refreshed 17521 CVEs.","t":"2021-09-24T08:18:42.475919+09:00"}
{"lvl":"info","msg":"Fetching CVE information from NVD(2021).","t":"2021-09-24T08:18:42.47594+09:00"}
{"lvl":"info","msg":"Fetching... https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2021.json.gz","t":"2021-09-24T08:18:42.475949+09:00"}
{"lvl":"info","msg":"Inserting fetched CVEs(2021)...","t":"2021-09-24T08:18:51.504318+09:00"}
11016 / 11016 [---------------------------------------------------------------------------] 100.00% 2899 p/s
{"lvl":"info","msg":"Refreshed 11016 CVEs.","t":"2021-09-24T08:18:55.504708+09:00"}
{"lvl":"info","msg":"Finished fetching NVD.","t":"2021-09-24T08:18:56.167961+09:00"}

Checklist:

• Write tests
• Write documentation
• Check that there aren't other open pull requests for the same issue/feature
• Format your source code by make fmt
• Pass the test by make test
• Provide verification config / commands
• Enable "Allow edits from maintainers" for this PR
• Update the messages below

Is this ready for review?: YES