feat: pull updates from mirror with --use-mirror flag

[b31ngd3v]

part of #2577

[codecov-commenter]

Codecov Report

Merging #2811 (a6e2851) into main (6af5701) will decrease coverage by 0.30%.
The diff coverage is 74.35%.

@@            Coverage Diff             @@
##             main    #2811      +/-   ##
==========================================
- Coverage   82.29%   82.00%   -0.30%     
==========================================
  Files         651      655       +4     
  Lines       10254    10420     +166     
  Branches     1382     1409      +27     
==========================================
+ Hits         8439     8545     +106     
- Misses       1446     1505      +59     
- Partials      369      370       +1     

Flag	Coverage Δ
longtests	82.00% <74.35%> (+0.19%)	⬆️
win-longtests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cve_bin_tool/cvedb.py	75.38% <20.00%> (+0.60%)	⬆️
cve_bin_tool/cli.py	67.46% <25.00%> (-1.42%)	⬇️
cve_bin_tool/fetch_json_db.py	70.23% <70.23%> (ø)
cve_bin_tool/error_handler.py	91.54% <100.00%> (+0.12%)	⬆️
test/test_fetch_json_db.py	100.00% <100.00%> (ø)

... and 18 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[anthonyharrison]

@b31ngd3v @terriko I think #2356 may also be relevent here.

[anthonyharrison]

@b31ngd3v @terriko What is the reason for pulling data from mirror if more than 7 days old? Should we not make this an evironment variable (the default would be 7 days) so if we want to change the period (shorter or longer), we can esily doso. How does this change work in offline mode as the mirror won't be available?

[terriko]

I'm not sure why 7 days either.

What I was envisioning was more like this:

• mirror gets data
• cve-bin-tool defaults to using the mirror (so no one needs to get a nvd key on first run of the product)
• cve-bin-tool has options to configure the mirror in use (presumably to choose a more "local" one, but also allowing people to use an internal company one or share a cache across machines in an air-gapped network)
• cve-bin-tool provides an option skip the mirror and go directly to nvd (i.e. the current default behaviour)
• in future: we figure out how to also deal with mirroring of gad/redhat/etc. and maybe how to configure mirroring of each of those separately/together

Maybe we could convert this PR to adding support for a flag for now? --use-mirror $url or something?

[b31ngd3v]

@terriko @anthonyharrison after fetching the data from mirror, should it incrementally update the database too? that should be the normal behavior when --use-mirror $url flag is used right?

[b31ngd3v]

hi @terriko looks like the mirror-sandbox repo is private, can you make the repo public so that i can setup an example where you can fetch the data using the mirror? thanks.

[terriko]

hi @terriko looks like the mirror-sandbox repo is private, can you make the repo public so that i can setup an example where you can fetch the data using the mirror? thanks.

should be fixed now, sorry!

[b31ngd3v]

@terriko i was trying to push the ci changes, then this happened

also from the github ui, there is no merge button

[terriko]

Sorry, I forgot that I hadn't set you as an admin on that repo. you should be able to do everything there now. You'll need to set up a new NVD API key for it and set it into secrets; I didn't want to re-use the one from the main repo.

[b31ngd3v]

@terriko can you please enable this option? i'm unable to change it! thanks!!

[terriko]

Wow, github permissions are weird. I had to change it in the org (the repo wouldn't let me change it either, just gave a no-entry icon🚫 ) but I think it may work now?

[b31ngd3v]

@terriko @anthonyharrison I think this PR is ready! you can test it with cve-bin-tool --use-mirror 'https://raw.githubusercontent.com/sec-data/mirror-sandbox/main/exported_data' --nvd-api-key YOUR_API_KEY_FOR_INCREMENTAL_UPDATE

[terriko]

Minor nit about return codes, but this is looking really good!

For this PR to get merged I'd like to see:

• documentation of the new flag
• a test (We could use the sandbox mirror but it's probably better to use mock and have it get a fake updated cve for testing that can immediately be deleted)

For either this PR or a future one, I think we need to add and validate a jsonschema to make sure that the mirror data isn't garbage. I think we'll also be doing some fancier stuff with storing mirrors in the config file too, but that's definitely future PR material and not needed yet.

But yeah, I'm really excited to see this coming together. Let's get docs and tests at minimum and get it merged so people can try it out!

[b31ngd3v]

@terriko done! re-requesting review! thanks!

[terriko]

This looks good enough to merge! I think we'll be iterating and refactoring a bit to add some validation but this should be good enough to get started!