test: basic execution test for EPSS #4484

[weichslgartner]

Adds test described in #4484.
Add a test to the cli tests to check the EPSS functionality: It first tests if the update of EPSS source runs without errors (regression test for #4473).
Then checks for an example SBOM if EPSS values are written to csv report.
I tried to disable the other data sources, but needed NVD. Any suggestions to minimize the test are welcome.

Should fail on current main, should pass after #4475 is merged.
Tested it locally with https://github.com/weichslgartner/cve-bin-tool/tree/basic_execution_test_for_EPSS

[terriko]

Looks like the test is failing:

 =========================== short test summary info ============================
FAILED test/test_cli.py::TestCLI::test_basic_epss - ValueError: could not convert string to float: ''
============= 1 failed, 22 passed, 11 skipped in 68.76s (0:01:08) ==============

[weichslgartner]

@terriko I branched from main when the other fix was not yet merged so this was expected. Yet it didn't fail during the update (as was skipped because of the cache), but only in the step when it checked if the EPSS value from the csv file. I will update with the main branch, then this test should pass.

[weichslgartner]

@terriko maybe

cve-bin-tool/test/test_cli.py

Line 169 in 2b15bc5

assert (

could be run with cap level error?

[terriko]

Ah, my bad -- for some reason I had it in my head that this had been updated but of course it hadn't. (that's what I get for multitasking too much this morning while scanning PRs.) Letting things re-run now so I can see what's up.

[terriko]

Still getting the same error:

=========================== short test summary info ============================
FAILED test/test_cli.py::TestCLI::test_basic_epss - ValueError: could not convert string to float: ''
============= 1 failed, 22 passed, 11 skipped in 68.77s (0:01:08) ==============

I haven't dug into it, but I do see that error often when we forget to trim whitespace on a string, so there may be a wayward " " somewhere? Or as you say, maybe the log level isn't giving you the correct messages and you'll need to swap it?

[weichslgartner]

@terriko It seems it didn't update the EPSS data because the cache seems to be still old (before the EPSS fix). The Value error comes from parsing values out of the generated csv files. The two last values should be epss_percentile and epss_probability (it also asserts the header and they didn't trigger). If EPSS failed there are no values there hence the conversion fails. (I will create a better error message here). I reproduced the this fail locally with an old cache with not epss data. Only if I do an "-u now" update it downloads the data correctly. So having a way to force epss update without have to download all the cve data would be good.

[terriko]

Hm, the cache was broken but I fixed it earlier this week and I did update last night. Let me go pull latest and then try re-running this.

[terriko]

(I'm wondering if there's something else up with caching that I'm not seeing, though, because I'm still having some weirdness on jobs.)

[terriko]

Okay, cache has updated. I'm going to try re-running the failing tests again and see if that unstuck it or if there's more to do.

[terriko]

Kicking off the tests again.

[weichslgartner]

@terriko
I did a little debugging under windows and seems there the csv file contains empty rows cause by (\n\n), first I thought is is a windows line ending \r\n issue, but splitlines should take care of the issue.
The file looks like:

vendor,product,version,location,cve_number,severity,score,source,cvss_version,cvss_vector,paths,remarks,comments,epss_probability,epss_percentile

gnu,glibc,2.11.1,NotFound,CVE-2009-5029,MEDIUM,6.8,NVD,2,AV:N/AC:M/Au:N/C:P/I:P/A:P,,NewFound,,0.00801,0.82134

gnu,glibc,2.11.1,NotFound,CVE-2009-5155,HIGH,7.5,NVD,3,CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H,,NewFound,,0.00469,0.76073

gnu,glibc,2.11.1,NotFound,CVE-2010-0296,HIGH,7.2,NVD,2,AV:L/AC:L/Au:N/C:C/I:C/A:C,,NewFound,,0.00044,0.11027

I added now a step to filter out empty lines and now the windows tests run at least locally on my windows machine.
Can check why there are empty lines in the csv file under windows tomorrow.

[weichslgartner]

seems to be this issue: https://stackoverflow.com/a/30930022

[weichslgartner]

I opened #4557 for the csv issue under windows

[terriko]

kicking off the tests again. Not sure if I need to rebase the branch to get the other fix in so I'm flagging this for myself so I double-check if it fails.

[terriko]

K, so the long tests failed again. Going to rebase the branch and try again.

Dismissing review in case it's what's blocking the tests from re-running.

[terriko]

One last time! I hope! I'm going to bump the branch for the new split up long tests and see where we're at now that they're actually behaving consistently.

[terriko]

Okay, I think we're finally ready to merge. Thank you so much for working on this and yoru extreme patience with the CI system!