Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: basic execution test for EPSS #4484

Open
terriko opened this issue Sep 30, 2024 · 3 comments
Open

test: basic execution test for EPSS #4484

terriko opened this issue Sep 30, 2024 · 3 comments
Assignees
Labels
good first issue Good for newcomers hacktoberfest good issue for hacktoberfest participation

Comments

@terriko
Copy link
Contributor

terriko commented Sep 30, 2024

EPSS tests are surprisingly hard because the scores change constantly, and we had to disable some of the tests originally written because they were too unstable. But as a result, the bug in #4473 slipped through because we didn't have a check for the most basic of errors.

What I'd like is a test that runs a scan with epss enabled, then checks the logs to make sure no errors appear in the log, I believe this should be pretty easy, since you can use caplog.set_level(logging.ERROR) and check that nothing occurs in there. You should be able to look at the existing test in test__source_epss.py to give you a hint on how to call just the epss related code, or you could look at test_cli.py for how to do a full command line run. Remember to strip down what you test as much as possible to make the test run faster: ideally we want to look up as little as possible because this test is mostly about making sure there's no obvious errors we're missing.

This may require some reading to make sure you understand how to use pytest and how to call the correct code, but I think it should be beginner-friendly so I'm tagging it as a "good first issue" and will put the new contributor tips below. I'm also tagging it for hacktoberfest. If you're interested in doing this as part of the contest, make sure you do it within their timeframe and follow their rules. (Most notably: don't create the PR before October 1 or it won't count)

Short tips for new contributors:

  • cve-bin-tool's contributor docs
  • If you've contributed to open source but not this project, you might just want our checklist for a great pull request
  • cve-bin-tool uses https://www.conventionalcommits.org/ style for commit messages, and we have a test that checks the title of your pull request (PR). A good potential title for this one is in the title of this issue.
  • You can make an issue auto close by including a comment "fixes #ISSUENUMBER" in your PR comments where ISSUENUMBER is the actual number of the issue. This "links" the issue to the pull request.

Claiming issues:

  • You do not need to have an issue assigned to you before you work on it. To "claim" an issue either make a linked pull request or comment on the issue saying you'll be working on it.
  • If someone else has already commented or opened a pull request, assume it is claimed and find another issue to work on.
  • If it's been more than 1 week without progress, you can ask in a comment if the claimant is still working on it before claiming it yourself (give them at least 3 days to respond before assuming they have moved on).
@terriko terriko added good first issue Good for newcomers hacktoberfest good issue for hacktoberfest participation labels Sep 30, 2024
@weichslgartner
Copy link
Contributor

I would be happy to work on this issue.

@terriko
Copy link
Contributor Author

terriko commented Oct 14, 2024

@weichslgartner Are you still working on this? Did you need some help?

@weichslgartner
Copy link
Contributor

@terriko sorry for the delay, I will work on the issue this week.

weichslgartner added a commit to weichslgartner/cve-bin-tool that referenced this issue Oct 14, 2024
Add a test to the cli tests to check the EPSS functionality:
It first tests if the the update of EPSS source runs without errors
(regression test for intel#4473).
Then checks for an example SBOM if EPSS values are written to csv report.

test: fix Delete epss testfile if exists
weichslgartner added a commit to weichslgartner/cve-bin-tool that referenced this issue Oct 14, 2024
Add a test to the cli tests to check the EPSS functionality:
It first tests if the the update of EPSS source runs without errors
(regression test for intel#4473).
Then checks for an example SBOM if EPSS values are written to csv report.
weichslgartner added a commit to weichslgartner/cve-bin-tool that referenced this issue Oct 14, 2024
Add a test to the cli tests to check the EPSS functionality:
It first tests if the the update of EPSS source runs without errors
(regression test for intel#4473).
Then checks for an example SBOM if EPSS values are written to csv report.
weichslgartner added a commit to weichslgartner/cve-bin-tool that referenced this issue Oct 14, 2024
Add a test to the cli tests to check the EPSS functionality:
It first tests if the the update of EPSS source runs without errors
(regression test for intel#4473).
Then checks for an example SBOM if EPSS values are written to csv report.
weichslgartner added a commit to weichslgartner/cve-bin-tool that referenced this issue Oct 14, 2024
Add a test to the cli tests to check the EPSS functionality:
It first tests if the update of EPSS source runs without errors
(regression test for intel#4473).
Then checks for an example SBOM if EPSS values are written to csv report.
terriko added a commit that referenced this issue Dec 18, 2024
* test: basic execution test for EPSS #4484

Add a test to the cli tests to check the EPSS functionality:
It first tests if the update of EPSS source runs without errors
(regression test for #4473).
Then checks for an example SBOM if EPSS values are written to csv report.

* test: Added sugestion to use -u never instead of -u now

* Adds better assert messages on failure and filters out empty lines in windows csv files cause by double newlines in csv file

---------

Co-authored-by: Terri Oda <terri.oda@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers hacktoberfest good issue for hacktoberfest participation
Projects
None yet
Development

No branches or pull requests

2 participants