Expand matching of requirements.txt

[alfredodeza]

Currently, grype matches on requirements.txt files for that file only, and I think it is valid to match on more combinations of requirements.txt like:

• requirements-dev.txt
• requirements-prod.txt
• test-requirements.txt

This grep found those variations in (my) local repositories that would benefit from getting inspected. This will probably need only the globs to be updated for Python.

[dakaneye]

@alfredodeza @luhring @wagoodman I spent some time looking through the Grype code, and it seems like Grype is just doing the matching based on the packages returned from syft, which is looking in any requirements.txt file.

I believe if we change that glob expression from **/requirements.txt to **/*requirements*.txt it should allow syft to look through multiple files, hence returning packages from all of them for Grype to scan and match against vulnerabilities.

[luhring]

Sounds cool, @dakaneye! I don't know the Python ecosystem very well. The only risk I could imagine is picking up false positives. But assuming we think there's a low chance that we'll come across txt files with "requirements" somewhere in the name that don't specify Python dependencies, I think this is a fantastic solution.

[dakaneye]

@luhring I think mostly the use case for having multiple types of requirements files is to add debugging tools or other things that won't go into you're production binary / app environment. The value add here is pretty small, to be honest, I was just looking for a "Good First Issue" to do some Golang! You'd probably still want to know if a package there was vulnerable, as long as you could trace it back to which file it was declared in.

[wagoodman]

I think it would be relatively safe to loosen the pattern to **/*requirements*.txt since false positives would require the filename to be the same as well as the internal format (lines with name == some.sem.ver), which seems unlikely to me. What I do wonder about is if the contents of requirements-dev.txt is really valid? That is, a service probably isn't vulnerable in prod just because a dev package is vulnerable. For a container, no one should be packaging requirements-dev.txt files anyway, so it shouldn't be an issue... for a dir scan I could go either way.

I ultimately see that having a vulnerable package is undesirable and the point of the tool is to notify the user of vulnerabilities outside of any dev/prod context, so the change is probably worth it.

[alfredodeza]

I don't think we do any distinction about "is this package in prod? or is it in a dev environment?" to selectively say something is vulnerable and worth notifying.

The whole directory approach is based on the suspicion that packages will get installed. A development workflow for example, regardless of a containerized output or not, could include adding new (or newer) dependencies to a separate requirements file, so that it can be promoted later. Catching it early, and reporting it, is valuable information, to prevent churn at the end of the process.

I think @wagoodman is basically on the same page, but wanted to emphasize that it is important to catch these things even in seemingly non-prod environments.

[wagoodman]

agreed