Catalog packages from source build.gradle during directory scans

[nedenwalker]

What would you like to be added:

During directory scans (e.g. w/ the dir: scheme), let's scan build.gradle files on disk as a means of learning about dependencies, and let's surface those dependencies as packages. Today Syft does parse build.gradle files, but only within the context of .jar/.war/etc. files.

Available with Gradle is the ability to create a dependency graph via ./gradlew dependencies.

Why is this needed:

This allows for source code detection of dependencies, even when no dependencies have been installed yet.

Additional context:

For an example repo that has dependencies listed in pom.xml, check out: https://github.com/nedenwalker/spring-boot-app-using-gradle

Note: Once this issue is implemented, we should pull this into Grype to improve Grype's scans of local code repositories.

[luhring]

One note for developers — we may be able to deliver more value sooner by breaking this up into two iterations. Today, there's not a precedent in Syft for dynamic dependency look-ups via network calls.

The value of doing these look-ups is apparent, especially for developer-focused workflows. So we should strongly consider adding a system for this into Syft at some point.

But until then, a lighter weight improvement would be to surface dependencies that are listed explicitly, and accept that transient dependencies won't be discovered until the second iteration.

[bolshoytoster]

I've started working on this, I might also add #676 as well.

[luhring]

@bolshoytoster Wow, that's great — thanks! Let us know if we can do anything to help.

[nedenwalker]

Nice, in researching this, also came across some mention that a Gradle lockfile can be created within a project, and so some users may have that. Maven apparently cannot have a lockfile. But we were going to run through this doc and see how it works with the demo spring boot app in the description: https://medium.com/@anthogez/improved-security-testing-for-git-based-gradle-projects-using-lockfile-f0af772a9e25

[bolshoytoster]

@luhring

Today, there's not a precedent in Syft for dynamic dependency look-ups via network calls.

I might have to implement this since the build.gradle file only includes dependency names and versions.

[nedenwalker]

@bolshoytoster I followed the article I linked about adding a gradle dependency lockfile (and checked the code change into the sample gradle app: https://github.com/nedenwalker/spring-boot-app-using-gradle). My sense is that if a project has a gradle.lockfile present, then it could be used for scanning, without requiring any network calls. If not present, then network calls would be required as you mentioned.

[bolshoytoster]

@nedenwalker

@bolshoytoster I followed the article I linked about adding a gradle dependency lockfile. My sense is that if a project has a gradle.lockfile present, then it could be used for scanning, without requiring any network calls. If not present, then network calls would be required as you mentioned.

I can probably do that, I'm not entirely sure if the current package scanning system would allow that but I'll give it a try.

[wagoodman]

Today, there's not a precedent in Syft for dynamic dependency look-ups via network calls.

I might have to implement this since the build.gradle file only includes dependency names and versions.

Just a heads up, today syft takes the philosophy of not needing to reach out to external sources to enrich the SBOM results. We hope to start doing this in the future, as this would be extremely valuable, however, we need to add it in a way that makes sense for all ecosystems that syft supports.

That is, if we add this feature for java, extending the same concept to python, go, ruby, javascript, etc should be "easy" if the architecture is done correctly.

A good first step for this issue may be to add what can be added from the local files that are available first (though imperfect) and build from there. This would be consistent with the other ecosystem catalogers.

[bolshoytoster]

I've opened a pr for this (#707). It doesn't parse gradle.lockfile, I could add that if needed.

I wasn't sure about writing tests, so I left them out. Also the code could probably be optimised a bit more.

Aside from that, it works; it gets the name and version of each dependency and returns it as

pkg.Package{
    Name:         name,
    Version:      version,
    FoundBy:      "java-gradle-cataloger",
    Language:     pkg.Java,
    Type:         pkg.JavaPkg,
})

[nedenwalker]

Hi there, just tested the flow again using the sample repo in the description. The directory scan continues to show no vulns. And then once the binary is built, vulns are found. PR #707 seems to partially address this? But it has only been merged into main and it has not been released yet? As it is, what would next steps be?

[spiffcs]

Hey @bolshoytoster! I saw you had closed #707.

As of right now, we have merged in the pom.xml support. If you're interested I think we can call this issue closed after merging or working through any changes needed for #707?

I'm happy to work with you on it. Also @nedenwalker if you're interested in reopening or helping with the grade.lockfile support that would be awesome!