Detect dependencies from build.gradle files

[pombredanne]

* Create parser for dependency info from build.gradle files

Signed-off-by: Jono Yang jyang@nexb.com

[tardyp]

Thanks for this. I thinks its going to be useful. I can foresee some problems with this:

• we need tests with some example files, to see what works exactly. I can provide some twisted examples from my twisted kotlin dev collegues :)

• build.gradle can be either groovy or kotlin.

• those are programmation langage so developers are factorizing deps, so for me it just unreliable to just parse the file, you have to execute it.

• gradle is supporting transiant dependencies. So build.gradle only contains rank 1 deps.

• ort, whitesource do run gradle to work. this is painful as you have to provision your build container with your build toolchain and the scanner toolchain

For those reason I advise my devs to use lockfiles like nebula.
Not all of them have made the switch so this will certainly be useful.

[tardyp]

interresting use of pygments :)

[pombredanne]

@tardyp the idea of https://github.com/nexB/pygmars/ is to be used in tandem with pygments to create quickly some lightweight parsers for programming languages.

This is used for instance to lex bash in https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/bashlex.py and parse it in https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/bashparse.py
In this case this is used to enable lightweight Bash parameter extension to parse the Apkbuild scripts from Alpine Linux (to collect metadata) in https://github.com/nexB/scancode-toolkit/blob/9ed2cb4e78a8b1b138bcb201f50203b43098febb/src/packagedcode/alpine.py#L161

The same approach will be used whenever there is a need to parse code to extract structured data as this is easier to read and maintain than a bunch of regex. The upcoming Yocto/Bitbake support likely can benefit from some of it too... and we should be able to parse also Autotools scripts that contain interesting metadata.

[pombredanne]

Note that this is not perfect... and still only static. Next up will be to enable dependency resolution on the deps collected (which may require some live network access ... but this will be an option with proxy support for airgapped envt. ;) )

[JonoYang]

@tardyp Thanks for the review! I've got a script going that uses pygmars to roughly glean what I can from build.gradle files. I'll bring the code over.

[pombredanne]

@tardyp Some more thoughts in reply to your insightful comments:

Thanks for this. I thinks its going to be useful. I can foresee some problems with this:

* we need tests with some example files, to see what works exactly. I can provide some twisted examples from my twisted kotlin dev collegues :)

This would be great if you can attach these.

* build.gradle can be either groovy or kotlin.

This is may work also with kotlin, but the devils is in the details.

* those are programmation langage so developers are factorizing deps, so for me it just unreliable to just parse the file, you have to execute it.

yeah, that's the problem with these executable scripts....

* gradle is supporting transiant dependencies. So build.gradle only contains rank 1 deps. 
* ort, whitesource do run gradle to work. this is painful as you have to provision your build container with your build toolchain and the scanner toolchain

So the plan here would be to:

1. enable resolving deps WITHOUT running gradle (or in general a package manager) using DependentCode, univers and purl/vers

This is not going to be a perfect solution in all cases.

2. also run tools using gradle minimally with containers (possibly borrowing or reusing code from ORT) to collect a full dependency tree

3. process also the lockfiles...

For those reason I advise my devs to use lockfiles like nebula. Not all of them have made the switch so this will certainly be useful.

which would be super useful to have as a parser and having some test files would be super useful too. Do you think nebula is the most common one? Any other common lockfile format otherwise?

[pombredanne]

LGTM!

[pombredanne]

See also #2761

[pombredanne]

I am keeping #793 (comment) open until we have explicit tests for Kotlin

[tardyp]

which would be super useful to have as a parser and having some test files would be super useful too. Do you think nebula is the most common one? Any other common lockfile format otherwise?

FWIW, we do have some nebula support code, and we do need to find some slot in our agenda to upstream it :-}

for the rest, sounds like a plan!

[pombredanne]

FWIW, we do have some nebula support code, and we do need to find some slot in our agenda to upstream it :-}

Awesome... I look forward to it!