Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(java): add support licenses and graph for gradle lock files #6140

Merged
merged 31 commits into from
Mar 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
7e6fe52
feat(gradle): check cache dir to get licenses
DmitriyLewen Feb 14, 2024
cc404bd
refactor: move gradleLockAnalyzer to PostAnalyzer
DmitriyLewen Feb 15, 2024
b7cfa38
refactor
DmitriyLewen Feb 15, 2024
26a51b0
refactor: move pom struct to `types.go`
DmitriyLewen Feb 15, 2024
5111f89
fix linter errors
DmitriyLewen Feb 15, 2024
d46a093
test: update tests
DmitriyLewen Feb 15, 2024
bd66906
feat: add dependencies for pom
DmitriyLewen Feb 15, 2024
4ccc955
feat: fill DependsOn from poms
DmitriyLewen Feb 16, 2024
4f31a54
refactor
DmitriyLewen Feb 16, 2024
460a0fa
chore(deps): go.mod tidy
DmitriyLewen Feb 16, 2024
5679b16
Merge branch 'main' into 'feat-gradle/license-support)'
DmitriyLewen Feb 19, 2024
ef7273f
refactor: move pom logic from types.go to pom.go
DmitriyLewen Feb 19, 2024
3360c81
test: add pom tests
DmitriyLewen Feb 19, 2024
346acbd
feat: add parser for build.gradle
DmitriyLewen Feb 19, 2024
073048b
add test for parsing build.gradle
DmitriyLewen Feb 19, 2024
6726c42
feat: add build.gradle logic to lock file
DmitriyLewen Feb 19, 2024
cd6be51
add tests
DmitriyLewen Feb 19, 2024
b450dbb
feat: add support of build.gradle.kts files
DmitriyLewen Feb 19, 2024
ebeb37a
test: add empty test for build.gradle
DmitriyLewen Feb 19, 2024
55d3b96
add support of single line "dependencies"
DmitriyLewen Feb 19, 2024
a75068a
fix linter error
DmitriyLewen Feb 19, 2024
0cf9360
Merge branch 'main' into 'feat-gradle/license-support'
DmitriyLewen Feb 20, 2024
8d71b35
docs(java): add info about cache dir and build.gradle files
DmitriyLewen Feb 20, 2024
76f78a8
refactor
DmitriyLewen Feb 20, 2024
580f12d
feat(build.gradle): add excludes support
DmitriyLewen Feb 20, 2024
46d0105
refactor: remove logic for `build.gradle`
DmitriyLewen Feb 21, 2024
2845964
refactor(parser): mark all dependencies as Indirect
DmitriyLewen Feb 21, 2024
1cd367f
refactor: remove unused variables
DmitriyLewen Feb 21, 2024
c081e1b
docs: add gradle.lockfile to dependency tree support
DmitriyLewen Feb 21, 2024
7a378b0
Merge branch 'main' of github.com:DmitriyLewen/trivy into feat-gradle…
DmitriyLewen Mar 18, 2024
81d500b
fix import
DmitriyLewen Mar 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/docs/configuration/reporting.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ The following languages are currently supported:
| Go | [go.mod][go-mod] |
| PHP | [composer.lock][composer-lock] |
| Java | [pom.xml][pom-xml] |
| | [*gradle.lockfile][gradle-lockfile] |
| Dart | [pubspec.lock][pubspec-lock] |

This tree is the reverse of the dependency graph.
Expand Down Expand Up @@ -445,5 +446,6 @@ $ trivy convert --format table --severity CRITICAL result.json
[go-mod]: ../coverage/language/golang.md#go-modules
[composer-lock]: ../coverage/language/php.md#composer
[pom-xml]: ../coverage/language/java.md#pomxml
[gradle-lockfile]: ../coverage/language/java.md#gradlelock
[pubspec-lock]: ../coverage/language/dart.md#dart
[cargo-binaries]: ../coverage/language/rust.md#binaries
34 changes: 24 additions & 10 deletions docs/docs/coverage/language/java.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,19 @@ Trivy supports three types of Java scanning: `JAR/WAR/PAR/EAR`, `pom.xml` and `*

Each artifact supports the following scanners:

| Artifact | SBOM | Vulnerability | License |
| ---------------- | :---: | :-----------: | :-----: |
| JAR/WAR/PAR/EAR | ✓ | ✓ | - |
| pom.xml | ✓ | ✓ | ✓ |
| *gradle.lockfile | ✓ | ✓ | - |
| Artifact | SBOM | Vulnerability | License |
|------------------|:----:|:-------------:|:-------:|
| JAR/WAR/PAR/EAR | ✓ | ✓ | - |
| pom.xml | ✓ | ✓ | ✓ |
| *gradle.lockfile | ✓ | ✓ | |

The following table provides an outline of the features Trivy offers.

| Artifact | Internet access | Dev dependencies | [Dependency graph][dependency-graph] | Position |
|------------------|:---------------------:|:----------------:|:------------------------------------:|:--------:|
| JAR/WAR/PAR/EAR | Trivy Java DB | Include | - | - |
| pom.xml | Maven repository [^1] | Exclude | ✓ | ✓[^7] |
| *gradle.lockfile | - | Exclude | - | ✓ |
| *gradle.lockfile | - | Exclude | | ✓ |

These may be enabled or disabled depending on the target.
See [here](./index.md) for the detail.
Expand Down Expand Up @@ -64,18 +64,32 @@ If you need to show them, use the `--include-dev-deps` flag.


## Gradle.lock
`gradle.lock` files contain all necessary information about used dependencies.
Trivy simply parses the file, extract dependencies, and finds vulnerabilities for them.
It doesn't require the internet access.
`gradle.lock` files only contain information about used dependencies.

!!!note
All necessary files are checked locally. Gradle file scanning doesn't require internet access.

### Dependency-tree
!!! warning "EXPERIMENTAL"
This feature might change without preserving backwards compatibility.
Trivy finds child dependencies from `*.pom` files in the cache[^8] directory.

But there is no reliable way to determine direct dependencies (even using other files).
Therefore, we mark all dependencies as indirect to use logic to guess direct dependencies and build a dependency tree.

### Licenses
Trity also can detect licenses for dependencies.

Make sure that you have cache[^8] directory to find licenses from `*.pom` dependency files.

[^1]: https://github.com/aquasecurity/trivy-java-db
[^1]: Uses maven repository to get information about dependencies. Internet access required.
[^2]: It means `*.jar`, `*.war`, `*.par` and `*.ear` file
[^3]: `ArtifactID`, `GroupID` and `Version`
[^4]: e.g. when parent pom.xml file has `../pom.xml` path
[^5]: When you use dependency path in `relativePath` field in pom.xml file
[^6]: `/Users/<username>/.m2/repository` (for Linux and Mac) and `C:/Users/<username>/.m2/repository` (for Windows) by default
[^7]: To avoid confusion, Trivy only finds locations for direct dependencies from the base pom.xml file.
[^8]: The supported directories are `$GRADLE_USER_HOME/caches` and `$HOME/.gradle/caches` (`%HOMEPATH%\.gradle\caches` for Windows).

[dependency-graph]: ../../configuration/reporting.md#show-origins-of-vulnerable-dependencies
[maven-invoker-plugin]: https://maven.apache.org/plugins/maven-invoker-plugin/usage.html
4 changes: 4 additions & 0 deletions pkg/dependency/parser/gradle/lockfile/parse.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ func (Parser) Parse(r xio.ReadSeekerAt) ([]types.Library, []types.Dependency, er
EndLine: lineNum,
},
},
// There is no reliable way to determine direct dependencies (even using other files).
// Therefore, we mark all dependencies as Indirect.
// This is necessary to try to guess direct dependencies and build a dependency tree.
Indirect: true,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see a second way:
we can mark Gradle dependencies as indirect when before building the dependency tree -

func (r *vulnerabilityRenderer) renderDependencyTree() {

})

}
Expand Down
21 changes: 12 additions & 9 deletions pkg/dependency/parser/gradle/lockfile/parse_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@ func TestParser_Parse(t *testing.T) {
inputFile: "testdata/happy.lockfile",
want: []types.Library{
{
ID: "cglib:cglib-nodep:2.1.2",
Name: "cglib:cglib-nodep",
Version: "2.1.2",
ID: "cglib:cglib-nodep:2.1.2",
Name: "cglib:cglib-nodep",
Version: "2.1.2",
Indirect: true,
Locations: []types.Location{
{
StartLine: 4,
Expand All @@ -32,9 +33,10 @@ func TestParser_Parse(t *testing.T) {
},
},
{
ID: "org.springframework:spring-asm:3.1.3.RELEASE",
Name: "org.springframework:spring-asm",
Version: "3.1.3.RELEASE",
ID: "org.springframework:spring-asm:3.1.3.RELEASE",
Name: "org.springframework:spring-asm",
Version: "3.1.3.RELEASE",
Indirect: true,
Locations: []types.Location{
{
StartLine: 5,
Expand All @@ -43,9 +45,10 @@ func TestParser_Parse(t *testing.T) {
},
},
{
ID: "org.springframework:spring-beans:5.0.5.RELEASE",
Name: "org.springframework:spring-beans",
Version: "5.0.5.RELEASE",
ID: "org.springframework:spring-beans:5.0.5.RELEASE",
Name: "org.springframework:spring-beans",
Version: "5.0.5.RELEASE",
Indirect: true,
Locations: []types.Location{
{
StartLine: 6,
Expand Down
88 changes: 80 additions & 8 deletions pkg/fanal/analyzer/language/java/gradle/lockfile.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,36 +2,104 @@ package gradle

import (
"context"
"fmt"
"io"
"io/fs"
"os"
"sort"
"strings"

"github.com/samber/lo"
"golang.org/x/xerrors"

"github.com/aquasecurity/trivy/pkg/dependency/parser/gradle/lockfile"
godeptypes "github.com/aquasecurity/trivy/pkg/dependency/types"
"github.com/aquasecurity/trivy/pkg/fanal/analyzer"
"github.com/aquasecurity/trivy/pkg/fanal/analyzer/language"
"github.com/aquasecurity/trivy/pkg/fanal/types"
"github.com/aquasecurity/trivy/pkg/log"
"github.com/aquasecurity/trivy/pkg/utils/fsutils"
)

func init() {
analyzer.RegisterAnalyzer(&gradleLockAnalyzer{})
analyzer.RegisterPostAnalyzer(analyzer.TypeGradleLock, newGradleLockAnalyzer)
}

const (
version = 1
version = 2
fileNameSuffix = "gradle.lockfile"
)

// gradleLockAnalyzer analyzes '*gradle.lockfile'
type gradleLockAnalyzer struct{}
type gradleLockAnalyzer struct {
parser godeptypes.Parser
}

func newGradleLockAnalyzer(_ analyzer.AnalyzerOptions) (analyzer.PostAnalyzer, error) {
return &gradleLockAnalyzer{
parser: lockfile.NewParser(),
}, nil
}

func (a gradleLockAnalyzer) Analyze(_ context.Context, input analyzer.AnalysisInput) (*analyzer.AnalysisResult, error) {
p := lockfile.NewParser()
res, err := language.Analyze(types.Gradle, input.FilePath, input.Content, p)
func (a gradleLockAnalyzer) PostAnalyze(_ context.Context, input analyzer.PostAnalysisInput) (*analyzer.AnalysisResult, error) {
poms, err := parsePoms()
if err != nil {
return nil, xerrors.Errorf("%s parse error: %w", input.FilePath, err)
log.Logger.Warnf("Unable to get licenses and dependsOn: %s", err)
}

required := func(path string, d fs.DirEntry) bool {
return a.Required(path, nil)
}
return res, nil

var apps []types.Application
err = fsutils.WalkDir(input.FS, ".", required, func(filePath string, _ fs.DirEntry, r io.Reader) error {
var app *types.Application
app, err = language.Parse(types.Gradle, filePath, r, a.parser)
if err != nil {
return xerrors.Errorf("%s parse error: %w", filePath, err)
}

if app == nil {
return nil
}

libs := lo.SliceToMap(app.Libraries, func(lib types.Package) (string, struct{}) {
return lib.ID, struct{}{}
})

for i, lib := range app.Libraries {
pom := poms[lib.ID]

// Fill licenses from pom file
if len(pom.Licenses.License) > 0 {
app.Libraries[i].Licenses = lo.Map(pom.Licenses.License, func(license License, _ int) string {
return license.Name
})
}

// File child deps from pom file
var deps []string
for _, dep := range pom.Dependencies.Dependency {
id := packageID(dep.GroupID, dep.ArtifactID, dep.Version)
if _, ok := libs[id]; ok {
deps = append(deps, id)
}
}
sort.Strings(deps)
app.Libraries[i].DependsOn = deps
}

sort.Sort(app.Libraries)
apps = append(apps, *app)
return nil
})
if err != nil {
return nil, xerrors.Errorf("walk error: %w", err)
}

return &analyzer.AnalysisResult{
Applications: apps,
}, nil
}

func (a gradleLockAnalyzer) Required(filePath string, _ os.FileInfo) bool {
Expand All @@ -45,3 +113,7 @@ func (a gradleLockAnalyzer) Type() analyzer.Type {
func (a gradleLockAnalyzer) Version() int {
return version
}

func packageID(groupId, artifactId, ver string) string {
return fmt.Sprintf("%s:%s:%s", groupId, artifactId, ver)
}
99 changes: 79 additions & 20 deletions pkg/fanal/analyzer/language/java/gradle/lockfile_test.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package gradle

import (
"context"
"os"
"testing"

Expand All @@ -13,54 +14,112 @@ import (

func Test_gradleLockAnalyzer_Analyze(t *testing.T) {
tests := []struct {
name string
inputFile string
want *analyzer.AnalysisResult
name string
dir string
cacheDir string
want *analyzer.AnalysisResult
}{
{
name: "happy path",
inputFile: "testdata/happy.lockfile",
name: "happy path",
dir: "testdata/lockfiles/happy",
cacheDir: "testdata/cache",
want: &analyzer.AnalysisResult{
Applications: []types.Application{
{
Type: types.Gradle,
FilePath: "gradle.lockfile",
Libraries: types.Packages{
{
ID: "junit:junit:4.13",
Name: "junit:junit",
Version: "4.13",
Indirect: true,
Locations: []types.Location{
{
StartLine: 4,
EndLine: 4,
},
},
Licenses: []string{
"Eclipse Public License 1.0",
},
DependsOn: []string{
"org.hamcrest:hamcrest-core:1.3",
},
},
{
ID: "org.hamcrest:hamcrest-core:1.3",
Name: "org.hamcrest:hamcrest-core",
Version: "1.3",
Indirect: true,
Locations: []types.Location{
{
StartLine: 5,
EndLine: 5,
},
},
},
},
},
},
},
},
{
name: "happy path without cache",
dir: "testdata/lockfiles/happy",
want: &analyzer.AnalysisResult{
Applications: []types.Application{
{
Type: types.Gradle,
FilePath: "testdata/happy.lockfile",
FilePath: "gradle.lockfile",
Libraries: types.Packages{
{
ID: "com.example:example:0.0.1",
Name: "com.example:example",
Version: "0.0.1",
ID: "junit:junit:4.13",
Name: "junit:junit",
Version: "4.13",
Indirect: true,
Locations: []types.Location{
{
StartLine: 4,
EndLine: 4,
},
},
},
{
ID: "org.hamcrest:hamcrest-core:1.3",
Name: "org.hamcrest:hamcrest-core",
Version: "1.3",
Indirect: true,
Locations: []types.Location{
{
StartLine: 5,
EndLine: 5,
},
},
},
},
},
},
},
},
{
name: "empty file",
inputFile: "testdata/empty.lockfile",
name: "empty file",
dir: "testdata/lockfiles/empty",
want: &analyzer.AnalysisResult{},
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
f, err := os.Open(tt.inputFile)
if tt.cacheDir != "" {
t.Setenv("GRADLE_USER_HOME", tt.cacheDir)
}

a, err := newGradleLockAnalyzer(analyzer.AnalyzerOptions{})
require.NoError(t, err)
defer func() {
err = f.Close()
assert.NoError(t, err)
}()

a := gradleLockAnalyzer{}
got, err := a.Analyze(nil, analyzer.AnalysisInput{
FilePath: tt.inputFile,
Content: f,
got, err := a.PostAnalyze(context.Background(), analyzer.PostAnalysisInput{
FS: os.DirFS(tt.dir),
})

assert.NoError(t, err)
Expand Down
Loading