Add java.Matcher configuration to includes maven upstream sha1 query #714

spiffcs · 2022-04-08T18:56:03Z

Add Grype Matcher configuration for upstream maven query based on SHA1

Syft is now able to generate sha1 digests for discovered java archives. These digests are included along with the java archive metadata output.
anchore/syft#941

This PR adds the ability for the user to configure the java.Matcher to query the maven upstream repository for additional data that may not be present at time of SBOM generation.

Specifically, sboms of java archives are sometimes limited in their details when pom.xml or pom.properties are not present at the time of analysis. Querying maven given an archives sha1 digest value allows users to add fidelity to their vulnerability report when matching against data not relying on CPE generation (GHSA or GL Advisory Database).

It should be noted that a NEW package is not generated for the grype output. Matches found with this method are assigned directly to the java package that presented the sha1 for search. No duplication should be caused from doing this additional query.

The .grype.yaml configuration has been updated to include the following block:

external-sources:
  enable: true
  maven:
    search-maven-upstream: true
    base-url: "https://search.maven.org/solrsearch/select"

Features added

Matcher Config options at the app config level
Maven upstream package search with vulnerability comparison

TODO

Test Updates

Demo

Disabled - 606 vulnerabilities reported:

Enabled - 682 vulnerabilities reported:

Yardstick Analysis vs current release and branch:

This shows all of the additional vulnerabilities captured when enabling upstream matching:

None of these findings appear to be FP and can all be accurately attributed to the image.

What's interesting here is that after running yardstick the new branch shows 22 new vulnerabilities reported. The output of the respective commands (current release vs branch) shows a difference of 76 when comparing the Scanned image [xxx vulnerabilities] field (606 vs 682). This disparity has been found to be related to additional relatedVulnerabiities being populated as a result of enabling the upstream source. When searching by sha and matching with GHSA is also will generate a new relatedVulnerability for the NVD source. New vulnerabilities PLUS packages that can now be found via the GHSA and not just CPE account for this increase in the output.

I have found that the branch results will overwrite exact-direct-match with exact-indirect-match if a vulnerability is found upstream via the checksum search.

Signed-off-by: Christopher Phillips christopher.phillips@anchore.com

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs · 2022-04-11T19:30:12Z

go.mod

@@ -9,8 +9,8 @@ require (
 	github.com/anchore/go-testutils v0.0.0-20200925183923-d5f45b0d3c04
 	github.com/anchore/go-version v1.2.2-0.20210903204242-51efa5b487c4
 	github.com/anchore/packageurl-go v0.1.1-0.20220314153042-1bcd40e5206b
-	github.com/anchore/stereoscope v0.0.0-20220330165332-7fc73ee7b0f0
-	github.com/anchore/syft v0.43.0


Update on next syft release

kzantow

Only other thing I'll note at the moment is that we'll probably want to update the README with this one. Removing global state is MEGA-👍, though!

grype/pkg/java_metadata.go

internal/config/datasources.go

grype/lib.go

grype/matcher/java/matcher.go

grype/lib.go

grype/matcher/java/matcher.go

grype/matcher/matchers.go

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

wagoodman

great maven search addition, and love that this paves the way to exposing user-configuration for matchers in the future

wagoodman · 2022-04-12T19:46:43Z

internal/config/datasources.go

+
+func (cfg externalSources) loadDefaultValues(v *viper.Viper) {
+	v.SetDefault("external-sources.enable", false)
+	v.SetDefault("external-sources.maven.search-maven-upstream", false)


nit: since we have the master switch (which is great) we could have the maven search option default to on

I strongly +1 this

wagoodman · 2022-04-12T19:49:41Z

internal/config/datasources.go

+}
+
+type maven struct {
+	SearchMavenUpstream bool   `yaml:"search-maven-upstream" json:"search_maven_upstream" mapstructure:"search-maven-upstream"`


nit: the full config path stutters: external-sources.maven.search-maven-upstream with maven showing up twice.
maybe rename the field to search-upstream? Or more descriptive to the what or how of the operation, such as search-by-sha1 or find-artifact-group-id (not saying these suggestions are necessarily better, but wanted to give an idea)

Yea nice suggestion. I'll update the config here.

spiffcs · 2022-04-12T20:01:31Z

TODO before merge and thanks for the 🟢 @wagoodman and @kzantow :

config update to not stutter
default updates
test behavior correction
update syft release

kzantow · 2022-04-12T20:11:34Z

@spiffcs I'd like to see two more things:

make sure the appropriate environment variables work
make sure the README is updated

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs · 2022-04-13T15:51:55Z

Still debugging these env variables

spiffcs · 2022-04-13T15:56:42Z

@kzantow this should work for you GRYPE_EXTERNAL_SOURCES_ENABLE=true I think the env variables need to be prefixed with GRYPE

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs added 4 commits April 8, 2022 11:32

update java metadata decoding for syft-json

fe61325

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

add mavenSearch implementation for java.Matcher

95f7482

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

add matchUpstreamMavenPackages method for java.Match

17d4848

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

enable maven upstream matching

90fa26b

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs changed the title ~~update java metadata decoding for syft-json~~ Add grype java.Matcher configuration with ability to query upstream maven repository Apr 8, 2022

spiffcs changed the title ~~Add grype java.Matcher configuration with ability to query upstream maven repository~~ Add grype java.Matcher configuration (Includes maven upstream sha1 query) Apr 8, 2022

spiffcs and others added 3 commits April 8, 2022 15:24

bump syft to pull in merged metadata changes

f53be80

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

Merge branch 'main' into 711-External-Maven-Data-Source

a65af3b

sum updates and comment removal

b5c98f1

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs force-pushed the 711-External-Maven-Data-Source branch from 7fd3f09 to b5c98f1 Compare April 8, 2022 19:42

spiffcs linked an issue Apr 11, 2022 that may be closed by this pull request

Optional External Data Source Reference for Maven Packages #711

Closed

add user config options for upstream data

8a9913a

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs force-pushed the 711-External-Maven-Data-Source branch from 8323017 to 2884e42 Compare April 11, 2022 18:59

refactor matchers to be composed at the library level

c3b3276

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs force-pushed the 711-External-Maven-Data-Source branch from 2884e42 to c3b3276 Compare April 11, 2022 19:08

spiffcs marked this pull request as ready for review April 11, 2022 19:26

respect global feature configuration

8a3c573

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs requested a review from a team April 11, 2022 19:29

spiffcs commented Apr 11, 2022

View reviewed changes

kzantow reviewed Apr 11, 2022

View reviewed changes

spiffcs added 3 commits April 11, 2022 17:23

pr feedback

b16b6e0

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

update map lookup for package type

f10f346

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

stub out test for matcher

ab786ce

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs force-pushed the 711-External-Maven-Data-Source branch from 08bc8c6 to ab786ce Compare April 12, 2022 16:43

stub out maven searcher

c498a27

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

wagoodman approved these changes Apr 12, 2022

View reviewed changes

wagoodman reviewed Apr 12, 2022

View reviewed changes

complete test assertion for new matcher

5522623

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

update config name to not stutter

56bbeff

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs force-pushed the 711-External-Maven-Data-Source branch from dab92ea to 56bbeff Compare April 13, 2022 15:57

bump to latest syft

0e32cfc

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs changed the title ~~Add grype java.Matcher configuration (Includes maven upstream sha1 query)~~ Add java.Matcher configuration to includes maven upstream sha1 query Apr 13, 2022

spiffcs merged commit 95f68b4 into main Apr 13, 2022

spiffcs deleted the 711-External-Maven-Data-Source branch April 13, 2022 17:01

ibreakthecloud mentioned this pull request Apr 19, 2022

add external-sources to enhance java results deepfence/ThreatMapper#408

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add java.Matcher configuration to includes maven upstream sha1 query #714

Add java.Matcher configuration to includes maven upstream sha1 query #714

spiffcs commented Apr 8, 2022 •

edited

Loading

spiffcs Apr 11, 2022

kzantow left a comment

wagoodman left a comment

wagoodman Apr 12, 2022

kzantow Apr 12, 2022

wagoodman Apr 12, 2022 •

edited

Loading

spiffcs Apr 12, 2022

spiffcs commented Apr 12, 2022 •

edited

Loading

kzantow commented Apr 12, 2022

spiffcs commented Apr 13, 2022

spiffcs commented Apr 13, 2022 •

edited

Loading

Add java.Matcher configuration to includes maven upstream sha1 query #714

Add java.Matcher configuration to includes maven upstream sha1 query #714

Conversation

spiffcs commented Apr 8, 2022 • edited Loading

Add Grype Matcher configuration for upstream maven query based on SHA1

Features added

TODO

Demo

Yardstick Analysis vs current release and branch:

spiffcs Apr 11, 2022

Choose a reason for hiding this comment

kzantow left a comment

Choose a reason for hiding this comment

wagoodman left a comment

Choose a reason for hiding this comment

wagoodman Apr 12, 2022

Choose a reason for hiding this comment

kzantow Apr 12, 2022

Choose a reason for hiding this comment

wagoodman Apr 12, 2022 • edited Loading

Choose a reason for hiding this comment

spiffcs Apr 12, 2022

Choose a reason for hiding this comment

spiffcs commented Apr 12, 2022 • edited Loading

kzantow commented Apr 12, 2022

spiffcs commented Apr 13, 2022

spiffcs commented Apr 13, 2022 • edited Loading

spiffcs commented Apr 8, 2022 •

edited

Loading

wagoodman Apr 12, 2022 •

edited

Loading

spiffcs commented Apr 12, 2022 •

edited

Loading

spiffcs commented Apr 13, 2022 •

edited

Loading