fix(java): update logic to detect `pom.xml` file snapshot artifacts from remote repositories #6412

DmitriyLewen · 2024-03-28T11:45:43Z

Description

Maven splits release and snapshot repositories - https://github.com/apache/maven/blob/259404701402230299fe05ee889ecdf1c9dae816/maven-core/src/main/java/org/apache/maven/project/MavenProject.java#L144-L146.
Maven uses snapshot remote repos for snapshot artifacts¹ and release remote repos + maven central for other artifacts:

We need reproduce this logic:

snapshot artifact
- check snapshot repositories from pom file (metadata.xml -> *.pom)
other artifacts
- check releases repositories from pom file (only *.pom)
- check maven central

Tests with mvn and trivy:

snapshot version + snapshot repo enable

test pom.xml file:

...
<dependencies>
  <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.17.0-SNAPSHOT</version>
  </dependency>      
</dependencies>

<repositories>
  <repository>
    <id>sonatype-nexus-snapshots</id>
    <name>Sonatype Nexus Snapshots</name>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    <releases><enabled>false</enabled></releases>
    <snapshots><enabled>true</enabled></snapshots>
  </repository>
</repositories>
...

mvn (jackson-annotations is child dependency of jackson-databind):

➜  6355 mvn dependency:resolve | grep jackson-annotations
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/maven-metadata.xml
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/maven-metadata.xml (1.2 kB at 7.3 kB/s)
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.pom
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.pom (7.0 kB at 42 kB/s)
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.jar
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.jar (79 kB at 454 kB/s)
[INFO]    com.fasterxml.jackson.core:jackson-annotations:jar:2.17.0-SNAPSHOT:compile -- module com.fasterxml.jackson.annotation

trivy (need add logic for metadata.xml):

➜ ./trivy -d fs pom.xml | grep jackson-databind
...
2024-03-29T09:32:49.948+0600	DEBUG	Resolving com.fasterxml.jackson.core:jackson-databind:2.17.0-SNAPSHOT...
2024-03-29T09:32:53.265+0600	DEBUG	Failed to fetch from oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-databind/2.17.0-SNAPSHOT/jackson-databind-2.17.0-SNAPSHOT.pom

snapshot version (maven central contains this version) + repositories disable

!!! This test shows that maven doesn't check central for snapshots.

test pom.xml file (https://repo.maven.apache.org/maven2/ai/kassette/kassette-java-sdk/1.0.4-SNAPSHOT/kassette-java-sdk-1.0.4-SNAPSHOT.pom):

...
<dependencies>
  <dependency>
    <groupId>ai.kassette</groupId>
    <artifactId>kassette-java-sdk</artifactId>
    <version>1.0.4-SNAPSHOT</version>
  </dependency>      
</dependencies>
...

mvn:

➜  6355 mvn dependency:resolve                           
[ERROR] Failed to execute goal on project pom: Could not resolve dependencies for project root:pom:jar:1.0.0: The following artifacts could not be resolved: ai.kassette:kassette-java-sdk:jar:1.0.4-SNAPSHOT (absent): Could not find artifact ai.kassette:kassette-java-sdk:jar:1.0.4-SNAPSHOT -> [Help 1]

trivy:

➜ ./trivy -d fs pom.xml                        
2024-03-29T09:42:57.580+0600	DEBUG	Resolving ai.kassette:kassette-java-sdk:1.0.4-SNAPSHOT...
2024-03-29T09:42:57.580+0600	DEBUG	ai.kassette:kassette-java-sdk:1.0.4-SNAPSHOT was not found in local/remote repositories

release version + release repository enable

!!! This test shows that maven checks repositories from pom file before maven central

test pom.xml file:

...
<dependencies>
  <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.17.0</version>
  </dependency>      
</dependencies>

<repositories>
  <repository>
    <id>sonatype-nexus-snapshots</id>
    <name>Sonatype Nexus Snapshots</name>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    <releases><enabled>true</enabled></releases>
    <snapshots><enabled>false</enabled></snapshots>
  </repository>
</repositories>

...

mvn:

➜ mvn dependency:resolve
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-databind/2.17.0/jackson-databind-2.17.0.pom
Downloading from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.17.0/jackson-databind-2.17.0.pom
Downloaded from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.17.0/jackson-databind-2.17.0.pom (21 kB at 52 kB/s)

trivy:

➜ ./trivy -d fs pom.xml -f json --list-all-pkgs | grep jackson-annotations 
...
2024-03-29T10:10:24.421+0600	DEBUG	Resolving com.fasterxml.jackson.core:jackson-databind:2.17.0...
2024-03-29T10:10:25.149+0600	DEBUG	Failed to fetch from oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-databind/2.17.0/jackson-databind-2.17.0.pom
...
        "ID": "com.fasterxml.jackson.core:jackson-annotations:2.17.0",

Changes for another PR:

Maven uses metadata.xml file to find *.pom file for snapshots.
e.g. (https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/):

Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/maven-metadata.xml
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/maven-metadata.xml (1.2 kB at 171 B/s)
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.pom
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.pom (7.0 kB at 33 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.17/jackson-parent-2.17.pom
Downloaded from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.17/jackson-parent-2.17.pom (6.5 kB at 22 kB/s)

Related issues

Close 0.50.0 : big time increase of FS scanning vs 0.49.1 #6374

Related PRs

feat(java): add support for fetching packages from repos mentioned in pom.xml #6171

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

Maven detects snapshot artifacts by artifact version - https://github.com/apache/maven/blob/259404701402230299fe05ee889ecdf1c9dae816/maven-artifact/src/main/java/org/apache/maven/artifact/DefaultArtifact.java#L482-L486 ↩

- divide remote repositories into releases and snapshots - detect snapshots from the artifact version

knqyf263 · 2024-04-16T06:33:38Z

This change looks awesome! I have one question.

snapshot artifact
check snapshot repositories from pom file (metadata.xml -> *.pom)

What is metadata.xml? I didn't find it tried to extract repositories from metadata.xml. I might be missing something.

DmitriyLewen · 2024-04-16T06:45:10Z

IIUC metadata.xml file contains info about latest versions of jar, pom, etc. for snapshot version.
e.g. https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/maven-metadata.xml

Take a look this mvn log:

Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/maven-metadata.xml
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/maven-metadata.xml (1.2 kB at 171 B/s)
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.pom
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.pom (7.0 kB at 33 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.17/jackson-parent-2.17.pom
Downloaded from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.17/jackson-parent-2.17.pom (6.5 kB at 22 kB/s)

mvn checks https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/maven-metadata.xml -> takes version for pom (<value>2.17.0-20240312.035235-10</value>) -> checks url with this version (https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-20240312.035235-10.pom)

knqyf263 · 2024-04-16T07:31:26Z

Thanks for explaining. I got it. So, it sounds like we also need to fetch metadata.xml to identify the latest version. I'm wondering if our current logic works.

trivy/pkg/dependency/parser/java/pom/parse.go

Lines 651 to 660 in 6c19ae7

    
           // try all remoteRepositories 
        
           for _, repo := range remoteRepos { 
        
           	fetched, err := p.fetchPOMFromRemoteRepository(repo, paths) 
        
           	if err != nil { 
        
           		return nil, xerrors.Errorf("fetch repository error: %w", err) 
        
           	} else if fetched == nil { 
        
           		continue 
        
           	} 
        
           	return fetched, nil 
        
           }

DmitriyLewen · 2024-04-16T07:38:21Z

I don't have much experience with custom repositories.
But I think that if repository uses structure as maven central (without metadata.xml) (if this is possible) - our logic should work.

it sounds like we also need to fetch metadata.xml to identify the latest version

I thought I'd add this to another PR. But I can do it in this PR.

knqyf263 · 2024-04-16T07:41:14Z

But I think that if repository uses structure as maven central (without metadata.xml) (if this is possible) - our logic should work.

I asked about snapshot repositories since the structure seems different from that of release repositories. We just replace URLs here even for snapshots. But it's ok if you work on it in another PR.

knqyf263 · 2024-04-16T07:48:15Z

When the artifact is org.example:example-api:2.0.0-SNAPSHOT, the current logic tries to access https://oss.sonatype.org/content/repositories/snapshots/org/example/example-api/2.0.0-SNAPSHOT/example-api-2.0.0-SNAPSHOT.pom, right? But it should be https://oss.sonatype.org/content/repositories/snapshots/org/example/example-api/2.0.0-SNAPSHOT/maven-metadata.xml. Am I correct?

knqyf263 · 2024-04-16T07:49:48Z

Anyway, it's not a bug from this PR. I'll merge this one, and we'll implement the snapshot logic later.

DmitriyLewen · 2024-04-16T07:51:19Z

When the artifact is org.example:example-api:2.0.0-SNAPSHOT, the current logic tries to access https://oss.sonatype.org/content/repositories/snapshots/org/example/example-api/2.0.0-SNAPSHOT/example-api-2.0.0-SNAPSHOT.pom, right? But it should be https://oss.sonatype.org/content/repositories/snapshots/org/example/example-api/2.0.0-SNAPSHOT/maven-metadata.xml. Am I correct?

mvn first checks https://oss.sonatype.org/content/repositories/snapshots/org/example/example-api/2.0.0-SNAPSHOT/maven-metadata.xml
But if this file doesn't exist - mvn will try to check https://oss.sonatype.org/content/repositories/snapshots/org/example/example-api/2.0.0-SNAPSHOT/example-api-2.0.0-SNAPSHOT.pom

UPD:
example (using incorrect version - jackson-databind-2.17.012341-SNAPSHOT):

Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-databind/2.17.012341-SNAPSHOT/maven-metadata.xml
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-databind/2.17.012341-SNAPSHOT/jackson-databind-2.17.012341-SNAPSHOT.pom
[WARNING] The POM for com.fasterxml.jackson.core:jackson-databind:jar:2.17.012341-SNAPSHOT is missing, no dependency information available
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-databind/2.17.012341-SNAPSHOT/jackson-databind-2.17.012341-SNAPSHOT.jar

knqyf263 · 2024-04-16T07:55:47Z

So, Trivy currently accesses the following URL and fails as it's 404.
https://oss.sonatype.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-annotations/2.17.0-SNAPSHOT/jackson-annotations-2.17.0-SNAPSHOT.pom

In short, Trivy tries snapshot repositories and most likely fails because Trivy doesn't support maven-metadata.xml. Please correct me if I'm wrong. I'm just trying to understand the current situation.

DmitriyLewen · 2024-04-16T07:59:11Z

In short, Trivy tries snapshot repositories and most likely fails because Trivy doesn't support maven-metadata.xml

if the repository uses maven-metadata.xml files, you are right.

thus oss doesn't currently support

knqyf263 · 2024-04-16T08:17:53Z

I randomly picked up some artifacts in the snapshot repository. All of them use maven-metadata.xml.
https://oss.sonatype.org/content/repositories/snapshots/com/zendesk/jazon/jazon-core/maven-metadata.xml

Is it possible that a snapshot repository doesn't use maven-metadata.xml?

DmitriyLewen · 2024-04-16T08:24:05Z

as i wrote in #6412 (comment):
mvn checks <artifactid>-<version>.pom file if maven-metadata.xml file doesn't exist - so i think this is possible, but i didn't see these repositories 😄

…rom remote repositories (#6412)

…rom remote repositories (aquasecurity#6412)

refactor(pom): update logic for snapshots:

31255ea

- divide remote repositories into releases and snapshots - detect snapshots from the artifact version

DmitriyLewen self-assigned this Mar 28, 2024

DmitriyLewen mentioned this pull request Mar 28, 2024

fix(java): add flag for java scan options #6375

Closed

7 tasks

DmitriyLewen added 4 commits March 28, 2024 18:32

test: add test for snapshot artifact

6e78b40

refactor

3214cff

fix typo

d2bd515

docs: update java coverage page

cbae196

DmitriyLewen marked this pull request as ready for review March 29, 2024 07:21

DmitriyLewen requested a review from knqyf263 as a code owner March 29, 2024 07:21

DmitriyLewen added 2 commits April 16, 2024 12:53

Merge branch 'main' into 'fix-pom/snapshot-and-releases-repos)'

1d74ab2

refactor: fix typo

6c19ae7

knqyf263 approved these changes Apr 16, 2024

View reviewed changes

knqyf263 added this pull request to the merge queue Apr 16, 2024

Merged via the queue into aquasecurity:main with commit 34ab09d Apr 16, 2024
17 checks passed

DmitriyLewen deleted the fix-pom/snapshot-and-releases-repos branch April 16, 2024 08:11

DmitriyLewen mentioned this pull request Apr 17, 2024

feat(pom.xml): parse maven-metadata.xml files for snapshot repositories. #6508

Closed

knqyf263 pushed a commit that referenced this pull request Apr 22, 2024

fix(java): update logic to detect pom.xml file snapshot artifacts f…

2ad8e33

…rom remote repositories (#6412)

DmitriyLewen mentioned this pull request Apr 23, 2024

0.50.0 : big time increase of FS scanning vs 0.49.1 #6374

Closed

fl0pp5 pushed a commit to altlinux/trivy that referenced this pull request May 6, 2024

fix(java): update logic to detect pom.xml file snapshot artifacts f…

5643473

…rom remote repositories (aquasecurity#6412)

aqua-bot mentioned this pull request May 29, 2024

release: v0.52.0 [main] #6809

Merged

DmitriyLewen mentioned this pull request Jun 17, 2024

feat(java): add support for maven-metadata.xml files for remote snapshot repositories. #6950

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(java): update logic to detect `pom.xml` file snapshot artifacts from remote repositories #6412

fix(java): update logic to detect `pom.xml` file snapshot artifacts from remote repositories #6412

DmitriyLewen commented Mar 28, 2024 •

edited

Loading

knqyf263 commented Apr 16, 2024

DmitriyLewen commented Apr 16, 2024 •

edited

Loading

knqyf263 commented Apr 16, 2024 •

edited

Loading

DmitriyLewen commented Apr 16, 2024

knqyf263 commented Apr 16, 2024 •

edited

Loading

knqyf263 commented Apr 16, 2024

knqyf263 commented Apr 16, 2024

DmitriyLewen commented Apr 16, 2024 •

edited

Loading

knqyf263 commented Apr 16, 2024 •

edited

Loading

DmitriyLewen commented Apr 16, 2024

knqyf263 commented Apr 16, 2024

DmitriyLewen commented Apr 16, 2024

fix(java): update logic to detect pom.xml file snapshot artifacts from remote repositories #6412

fix(java): update logic to detect pom.xml file snapshot artifacts from remote repositories #6412

Conversation

DmitriyLewen commented Mar 28, 2024 • edited Loading

Description

Changes for another PR:

Related issues

Related PRs

Checklist

Footnotes

knqyf263 commented Apr 16, 2024

DmitriyLewen commented Apr 16, 2024 • edited Loading

knqyf263 commented Apr 16, 2024 • edited Loading

DmitriyLewen commented Apr 16, 2024

knqyf263 commented Apr 16, 2024 • edited Loading

knqyf263 commented Apr 16, 2024

knqyf263 commented Apr 16, 2024

DmitriyLewen commented Apr 16, 2024 • edited Loading

knqyf263 commented Apr 16, 2024 • edited Loading

DmitriyLewen commented Apr 16, 2024

knqyf263 commented Apr 16, 2024

DmitriyLewen commented Apr 16, 2024

fix(java): update logic to detect `pom.xml` file snapshot artifacts from remote repositories #6412

fix(java): update logic to detect `pom.xml` file snapshot artifacts from remote repositories #6412

DmitriyLewen commented Mar 28, 2024 •

edited

Loading

DmitriyLewen commented Apr 16, 2024 •

edited

Loading

knqyf263 commented Apr 16, 2024 •

edited

Loading

knqyf263 commented Apr 16, 2024 •

edited

Loading

DmitriyLewen commented Apr 16, 2024 •

edited

Loading

knqyf263 commented Apr 16, 2024 •

edited

Loading