Maven dependency resolution performance

[AB-xdev]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab

Please tell us more about your question or problem

I likely found 2 improvements regarding Maven dependency resolution for Maven Central (https://repo1.maven.org/maven2/).
I noticed these problems after further inspecting logs due to #27621 / #27759 and finding out that a small Maven project with ~30 dependencies sent around 3k request to maven central:

1. Renovate queries index.html to lookup Maven releases (e.g. https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/index.html), however Maven Central hosts no such file and the requests always fail.
2. I noticed that Renovate (without cache) sends extreme amounts of HEAD request to Maven central to detect/check all existing releases. However Artifacts on Maven Central are usually never revoked and if this should ever happen it will likely also be visible inside the maven-metadata.xml file (that is used for enumerating all dependency versions).

I recommend the following changes for https://repo1.maven.org/maven2/ and they likely should be ship by default with renovate:

1. Do not use index.html for lookup
2. Do not send HEAD requests for further validation of each dependency version

Logs (if relevant)

Logs

[2024-03-07T08:55:33.659Z] DEBUG: http statistics (repository=redacted)
[2024-03-07T08:55:33.659Z]        "urls": {
[2024-03-07T08:55:33.660Z]          "https://api.github.com/graphql (POST,200)": 7,
...
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.0/testcontainers-bom-1.10.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.1/testcontainers-bom-1.10.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.2/testcontainers-bom-1.10.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.3/testcontainers-bom-1.10.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.4/testcontainers-bom-1.10.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.5/testcontainers-bom-1.10.5.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.6/testcontainers-bom-1.10.6.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.10.7/testcontainers-bom-1.10.7.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.0/testcontainers-bom-1.11.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.1/testcontainers-bom-1.11.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.2/testcontainers-bom-1.11.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.3/testcontainers-bom-1.11.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.11.4/testcontainers-bom-1.11.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.0/testcontainers-bom-1.12.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.847Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.1/testcontainers-bom-1.12.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.2/testcontainers-bom-1.12.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.3/testcontainers-bom-1.12.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.4/testcontainers-bom-1.12.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.12.5/testcontainers-bom-1.12.5.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.13.0/testcontainers-bom-1.13.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.14.0/testcontainers-bom-1.14.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.14.1/testcontainers-bom-1.14.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.14.2/testcontainers-bom-1.14.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.14.3/testcontainers-bom-1.14.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.0-rc1/testcontainers-bom-1.15.0-rc1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.0-rc2/testcontainers-bom-1.15.0-rc2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.0/testcontainers-bom-1.15.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.1/testcontainers-bom-1.15.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.2/testcontainers-bom-1.15.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.15.3/testcontainers-bom-1.15.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.16.0/testcontainers-bom-1.16.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.16.1/testcontainers-bom-1.16.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.16.2/testcontainers-bom-1.16.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.16.3/testcontainers-bom-1.16.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.848Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.0/testcontainers-bom-1.17.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.1/testcontainers-bom-1.17.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.2/testcontainers-bom-1.17.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.3/testcontainers-bom-1.17.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.4/testcontainers-bom-1.17.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.5/testcontainers-bom-1.17.5.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.17.6/testcontainers-bom-1.17.6.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.18.0/testcontainers-bom-1.18.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.18.1/testcontainers-bom-1.18.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.18.2/testcontainers-bom-1.18.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.18.3/testcontainers-bom-1.18.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.0/testcontainers-bom-1.19.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.1/testcontainers-bom-1.19.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.2/testcontainers-bom-1.19.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.3/testcontainers-bom-1.19.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.4/testcontainers-bom-1.19.4.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.5/testcontainers-bom-1.19.5.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.6/testcontainers-bom-1.19.6.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.7/testcontainers-bom-1.19.7.pom (GET,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.19.7/testcontainers-bom-1.19.7.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.7.0/testcontainers-bom-1.7.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.7.1/testcontainers-bom-1.7.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.849Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.7.2/testcontainers-bom-1.7.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.7.3/testcontainers-bom-1.7.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.8.0/testcontainers-bom-1.8.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.8.1/testcontainers-bom-1.8.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.8.2/testcontainers-bom-1.8.2.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.8.3/testcontainers-bom-1.8.3.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.9.0/testcontainers-bom-1.9.0.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/1.9.1/testcontainers-bom-1.9.1.pom (HEAD,200)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/index.html (GET,404)": 1,
[2024-03-07T08:55:33.850Z]          "https://repo.maven.apache.org/maven2/org/testcontainers/testcontainers-bom/maven-metadata.xml (GET,200)": 1,
...
[2024-03-07T08:55:33.850Z]        },
[2024-03-07T08:55:33.850Z]        "hostStats": {
[2024-03-07T08:55:33.850Z]          "api.github.com": {"requestCount": 14, "requestAvgMs": 227, "queueAvgMs": 2},
[2024-03-07T08:55:33.850Z]          "auth.docker.io": {"requestCount": 4, "requestAvgMs": 886, "queueAvgMs": 4},
[2024-03-07T08:55:33.850Z]          "ghcr.io": {"requestCount": 7, "requestAvgMs": 219, "queueAvgMs": 26},
[2024-03-07T08:55:33.850Z]          "gitlab.xdev.cloud": {
[2024-03-07T08:55:33.850Z]            "requestCount": 7,
[2024-03-07T08:55:33.850Z]            "requestAvgMs": 757,
[2024-03-07T08:55:33.850Z]            "queueAvgMs": 4
[2024-03-07T08:55:33.850Z]          },
[2024-03-07T08:55:33.850Z]          "index.docker.io": {"requestCount": 8, "requestAvgMs": 551, "queueAvgMs": 8},
[2024-03-07T08:55:33.851Z]          "repo.maven.apache.org": {
[2024-03-07T08:55:33.851Z]            "requestCount": 3092,
[2024-03-07T08:55:33.851Z]            "requestAvgMs": 21,
[2024-03-07T08:55:33.851Z]            "queueAvgMs": 90
[2024-03-07T08:55:33.851Z]          }
[2024-03-07T08:55:33.851Z]        },

[AB-xdev]

Ok so I did a bit of further research and found out the following:

1.

There is an environment variable called RENOVATE_EXPERIMENTAL_NO_MAVEN_POM_CHECK that can be used to disable the HEAD request checks.

I enabled it, now my Renvoate needs 30s for dependency resolution instead of 90s and sends 100 requests to Maven Central instead of ~3k.

2.

I think that you should get completely rid of the HEAD request check:
Historically the HEAD request was introduced with #5614 (as "does artifact exist"-check).

This PR addresses #4471 and renovatebot/config-help#529 which are both cases where people are using a misconfigured Maven Repository where the Artifact Level maven-metadata.xml files are corrupted (likely someone deleted the artifacts manually and forgot to update the metadata file).

IMHO the solution here is to fix the broken maven-metadata.xml on the server and it's not Renovates job to workaround it by sending a ton of HEAD requests.

If anybody tries to tell's you "no it's not broken" tell them that they should go read the Maven Specs as it clearly describes:

The artifactId level metadata (found in directory of artifactId) serves purpose of version discovery. This metadata contains list of versions of given GA coordinates.

[rarkins]

Let's switch it to opt in for v38. Can you create the PR?

[zharinov]

However, before we turn it off, we could do great job of detecting all the broken metadata files we've encountered across Maven central

[rarkins]

Maybe we can build another crowdsourced list built into the repo so users can add to it

[zharinov]

I'll take a look how could we pre-populate it based on our current warm cache...

[rarkins]

I think it would be useful to print information about how many versions were found to be invalid, and then if we wish to build from that data then we could log them too so that we can pull out and deduplicate the list from the Mend backend to bootstrap this list.

[AB-xdev]

Update Nov 2024:

• The situation has slightly improved with f182fb3
• The general problems still persist. I created feat(datasource/maven): Introduce option to assume that looked up maven data is specification compliant #32651 which should address them.

[rarkins]

@AB-xdev thanks for your persistence in this. Could you start a new section below, or perhaps even a new discussion altogether summarizing what you think is the next step? I believe for example that the HEAD requests are no longer being applied to all versions, but does there remain any:

• significant inefficiency (e.g. of API requests), or
• inaccuracies/wrong results?

[AB-xdev]

@rarkins

@AB-xdev thanks for your persistence in this. Could you start a new section below, or perhaps even a new discussion altogether summarizing what you think is the next step? I believe for example that the HEAD requests are no longer being applied to all versions, but does there remain any:

• significant inefficiency (e.g. of API requests), or
• inaccuracies/wrong results?

Ok so when I read the code correctly the HEAD requests are no longer done 👍

However the latest maven release is still post processed and checked if it exists.
This request is not necessary and should be avoided.

Regarding "inaccuracies/wrong results":
There never were any. The messed up, non-specification compliant maven repositories are the problem here.

I think the best course of action here is to disable the postprocessRelease-check by default - as it's only required in a extremly small amount of cases (if any) - and provide an experimental flag that allows re-enabling it. This flag can then be removed at a later point.

[AB-xdev]

The changes could look like this: main...AB-xdev:renovate:maven-datasource-specification-compliant-v2

[rarkins]

However the latest maven release is still post processed and checked if it exists. This request is not necessary and should be avoided.

It not only checks that the release exists - but it also is used to determine the release timestamp for the release, which is important information for many users.

Additionally, this check is:

• only performed for versions which are contained in an update suggestion (i.e. not for every version of a package), and
• cached for 24 hours. We could extend this if we think that the release timestamp in particular very rarely changes. I'm not sure if Maven repositories are "immutable" when it comes to releases, like npmjs is meant to be?

Regarding "inaccuracies/wrong results": There never were any. The messed up, non-specification compliant maven repositories are the problem here.

Your two sentences contradict themselves. Yes, it's the repository's fault, and that causes the wrong results. But in reality it becomes Renovate's problem if it proposes updates to releases which can't be installed. And we received plenty of such examples in the early days of supporting Maven.

I think the best course of action here is to disable the postprocessRelease-check by default - as it's only required in a extremly small amount of cases (if any) - and provide an experimental flag that allows re-enabling it. This flag can then be removed at a later point.

1. Doing so loses release timestamps, so that's something which can't just be dropped
2. Experimental flag is not the best approach even if we did this. Experimental flags can be set only by bot administrators, whereas registry problems can be per-org, per-repo, or even per-dependency. Any such change must be repo-configurable, and non-experimental. Additionally, there's nothing experimental about this - it's existing functionality and would be retained for backwards compatibility.

I could live with a user-configurable (including in packageRules) new setting to opt out of this, for those who want less API requests and can sacrifice release timestamps. As a user, you would be able to opt out with a single line of config. The default settings or what others do should not be your concern.

[AB-xdev]

Thank you for the feedback.

used to determine the release timestamp

Ok I didn't have a look at that (was that always there?).

However:

• Maven has no release timestamps in the specification (However maven-metadata.xml has an lastUpdated field for all versions)
• You are currently getting the value using the last-modified-header. This is not specification compliant and may randomly change e.g. when servers/files are migrated. Be aware of that.
  • Update: There is already a warning in the docs 👍

I'm not sure if Maven repositories are "immutable" when it comes to releases

Maven releases are meant to be immutable: https://stackoverflow.com/a/61304251 (btw khmarbaise is a Maven dev ;) )
Maven Central definitely enforces this.

1. Doing so loses release timestamps, so that's something which can't just be dropped
2. Experimental flag is not the best approach even if we did this. Experimental flags can be set only by bot administrators, whereas registry problems can be per-org, per-repo, or even per-dependency. Any such change must be repo-configurable, and non-experimental. Additionally, there's nothing experimental about this - it's existing functionality and would be retained for backwards compatibility.

I could live with a user-configurable (including in packageRules) new setting to opt out of this, for those who want less API requests and can sacrifice release timestamps. As a user, you would be able to opt out with a single line of config. The default settings or what others do should not be your concern.

Having to agree here, these points make sense.

---

After reading this, I think the current situation is fine as it is.
There are still some additional requests but these aren't that many.
Also compared to the implementation effort required it will likely not pay off.

So from my perspective this is resolved and we can close this topic.