Integration Test failures related to the pluginRepositories order

[dduportal]

Since jenkins-infra/jenkins-infra#3041 was deployed to ci.jenkins.io, the Integration Tests are failing with this kind of errors:

[INFO] Building: check-core-version-failure\pom.xml
[INFO] run post-build script verify.groovy
[INFO]   Assertion failed: 

assert new File(basedir, 'build.log').getText('UTF-8').contains("Dependency org.jenkins-ci.plugins:credentials:jar:1214.v1de940103927 requires Jenkins 2.375 or higher.")
       |        |                     |                |
       |        |                     |                false
       |        |                     '-XX:G1ConcRefinementThreads=4 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=134200512 -XX:MaxHeapSize=268435456 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:-UseLargePagesIndividualAllocation \r\n[INFO] Scanning for projects...\r\n[ERROR] [ERROR] Some problems were encountered while processing the POMs:\r\n[FATAL] Non-resolvable parent POM for org.jenkins-ci.plugins:check-core-version-failure:1.0-SNAPSHOT: The following artifacts could not be resolved: org.jenkins-ci.plugins:plugin:pom:4.61 (absent): org.jenkins-ci.plugins:plugin:pom:4.61 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced and \'parent.relativePath\' points at no local POM @ line 4, column 11\r\n @ \r\n[ERROR] The build could not read 1 project -> [Help 1]\r\n[ERROR]   \r\n[ERROR]   The project org.jenkins-ci.plugins:check-core-version-failure:1.0-SNAPSHOT (C:\\Jenkins\\agent\\workspace\\Plugins_maven-hpi-plugin_master\\target\\its\\check-core-version-failure\\pom.xml) has 1 error\r\n[ERROR]     Non-resolvable parent POM for org.jenkins-ci.plugins:check-core-version-failure:1.0-SNAPSHOT: The following artifacts could not be resolved: org.jenkins-ci.plugins:plugin:pom:4.61 (absent): org.jenkins-ci.plugins:plugin:pom:4.61 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced and \'parent.relativePath\' points at no local POM @ line 4, column 11 -> [Help 2]\r\n[ERROR] \r\n[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.\r\n[ERROR] Re-run Maven using the -X switch to enable full debug logging.\r\n[ERROR] \r\n[ERROR] For more information about the errors and possible solutions, please read the following articles:\r\n[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException\r\n[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException\r\nRunning post-build script: C:\\Jenkins\\agent\\workspace\\Plugins_maven-hpi-plugin_master\\target\\its\\check-core-version-failure\\verify.groovy\r\n'
       |        C:\Jenkins\agent\workspace\Plugins_maven-hpi-plugin_master\target\its\check-core-version-failure
       C:\Jenkins\agent\workspace\Plugins_maven-hpi-plugin_master\target\its\check-core-version-failure\build.log

[INFO]           check-core-version-failure\pom.xml ............... FAILED (3.801 s)

It was caught by @MarkEWaite in #529 and there has been a first tentative in #532.

As per #532 (comment) this issue must be fixed to avoid blocking jenkins-infra/helpdesk#3599

[dduportal]

A bit more of analysis:

The integration tests of this project are specifying a custom settings.xml in

maven-hpi-plugin/pom.xml

Line 242 in 043e1e0

<settingsFile>src/it/settings.xml</settingsFile>

which goal is to benefit from a local maven repository provided by the mrm-maven-plugin ("Mock Repository Manager" - https://www.mojohaus.org/mrm/mrm-maven-plugin/) Maven plugin.

My understanding is that it avoid re-downloading dependencies for each Integration Test execution.

This settings.xml is located in the repository:

maven-hpi-plugin/src/it/settings.xml

Lines 27 to 30 in de738f8

	<id>mrm-maven-plugin</id>
	<name>Mock Repository Manager</name>
	<url>@repository.proxy.url@</url>
	<mirrorOf>*</mirrorOf>

The idea I have is to simulate the behavior of the Jenkins Infra's settings.xml deployed in jenkins-infra/jenkins-infra#3041 with the IT's settings.xml file here.

[dduportal]

I commit to drive this to resolution as soon as posible as discussed in #532 (comment).

[basil]

For what it's worth, I asked Maven developer Tamás Cservenák to audit this plugin a few months back, and here is what he had to say:

I'd avoid the use of mrm plugin. You already pull Jetty, simplest (and most straightforward and least error prone) is to just fire up Jetty to serve files from somewhere as "remote repository" as in that case you are 100% in control of what is being served, while with MRM you are not.

So I don't think we're particularly attached to mrm-maven-plugin. In fact, maybe we could simply remove any and all references to mrm-maven-plugin and repository.proxy.url without any negative effects — this works fine for me locally, pulling about 696 MB of content into target/local-repo every time I run mvn clean verify -Prun-its. I can't see any particular reason to have mrm-maven-plugin enabled, and commit 6a2e160 only says "Tidy up invoker based tests" which is hardly informative. From my perspective this task could be completed by filing a PR with MRM removal and verifying the CI build behaves as expected — that the 696 MB of content is downloaded only once for each branch of the CI build (which I would expect to be the case, since it was the case for me locally) and that the 696 MB of content is downloaded from the artifact caching proxy if possible (although this might be a pre-existing bug which could be considered out of scope).

[dduportal]

For what it's worth, I asked Maven developer Tamás Cservenák to audit this plugin a few months back, and here is what he had to say:

I'd avoid the use of mrm plugin. You already pull Jetty, simplest (and most straightforward and least error prone) is to just fire up Jetty to serve files from somewhere as "remote repository" as in that case you are 100% in control of what is being served, while with MRM you are not.

So I don't think we're particularly attached to mrm-maven-plugin. In fact, maybe we could simply remove any and all references to mrm-maven-plugin and repository.proxy.url without any negative effects — this works fine for me locally, pulling about 696 MB of content into target/local-repo every time I run mvn clean verify -Prun-its. I can't see any particular reason to have mrm-maven-plugin enabled, and commit 6a2e160 only says "Tidy up invoker based tests" which is hardly informative. From my perspective this task could be completed by filing a PR with MRM removal and verifying the CI build behaves as expected — that the 696 MB of content is downloaded only once for each branch of the CI build (which I would expect to be the case, since it was the case for me locally) and that the 696 MB of content is downloaded from the artifact caching proxy if possible (although this might be a pre-existing bug which could be considered out of scope).

That would simplify the work required! Thanks for sharing this and the explanation (and pre-work!) @basil , it is really appreciated. I'm trying this angle then

[basil]

Based on https://www.mojohaus.org/mrm/mrm-maven-plugin/examples/invoker-tests.html it seems the reason for doing this was originally

to allow the integration tests of a Maven Plugin to be run against a test local repository thereby ensuring that the plugin does not get installed into the local repository until after the integration tests have been confirmed as passing

but I just ran mvn clean verify -Prun-its with mrm-maven-plugin removed and my local ~/.m2 did not seem to get polluted, so I don't think this is a concern anymore.

[dduportal]

Based on https://www.mojohaus.org/mrm/mrm-maven-plugin/examples/invoker-tests.html it seems the reason for doing this was originally

to allow the integration tests of a Maven Plugin to be run against a test local repository thereby ensuring that the plugin does not get installed into the local repository until after the integration tests have been confirmed as passing

but I just ran mvn clean verify -Prun-its with mrm-maven-plugin removed and my local ~/.m2 did not seem to get polluted, so I don't think this is a concern anymore.

Just got the same result \o/

[dduportal]

Sorry for the delay, I got unexpected tasks, still on it

[dduportal]

Oh, while trying the ci.jenkins.io's settings.xml locally to verify the behavior without mrm , I still got error:

[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.jenkins-ci.plugins:check-core-version-success:1.0-SNAPSHOT: The following artifacts could not be resolved: org.jenkins-ci.plugins:plugin:pom:4.61 (absent): Could not find artifact org.jenkins-ci.plugins:plugin:pom:4.61 in local.central (file:///Users/dadou/.m2/repository) and 'parent.relativePath' points at no local POM @ line 4, column 11
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.jenkins-ci.plugins:check-core-version-success:1.0-SNAPSHOT (/Users/dadou/workspace/jenkins-infra/maven-hpi-plugin/target/its/check-core-version-success/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM for org.jenkins-ci.plugins:check-core-version-success:1.0-SNAPSHOT: The following artifacts could not be resolved: org.jenkins-ci.plugins:plugin:pom:4.61 (absent): Could not find artifact org.jenkins-ci.plugins:plugin:pom:4.61 in local.central (file:///Users/dadou/.m2/repository) and 'parent.relativePath' points at no local POM @ line 4, column 11 -> [Help 2]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

I want to work it a little bit more before opening a PR