Publish plugin distribution ZIPs to Maven, get rid of checked in snapshot builds

[dblock]

Is your feature request related to a problem? Please describe

Coming from #20 (comment), publish distribution zips to maven.

Maven can be used for things other than java dependencies and is also actively used in such cases.

Describe the solution you'd like

• Publish plugin distribution ZIPs to Maven
• Publish Snapshot builds for plugin zips to Maven
• Get rid of checked in snapshot plugin zips

Dependencies with plugins

• [BUG] Checked in versions of job-scheduler-*-SNAPSHOT.zip  anomaly-detection#17
• [BUG] Replace checked in reports-scheduler/src/test/resources/job-scheduler/opensearch-job-scheduler-1.2.3.0-*.zip with a dynamic dependency reporting#263
• [BUG] Project integration tests require same version of plugins to be checked in index-management#123
• [BUG] Use snapshots from distribution/maven for dependencies  cross-cluster-replication#274

Describe alternatives you've considered

No response

Additional context

Some examples of others which also publish ZIPs on maven:

• Search Guard: https://docs.search-guard.com/latest/search-guard-versions
• Wildfly: https://repo1.maven.org/maven2/org/wildfly/wildfly-dist/24.0.1.Final/

Acceptance Criteria

• All the distribution zips for each plugin should be available in maven

[rursprung]

to reply to @dblock's comment from the original issue:

@rursprung Another suggestion, while still leveraging an authoritative link, how about scanning https://github.com/opensearch-project/project-website/tree/main/_artifacts for opensearch-* and then into the the version description?

opensearch-1.1.0-min-linux-x64.markdown:

---
role: daemon
artifact_id: opensearch-min
version: 1.1.0
platform: linux
architecture: x64
artifact_url: https://artifacts.opensearch.org/releases/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz
slug: opensearch-1.1.0-min-linux-x64
category: opensearch
type: targz
signature: https://artifacts.opensearch.org/releases/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz.sig
security_warning: true
indirect: true
---

this would require the build to connect to the "outside world" every time - using maven allows us to cache things locally with a well-established process (our internal maven server has the public maven servers as remotes and caches all packages downloaded from there).
if we have to scan for files that'd be a home-grown process which is then brittle and risks falling apart whenever something changes (e.g. the structure of the markdown file you linked - that's currently not a public API, if we'd scan it you'd have to treat it as such).

[dblock]

One more reason to do this is to (help) fix these bugs:

• [BUG] Checked in versions of job-scheduler-*-SNAPSHOT.zip  anomaly-detection#17
• [BUG] Replace checked in reports-scheduler/src/test/resources/job-scheduler/opensearch-job-scheduler-1.2.3.0-*.zip with a dynamic dependency reporting#263
• [BUG] Project integration tests require same version of plugins to be checked in index-management#123
• [BUG] Use snapshots from distribution/maven for dependencies  cross-cluster-replication#274

[dblock]

@rursprung if you or someone else wants to contribute this change, starting with snapshots, we'd take it given the proliferation of checked in zips; I think the work is to include the contents of plugins when publishing to maven in distribution-build.jenkinsfile

[dblock]

I'd like us to consider this for 1.3.0, labelled as such.

[gaiksaya]

Is publishing the core-plugins zips to maven in scope of this change?
Today we are publishing everything for them except the zip on maven https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/discovery-ec2/1.2.4-SNAPSHOT/

[dblock]

Is publishing the core-plugins zips to maven in scope of this change? Today we are publishing everything for them except the zip on maven https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/discovery-ec2/1.2.4-SNAPSHOT/

I would imagine that yes.

[peterzhuamazon]

[Grooming] Need to get more information on the scenario that we need this from Maven.
Is the maven dependency a required one for BWC tests?
Or the existing cloudfront in front of S3 is enough?

@VachaShah @adnapibar

[dblock]

[Grooming] Need to get more information on the scenario that we need this from Maven. Is the maven dependency a required one for BWC tests? Or the existing cloudfront in front of S3 is enough?

@VachaShah @adnapibar

The issue is to publish zips to maven so that maven can be used for dependency management.

[bbarani]

Can we not use artifacts.opensearch.org for dependency management? BWC tests should be able to use the artifacts directly from artifacts.opensearch.org.

[dblock]

Can we not use artifacts.opensearch.org for dependency management? BWC tests should be able to use the artifacts directly from artifacts.opensearch.org.

It would ideally be a maven repo, which is a system with a bunch of well designed dependency management tools. We could get away with artifacts.opensearch.org, too, but we've been asked for maven explicitly in a few issues linked above. We're reinventing the wheel and using different tools for the same purpose.

[jmazanec15]

@gaiksaya @peterzhuamazon I think core-plugins zips would be helpful. I want to be able to run integration tests for k-NN on Mac. The INTEG_TEST testDistibution only ships with the transport modules (ARCHIVE does not work on mac).

I need the lang-painless module because it needs to be installed on the cluster for k-NN to work. I can pick up the artifacts for lang-painless from https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/ but I am unable to install the module on the integ test cluster. I tried zipping the lang-painless artifacts up in a custom gradle task, but it still fails to install due to a missing plugin-descriptors file, which is not included in the artifacts link above.

It would be nice to be able to declare dependencies on core plugins and easily be able to install them on the integ test clusters. I am not sure if there is a better workaround to this (if so, please let me know). The TestClusters plugin already has the ability to add modules/plugins from zips.

More details in OpenSearch issue for what I have tried in particular: opensearch-project/OpenSearch#1676 (comment).

[ylwu-amzn]

#1689, as we support latest artifact URL, we will try latest link first, like https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.1/latest/linux/x64/builds/opensearch/plugins/opensearch-job-scheduler-1.3.1.0.zip, https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.1/latest/linux/x64/builds/opensearch/plugins/opensearch-ml-1.3.1.0.zip ,

@amitgalitz will work on removing job-scheduler local zip in AD, @jackiehanyang Can you take a look for moving local ml-commons to this latest URL in SQL plugin.

[prudhvigodithi]

I can see with current setup that publishes maven jars to snapshot repo has an added date suffix format example 2.0.0.0-alpha1-SNAPSHOT/common-utils-2.0.0.0-alpha1-20220331.205721-1-javadoc.jar, (https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch/2.0.0-alpha1-SNAPSHOT/) in order to upload the generated plugin zips under builds/opensearch/plugins, we can directly publish them or go with same like adding date suffix before we publish.

The publish location can be as follows:

plugin zips for snapshot type to https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/<plugin_name>/<2.0.0.0-alpha1-SNAPSHOT>/.zip

I already see this path https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin for core plugins, we can leverage this path or they can go under https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch//

The main releases plugin zips, once finalized to releases repo, once the path is finalized
https://aws.oss.sonatype.org/content/repositories/releases/
@dblock @bbarani

[dblock]

@prudhvigodithi Leave that decision to maven. Maven adds these suffixes to allow multiple -SNAPSHOT versions to be uploaded, so for .zips I would do the exact same thing as for other artifacts such as JARs. For the released version of the artifact I imagine the suffix is dropped.

[prudhvigodithi]

Hey @dblock
for jars the date suffix as (20220331)

   - maven/org/opensearch/plugin/percolator-client/2.0.0-alpha1-SNAPSHOT/percolator-client-2.0.0-alpha1-20220331.205538-1.jar.md5
   - maven/org/opensearch/plugin/percolator-client/2.0.0-alpha1-SNAPSHOT/percolator-client-2.0.0-alpha1-20220331.205538-1.jar.sha512
   - maven/org/opensearch/plugin/opensearch-scripting-painless-spi/2.0.0-alpha1-SNAPSHOT/opensearch-scripting-painless-spi-2.0.0-alpha1-20220331.205538-1.pom
   - maven/org/opensearch/plugin/opensearch-scripting-painless-spi/2.0.0-alpha1-SNAPSHOT/maven-metadata.xml.sha256
   - maven/org/opensearch/plugin/opensearch-scripting-painless-spi/2.0.0-alpha1-SNAPSHOT/opensearch-scripting-painless-spi-2.0.0-alpha1-20220331.205538-1-javadoc.jar

but for plugin zips this date is not added (currently)Example as

 opensearch-job-scheduler-2.0.0.0-alpha1-SNAPSHOT.zip
 opensearch-ml-2.0.0.0-alpha1-SNAPSHOT.zip
 opensearch-knn-2.0.0.0-alpha1-SNAPSHOT.zip

So when we upload the zips (mvn deploy )after every distribution build under snapshot repo, with each build the zip files will be overridden as they constantly generate with same suffix as -2.0.0.0-alpha1-SNAPSHOT.zip considering above example. But if we are ok to always pick the latest build zips like under 2.0.0.0-alpha1-SNAPSHOT folder, then it should be good to be like opensearch-job-scheduler-2.0.0.0-alpha1-SNAPSHOT.zip (as example).

[dblock]

I am not super familiar with how it works, but I would want to take a maven dependency on opensearch-ml 2.0.0.0-alpha1-SNAPSHOT and get the latest. File names seems like an implementation detail.

[prudhvigodithi]

Proposed Solution

Almost inclined with the same setup on how the current jars are published to maven repo, but the key difference is instead of creating a directory(as maven coordinates and then deploy) we allow maven to do it, by creating local file system as temp maven repo.

Example:

mvn install:install-file -DgroupId=org.opensearch.plugin -DartifactId=pgplugin -Dversion=0.0.1 -Dpackaging=zip -Dfile=package.zip -DlocalRepositoryPath=../loacalrepo/

With the above cli command now we have folder created as maven coordinates
loacalrepo/org/opensearch/plugin/pgplugin/0.0.1

This will have pom file and zip file stored in loacalrepo/org/opensearch/plugin/pgplugin/0.0.1

Snapshot zips:

Upload the snapshot plugin zips to https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/
<plugin_name>/-SNAPSHOT/<.zip_file>
Edit the plan https://github.com/prudhvigodithi/opensearch-build/blob/slack-publish-v1/jenkins/opensearch/distribution-build.jenkinsfile#L82 to upload zips to nexus maven repo

settings.xml

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      https://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <server>
      <id>nexus</id>
      <username>${SONATYPE_USERNAME}</username>
      <password>${SONATYPE_PASSWORD}</password>
    </server>
  </servers>
</settings>

mvn --settings="${mvn_settings}" deploy:deploy-file -DgeneratePom=false -DrepositoryId=nexus -DpomFile=loacalrepo/org/opensearch/plugin/pgplugin/0.0.1/pgplugin-0.0.1.pom -Dfile=loacalrepo/org/opensearch/plugin/pgplugin/0.0.1/pgplugin-0.0.1.zip -Durl=$repo_url

Release zips:
The Release candidates will be signed before staging to make sure files are unaltered.

settings.xml

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      https://maven.apache.org/xsd/settings-1.0.0.xsd">
                    
  <servers>
    <server>
      <id>nexus</id>
      <username>${SONATYPE_USERNAME}</username>
      <password>${SONATYPE_PASSWORD}</password>
    </server>
  </servers>
</settings>

Create a staging repo nexus-staging:rc-open this will create a new new staging repository on the remote Nexus
Upload zips from the local filesystem to a staging repository using plugin in built deploy-staged-repository
Close the created staging repo nexus-staging:rc-close

staging_repo_id = $(mvn --settings="${mvn_settings}" \
    org.sonatype.plugins:nexus-staging-maven-plugin:rc-open \
    -DnexusUrl=$REPO_URL \
    -DserverId=nexus \
    -DstagingProfileId="${STAGING_PROFILE_ID}" \
    -DstagingDescription="Staging artifacts for build ${BUILD_ID}" \
    -DopenedRepositoryMessageFormat="opensearch-staging-repo-id=%s" |
    grep -E -o 'opensearch-staging-repo-id=.*$' | cut -d'=' -f2)
    

mvn --settings="${mvn_settings}" \
  org.sonatype.plugins:nexus-staging-maven-plugin:1.6.5:deploy-staged-repository \
  -DnexusUrl="$REPO_URL \
  -DrepositoryDirectory="${staged_repo}"
  -DautoReleaseAfterClose=false \
  -DstagingProgressTimeoutMinutes=30 \
  -DstagingProfileId="${STAGING_PROFILE_ID}" \
  -DstagingRepositoryId="${staging_repo_id}" 

staged_repo=loacalrepo (file system folder)
REPO_URL = "https://aws.oss.sonatype.org/"
STAGING_PROFILE_ID = "${SONATYPE_STAGING_PROFILE_ID}"
staging_repo_id from above mvn cli staging command

@dblock

[rursprung]

i just re-read this ticket to see whether it'll land in 2.0 and have some comments to add:

• first of all: thanks for working on this! sorry for not having helped out here, somehow time is running through my fingers and i never got around to it :(
• i mainly (only?) see comments about publishing plugin ZIPs to maven, but please note that my request was also explicitly for the final distributions (i.e. the ones listed for download here: https://opensearch.org/downloads.html) as well
• IMHO it'd make sense to put this on the distributions roadmap: https://github.com/orgs/opensearch-project/projects/13
• is the maven repository to which you plan to publish hooked up to maven central & co. or will consumers have to add yours as an explicit remote? since other opensearch artifacts can be found here i think this will be fine as long as you publish on the same maven server: https://mvnrepository.com/artifact/org.opensearch

[prudhvigodithi]

Hey @rursprung
The target for plugins to be part of the maven is to allow download of zip's by dependency model, off course once they are in maven they can even be downloaded using HTTP/S methods to run the required tests or to install the plugin in the cluster.
Distributions (the tar format files) are good to be part of maven, but I don't see a strong need to download them using dependency model, I mean by using maven coordinates (Please correct me here if i'm wrong). They just need to download using a HTTP method and install them for desired purpose.

Coming to the Roadmap, yes I agree it would make sense to add to the roadmap. Right now the plugin creation part is done that does the zip publications to maven, the target is to add this to each existing plugin repo, that follow certain guidelines (PR for document is under review) and make sure the zips are added to maven.

The maven for releases should be central maven along with nexus.

@dblock @bbarani any thoughts to add ?

[rursprung]

Distributions (the tar format files) are good to be part of maven, but I don't see a strong need to download them using dependency model, I mean by using maven coordinates (Please correct me here if i'm wrong). They just need to download using a HTTP method and install them for desired purpose.

as i had already mentioned in #20 (comment) and #20 (comment) (and i think also various other places) we'd actually want to pull in the opensearch distribution also via maven as it brings a ton of benefits (caching amongst them).

[prudhvigodithi]

from the comment, you mean the TAR packaged files?
we could also ship them along with plugin zip's based on further discussion 👍
(But 1st lets target the plugin zips and we can add TAR packaged files to this META)

[rursprung]

from the comment, you mean the TAR packaged files?

you're right, i mean the .tar.gz files, e.g. this one here: https://artifacts.opensearch.org/releases/core/opensearch/1.3.1/opensearch-min-1.3.1-linux-x64.tar.gz
once you have windows distributions i guess that would be a ZIP, though i personally don't care about that one (then again, for consistency reasons that one should also be published)

But 1st lets target the plugin zips and we can add TAR packaged files to this META

let's do that! will you edit the ticket description or should i add a comment there?

[dblock]

@rursprung @prudhvigodithi Looks like current issues talk about publishing "plugin zips". After @prudhvigodithi is done with that all those will get closed rather than expanding their scope. I think we need two more related issues: one to publish the min distribution (e.g. https://artifacts.opensearch.org/releases/core/opensearch/1.3.1/opensearch-min-1.3.1-linux-x64.tar.gz) to maven, and another to publish the distribution zip (e.g. https://artifacts.opensearch.org/releases/bundle/opensearch/1.3.1/opensearch-1.3.1-linux-x64.tar.gz) to maven, in this repo. Note that there are cons to doing that because it effectively creates 2 different places where we publish signed binaries. If you agree, care to open them @rursprung?

[rursprung]

i was going to raise a new ticket, but then i realised again that the title of this ticket here literally is "publish distribution ZIPs to Maven", so based on that i think it should cover it? and you raised this ticket here out of our discussion over on #20 (comment)

i'm fine with raising a new ticket (or even two, as you suggested), just wondering on how to word it so that we can ensure that we then are talking about the same thing and don't get a scope change again. because so far i fully expected this ticket here to contain the distributions (min & full) and the plugins just as a (very welcome & also needed!) side-effect ;)

[dblock]

@rursprung yeah let's not assume side effects :) you're right though it was all zips in this issue - I decided to scope this down and rename this issue to one to only focus on publishing plugin zips - care to open 2 more for the other kind of artifacts, one for each?

[rursprung]

done: #2068 & #2069, i hope they contain all relevant information.

PS: that was a sneaky change of the title on this ticket ;)

[dblock]

done: #2068 & #2069, i hope they contain all relevant information.

PS: that was a sneaky change of the title on this ticket ;)

I "clarified" and "simplified" it :) thank you!

[prudhvigodithi]

Hey @dblock we have now, a mechanism to publish the zips to maven repo using opensearch.pluginzip, can we close this issue ?
@bbarani

[dblock]

This issue is about leveraging this mechanism. At least these two need to be resolved, and we need to check that we're not doing something similar elsewhere before closing this.

• [BUG] Project integration tests require same version of plugins to be checked in index-management#123
• [BUG] Use snapshots from distribution/maven for dependencies  cross-cluster-replication#274

[prudhvigodithi]

Hey Starting with release 2.1.0, we now have plugin zips in maven repo
#1916 (comment)

Pending items related to this issue before it can be closed:
opensearch-project/index-management#123
opensearch-project/cross-cluster-replication#274

@dblock @bbarani

Thank you

[prudhvigodithi]

Closing this issue as associated tasks are marked as complete, please feel free to re-open this issue if required. :)
Thank you