Publish snapshots and release artifacts to maven

[dblock]

Currently plugins check out and build OpenSearch in their CI. They should be consuming dependent JARs published into a maven repository that hosts nightly / SNAPSHOT builds and release artifacts.

• Either sonatype or self-hosted Maven on top of S3 with credentials managed by infra
• Nightly or even after each check-in builds of OpenSearch, plugins, etc., for a list of well known branches
• All artifacts for published beta, RCs and releases
• README explains how to onboard any project to build and publish nightlies into Maven (ticket? issue?)
• README explains how to add a branch to be built and published
• Visibility into the builds - #102

[dblock]

cc: @camerski since I can't assign it

[mch2]

Would like to take a stab at this. We have an issue to do this for lucene as well here. As a first step, am thinking of putting together a small cdk project to create the resources. Will then look at updating infra to publish a nightly.

[camerski]

@mch2 has been prototyping some options, and it looks like we have a path forward for automatically publishing everything through Sonatype with no human intervention needed. We probably don't need a self-hosted repository.

[mch2]

Updated with a simple configuration to push to our Sonatype repo with ./gradlew publish. I tested this with a locally running instance of nexus.

I do not think we need [#41] anymore, closing that out.

After this is added to CI, I am thinking we can extend this with a custom script in opensearch-build that builds/publishes each repo we would like to push.

[mch2]

We did a successful test push here of only the opensearch repo - https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch. @camerski is in the process of adding this to the nightly.

[dblock]

See https://discuss.opendistrocommunity.dev/t/maven-repository-artifacts-for-plugin-development/6406 for another ask for this.

[frotsch]

Pls. also refer to opensearch-project/OpenSearch#939

[frotsch]

Why only snapshots and not also the released artifacts?

For plugin developers it's IMHO an absolute must have. Local maven repo does not work well with distributed CI environments etc ...

So I like to ask to publish all the artifacts (releases and snapshots) to maven central like elastic did it/does it.

[dblock]

Why only snapshots and not also the released artifacts?

For plugin developers it's IMHO an absolute must have. Local maven repo does not work well with distributed CI environments etc ...

So I like to ask to publish all the artifacts (releases and snapshots) to maven central like elastic did it/does it.

Absolutely. I edited the issue accordingly.

[frotsch]

https://discuss.opendistrocommunity.dev/t/how-will-the-opensearch-plugins-be-delivered-in-ga-release/6250

[mch2]

1.0 artifacts for core, common-utils, job-scheduler-spi, and notification are now available in maven central.

Nightly snapshot builds are pushed here by this job.

I am in the process of updating the README with information for repo onboarding & details on the bundle build system.

Visibility into the builds can be tracked here.

[newUserForTesting]

Hello Experts,

Could you please make this also available in maven central: https://github.com/opensearch-project/sql/

Appreciate your help!

[rursprung]

from what i can see you only publish the individual JARs & co.
could you please also publish the ZIP files for the actual distribution (incl. the -min distribution!)? that's the one i'd need on maven.

[dblock]

Do you need it in a Maven repo, or do you need it "somewhere", and if it's the former, how do you want to use it?

[rursprung]

Do you need it in a Maven repo, or do you need it "somewhere", and if it's the former, how do you want to use it?

in a maven repo. we use gradle to orchestrate our builds, incl. building docker images. so the gradle build pulls in the artifacts (e.g. the ZIP files) from the maven repo and puts it into the build directory, for the Dockerfile it then looks as if the file would be in the same directory (so you can just do COPY foo.zip /tmp/foo.zip even though foo.zip is not present in your git repo & working tree).

right now i have to manually download the ZIP file and then manually upload it to our internal maven repository, which is very cumbersome since i need to repeat this for every release of every such ZIP file (elasticsearch / opensearch, kibana / opensearch dashboards, plugins, etc.).

[rursprung]

is there any indication if this will make it for 1.1 next week?

[dblock]

Do you mean zips @rursprung? We haven’t been publishing zips to maven, and I am still not convinced we should (maven is really for Java dependencies - who else publishes zips?), but all the Java artifacts are in AWS OSS Sonatype and will be promoted to maven central.

https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#consume-maven-artifacts-in-order

[rursprung]

Do you mean zips @rursprung?

sorry for not being more specific. yes, i meant the distribution ZIPs

We haven’t been publishing zips to maven, and I am still not convinced we should (maven is really for Java dependencies - who else publishes zips?), but all the Java artifacts are in AWS OSS Sonatype and will be promoted to maven central.

https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#consume-maven-artifacts-in-order

some examples of others which also publish ZIPs on maven:

• Search Guard: https://docs.search-guard.com/latest/search-guard-versions (all links go to their maven server) and we're actively using that for downloads
• Wildfly: e.g. https://repo1.maven.org/maven2/org/wildfly/wildfly-dist/24.0.1.Final/

maven can be used for things other than java dependencies and is also actively used in such cases.

[dblock]

I'd like to know what @nknize @mch2, @peternied and @bbarani think?

[dblock]

My concerns are:

• The fact that we will no longer have a single authoritative source for artifacts and there will be two ways to get the same thing from two different places.
• Not an ask from many people in the community. Maybe we/you want to bring it up on the forums?
• Lesser of a concern, the proliferation of snapshot artifacts, times every build.

Note that OpenSearch even inherited DistributionDownloadPlugin that says that the authors didn't want to put package artifacts into maven.

Finally, others can also ask for artifacts to be published into and then would we say no?

[peternied]

we use gradle to orchestrate our builds, incl. building docker images.

@rursprung What about a docker image instead of a zip file, if there was a -min version of OpenSearch would that be more in alignment with your process?

[rursprung]

* The fact that we will no longer have a single authoritative source for artifacts and there will be two ways to get the same thing from two different places.

one approach would be the one from search guard: publish on maven and use that link everywhere as the download link

we use gradle to orchestrate our builds, incl. building docker images.

@rursprung What about a docker image instead of a zip file, if there was a -min version of OpenSearch would that be more in alignment with your process?

i think we had this discussion in the forum a while ago, but i'd have to dig up the exact post again if you'd want the details.
if we build the image ourselves we can control which base image we're using, which java version we're using, etc. - we control the whole stack and can ensure that it's the same for all of our services and it also allows us to react fast to relevant changes, e.g. updates in case of security findings.
process-wise it is easy for us to build our own docker image, getting the approval to use a 3rd party docker image is a bit more complicated because our CI tools will report it as non-compliant.

[peternied]

@rursprung Another suggestion, while still leveraging an authoritative link, how about scanning https://github.com/opensearch-project/project-website/tree/main/_artifacts for opensearch-* and then into the the version description?

opensearch-1.1.0-min-linux-x64.markdown:

---
role: daemon
artifact_id: opensearch-min
version: 1.1.0
platform: linux
architecture: x64
artifact_url: https://artifacts.opensearch.org/releases/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz
slug: opensearch-1.1.0-min-linux-x64
category: opensearch
type: targz
signature: https://artifacts.opensearch.org/releases/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz.sig
security_warning: true
indirect: true
---

[dblock]

I've opened #716 specifically for publishing ZIPs to maven, we can keep adding there.

[dblock]

I am going to close this issue. We are publishing snapshots and release artifacts to maven with increased automation. The documentation in the repo's README is sufficiently well documenting the end-to-end process and https://github.com/opensearch-project/opensearch-build/blob/main/ONBOARDING.md describes what plugins need to do. There are separate action items on making the builds better visible.

[dblock]

More process documentation: #701