-
Notifications
You must be signed in to change notification settings - Fork 276
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Meta] Mechanism to publish the plugin zips to maven repo. #1916
Comments
Looks like a dup of #716 |
Hey @dblock, since there are two aspects for this maven publish, one for snapshots repo and other for actual release repo, the two repo workflows are not constant, as release repo has more wider process, like pre staging, promoting, signing, additional maven tags and then releasing, so I have created a meta issue #1916 that expands with more details based on the issue #716. |
The proposal in this issue is attempting to re-create a separate maven publishing process for any set of ZIPs. This may not be what we want. Each plugin already publishes artifacts to maven. For example, job-scheduler publishes job-scheduler-spi.jar. This is done following these instructions. It seems to make sense to extend the publication of the JAR to include other artifacts, such as the distribution ZIP. This can be accomplished by adding @mch2 @reta have mucked with all of this quite a bit, maybe you have better ideas? |
Thanks @dblock, more over looks to me your solution is like working backward, with this we need to add this logic across all repo components to have the zips published in same way, but it will ease of not re-doing the mvn publishing process again. |
@dblock I am hesitating a bit to acknowledge that publishing plugin ZIPs to Maven repositories is a good idea (it is certainly possible - no doubts). But from other side - how people are supposed to use it? It looks to me we should align the publishing (at least, of the standard plugins) with the In scope of this issue, if you need those dependencies in Maven projects - the direct URL to the catalog could be used along with plugin (fe [1]), they should not be only artifacts. Also, it covers well external 3rd party plugins, often hosted on Github and not published to Maven repos (afaik). Hope I didn't introduce the confusion, what do you think? [1] https://github.com/maven-download-plugin/maven-download-plugin |
@reta I think as a consumer I would ideally want to do something like this in Gradle:
Latest snapshot build Is this crazy? Something like maven-download-plugin works well as a middle-ground solution, but @prudhvigodithi it looks like you chose the maven route, I imagine you have a clear idea of how those artifacts will be used? And yes, #1823 is waaay less work for similar effect where you just download from a |
Hey @dblock, I see this PR overrides the requirement to download a zip in a maven way, it uses directly GET protocol to download the latest zip, also as suggested by @reta maven-download-plugin download different files on different protocol and this can connect with above change, a user can download straight from URL without giving maven coordinates as
So from consumer perspective if we can document this we can skip zips publish to maven, else once its in maven a user can always download using maven coordinates as well as with GET protocols. |
Once ZIPs are published to maven the URL will be changing every time, you'll need to be able to resolve something like |
No, it is not, but:
There are plenty of projects which use this route (publishing ZIP/TAR/...), it is on the table for sure and it works. BUT that would mean, very likely, we have to ask all external plugin authors to do that for their plugins (and as such, they would have to look for publishing options). |
@dblock I dont see date format is added to zips with current setup, the plugins are just dumped as regular files without date, so you are saying if we add |
Yes, it will add the date to the file name, that's how maven works for snapshots. Take a look at https://aws.oss.sonatype.org/service/local/repositories/snapshots/content/org/opensearch/opensearch-job-scheduler-spi/2.0.0.0-alpha1-SNAPSHOT/, that's all the artifacts for many snapshot builds. To find out what the latest one is, maven looks in this file. Regardless of how you publish to maven, SNAPSHOT builds will end up having a date/time in the filename. While you can disable it, there's no folder called "latest", so you'd have hard time getting a direct download URL to the latest build. When we publish a final released version of 2.0, there's only 1, so maven doesn't need to append the date or anything. |
> > Is this crazy?
Maven could use maven-download-plugin, etc. Either way we're only debating whether ZIPs are published like JARs in maven, or they are published as ZIPs somewhere else. We can do both and let people decide what they like.
It piggybacks on maven repository resolution 100%. It's effectively a fancy wrapper over (the equivalent of) maven-download-plugin, a syntax sugar that knows to install the plugin into an OpenSearch cluster before running integ tests. But maybe that's just over-engineering ... |
@dblock sounds good, as far as different tools pull ZIPs consistently using same conventions, we could use Maven as the distribution mechanism. |
Hey @dblock, I have tested the idea to work backward from individual repo, to be able to add assembled zip's also to maven format just like how the jars are added. I was able to do this Draft. Please check the following implementation steps
Now with this a user should be able to download opensearch-job-scheduler plugin zip with maven coordinates as
The above values of groupId, artifactId, version are all inherited from default job-scheduler build system, as all heavy lifting is already been done here. |
Thanks for the update @prudhvigodithi ,
I think there should be a way to strip those artifacts (javadoc and sources) from ZIP publishing without CLI scripting. May be we should strip them by default and only configure with JAR publisher? |
I like the draft! I obviously dislike the custom elimination of sources and jar, which is hopefully something that can be disabled in shadow. Then, job scheduler has this spi vs. non-spi build gradle so this may be muddying the waters. Should this be going into root buld.gradle? Should this be in the distribution gradle? A new subfolder dedicated to publishing the zip to maven? I think you should iterate on a PR into job-scheduler for this and let's see if we can merge something? |
@reta, yes we can directly push them with jar, then all the zips, jar javadoc and sources jars's will be under same maven location, which could confuse users to use those zip plugins, hence I wanted to separate the maven location for plugin zips, there already exists plugin prefix. ya thats a good point @reta I have this question as well, do we actually use these javadoc and sources jars's ? @dblock @bbarani @reta @dblock I liked the idea for a new subfolder, let me explore this and yes my idea was 1st once we have the job-scheduler working, then we can have the same implementation for other components as well. 👍 |
they are useful for developers (for example if you develop a plugin which depends on another plugin), so it is better to have them published.
We could actually follow the other projects and publish ZIPs as [1] https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.4.9.v20180320/ |
Cool, thanks for the info @reta |
I see whats happening so this block of code eliminates from uploading them :) |
@dblock @prudhvigodithi since I am lacking the context here, sorry for possible derailing, but using custom made scripts for publishing artifacts is not what the community (since we are discussing the plugins) would expect: |
Wanted to chime in with a few callouts & questions.
Agree 100%. This stems from a current limitation with our signing process not wired into gradle. So the build system builds and pushes the artifacts that are then after the fact picked up, signed, and promoted with these scripts.
We've had an issue out for killing this script for a while. We should be pushing directly from As far as placing maven publications in the same group/artifact id with different packaging. I'm hesitant about this because I'm not sure the jars produced by each plugin match the produced plugin zips that are installed today. Job-scheduler-spi vs job-scheduler as an example. Meaning the artifact is inherently different and not simply differentiated by packaging type. If this is the case I think we should treat them as two separate artifacts under the same group ID. I would also explore updating the publish plugin with a task where all plugins need to specify is the output path of the zip and any plugin specific fields required in the pom (scm etc). That task could then generate the pom and publish in one go. This would make it easier to onboard and less custom logic in each plugin script. |
Hey @dblock, @reta, please check this Draft, I have modified to add the zip publish task to root gradle project. With this I see the maven coordinates for zip plugin being created as Example
Since this is added as part of root project, we call a task So my plan is to get this merged to job-scheduler and with no other code change we should be see job-scheduler plugin zips under nexus maven repo plugin location. |
Hey @reta, thats true, we should not use same groupID for different package types, I have separated the zips's as
|
@prudhvigodithi dB looks to be blocking that PR until all plugins get this change in to avoid adding custom logic to the default build script, but that would mean introducing a custom build script in every plugin, then changing the default build script and then removing the custom build script from each plugin. I would prefer to only add the publishing zip task for plugins with the default build script, and then once all plugins have onboarded, the default build script can change to introduce the change without requiring an interim build script for all plugins. |
What @downsrob is proposing works fine as well - get all plugins to onboard without adding a custom build script, then change the default to enable it for everyone. |
Hey @dblock @downsrob, if I understood correct, 1st make sure all plugins use the gradle plugin |
Yes
i am not sure i understand what you’re saying :) once you’ve added the task to the default build.sh without conditionals it will work for all plugins Note that the default script is still used for older versions, so we may not be able to modify it after all |
I'm fine until all plugins are onboarded to use |
If you want a published zip of the older versions anyways, say, for BWC testing, then I can see the merit in backporting the task to all versions and moving on. However, it seems natural that eventually the default |
That's proposed in #1975, add your comments there. I want to make sure we don't try to solve the kitchen sink at the same time, so maybe cloning the default build script for your plugin is the most expedient path? |
Starting with release 2.1.0, we now have plugin zips in maven repo Thank you |
Is your feature request related to a problem? Please describe
With current setup the plugins zips that are used as dependencies cannot be downloaded dynamically during runtime, as they are not part of version controlled maven system. Only the respective Java jars are available to download using maven coordinates.
More details on #716
Describe the solution you'd like
Publish the release candidate zips, distribution build zips (snapshots) to maven repo, so that a user who is dependent on the plugin zip, can download using maven coordinates. There needs to be a process implemented for uploading zips to snapshot and release repos
Proposed Solution [1]:
With the current build system the plugin zips are stored under
builds/plugins/
, these are not stored or ordered in maven folders, so that maven can publish them with right artifactID, groupID and version.for each plugin use
install:install-file
to create the maven coordinates and publish them to local file system maven repo, passing the right artifactID, groupID and version along with proper qualifier if required.Then from local file system publish them to respective snapshot or release repos, adhering to existing workflows.
To achieve this integrate this solution with existing CI.
Proposed Solution [2]: ---> (The chosen one)
Add a gradle sub-project in each component, so that root gradle project can include them, that converts the zips generated inside distribution folder to maven format, then publish them to local maven repo and staging repo, this way all the maven coordinate's (groupID, artifactID, version) generation and other maven heavy lifting is done at component (plugin) level and using CI with already existing code, publish them to maven nexus repo.
Additional context
For Proposed Solution [1]:
Need to come up with an idea to add either a python script, a groovy shared library or a gradle sub tasks for publishing the zips.
Snapshot zips:
Upload the snapshot plugin zips to https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/
<plugin_name>/-SNAPSHOT/<.zip_file>
Edit the plan https://github.com/prudhvigodithi/opensearch-build/blob/slack-publish-v1/jenkins/opensearch/distribution-build.jenkinsfile#L82 to upload zips to nexus maven repo
Release zips:
The Release candidates will be signed before staging to make sure files are unaltered.
Also there needs to be an additional tags inside the POM as example for the zips pom file
https://aws.oss.sonatype.org/service/local/repositories/releases/content/org/opensearch/common-utils/1.0.0.0/common-utils-1.0.0.0.pom
Create a staging repo nexus-staging:rc-open this will create a new new staging repository on the remote Nexus
Upload zips from the local filesystem to a staging repository using plugin in built deploy-staged-repository
Close the created staging repo nexus-staging:rc-close
For Proposed Solution [2]:
Add the gradle subproject to each repo and include in root project, for the generated zip to get published to local staging maven repo.
Implementation tasks
Associated Issues
Issue with integrating the gradle java custom plugin:
opensearch-plugin-template-java jdk and gradle version version upgrade
Document Create
Campaign Issues:
Associated PR's
With custom plugin:
Plugin design and usage doc
opensearch-plugin-template-java jdk and gradle version version upgrade
opensearch-plugin-template-java add
opensearch.pluginzip
details to templateRelated Plugin Issues
plugin/
folder. sql#660The text was updated successfully, but these errors were encountered: