[Bug] Cannot get latest artifacts for CI

[peternied]

In the Security team, we've got two tightly coupled repositories, our plugin for OpenSearch and a plugin for OpenSearch Dashboards. During out CI workflow we depend on artifacts from the distribution build [1] [2]. If these artifacts are out of date or not present it creates a considerable amount of churn to figure out why they aren't up to date and then outreach to trigger a 'minimal' distribution build, latest blocked PRs [3] [4]. This is compounded in plugins for OpenSearch Dashboards where we might need to make a fix for a new version of OpenSearch in the plugin, but the distribution has to build so we can download the most recent artifacts.

Problem

We are at a point where it is not productive to depend on distribution artifacts for our CI workflow, due to requesting / waiting on a 'partial' distribution build [5]. We would like to start producing snapshot artifacts for use between these repositories. This gives plugin teams control over what artifacts are used and how they are updated.

Proposal

We are proposing that some artifacts are not built through OpenSearch-build systems. We would like to publish these artifacts through maven - can we do that independently of the https://build.ci.opensearch.org infrastructure? Has someone already done this and we can repeat their process?

• [1] https://github.com/opensearch-project/security/blob/bae002b0041e43efdcbd56eef24db142d3f6e832/.github/actions/start-opensearch-with-one-plugin/action.yml#L27-L32
• [2] https://github.com/opensearch-project/security-dashboards-plugin/blob/8e9c4b70f4651a5c733e6b823c51cd4994257ccd/.github/actions/download-plugin/action.yml#L20-L24
• [3] Add indices:admin/close* to list of permissible index permissions security-dashboards-plugin#1323
• [4] Synchronize all permissions from OpenSearch security-dashboards-plugin#1325
• [5] https://build.ci.opensearch.org/job/distribution-build-opensearch/7079/parameters/

[peternied]

FYI @opensearch-project/security

[dblock]

+1 on moving snapshot builds into their own repo GHA workflows, including OpenSearch

[cwperks]

@peternied I filed a related issue a while ago in the security repo with stale maven jars. I think security should checkout core and perform a ./gradlew publishToMavenLocal to ensure the build is performed with the latest SNAPSHOT. opensearch-project/security#2276

[bbarani]

We have already decided to move snapshots out of distribution build jobs and created an issue but we haven't prioritized it yet due to bandwidth issues. @gaiksaya already has the steps / information required to make this happen.

[joshpalis]

+1 @peternied we have experienced similar issues with stale snapshots and the resulting churn has impacted our velocity as well. @saratvemulapalli has opened a proposal here : #3180

[gaiksaya]

Sample workflow to publish snapshots via GHA https://github.com/opensearch-project/opensearch-sdk-java/blob/main/.github/workflows/publish-snapshots.yml
Some additional code that needs to go in opensearch-project/opensearch-sdk-java@0ce8a0d

[bbarani]

Closing this issue as we are tracking the progress to move the snapshot job to GHA workflow using this issue