[Bug]: How can we download the OpenSearch min artifacts for windows

[peternied]

Describe the bug

Note; this the windows platform specific security demo script opensearch-project/security#2161

We are trying to setup builds and validation of OpenSearch for the Security Plugin, ./gradlew.bat build functions on a windows machine, but need the minimum artifact, just like how they are obtained for linux with

opensearch_version=`./gradlew properties -q | grep "opensearch_version:" | awk '{print $2}' | sed 's/-SNAPSHOT//g'`
wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/$opensearch_version/latest/linux/x64/tar/builds/opensearch/dist/opensearch-min-$opensearch_version-linux-x64.tar.gz

I assume there would be a pattern that would look something like the following. Is there a url we can use to get these artifacts?

[1] Possible windows artifact pattern https://.ci.opensearch.org/ci/dbc/distribution-build-opensearch/$opensearch_version/latest/_windows/x64/zip/_builds/opensearch/dist/opensearch-min-$opensearch_version-windows-x64.zip

Notes

I Inspected a recent 3.0.0 build, https://build.ci.opensearch.org/job/distribution-build-opensearch/6226/flowGraphTable/ Do not see a step where artifacts are uploaded.

[bbarani]

@peternied You can download the Windows -min artifacts from here. @peterzhuamazon Can you confirm if its updated link?

[peternied]

Thanks @bbarani it looks like we can get https://artifacts.opensearch.org/snapshots/core/opensearch/2.4.0-SNAPSHOT/opensearch-min-2.4.0-SNAPSHOT-windows-x64-latest.zip can keep us unblocked.

I notice this is different than the previous urls we were using prefixed with https://build.ci.opensearch.org/job/distribution-build-opensearch/, should we broadly us one over the other?

[peternied]

FYI @scrawfor99 ^

[peterzhuamazon]

@peternied unblocked, link available now.

$ curl -vvv https://artifacts.opensearch.org/snapshots/core/opensearch/2.4.0-SNAPSHOT/opensearch-min-2.4.0-SNAPSHOT-windows-x64-latest.zip
*   Trying <>
* Connected to artifacts.opensearch.org (<>) port 443 (#0)

......

< HTTP/2 200
< content-type: application/zip
< content-length: 219749189
......

Thanks.

[peternied]

I notice this is different than the previous urls we were using prefixed with https://build.ci.opensearch.org/job/distribution-build-opensearch/, should we broadly us one over the other?

Can we please keep this open until we have an answer to this question?

[dblock]

Agreed, I'll reopen.

[gaiksaya]

Hi @peternied, the snapshots are accessible via artifacts.opensearch.org. However for other components like plugin zips, etc https://ci.opensearch.org/ci/dbc/distribution-build-opensearch was used.
More details in README https://github.com/opensearch-project/opensearch-build/blob/main/README.md#min-snapshots

[peternied]

Thanks @gaiksaya, glad to see this is documented! According to the documentation there are two different ways to get artifacts, Latest Distribution Url vs Min Snapshots. Are these functionality the same, or are there considerations teams should have to use one vs the other?

[gaiksaya]

The latest distribution url is more towards distribution artifacts like tarballs, zips, manifest where as the Min Snapshots only focuses on min artifact and will always be latest as it gets overridden in each run. I hope this helps.

[peternied]

Cool, so it seems like for our build processes we should use the latest distribution url. It looks like the windows build isn't being published to that bucket. Can we use this issue to track that we need to update the build process to mirror the linux builds that are being published

[peterzhuamazon]

Cool, so it seems like for our build processes we should use the latest distribution url. It looks like the windows build isn't being published to that bucket. Can we use this issue to track that we need to update the build process to mirror the linux builds that are being published

It is already published as part of the bundle build.
The artifacts path is more of a old method before ci being introduced.

[peternied]

Does that mean there is a modification to the following URL that will get the latest windows build? https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/6234/windows/x64/zip/builds/opensearch/dist/opensearch-min-3.0.0-windows-x64.zip

Could you help me figure out what that url is?

[bbarani]

@peternied it should follow the below format but it looks like we don't have successful full distribution build for 3.0 yet but the below URL works for 2.3.1 at this point in time. @peterzhuamazon can you please confirm?

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.3.1/latest/windows/x64/zip/dist/opensearch/opensearch-2.3.1-windows-x64.zip

[peternied]

@scrawfor99 Checkout the most recent comment [1] to see how we can get the current windows build url

[1] #2754 (comment)

[peternied]

Issue seems resolved for me, thanks for all the help tracking this down