Pre-compile and host native plugins on artifacts.opensearch.org

[bbarani]

Is your feature request related to a problem? Please describe.
Users are finding it hard to install the native plugins as they are not readily available to download when running sh opensearch-plugin install command

Describe the solution you'd like
All the internal plugins under /OpenSearch/tree/main/plugins are pre-compiled and hosted on https://artifacts.opensearch.org/releases/plugins/* directory

Describe alternatives you've considered
Manually building the plugins on local and making changes to point the install script to use the local version
opensearch-project/OpenSearch#951 (comment)

Reference:
opensearch-project/OpenSearch#951

[peterzhuamazon]

@mch2

Run the command in both snapshots and releases version of min shows 2 different behaviors:

# Snapshots
$ ./opensearch-plugin install repository-s3
-> Installing repository-s3
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
ERROR: attempted to install release build of official plugin on snapshot build of OpenSearch

# Releases
$ ./opensearch-plugin install repository-s3
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/releases/plugins/repository-s3/1.0.0/repository-s3-1.0.0.zip
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1932)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1528)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at org.opensearch.plugins.InstallPluginCommand.downloadZip(InstallPluginCommand.java:438)
	at org.opensearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:515)
	at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:306)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:260)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:237)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
	at org.opensearch.cli.Command.main(Command.java:103)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

[peterzhuamazon]

Related issues during release of these artifacts:
opensearch-project/OpenSearch#996

[pkalliok-aiven]

We are also very much looking for mapper-size and analysis plugins. Is there any way to install them currently? Just build from source?

[tlfeng]

We are also very much looking for mapper-size and analysis plugins. Is there any way to install them currently? Just build from source?

Hi @pkalliok-aiven I think currently the the artifacts of the native plugins exist in artifacts.opensearch.org, but there is an issue when installing a plugin with its name (mentioned in issue opensearch-project/OpenSearch#996).
However, you could install by the URL, for example:
./opensearch-plugin install https://artifacts.opensearch.org/releases/plugins/mapper-size/1.0.0/mapper-size-1.0.0.zip

[AmiStrn]

It would be great to have a fix for this backported as far as possible. since some users would not be able to move to v1.0.0 without this fixed (therefore it would be a major breaking change between Elasticsearch and OpenSearch).
Additionally, those that are already using the workaround would benefit from this as they could avoid the pains of upgrading if they are not interested in v1.1.0 yet.

[peterzhuamazon]

We have fixed the issue in opensearch-project/OpenSearch#1233.
Native plugin for 1.1.0 will be available once 1.1.0 release.
Thanks

[peterzhuamazon]

native-plugins-1.1.0.log

[peterzhuamazon]

Push to candidate channel and did a test on repository-s3, which works:

$ ./opensearch-plugin install repository-s3
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-s3 with folder name repository-s3

$ ll ../plugins/ | grep repository-s3
drwxr-xr-x 2 opensearch opensearch 4096 Oct  1 21:06 repository-s3

[peterzhuamazon]

The build is completed for now.
We will use #686 to track the release of native plugins in 1.1.0.

[peterzhuamazon]

1.1.0 Native Plugins released for now.