[RFC] Adding new plugins to OpenSearch: core or separate repository?

[reta]

Is your feature request related to a problem? Please describe

As the OpenSearch ecosystem continues to evolve and to grow, new plugins are being developed all the time. Respectively, the reasonable question arises where should such plugins be hosted:

1. Outside the OpenSearch project? (natural choice for third party plugins)
2. Under OpenSearch project umbrella but separate repository?
3. In the OpenSearch core repository?

The 1st option is really the way most plugin developers start off and it (arguably) should be the preferred route, at least till the plugin in question gains the community support to warrant other options. The 2nd and 3rd options are not really clearly defined but should.

Describe the solution you'd like

To eliminate the confusion and have a clear guidance for the internal (AWS) contributors, the suggestion is to always start off with separate repository under OpenSearch project organization for all non essential plugins. The essential plugins have special meaning in OpenSearch core and called modules, they are essential for service to function (fe network transport, painless scripting) and always supplied preinstalled with the distribution by default.

Why separate repository?

Primarily, adding more plugins to OpenSearch core incur significant maintenance cost, not to mentioned the need to bring the necessary expertise. It also bloats the OpenSearch core repository, increases the time of the builds and CI checks, complicates releasing the critical patches / bugfixes, ... to name a few reasons. Having separate repository with targeted team of maintainers not only eliminates the concerns from above, but allows the development at the pace the maintainers are comfortable with.

How to get it?

There are a few obstacles along the way, often cited as the reason to consider 3rd option right away. Only OpenSearch project admins could create a new repository, so the first step is to submit a request to #admin-requests Slack channel. After that, the long process takes place on AWS side before the repository appears within OpenSearch project organization:

• Ticket validation (? days)
• Tech review (2-3 days)
• IP review (3-5 days)
• Final review (? days)

This process is not familiar to non-AWS contributors, but even for internal contributors it is a painful one. That is one of the reasons to involve OpenSearch Leadership Committee here to help streamline the procedure.

Related component

Plugins

Describe alternatives you've considered

Once the agreement is reached, the final guidelines will be published to OpenSearch core.

Additional context

• Query insights plugin implementation #11903 (comment)
• [RFC] Query insights framework  #11429

[reta]

@AmiStrn @msfroh @andrross @peternied @dblock would appreciate your input folks, thank you.

[andrross]

@reta I agree with your recommendation. Another way to phrase is is that adding new plugins in the plugins folder of the OpenSearch repository should no longer happen going forward. If the feature is optional, then it should be a separate repository. If the feature is not optional, then it should be a module. Of course we can allow for exceptions if it truly makes sense, but this should be the default policy going forward.

not to mentioned the need to bring the necessary expertise

The events-correlation-engine is an example of this problem. This feature was extracted out of the security analytics plugin as a more generic capability that can be used in other domains. However, the effort has stalled out primarily because none of the core maintainers have any expertise related to this particular feature. I believe it would be able to iterate faster and ultimately be more successful as a separate repository.

[dblock]

My recommendation would be to start plugins outside of opensearch-project in the creator's personal GitHub account. The entire overhead of having to deal with the existing org maintainers goes away if you do, and there are no downsides IMHO. There's no value in being part of the opensearch-project org other than for discoverability until the plugin needs to build and release with the distribution. However every new repo creates work for the organization, e.g. security vulnerabilities will need to be dealt with.

Aside from plugins, we need a more general public process of what repo can be created/moved into the organization. Currently this burden is on the org admins (AWS).

Whichever avenue we propose, how about turning this proposal into some text and PRing it into https://github.com/opensearch-project/opensearch-plugins?

[AmiStrn]

Whichever avenue we propose, how about turning this proposal into some text and PRing it into

@dblock @reta I can take this if that is ok with you.
ill add the guidelines into this section: https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#developing-plugins-for-opensearch

probably just a short paragraph about "where to create your plugin?"

regarding the elephant in the room - which plugins get to be ported into the project?
this is mostly case-by-case, but it would be best if they all started outside the project.

[reta]

@dblock @reta I can take this if that is ok with you.

💯% OK with me, thanks @AmiStrn !

[peternied]

[Triage - attendees 1 2 3 4 5 6 7]
@reta Thanks for creating this RFC, looking forward to seeing how this resolves.

[reta]

Resolved by opensearch-project/opensearch-plugins#233