[PROPOSAL] Separate OSS functionality from vendor specific one

[reta]

What/Why

What are you proposing?

Since the fork has happened, we are constantly facing this dilemma: which features OpenSearch has to offer out of the box? AWS by all means is in very privileged position here and there are few features that slipped in into what to be thought of vendor neutral project:

Examples:

• [FEATURE] Modify integration tests to run with OpenSearch vendors (multi-vendor IT abilities ) opensearch-spark#647
• [FEATURE] Keep the OpenAPI specification vendor-neutral / vendor-agnostic opensearch-api-specification#522
• [FEATURE] Add IAM authentication option / support for SigV4 Signer opensearch-java#55
• Support for serverless implementations opensearch-benchmark#619

There could be more but those I am personally aware of.

What users have asked for this feature?

They do since many OpenSearch users are AWS customers.

What problems are you trying to solve?

We should keep OpenSearch codebase vendor neutral. In practice though, vendor specific extensions are required but have to be handled differently. What could be the plan?

• create OpenSearch-contrib organization and host all vendor (or non-core) extensions there
• create a separate repositories for vendors (under OpenSearch-project) and move all vendor extensions there

What is the developer experience going to be?

Nothing unusual, the vendor specific extensions would be managed as regular dependencies.

Are there any security considerations?

The vendors could take care of supporting their own extensions.

Are there any breaking changes to the API

No

What is the user experience going to be?

N/A

Are there breaking changes to the User Experience?

No

Why should it be built? Any reason not to?

The vendor specific extensions incur significant overhead not only from maintenance perspective but often require the monetary spending to provision the resources in question (to fix a bug or add new features).

What will it take to execute?

I think it won't require much efforts at this point.

Any remaining open questions?

N/A

[reta]

@dblock @AmiStrn FYI folks :)

[dblock]

I agree that vendor-specific extensions should be vendor-specific in principle. There are questions of whether it's ok to have code in the same repo, having releases that include that code loaded by default vs. included in the distribution, etc. I propose spelling this somewhere and opening issues for each case where we believe the principles are broken and enforcing it for anything new moving forward.

Personally, I don't mind having plugins such as repository-s3 or discovery-ec2 in OpenSearch core repo (even if I dislike bloat), or having sigv4 supported code in clients repo, as much as I don't mind support for code that supports other cloud vendors. We have enough complexity without splitting everything up. To me the runtime dependency is more important, I do not want vendor-specific runtime dependency to be dragged in by default.

[dblock]

[Catch All Triage - 1, 2]

[YANG-DB]

@dblock @reta
I agree with the following concepts:

• allow vendor specific code contributions in our own rep (similar to OpenTelemetry - contrib repository)
• no direct build or runtime dependencies in these contrib components
• add an additional abstract layer to allow the specific vendor implementation without explicitly depending on it (the next example is NOT a good separations of concerns IMO )