[RFC][Build] extra plugins and sourcing functionality in OSD that is not available in non-min artifact OpenSearch Core

[kavilla]

Glossary

• OSD: Abbreviation for OpenSearch Dashboards
• Third-party plugin: Any plugin not included in the core repository, officially referred to as "third-party plugin" (literally within code)

Problem

Historically, the inclusion of features within the OSD repository has been guided by whether the minimal distribution of OpenSearch offers those features without necessitating a third-party plugin. If not, such features should be available through an OSD third-party plugin that isn't an internal one.

This guideline aimed to preserve the integrity of default experiences for projects and developers forking from OSD, ensuring seamless operations in their instances.

This approach has generally been effective, facilitating:

• Clear ownership and maintenance models for contributors
• Assurance that the open-source minimal distribution remains free of non-open-source features
• Maintenance of a 1:1 feature parity with upstream OpenSearch dependencies
• Development stability, offering a consistent default state without additional configurations or setup, and avoiding version mismatches
• Enabling developers to utilize capability APIs or plugin checks for conditional logic
• Allowing extension of existing OSD functionalities for additional registrable functions

However, this guideline has introduced several challenges:

• Necessity for conditional logic or code duplication by plugins for functionalities not available by default
• Potential for circular plugin dependencies
• Extension requirements for unavailable features or components, leading to increased ramp-up times
• Development and build complications due to local path and reference discrepancies
• Lack of guaranteed runtime precedence for third-party plugins over core plugins, leading to non-standard solutions
• Unlike OpenSearch, plugin interactions with each other and the core through APIs are not fully shared

Proposed Solution

The reintroduction of extras plugins (and possibly packages) to OSD is proposed. This concept has never been entirely phased out from the repository [example]

The proposal includes:

Building all plugins by default, including those requiring non-minimal OpenSearch functionalities
Serving all plugins by default in distribution processes, leveraging existing OSD core logic
Excluding extra plugins by default in development mode
Introducing a flag to exclude extra plugins from the build process, catering to partners preferring a minimal distribution
Offering a flag to prevent serving extra plugins in distributions, allowing partners to limit access
Providing a flag to enable all extra plugins in development mode
While the specifics of these flags and default settings are open for discussion, the overarching goal is to facilitate access to foundational logic not present in the minimal distribution but utilized by multiple plugins, benefiting partners and developers alike.

This approach also aims to centralize foundational logic, ensuring application stability through comprehensive linting and automation, thereby preventing potential issues and bad practices from downstream plugins.

Related PR adding some of this solution: #7080

Other info

This is targeting https://github.com/kavilla/queryEnhancements which registers PPL and SQL languages to the data plugin

[ruanyl]

Hi @kavilla, I have a few questions:

1. What does "extra plugins" refer to?
2. Third-party plugins are plugins not in OSD core src/plugins, but are available in OSD full distribution, am I right?

Building all plugins by default

Does this mean run yarn build will build OSD with all 3rd party plugins?

[seraphjiang]

build, develop, test, and release plugin outside of osd required significant effort and full knowledge, that will not only increase learning curve for new contributor and increasing the workload to the plugin owner for each release.

we should adopt more core plugin into osd, so developer could leverage what we have built the infra/automation for them, and they could focus on innovation less on process.

Only two cases i feel it is worth to make a separate module/package

case 1: plugin/module is reusable component. could contribute to other open source project.
case 2: plugin/module is belong to a vertical domain, e.g. alerting, machine learning.

in all other case(or by default), we should adopt more, rather than push pluign/code to separate repository.

[ashwin-pc]

focus on innovation less on process.

I agree with this statement, but innovation also does not mean carelessly dumping code into a repo.

we should adopt more core plugin into osd

I agree with this only if the added functionality makes sense. The specific case being talked about here is functionality that does not work work in isolation and need a corresponding backend that isnt the OS engine to function. The plugin in question here cannot work with the OS engine core codebase.

That being said, since OSD consumes the OS min distribution and if we take its API as the contract for a minimum product backend for a feature to work, I can be convinced that this should be a core plugin. And I want us to consider how we will treat similar contributions to query languages or other features specific to other non AWS platforms. Right now the core is unbiased and is an extensible abstraction. This plugin introduces bias

[ashwin-pc]

I would also want to draw parallels from a similar conversation in OS Core where something similar was proposed:
opensearch-project/OpenSearch#12434

[kavilla]

@ruanyl

1. What does "extra plugins" refer to?

It would be plugins housed in this repo that can be configured to not be built and not be ran in a single flag.

The "extra" being used as the name as we haven't cleaned out "extra" from when we forked:

OpenSearch-Dashboards/packages/osd-pm/src/config.ts

Line 59 in 4f54049

projectPaths.push(resolve(rootPath, '../opensearch-dashboards-extra/*'));

2. Third-party plugins are plugins not in OSD core `src/plugins`, but are available in OSD full distribution, am I right?

Yes and no, third party plugins are plugins not build in OSD core src/plugins. Within the OpenSearch project, the full distribution includes all the 3rd party plugins produced by OpenSearch project. Meaning partners or non-OpenSearch project plugins are 3rd party plugins and treated equally. Which has implications on what OpenSearch project plugins can actually do to core functionality even though we expect to ship it as a whole build.

Does this mean run yarn build will build OSD with all 3rd party plugins?

Sorry about that more like all plugins with src/plugins and extra plugins sourced in here. Any third party plugin will not be built still.

[AMoo-Miki]

@kavilla, you noted:

However, this guideline has introduced several challenges:

• Necessity for conditional logic or code duplication by plugins for functionalities not available by default
• Potential for circular plugin dependencies
• Extension requirements for unavailable features or components, leading to increased ramp-up times
• Development and build complications due to local path and reference discrepancies
• Lack of guaranteed runtime precedence for third-party plugins over core plugins, leading to non-standard solutions
• Unlike OpenSearch, plugin interactions with each other and the core through APIs are not fully shared

All of these essentially boil down to:

1. Avoiding code duplication: The problem of plugins' desire to share code without its duplication is a valid concern and a common challenge in software development. However, this can effectively and effortlessly be solved by employing the existing modular approach of OSD plugins by porting the common functionality into a plugin of its own. By creating shared plugins that encapsulate commonly needed functionality, other plugins can leverage and reuse this code, eliminating the need to bloat core OSD's codebase; this prevents ownership complications and promotes reusability and maintainability.
2. Having some plugins initialize before core plugins: While this limitation is by design, we have seen developers fight it using creative solutions. While I am not against allowing runtime precedence for third-party plugins over core plugins, I am not a fan of it either. I prefer understanding the reasons developers felt they had to fight the system and provide them the right solutions. If developers need more extensibility APIs, we should provide them that, as opposed to allowing them to modify or override stuff before core plugins run.

---

@kavilla, you also mentioned:

This approach also aims to centralize foundational logic, ensuring application stability through comprehensive linting and automation, thereby preventing potential issues and bad practices from downstream plugins.

Centralizing foundational code introduces inflexibility and runs against the extensibility the plugin ecosystem of OSD is trying to provide. A centralized codebase for non-core functionality will make it more challenging to evolve in response to new use-cases and requirements. This can easily veer into a monolithic architecture where version decoupling would be impossible. Building on top of the existing plugin ecosystem and raising awareness about the CrossCompatibilityService, we can continue to promote flexibility, scalability, and ease of maintenance.

---

@seraphjiang, you wrote:

build, develop, test, and release plugin outside of osd required significant effort and full knowledge, that will not only increase learning curve for new contributor and increasing the workload to the plugin owner for each release.

That initially sounded reasonable but the actual process of setting up a development environment for plugins is relatively straightforward and well-documented. Developing a plugin can be as simple as placing the code in a specific folder and following the prescribed structure and conventions. This approach, although primitive, is functional and effective for most use cases, especially for experienced developers familiar with the ecosystem.

IMO, the best way to streamline and enhance the plugin development experience, particularly for new contributors, is to have an SDK that provides standardized tooling, utilities, and a more structured development workflow; this reduces the learning curve and promotes consistency across different plugins.

OSD has its legitimate reasons to not prioritize the creation of the SDK for plugins and while that is a better solution in the long run, the current approach should not be dismissed as overly burdensome or lacking in documentation, as it serves its purpose adequately. Furthermore, the shared development environment already provides plugins with building, testing, and releasing tools out of the box.

---

@seraphjiang, you also mentioned:

we should adopt more core plugin into osd, so developer could leverage what we have built the infra/automation for them, and they could focus on innovation less on process.
...
in all other case(or by default), we should adopt more, rather than push pluign/code to separate repository.

I touched on this in my response to Rocky. Adopting a centralized approach to OSD plugins could potentially lead to an overly monolithic architecture, hindering the flexibility and innovation that a decentralized plugin ecosystem fosters. An OSD core that imposes architectural decisions and constraints will limit the freedom and creativity of plugin developers.
This will also complicate ownership and inhibit parallel development that is facilitated by having code reside in separate repositories.

---

@kavilla, while responding to Yulong, you said:

Within the OpenSearch project, the full distribution includes all the 3rd party plugins produced by OpenSearch project.
...
Any third party plugin will not be built still.

You and Yolong almost touched upon a crucial challenge with the OSD ecosystem: the discoverability of plugins, both developed inside and outside the OpenSearch repositories. Currently, OSD lacks a centralized mechanism to identify and showcase plugins, leading to a significant gap in awareness and accessibility. This prevents users from discovering potentially transformative functionalities offered by plugins, limiting their ability to leverage innovative solutions. Simultaneously, it hinders plugin developers from effectively promoting and sharing their groundbreaking innovations with the broader community, diminishing incentives for continuous collaboration and progress. A marketplace for plugins with one-click install capability and dependency management will be an answer to many prayers.

[seraphjiang]

@AMoo-Miki build and generate plugin doesn't straitforward doesn't mean effort is less.

also see my recommendation for criteria to determine

case 1: plugin/module is reusable component. could contribute to other open source project.
case 2: plugin/module is belong to a vertical domain, e.g. alerting, machine learning.

to simplify the discussion with real example.

if we want to push to create https://github.com/kavilla/queryEnhancements, i'm not seeing strong justification.

[ashwin-pc]

Based on the thread here I think we agree that we dont want a 3rd place for plugins to exist. I think i agree with @seraphjiang 's statement here about criteria but we need to make that criteria more explicit. Can we agree on something like this?

Core Plugin Criteria

A plugin should be considered core if it meets ALL of these criteria:

1. Fundamental Functionality: The plugin provides basic, essential functionality for data querying, visualization, or dashboard management.
2. Universal Applicability: The feature is useful across various use cases and is not specific to a particular domain or industry.
3. Tight Backend Integration: It integrates closely with core OpenSearch features and doesn't require extensive additional backend services.
4. Limited Scope: The plugin has a focused purpose and doesn't constitute a large, complex application on its own.
5. Central to User Experience: Removing the plugin would significantly impair the basic functionality of OpenSearch Dashboards for most users.

External Plugin Criteria

A plugin should be considered external if it meets ANY of these criteria:

1. Specialized Application: The plugin constitutes a larger, more complex application serving a specific use case or domain.
2. Extended Functionality: It provides advanced features that go beyond basic data exploration and visualization.
3. Additional Dependencies: The plugin requires significant additional backend services or databases beyond the core OpenSearch.
4. Domain-Specific: It's designed for particular industries or specialized use cases not applicable to all users.
5. Independent Operation: The plugin can function as a somewhat standalone application within the OSD framework.

Decision Process

1. Evaluate the plugin against both sets of criteria.
2. If the plugin meets ALL core criteria, categorize it as core.
3. If the plugin meets ANY external criteria, categorize it as external.
4. In ambiguous cases, lean towards categorizing as external to keep the core lean.
5. Document the reasoning behind each decision for future reference and consistency.

My main concern here is to keep the minimal experience small while moving most of the larger features out of core. Similar to how many linux distributions that have a min and a full experience.

[AMoo-Miki]

@ashwin-pc, even though I would have liked a more rigid and aggressive criteria for core plugins, I think this is a good start / compromise.