[PROPOSAL] Dashboards anywhere for OpenSearch eco-system

[seraphjiang]

Dashboards anywhere for OpenSearch eco-system envision and strive to build cloud native open source analytics service to enrich OpenSearch project stack and eco-system.

What kind of business use case are you trying to solve? What are your requirements?

In order to run, customize and extend OpenSearch Dashboards require deep knowledge on dashboards and opensearch. People are looking for out-of-box dashboards service.

What is the problem? What is preventing you from meeting the requirements?
Configure dashboards, customize, extend dashboards are difficult and no standard

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?

• Cloud native infrastructure to host a dashboards service
  [FEATURE] Create a simple command line tool to deploy a Dashboards cluster onto a Kubernetes cluster #2

• Dashboards need to decouple from OpenSearch, could be run separately

opensearch-project/OpenSearch-Dashboards#413

• Pluggable Data connector to connect dashboards to different data source include OpenSearch
  [RFC] Enable OpenSearch Dashboards to support multiple OpenSearch clusters OpenSearch-Dashboards#1388

• Plug & play mechanism to extend dashboards from plugin center

• Security out-of-box
  [feature-proposal] Dashboards security out of box OpenSearch-Dashboards#1215

What are your assumptions or prerequisites?

What are remaining open questions?

1. What's options for Infrastructure as code(CDK | terraform | ??)
2. What's story for update software?
3. How could we handle specific user's config at scale
4. How many components we need to support? Infra for service, meta service to store all data, centralized identity provide, integration with other software like slack, github. linkedin, facebook, google drive.
5. What's log, metric, and distributed tracking story cross all components, could we dogfood our own stack
6. what's project roadmap
7. which version of k8s do we support
8. do we support Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS), and Amazon Elastic Kubernetes Service (EKS) ?

[sichend]

Thanks for setting up the first issue and kick off the discussion. Great topic! Here are some of my initial thoughts.

What are your assumptions or prerequisites?
The prerequisites are largely depending on the target customers of Dashboards Anywhere platform. If we targeting self-hosted customers, having access to a cloud service, such as AWS, Google Cloud, or Azure, would be a prerequisite.

What are remaining open questions?

What's options for Infrastructure as code(CDK | terraform | ??)
If the goal of the product is to be able to host Dashboards service on any cloud services, the maintenance cost will be higher if we directly use CDKs from each cloud platform. Terraform is an open source platform that provide plugins for different cloud providers. However, we do need to separate the responsibility of infrastructure and service. For example, VM to host service V.S containers that is actually part of the service V.S the service topology. Depending on the underlying platform to implement the Dashboards Anywhere, Helm chart could also be a solution to define service structure and deployment. We currently have the OpenSearch Helm project for Kubernetes deployments. How could we leverage this project as part of the Dashboard Anywhere initiative.

What's story for update software?
We will support rolling update for software in a stack. Operators can easily sandbox testing, rolling update at configure pace, monitoring and rolling back comfortably during an update deployment without impact to the availability of the

How could we handle specific user's config at scale?
From service experience perspective, we would like to be easily override the configuration for each clusters. This is somehow similar to the update and patching software experience. Operator could easily manage the configurations, sandbox testing, roll out new configurations, tracking deployed versions, and roll back confidently.

What's log, metric, and distributed tracking story cross all components, could we dogfood our own stack?
In order to provide a platform that is production-ready, we will need to consider logs, metrics, tracing, notifications, backup/restore into the platform. The questions how would the extendibility look like with these components. How could we define a extendable interface and use different types of plugins to implement the concrete integration with different service providers, for instance, plugin for object storages for backups and logs, Slack integration for notifications, and so on. This naturally leads to the question of how we can build an extendable plugin platform and how does developers and operators to use the Dashboards anywhere system to develop, deploy, and manage the plugins in/for productions systems. Kubernetes is a great example for a successful plugin system both enabling developers and simply operations.

What's project roadmap?
The Dashboards Anywhere vision is grand.

1. We can start with incremental progress of define the underlying service to implement Dashboards Anywhere. Being able to easily setup a Dashboards cluster in major cloud service providers. An important goal is to join Cloud Native Computing Foundation and being certified by CNCF
2. Considering defining the plugin mechanism and platform, and implementing initial plugins for log, metrics, backup/restore, and notifications.
3. Defining the rolling update experience without affecting cluster performance.
4. How could we utilize this platform to extent the product offering to benefit to a larger customer groups in the format of SaaS. At this stage, we can be a customer of our own Dashboards Anywhere platform, and managing large scale Dashboards clusters.

[seraphjiang]

cli for dashboards service infra
#2

[seraphjiang]

Thanks for setting up the first issue and kick off the discussion. Great topic! Here are some of my initial thoughts.

What are your assumptions or prerequisites? The prerequisites are largely depending on the target customers of Dashboards Anywhere platform. If we targeting self-hosted customers, having access to a cloud service, such as AWS, Google Cloud, or Azure, would be a prerequisite.

The dashboards is design to support cloud native but we want to provide solution for on-premise as well. We will start from support k8s which is supported by most of cloud provide as well as on-premise deployment.

Self-hosted customers doesn't mean conflict with cloud. Our understanding is customer choose to deploy their own cluster other than use the managed service.

What are remaining open questions?

What's options for Infrastructure as code(CDK | terraform | ??) If the goal of the product is to be able to host Dashboards service on any cloud services, the maintenance cost will be higher if we directly use CDKs from each cloud platform. Terraform is an open source platform that provide plugins for different cloud providers. However, we do need to separate the responsibility of infrastructure and service. For example, VM to host service V.S containers that is actually part of the service V.S the service topology. Depending on the underlying platform to implement the Dashboards Anywhere, Helm chart could also be a solution to define service structure and deployment. We currently have the OpenSearch Helm project for Kubernetes deployments. How could we leverage this project as part of the Dashboard Anywhere initiative.

We need to explore a little bit here to pick up IoC, there might be multiple answer. We will choose specific one first to gain knowledge, then convert to generic solution. It won't be perfect since day one.

What's story for update software? We will support rolling update for software in a stack. Operators can easily sandbox testing, rolling update at configure pace, monitoring and rolling back comfortably during an update deployment without impact to the availability of the

sounds good update story, let's create issue to track effort

How could we handle specific user's config at scale? From service experience perspective, we would like to be easily override the configuration for each clusters. This is somehow similar to the update and patching software experience. Operator could easily manage the configurations, sandbox testing, roll out new configurations, tracking deployed versions, and roll back confidently.

What does each clusters mean here?
there should be only one clusters for dashboards service with multiple nodes. The artifacts of each nodes are same, but config might be different per user, or other group rule.

What's log, metric, and distributed tracking story cross all components, could we dogfood our own stack? In order to provide a platform that is production-ready, we will need to consider logs, metrics, tracing, notifications, backup/restore into the platform. The questions how would the extendibility look like with these components. How could we define a extendable interface and use different types of plugins to implement the concrete integration with different service providers, for instance, plugin for object storages for backups and logs, Slack integration for notifications, and so on. This naturally leads to the question of how we can build an extendable plugin platform and how does developers and operators to use the Dashboards anywhere system to develop, deploy, and manage the plugins in/for productions systems. Kubernetes is a great example for a successful plugin system both enabling developers and simply operations.

good questions, let's find answers for dog food ourself for log and analytic solution

What's project roadmap? The Dashboards Anywhere vision is grand.

1. We can start with incremental progress of define the underlying service to implement Dashboards Anywhere. Being able to easily setup a Dashboards cluster in major cloud service providers. An important goal is to join Cloud Native Computing Foundation and being certified by CNCF
2. Considering defining the plugin mechanism and platform, and implementing initial plugins for log, metrics, backup/restore, and notifications.
3. Defining the rolling update experience without affecting cluster performance.
4. How could we utilize this platform to extent the product offering to benefit to a larger customer groups in the format of SaaS. At this stage, we can be a customer of our own Dashboards Anywhere platform, and managing large scale Dashboards clusters.