[FEATURE] Dashboards Visualization Integration to Operational Panels

[ps48]

Operational Panels support for OSD Visualizations

NOTE: Please make sure you go through the glossary before diving into the design with “dashboards” word being overloaded things can get pretty confusing.

Glossary

• OS → OpenSearch
• OSD → OpenSearch Dashboards
• PPL → OpenSearch Piped Processing Language
• DSL → OpenSearch query domain-specific language
• Dashboards → Dashboard plugin in OpenSearch Dashboards
• OSD Visualizations → Visualizations that are created with Visualize plugin in OpenSearch Dashboards
• PPL Saved Visualizations → Visualizations that are created in Events Analytics component of Observability Plugin using PPL
• PPL Runtime Visualizations → Visualizations that are created using PPL Query without being saved in Events Analytics. These include visualizations from different data sources like Prometheus
• Operational Panels → Observability plugin component to view PPL visualizations in a single pane.

Overview

Today, Operational Panels enables users to dive into multiple PPL saved visualizations for RCA analysis, do regular monitoring of visualizations derived from logs for a given time frame and compare these saved visualizations with global PPL filters. This design document introduces new support for existing OSD visualizations in Operational Panels. We dive into implementation design on pushing down PPL filters, date filters and UI filters(discussed more below) to both PPL Saved Visualizations and OSD Visualizations.

Motivation

Observability Plugin was introduced in OpenSearch 1.2.0 version, this included original components of Trace Analytics, Notebooks, Operational Panels and Events Analytics. With the latest released of 2.4.0 version of OpenSearch, the Observability plugin stands with new components like Metrics Analytics and App Analytics along with many small but high value features like Log-Trace Correlation, Log Patterns with Anomalies, View surrounding documents, Live Tail, etc.

The OpenSearch community has many customers who are legacy users using our offerings from Open-Distro for ElasticSearch. They have pre-built Dashboards and OSD Visualizations for all of their workloads. It is difficult for these users to switch to the Observability Plugin even when they upgrade to the latest OpenSearch versions. The transition after upgrade is difficult due to the following reasons:

1. Lack of a common place that can support both OSD visualizations to PPL visualizations.
2. Time and effort required by users to convert Dashboards to Operational Panels.
3. Time and effort required by users to convert OSD visualizations to PPL visualizations.

With Operational Panels supporting both PPL and DSL visualization, more users will be open to adopt the Observability plugin. This would be the first step from our users to use the new Observability Plugin features, pointing towards increase in adoption of the Observability plugin.

Requirements

Functional Requirements

1. As a user, I should be able to view existing OSD Visualizations to be added in the Operational Panels side menu
2. As a user, I should be able to view OSD Visualizations in preview before adding to Operational Panels
3. As a user, I should be able to resize OSD Visualizations in Operational Panels
4. As a user, I should be able to drag & drop OSD Visualizations in Operational Panels
5. As a user, I should be able to use PPL filters for OSD Visualizations
6. As a user, I should be able to use time-filters on OSD Visualizations
7. As a user, I should be able to save Operational Panels with OSD Visualizations
8. As a user, I should be able to view data from OSD Visualizations, similar to how it is done is notebooks
9. As a user, I should be able to move to the visualize window for editing the selected OSD Visualization
10. As a user, I should be able to add text filters, similar to how it is done in dashboards apart from the existing PPL Filters.
11. As a user, I should be able to enable all, disable all, pin all, unpin all, invert inclusions, invert enabled/disabled and remove all text filters.
12. As a user, I should be able to remove individual text filters on the Operational Panels view page.

Non-Functional Requirements

1. A new visualization container should be created for Operational Panels with support of Query Manager.
2. The new visualization container should support OSD Visualizations, PPL Saved Visualizations and PPL Runtime Visualizations.
3. The new visualization container created for Operational Panels should be re-used for all Observability components Event Analytics, App Analytics, Notebooks and Metrics Analytics.

UI Mockups



Architecture

Components

1. Panels Visualization Grid

The react grid container component comprising of all the visualization containers. This grid component uses React-Grid-Layout(RGL) npm module to create resizable, drag, and drop features. RGL is used by Operational Panels as well Dashboards for easy movement features. We look at re-using this container from Operational Panels to fit in OSD visualizations.

Today, Observability component Notebooks has a similar usage for OSD Visualizations. It spawns up a RGL component and fits in the OSD visualizations this concept is OSD is known as Embeddables(more on Embeddables in Appendix 2.). While, this approach is good for single visualizations, it is not performant when we have multiple visualizations created with their own RGL container. To have good load time performance we need to use the visualizations layer directly rather then using the Dashboards Container.

2. Combined Visualization Container

• As of OpenSearch version 2.4.0, we have 2 observability visualization rendering components: One for Events Analytics that is well integrated with the Query Manager and Second is a common Viz. Container for Operational Panels, App Analytics, Notebooks and Metrics Analytics.

2.1 PPL Saved Visualizations

• These are user generated from Logs in Events Analytics component of Observability. Users write down a PPL Query or use the drag & drop feature along with other advanced options to create this visualization. The visualizations meta-data (Query, index, time-range) are stored in the observability index.

2.2 PPL RunTime Visualizations

• These visualizations are automatically generated with external data sources from the SQL federation engine. For example, the Prometheus use-case in Metrics Analytics uses Runtime Visualizations to show Prometheus metrics in Event Analytics. The meta-data for these visualizations is not saved rather gathered in run-time and they use default visualization options for display.

2.3 OSD Visualizations

• The visualizations created by users using the visualize plugin. The visualizations are then further used in Dashboards plugin via embeddable framework.
• Notebooks too, uses these OSD visualizations via the embeddable framework and rendering the viz, with a dashboard container component.

2.4 Query Manager

• This component parses the UI component based on Antlr, which acts as a wrapper sitting on top of Antlr solution for managing internal modules, and exposes various interfaces for query parsing/building use cases to consumers.
• Events Analytics visualization container has a tight integration with Query Manager for all its visualizations. We should use this integration for all the visualization components in Observability.
• PPL Saved Visualizations - Query Manager would parse PPL Query and the stored meta-data to finalize visualization rendering components.
• PPL Runtime Visualizations - Query Manager would parse PPL Query and use the default meta-data to finalize visualization rendering components.
• OSD Visualizations - Currently these viz. do not support PPL Query, but this integration will be helpful in future.

3. Language Translator Module

We will two different types of filter in Operational Panels:

• PPL Query Filter: This is the PPL where clause filter that is present today in Operational Panels. It pushes the where clause to all the PPL Saved Visualizations.
• Text Filter: This is the new filter that we are creating based on the implementation in Dashboards. This filter is created by the user with a UI pop that enables them to add match queries or range queries to their visualizations. The filter results in a DSL Query that is then pushed to all visualizations. This usually consists on match and range queries.

We have PPL and DSL working with their own visualizations and push downs. Now, to combine these filters and push them down in individual visualizations we need to use language translation module. We’ll have 3 layers of language translation:

3.1 Text to PPL

• This converts text filters from the UI component to PPL Query. We would need to support functions as follows: match, range, exists and negation. This layer is mainly used to push down text filters to PPL Saved Visualizations.

3.2 Text to DSL

• This layer converts text filters from the UI component to DSL Query. This has been already implemented in OSD. We should plan to re-use the whole layer as is. We need this layer to push down text filters to OSD Visualizations.

3.3 PPL to DSL

• This layer converts PPL where clauses to DSL. This is needed to push down PPL clauses to OSD visualizations.
• Already SQL plugin. We need see how to do it for where clauses

3.4 Combinator

• This layer combines the PPL clauses with Pipe and the DSL queries with bool functions for respective push downs in the visualizations.

User WorkFlows

View & Add Visualizations

When users request for available visualizations, we make 2 requests: One for the OSD Visualizations using a custom route(already existing in notebooks) to query the .kibana index and the other for Saved PPL Visualizations using Observability route(already existing in Operational panels) to query the .opensearch-observability index.

Adding Visualization workflows enable users to add OSD Visualization or PPL Saved Visualization to Operational Panels. When an user clicks on the add visualization button they have two options either add an existing visualization or create a new visualization. With this new feature, the add existing visualization option will show both the OSD Visualizations and the PPL Saved Visualizations in the dropdown list. Once the users select and add either of them, the backend would saved the visualization ID and other meta data (mentioned in the data model).

Edit Text/PPL Filters

When users edit the text filters the language translator module in the Observability UI kicks in. Post translation, we push down the respective filters to individual visualizations.

Data Model

Current Data Model for Operational Panels

// OperationalPanel main data class.
// JSON format
   {
     "operationalPanel": {
       "name": "Demo Panel 1",
       "visualizations": [
         {
           "id": "panelViz_7ba28e34-6fd8-489d-9b9f-1f83e006fb17",
           "savedVisualizationId": "oyuecXwBYVazWqOOde0o",
           "x": 0,
           "y": 0,
           "w": 10,
           "h": 10
         },
         {
           "id": "panelViz_7ba28e34-6fd8-489d-9b9f-165fdv6wd611",
           "savedVisualizationId": "oiuccXwBYVazWqOO1e06",
           "x": 20,
           "y": 20,
           "w": 30,
           "h": 20
         }
       ],
       "timeRange": {
         "to": "now",
         "from": "now-1d"
       },
       "queryFilter": {
         "query": "| where Carrier='OpenSearch-Air'",
         "language": "ppl"
       },
       "applicationId": "KE1Ie34BbsTr-CsB4G6Y"
     }
   }
 }

New Data Model

// OperationalPanel updated main data class.
// JSON format
   {
     "operationalPanel": {
       "name": "Demo Panel 1",
       "visualizations": [
         {
           "id": "panelViz_7ba28e34-6fd8-489d-9b9f-1f83e006fb17",
           "savedVisualizationId": "oyuecXwBYVazWqOOde0o",
           "x": 0,
           "y": 0,
           "w": 10,
           "h": 10, 
           "visualizationType": "observability"
         },
         {
           "id": "panelViz_7ba28e34-6fd8-489d-9b9f-165fdv6wd611",
           "savedVisualizationId": "oiuccXwBYVazWqOO1e06",
           "x": 20,
           "y": 20,
           "w": 30,
           "h": 20,
           "visualizationType": "dashboards"
         }
       ],
       "timeRange": {
         "to": "now",
         "from": "now-1d"
       },
       "queryFilter": {
         "query": "| where Carrier='OpenSearch-Air'",
         "language": "ppl"
       },
       "textFilter":[
            {
              "meta": {
                "params": {
                "gte": 200,
                "lt": 5000
                },
                "alias": null,
                "negate": false,
                "disabled": false,
                "type": "range",
                "key": "bytes",
                "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index"
             }
           }
       ],
       "applicationId": "KE1Ie34BbsTr-CsB4G6Y"
     }
   }
 }

OpenSearch Observability APIs

TBA

Milestones

Project	Estimate	Owner	Comments
[PoC] - OSD viz in panels with dashboards container	2 weeks	@ps48	MVP without filter pushdown
OSD viz in panels without dashboards container			Optimized UI container without Dashboard Container, with filter pushdown removed in favor of above PoC
Add Text filter to Panels UI	1 week		Re-use text filters react component from Dashboards Core
Language Translater Text filter to DSL/PPL	4 weeks
Language Translater PPL to DSL	3 weeks		Re-use PPL push-down translation from Engine backend
Combined Visualization Container with Query Manager	2 weeks		Combine charting/viz react components for all Observability components
Add OSD viz support to Combined Visualization Container	1 week
Combine components for end to end workflow	1 week
Add Jest & Junit unit tests	3 days
Add Cypress & Integration tests	1 week
Bug Bash - Pre release	2 days

Appendix

1. Dashboards Data model to save DQL & Text filters

• The DQL queries are saved in query field. Whereas rest of the Text filters are stored in filter field as array of objects
• These Data model object can be viewed in Stack management → Saved Objects → Type:(dashboards)

// Dashboards Data model to save DQL & Text filters
    {
      "query": {
        "language": "kuery",
        "query": ""
      },
      "highlightAll": true,
      "version": true,
      "filter": [
        {
          "$state": {
            "store": "appState"
          },
          "meta": {
            "alias": null,
            "disabled": false,
            "key": "bytes",
            "negate": false,
            "params": {
              "gte": 200,
              "lt": 5000
            },
            "type": "range",
            "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index"
          },
          "range": {
            "bytes": {
              "gte": 200,
              "lt": 5000
            }
          }
        },
        {
          "$state": {
            "store": "appState"
          },
          "exists": {
            "field": "agent"
          },
          "meta": {
            "alias": null,
            "disabled": false,
            "key": "agent",
            "negate": false,
            "type": "exists",
            "value": "exists",
            "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index"
          }
        },
        {
          "$state": {
            "store": "appState"
          },
          "meta": {
            "alias": null,
            "disabled": false,
            "key": "geo.src",
            "negate": false,
            "params": {
              "query": "US"
            },
            "type": "phrase",
            "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[2].meta.index"
          },
          "query": {
            "match_phrase": {
              "geo.src": "US"
            }
          }
        },
        {
          "$state": {
            "store": "appState"
          },
          "meta": {
            "alias": null,
            "disabled": false,
            "key": "memory",
            "negate": true,
            "params": {
              "gte": 20,
              "lt": 30
            },
            "type": "range",
            "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[3].meta.index"
          },
          "range": {
            "memory": {
              "gte": 20,
              "lt": 30
            }
          }
        }
      ]
    }

2. Dashboards Embeddable API & Embedding Visualizations

Embeddable API

• Embeddables are re-usable widgets that can be rendered in any environment or plugin. Developers can embed them directly in their plugin. End users can dynamically add them to any embeddable containers.
• Containers are a special type of embeddable that can contain nested embeddables. Embeddables can be dynamically added to embeddable containers. Currently only dashboard uses this interface.

Credits
Code: https://github.com/opensearch-project/OpenSearch-Dashboards/tree/master/src/plugins/embeddable
README: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/master/src/plugins/embeddable/README.md

1. Visualizations, Saved Search and Dashboard embeddable are part of this API now.
2. Embeddable Factory allows to create objects:
   1. with “.create()” menthod → needs input of data/source/query/time range explicitly
   2. with “.createFromSavedObject()” method → either inherits values from containers or takes from explicit input provided
3. Each of the above has a implementation has to inherit an embeddable & Factory API like:
   1. Viz. Embeddable & Factory
   2. Creating Custom Embeddable Example & Factory by Value
   3. Creating Custom Embeddable Example & Factory by reference
4. Visualizations Embeddable API Code
5. Dashboard Container is exposed as an embeddable - to have multiple embeddable in a GRID like structure just like the Dashboard Plugin.

Embeddable Examples

• Examples folder in Dashboards has all the usage samples for new APIs
• Use to create new embeddable objects
• Embeddable Examples shows how to create new embeddable inheriting the API
• Embeddable Explorershows usage of these embeddable examples in a Panel Container
• Dashboard Embeddable shows usage of these embeddable examples in a Dashboard Container

Embeddable Renderer

• The Dashboards react Element/Prop to create new embeddable objects: Code
• Embeddable container use the renderer to create/update each child(an embeddable object)
  • Example Dashboard Container
  • Example of Static Embedding (without factory)
  • Example of Embedding with factory.create() method(with factory)

Future Work

One-click migrations

• Automatic migrations from Dashboards to Operational Panels. We would convert saved Dashboards Objects from .kibana index and then merge them into Observability saved Objects.
• This would require more work on Object versioning and thoughts on to what version back do we support Open-Distro for ElasticSearch Objects.

Reporting Integration

• Dashboards works with OpenSearch Reporting plugin to generate PDFs and PNGs in run-time. We would extend the same functionality to Operational Panels.
• This would include passing time and filters using the URL params like it is done today in dashboards.

[ps48]

PoC demo -> Panels integration with embedded Dashboards Container.

panels_osd_integration_poc1.mov

Perfomance Stats

• panels-observability-viz-5: Represents Operational Panels with 5 Observability Visualizations
• panels-mixed-viz-5: Represents Operational Panels with 5 Visualizations with 60-40 split [i.e 3 dashboards visualization and 2 observability visualizations]

Panel Type	Loading	Scripting	Rendering	Painting	System	Idle	Total
panels-observability-viz-5	10	3187	85	26	1292	939	5539 ms
panels-mixed-viz-5	10	3316	117	25	1309	813	5590 ms
panels-observability-viz-15	11	3283	104	41	1048	1146	5633 ms
panels-mixed-viz-15	10	3485	141	38	1315	805	5794 ms
panels-observability-viz-50	10	3118	73	37	1560	927	5925 ms
panels-mixed-viz-50	10	3431	53	31	1484	1015	6024 ms

Notes:

• The performance run includes loading the whole page, this includes loading lots of JS libraries not relevant to the panels plugin.
• All mixed viz panels have 60-40 split for dashboards visualization and observability visualization respectively.
• All the numbers above have been taken as an average of 3 runs.
• Below is a sample run of performance on chrome dev-tools.

[ahopp]

Hey team! Sorry I'm late to the game on reviewing this feature proposal but now that I'm taking a look I had some questions.

First, it seems that the main thesis is that users have "pre-built Dashboards and OSD Visualizations for all of their workloads" but since the Observability Plugin has different patterns of visualizations specifically around PPL, but the conclusion seems to be backward to me. Rather than making users migrate from the tool they are using, why wouldn't we enable the tool to support PPL visualizations? It seems asking users to covert, migrate, or otherwise remediate to a new tool is not the preferred approach - am I missing something?

Second, while it is likely true that supporting both PPL and DSL visualization more users will be open to adopt the Observability plugin, would it not also be true that building this functions into the tool they've already adopted (i.e., Dashboard core) would be more amplifying to the user then splitting the experience further? It seems the goal shouldn't be the adoption of a plugin via navigation but rather the adoption of a plugin via use. And it seems the more direct path for supporting users using this functionality is generalization in the core experience not forking the user flows, correct?

Can you help me understand why we wouldn't being using the core dashboard experience as the baseline over over the observability plugin? If we're talking adoption and impact, it seems we're doing it in the opposite way I think would result in the higher usage.

[ps48]

@ahopp Thanks for taking time and adding these comments.

For the first point, Yes we have discussed these options and have not ruled them out. Implementing our features to the dashboards-core is an alternative approach of solving the same problem. We still need to dig the details of this. As a thought, It looks straight forward to move Observability Plugin Objects to the core but, we need to see how extensible are the existing component of dashboards-core. These include extending current filters option to support PPL with auto-complete, support for PPL visualizations to have DSL/DQL or lucene filters, etc. Also, we are already having discussions in the team to support PPL visualizations in dashboards-core. Will keep the thread updated once we finalize on it.

Secondly, The navigation experience would be changed soon. We already have a more holistic integration for the observability plugin happening with dashbords-core ongoing here:

1. [FEATURE] Dashboards List Extensions OpenSearch-Dashboards#1866
2. Dashboard Lists Integration OpenSearch-Dashboards#3090
3. [Feature] Dashboards-List integrations #151