[D&D] Visualization type pipeline

[ashwin-pc]

This is a child task to track the implementation of "Expressions" in the Visualization service that defines how the visualization is rendered. Original Design.

The visualization service is responsible for two things.

1. Defining the UI fields for each visualization type
2. How the values set for those fields result in a visualization.

Here we discuss how that latter of the two will work.

Approach

To render the visualizations, we will be utilizing the expressions plugin. Once the UI fields are set using drag and drop or the inputs defined, the visualization service passes the state information to the viz type and gets an expression pipeline string in return. The visualization is responsible for translating the UI parameters into the appropriate expression. This expressions is fed into the workspace’s expression renderer to render the visualization.

Pipeline

Since expressions are quite flexible in what they can render and will be specific to each visualization, I will only cover how the pipeline for the default visualizations will work since they are similar. This is the proposed structure of that pipeline:

opensearchDashboards 
| opensearchaggs <aggregation_arguments>
| viz_fn <visualization_arguments>

It is similar to the pipeline used by the current visualize app. The different functions of the pipeline work as follows:

• opensearchDashboards: Used to pass on the search context information to perform aggregations.
  • Input Type: null
  • Output type: opensearch_dashboards_context
• opensearchaggs: Uses the dashboards search context and aggregation config to fetch the appropriate aggregation data and pass it on as a data-table.
  • Input Type: opensearch_dashboards_context
  • Output type: opensearch_dashboards_datatable
• viz_fn: This is the visualization specific expression render function. OpenSearch Dashboards has registered functions for all the visualizations currently supported in the application.
  • Input Type: opensearch_dashboards_datatable
  • Output type: render

All the visualizations planned in the initial implementation will use this pipeline but it is not necessary that visualizations follow this pipeline and any expression pipeline that can be rendered by the expression renderer is allowed.

Constructing the pipeline

Since the pipeline is quite specific to the visualization, here are some of the general principles on how the default visualizations will construct it.

For the default visualizations, this will rely heavily on how the current pipeline for visualize is constructed in build_pipeline. Based on the fields selected, the AggService will map the aggregations to aggconfig objects that will be passed on as arguments to the opensearchaggs expression function. The AggService will also provide the mappings that can be used to select the correct data from the resulting data-table. The resulting data-table with the mappings will be passed on as in input and vizConfig parameters to the viz render function respectively. The viz render function will use the data-table and mappings to render the visualization

FAQ

Custom visualizations

Adding custom visualizations is as easy as registering a new expression render function and using a pipeline that ends with the custom render function.

Other OpenSearch clusters

All the logic for connecting to other opensearch clusters will be introduced directly in the index management data model and encapsulated within the data plugin. The Drag and Drop app and other plugins that rely on this should not have to change their logic to support it.

Other non OS datasources

There is still no clear roadmap on how this will be implemented and is too early to factor this into the Drag and Drop design

Support for SQL/PPL or other query languages

There are two instances where query languages can be introduced in OSD

1. In the search bar that the user can input
2. In the query generated by the data plugin to fetch data

TL;DR: Since most of the logic to support these changes will be in the data plugin and in the OpenSearch cluster, there should be little to no changes needed for Drag and Drop to support these changes.

Long version. For both options the flow used by the Drag and Drop app will be similar to that of the Visualize app and any changes that need to be made to support these languages should work for Drag and Drop.

For the first instance, the query from the search bar is passed onto the opensearchaggs which is responsible for using that and aggregating for the appropriate data. This is also the expression function used by the existing visualize app and any modifications that need to be made outside the data plugin to support this will also work with Drag and Drop.

In the second instance, it is an under the hood implementation detail of how the data plugin converts the aggregation requests into queries. The API’s used by plugins will likely not change, and if they do, there should be a roadmap for how plugins need to migrate their API’s and drag and Drop would simply follow the same since this is a breaking change that isn't limited to it.

Open Questions

• How to support sub aggregations? When and how are they used in OpenSearch today
• What is a good level of abstraction to combine some of the functions in building an expression pipeline for the default visualizations
• How to map split row/split columns and other such complex combinations of data visualizations from UI to expression