[Design] Drag & Drop Visualization CX

[ashwin-pc]

Background

Today, visualization and exploration in OpenSearch Dashboard is harder than it needs to be. Users need to preselect both the chart type and index pattern they will use to create a visualization. Additionally, users may need to navigate between multiple screens (e.g. discovery → visualize → discover → visualize, etc.) to find the right feature, chart, index to include in their dashboard. Not only is these more work then it should be, it prevents the serendipitous discovery that easy exploration allows. This document outlines the high level design recommendation for a visualization experience that makes exploring data and creating visualizations much easier.

Tenets

• Minimal learning curve
• Easy of use
• No disruption to existing user workflow
• Get feedback early
• Progress not perfection

Out of Scope

• This design only covers the initial version of the tool and should not be seen as final
• This is only a high level design and doesn’t include component-level design

Recommended Design

The recommendation is to create the Drag and Drop experience as a first party plugin within the Opensearch Dashboards repository (i.e. explain to layman). It will act as an additional way to create visualizations alongside the exiting tools within the current visualizations plugin. The tool will be incremental to the visualization tools available to the user in OpenSearch Dashboards today. Due to the large nature of the design, I will divide it into 4 major sections:

• The user flow
• Tool features
• Architecture
• Aggregation logic

While we considered many options for each of our choices, we ultimately recommended this approach so that we could have minimal disruption to existing users workflow while also providing a seamless experience to the users who wish to create visualizations easily. It also lets us iterate and evolve over time.

The user flow

There are 3 main ways the user can reach the new tool.

1. Through the “New Visualization” option within the Visualize page.
2. Via the edit option for a visualization created by this tool in the dashboard
3. From the saved object viewer

For each of these flows the user is navigated to a full page view of the drag and drop tool that lets the user create or edit the visualization. After creating the visualization, the user can either choose to save the visualization or add the visualization directly to the preferred dashboard (existing or new).

Alternatives

1. Upgrade the existing visualize plugin to support the drag and drop:
   This approach is to update the exiting visualize plugin to be more easy to use and support some of the features this design proposes.
   1. Pros:
      1. No need to maintain two plugins
   2. Cons:
      1. Limits the scope of the features we can introduce. Some features such as the need to select the the visualization type before hand or the need to know the type of aggregation first go against the features introduced by this design. Changing it to align with the design breaks the experience for users that rely on the current experience.

Tool features

This tool features:

1. Top Nav: To search for data within an index pattern. This is identical to the top nav element within the Discover plugin and allows the user to search and filter the index pattern data for available fields and its data. It also allows the user to save the visualization as a saved object or save and add it directly to a dashboard.
2. Data Panel: A panel for accessing the Index pattern and its available fields. The data panel will feature an index pattern selector, a search bar to filter all the fields available within the index pattern for the given Top Nav filters, and the index pattern fields broadly subdivided (e.g. categorical fields, numeric fields and meta fields). The main features of this panel are:
   1. Index pattern selector: To select the index pattern
   2. Field filter: Filter the list of available fields. Helpful if the list of available fields is very large.
   3. Field summary: Each field, when hovered over or clicked, will show us a summary of the data in the field, similar to the discovery experience.
   4. Add to canvas: There will also be a button within each field to add the index pattern to the canvas.
3. Chart type selector: To allow the user to switch chart types on the fly, we will have a chart type selector. The initial implementation will prioritize bar (vertical and horizontal + stacked), line, area, and pie as well as metric and data table.
4. Visualization Config Panel: The panel is where the fields to visualize are added. It also has options to configure the type of aggregation for a field and serves as drop targets for the draggable fields.
5. Visualization Style Editor: A panel to edit the overall styles of the visualization such as the title, axis labels, legend, etc.
6. Visualization Canvas: The canvas serves as an area to see the final visualization. It also doubles as a drop target for the different draggable fields if a suitable aggregation is possible.
7. Drag and drop: All fields available in the Data Pane can be dragged and dropped into either the Visualization Canvas or the Visualization config panel. Doing so triggers an appropriate aggregation call described in the Aggregation section.

Architecture

Workflow

The app uses the fields selected or dropped into the chart context by the user to determine the best aggregation logic (outlined in a later section) to generate the appropriate DSL Query(s) and fetch the data to render the visualization.

Saved Object & Embeddable

The tool will introduce a new saved object type and embeddable to render its visualizations. This means that existing visualization will not be compatible with the new tool and separate factory. This is done to prevent breaking existing visualizations while allowing new visualizations to be created easily without much overhead.

Saved object structure

const wizardApp: SavedObjectsType = {
  name: WIZARD_SAVED_OBJECT,
  hidden: false,
  // ....
  migrations: {},
  mappings: {
    properties: {
      title: {
        type: 'text',
      },
      description: {
        type: 'text',
      },
      // Contains the state of the visualizaion
      state: {
        type: 'text',
        index: false,
      },
    },
  },
};

The one big drawback to this approach is not having the ability to edit existing visualizations using the new experience. Since the existing experience has a lot of visualization types that it already supports and a lot of customizations that we will not support for the initial launch and in future, reusing the existing schema will be rather tedious.

If the ability to edit existing visualizations is important, we can optionally provide an api/interface to selectively convert or migrate exiting and compatible visualization saved objects to the new saved object type. This will duplicate the existing visualization to the new format so that if there are any issues with the migration, the existing visualization remains unaffected. This approach also allows us to selectively enable migration for the visualizations that can be safely supported.

Alternatives:
Reuse existing visualization schemas: This approach lets us keep the existing schema and all new visualizations created using the new experience uses the existing visualization saved object type and embeddable. Supported visualization types will open the new experience when edited while the old experience is used for unsupported visualizations. This allows existing visualizations to use the new experience easily and allows wider adoption with no migration required, but not all options supported by the existing experience will be supported in the new experience. This may break workflow for current users. Additionally, this would break existing visualizations when edited and some existing visualizations may never be supported by the new experience (e.g. Vega). Lastly, inheriting a schema that is not optimized for the current experience may limit future expansion.

Charting Library

The app will use the EUI Charts library to create the default supported visualizations. This to to maintain visual parity with existing visualization while changing how they are generated. In future we expect all visualization to be created using this app.

While the default visualization types use EUI, there is no restrictions on what charting library can be used since each visualization type will be responsible for rendering itself and defining the fields and config types it supports it need to render correctly. In future this method can be used by other plugins to register their own visualization to exist alongside the default visualization types.

Drag and Drop

For the initial release there will only be 2 primary drop targets:

1. Visualization Config Panel
2. Visualization Canvas

The field cards can be dragged from the Data panel into either drop target and the page will automatically provide context to the user based on where the user is dragging the field.

Based on the dropped region an appropriate aggregation strategy will be used to fetch the data and render a visualization is possible. When a particular field cannot be dropped into a particular region, the drop cue does not display.

Aggregation Logic

One of the key problems we will need to address with the tool is how different interactions render a corresponding visualization that is both useful and makes sense to the user. Each existing visualization tool out there today does this in their own way and have their own strengths and drawbacks. To simplify this problem for our initial release and to get better user feedback on the approach we need to take here, my recommended is to allow the user to only be able to drag and drop a field into the canvas for the first field they wish to visualize. The Visualization generated will be based on a simple lookup below.

Field Type	Visualization Canvas
Numeric	median
Date	count of records (histogram)
String and others	count for top values

The Visualization Config panel will update with the appropriate fields and aggregation functions and the Visualization canvas will no longer act as a drop target. All subsequent fields need to be added directly to the Visualization Config Panel which assumes an appropriate aggregation function for the Chart axis and field type (similar to the existing visualize tool today). The User will also still be able to configure the visualization by Dragging and dropping fields directly onto the Visualization Config Panel. This approach makes the initial discovery of the data easy while keeping the reason for why a particular visualization was rendered intuitive.

Alternatives

1. The smart alternative: Similar to Tableau and Lens, we can allow the user to drop in any valid combination of fields into any valid drop target to generate an appropriate visualization. The biggest challenge here is to breakdown the problem such that it makes intuitive sense to the user why a particular visualization was rendered given the fields they dropped in. Given the number of drop targets, field types and aggregation functions Opensearch supports, the number of permutations quickly explode. Hence this approach is not recommended for the first iteration
2. The super simple approach: In this approach, we do not allow the user to drop anything into the canvas directly, but instead only allow them the pick the field they want to visualize directly in the Visualization config editor. This removes any ambiguity about what field and aggregation is used to render the visualization. It is not ideal since the user will now have to rely on the field summary in the data panel to explore the data, but it is still better than the experience today since the rest of the tool still makes intuitive sense.

The recommended solution is a meets both these approaches in the middle.

Future scope

1. Replace the most existing visualization tools and make this the default
2. Allow integration for other query languages
3. Migrate existing visualizations to the new tool
4. Ability to drag and drop any field into the canvas to create visualizations
5. Realtime smart suggestions for visualizations

Open Questions

1. Should we prioritize converting visualizations to work between the current visualizations experience and the proposed experience?
   1. Is the ability edit existing visualizations using the new experience more important?
   2. Is the ability to edit a visualization created using the proposed experience with the current experience important?
   3. Are they both important?
2. Visualization types to prioritize:
   1. Bar
   2. Line
   3. Metric
   4. Data
   5. Area
   6. Pi
3. How can we provide a preview or beta.
4. Is there anything in the future scope that we should add to the initial release
5. Any core feature we have missed for the initial iteration
6. Should this be a part of Opensearch Dashboards, the Dashboards-visualizations plugin, or be a separate plugin

[JacobBrandt]

Im only seeing index pattern as an option for the data panel. I've mentioned several times before that saved searches should be there as well. Especially since allowing the user to choose a saved search would have no impact to the generic design I had hoped was going to be built. Removing this ability would make this the only visualization that can't use saved searches just like Kibana did with Lens and Maps and seems needlessly limiting. Is the user going to be able to choose from a saved search as well?

The app will use the EUI Charts library to create the default supported visualizations. This to to maintain visual parity with existing visualization while changing how they are generated. In future we expect all visualization to be created using this app.

Wait, maintain visual parity with existing visualization? The majority of OpenSearch Dashboard visualizations DO NOT use EUI charts. When this was forked from Kibana, EUI Charts was barely used. I believe, if I remember right, only Discover and some of TSVB used EUI Charts. If you mean to say visual parity with latest Kibana, okay I can buy that. But is this visual parity requirement an Amazon need or a community need?

There's a lot to consider when bringing in another Elastic built repository now behind non open source licenses besides where the library will be hosted (this repo or separate). For instance, what are the number of bugs/regressions that are being inherited by creating EUI Charts from the version that can be forked? What's the plan to fix all of these regressions and bugs that have been inherited? Has the team considered the missing features in Lens that are missing because of its use of EUI Charts, which in turn limits this tools potential in the future? I'm curious, what alternative charting libraries were discussed with the community and team?

[ashwin-pc]

Thanks for the feedback @JacobBrandt! You have brought up some very good points here. Hopefully this addresses your concerns.

Im only seeing index pattern as an option for the data panel. I've mentioned several times before that saved searches should be there as well.

Sorry, I totally missed saved searches while writing up the design. Yes, using saved searches is definitely something that we should have and will introduce in future iterations. I’m hesitant to include it in the first iteration though because it does have some challenges that we need to solve for. I’d love to hear your thoughts if you disagree and think that we need this feature in the first iteration of this experience.

Wait, maintain visual parity with existing visualization? The majority of OpenSearch Dashboard visualizations DO NOT use EUI charts. When this was forked from Kibana, EUI Charts was barely used. I believe, if I remember right, only Discover and some of TSVB used EUI Charts.

Shoot you are right! Thanks for catching that. TSVB also happens to be the chart type that I looked into while writing this. I will correct that text. What i meant was that we will use the same chart components used by the visualize plugin today (which i had incorrectly assumed was EUI) to maintain visual parity.

But is this visual parity requirement an Amazon need or a community need?

I mention visual parity because the main goal of the drag and drop experience is to offer a much better way to explore the data and create visualizations, and not to immediately change the end visualizations created. This will help the user use this tool as a drop in replacement for the existing visualization plugin. That being said, this decision does not limit us in future from moving to a different visualization library since the visualization layer will be abstracted away from the core of the experience.

I'm curious, what alternative charting libraries were discussed with the community and team?

We did consider alternatives such as Plotly.JS and i’m pretty certain we will migrate away from the default charting libraries in future. But for reasons mentioned above I don’t recommend us doing so initially.

[ahopp]

But is this visual parity requirement an Amazon need or a community need?

To add on to what @ashwin-pc mentioned, we're trying to make it consistent with the current experience from an iteration perspective - I am personally advocating the "progress not perfection" path and we saw this as a good first step. This is not driven by any particular vendor.

Especially since allowing the user to choose a saved search would have no impact to the generic design I had hoped was going to be built.

Nothing is preventing this from being added, but we're trying to prioritize an initial version. But maybe this would be a good place to plug in on the dev side @JacobBrandt?

[JacobBrandt]

@ashwin-pc @ahopp Thanks for responding, and sorry for not reaching back out until now.

RE: Saved searches
I disagree with waiting to have this feature because I agree with this statement.

This will help the user use this tool as a drop in replacement for the existing visualization plugin.

Since we all agree with this statement above perhaps we can figure out what challenges are causing this feature to get cut from the design that is presented here. If it's not there from the start you've cut away half of the capabilities from all the other visualizations and makes adoption much harder for end users. From experience, using saved searches in visualizations has been relatively easy and straight forward. For example, the only difference from a search standpoint between using just an index pattern is that the SearchSource making the request should include the SearchSource of the saved search as its parent.

I honestly think visual parity will present more challenges than saved searches and I'm confident that we can figure something out to have saved searches right away.

RE: Visual parity
Ah ok that explains it. My response to this was related to the design saying this tool will use EUI charts to have visual parity. I get why the design would want visual parity but was confused by that statement which is why I was pointing out that using EUI charts doesn't really achieve visual parity and was wondering if there was some other reason I was missing for why it was going to be used. Thanks for clearing that up.

[ashwin-pc]

RE: Saved Searches

@JacobBrandt yes, saved searches should in theory be relatively straightforward. I'll admit that I've limited experience with it which could also be the reason for most of my concerns, however there are two challenges it introduces with our current design that I would ideally like to solve before introducing it to the experience.

1. The UX: Adding the option to pick a saved search from the data source picker should be relatively simple. However the other UI elements need to be ironed out such as:
   • letting the user know if the visualization is linked
   • how the visualization changes when the data source is changed and most importantly
   • what the saved search parameters currently are etc.
     While the last is not strictly needed because we don't have it today, it was something that confused me initially and still makes me jump between screens to view or modify the search. Given that this experience is trying to make such interactions unnecessary, I'd rather have it be implemented well than just introduce it for for the sake of parity.
2. The backend: True, this might be as simple as using the SearchSource of the saved search as its parent and that aligns with what i know about it. I haven't worked saved searches before so I am cautious about suggesting that is that simple too soon.

After reading your response and thinking about this a bit more, I believe that both of these concerns are relatively minor and can easily be solved. I think the question now is how important is this feature and should it be prioritized over some of the other features mentioned in the design. @ahopp what are your thoughts?

[JacobBrandt]

@ashwin-pc @ahopp One thing that is not clear here is how long external contributors will have to wait before providing enhancements and modifications to the baseline. As I was talking with another colleague about this I realized that this is the reason why I've been concerned about saved searches and brought it up.

I might be misunderstanding this, but right now it seems like external contributors would have to wait until this design, as outlined, is finished. I say this because the design and the team talks about all these things being necessary in the first iteration/initial version and other future work, including saved searches, could come later when it's complete. I take that as everything as outlined in the design will become the first publicly accessible baseline to end users and external contributors. There's a lot there to try and accomplish in one iteration before external contributions. Ideally, I'd rather have a more iterative approach with the community where the generic abstractions get pushed in place sooner rather than later so that external contributors don't have to wait. Thus mitigating the risk of waiting indeterminate amount of time to help with development and provide additional capabilities like saved searches.

@ahopp You mentioned could saved search be a good place to plug in. Yes it could, but because of the unknowns I mentioned above I was concerned about when that would be possible? If contributors don’t have to wait for everything as outlined in the design to be finished than this sounds like a great way to implement this feature and we could move forward that way.

[ashwin-pc]

@JacobBrandt I see your concern. You are right about the end user having to wait the design to be completed, but external contributors will not have to wait for that. We will be opening a branch on the Dashboards repo soon for the design and will have all the development happen directly on it so that it is accessible external contributors immediately. We will only hold merging it to the mainline until we are confident that we have an experience that we are comfortable sharing with users, but this should not limit contributors from either playing with it or contributing to it right away. We will also be opening the issues for each feature here. Does this alleviate your concern?

[JacobBrandt]

@ashwin-pc Perfect. That all sounds great. Sorry for my misunderstanding.

[jgough]

In the list of Available Fields to select from it would be extremely useful to replicate the Popular section as used on the Discover tab:

This is an extremely useful feature for both novice and advanced users as it helps them easily locate the fields they require most often very quickly and easily.

[tmarkley]

My comments/questions:

• Thanks for adding the alternatives for each of the sections! Can you have pros/cons lists for each of them to make that information easier to parse?
• Can we clarify what features we want to be in and out of scope for the initial release (this could be as simple as two lists)? A quick justification and priority for those would benefit the design in my opinion. Including each of the features/requests from @JacobBrandt in his previous comment would be great: Drag & Drop visualization creation #379 (comment)
• Can we create a meta issue that links each of the remaining issues to track the work that needs to be done and provides the community with a way to view progress and pick up work themselves? I know the label exists but I think it would be worthwhile to have a dedicated issue.

• Should we prioritize converting visualizations to work between the current visualizations experience and the proposed experience

  • My short answer would be yes. If our long-term plan is to Replace the most existing visualization tools and make this the default (also, what do you mean by "most"?) then I think migration options are required.

• How can we provide a preview or beta.

  • @kgcreative has mentioned wanting to work on defining how to release a "preview" or "beta" features and what the process should be for them. I'm happy to contribute as well. It'd be great to have a proper mechanism to gather feedback from the community and iterate over time (while still remaining backwards compatible).

• Should this be a part of OpenSearch Dashboards, the Dashboards-visualizations plugin, or be a separate plugin

  • If the goal is to have this become the default option then wouldn't it be best for this to exist in the Dashboards repository? I think that's what you're going for here, but just wanted to confirm.

[ahopp]

Can we create a meta issue that links each of the remaining issues to track the work that needs to be done and provides the community with a way to view progress and pick up work themselves? I know the label exists but I think it would be worthwhile to have a dedicated issue.

Sure! @tmarkley is there a example issues you can share that we can use as a template to what you'd like to see?

Can we clarify what features we want to be in and out of scope for the initial release (this could be as simple as two lists)?

I think you're describing a list of in-scope features plus a backlog? If so, I think we can do something like that. There are few features we'd be happy to add for the initial release if someone wants to build them, but are prioritizing other features with available resources so the backlog might be more like aspiration P1/P2 features rather than a traditional backlog. Where would you want to see this? The meta issues?

It'd be great to have a proper mechanism to gather feedback from the community and iterate over time (while still remaining backwards compatible).

100% agree. We should propose something soon.

Also as an FYI we're planning on closing the design on 12/17. We can obviously continue to iterate but I think the high-level design has been resolved and we're already working on development.

[tmarkley]

Sure! @tmarkley is there a example issues you can share that we can use as a template to what you'd like to see?

It doesn't have to be anything fancy, a prioritized list of tasks and regular status updates would be great. #920 is the first thing I thought of that was similar.

I think you're describing a list of in-scope features plus a backlog?

Yeah that would be great.

... if someone wants to build them ... Where would you want to see this? The meta issues?

Yes I think that would work to have it in the meta issue(s). I think we should make sure community members understand what is being prioritized and what else they can pick up and contribute to for the project.

[ashwin-pc]

Thanks for the feedback everyone. To summarize the conversation:

1. Saved searches are not in the initial proposal, but given the iterative approach we are taking, that should not be an issue. We can prioritize it in a future release or can be picked up by the community.
2. The design mentions that we will use eui as the charting library of choice. This is incorrect, we will strive to achieve visual parity. The library we will use will most likely be the one currently used by visualize.
3. All the features mentioned in the design are still in-scope for the initial release. All other features such as saved searches are not but they can be picked up by the community if needed.
4. A meta issue to link all the subtasks under Drag and Drop and track them will be created.