[RFC] What type of visualization abstractions do developers need?

[joshuarrrr]

Is your feature request related to a problem? Please describe.

OpenSearch Dashboards should provide users with a consistent, unified, and intuitive visualization experience throughout the application. To achieve that goal, we need better, easier-to-use visualization tools and interfaces for the developers building and creating those experiences.

Currently, it's difficult for developers to interact with and leverage the existing visualization plugins and frameworks in OpenSearch Dashboards, and the abstractions we do have don't clearly map to actual developer needs. Before we begin developing new abstractions, we first want to better understand, collect, and categorize all the existing visualization pain points and needs.

Describe the solution you'd like

Why invest in an abstraction layer?

1. Allows opinionated and consistent design decisions "baked-in"
2. Prevents dependency lock-in - we'll be better positioned to migrate to or support other charting libraries in the future
3. Frees feature designers and developers from low-level implementation details, which increases feature velocity

Some initial potential abstraction needs we've identified:

1. Opinionated components and interfaces that implement/inject the OpenSearch Design System visualization guidelines (look and feel, interaction patterns)
2. Visualization theming interfaces (e.g. for consistent color assignment across multiple visualization types. See also #1165, #1241, #1466, #1775)
3. High-level React components for making common visualization creation easy, with minimal configuration
4. A single wrapper interface/integration point for annotating, extending, and otherwise modifying existing visualizations
5. Generic web components to empower server-side extensions
6. Guidance or contracts for plugins/features that use alternative rendering libraries
7. New visualization type plugin scaffolding/templates

Did we leave something out? Do you feel strongly about one of these areas or have solution ideas in these areas? Let us know in the comments and we'll update and expand the list.

Describe alternatives you've considered

Abstractions are not free; they carry a an additional cost to build, maintain, and document features that may already be available in the underlying charting library or rendering layer. But without an abstraction layer, all feature developers would need to learn how to effectively use the Vega-Lite grammar. They would also need to be familiar with the OpenSearch Design System, and be responsible for implementing it correctly and consistently.

Additional context

Much of the need for visualization abstractions came out of research into consolidating OpenSearch Dashboards visualization rendering to Vega-Lite.

[seanneumann]

@ahopp - Do you have some thoughts about circulating this to encourage some more feedback?

[ahopp]

@seanneumann @joshuarrrr I think we should try to get on the calendar for the Community Meeting. Could we also highlight this issue in other relevant repos?

[ahopp]

I also think we can summon some of the folks that have had opinions previously... like @jkowall @galangel @JacobBrandt @AmiStrn might have some good perspective! ;)

[ahopp]

Couple of questions/ideas relative to the current requirements;

• Should we have requirements around testing at functional and visualization levels, i.e., unit and visualization integration and regression testing available in the abstraction layer?
• Should we add some requirements specifying how customizable the end state will be? We will likely get feedback that exposing the highest layer of abstraction may not enough to facilitate the development of complex visualizations (it may be argued that in some scenarios it becomes necessary to modify functionality rooted at different levels of abstraction), so should we add some specification on where the customization can happen (outside of direct contribution to the OpenSearch Dashboards)?
• Is it to early to add requirements for co-locating code and documentation for any abstraction layer(s)?
• Should we add a specific requirement around interoperability with different usage models, e.g., do we specific that it will need to support plugins and/or extensions as well as core.
• Should we add a specific requirement around the need to support new contributions to abstraction in a central manner, e.g., this will be in OpenSearch Dashboards and be treated as a platform feature?

Lastly, is there a technical reason we call out "High-level React components" specifically? Seems like React is a implementation detail and the core requirement is the availability of a library of common components with minimal configuration requirements. Just curious on the specificity here.

[jkowall]

Thanks for putting this together. I will pass this off to our architect who is working on the dashboards side more, and he can bring in the right engineers. I, personally, don't have any comments since this is too detailed for my depth of knowledge on how Dashboards works.

Thanks.

[ashwin-pc]

2 features I'd like to call out is two way databinding and fine grained drag and drop support. e.g.

1. Clicking or drawing on the chart should result in actions/events that the application can use
2. Dragging and dropping items in specific target drop locations on the visualization must be possible.

[AmiStrn]

I would like a way to add lookup lists so that uuid's returning from log data can be replaced with human readable data such as "user name".