Dataset user experience improvements

[ghost]

Hi all,

Preset is exploring making changes to the dataset user flows and wanted to share our UX proposal as a discussion topic before posting the full SIP with implementation details. Thanks for looking!

Proposal for improved datasets user experience

Motivation
In the current Superset user experience, creating a dataset is a required step in creating a data visualization. Research and feedback from users has revealed this to be a sub-optimal user experience for the following reasons:

1. Oftentimes users want to create a quick data visualization without saving a permanent dataset in the system.
2. Users don’t understand the purpose and role of a dataset, so the requirement to create one before creating a visualization becomes a confusing stumbling block in their workflow.

Proposed Change
The following are proposed changes to the user flow and functionality that will create a better dataset user experience:

1. During chart creation, the user can select a table directly from a connected database, a dataset, or a saved SQL query as the data source for their visualization.
2. The user can add a database or create a dataset directly from the chart creation page instead of having to navigate away to databases or datasets.
3. The user can open Explore directly from a query in the SQL editor without having to create a dataset first.
4. The user can open a saved query in Explore from the saved query CRUD view.
5. Clicking on a dataset name in the dataset CRUD view will open the edit dataset modal, not Explore (it will still be possible to open dataset in Explore from dataset CRUD view).
6. Making changes to the column metadata of a table or a query (e.g. needing to mark a column as temporal) will prompt the user to first save their dataset.
7. When saving a chart that was created from a table or a query, the user is prompted to save their underlying table or query as a dataset.
8. During database creation, user is prompted to create a dataset or query in SQL Lab.

We also plan to explore/test the renaming of “dataset” to something more descriptive such as “shaped data” or “enriched tables”.

Please refer to this link or the below image for the proposed flow changes.

New or Changed Public Interfaces
The following interfaces will be updated as part of this effort (UI mocks to be provided in a later stage):
Revamp dataset creation/edit modal UI for improved usability.
Change the “Create chart” setup from a single screen to a multi-screen experience to accommodate expanded availability of data sources.

1. Add navigation element to the Explore experience to make it easier for the user to toggle between visualization types, data sources, and chart configuration interfaces.
2. Change or move dataset area of Explore (top left of screen) to be more inclusive of expanded availability of data sources.
3. Add a button to the SQL editor that allows users to open their query directly in Explore without creating a dataset.
4. Add dataset education and call-to-action to the end of the database connection flow.
5. Update chart, dataset, and saved query CRUD views to reflect proposed changes.
6. Minor changes to save/confirmation modals — for example, in Explore, prompt users to save a dataset and in SQL editor, give the option of creating a visualization when saving query as a dataset.

Migration Plan and Compatibility
Implementation will be broken up into 3 parts:

1. Allow users to create visualizations from SQL editor and save query as a dataset in Explore.
2. Allow users to create visualizations from tables or saved queries and change between those data sources in Explore.
3. Update dataset creation and editing UI.

Rejected Alternatives
We considered automatically creating a permanent dataset based on a selected database table or SQL query for the user instead of requiring them to make one themselves. However, this would still lead to a cluttered system of 1:1 chart:dataset relationships, diluting the true power of a dataset to the user.

[nishantmonu51]

Datasets is powerful concept but current flow can get overwhelming for new users using superset for the first time.
Like the idea of simplifying the viz creation flow. Thanks for the proposal.

[cccs-rc]

Our users have been able to do a lot with datasets, but I can see the benefits of allowing users to build a visualization before committing to saving a dataset!

A few thoughts/comments/questions:

1. I see a few spots on the diagram where it looks like there might be counts of the number of rows in a table. This seems risky to me. We have some very big tables (TBs of data), so computing that row count could be quite costly.
2. When a user saves a dataset after developing the chart, will that be a virtual dataset?
3. I'm guessing the design for what the modals will look like isn't super solid yet, and I do see that there's a mention of moving to a multi-page wizard, but just wanted to mention that some of our users don't have permissions to create database connections, so hopefully they won't see an option to add a DB! Similarly, we have many (hundreds) of datasets, so I'm hoping the dataset list will be easily searchable.
4. Having the dataset editor open when clicking on the dataset name feels like a big change. Just hoping that gets well publicized if/when it's released so that we have time to prepare our users.

Thanks for providing the lovely workflow diagram! It does a lot to make the plans clear! :)

[eschutho]

I see a few spots on the diagram where it looks like there might be counts of the number of rows in a table. This seems risky to me. We have some very big tables (TBs of data), so computing that row count could be quite costly.

@cccs-rc Thanks for this feedback. If we were to only show the total row counts on a table for one table at a time, only when you click on the table itself, do you think that would be acceptable?

[cccs-rc]

@eschutho For us, our biggest tables have trillions of rows, so I worry about what would happen when a user clicked on one of those tables! Would it be possible to expose a config option to turn the count function off in this view? Our users are often not interested in the number of rows in tables (1.2B rows vs 1.3B rows doesn't make much difference to them!), so we would rather avoid the expensive count in Superset.

Alternatively, if showing the row count is really important for others' use cases and a config option would be unpalatable, maybe there's a performant way to get the row count only when that row count is under some reasonable threshold and otherwise we're just told that the table has more than [threshold] rows?

[eschutho]

thanks @cccs-rc, yes we can definitely look into either of those options. I appreciate hearing about your use cases.. it's very helpful!

[EBoisseauSierra]

Looks really great, thanks!

Something slightly tangential to this workflow, yet IMHO still relevant in terms of “create charts without ready-bake dataset”: a UX our users liked a lot in Metabase is their so-called Query Builder, that is a GUI allowing user to “write” a query without any SQL:

(source)

I have personally found it quite powerful (possibility to add multiple layers of JOINs, WHERE, GROUP BY), and intuitive to use.
A caveat, though, is that one tend to generate the already-aggregated query-result — so it wouldn't fit natively with the current Explore panel, where the last-mile aggregation is done.

[ghost]

Thanks for sharing, @EBoisseauSierra! This is a great pattern and we're always looking for ways to make Superset easier for less-technical folks.

[EBoisseauSierra]

Note that this goes quite against Max's case for Dataset-centric visualisation as it's quite query-centric — unless we add it as a noob-friendly alternative to the SQL Lab to build virtual datasets/views.

[eschutho]

Hi @EBoisseauSierra, yes we still want to encourage users to create datasets, but at the same time, give them an easy way to visualize their data from a query in the beginning stages of their data exploration. To accomplish both of these goals, we will be requiring users to create a dataset upon saving their chart.