Proposals (mainly breaking ones) for 3.0

[rusackas]

We want to start gathering some attainable wishlist items for 3.0. It's early at this point, so maybe we can gather ideas here (as threads for each proposal), and when we get to the right place in terms of calendar and wishlist, we can open a SIP and a project board. Jump on in!

I'm saying these are mainly breaking changes because things that are not considered breaking can happen at any time and are not held back by a major version release. However, we MAY want to consider some larger/impactful features as a major version milestone, and have a more feature-driven release.

[rusackas]

Deprecate drag and drop feature flags: ENABLE_EXPLORE_DRAG_AND_DROP & ENABLE_DND_WITH_CLICK_UX and remove any code paths possible around the old UX when these flags were disabled.

[rusackas]

Added to proposals board

[rusackas]

Deprecate and remove the UX_BETA feature flag.

[rusackas]

Added to proposals board

[rusackas]

Set GENERIC_CHART_AXES Feature flag to True by default

[rusackas]

Added to proposals board

[rusackas]

Remove Filter Box component

[rea725]

Nevermind! I see the new goodness in master!

[rea725]

I am going to miss some aspects of the current filterbox functionality:
1.) I like the new filter pane overall, but for some of our use cases, it would be nice to keep the filters in main dashboard layout. I'm wondering the there might be a way for the new filters to also have an option be moved about, even if the default behavior is to position them in the filter pane.
2.) In my initial exploring latest master, I didn't see an option to keep the Instant Filtering behavior that exists with the legacy filters. I really like the instant gratification that comes with being able to make a change and have the dashboard update instantly without the extra step of clicking "Apply Filters" at the bottom. I think that for small datasets or dashboards that are run on performant databases (such as a Greenplum cluster), the instant filtering option remains very desirable.

[kasiazjc]

@rea725 We are actually planning to introduce more flexible filtering (adding filters directly to the dashboard) and we have it planned as a Stage 2 of dashboard filters redesign. We just need to define when it will happen roadmap-wise.
I think there was a possibility to instantly apply some dashboard filters, but the functionality was confusing before and we need to revisit!

[kamalkeshavani-aiinside]

+1 for allowing Instant Filtering in the new setup.
Also, even though filter pane is good, users always like filters next to the charts when that filter is just for that one chart. So the native filter experimental feature should be further explored and allow filters to be inside the dashboard as well.

[rusackas]

Added to proposals board for consideration/consensus

[dpgaspar]

Remove all model views service side rendering code, replace by React list views

[michael-s-molina]

Very important for Theming as well.

[nytai]

the legacy datasource editor still has some behavior that is missing in the new model based editor.

eg, #19768

[rusackas]

Added to proposals board

[dpgaspar]

Remove all superset/* REST APIs marked has deprecated

[rusackas]

Added to proposals board

[dpgaspar]

Make all API v1 resource names plural. Ex: /api/v1/dashboard/ -> /api/v1/dashboards/

[rusackas]

Added to proposals board

[zhaoyongjie]

Move time range to adhoc filter and move time grain to columns

[rusackas]

Can you elaborate more on the "move time grain to columns" aspect?

[zhaoyongjie]

Move Advanced Analysis to Metrics

[rusackas]

Can you elaborate? Do you have time/intent to contribute this still?

[cdmikechen]

I wonder if this feature can be added to the new version as a new extension of superset
Let Superset provide another canvas based presentation/让Superset提供另一种基于画布方式的展现形式

[rusackas]

This is a super exciting proposal, but I think it's far too big of a change for the next (imminent) 3.0 release. I'd love to see more traction on the Discussion thread and on Slack, and get more design input to figure out how we can make a change like this gradually. It'd be a HUGE change, so what might be the incremental steps? This is probably best to put out there as a SIP with a full implementation plan if possible.

[eschutho]

remove OLD_API_CHECK_DATASET_OWNERSHIP

[rusackas]

Added to proposals board

[eschutho]

Create export/import capabilities : roles / users /row level security

[eschutho]

@amitmiran137

[dpgaspar]

note: FAB can export and import roles and permissions already: https://flask-appbuilder.readthedocs.io/en/latest/cli.html

flask fab export-roles --path roles.json

[rusackas]

Added to proposals board

[eschutho]

Remove Pivot Table in favor of Pivot Table V2

[eschutho]

@rusackas

[rusackas]

Heck yes... I'm not sure of all the breaking changes, but a couple of reported ones include:
• V2 renders NULL/NaN in cells, where V1 displays empty cells
• V2 does not render HTML in cells, where V1 does.

[rusackas]

Remove the /csrf_token/ route from core.py, as instructed by a todo comment there

[rusackas]

@suddjian for added context

[rusackas]

I'm not even sure what this refers to anymore. I don't see the TODO. Am I missing something?

[eschutho]

Remove react-select

[eschutho]

@michael-s-molina

[rusackas]

Added to proposals board

Requires the removal of the FilterBox component.

[RAhuja2015]

Feature to add tags to dashboard or folders to categorize published dashboards/queries

[rusackas]

There's an open PR on this... it's coming soon!

[rusackas]

This hasn't merged yet, but is a non-breaking feature.

[cemremengu]

Add a path prefix like /superset to all pages for making path handling easier in reverse-proxied environments. i.e. will prevent conflicts with other apps/paths

[dpgaspar]

We could probably make this optional so it would not be a breaking change

[rusackas]

If it's not a breaking change, I'm not going to add it to the 3.0 project board right now. We can take the PR whenever the time is right.

[itziarmj]

Allow customization of the prefix /superset added to all pages, so that several Supersets with different contents could be easily added to the same domain.
This is doable for example in Pentaho Dashboards.

[cemremengu]

Replace pandas with polars

[rusackas]

Could you explain why, or if this constitutes a breaking change?

[cwegener]

@rusackas Let me try and explain.
Pandas does not scale well as the number of rows returned from the DB increases, slowing down post-processing in SuperSet.
https://pola-rs.github.io/polars-book/user-guide/#performance-

EDIT Maybe worth to get an opinion from @betodealmeida as well.

[tiexiw]

Hope to integrate Alibaba's chart antv, compatible with H5 and mobile phones

[rusackas]

We're still working on ECharts completion, which was voted in as the "official" viz library. Superset's mobile capabilities are... well... not quite there yet. But if you're interested in helping to build new plugins, I'd love to speak with you on Slack!

[markostefanovic1]

Caching virtual dataset results - if multiple charts on the dashboard are using the same dataset results (but they just group them differently as required for different charts), then send only one query to the database (1 query per dataset), cache the dataset results and allow charts to use those results in chart creation. This should prevent sending a separate query to the database for each chart when the filters are applied.

[villebro]

This would require adding database functionality to Superset, as the full result set of the virtual dataset query would need to be persisted and then having the outer query executed against that result set. In addition to needing to add a SQL engine to Superset, this would be infeasible especially for large datasets (imagine moving 1 TB of query results from the analytical database to the Superset cache and then performing a new query on those results)

[markostefanovic1]

I'm not really well versed in this subject, but I think I understand what you mean.

Additional question - would it make more sense if the analytical database's resources were somehow used for caching virtual tables instead of using Superset's resources, OR to use some 3rd party software as a caching layer between the database and Superset?

[villebro]

The majority of databases can do this optimization by themselves. Imagine the following virtual dataset:

select str_col_1, str_col_2, num_col_1, num_col_2 from my_table where col3 = 'foo'

Next imagine we have two charts, one grouping by str_col_1 and summing on num_col_1, and the other doing the same on str_col_2 and num_col_2 respectively:

First query executed by Superset:

SELECT str_col_1, SUM(num_col_1) AS my_metric FROM (select str_col_1, str_col_2, num_col_1, num_col_2 from my_table where col_3 = 'foo') virtual_table GROUP BY str_col_1

Second query executed by Superset:

SELECT str_col_2, SUM(num_col_2) AS my_metric FROM (select str_col_1, str_col_2, num_col_1, num_col_2 from my_table where col_3 = 'foo') virtual_table GROUP BY str_col_2

In most modern databases, the subquery is just "syntactic sugar" and will be unrolled to the following queries:

First query:

SELECT str_col_1, SUM(num_col_1) AS my_metric FROM my_table WHERE col_3 = 'foo' GROUP BY str_col_1

Second query:

SELECT str_col_2, SUM(num_col_2) AS my_metric FROM my_table WHERE col_3 = 'foo' GROUP BY str_col_2

In other words, the virtual dataset usually doesn't add any execution overhead compared to a regular physical dataset.

[HeathLee]

New export/import method and add a copy dataset method to allow change dataset/dashboard when copy. Current export/import charts/dashboard require chart/dashboard name to contain ASCII only (due to secure_filename limit)， if the name use non-ascii like a Chinese character, export or import will not work correctly.

[cccs-rc]

Hey @HeathLee, just fyi, there's a discussion and draft PR for copying datasets here: #20245
The author mentioned it in Slack as well: https://apache-superset.slack.com/archives/C014LS99C1K/p1654718739800239

[rusackas]

This was merged, just to note it here, so we won't worry about it in the 3.0 conversation going forward.

[dpgaspar]

convert ENABLE_BROAD_ACTIVITY_ACCESS and MENU_HIDE_USER_INFO into feature flags

[rusackas]

Added to proposals board

[sfirke]

Move the general SEARCH box to be on the left of the other boxes? I find that it is the most-used of the input fields.

[cemremengu]

I opened a PR for this sometime ago #19721

[rusackas]

This was merged, just to note it here, so we won't worry about it in the 3.0 conversation going forward.

[eschutho]

side note on this @sfirke, we're planning to convert these crud view tables to the new antd table that has filtering built in, so you may see more improvements here soon.

[kamalkeshavani-aiinside]

Allow increasing horizontal grid columns from current fixed 12.

[rusackas]

A lot of folks complain about those columns being too narrow as it is, with the card titles getting truncated, and filter icons competing for room. Can you explain any design changes you'd have in mind here, or the motivations for additional columns? This might be part of a bigger effort (i.e. revisiting the dashboard layout feature holistically), so I'd be curious to learn more.

[kamalkeshavani-aiinside]

Here is my thought on how to implement this.(I tried to implement it myself, but couldn't figure it out)
#16140 (comment)
Motivation is very simple, if I want to add 5 'big number' charts of same width in a row, I can't do that without leaving a blank space in the row. Same goes for 7.

[rusackas]

I'm not sure... I guess we could in theory go to 24, but 12 seems to be sort of a best practice in similar tools (datadog springs to mind) as additional resolution may cause as much annoyance as utility. Plus, I know we'll start seeing more issues about where the card layout crumbles in such small settings with the current design. My personal thought here is that we should start a separate Discussion around what a revamped layout editor for Dashboards might look like. I think such a project might be too big for 3.0, as it could get pretty hairy.

[rusackas]

If you want to open a PR to make it configurable, that sounds amenable, so long as the default is 12 like you pointed out in your comment. I do worry about changing that setting causing a migration of sorts and breaking a whole bunch of dashboards. This would need pretty comprehensive tests.

[rusackas]

This seems like a long shot, since it's pretty intricate, but maybe adding mobile/responsive layout support for Dashboards. Just adding it to the convo here since it keeps cropping up on Slack. We proved in a hackathon that it's both doable and... well... quirky.

[rusackas]

3.0 needs to come out sooner rather than later, so I'm effectively rescinding this proposal, and we can revisit at any time in the future, so long as it's a non-breaking change.

[Synarcs]

One thing I particularly look in superset, is efficient version control, in an multi tenant environment. For example when some role does changes to chart, it tracks local changes, and once he is done the chart can be committed to a superset registry (same like apache nifi and nifi registry).
I assume in the registry the chart query context and raw form data can be saved and when importing from registry associated certification can be asked while importing. I know their are other tools like dbt but expect something from core superset as well.

[rusackas]

So essentially version control for charts? This has been discussed in various approaches (also for dashboards and queries), but I think the design was never quite finalized, and nobody ever stepped up to volunteer to actually build such a feature as part of their roadmap. If you have interest in contributing this engineering effort, I'd love to see (and be tagged on) a GIthub Discussion (in the "Ideas" category), so we can take it up as a community.

[eschutho]

We can probably remove this from the frontend extension registry:

// Exporting this under the old name for backwards compatibility.
// After downstream folks have migrated to `getExtensionsRegistry`, we should remove this.
export const getUiOverrideRegistry = getExtensionsRegistry;

[rusackas]

Added to the board :)

[eschutho]

This has been deprecated, so we can remove it from the docker config: ENABLE_SCHEDULED_EMAIL_REPORTS

[rusackas]

Added to the board.

[EugeneTorap]

Move SQL Lab to SPA 🙏 🙏 🙏

[rusackas]

Added to the board. Are you willing/able to work on this, or is it more of a request?

[rusackas]

All items have been moved to the board, which is in flight. We'll take this up in a new thread when 4.0 comes.