[WIP] Add support for Vega and Vega-lite #3931

ktmud · 2019-06-26T05:28:20Z

What type of PR is this? (check all applicable)

Feature

Description

Vega and Vega-Lite allow a user to specify a JSON configuration
to declaratively define a complex interactive visualization.

It could theoretically become the foundation of many visualizations
we have in Redash.

This PR adds Vega and Vega-Lite and a new visualization type,
and allows users to directly edit a Vega or Vega-Lite spec in either JSON or YAML. A lot of code is borrowed from the official Vega Editor.

While most of the functionalities are working, this is not ready for shipment yet. Some known issues:

Vega allows only fixed container size. We need to figure out how to do autoresize.
Not all themes are useful to the majority of the users.
Haven't tested tooltips.
~~Vega-Embed allows downloading PNG files, which can be very useful.~~ Will add support in future PR.
~~Was trying to add monaco-yaml so the YAML editor can also have the very nice IntelliSense, including autocomplete and linting. But that Monaco plugin isn't working well with ESM yet.~~ Will consider migrating to Monaco in future work.

Related Tickets & Documents

https://discuss.redash.io/t/add-vega-grammar-support-for-advanced-visualizations/3020

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

Visualization editor:

Visualization renderer (with a different Vega theme)

client/app/visualizations/vega/theme.js

kravets-levko · 2019-06-26T16:36:52Z

Hi @ktmud! Thanks for your incredible work! I know that this PR is still WIP, but I have some notices that would be better to fix on early stage:

don't pass query object to each visualization; you need only URLs there - so pass them; probably, data object is a good place;
please try to reuse existing dependencies; I like Monaco as well, but Redash already has an editor library (Ace), so please use it; also, do you really need to update all that dependencies?

Also, please see my comments to the code. Thanks!

client/app/services/query.js

client/app/visualizations/VisualizationRenderer.jsx

kravets-levko · 2019-06-26T16:40:25Z

client/app/visualizations/index.js

+  return find(
+    registeredVisualizations,
+    viz => !viz.isDeprecated && viz.isDefault,
+  );


Now this code assumes that at least one visualization has isDefault attribute; not sure if it's right or not, but I'd add a fallback for this case (if no isDefault visualization - return first non-deprecated). That's just a suggestion

Makes sense. Maybe we should pre-sort the registered visualizations and make the default visualization a constant instead of calling find() every time?

This is just a quick hack to avoid Vega showing up as the default editor. If there's any other way of doing it, I'd be happy to avoid adding a new config field.

Adding new config field is a good solution, let's keep it. Calling find every time is not a problem because ATM it is used only in Edit Visualization dialog and is called there only once (on dialog open).

@ktmud I created a separate PR for this feature (#3944) because we needed it right now in few other PRs to fix tests.

redash/handlers/admin.py

kravets-levko · 2019-06-26T16:41:43Z

redash/handlers/query_results.py

-                'attachment; filename="{}"'.format(filename.encode("utf-8"))
-            )
+            if filetype in ('csv', 'json') and request.args.get('download') in ['false', '0']:
+                response.headers.set('Content-Type', 'text/plain; charset=UTF-8')


Can you please explain these changes (what are they needed for?). Thanks!

This is for allowing users to click on the data source url that populated in Vega schema and review the data directly in browser without automatically downloading a file.

I think in this case you have to set a proper MIME type for each extension. E.g. I have a chrome extensions that pretty-print both CSV and JSON (see screenshot with example), and both that extensions detect their targets by MIME type, so changes like this will break them.

I also have a JSON prettier extension installed, but couldn't find a good one for CSV. Which one are you using?

Since this change added a new URL parameter "download=false", it shouldn't break any existing links. I'm inclined to keep it. What do you think?

ktmud · 2019-06-26T17:02:56Z

Hi Levko, thanks for reviewing!

please try to reuse existing dependencies; I like Monaco as well, but Redash already has an editor library (Ace), so please use it; also, do you really need to update all that dependencies?

Can we actually replace Ace Editor with Monaco? Monaco comes with some highly desirable advanced features such as inline validation of JSON schemas, which is quite essential for a Vega editor.

I had to upgrade Webpack because the latest version of Monaco wouldn't build.

kravets-levko · 2019-06-26T18:01:02Z

Can we actually replace Ace Editor with Monaco?

Well, yes, we can, of course. But definitely not within this PR - we use a lot of Ace features in query editor and all of them should be kept with Monaco.

ktmud · 2019-06-26T19:25:51Z

Can we actually replace Ace Editor with Monaco?

Well, yes, we can, of course. But definitely not within this PR - we use a lot of Ace features in query editor and all of them should be kept with Monaco.

Sure! I think I'll remove Monaco dependency for now and replace it with Ace Editor. There was some building issues with Monaco as well (it takes forever to build client/dist for Docker image).

ktmud · 2019-07-02T02:57:58Z

client/app/visualizations/vega/Editor.jsx

+//   monaco.languages.json.jsonDefaults.setDiagnosticsOptions(monacoDiagnostics);
+//   monaco.languages.registerDocumentFormattingEditProvider('json', jsonFormatter);
+//   monaco.languages.registerDocumentFormattingEditProvider('yaml', yamlFormatter);
+// }


Keeping the code for Monaco here for now, so later migration will be easier (if we are going to do it)

ktmud · 2019-07-02T02:59:32Z

client/app/visualizations/vega/Renderer.jsx

+      vPadding += (spec.padding.top || 0) + (spec.padding.bottom || 0);
+    }
+    width = width || specWidth || Math.max(parentSize.width - hPadding, 100);
+    height = height || specHeight || Math.min(400, Math.max(parentSize.height - vPadding, 300));


By default, a Vega visualization has min-height 300px and max-height 400px. This is inline with what Plot.ly currently has, although I was not able to find where we set the height for Plot.ly...

We don't set height for Plotly charts: on Dashboard page chart fits it's container in widget, and on Query page it has fixed height (IIRC, 500px) and width 100%.

So the default height 500px comes from Plot.ly library itself?

I implemented 100% height for Vega charts everywhere.

ktmud · 2019-07-02T08:31:17Z

@kravets-levko @arikfr I think this PR is more or less ready now. Can you please take another look? It's actually quite fun to play with Vega specs.

kravets-levko · 2019-07-02T08:33:27Z

client/app/visualizations/vega/Renderer.jsx

+import { Handler } from 'vega-tooltip';
+import { Alert, Icon } from 'antd';
+import LZString from 'lz-string';
+import memoize from 'memoize-one';


_.memoize? useMemo?

_.memoize does not cache only one return value, having higher risk of memory leaks.
useMemo does not work with class components. I don't see the added benefits of rewriting my class to a function component.

memoize-one only has 45 LOC.

rastafarra · 2019-09-24T12:24:46Z

hi!

can you please check conflicts?

bazhenov · 2020-03-30T00:46:30Z

Hi.

We are very interested in this functionality, as standard visualisations types are quite limiting for us. What is the status of this MR and do you need any help? Maybe some functionality should be implemented before it considered "done"?

kravets-levko · 2020-03-30T11:30:06Z

Hi @ktmud! I'd like to review your PR once more, so can you please resolve conflicts? Also, I'd ask you to remove changes that are not related to subject of this PR and commented out fragments. Thank you!

P.S. Please ping me when it's ready for review 🙇‍♂️

ktmud · 2020-03-31T04:50:53Z

Hi all, I just pushed a quick fix to resolve the conflicts. Things might still not work. Will pick this up again during the weekend.

ktmud · 2020-03-31T04:51:24Z

I haven't worked on Redash for a while, so things might be a little slow.

itssimon · 2020-08-12T12:05:32Z

This looks amazing @ktmud! Do you think you'll get a chance to get this across the line? Happy to help!

ktmud · 2020-08-14T09:07:46Z

Just updated the branch and it is now 90% done.

It depends on #5104 , but the dependency can be easily removed if that PR isn't going anywhere.

Optimize typescript configuration Fix webpack for prod Adjust cypress timeout Fix viz-lib build

Vega and Vega-Lite allow users to specify a JSON configuration to declaratively define a complex interactive visualization. It could thoeretically become the foundation of many visualizations we already have. This commit adds integration with Vega and Vega-Lite, with a lot of code borrowed from the official Vega Editor[1]. [1] https://vega.github.io/editor/

vadimcn · 2023-02-02T02:08:56Z

Hi! What happened to this PR? Is it abandoned?

guidopetri · 2023-07-15T19:44:54Z

@ktmud , thanks for the PR! We've updated a lot of things now that we're Community-driven so - if you're still interested in getting this merged - would you mind rebasing off master to re-run the CI?

We're trying to clean up our PR todo list, so if you're not interested, that's fine - we'll close the PR in about a week if we don't hear back. If you're interested in reopening the PR afterwards, we would also very much welcome that.

ktmud force-pushed the vega branch from 697b503 to 59b2bad Compare June 26, 2019 05:31

ktmud commented Jun 26, 2019

View reviewed changes

client/app/visualizations/vega/theme.js Outdated Show resolved Hide resolved

ktmud changed the title ~~Add support for Vega and Vega-lite~~ [WIP] Add support for Vega and Vega-lite Jun 26, 2019

kravets-levko reviewed Jun 26, 2019

View reviewed changes

client/app/services/query.js Outdated Show resolved Hide resolved

kravets-levko reviewed Jun 26, 2019

View reviewed changes

client/app/visualizations/VisualizationRenderer.jsx Outdated Show resolved Hide resolved

kravets-levko reviewed Jun 26, 2019

View reviewed changes

redash/handlers/admin.py Outdated Show resolved Hide resolved

kravets-levko reviewed Jun 26, 2019

View reviewed changes

ktmud force-pushed the vega branch 5 times, most recently from 92e56d4 to 2e522cc Compare July 1, 2019 21:07

ktmud commented Jul 2, 2019

View reviewed changes

ktmud changed the title ~~[WIP] Add support for Vega and Vega-lite~~ Add support for Vega and Vega-lite Jul 2, 2019

ktmud force-pushed the vega branch from 4dcc779 to 5a22339 Compare July 2, 2019 03:26

kravets-levko reviewed Jul 2, 2019

View reviewed changes

ktmud mentioned this pull request Jul 8, 2019

Vegalite charts apache/superset#4558

Closed

ktmud force-pushed the vega branch from da21310 to c3394e4 Compare July 9, 2019 17:22

ktmud force-pushed the vega branch 2 times, most recently from 8e1c0a4 to e7f78a4 Compare September 26, 2019 00:43

weekly-digest bot mentioned this pull request Sep 30, 2019

Weekly Digest (23 September, 2019 - 30 September, 2019) #4196

Closed

kravets-levko self-requested a review March 30, 2020 07:50

ktmud force-pushed the vega branch from e7f78a4 to ea7c3c3 Compare March 31, 2020 04:49

ktmud changed the title ~~Add support for Vega and Vega-lite~~ [WIP] Add support for Vega and Vega-lite Mar 31, 2020

restyled-io bot mentioned this pull request Mar 31, 2020

Restyle Add support for Vega and Vega-lite #4762

Closed

ktmud force-pushed the vega branch 2 times, most recently from 802c9dc to 2b6427e Compare August 14, 2020 09:05

ktmud force-pushed the vega branch 3 times, most recently from 01ae06d to 96d38c6 Compare August 14, 2020 18:13

ktmud and others added 6 commits August 14, 2020 11:20

Add React hot refresh

5f65c51

Upgrade React

6f93382

Upgrade babel and webpack

e85613a

Add proper exports for pages

f2d0355

Fix linter

2e47f42

Optimize typescript configuration Fix webpack for prod Adjust cypress timeout Fix viz-lib build

ktmud force-pushed the vega branch from 96d38c6 to 65191ff Compare August 14, 2020 18:23

guidopetri added the 1-week-review label Jul 15, 2023

guidopetri closed this Jul 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add support for Vega and Vega-lite #3931

[WIP] Add support for Vega and Vega-lite #3931

ktmud commented Jun 26, 2019 •

edited

Loading

kravets-levko commented Jun 26, 2019

kravets-levko Jun 26, 2019

ktmud Jun 26, 2019

kravets-levko Jun 26, 2019

kravets-levko Jun 30, 2019

kravets-levko Jun 26, 2019

ktmud Jun 26, 2019

kravets-levko Jun 26, 2019

ktmud Jun 30, 2019

ktmud commented Jun 26, 2019

kravets-levko commented Jun 26, 2019

ktmud commented Jun 26, 2019

ktmud Jul 2, 2019

ktmud Jul 2, 2019

kravets-levko Jul 2, 2019

ktmud Jul 2, 2019

ktmud commented Jul 2, 2019

kravets-levko Jul 2, 2019

ktmud Jul 2, 2019

rastafarra commented Sep 24, 2019

bazhenov commented Mar 30, 2020

kravets-levko commented Mar 30, 2020

ktmud commented Mar 31, 2020

ktmud commented Mar 31, 2020

itssimon commented Aug 12, 2020

ktmud commented Aug 14, 2020

vadimcn commented Feb 2, 2023

guidopetri commented Jul 15, 2023

[WIP] Add support for Vega and Vega-lite #3931

[WIP] Add support for Vega and Vega-lite #3931

Conversation

ktmud commented Jun 26, 2019 • edited Loading

What type of PR is this? (check all applicable)

Description

Related Tickets & Documents

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

Visualization editor:

Visualization renderer (with a different Vega theme)

kravets-levko commented Jun 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ktmud commented Jun 26, 2019

kravets-levko commented Jun 26, 2019

ktmud commented Jun 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ktmud commented Jul 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rastafarra commented Sep 24, 2019

bazhenov commented Mar 30, 2020

kravets-levko commented Mar 30, 2020

ktmud commented Mar 31, 2020

ktmud commented Mar 31, 2020

itssimon commented Aug 12, 2020

ktmud commented Aug 14, 2020

vadimcn commented Feb 2, 2023

guidopetri commented Jul 15, 2023

ktmud commented Jun 26, 2019 •

edited

Loading