[Vega] Update to most recent version (4.2 at least)

[thatguy7]

In the most recent release, 4.2, the Vega library added dynamic data loading. E.i. it's possible to set signals as data urls.

We would benefit here from the possibility to apply global filters to a certain index, and then use the response to filter other indices depending on the result of the first query.

In my use case, I have two types of indices, ones with lots of fields (500+), which I want to filter by (metadata), and ones with few fields, that contain data I want to visualize (data). The documents containing actual data outnumber the ones with the metadata by a factor of over 1000. So attaching the metadata to each data document creates a huge drop in performance.

This will make it possible to take advantage of relationships between indices and essentially provide the possibility to leverage nested like behavior, without actually nesting documents.

[timroes]

@nyurik is this something that's easily done by upgrading dependencies, or do we need to hook into the dynamic data loading to make it work with our layer of querying?

[nyurik]

Our current data loader works in a different way - we pre-parse Vega spec, find any elasticsearch queries, and replace them with the result data before even passing it to the Vega lib. I do not like this approach (initially we had this the other way around - Vega lib would call our code to indicate that it wants to load X, and we would parse that X into the data). Unfortunately, this did not fit our data model - with render() being called with all the data already in place.

We would have to rework this somehow to make render() be able to make data requests too.

[legrego]

I had a discuss user asking about this today: https://discuss.elastic.co/t/kibana-information-in-tooltip/166018/3

[flash1293]

@timroes Regarding @nyurik s latest response, making dynamic data loading possible seems to require major refactoring of the whole way the vega plugin works. Should we split this issue into vega update with specifically (and documented) no support for dynamic data loading as part of the stabilizing (7.0 or at least 7.1) and a second issue which tracks the refactoring and inversion of control flow?

[timroes]

I think we should clearly state in our documentation, we don't support dynamic data loading in Vega for now, and open a separate enhancement issue to support it. I think there is already a benefit for the user having the newest version even if we don't support that feature (yet).

[timroes]

I would suggest we add either a section to this page: https://www.elastic.co/guide/en/kibana/current/vega-graph.html or create a whole new subpage: limitations, but since there won't be too much on it, I think we can just put a warning paragraph in that Vega front page.