[Lens/Discover/Dashboard] Support for ad-hoc data views

[angorayc]

As Lens embeddable only takes Data View id. Therefore, Lens embeddable is not compatible to the Sourcerer we have on our page, which allows users to filter the index patterns under the data view. So we are not going to replace our existing charts with Lens embeddable, we keep the existing charts and create the chart actions by ourselves.
#126507

security_solution_sourcerer.mp4

In order to keep the consistency of index patterns in Lens, I use _index filters to filter out the selected patterns from Security Solution's Sourcerer when open the chart in Lens, save Visualization, and attach to case.
Although the query is different from what we have when inspecting on Security Solution's page, this keep the result aligned.
But I'm still hoping that Lens / Lens Embeddable can take index patterns, this will allow Security Solution to use Lens Embeddable on our page with aligned queries and index patterns. We also have a plan to show custom Lens Visualization on Security Solution's page in 8.3 release, the change will empower this feature a lot. Thanks for considering.

_index_filter.mp4

Originally posted by @angorayc in #124792 (comment)

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[flash1293]

To keep this compatible with the Lens UI, we need to support ad-hoc data views in Lens (do not pick an existing data view from the list, but create it inline without saving). This has a dependency on @elastic/kibana-app-services as data views have to implement the persistable state interface so they can be embedded as part of a Lens configuration.

[mattkime]

@angorayc Whats the best way for me to get up to speed on how data views are being used by security solution?

The request for this feature seems necessitated by the filtering of index patterns (on data views). This likely limits interoperability with other kibana apps. I'd like to make sure I understand this before creating additional functionality.

We have flexibility in how we resolve this so I'd like to make sure we're taking the widest view possible before committing to a particular implementation.

[angorayc]

@mattkime , yes, in Security Solution's use cases, our data view allows users to filter index patterns under the given data view, this allow us to have better performance when loading data. However this also brings the inconsistency between Security Solution and other apps like Observability and Lens, as the data view only takes data view id. I understand that we can apply _index filter when open our charts in Lens to have the consistent result, but users may be confused as when they clicking on Inspect in Security Solution, they see the selected index patterns are applied.

But when open the chart in Lens, they see all the index patterns under the data view id, and they have to go to they query tab to figure out actually _index is applied for selected index patterns.

video:

inspect.mp4

[angorayc]

Please feel free to visit https://kibana.endpoint.elastic.dev/app/security/hosts/allHosts and play around with the data view. Thank you!

[angorayc]

We have another use case that needs ad-hoc data views:
Host risk score over time consumes ml_host_risk_score_default index, which does not exist in kibana data view, and thus doesn't have data view id either. We prefer not to create the data view under users' behave, therefore we cannot open it in Lens atm.

[mattkime]

@angorayc

We have another use case that needs ad-hoc data views:

Is this another case where you want to use lens?

[angorayc]

@mattkime Yes, Host risk score over time is an example, but we have more charts on Security Solution that consume an index that doesn't exist in Kibana data views.

[stratoula]

@angorayc now Lens supports adhoc dataviews. Does this solve your requirements? Can we close this issue?

[angorayc]

Thanks so much for the help! Using it in our current feature, looking good!