Proposal: OpenSearch Dashboards Search Multiple Indices

[jkowall]

Requirements - What kind of business use case are you trying to solve?

The ability to search multiple indices at the same time with a single query

Problem - What blocks you from solving this today?

Today the user interface in Kibana allows for the query to only run against a single index.

Proposal - what do you suggest to solve the problem or improve the existing situation?

We should modify Kibana to select multiple indices at the same time via a checkbox and run the corresponding query. Today you can only pick one at a time.

Assumptions

Installation of Kibana

Any open questions to address

None

Thanks from @jkowall and @galangel

[mihirsoni]

Trying to understand more on proposal and use cases behind it, so with this change we can execute queries against multiple indices (may be completely different mapping) and display results on it. This field indicates on index patterns, does it also include searching over multiple index patterns ?

If we make this change allowing users to select multiple indices or indices patterns , how filters will be applied ? What if users they both have completely different mapping ?

Would it be possible to add some use cases in issue for better context understanding ?

[galangel]

Hi @mihirsoni, we want to develop the ability to make a single query run on multiple indices and get all of the results,
So yes, that includes searching over multiple indices patterns.

Yes, field types should not conflict. If they do, there are solutions for that. Preemptively, we can validate before searching and indicate in the UI, or we could return an informative error.

A use-case that I am a fan of is searching for a "Request-ID" (that we generate and attach to logs and requests) and track and visualize the flow across multiple services.

Right now, at Logz.io, we allow users to have multiple “accounts” or indices and search across one, many, or all of them, Our customers benefit a great deal from it, and we believe it could be a great contribution to the open search project.

[JacobBrandt]

If you allow users to create their own aliases then Voilà all search and filtering is just going to work for everything that searches within the tool not just Discover.

EDIT: I want to clarify what I'm trying to say. The following statement in this ticket is inaccurate.

Today the user interface in Kibana allows for the query to only run against a single index.

That is not true. You query against an index pattern which is not the same as a single index in Elasticsearch. Your index pattern can be an alias and by extension aliases can be tied to multiple indices.

[jkowall]

I guess you could do that @JacobBrandt, but then if you have hundreds or in our case thousands of indices how would the user query multiple indices at the same time? I am not familiar with index patterns, but I assume there has to be some kind of regex match?

@galangel What do you think?

[JacobBrandt]

@jkowall I have hundreds of indices as well. Look into Aliases on Elasticsearch. You can have 1 alias for hundreds or in your case thousands of indices. You then create an index pattern from the alias, which you can give any name you want. It's literally why it was designed this way. So that you can query multiple indices, think time based indices in a cluster. You don't want users needing to specifically pick the month of that index to get to their data. They pick the named alias.

[JacobBrandt]

This alias mechanism is also not limited to only indices with the same mapping template. I use it to group like minded indices together. They share similarly mapped fields (like you mentioned with Request-ID) but also have many other various fields not in the other index. Kibana handles all of this including letting you know when there is conflicting fields when you go to create the index pattern and the indices in the alias have conflicting mappings.

[jkowall]

I had a look at that feature. The only challenge is that your index needs to have a certain type of name associated with it. So you can do a wild card. In this case we want to be able to select multiple indices that do not have the same pattern within their names.

[JacobBrandt]

@jkowall Again what you are saying is wrong. An alias can be named whatever you want it to be and the indices you add to the alias can be whatever name as well.

See Elastic's example

The alias is named alias1 and the index added to this alias is my-index-000001. You could then create an index pattern in Kibana with that named alias by typing alias1 and creating the index pattern.

The name "index pattern" in Kibana does not mean that the name you enter into the form to create an index pattern must be a regex pattern or even the name of an index. There is a large discussion even the Kibana developers have about the confusing name. I hope this helps.

[mihirsoni]

I think @JacobBrandt @jkowall is trying to propose search across multiple index pattern, and that's why I raised the question over the filters.

So for a reason I feel this is still valid use case

• A customer is ingesting all micro-services data within same OpenSearch Cluster
• One create index patterns based on different micro services
• Now let's say if you would have to dive deep based on the request id, as of today you can just do it over single index pattern and not across the index pattern
  So I agree, there is a way to search across multiple indices using index pattern, but there is no way currently to search across the index patterns, which I feel could be useful.

Despite, we might need to dive deep on user experience and filter experience with it, that could become tricky and challenging.

[mihirsoni]

Also I agree with @JacobBrandt that we could achieve the same functionality from creating an alias:

• User can create alias, which points to multiple index regex. i.e

POST /_aliases
{
  "actions" : [
    { "add" : { "index" : "serive-1*,service-2*, proxy*", "alias" : "logs" } }
  ]
}

• You could now create an index pattern using the logs* and it would ensure to search across the fields.

Does this solve what you would have to achieve @jkowall @galangel

[AmiStrn]

@JacobBrandt you are correct regarding creating the aliases as being related to this feature, however:

• in terms of security Elasticsearch also mention:

If the Elasticsearch security features are enabled, you must have the following index privileges:
To use the add or remove action, you must have the manage index privilege for both the index and index alias.
To use the remove_index action, you must have the manage index privilege for the index.

This is for good reason - if some OpenSearch users want secure multitenant clusters then they would disable the ability of most users to create/delete aliases. Therefore, if you want to put a frame around the different indices the user can group together for a request this proposed feature is offering that exactly.

• Secondly, thinking in terms of the UX, this is by far more comfortable for the majority of Kibana users. It keeps the suggested alias creations simple and accesable. We have also found this to be useful in real-life scenarios, as @galangel has mentioned. Most users are not savvy enough to create the correct alias they think they want, nor understand the implications of the mapping errors.
• I do agree that the alias is providing an equivalent outcome, but in a less user-friendly way.

[galangel]

Thank you all for your comments,
We want to solve this without having the user learn how to use an alias, or maintaining an index naming scheme for every possible search.

What we want is to search multiple indexes On Demand, if we'll go with the alias, then I want to make it on the fly.
Not making aliases in advance for every situation.

[JacobBrandt]

This is an Open Source project
This is a good thing and important to remember.

Not user friendly

I do agree that the alias is providing an equivalent outcome, but in a less user-friendly way.

Solvable with a UI experience that allows users to create aliases. Aliases, do more than just allowing to group indices together, like filtering and routing, so this approach provides more functionality then your proposed suggestion to the non-issue.

Technical challenges of the proposals suggestion
There are multiple technical challenges with the approach you are presenting with selecting multiple index patterns within the Discover view:

• SearchSource operates on a single named search target
• These Discover views become saved searches
  • Saved searches can be used in visualizations and visualizations understand a single named search target because they make requests with a SearchSource which like I just mentioned operates on a single named search target and not multiple names
• When fields are named the same between these index patterns:
  • What formatter are you going to use?
  • What popularity?

I'm looking at the big picture here instead of just Logz.io use case.

[AmiStrn]

@JacobBrandt I am going to refer to what you wrote from

Not user friendly

Onward, As the first paragraphs don't present yourself in the best light. Let's try to keep things on an engineering level here. If you want to talk about a specific company then I believe this is not the place. Feel free to dm me about questions regarding logz.io.

Regarding the option to add a ui to create aliases rather than a "simple" grouping --
Awesome! I really like this and think that is a way to get the users to benefit from both worlds:)

Regarding the challenges - @galangel perhaps these should be added to the proposal as open issues?

Thank you @JacobBrandt for the important feedback.

[JacobBrandt]

@AmiStrn You're right, I apologize then. Removed them so we can keep this content at the engineering level of the proposal.

The challenges I listed only present themselves if the plan is to allow users to select multiple index patterns from Discover view. If instead the tool is used as designed they don't exist.

[AmiStrn]

I like this dialogue. I think the feature description needs to speak for itself so you are correct @JacobBrandt that something is missing. I apologize for any frustration, I appreciate your involvement.

[jkowall]

Closing this out until we modify the plugin system, proposal is coming soon.