[Observability] Query bar should, by default, only show relevant fields

[andrewvc]

The query bar, today, basically lists the entirety of ECS when the autocomplete drops down, making the autocomplete less than useful. This makes sense in the discover app where the kuery bar comes from, but in a curated solution it should generally limit itself to fields a user is likely to want. See the image below for an example of the poor field names that come up by default. No one has ever wanted to query by agent.build.original I'm fairly certain, and yet it is our second result.

The proposal here is to change the kuery bar behavior as follows:

• Create a manually curated list per app of common fields, no more than a couple dozen, that users would actually like to see in the search. This would enhance discoverability
• Only show these fields by default in the kuery bar, add a toggle somewhere to show all
  - As a bonus, using local storage, keep track of uncommon fields the user has used historically, and add those to auto-complete

I crossed out the last item because it turns out the kuery bar already does this as @sqren points out below

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[formgeist]

Thanks for creating this @andrewvc! I like the simplicity of the proposals 👍 I wanted to mention that we've spoken to the Security team in our filtering explorations, and I think this is also useful for their search experience.

[sorenlouv]

I think this would be great.

@andrewvc Have you thought about what subset of fields to show? Would it be a manually curated list of fields, or is there a heuristic to determine which fields to show/hide?

Nvm, you already answered this:

Create a manually curated list per app of common fields

[sorenlouv]

As a bonus, using local storage, keep track of uncommon fields the user has used historically, and add those to auto-complete

It looks like the kuery bar already keeps a history (I previously searched for transaction.duration.us >1000)

[andrewvc]

That's great about the history, it's just the fields then!

[andrewvc]

I'm wondering @shahzad31 , could we drive this list of curated fields via the curated field list in Exploratory view? It makes sense to maintain a high level schema above the actual literal elasticsearch schema I think. Ideally, we'd search with more friendly names, so instead of monitor.duration.us you'd just get Monitor Execution Time or similar, and an easy way to deal with time units.

CC @paulb-elastic @drewpost @formgeist

[sorenlouv]

Related: #95558

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gbamparop]

Related https://github.com/elastic/apm-dev/issues/934 (internal)

[roshan-elastic]

I love this...

@neptunian - do we have scope to affect the universal search for metrics UI only or do we need an observability-wide solution?

[neptunian]

@roshan-elastic If I understand you correctly, as stated in the description this would be a "manually curated list per app".

[sorenlouv]

Crosslinking a somewhat related issue: #158986

tldr: the search bar was not available until the data view had loaded. This could take 10-20 sec on big clusters. The quick fix we agreed on was to make the data view optional and show the search bar immediately. However, a further improvement was suggested by @kertal that I think would solve this issue as well:

there might be cases like the given example, where just a subset of fields are of interest, maybe already known. if this would be the case, using a temporary data view that can be initialized with a given set of known fields could be an option, so there would be no fields fetching necessary, and the UI would therefor render within milliseconds instead of waiting seconds to return results.

By adding this temporary data view we will both be able to show suggestions to the user immediately, and the suggestions will more more relevant than the fields we show today. Improved speed and relevance = win-win!

cc @teresaalvarezsoler @stratoula

[roshan-elastic]

@roshan-elastic If I understand you correctly, as stated in the description this would be a "manually curated list per app".

Ah yes, sorry - missed that. Correct.

Let me have a think about this...I certainly agree with the goal of this issue. Not sure how we'd choose the right fields right now...

[sorenlouv]

Not sure how we'd choose the right fields right now...

APM should be able to come up with a handful of relevant fields that we want to show first, so from our perspective, having this would be an improvement of the user experience.

[gbamparop]

Let me have a think about this...I certainly agree with the goal of this issue. Not sure how we'd choose the right fields right now...

Telemetry about the fields that are being used in the search bar might be a good starting point. We plan to add this data point in APM as part of #146603.

• Only show these fields by default in the kuery bar, add a toggle somewhere to show all

It seems that most teams within Observability are interested in this. Shall we create a request for the unified search team to add support for it? cc @stratoula

[stratoula]

@gbamparop so you want to keep track of the fields that are used in the KQL bar in order to have a static list of the most popular fields? Do I get it correctly?
In discover fields that are used mostly are identified as popular (also saved in the dataview). I wonder if a mechanism like that is more powerful that gathering telemetry of the fields. Wdyt? (I guess that each user has different favorite fields, right?)

[stratoula]

@lukasolson I know that we are tracking some things with the KQL telemetry. Maybe we already can share some insights?

[gbamparop]

In discover fields that are used mostly are identified as popular (also saved in the dataview). I wonder if a mechanism like that is more powerful that gathering telemetry of the fields. Wdyt? (I guess that each user has different favorite fields, right?

@stratoula that's a good idea, is this being used to populate the history section mentioned here or it's displayed somewhere else?

Also, can we limit the amount of fields displayed in the suggestions right now?

[stratoula]

@gbamparop the popular fields in Discover are atm only a Discover app thing. They are being displayed as Popular fields in the field list (the one in the left).

Also, can we limit the amount of fields displayed in the suggestions right now?

No if you don't write anything you see all fields, as you start typing the fields are filtered based on your text. (unless I am wrong, I can check the code)

[sorenlouv]

I don't think the "popular fields" is what we want. Popular fields also have to be loaded from ES and I imagine will be subject to the same delays as the data view - so could be quite slow on big clusters.

Furthermore, popular fields don't help new users who don't know which fields to query.

Instead, I think it should be possible to supply unified search with a list of Elastic defined fields, that we expect users to want to search on.

[lukasolson]

@lukasolson I know that we are tracking some things with the KQL telemetry. Maybe we already can share some insights?

I think right now we're only collecting when users opt in/out of using KQL altogether (this telemetry was added quite a long time ago).

Folks on this issue may also be interested in the recent work by @XavierM to add support for a "suggestions abstraction", which allows you to specify a list of fields to suggest: #158106

[sorenlouv]

Folks on this issue may also be interested in the recent work by @XavierM to add support for a "suggestions abstraction", which allows you to specify a list of fields to suggest: #158106

Very much! From that PR it looks like it is possible to supply a list of suggestions like:

   <SearchBar
      suggestionsAbstraction={suggestions}

It is not clear to me exactly how the suggestions and displayed eg. if they are given preference. I hope they show up on top even after the data view has loaded.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)