Better field summaries

[joshuarrrr]

Is your feature request related to a problem? Please describe.

One of the fundamental tasks in data exploration is understanding the "shape" of the data. Index patterns may have many different fields, and figuring out which fields are interesting, useful, or complete is the first step in separating signal from noise. Field titles and types alone are not sufficient (titles alone may even be misleading, depending on the data source). So Discover allows users to click on each field title to see a quick summary:

The problem is that, for many fields, these summaries are essentially useless:

Describe the solution you'd like

1. Summaries should be useful for the given field type. While "Top 5 Values" is fine for string fields, it makes no sense for many other field types (timestamp, as seen above, or _id are particularly egregious examples). Instead, we should provide useful summary data. A numerical field summary might include some sort of distribution plot and min, max, median labels. A boolean field might be better summarized with a pie chart.
   
2. Make the "Visualize" button useful. The current implementation always takes a user to a bar chart, which has all of the same problems as 1. Now that we're building a new visualization creation flow, it should be even easier to pre-load a suggested visualization, while giving users an easy way to switch it.
3. Sampled data should be obvious. Sampled summaries can still be useful, but may not accurately reflect the current search context (by default, Discover's field summaries only count the first 500 matching records).
4. Empty/missing values matter. The current implementation does list how many of the sampled records "exist", but we could make it easier to digest at a glance.
5. Can we summarize all fields? Geo and attachment fields are currently excluded from summarization. But is there something useful we could say about them?

Note that we're also adding similar field summaries to the new visualization creation process, so all of the above (excluding 2) apply equally to that tool.

Describe alternatives you've considered

Most of the above suggestions could be implemented with the same basic UI components, it's just a matter of what information and visualization to put in the popover. But we may also want to explore completely different UI flows for summarizing/inspecting a field, particularly if rethinking the entire Discover experience.

[kavilla]

@opensearch-project/opensearch-ux can you check this out?

[KrooshalUX]

@joshuarrrr

• Can you further elaborate on point 1 "Summaries should be useful for the given field type" ? In order to make further UI recommendations, it might be helpful to put together a matrix of field types and specifically what would provide value to display. Perhaps we can tease this information out together.

• In regards to point 2, do we have the ability to specify paths per field type? Perhaps while we continue to add chart types in the experimental Drag and Drop feature release, we can prescribe a targeted vis type for a p0.

[joshuarrrr]

Starting with the second question first:

In regards to point 2, do we have the ability to specify paths per field type?

Yes, we have the ability to, because we know the field type when it's clicked and can have different summary experiences (or visualization paths) for different types. Currently we don't really utilize that info, except to show the "Analysis is not available..." message for geo and attachment fields.

Can you further elaborate on point 1 "Summaries should be useful for the given field type" ? In order to make further UI recommendations, it might be helpful to put together a matrix of field types and specifically what would provide value to display.

Yeah, I totally agree that we'd want to first list all the field types and identify which ones don't make sense for the current "Top 5" summary. Once we have those, we can start mapping from field type -> summary display.

But just a few examples to demonstrate what I have in mind:

Boolean - the visualization could be a single "part of the whole" bar chart or a pie chart (excuse these very rough mocks!):

Numerical - we could explore different ways of visualizing the distribution and highlighting key summary stats. Not this, but something in this direction would be much more useful than the top 5 values: