Unable to sort if there are blank empty fields

[desean1625]

Kibana version:
8.10.3
Elasticsearch version:
8.10.3
Describe the bug:
When sorting in discover if you have an empty field, you get an error saying the field has incompatible sort types.

This appears to be more of an issue with changes to elastic search, but can also be mitigated on the Kibana side by sending the correct type as the sort "unmapped_type" that comes back from the dataview fields types.

Currently discover sets all unmapped types to boolean in the query. If you set it to the field type the query works as expected.
Change

POST kibana_*/_search
{
    "sort": [
        {
            "AvgTicketPrice": {
                "order": "desc",
                "unmapped_type": "boolean"
            }
        }
    ],
    "size": 500
}

to

POST kibana_*/_search
{
    "sort": [
        {
            "AvgTicketPrice": {
                "order": "desc",
                "unmapped_type": "float"
            }
        }
    ],
    "size": 500
}

See elasticsearch issue elastic/elasticsearch#102723

Steps to reproduce:
Steps to recreate issue.
From kibana add sample data sets ['Sample flight data','Sample eCommerce orders']
Create dataview
name:test
index patter:kibana_*
timefield: --- I don't want to use the time filter ---

From discover sort by AvgTicketPrice

Expected behavior:
Records return

Any additional context:
Worked in 7.17 and 8.6.4

[ndmitch311]

@Villegas12 we opened this ticket also which relates to elastic/elasticsearch#102723

[benwtrent]

Discover should not do this. Elasticsearch is doing the correct thing. If you give us an unmapped type that is different than the mapped type, I don't see how we can accurately sort the values.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[benwtrent]

FYI, I have fixed behavior to be more forgiving on the ES side. elastic/elasticsearch#102723

Basically, if there are no docs for a conflicting type, it is not considered part of the sort, and things continue as normal.

This may fix the bug that is seen here. My concern is that if we are defaulting to this "unmapped type" but there actually IS an unmapped doc, what happens. Seems like a bug if we default to an invalid unmapped type.

[davismcphee]

Related: #167375, #4593.

The "unmapped_type": "boolean" default (not sure why exactly boolean was chosen originally) is static but can be configured using sort:options in Advanced Settings:

There are discussions in the linked issues around dynamically setting this value, but I believe the outcome was that this would be tricky since a data view's index pattern may target fields with multiple conflicting types, so we wouldn't know which value to set. But maybe there's an opportunity to improve this in cases where the index pattern doesn't have conflicting types?

Or does elastic/elasticsearch#102723 remove the need for this entirely and we should instead consider just dropping the default unmapped_type setting?

[benwtrent]

@davismcphee I would be hesitant around removing this. The change I made simply changes ES to behave like it did < 8.4 but with better error messages.

Indeed, it is weird to me that there is a global default for this at all as sort by fields each have their different type and you always want the unmapped type to equal the expected type for the field.

[desean1625]

fixed in elastic/elasticsearch#102779