[BUG] Alias fields not visible in Discover tab for non-date field types

[ananzh]

Describe the bug
Alias fields in OpenSearch are not consistently visible in the Discover . The visibility depends on the data type of the field being aliased. Specifically, alias fields are visible when the aliased field is of type "date", but not visible when the aliased field is of type "keyword".

To Reproduce

1. Create two indices with different mappings:

   Index 1 (keyword type):
   PUT movies-1
   {
     "mappings": {
       "properties": {
         "year": {
           "type": "keyword"
         },
         "release_date_movies": {
           "type": "alias",
           "path": "year"
         }
       }
     }
   }

   Index 2 (date type):
   PUT movies-2
   {
     "mappings": {
       "properties": {
         "year": {
           "type": "date"
         },
         "release_date_movies": {
           "type": "alias",
           "path": "year"
         }
       }
     }
   }

2. Add sample data to each index:

   For movies-1:
   POST movies-keyword/_doc
   {
     "year": "2024"
   }

   For movies-2:
   POST movies-date/_doc
   {
     "year": "2024-01-01"
   }

3. Open Discover.
4. Select the index pattern for movies-1 and observe the available fields.

5. Select the index pattern for movies-2 and observe the available fields.

Expected behavior
Both "year" and "release_date_movies" fields should be visible in the Discover for both indices.

Actual Behavior:

• For movies-1: Only the "year" field is visible in the Discover tab.
• For movies-2: Both "year" and "release_date_movies" fields are visible in the Discover tab.

OpenSearch Version
currently only try main

Dashboards Version
currently only try main

[huyaboo]

Will triage and release feedback today

[ashwin-pc]

We are currently working on supporting auto complete for Discover for 2.17. That should help mitigate some of the pain here until we dig deeper into this issue.

[kavilla]

One thing that I noticed that the type of the first screenshot of the release date movies is of type text indicated by the 't' and doesn't display.

While the second screenshot indicates it is a date type indicated by the calendar icon.

.

Within OpenSearch Dashboards we map mappings for OpenSearch to specific types here: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/data/common/osd_field_types/osd_field_types_factory.ts#L46.

Which is why without aliasing it wouldn't display something like release_date_movies.keyword as that serves a functional purpose. But for me that would indicate that even though in OpenSearch Dashboards, it doesn't look like a nested object to users but it might actually be being treated as a nested object. Which means it could be breaking one of the rules here:

https://opensearch.org/docs/latest/field-types/supported-field-types/alias/

Do you mind setting it to an alias with the same level of nested structure?

[LDrago27]

Why are we not showing the results of alias fields when it is not date-time field ?

We are not receiving those values from OpenSearch in the call hence we don’t show those data points.

Request Sent by Discover

{
   "sort": [{
       "_score": {
           "order": "desc"
       }
   }],
   "size": 500,
   "version": true,
   "stored_fields": [
       "*"
   ],
   "script_fields": {},
   "docvalue_fields": [],
   "_source": {
       "excludes": []
   },
   "query": {
       "bool": {
           "must": [],
           "filter": [{
               "match_all": {}
           }],
           "should": [],
           "must_not": []
       }
   },
   "highlight": {
       "pre_tags": [
           "@opensearch-dashboards-highlighted-field@"
       ],
       "post_tags": [
           "@/opensearch-dashboards-highlighted-field@"
       ],
       "fields": {
           "*": {}
       },
       "fragment_size": 2147483647
   }
}

Response

{
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 0,
        "hits": [{
            "_index": "movies-1",
            "_id": "2weMgJEBHfJrXCuUH_dU",
            "_version": 1,
            "_score": 0,
            "_source": {
                "year": "2024-01-01"
            }
        }]
    }
}

Why are we seeing date-time alias fields ?

For any date time fields we have a logic that populates the datetime field in the “docvalue_fields": [] explicitly which causes the doctype fields to be part of the queries response , Hence Discover is able to show them

Request

{
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    }
  ],
  "size": 500,
  "version": true,
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "release_date_movies",
      "format": "date_time"
    },
    {
      "field": "year",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@opensearch-dashboards-highlighted-field@"
    ],
    "post_tags": [
      "@/opensearch-dashboards-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}

Response

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0,
    "hits": [
      {
        "_index": "movies-2",
        "_id": "3AeMgJEBHfJrXCuUKPfb",
        "_version": 1,
        "_score": 0,
        "_source": {
          "year": "2024-01-01"
        },
        "fields": {
          "release_date_movies": [
            "2024-01-01T00:00:00.000Z"
          ],
          "year": [
            "2024-01-01T00:00:00.000Z"
          ]
        }
      }
    ]
  }
}

The difference is the docvalue fields array between the 2 requests.

What is docvalue search parameter ?

DocValues are essentially a data structure used by Elasticsearch to store the fields to support efficient sorting and searching operations.

Why are we using docvalue fields only for date-time field type ?

date-time fields can have different formats in original source documents, but we want a common format that can be used for filtering and aggregation. Therefore instead of showing the original source form of those we are retuning the docvalue or the common way how they are stored internally in ElasticSearch.

Code snippet where we are populating the docValueFields.

const docvalueFields = reject(this.fields.getByType('date'), 'scripted').map(
      (dateField: any) => {
        return {
          field: dateField.name,
          format:
            dateField.esTypes && dateField.esTypes.indexOf('date_nanos') !== -1
              ? 'strict_date_time'
              : 'date_time',
        };
      }
    );

cc @ashwin-pc

[himanshuramchandanisonos]

This seems to be fixed in 2.15 now.