[Data Services] Support fields parameter in high level search apis

[stacey-gammon]

Many developers currently access field values directly from _source (this object is accessible in each hit from the hits array in a search request response). This exposes the raw source fields, which is often not actually what is desired. What is often desired is the mapped field values. This subtle difference leads to issues like this and this.

Recently Elasticsearch has recently exposed a way to access these mapped field values.

The data search services needs to provide a way for developers to use this new search parameter, in order to solve problems like runtime field values showing up blank in discover.

There are some differences in values retrieved from _source vs the fields array:

	_source	fields array
Includes alias fields	No	Yes
Includes runtime fields	No	Yes
Objects are flattened	No	Yes
Nested fields are flattened	No	Yes

There are performance considerations as well. If a searcher wants to get values for certain fields, maybe they want to turn _source off.

The bare minimum we'll need is the ability to allow users to define the list of fields returned in the fields array. Unfortunately an initial investigation revealed that it may not be straightforward to create a good DX, given that searchSource.fields(..) is already used for something else.

Additional possibilities that probably don't need to be included in the initial API:

• ability for users to turn off _source to improve performance
• ability for users to configure flattening (not yet exposed in ES but could be if desired).

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[ppisljar]

with esdocs our approach was to use values from fields array and fallback to _source if fields array is not present.

[ppisljar]

searchSource already has a _source parameter which can be set to false to not return anything under _source

[ppisljar]

fields is already used to signal we want to retrieve only specific fields.

currently:

• we make sure we don't request script_fields that are not listed in fields parameter
• we make sure we don't request 'docvalue_fields' that are not listed in fields parameter
• we make sure only the fields listed in fields parameter are requested on stored_fields and _source

elasticsearch says this in their docs:

fields
(object) Contains field values for the documents. These fields must be specified in the request using one or more of the following request parameters:

docvalue_fields
script_fields
stored_fields
This property is returned only if one or more of these parameters are set.

so it seems we are doing everything correctly, the response should contain fields array

[ppisljar]

i am looking for this fields paramater for es search api, can somebody please link me to docs as i can't find it ?

[ppisljar]

checked the response and it does contain fields array, but not with all the fields.

[ppisljar]

could something like this work ? #77348

[stacey-gammon]

ES docs: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/search-fields.html#search-fields

In the docs it's referred to as "the fields parameter on a search request", but internally it's more often referred to as the "Fields API". The former is more accurate, but be aware of the latter because you'll probably hear it a lot.

[lukeelmers]

I have a POC up which introduces the fields API in SearchSource: #82383

It isn't intended to merge as-is, but rather to align on the exact behavior we want so that we can move forward with a final implementation and unblock #77107.