From bfe7850ce0c6904d1939b9e176bf0312c87ab241 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Thu, 11 Jun 2020 10:57:15 -0400 Subject: [PATCH] [DOCS] Move search API's `docvalue_fields` examples (#57760) Changes: * Condenses and relocates the `docvalue_fields` example to the 'Run a search' page. * Adds docs for the `docvalue_fields` request body parameter. * Updates several related xrefs. Co-authored-by: debadair --- docs/plugins/mapper-size.asciidoc | 2 +- .../metrics/tophits-aggregation.asciidoc | 4 +- .../mapping/fields/source-field.asciidoc | 2 +- docs/reference/mapping/params/store.asciidoc | 2 +- docs/reference/mapping/types/nested.asciidoc | 4 +- docs/reference/redirects.asciidoc | 9 +- .../search/request/docvalue-fields.asciidoc | 73 +---------- .../search/request/inner-hits.asciidoc | 2 +- .../search/request/source-filtering.asciidoc | 2 +- .../search/request/stored-fields.asciidoc | 4 +- docs/reference/search/search-fields.asciidoc | 120 +++++++++++++++--- docs/reference/search/search.asciidoc | 51 ++++++++ .../suggesters/completion-suggest.asciidoc | 2 +- .../apis/put-repo-api.asciidoc | 2 +- 14 files changed, 174 insertions(+), 105 deletions(-) diff --git a/docs/plugins/mapper-size.asciidoc b/docs/plugins/mapper-size.asciidoc index 36cce29436dcc..9776c5c8f277c 100644 --- a/docs/plugins/mapper-size.asciidoc +++ b/docs/plugins/mapper-size.asciidoc @@ -86,7 +86,7 @@ GET my_index/_search {ref}/search-request-body.html#request-body-search-script-fields[script field] to return the `_size` field in the search response. <5> Uses a -{ref}/search-request-body.html#request-body-search-docvalue-fields[doc value +{ref}/run-a-search.html#docvalue-fields[doc value field] to return the `_size` field in the search response. Doc value fields are useful if {ref}/modules-scripting-security.html#allowed-script-types-setting[inline diff --git a/docs/reference/aggregations/metrics/tophits-aggregation.asciidoc b/docs/reference/aggregations/metrics/tophits-aggregation.asciidoc index 2ae3b8a658a41..8fff7365aa71c 100644 --- a/docs/reference/aggregations/metrics/tophits-aggregation.asciidoc +++ b/docs/reference/aggregations/metrics/tophits-aggregation.asciidoc @@ -20,10 +20,10 @@ The top_hits aggregation returns regular search hits, because of this many per h * <> * <> * <> -* <> +* <> * <> * <> -* <> +* <> * <> * <> diff --git a/docs/reference/mapping/fields/source-field.asciidoc b/docs/reference/mapping/fields/source-field.asciidoc index 17526438b79d0..ded8fc3a6b41b 100644 --- a/docs/reference/mapping/fields/source-field.asciidoc +++ b/docs/reference/mapping/fields/source-field.asciidoc @@ -76,7 +76,7 @@ stored. WARNING: Removing fields from the `_source` has similar downsides to disabling `_source`, especially the fact that you cannot reindex documents from one Elasticsearch index to another. Consider using -<> instead. +<> instead. The `includes`/`excludes` parameters (which also accept wildcards) can be used as follows: diff --git a/docs/reference/mapping/params/store.asciidoc b/docs/reference/mapping/params/store.asciidoc index 33debc03e58a4..42aefefefbc2f 100644 --- a/docs/reference/mapping/params/store.asciidoc +++ b/docs/reference/mapping/params/store.asciidoc @@ -9,7 +9,7 @@ Usually this doesn't matter. The field value is already part of the <>, which is stored by default. If you only want to retrieve the value of a single field or of a few fields, instead of the whole `_source`, then this can be achieved with -<>. +<>. In certain situations it can make sense to `store` a field. For instance, if you have a document with a `title`, a `date`, and a very large `content` diff --git a/docs/reference/mapping/types/nested.asciidoc b/docs/reference/mapping/types/nested.asciidoc index 87e75b9aefc31..6ec31be812d2e 100644 --- a/docs/reference/mapping/types/nested.asciidoc +++ b/docs/reference/mapping/types/nested.asciidoc @@ -180,8 +180,8 @@ For instance, if a string field within a nested document has during the highlighting, these offsets will not be available during the main highlighting phase. Instead, highlighting needs to be performed via <>. The same consideration applies when loading -fields during a search through <> -or <>. +fields during a search through <> or <>. ============================================= diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index 7b65a254da17b..6dd7b2d24c879 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -910,5 +910,10 @@ See <>. [role="exclude",id="request-body-search-source-filtering"] ==== Source filtering -See <>. -//// \ No newline at end of file +See <>. + +[role="exclude",id="request-body-search-docvalue-fields"] +==== Doc value fields + +See <>. +//// diff --git a/docs/reference/search/request/docvalue-fields.asciidoc b/docs/reference/search/request/docvalue-fields.asciidoc index 45e09dce78a60..14aaa6ec2691d 100644 --- a/docs/reference/search/request/docvalue-fields.asciidoc +++ b/docs/reference/search/request/docvalue-fields.asciidoc @@ -1,73 +1,4 @@ [[request-body-search-docvalue-fields]] -==== Doc value Fields +==== Doc value fields -Allows to return the <> representation of a field for each hit, for -example: - -[source,console] --------------------------------------------------- -GET /_search -{ - "query" : { - "match_all": {} - }, - "docvalue_fields" : [ - "my_ip_field", <1> - { - "field": "my_keyword_field" <2> - }, - { - "field": "my_date_field", - "format": "epoch_millis" <3> - } - ] -} --------------------------------------------------- - -<1> the name of the field -<2> an object notation is supported as well -<3> the object notation allows to specify a custom format - -Doc value fields can work on fields that have doc-values enabled, regardless of whether they are stored - -`*` can be used as a wild card, for example: - -[source,console] --------------------------------------------------- -GET /_search -{ - "query" : { - "match_all": {} - }, - "docvalue_fields" : [ - { - "field": "*_date_field", <1> - "format": "epoch_millis" <2> - } - ] -} --------------------------------------------------- - -<1> Match all fields ending with `field` -<2> Format to be applied to all matching fields. - -Note that if the fields parameter specifies fields without docvalues it will try to load the value from the fielddata cache -causing the terms for that field to be loaded to memory (cached), which will result in more memory consumption. - -[float] -====== Custom formats - -While most fields do not support custom formats, some of them do: - - - <> fields can take any <>. - - <> fields accept a https://docs.oracle.com/javase/8/docs/api/java/text/DecimalFormat.html[DecimalFormat pattern]. - -By default fields are formatted based on a sensible configuration that depends -on their mappings: `long`, `double` and other numeric fields are formatted as -numbers, `keyword` fields are formatted as strings, `date` fields are formatted -with the configured `date` format, etc. - -NOTE: On its own, `docvalue_fields` cannot be used to load fields in nested -objects -- if a field contains a nested object in its path, then no data will -be returned for that docvalue field. To access nested fields, `docvalue_fields` -must be used within an <> block. \ No newline at end of file +See <>. diff --git a/docs/reference/search/request/inner-hits.asciidoc b/docs/reference/search/request/inner-hits.asciidoc index 2d0b0d63e25c0..21f2b06b46eaf 100644 --- a/docs/reference/search/request/inner-hits.asciidoc +++ b/docs/reference/search/request/inner-hits.asciidoc @@ -74,7 +74,7 @@ Inner hits also supports the following per document features: * <> * <> * <> -* <> +* <> * <> * <> diff --git a/docs/reference/search/request/source-filtering.asciidoc b/docs/reference/search/request/source-filtering.asciidoc index 39f478331775b..dcad5d2ee9054 100644 --- a/docs/reference/search/request/source-filtering.asciidoc +++ b/docs/reference/search/request/source-filtering.asciidoc @@ -1,4 +1,4 @@ [[request-body-search-source-filtering]] ==== Source filtering -See <>. +See <>. diff --git a/docs/reference/search/request/stored-fields.asciidoc b/docs/reference/search/request/stored-fields.asciidoc index 9515fc8cb78b6..4224343455492 100644 --- a/docs/reference/search/request/stored-fields.asciidoc +++ b/docs/reference/search/request/stored-fields.asciidoc @@ -3,7 +3,7 @@ WARNING: The `stored_fields` parameter is about fields that are explicitly marked as stored in the mapping, which is off by default and generally not recommended. -Use <> instead to select +Use <> instead to select subsets of the original source document to be returned. Allows to selectively load specific stored fields for each document represented @@ -62,5 +62,5 @@ GET /_search } -------------------------------------------------- -NOTE: <> and <> parameters cannot be activated if `_none_` is used. +NOTE: <> and <> parameters cannot be activated if `_none_` is used. diff --git a/docs/reference/search/search-fields.asciidoc b/docs/reference/search/search-fields.asciidoc index 77010912e3562..1c8b60b4c22fe 100644 --- a/docs/reference/search/search-fields.asciidoc +++ b/docs/reference/search/search-fields.asciidoc @@ -1,12 +1,43 @@ [discrete] [[search-fields]] -=== Return fields in a search +=== Retrieve selected fields By default, each hit in the search response includes the document <>, which is the entire JSON object that was -provided when indexing the document. If you only need certain fields in the -search response, you can use the `_source` parameter to restrict what parts of -the source are returned. This is called _source filtering_. +provided when indexing the document. If you only need certain source fields in +the search response, you can use the <> to +restrict what parts of the source are returned. + +Returning fields using only the document source has some limitations: + +* The `_source` field does not include <> or +<>. Likewise, a field in the source does not contain +values copied using the <> mapping parameter. +* Since the `_source` is stored as a single field in Lucene, the whole source +object must be loaded and parsed, even if only a small number of fields are +needed. + +To avoid these limitations, you can: + +* Use the <> +parameter to get values for selected fields. This can be a good +choice when returning a fairly small number of fields that support doc values, +such as keywords and dates. +* Use the <> parameter to get the values for specific stored fields. (Fields that use the <> mapping option.) + +You can find more detailed information on each of these methods in the +following sections: + +* <> +* <> +* <> + +[discrete] +[[source-filtering]] +=== Source filtering + +You can use the `_source` parameter to select what fields of the source are +returned. This is called _source filtering_. .*Example* [%collapsible] @@ -91,23 +122,74 @@ GET /_search ---- ==== -Returning fields using only the document source has some limitations: -* The `_source` field does not include <> or -<>. Likewise, a field in the source does not contain -values copied using the <> mapping parameter. -* Since the `_source` is stored as a single field in Lucene, the whole source -object must be loaded and parsed, even if only a small number of fields are -needed. +[discrete] +[[docvalue-fields]] +=== Doc value fields -{es} supports some alternative methods for returning fields that help avoid -these downsides: +You can use the <> parameter to return +<> for one or more fields in the search response. -* The <> -parameter allows for loading fields from their doc values. This can be a good -choice when returning a fairly small number of fields that support doc values, -such as keywords and dates. -* It's also possible to store an individual field's values by using the +Doc values store the same values as the `_source` but in an on-disk, +column-based structure that's optimized for sorting and aggregations. Since each +field is stored separately, {es} only reads the field values that were requested +and can avoid loading the whole document `_source`. + +Doc values are stored for supported fields by default. However, doc values are +not supported for <> or +{plugins}/mapper-annotated-text-usage.html[`text_annotated`] fields. + +.*Example* +[%collapsible] +==== +The following search request uses the `docvalue_fields` parameter to +retrieve doc values for the following fields: + +* Fields with names starting with `my_ip` +* `my_keyword_field` +* Fields with names ending with `_date_field` + +[source,console] +---- +GET /_search +{ + "query": { + "match_all": {} + }, + "docvalue_fields": [ + "my_ip*", <1> + { + "field": "my_keyword_field" <2> + }, + { + "field": "*_date_field", + "format": "epoch_millis" <3> + } + ] +} +---- + +<1> Wildcard patten used to match field names, specified as a string. +<2> Wildcard patten used to match field names, specified as an object. +<3> With the object notation, you can use the `format` parameter to specify a + format for the field's returned doc values. <> support a + <>. <> support a + https://docs.oracle.com/javase/8/docs/api/java/text/DecimalFormat.html[DecimalFormat + pattern]. Other field datatypes do not support the `format` parameter. +==== + +TIP: You cannot use the `docvalue_fields` parameter to retrieve doc values for +nested objects. If you specify a nested object, the search returns an empty +array (`[ ]`) for the field. To access nested fields, use the +<> parameter's `docvalue_fields` +property. + + +[discrete] +[[stored-fields]] +=== Stored fields + +It's also possible to store an individual field's values by using the <> mapping option. You can use the -<> parameter to return +<> parameter to include these stored values in the search response. diff --git a/docs/reference/search/search.asciidoc b/docs/reference/search/search.asciidoc index 811df60efe5e9..393994e0aa44f 100644 --- a/docs/reference/search/search.asciidoc +++ b/docs/reference/search/search.asciidoc @@ -270,6 +270,35 @@ If `true`, returns document version as part of a hit. Defaults to `false`. [[search-search-api-request-body]] ==== {api-request-body-title} +[[search-docvalue-fields-param]] +`docvalue_fields`:: +(Optional, array of strings and objects) +Array of wildcard (`*`) patterns. The request returns doc values for field names +matching these patterns in the `hits.fields` property of the response. ++ +You can specify items in the array as a string or object. +See <>. ++ +.Properties of `docvalue_fields` objects +[%collapsible] +==== +`field`:: +(Required, string) +Wildcard pattern. The request returns doc values for field names matching this +pattern. + +`format`:: +(Optional, string) +Format in which the doc values are returned. ++ +For <>, you can specify a date <>. For <> fields, you can specify a +https://docs.oracle.com/javase/8/docs/api/java/text/DecimalFormat.html[DecimalFormat +pattern]. ++ +For other field datatypes, this parameter is not supported. +==== + [[request-body-search-explain]] `explain`:: (Optional, boolean) If `true`, returns detailed information about score @@ -533,6 +562,28 @@ Original JSON body passed for the document at index time. + You can use the `_source` parameter to exclude this property from the response or specify which source fields to return. + +`fields`:: ++ +-- +(object) +Contains field values for the documents. These fields must be specified in the +request using one or more of the following request parameters: + +* <> +* <> +* <> + +This property is returned only if one or more of these parameters are set. +-- ++ +.Properties of `fields` +[%collapsible%open] +====== +``:: +(array) +Key is the field name. Value is the value for the field. +====== ===== ==== diff --git a/docs/reference/search/suggesters/completion-suggest.asciidoc b/docs/reference/search/suggesters/completion-suggest.asciidoc index 54d2fee0cc8cd..e36af188af76e 100644 --- a/docs/reference/search/suggesters/completion-suggest.asciidoc +++ b/docs/reference/search/suggesters/completion-suggest.asciidoc @@ -219,7 +219,7 @@ The configured weight for a suggestion is returned as `_score`. The return the full document `_source` by default. The size of the `_source` can impact performance due to disk fetch and network transport overhead. To save some network overhead, filter out unnecessary fields from the `_source` -using <> to minimize +using <> to minimize `_source` size. Note that the _suggest endpoint doesn't support source filtering but using suggest on the `_search` endpoint does: diff --git a/docs/reference/snapshot-restore/apis/put-repo-api.asciidoc b/docs/reference/snapshot-restore/apis/put-repo-api.asciidoc index 66f8c9f3356d4..773679aca9c8b 100644 --- a/docs/reference/snapshot-restore/apis/put-repo-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/put-repo-api.asciidoc @@ -113,7 +113,7 @@ source-only snapshots that take up to 50% less space on disk. + Source-only snapshots are only supported if the <> is enabled and no -<> is applied. +<> is applied. + WARNING: Source-only snapshots contain stored fields and index metadata. They do not include index or doc values structures and are not searchable when restored.