Merge branch '8.16' into backport/8.16/pr-114946

dakrone · Oct 21, 2024 · c1978ba · c1978ba
2 parents 1fbec14 + a6fe784
commit c1978ba
Show file tree

Hide file tree

Showing 100 changed files with 2,190 additions and 569 deletions.
diff --git a/docs/changelog/112250.yaml b/docs/changelog/112250.yaml
@@ -0,0 +1,5 @@
+pr: 112250
+summary: Do not exclude empty arrays or empty objects in source filtering
+area: Search
+type: bug
+issues: [109668]
diff --git a/docs/changelog/114295.yaml b/docs/changelog/114295.yaml
@@ -0,0 +1,5 @@
+pr: 114295
+summary: "Reprocess operator file settings when settings service starts, due to node restart or master node change"
+area: Infra/Settings
+type: enhancement
+issues: [ ]
diff --git a/docs/changelog/115031.yaml b/docs/changelog/115031.yaml
@@ -0,0 +1,5 @@
+pr: 115031
+summary: Bool query early termination should also consider `must_not` clauses
+area: Search
+type: enhancement
+issues: []
diff --git a/docs/changelog/115048.yaml b/docs/changelog/115048.yaml
@@ -0,0 +1,5 @@
+pr: 115048
+summary: Add timeout and cancellation check to rescore phase
+area: Ranking
+type: enhancement
+issues: []
diff --git a/docs/changelog/115147.yaml b/docs/changelog/115147.yaml
@@ -0,0 +1,5 @@
+pr: 115147
+summary: Fix IPinfo geolocation schema
+area: Ingest Node
+type: bug
+issues: []
diff --git a/docs/changelog/115194.yaml b/docs/changelog/115194.yaml
@@ -0,0 +1,7 @@
+pr: 115194
+summary: Update APM Java Agent to support JDK 23
+area: Infra/Metrics
+type: upgrade
+issues:
+ - 115101
+ - 115100
diff --git a/docs/plugins/mapper-annotated-text.asciidoc b/docs/plugins/mapper-annotated-text.asciidoc
@@ -155,11 +155,6 @@ be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 
-`annotated_text` fields support {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`] if they have
-a {ref}/keyword.html#keyword-synthetic-source[`keyword`] sub-field that supports synthetic
-`_source` or if the `annotated_text` field sets `store` to `true`. Either way, it may
-not have {ref}/copy-to.html[`copy_to`].
-
 If using a sub-`keyword` field then the values are sorted in the same way as
 a `keyword` field's values are sorted. By default, that means sorted with
 duplicates removed. So:

diff --git a/docs/reference/how-to/knn-search.asciidoc b/docs/reference/how-to/knn-search.asciidoc
@@ -59,8 +59,7 @@ since it relies on separate data structures to perform the search. Before
 using the <<include-exclude, `excludes`>> parameter, make sure to review the
 downsides of omitting fields from `_source`.
 
-Another option is to use  <<synthetic-source,synthetic `_source`>> if all
-your index fields support it.
+Another option is to use  <<synthetic-source,synthetic `_source`>>.
 
 [discrete]
 === Ensure data nodes have enough memory

diff --git a/docs/reference/inference/delete-inference.asciidoc b/docs/reference/inference/delete-inference.asciidoc
@@ -2,16 +2,11 @@
 [[delete-inference-api]]
 === Delete {infer} API
 
-experimental[]
-
 Deletes an {infer} endpoint.
 
-IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in
-{ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI or
-Hugging Face. For built-in models and models uploaded through Eland, the {infer}
-APIs offer an alternative way to use and manage trained models. However, if you
-do not plan to use the {infer} APIs to use these models or if you want to use
-non-NLP models, use the <<ml-df-trained-models-apis>>.
+IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in {ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face.
+For built-in models and models uploaded through Eland, the {infer} APIs offer an alternative way to use and manage trained models.
+However, if you do not plan to use the {infer} APIs to use these models or if you want to use non-NLP models, use the <<ml-df-trained-models-apis>>.
 
 
 [discrete]

diff --git a/docs/reference/inference/get-inference.asciidoc b/docs/reference/inference/get-inference.asciidoc
@@ -2,16 +2,11 @@
 [[get-inference-api]]
 === Get {infer} API
 
-experimental[]
-
 Retrieves {infer} endpoint information.
 
-IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in
-{ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI or
-Hugging Face. For built-in models and models uploaded through Eland, the {infer}
-APIs offer an alternative way to use and manage trained models. However, if you
-do not plan to use the {infer} APIs to use these models or if you want to use
-non-NLP models, use the <<ml-df-trained-models-apis>>.
+IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in {ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face.
+For built-in models and models uploaded through Eland, the {infer} APIs offer an alternative way to use and manage trained models.
+However, if you do not plan to use the {infer} APIs to use these models or if you want to use non-NLP models, use the <<ml-df-trained-models-apis>>.
 
 
 [discrete]

diff --git a/docs/reference/inference/images/inference-landscape.jpg b/docs/reference/inference/images/inference-landscape.jpg
diff --git a/docs/reference/inference/images/inference-landscape.png b/docs/reference/inference/images/inference-landscape.png
diff --git a/docs/reference/inference/inference-apis.asciidoc b/docs/reference/inference/inference-apis.asciidoc
@@ -2,8 +2,6 @@
 [[inference-apis]]
 == {infer-cap} APIs
 
-experimental[]
-
 IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in
 {ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure,
 Google AI Studio or Hugging Face. For built-in models and models uploaded
@@ -25,7 +23,7 @@ the following APIs to manage {infer} models and perform {infer}:
 
 [[inference-landscape]]
 .A representation of the Elastic inference landscape
-image::images/inference-landscape.png[A representation of the Elastic inference landscape,align="center"]
+image::images/inference-landscape.jpg[A representation of the Elastic inference landscape,align="center"]
 
 An {infer} endpoint enables you to use the corresponding {ml} model without
 manual deployment and apply it to your data at ingestion time through
@@ -54,3 +52,4 @@ include::service-google-vertex-ai.asciidoc[]
 include::service-hugging-face.asciidoc[]
 include::service-mistral.asciidoc[]
 include::service-openai.asciidoc[]
+include::service-watsonx-ai.asciidoc[]
diff --git a/docs/reference/inference/post-inference.asciidoc b/docs/reference/inference/post-inference.asciidoc
@@ -2,16 +2,11 @@
 [[post-inference-api]]
 === Perform inference API
 
-experimental[]
-
 Performs an inference task on an input text by using an {infer} endpoint.
 
-IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in
-{ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI or
-Hugging Face. For built-in models and models uploaded through Eland, the {infer}
-APIs offer an alternative way to use and manage trained models. However, if you
-do not plan to use the {infer} APIs to use these models or if you want to use
-non-NLP models, use the <<ml-df-trained-models-apis>>.
+IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in {ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face.
+For built-in models and models uploaded through Eland, the {infer} APIs offer an alternative way to use and manage trained models.
+However, if you do not plan to use the {infer} APIs to use these models or if you want to use non-NLP models, use the <<ml-df-trained-models-apis>>.
 
 
 [discrete]

diff --git a/docs/reference/inference/put-inference.asciidoc b/docs/reference/inference/put-inference.asciidoc
@@ -2,19 +2,12 @@
 [[put-inference-api]]
 === Create {infer} API
 
-experimental[]
-
 Creates an {infer} endpoint to perform an {infer} task.
 
 [IMPORTANT]
 ====
-* The {infer} APIs enable you to use certain services, such as built-in
-{ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Mistral,
-Azure OpenAI, Google AI Studio, Google Vertex AI, Anthropic or Hugging Face.
-* For built-in models and models uploaded through Eland, the {infer} APIs offer an
-alternative way to use and manage trained models. However, if you do not plan to
-use the {infer} APIs to use these models or if you want to use non-NLP models,
-use the <<ml-df-trained-models-apis>>.
+* The {infer} APIs enable you to use certain services, such as built-in {ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Mistral, Azure OpenAI, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face.
+* For built-in models and models uploaded through Eland, the {infer} APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the {infer} APIs to use these models or if you want to use non-NLP models, use the <<ml-df-trained-models-apis>>.
 ====
 
 
@@ -71,6 +64,7 @@ Click the links to review the configuration details of the services:
 * <<infer-service-hugging-face,Hugging Face>> (`text_embedding`)
 * <<infer-service-mistral,Mistral>> (`text_embedding`)
 * <<infer-service-openai,OpenAI>> (`completion`, `text_embedding`)
+* <<infer-service-watsonx-ai>> (`text_embedding`)
 
 The {es} and ELSER services run on a {ml} node in your {es} cluster. The rest of
 the services connect to external providers.
diff --git a/docs/reference/inference/service-watsonx-ai.asciidoc b/docs/reference/inference/service-watsonx-ai.asciidoc
@@ -0,0 +1,115 @@
+[[infer-service-watsonx-ai]]
+=== Watsonx {infer} service
+
+Creates an {infer} endpoint to perform an {infer} task with the `watsonxai` service.
+
+You need an https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-provisioning&interface=api[IBM Cloud® Databases for Elasticsearch deployment] to use the `watsonxai` {infer} service.
+You can provision one through the https://cloud.ibm.com/databases/databases-for-elasticsearch/create[IBM catalog], the https://cloud.ibm.com/docs/databases-cli-plugin?topic=databases-cli-plugin-cdb-reference[Cloud Databases CLI plug-in], the https://cloud.ibm.com/apidocs/cloud-databases-api[Cloud Databases API], or https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/database[Terraform].
+
+
+[discrete]
+[[infer-service-watsonx-ai-api-request]]
+==== {api-request-title}
+
+`PUT /_inference/<task_type>/<inference_id>`
+
+[discrete]
+[[infer-service-watsonx-ai-api-path-params]]
+==== {api-path-parms-title}
+
+`<inference_id>`::
+(Required, string)
+include::inference-shared.asciidoc[tag=inference-id]
+
+`<task_type>`::
+(Required, string)
+include::inference-shared.asciidoc[tag=task-type]
++
+--
+Available task types:
+
+* `text_embedding`.
+--
+
+[discrete]
+[[infer-service-watsonx-ai-api-request-body]]
+==== {api-request-body-title}
+
+`service`::
+(Required, string)
+The type of service supported for the specified task type. In this case, 
+`watsonxai`.
+
+`service_settings`::
+(Required, object)
+include::inference-shared.asciidoc[tag=service-settings]
++
+--
+These settings are specific to the `watsonxai` service.
+--
+
+`api_key`:::
+(Required, string)
+A valid API key of your Watsonx account.
+You can find your Watsonx API keys or you can create a new one https://cloud.ibm.com/iam/apikeys[on the API keys page].
++
+--
+include::inference-shared.asciidoc[tag=api-key-admonition]
+--
+
+`api_version`:::
+(Required, string)
+Version parameter that takes a version date in the format of `YYYY-MM-DD`.
+For the active version data parameters, refer to the https://cloud.ibm.com/apidocs/watsonx-ai#active-version-dates[documentation].
+
+`model_id`:::
+(Required, string)
+The name of the model to use for the {infer} task.
+Refer to the IBM Embedding Models section in the https://www.ibm.com/products/watsonx-ai/foundation-models[Watsonx documentation] for the list of available text embedding models.
+
+`url`:::
+(Required, string)
+The URL endpoint to use for the requests.
+
+`project_id`:::
+(Required, string)
+The name of the project to use for the {infer} task.
+
+`rate_limit`:::
+(Optional, object)
+By default, the `watsonxai` service sets the number of requests allowed per minute to `120`.
+This helps to minimize the number of rate limit errors returned from Watsonx.
+To modify this, set the `requests_per_minute` setting of this object in your service settings:
++
+--
+include::inference-shared.asciidoc[tag=request-per-minute-example]
+--
+
+
+[discrete]
+[[inference-example-watsonx-ai]]
+==== Watsonx AI service example
+
+The following example shows how to create an {infer} endpoint called `watsonx-embeddings` to perform a `text_embedding` task type.
+
+[source,console]
+------------------------------------------------------------
+PUT _inference/text_embedding/watsonx-embeddings
+{
+    "service": "watsonxai",
+    "service_settings": {
+        "api_key": "<api_key>", <1>
+        "url": "<url>", <2>
+        "model_id": "ibm/slate-30m-english-rtrvr",
+        "project_id": "<project_id>", <3>
+        "api_version": "2024-03-14" <4>
+    }
+}
+
+------------------------------------------------------------
+// TEST[skip:TBD]
+<1> A valid Watsonx API key.
+You can find on the https://cloud.ibm.com/iam/apikeys[API keys page of your account].
+<2> The {infer} endpoint URL you created on Watsonx.
+<3> The ID of your IBM Cloud project.
+<4> A valid API version parameter. You can find the active version data parameters https://cloud.ibm.com/apidocs/watsonx-ai#active-version-dates[here].
diff --git a/docs/reference/inference/update-inference.asciidoc b/docs/reference/inference/update-inference.asciidoc
@@ -2,11 +2,9 @@
 [[update-inference-api]]
 === Update inference API
 
-experimental[]
-
 Updates an {infer} endpoint.
 
-IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in {ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI or Hugging Face.
+IMPORTANT: The {infer} APIs enable you to use certain services, such as built-in {ml} models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face.
 For built-in models and models uploaded through Eland, the {infer} APIs offer an alternative way to use and manage trained models.
 However, if you do not plan to use the {infer} APIs to use these models or if you want to use non-NLP models, use the <<ml-df-trained-models-apis>>.
 

diff --git a/docs/reference/mapping/types/aggregate-metric-double.asciidoc b/docs/reference/mapping/types/aggregate-metric-double.asciidoc
@@ -259,9 +259,6 @@ be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 
-`aggregate_metric-double` fields support <<synthetic-source,synthetic `_source`>> in their default
-configuration.
-
 For example:
 [source,console,id=synthetic-source-aggregate-metric-double-example]
 ----

diff --git a/docs/reference/mapping/types/binary.asciidoc b/docs/reference/mapping/types/binary.asciidoc
@@ -63,7 +63,7 @@ be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 
-`binary` fields support <<synthetic-source,synthetic `_source`>> only when <<doc-values,`doc_values`>> are enabled. Synthetic source always sorts `binary` values in order of their byte representation. For example:
+Synthetic source may sort `binary` values in order of their byte representation. For example:
 [source,console,id=synthetic-source-binary-example]
 ----
 PUT idx

diff --git a/docs/reference/mapping/types/boolean.asciidoc b/docs/reference/mapping/types/boolean.asciidoc
@@ -241,10 +241,9 @@ any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 
 `boolean` fields support <<synthetic-source,synthetic `_source`>> in their
-default configuration. Synthetic `_source` cannot be used together with
-<<copy-to,`copy_to`>> or with <<doc-values,`doc_values`>> disabled.
+default configuration.
 
-Synthetic source always sorts `boolean` fields. For example:
+Synthetic source may sort `boolean` field values. For example:
 [source,console,id=synthetic-source-boolean-example]
 ----
 PUT idx

diff --git a/docs/reference/mapping/types/date.asciidoc b/docs/reference/mapping/types/date.asciidoc
@@ -247,11 +247,7 @@ be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 
-`date` fields support <<synthetic-source,synthetic `_source`>> in their
-default configuration. Synthetic `_source` cannot be used together with
-<<copy-to,`copy_to`>> or with <<doc-values,`doc_values`>> disabled.
-
-Synthetic source always sorts `date` fields. For example:
+Synthetic source may sort `date` field values. For example:
 [source,console,id=synthetic-source-date-example]
 ----
 PUT idx

diff --git a/docs/reference/mapping/types/date_nanos.asciidoc b/docs/reference/mapping/types/date_nanos.asciidoc
@@ -150,12 +150,7 @@ be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 
-`date_nanos` fields support <<synthetic-source,synthetic `_source`>> in their
-default configuration. Synthetic `_source` cannot be used together with
-<<copy-to,`copy_to`>>, <<ignore-malformed,`ignore_malformed`>> set to true
-or with <<doc-values,`doc_values`>> disabled.
-
-Synthetic source always sorts `date_nanos` fields. For example:
+Synthetic source may sort `date_nanos` field values. For example:
 [source,console,id=synthetic-source-date-nanos-example]
 ----
 PUT idx

diff --git a/docs/reference/mapping/types/flattened.asciidoc b/docs/reference/mapping/types/flattened.asciidoc
@@ -325,10 +325,9 @@ any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 
 Flattened fields support <<synthetic-source,synthetic`_source`>> in their default
-configuration. Synthetic `_source` cannot be used with <<doc-values,`doc_values`>>
-disabled.
+configuration.
 
-Synthetic source always sorts alphabetically and de-duplicates flattened fields.
+Synthetic source may sort `flattened` field values and remove duplicates.
 For example:
 [source,console,id=synthetic-source-flattened-sorting-example]
 ----

diff --git a/docs/reference/mapping/types/geo-point.asciidoc b/docs/reference/mapping/types/geo-point.asciidoc
@@ -219,11 +219,7 @@ be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 
-`geo_point` fields support <<synthetic-source,synthetic `_source`>> in their
-default configuration. Synthetic `_source` cannot be used together with <<copy-to,`copy_to`>> or with
-<<doc-values,`doc_values`>> disabled.
-
-Synthetic source always sorts `geo_point` fields (first by latitude and then
+Synthetic source may sort `geo_point` fields (first by latitude and then
 longitude) and reduces them to their stored precision. For example:
 [source,console,id=synthetic-source-geo-point-example]
 ----

diff --git a/docs/reference/mapping/types/geo-shape.asciidoc b/docs/reference/mapping/types/geo-shape.asciidoc
@@ -502,6 +502,3 @@ synthetic `_source` is in technical preview. Features in technical preview may
 be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
-
-`geo_shape` fields support <<synthetic-source,synthetic `_source`>> in their
-default configuration.