Merge branch 'main' into commit_data_counts_on_close

.ci/jobs.t/elastic+elasticsearch+periodic+packaging-tests.yml

@@ -2,7 +2,8 @@
 - job:
  name: elastic+elasticsearch+%BRANCH%+periodic+packaging-tests
  display-name: "elastic / elasticsearch # %BRANCH% - packaging tests"
- description: "Testing of the Elasticsearch %BRANCH% branch packaging tests.\n"
+ description: "This job has been migrated to Buildkite.\n"
+ disabled: true
  project-type: multijob
  node: master
  vault: []

docs/changelog/101120.yaml

@@ -0,0 +1,6 @@
+pr: 101120
+summary: "ESQL: Fix escaping of backslash in LIKE operator"
+area: ES|QL
+type: bug
+issues:
+ - 101106

docs/reference/esql/esql-enrich-data.asciidoc

@@ -0,0 +1,126 @@
+[[esql-enrich-data]]
+=== Enrich data
+
+++++
+<titleabbrev>Enrich data</titleabbrev>
+++++
+
+You can use {esql}'s <<esql-enrich>> processing command to enrich a table with
+data from indices in {es}.
+
+For example, you can use `ENRICH` to:
+
+* Identify web services or vendors based on known IP addresses
+* Add product information to retail orders based on product IDs
+* Supplement contact information based on an email address
+
+[[esql-how-enrich-works]]
+==== How the `ENRICH` command works
+
+The `ENRICH` command adds new columns to a table, with data from {es} indices.
+It requires a few special components:
+
+image::images/esql/esql-enrich.png[align="center"]
+
+[[esql-enrich-policy]]
+Enrich policy::
++
+--
+A set of configuration options used to add the right enrich data to the input
+table.
+
+An enrich policy contains:
+
+include::../ingest/enrich.asciidoc[tag=enrich-policy-fields]
+
+After <<esql-create-enrich-policy,creating a policy>>, it must be
+<<esql-execute-enrich-policy,executed>> before it can be used. Executing an
+enrich policy uses data from the policy's source indices to create a streamlined
+system index called the _enrich index_. The `ENRICH` command uses this index to
+match and enrich an input table.
+--
+
+[[esql-source-index]]
+Source index::
+An index which stores enrich data that the `ENRICH` command can add to input
+tables. You can create and manage these indices just like a regular {es} index.
+You can use multiple source indices in an enrich policy. You also can use the
+same source index in multiple enrich policies.
+
+[[esql-enrich-index]]
+Enrich index::
++
+--
+A special system index tied to a specific enrich policy.
+
+Directly matching rows from input tables to documents in source indices could be
+slow and resource intensive. To speed things up, the `ENRICH` command uses an
+enrich index.
+
+include::../ingest/enrich.asciidoc[tag=enrich-index]
+--
+
+[[esql-set-up-enrich-policy]]
+==== Set up an enrich policy
+
+To start using `ENRICH`, follow these steps:
+
+. Check the <<enrich-prereqs, prerequisites>>.
+. <<esql-create-enrich-source-index>>.
+. <<esql-create-enrich-policy>>.
+. <<esql-execute-enrich-policy>>.
+. <<esql-use-enrich>>
+
+Once you have enrich policies set up, you can <<esql-update-enrich-data,update
+your enrich data>> and <<esql-update-enrich-policies, update your enrich
+policies>>.
+
+[IMPORTANT]
+====
+The `ENRICH` command performs several operations and may impact the speed of
+your query.
+====
+
+[[esql-enrich-prereqs]]
+===== Prerequisites
+
+include::{es-repo-dir}/ingest/apis/enrich/put-enrich-policy.asciidoc[tag=enrich-policy-api-prereqs]
+
+[[esql-create-enrich-source-index]]
+===== Add enrich data
+
+include::../ingest/enrich.asciidoc[tag=create-enrich-source-index]
+
+[[esql-create-enrich-policy]]
+===== Create an enrich policy
+
+include::../ingest/enrich.asciidoc[tag=create-enrich-policy]
+
+[[esql-execute-enrich-policy]]
+===== Execute the enrich policy
+
+include::../ingest/enrich.asciidoc[tag=execute-enrich-policy1]
+
+image::images/esql/esql-enrich-policy.png[align="center"]
+
+include::../ingest/enrich.asciidoc[tag=execute-enrich-policy2]
+
+[[esql-use-enrich]]
+==== Use the enrich policy
+
+After the policy has been executed, you can use the <<esql-enrich,`ENRICH`
+command>> to enrich your data.
+
+image::images/esql/esql-enrich-command.png[align="center",width=50%]
+
+include::processing-commands/enrich.asciidoc[tag=examples]
+
+[[esql-update-enrich-data]]
+==== Update an enrich index
+
+include::{es-repo-dir}/ingest/apis/enrich/execute-enrich-policy.asciidoc[tag=update-enrich-index]
+
+[[esql-update-enrich-policies]]
+==== Update an enrich policy
+
+include::../ingest/enrich.asciidoc[tag=update-enrich-policy]

docs/reference/esql/esql-language.asciidoc

@@ -10,7 +10,9 @@ Detailed information about the {esql} language:
 * <<esql-syntax>>
 * <<esql-multivalued-fields>>
 * <<esql-metadata-fields>>
+* <<esql-enrich-data>>
 
 include::esql-syntax.asciidoc[]
 include::multivalued-fields.asciidoc[]
-include::metadata-fields.asciidoc[]
+include::metadata-fields.asciidoc[]
+include::esql-enrich-data.asciidoc[]

docs/reference/esql/processing-commands/enrich.asciidoc

@@ -1,8 +1,55 @@
 [discrete]
 [[esql-enrich]]
 === `ENRICH`
-You can use `ENRICH` to add data from your existing indices to incoming records.
-It's similar to <<ingest-enriching-data, ingest enrich>>, but it works at query time.
+
+**Syntax**
+
+[source,txt]
+----
+ENRICH policy [ON match_field] [WITH [new_name1 = ]field1, [new_name2 = ]field2, ...]
+----
+
+*Parameters*
+
+`policy`::
+The name of the enrich policy. You need to <<esql-set-up-enrich-policy,create>>
+and <<esql-execute-enrich-policy,execute>> the enrich policy first.
+
+`ON match_field`::
+The match field. `ENRICH` uses its value to look for records in the enrich
+index. If not specified, the match will be performed on the column with the same
+name as the `match_field` defined in the <<esql-enrich-policy,enrich policy>>.
+
+`WITH fieldX`::
+The enrich fields from the enrich index that are added to the result as new
+columns. If a column with the same name as the enrich field already exists, the
+existing column will be replaced by the new column. If not specified, each of
+the enrich fields defined in the policy is added
+
+`new_nameX =`::
+Enables you to change the name of the column that's added for each of the enrich
+fields. Defaults to the enrich field name.
+
+*Description*
+
+`ENRICH` enables you to add data from existing indices as new columns using an
+enrich policy. Refer to <<esql-enrich-data>> for information about setting up a
+policy.
+
+image::images/esql/esql-enrich.png[align="center"]
+
+TIP: Before you can use `ENRICH`, you need to <<esql-set-up-enrich-policy,create
+and execute an enrich policy>>.
+
+*Examples*
+
+// tag::examples[]
+The following example uses the `languages_policy` enrich policy to add a new
+column for each enrich field defined in the policy. The match is performed using
+the `match_field` defined in the <<esql-enrich-policy,enrich policy>> and
+requires that the input table has a column with the same name (`language_code`
+in this example). `ENRICH` will look for records in the
+<<esql-enrich-index,enrich index>> based on the match field value. 
 
 [source.merge.styled,esql]
 ----
@@ -13,12 +60,8 @@ include::{esql-specs}/docs-ignoreCsvTests.csv-spec[tag=enrich]
 include::{esql-specs}/docs-ignoreCsvTests.csv-spec[tag=enrich-result]
 |===
 
-`ENRICH` requires an <<enrich-policy,enrich policy>> to be executed.
-The enrich policy defines a match field (a key field) and a set of enrich fields.
-
-`ENRICH` will look for records in the <<enrich-index,enrich index>> based on the match field value.
-The matching key in the input dataset can be defined using `ON <field-name>`; if it's not specified,
-the match will be performed on a field with the same name as the match field defined in the <<enrich-policy,enrich policy>>.
+To use a column with a different name than the `match_field` defined in the
+policy as the match field, use `ON <column-name>`:
 
 [source.merge.styled,esql]
 ----
@@ -29,9 +72,9 @@ include::{esql-specs}/docs-ignoreCsvTests.csv-spec[tag=enrich_on]
 include::{esql-specs}/docs-ignoreCsvTests.csv-spec[tag=enrich_on-result]
 |===
 
-
-You can specify which attributes (between those defined as enrich fields in the policy) have to be added to the result,
-using `WITH <field1>, <field2>...` syntax.
+By default, each of the enrich fields defined in the policy is added as a
+column. To explicitly select the enrich fields that are added, use 
+`WITH <field1>, <field2>...`:
 
 [source.merge.styled,esql]
 ----
@@ -42,8 +85,7 @@ include::{esql-specs}/docs-ignoreCsvTests.csv-spec[tag=enrich_with]
 include::{esql-specs}/docs-ignoreCsvTests.csv-spec[tag=enrich_with-result]
 |===
 
-
-Attributes can also be renamed using `WITH new_name=<field1>`
+You can rename the columns that are added using `WITH new_name=<field1>`:
 
 [source.merge.styled,esql]
 ----
@@ -54,8 +96,6 @@ include::{esql-specs}/docs-ignoreCsvTests.csv-spec[tag=enrich_rename]
 include::{esql-specs}/docs-ignoreCsvTests.csv-spec[tag=enrich_rename-result]
 |===
 
-
-By default (if no `WITH` is defined), `ENRICH` will add all the enrich fields defined in the <<enrich-policy,enrich policy>>
-to the result.
-
-In case of name collisions, the newly created fields will override the existing fields.
+In case of name collisions, the newly created columns will override existing
+columns.
+// end::examples[]

docs/reference/ingest/apis/enrich/execute-enrich-policy.asciidoc

@@ -77,8 +77,9 @@ and are <<indices-forcemerge,force merged>>.
 
 [WARNING]
 ====
-Enrich indices should be used by the <<enrich-processor,enrich processor>> only.
-Avoid using enrich indices for other purposes.
+Enrich indices should only be used by the <<enrich-processor,enrich processor>>
+or the <<esql-enrich,{esql} `ENRICH` command>>. Avoid using enrich indices for
+other purposes.
 ====
 // end::execute-enrich-policy-def[]
 

docs/reference/ingest/apis/enrich/put-enrich-policy.asciidoc

@@ -53,7 +53,7 @@ DELETE /_enrich/policy/my-policy
 ==== {api-prereq-title}
 
 // tag::enrich-policy-api-prereqs[]
-If you use {es} {security-features}, you must have:
+To use enrich policies, you must have:
 
 * `read` index privileges for any indices used
 * The `enrich_user` <<built-in-roles,built-in role>>

docs/reference/ingest/enrich.asciidoc

@@ -67,14 +67,16 @@ Directly matching incoming documents to documents in source indices could be
 slow and resource intensive. To speed things up, the enrich processor uses an
 enrich index.
 
+// tag::enrich-index[]
 Enrich indices contain enrich data from source indices but have a few special
 properties to help streamline them:
 
 * They are system indices, meaning they're managed internally by {es} and only
- intended for use with enrich processors.
+ intended for use with enrich processors and the {esql} `ENRICH` command.
 * They always begin with `.enrich-*`.
 * They are read-only, meaning you can't directly change them.
 * They are <<indices-forcemerge,force merged>> for fast retrieval.
+// end::enrich-index[]
 --
 
 [[enrich-setup]]
@@ -115,8 +117,9 @@ include::{es-repo-dir}/ingest/apis/enrich/put-enrich-policy.asciidoc[tag=enrich-
 [[create-enrich-source-index]]
 ==== Add enrich data
 
+// tag::create-enrich-source-index[]
 To begin, add documents to one or more source indices. These documents should
-contain the enrich data you eventually want to add to incoming documents.
+contain the enrich data you eventually want to add to incoming data.
 
 You can manage source indices just like regular {es} indices using the
 <<docs,document>> and <<indices,index>> APIs.
@@ -125,29 +128,38 @@ You also can set up {beats-ref}/getting-started.html[{beats}], such as a
 {filebeat-ref}/filebeat-installation-configuration.html[{filebeat}], to
 automatically send and index documents to your source indices. See
 {beats-ref}/getting-started.html[Getting started with {beats}].
+//end::create-enrich-source-index[]
 
 [[create-enrich-policy]]
 ==== Create an enrich policy
 
+// tag::create-enrich-policy[]
 After adding enrich data to your source indices, use the
-<<put-enrich-policy-api,create enrich policy API>> to create an enrich policy.
+<<put-enrich-policy-api,create enrich policy API>> or
+<<manage-enrich-policies,Index Management in {kib}>> to create an enrich policy.
 
 [WARNING]
 ====
 Once created, you can't update or change an enrich policy.
 See <<update-enrich-policies>>.
 ====
+// end::create-enrich-policy[]
 
 [[execute-enrich-policy]]
 ==== Execute the enrich policy
 
-Once the enrich policy is created, you can execute it using the
-<<execute-enrich-policy-api,execute enrich policy API>> to create an
+// tag::execute-enrich-policy1[]
+Once the enrich policy is created, you need to execute it using the
+<<execute-enrich-policy-api,execute enrich policy API>> or
+<<manage-enrich-policies,Index Management in {kib}>> to create an
 <<enrich-index,enrich index>>.
+// end::execute-enrich-policy1[]
 
 image::images/ingest/enrich/enrich-policy-index.svg[align="center"]
 
+// tag::execute-enrich-policy2[]
 include::apis/enrich/execute-enrich-policy.asciidoc[tag=execute-enrich-policy-def]
+// end::execute-enrich-policy2[]
 
 [[add-enrich-processor]]
 ==== Add an enrich processor to an ingest pipeline
@@ -208,9 +220,10 @@ Instead, you can:
 
 . Replace the previous enrich policy
  with the new enrich policy
- in any in-use enrich processors.
+ in any in-use enrich processors or {esql} queries.
 
-. Use the <<delete-enrich-policy-api, delete enrich policy>> API
+. Use the <<delete-enrich-policy-api, delete enrich policy>> API or
+<<manage-enrich-policies,Index Management in {kib}>>
  to delete the previous enrich policy.
 // end::update-enrich-policy[]
 

docs/reference/redirects.asciidoc

@@ -1952,4 +1952,3 @@ coming::[8.11.0]
 === Tutorial: Natural language processing (NLP)
 
 coming::[8.11.0]
-