Merge remote-tracking branch 'origin/main' into add-dataset-size-to-cat

dakrone · Aug 21, 2023 · e5ffcf6 · e5ffcf6
2 parents 78a2f2c + d7bcad0
commit e5ffcf6
Show file tree

Hide file tree

Showing 182 changed files with 1,927 additions and 892 deletions.
diff --git a/docs/build.gradle b/docs/build.gradle
@@ -100,7 +100,6 @@ testClusters.matching { it.name == "yamlRestTest"}.configureEach {
   systemProperty 'es.transport.cname_in_publish_address', 'true'
 
   requiresFeature 'es.index_mode_feature_flag_registered', Version.fromString("8.0.0")
-  requiresFeature 'es.dlm_feature_flag_enabled', Version.fromString("8.8.0")
 
   // build the cluster with all plugins
   project.rootProject.subprojects.findAll { it.parent.path == ':plugins' }.each { subproj ->
@@ -122,9 +121,6 @@ tasks.named("yamlRestTest").configure {
   doFirst {
     delete("${buildDir}/cluster/shared/repo")
   }
-  if (BuildParams.isSnapshotBuild() == false) {
-    systemProperty 'es.dlm_feature_flag_enabled', 'true'
-  }
 }
 
 tasks.named("forbiddenPatterns").configure {

diff --git a/docs/changelog/97319.yaml b/docs/changelog/97319.yaml
@@ -2,7 +2,19 @@ pr: 97319
 summary: Change pre-configured and cached analyzer components to use IndexVersion instead of Version
 area: "Analysis"
 type: breaking-java
-breaking: "Java API"
+breaking:
+  title: Change pre-configured and cached analyzer components to use IndexVersion instead of Version
+  area: Java API
+  details: |-
+    This PR changes the types used to obtain pre-configured components from Version to IndexVersion,
+    with corresponding changes to method names.
+
+    Prior to 8.10, there is a one-to-one mapping between node version and index version, with corresponding constants
+    in the IndexVersion class.
+    Starting in 8.10, IndexVersion is versioned independently of node version, and will be a simple incrementing number.
+    For more information on how to use IndexVersion and other version types, please see the contributing guide.
+  impact: Analysis components now take IndexVersion instead of Version
+  notable: false
 issues: []
 highlight:
   title: Change pre-configured and cached analyzer components to use IndexVersion instead of Version

diff --git a/docs/changelog/98635.yaml b/docs/changelog/98635.yaml
@@ -0,0 +1,5 @@
+pr: 98635
+summary: Fix NPE in `StableMasterHealthIndicatorService`
+area: Health
+type: bug
+issues: []
diff --git a/docs/changelog/98644.yaml b/docs/changelog/98644.yaml
@@ -0,0 +1,20 @@
+pr: 98644
+summary: GA the data stream lifecycle
+area: Data streams
+type: feature
+issues: []
+highlight:
+  title: The data stream lifecycle is now generally available
+  body: |-
+    This makes the data stream lifecycle generaly available. Data streams will 
+    be able to take advantage of a built-in simplified and resilient
+    lifecycle implementation. Data streams with a configured lifecycle will
+    be automatically rolled over and tail merged (a forcemerge implementation 
+    that's lightweight and only merges the long tail of small segments instead 
+    of the whole shard). With the shard and index maintenance tasks being 
+    handled automatically to ensure optimum performance, and trade-off between
+    indexing and searching, you'll be able to focus on the business related 
+    lifecycle aspects like data retention. The automatic maintenance tasks will
+    automatically be available for any new data stream that's not managed by 
+    ILM.
+  notable: true
diff --git a/docs/reference/data-management.asciidoc b/docs/reference/data-management.asciidoc
@@ -20,20 +20,12 @@ so you can move it to less expensive, less performant hardware.
 For your oldest data, what matters is that you have access to the data. 
 It's ok if queries take longer to complete.
 
-ifeval::["{release-state}"=="unreleased"]
 To help you manage your data, {es} offers you:
 
 * <<index-lifecycle-management, {ilm-cap}>> ({ilm-init}) to manage both indices and data streams and it is fully customisable, and
 * <<data-stream-lifecycle, Data stream lifecycle>> which is the built-in lifecycle of data streams and addresses the most
 common lifecycle management needs.
 
-preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will apply best effort to fix any issues, but this feature is not subject to the support SLA of official GA features."]
-endif::[]
-ifeval::["{release-state}"=="released"]
-To help you manage your data, {es} offers you <<index-lifecycle-management, {ilm-cap}>> ({ilm-init}) to manage both indices and data
-streams and it is fully customisable.
-endif::[]
-
 **{ilm-init}** can be used to manage both indices and data streams and it allows you to:
 
 * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
@@ -44,14 +36,12 @@ Data older than this period can be deleted by {es}.
 for your older indices while reducing operating costs and maintaining search performance.  
 * Perform <<async-search-intro, asynchronous searches>> of data stored on less-performant hardware.
 
-ifeval::["{release-state}"=="unreleased"]
 **Data stream lifecycle** is less feature rich but is focused on simplicity, so it allows you to easily:
 
 * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
 Data older than this period can be deleted by {es} at a later time.
 * Improve the performance of your data stream by performing background operations that will optimise the way your data
 stream is stored.
-endif::[]
 --
 
 include::ilm/index.asciidoc[]

diff --git a/docs/reference/data-streams/data-stream-apis.asciidoc b/docs/reference/data-streams/data-stream-apis.asciidoc
@@ -12,15 +12,13 @@ The following APIs are available for managing <<data-streams,data streams>>:
 * <<promote-data-stream-api>>
 * <<modify-data-streams-api>>
 
-ifeval::["{release-state}"=="unreleased"]
 [[data-stream-lifecycle-api]]
 The following APIs are available for managing the built-in lifecycle of data streams:
 
-* <<data-streams-put-lifecycle,Update data stream lifecycle>> preview:[]
-* <<data-streams-get-lifecycle,Get data stream lifecycle>> preview:[]
-* <<data-streams-delete-lifecycle,Delete data stream lifecycle>> preview:[]
-* <<data-streams-explain-lifecycle,Explain data stream lifecycle>> preview:[]
-endif::[]
+* <<data-streams-put-lifecycle,Update data stream lifecycle>> 
+* <<data-streams-get-lifecycle,Get data stream lifecycle>> 
+* <<data-streams-delete-lifecycle,Delete data stream lifecycle>>
+* <<data-streams-explain-lifecycle,Explain data stream lifecycle>>
 
 The following API is available for <<tsds,time series data streams>>:
 
@@ -43,14 +41,12 @@ include::{es-repo-dir}/data-streams/promote-data-stream-api.asciidoc[]
 
 include::{es-repo-dir}/data-streams/modify-data-streams-api.asciidoc[]
 
-ifeval::["{release-state}"=="unreleased"]
 include::{es-repo-dir}/data-streams/lifecycle/apis/put-lifecycle.asciidoc[]
 
 include::{es-repo-dir}/data-streams/lifecycle/apis/get-lifecycle.asciidoc[]
 
 include::{es-repo-dir}/data-streams/lifecycle/apis/delete-lifecycle.asciidoc[]
 
 include::{es-repo-dir}/data-streams/lifecycle/apis/explain-lifecycle.asciidoc[]
-endif::[]
 
 include::{es-repo-dir}/indices/downsample-data-stream.asciidoc[]
diff --git a/docs/reference/data-streams/lifecycle/apis/delete-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/delete-lifecycle.asciidoc
@@ -4,8 +4,6 @@
 <titleabbrev>Delete Data Stream Lifecycle</titleabbrev>
 ++++
 
-preview::[]
-
 Deletes the lifecycle from a set of data streams.
 
 [[delete-lifecycle-api-prereqs]]

diff --git a/docs/reference/data-streams/lifecycle/apis/explain-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/explain-lifecycle.asciidoc
@@ -4,8 +4,6 @@
 <titleabbrev>Explain Data Stream Lifecycle</titleabbrev>
 ++++
 
-preview::[]
-
 Retrieves the current data stream lifecycle status for one or more data stream backing indices.
 
 [[explain-lifecycle-api-prereqs]]

diff --git a/docs/reference/data-streams/lifecycle/apis/get-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/get-lifecycle.asciidoc
@@ -4,8 +4,6 @@
 <titleabbrev>Get Data Stream Lifecycle</titleabbrev>
 ++++
 
-preview::[]
-
 Gets the lifecycle of a set of data streams.
 
 [[get-lifecycle-api-prereqs]]

diff --git a/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc
@@ -4,8 +4,6 @@
 <titleabbrev>Put Data Stream Lifecycle</titleabbrev>
 ++++
 
-preview::[]
-
 Configures the data stream lifecycle for the targeted data streams.
 
 [[put-lifecycle-api-prereqs]]

diff --git a/docs/reference/data-streams/lifecycle/index.asciidoc b/docs/reference/data-streams/lifecycle/index.asciidoc
@@ -1,10 +1,7 @@
 [role="xpack"]
 [[data-stream-lifecycle]]
-ifeval::["{release-state}"=="unreleased"]
 == Data stream lifecycle
 
-preview::[]
-
 A data stream lifecycle is the built-in mechanism data streams use to manage their lifecycle. It enables you to easily
 automate the management of your data streams according to your retention requirements. For example, you could configure
 the lifecycle to:
@@ -63,4 +60,3 @@ because it is applied on the data stream level and not on the individual backing
 include::tutorial-manage-new-data-stream.asciidoc[]
 
 include::tutorial-manage-existing-data-stream.asciidoc[]
-endif::[]
diff --git a/.../reference/data-streams/lifecycle/tutorial-manage-existing-data-stream.asciidoc b/.../reference/data-streams/lifecycle/tutorial-manage-existing-data-stream.asciidoc
@@ -2,8 +2,6 @@
 [[tutorial-manage-existing-data-stream]]
 === Tutorial: Update existing data stream
 
-preview::[]
-
 To update the lifecycle of an existing data stream you do the following actions:
 
 . <<set-lifecycle>>

diff --git a/docs/reference/data-streams/lifecycle/tutorial-manage-new-data-stream.asciidoc b/docs/reference/data-streams/lifecycle/tutorial-manage-new-data-stream.asciidoc
@@ -2,8 +2,6 @@
 [[tutorial-manage-new-data-stream]]
 === Tutorial: Create a data stream with a lifecycle
 
-preview::[]
-
 To create a data stream with a built-in lifecycle, follow these steps:
 
 . <<create-index-template-with-lifecycle>>

diff --git a/docs/reference/esql/esql-functions.asciidoc b/docs/reference/esql/esql-functions.asciidoc
@@ -16,6 +16,7 @@ these functions:
 * <<esql-auto_bucket>>
 * <<esql-case>>
 * <<esql-cidr_match>>
+* <<esql-coalesce>>
 * <<esql-concat>>
 * <<esql-cos>>
 * <<esql-cosh>>
@@ -73,6 +74,7 @@ include::functions/atan2.asciidoc[]
 include::functions/auto_bucket.asciidoc[]
 include::functions/case.asciidoc[]
 include::functions/cidr_match.asciidoc[]
+include::functions/coalesce.asciidoc[]
 include::functions/concat.asciidoc[]
 include::functions/cos.asciidoc[]
 include::functions/cosh.asciidoc[]

diff --git a/docs/reference/esql/esql-syntax.asciidoc b/docs/reference/esql/esql-syntax.asciidoc
@@ -120,15 +120,23 @@ The following boolean operators are supported:
 
 For NULL comparison use the `IS NULL` and `IS NOT NULL` predicates:
 
-[source,esql]
+[source.merge.styled,esql]
 ----
-include::{esql-specs}/conditional.csv-spec[tag=is-null]
+include::{esql-specs}/null.csv-spec[tag=is-null]
 ----
+[%header.monospaced.styled,format=dsv,separator=|]
+|===
+include::{esql-specs}/null.csv-spec[tag=is-null-result]
+|===
 
-[source,esql]
+[source.merge.styled,esql]
 ----
-include::{esql-specs}/conditional.csv-spec[tag=is-not-null]
+include::{esql-specs}/null.csv-spec[tag=is-not-null]
 ----
+[%header.monospaced.styled,format=dsv,separator=|]
+|===
+include::{esql-specs}/null.csv-spec[tag=is-not-null-result]
+|===
 
 [discrete]
 [[esql-timespan-literals]]

diff --git a/docs/reference/esql/functions/coalesce.asciidoc b/docs/reference/esql/functions/coalesce.asciidoc
@@ -0,0 +1,13 @@
+[[esql-coalesce]]
+=== `COALESCE`
+
+Returns the first non-null value.
+
+[source.merge.styled,esql]
+----
+include::{esql-specs}/null.csv-spec[tag=coalesce]
+----
+[%header.monospaced.styled,format=dsv,separator=|]
+|===
+include::{esql-specs}/null.csv-spec[tag=coalesce-result]
+|===
diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc
@@ -1885,47 +1885,23 @@ Refer to <<fix-watermark-errors>>.
 [role="exclude",id="dlm-delete-lifecycle"]
 === Delete the lifecycle of a data stream
 
-ifeval::["{release-state}"=="unreleased"]
 Refer to <<data-streams-delete-lifecycle,Delete data stream lifecycle API>>.
 
-endif::[]
-ifeval::["{release-state}"=="unreleased"]
-This feature has not been released yet
-endif::[]
-
 [role="exclude",id="dlm-explain-lifecycle"]
 === Explain the lifecycle of a data stream
 
-ifeval::["{release-state}"=="unreleased"]
 Refer to <<data-streams-explain-lifecycle,Explain data stream lifecycle API>>.
 
-endif::[]
-ifeval::["{release-state}"=="unreleased"]
-This feature has not been released yet
-endif::[]
-
 [role="exclude",id="dlm-get-lifecycle"]
 === Get the lifecycle of a data stream
 
-ifeval::["{release-state}"=="unreleased"]
 Refer to <<data-streams-get-lifecycle,Get data stream lifecycle API>>.
 
-endif::[]
-ifeval::["{release-state}"=="unreleased"]
-This feature has not been released yet
-endif::[]
-
 [role="exclude",id="dlm-put-lifecycle"]
 === Update the lifecycle of a data stream
 
-ifeval::["{release-state}"=="unreleased"]
 Refer to <<data-streams-delete-lifecycle,Update data stream lifecycle API>>.
 
-endif::[]
-ifeval::["{release-state}"=="unreleased"]
-This feature has not been released yet
-endif::[]
-
 [role="exclude",id="get-synonyms"]
 === Get synonyms set API
 

diff --git a/docs/reference/release-notes.asciidoc b/docs/reference/release-notes.asciidoc
@@ -8,6 +8,7 @@ This section summarizes the changes in each release.
 
 * <<release-notes-8.11.0>>
 * <<release-notes-8.10.0>>
+* <<release-notes-8.9.1>>
 * <<release-notes-8.9.0>>
 * <<release-notes-8.8.2>>
 * <<release-notes-8.8.1>>
@@ -49,6 +50,7 @@ This section summarizes the changes in each release.
 
 include::release-notes/8.11.0.asciidoc[]
 include::release-notes/8.10.0.asciidoc[]
+include::release-notes/8.9.1.asciidoc[]
 include::release-notes/8.9.0.asciidoc[]
 include::release-notes/8.8.2.asciidoc[]
 include::release-notes/8.8.1.asciidoc[]

diff --git a/docs/reference/release-notes/8.8.2.asciidoc b/docs/reference/release-notes/8.8.2.asciidoc
@@ -1,8 +1,6 @@
 [[release-notes-8.8.2]]
 == {es} version 8.8.2
 
-coming[8.8.2]
-
 Also see <<breaking-changes-8.8,Breaking changes in 8.8>>.
 
 [[bug-8.8.2]]

diff --git a/docs/reference/release-notes/8.9.1.asciidoc b/docs/reference/release-notes/8.9.1.asciidoc
@@ -0,0 +1,51 @@
+[[release-notes-8.9.1]]
+== {es} version 8.9.1
+
+Also see <<breaking-changes-8.9,Breaking changes in 8.9>>.
+
+[[bug-8.9.1]]
+[float]
+=== Bug fixes
+
+Aggregations::
+* `GlobalAggregator` should call rewrite() before `createWeight()` {es-pull}98091[#98091] (issue: {es-issue}98076[#98076])
+
+Cluster Coordination::
+* Improve exception handling in Coordinator#publish {es-pull}97840[#97840] (issue: {es-issue}97798[#97798])
+
+EQL::
+* Backport fix for async missing events and re-enable the feature {es-pull}98130[#98130]
+
+ILM+SLM::
+* Ignore the `total_shards_per_node` setting on searchable snapshots in frozen {es-pull}97979[#97979]
+* Migrate to data tiers routing configures correct default for mounted indices {es-pull}97936[#97936] (issue: {es-issue}97898[#97898])
+
+Infra/Core::
+* Fix APM trace start time {es-pull}98113[#98113]
+
+Infra/Logging::
+* Add Configuration to `PatternLayout` {es-pull}97679[#97679]
+
+Machine Learning::
+* Fix failure processing Question Answering model output where the input has been spanned over multiple sequences {es-pull}98167[#98167] (issue: {es-issue}97917[#97917])
+
+Search::
+* `UnmappedFieldFetcher` should ignore nested fields {es-pull}97987[#97987] (issue: {es-issue}97684[#97684])
+
+[[enhancement-8.9.1]]
+[float]
+=== Enhancements
+
+Authentication::
+* Upgrade xmlsec to 2.1.8 {es-pull}97741[#97741]
+
+Infra/Core::
+* Enhance regex performance with duplicate wildcards {es-pull}98176[#98176]
+
+Machine Learning::
+* Add setting to scale the processor count used in the model assignment planner {es-pull}98296[#98296]
+
+Search::
+* Refactor nested field handling in `FieldFetcher` {es-pull}97683[#97683]
+
+