diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index e8886c038ca58..8e1e2178e9e72 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -89,6 +89,7 @@ When you are running Elasticsearch 7 with Java 8, you are not able to parse the timezone `GMT0` properly anymore. The reason for this is a bug in the JDK, which has not been fixed for JDK8. You can read more in the https://bugs.openjdk.java.net/browse/JDK-8138664[official issue] +This bug is fixed in JDK9 and later versions. [float] ==== Scripting with dates should use java time based methods @@ -125,3 +126,10 @@ and should be replaced. With the switch to java time, support for negative timestamps has been removed. For dates before 1970, use a date format containing a year. + + +[float] +==== Migration guide +For a detailed migration guide, see <>. + +include::migrate_to_java_time.asciidoc[] diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc new file mode 100644 index 0000000000000..2eab8e2de7f28 --- /dev/null +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -0,0 +1,410 @@ +[[migrate-to-java-time]] +=== Java time migration guide + +With 7.0, {es} switched from joda time to java time for date-related parsing, +formatting, and calculations. This guide is designed to help you determine +if your cluster is impacted and, if so, prepare for the upgrade. + +[discrete] +[[java-time-convert-date-formats]] +=== Convert date formats + +To upgrade to 7.0, you'll need to convert any joda-time date formats +to their java-time equivalents. + +To help track this effort, you can prefix java-time date formats with an `8` + in {es} 6.8 and later versions. + +For example, you can change the date format `YYYY-MM-dd` to `8yyyy-MM-dd` to +indicate the date format uses java time. + +{es} treats date formats starting with the `8` prefix differently depending on +the version: + +*6.8*: Date formats with an `8` prefix are handled as java-time formats. Date +formats without an `8` prefix are treated as joda-time formats. We recommend +converting these joda-time formats to java-time _before_ upgrading to 7.x. + +*7.x and later*: For indices created in 6.x, date formats without an `8` prefix +are treated as joda-time formats. For indices created in 7.x and later versions, +all date formats are treated as java-time formats, regardless of whether it +starts with an `8` prefix. + +[[java-time-migration-impacted-features]] +==== Impacted features +The switch to java time only impacts custom <> and +<> formats. + +These formats are commonly used in: + +* <> +* <> +* <> + +If you don't use custom date formats, you can skip the rest of this guide. +Most custom date formats are compatible. However, several require +an update. + +To see if your date format is impacted, use the <> +or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant]. + +[[java-time-migration-incompatible-date-formats]] +==== Incompatible date formats +Custom date formats containing the following joda-time literals should be +changed before upgrading. + +`Y` (Year of era):: ++ +-- +Replace with `y`. + +*Example:* +`YYYY-MM-dd` should become `yyyy-MM-dd`. + +In java time, `Y` is used for +https://docs.oracle.com/javase/8/docs/api/java/time/temporal/WeekFields.html[week-based year]. +Using `Y` in place of `y` could result in off-by-one errors in year calculation. + +For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z` will give `2019-01` +For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter-intuitive) because there is >4 days of that week in 2019 +-- + +`y` (Year):: ++ +-- +Replace with `u`. + +*Example:* +`yyyy-MM-dd` should become `uuuu-MM-dd`. + +In java time, `y` is used for year of era. `u` can contain non-positive +values while `y` cannot. `y` can also be associated with an era field. +-- + + +`C` (Century of era):: ++ +-- +Century of era is not supported in java time. +There is no replacement. Instead, we recommend you preprocess your input. +-- + +`x` (Week year):: ++ +-- +Replace with `Y`. + +In java time, `x` means https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html[zone-offset]. + +[WARNING] +==== +Failure to properly convert `x` (Week year) to `Y` could result in data loss. +==== +-- + +`Z` (Zone offset/id):: ++ +-- +Replace with multiple `X`'s. + +`Z` has a similar meaning in java time. However, java time expects different +numbers of literals to parse different forms. + +Consider migrating to `X`, which gives you more control over how time is parsed. +For example, the joda-time format `YYYY-MM-dd'T'hh:mm:ssZZ` accepts the following dates: + +``` +2010-01-01T01:02:03Z +2010-01-01T01:02:03+01 +2010-01-01T01:02:03+01:02 +2010-01-01T01:02:03+01:02:03 +``` + +In java time, you cannot parse all these dates using a single format +Instead, you must specify 3 separate formats: + +``` +2010-01-01T01:02:03Z +2010-01-01T01:02:03+01 +both parsed with yyyy-MM-dd'T'hh:mm:ssX + +2010-01-01T01:02:03+01:02 +yyyy-MM-dd'T'hh:mm:ssXXX + +2010-01-01T01:02:03+01:02:03 +yyyy-MM-dd'T'hh:mm:ssXXXXX +``` + + +The formats must then be delimited using `||`: +[source,txt] +-------------------------------------------------- +yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX +-------------------------------------------------- + +The same applies if you expect your pattern to occur without a colon (`:`): +For example, the `YYYY-MM-dd'T'hh:mm:ssZ` format accepts the following date forms: +``` +2010-01-01T01:02:03Z +2010-01-01T01:02:03+01 +2010-01-01T01:02:03+0102 +2010-01-01T01:02:03+010203 +``` +To accept all these forms in java time, you must use the `||` delimiter: +[source,txt] +-------------------------------------------------- +yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX +-------------------------------------------------- +-- + +`d` (Day):: ++ +-- +In java time, `d` is still interpreted as "day" but is less flexible. + +For example, the joda-time date format `YYYY-MM-dd` accepts `2010-01-01` or +`2010-01-1`. + +In java time, you must use the `||` delimiter to provide specify each format: + +[source,txt] +-------------------------------------------------- +yyyy-MM-dd||yyyy-MM-d +-------------------------------------------------- + +In java time, `d` also does not accept more than 2 digits. To accept days with more +than two digits, you must include a text literal in your java-time date format. +For example, to parse `2010-01-00001`, you must use the following java-time date format: + +[source,txt] +-------------------------------------------------- +yyyy-MM-'000'dd +-------------------------------------------------- +-- + +`e` (Name of day):: ++ +-- +In java time, `e` is still interpreted as "name of day" but does not parse +short- or full-text forms. + +For example, the joda-time date format `EEE YYYY-MM` accepts both +`Wed 2020-01` and `Wednesday 2020-01`. + +To accept both of these dates in java time, you must specify each format using +the `||` delimiter: + +[source,txt] +-------------------------------------------------- +cccc yyyy-MM||ccc yyyy-MM +-------------------------------------------------- + +The joda-time literal `E` is interpreted as "day of week." +The java-time literal `c` is interpreted as "localized day of week." +`E` does not accept full-text day formats, such as `Wednesday`. +-- + +`EEEE` and similar text forms:: ++ +-- +Support for full-text forms depends on the locale data provided with your Java +Development Kit (JDK) and other implementation details. We recommend you +test formats containing these patterns carefully before upgrading. +-- + +`z` (Time zone text):: ++ +-- +In java time, `z` outputs 'Z' for Zulu when given a UTC timezone. +-- + +[[java-time-migration-test]] +===== Test with your data + +We strongly recommend you test any date format changes using real data before +deploying in production. + +For help with date debugging, consider using +https://esddd.herokuapp.com/[https://esddd.herokuapp.com/.] + +[[java-time-migrate-update-mappings]] +==== Update index mappings +To update joda-time date formats in index mappings, you must create a new index +with an updated mapping and reindex your data to it. +You can however update your pipelines or templates. + +The following `my_index_1` index contains a mapping for the `datetime` field, a +`date` field with a custom joda-time date format. +//// +[source,console] +-------------------------------------------------- +PUT my_index_1 +{ + "mappings" : { + "properties" : { + "datetime": { + "type": "date", + "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis" + } + } + } +} +-------------------------------------------------- +//// + +[source,console] +-------------------------------------------------- +GET my_index_1/_mapping +-------------------------------------------------- +// TEST[continued] + +[source,console-result] +-------------------------------------------------- +{ + "my_index_1" : { + "mappings" : { + "properties" : { + "datetime": { + "type": "date", + "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis" + } + } + } + } +} +-------------------------------------------------- + + +To change the date format for the `datetime` field, create a separate index +containing an updated mapping and date format. + +For example, the following `my_index_2` index changes the `datetime` field's +date format to `8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis`. The `8` prefix +indicates this date format uses java time. + +[source,console] +-------------------------------------------------- +PUT my_index_2 +{ + "mappings": { + "properties": { + "datetime": { + "type": "date", + "format": "8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis" + } + } + } +} +-------------------------------------------------- +// TEST[continued] + +Next, reindex data from the old index to the new index. + +The following <> API request reindexes data from +`my_index_1` to `my_index_2`. + +[source,console] +-------------------------------------------------- +POST _reindex +{ + "source": { + "index": "my_index_1" + }, + "dest": { + "index": "my_index_2" + } +} +-------------------------------------------------- +// TEST[continued] + +If you use index aliases, update them to point to the new index. + +[source,console] +-------------------------------------------------- +POST /_aliases +{ + "actions" : [ + { "remove" : { "index" : "my_index_1", "alias" : "my_index" } }, + { "add" : { "index" : "my_index_2", "alias" : "my_index" } } + ] +} +-------------------------------------------------- +// TEST[continued] + +[[java-time-migration-update-ingest-pipelines]] +===== Update ingest pipelines +If your ingest pipelines contain joda-time date formats, you can update them +using the <> API. + +[source,console] +-------------------------------------------------- +PUT _ingest/pipeline/my_pipeline +{ + "description": "Pipeline for routing data to specific index", + "processors": [ + { + "date": { + "field": "createdTime", + "formats": [ + "8uuuu-w" + ] + }, + "date_index_name": { + "field": "@timestamp", + "date_rounding": "d", + "index_name_prefix": "x-", + "index_name_format": "8uuuu-w" + } + } + ] +} +-------------------------------------------------- + + +[[java-time-migration-update-index-templates]] +===== Update index templates + +If your index templates contain joda-time date formats, you can update them +using the <> API. + +[source,console] +-------------------------------------------------- +PUT _template/template_1 +{ + "index_patterns": [ + "te*", + "bar*" + ], + "settings": { + "number_of_shards": 1 + }, + "mappings": { + "_source": { + "enabled": false + }, + "properties": { + "host_name": { + "type": "keyword" + }, + "created_at": { + "type": "date", + "format": "8EEE MMM dd HH:mm:ss Z yyyy" + } + } + } +} +-------------------------------------------------- + +//// +[source,console] +-------------------------------------------------- +DELETE /_template/template_1 +-------------------------------------------------- +// TEST[continued] +//// + +[[java-time-migration-update-external-tools-templates]] +===== Update external tools and templates +Ensure you also update any date formats in templates or tools outside of {es}. +This can include tools such as {beats-ref}/getting-started.html[{beats}] or +{logstash-ref}/index.html[Logstash]. diff --git a/docs/reference/release-notes/7.0.asciidoc b/docs/reference/release-notes/7.0.asciidoc index 751c824207db1..2dffe3da4f564 100644 --- a/docs/reference/release-notes/7.0.asciidoc +++ b/docs/reference/release-notes/7.0.asciidoc @@ -6,6 +6,15 @@ releases of 7.0.0. Also see <>. +[float] +=== Known issues + +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + + [[breaking-7.0.0]] [float] === Breaking changes diff --git a/docs/reference/release-notes/7.1.asciidoc b/docs/reference/release-notes/7.1.asciidoc index ec93927513b85..5f9df16026363 100644 --- a/docs/reference/release-notes/7.1.asciidoc +++ b/docs/reference/release-notes/7.1.asciidoc @@ -3,6 +3,14 @@ Also see <>. +[float] +=== Known issues + +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + [[bug-7.1.1]] [float] === Bug fixes diff --git a/docs/reference/release-notes/7.2.asciidoc b/docs/reference/release-notes/7.2.asciidoc index 6f6c12ed65f7a..98252068cc51a 100644 --- a/docs/reference/release-notes/7.2.asciidoc +++ b/docs/reference/release-notes/7.2.asciidoc @@ -85,6 +85,14 @@ Security:: Also see <>. +[float] +=== Known issues + +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + [[breaking-7.2.0]] [float] === Breaking changes diff --git a/docs/reference/release-notes/7.3.asciidoc b/docs/reference/release-notes/7.3.asciidoc index 975f183514d75..0e7452202fe4e 100644 --- a/docs/reference/release-notes/7.3.asciidoc +++ b/docs/reference/release-notes/7.3.asciidoc @@ -125,6 +125,7 @@ Security:: * Use system context for looking up connected nodes {pull}43991[#43991] (issue: {issue}43974[#43974]) + [[upgrade-7.3.1]] [float] === Upgrades @@ -138,6 +139,16 @@ Infra/Packaging:: Also see <>. + +[float] +=== Known issues + +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + + [[breaking-7.3.0]] [float] === Breaking changes diff --git a/docs/reference/release-notes/7.4.asciidoc b/docs/reference/release-notes/7.4.asciidoc index 80550d53c8ee3..084d5224f94d2 100644 --- a/docs/reference/release-notes/7.4.asciidoc +++ b/docs/reference/release-notes/7.4.asciidoc @@ -116,6 +116,11 @@ Any attempt to log a slow search can throw an AIOOBE due to a bug that performs concurrent modifications on a shared byte array. (issue: {issue}/48358[#48358]) +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + [[breaking-7.4.0]] [float] === Breaking changes diff --git a/docs/reference/release-notes/7.5.asciidoc b/docs/reference/release-notes/7.5.asciidoc index 1faf181b92f24..5f115314cbe1d 100644 --- a/docs/reference/release-notes/7.5.asciidoc +++ b/docs/reference/release-notes/7.5.asciidoc @@ -98,6 +98,11 @@ Also see <>. If a {transform} is running during upgrade, the {transform} audit index might disappear. (issue: {issue}/49730[#49730]) +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + [[breaking-7.5.0]] [float] === Breaking changes diff --git a/f b/f new file mode 100644 index 0000000000000..e69de29bb2d1d