From 9bbfc8be588e6abcaf5a8c25ad8d3eff611e57bd Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Thu, 6 Feb 2020 10:57:28 +0100 Subject: [PATCH 01/81] draft --- .../migration/migrate_7_0/java_time.asciidoc | 7 + .../migrate_7_0/migrate_to_java_time.asciidoc | 130 ++++++++++++++++++ 2 files changed, 137 insertions(+) create mode 100644 docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index e8886c038ca58..3655b0bc6b1c9 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -89,6 +89,7 @@ When you are running Elasticsearch 7 with Java 8, you are not able to parse the timezone `GMT0` properly anymore. The reason for this is a bug in the JDK, which has not been fixed for JDK8. You can read more in the https://bugs.openjdk.java.net/browse/JDK-8138664[official issue] +The bug in JDK version 9 and newer. [float] ==== Scripting with dates should use java time based methods @@ -125,3 +126,9 @@ and should be replaced. With the switch to java time, support for negative timestamps has been removed. For dates before 1970, use a date format containing a year. + + +[float] +==== Migration Guide +You can find a detailed migration guide here +include::migrate_7_0/migrate_to_java_time.asciidoc[] diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc new file mode 100644 index 0000000000000..7ac6a2d2ed9ad --- /dev/null +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -0,0 +1,130 @@ +=== Java Time Migration Guide + +Since version 7.0 Elasticsearch is using java.time as an underlying implementation of dates parsing, formatting and calculations. +java.time is many aspects very similar to joda-time which we previously used. However there are some differences and this guide is meant to +help to become compatible. + +==== Identifying if you are affected +You are only affected if you have used any custom date formats in your mappings, pipelines or templates. +Most of the custom date formats are compatible, but with some there is a change of meaning or missing. More on that below. +A custom format could be specified on a date field like this: +[source,console] +-------------------------------------------------- +PUT my_index +{ + "mappings": { + "properties": { + "date": { + "type": "date", + "format": "YYYY-MM-dd" + } + } + } +} +-------------------------------------------------- +See more on: +https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html#multiple-date-formats + +To make sure if any of the date formats is deprecated you should run deprecation API or kibana upgrade assistant. +* <> +include::apis/deprecation.asciidoc[] + +Sample output of deprecation api will be as follows: +[source,console] +-------------------------------------------------- +This index has date fields with deprecated formats: [[type: doc, field: param2, format: yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis, suggestion: 'y' year should be replaced with 'u'. Use 'y' for year-of-era.]]. Prefix your date format with '8' to use the new specifier. +-------------------------------------------------- + +==== Incompatible formats +If your pattern contains literals below, that means your format is incompatible and needs to be changed. + +===== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is an big change as in 7.0 Y means week-based-year. +If you don't update your date pattern, you will have problems with search results, parsing and date calculations. +example: +YYYY-MM-dd should become yyyy-MM-dd + +====== The difference between year-of-era and week based year +For pattern YYYY-ww and date 2019-01-01T00:00:00.000Z will give 2019-01 +For pattern YYYY-ww and date 2018-12-31T00:00:00.000Z will give 2019-01 (counter intuitive) because there is >4 days of that year 2019 on that week + +===== y - in 6.8 meaning year. It should be replaced to 'u' in 7.0. This is a subtle change, but worth fixing. +As previously stated 'y' means year-of-era. 'u' means year. The difference is that year can contain non positive values, whereas 'y' can not. //clarify 0 year +year-of-era can also be associated with an era field. + +===== C - century of era is no longer supported in 7.0. There is no replacement, so you need to preprocess your input. +https://stackoverflow.com/questions/38354151/how-to-force-java-time-localdate-to-assume-19th-century-as-yy-year-pattern + +===== x - in 6.8 meaning week year. It should be replaced with 'Y' in 7.0. Again a big change as 'x' means zone-offset in 7.0. +You will get parsing, search problems if you won't update. + +===== Z - in 6.8 meaning time zone offset/id. It has similar meaning in 7.0 with a notable difference that it expects different number of literals +to parse different forms. +It also won't parse 'Z' for Zulu timezone. You should consider migrating to 'X' which give you more control of how your time is parsed. +For instance format YYYY-MM-dd'T'hh:mm:ssZZ +allowed all the forms below: +2010-01-01T01:02:03Z +2010-01-01T01:02:03+01 +2010-01-01T01:02:03+01:02 +2010-01-01T01:02:03+01:02:03 + +You cannot parse them all with one pattern in 7.0. You need to specify 3 separate patterns +2010-01-01T01:02:03Z +2010-01-01T01:02:03+01 +both parsed with yyyy-MM-dd'T'hh:mm:ssX //clarify Z + +2010-01-01T01:02:03+01:02 +yyyy-MM-dd'T'hh:mm:ssXXX + +2010-01-01T01:02:03+01:02:03 +yyyy-MM-dd'T'hh:mm:ssXXXXX + +If you expect all these combination to occur in your data you need to combine these patterns in 7.0 +yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX + +The same applies if you expected your pattern to occur without a colon: +Pattern YYYY-MM-dd'T'hh:mm:ssZ accepting these dates: +2010-01-01T01:02:03Z +2010-01-01T01:02:03+01 +2010-01-01T01:02:03+0102 +2010-01-01T01:02:03+010203 +should be migrated to +yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX + +===== d for day. +This one have the same meaning in 7.0 but are less flexible. +If your pattern was YYYY-MM-dd in 6.8 and you expects dates in form +2010-01-01 or 2010-01-1 +You will need to provide an alternative pattern for all of these. +yyyy-MM-dd||yyyy-MM-d +More then 2 digits are not allowed anymore. If you want to parse dates like 2010-01-00001 you need to create a pattern that expects a text literal '000' before 'd' +yyyy-MM-'000'dd + +===== e for name of day +Also has a similar meaning in 7.0. However was more flexible in 6.8 and allowed to parse short and full text forms. +if you used EEE YYYY-MM for a format and expected days in form: +Wed 2020-01 or Wednesday 2020-01 +then you have to use two combined patterns in 7.0 +cccc yyyy-MM||ccc yyyy-MM + +E vs c. E means day-of-week vs c is for localized day-of-week. The difference is that E won't behave correctly for full text forms like Wednesday +//todo confirm + +===== more on text forms EEEE and similar //todo + +===== 'z' +'z' time zone text. Will print 'Z' for Zulu given UTC timezone. +//todo expand + +==== Migrating affected mappings +Once an index is created with a mapping, it cannot change already existing fields. You need to create a new index with updated mapping +and reindex your data to it. +You can howeve update your pipelines or templates. + +- Create new_index_1 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis as required +- Reindex the deprecated format index to new_index_1 +- repoint aliases?? + +- upgrade pipeline +- upgrade template + +- Upgrade to 7.x From e1752b1f66eea1ad113ca864eefd8ca6014b43ec Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Thu, 6 Feb 2020 11:00:37 +0100 Subject: [PATCH 02/81] todo --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 7ac6a2d2ed9ad..097579eb08ade 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -128,3 +128,6 @@ You can howeve update your pipelines or templates. - upgrade template - Upgrade to 7.x + +//todo +on templates outside ES From 8fa601ca973e19219d0eddb4fcf1a737c968f2b3 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Tue, 25 Feb 2020 16:57:22 +0100 Subject: [PATCH 03/81] more docs --- .../migration/migrate_7_0/java_time.asciidoc | 2 +- .../migrate_7_0/migrate_to_java_time.asciidoc | 156 ++++++++++++++++-- 2 files changed, 143 insertions(+), 15 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index 3655b0bc6b1c9..2dd4f3fdc66cb 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -131,4 +131,4 @@ For dates before 1970, use a date format containing a year. [float] ==== Migration Guide You can find a detailed migration guide here -include::migrate_7_0/migrate_to_java_time.asciidoc[] +include::migrate_to_java_time.asciidoc[] diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 097579eb08ade..d507e9b252463 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -1,13 +1,13 @@ === Java Time Migration Guide Since version 7.0 Elasticsearch is using java.time as an underlying implementation of dates parsing, formatting and calculations. -java.time is many aspects very similar to joda-time which we previously used. However there are some differences and this guide is meant to -help to become compatible. +java.time is in many aspects very similar to joda-time which we previously used. However there are some differences and this guide is meant to +help you to become compatible. ==== Identifying if you are affected You are only affected if you have used any custom date formats in your mappings, pipelines or templates. Most of the custom date formats are compatible, but with some there is a change of meaning or missing. More on that below. -A custom format could be specified on a date field like this: +A custom format is specified on a date field like this: [source,console] -------------------------------------------------- PUT my_index @@ -25,7 +25,7 @@ PUT my_index See more on: https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html#multiple-date-formats -To make sure if any of the date formats is deprecated you should run deprecation API or kibana upgrade assistant. +To make sure if any of the date formats is deprecated run deprecation API or kibana upgrade assistant. * <> include::apis/deprecation.asciidoc[] @@ -36,16 +36,16 @@ This index has date fields with deprecated formats: [[type: doc, field: param2, -------------------------------------------------- ==== Incompatible formats -If your pattern contains literals below, that means your format is incompatible and needs to be changed. +If a pattern contains literals below, that means the pattern is incompatible and needs to be changed. -===== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is an big change as in 7.0 Y means week-based-year. +===== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is a big change because in 7.0 'Y' means week-based-year. If you don't update your date pattern, you will have problems with search results, parsing and date calculations. example: YYYY-MM-dd should become yyyy-MM-dd ====== The difference between year-of-era and week based year For pattern YYYY-ww and date 2019-01-01T00:00:00.000Z will give 2019-01 -For pattern YYYY-ww and date 2018-12-31T00:00:00.000Z will give 2019-01 (counter intuitive) because there is >4 days of that year 2019 on that week +For pattern YYYY-ww and date 2018-12-31T00:00:00.000Z will give 2019-01 (counter intuitive) because there is >4 days of that week in 2019 ===== y - in 6.8 meaning year. It should be replaced to 'u' in 7.0. This is a subtle change, but worth fixing. As previously stated 'y' means year-of-era. 'u' means year. The difference is that year can contain non positive values, whereas 'y' can not. //clarify 0 year @@ -81,7 +81,7 @@ yyyy-MM-dd'T'hh:mm:ssXXXXX If you expect all these combination to occur in your data you need to combine these patterns in 7.0 yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX -The same applies if you expected your pattern to occur without a colon: +The same applies if you expect your pattern to occur without a colon: Pattern YYYY-MM-dd'T'hh:mm:ssZ accepting these dates: 2010-01-01T01:02:03Z 2010-01-01T01:02:03+01 @@ -115,19 +115,147 @@ E vs c. E means day-of-week vs c is for localized day-of-week. The difference is 'z' time zone text. Will print 'Z' for Zulu given UTC timezone. //todo expand +==== Test with your data +All of the examples above should be tested with real data. It is likely that you do not need the flexibility joda was giving. +If you expected your timezone to only be in +01:00 form (XXX in java, ZZ in joda), +then there is no need to create a new java pattern with so many alternatives. + +Consider using this date debugging site for assistance https://esddd.herokuapp.com/ + ==== Migrating affected mappings Once an index is created with a mapping, it cannot change already existing fields. You need to create a new index with updated mapping and reindex your data to it. -You can howeve update your pipelines or templates. +You can however update your pipelines or templates. Remember to look for templates you use with tools outside ES. +If you specified a custom date format there, then you need to update it too. + +===== Example migration procedure +Let's assume that you have an index with a date field and custom format +[source,console] +---- +GET /my_index_1/_mapping +---- +// TEST[continued] + +[source,console-result] +---- +{ + "my_index_1" : { + "mappings" : { + "properties" : { + "datetime": { + "type": "date", + "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis" + } + } + } + } +} +---- + + +- Create my_index_2 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis +[source,console] +-------------------------- +PUT my_index_2 +{ + "mappings": { + "properties": { + "datetime": { + "type": "date", + "format": "8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis" + } + } + } +} +-------------------------- -- Create new_index_1 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis as required - Reindex the deprecated format index to new_index_1 -- repoint aliases?? +POST _reindex +{ + "source": { + "index": "my_index_1" + }, + "dest": { + "index": "my_index_2" + } +} +---- +// NOTCONSOLE + +- If you were using aliases, update them to a new index +[source,console] +-------------------------------------------------- +POST /_aliases +{ + "actions" : [ + { "remove" : { "index" : "my_index_1", "alias" : "my_index" } }, + { "add" : { "index" : "my_index_2", "alias" : "my_index" } } + ] +} +-------------------------------------------------- +// NOTCONSOLE + +===== Update before upgrading to ES7. +- update pipeline +If your pipelines were using a joda style patterns, they also have to be updated. There is no need to create a new pipeline. +Just update the already existing one. +[source,console] +-------------------------------------------------- +PUT _ingest/pipeline/mypipeline +{ + "description": "Pipeline for routing data to specific index", + "processors": [ + { + "date": { + "field": "createdTime", + "formats": [ + "8uuuu-w" + ] + }, + "date_index_name": { + "field": "@timestamp", + "date_rounding": "d", + "index_name_prefix": "x-", + "index_name_format": "8uuuu-w" + } + } + ] +} +-------------------------------------------------- +// NOTCONSOLE -- upgrade pipeline - upgrade template +If your tempalte was using joda date pattern it also should be updated before upgrading to ES7. +[source,console] +-------------------------------------------------- +PUT _template/template_1 +{ + "index_patterns": [ + "te*", + "bar*" + ], + "settings": { + "number_of_shards": 1 + }, + "mappings": { + "_source": { + "enabled": false + }, + "properties": { + "host_name": { + "type": "keyword" + }, + "created_at": { + "type": "date", + "format": "EEE MMM dd HH:mm:ss Z yyyy" + } + } + } +} +-------------------------------------------------- +// NOTCONSOLE - Upgrade to 7.x -//todo -on templates outside ES +===== External templates +Revisit other templates from elastic stack where you used a custom date pattern. From 8770f30a496bcca1dfa8422d27bf106235fc9b5b Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Tue, 25 Feb 2020 17:46:41 +0100 Subject: [PATCH 04/81] link --- docs/reference/migration/migrate_7_0/java_time.asciidoc | 1 + .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 ++ 2 files changed, 3 insertions(+) diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index 2dd4f3fdc66cb..03e22b6526d09 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -131,4 +131,5 @@ For dates before 1970, use a date format containing a year. [float] ==== Migration Guide You can find a detailed migration guide here +* <> include::migrate_to_java_time.asciidoc[] diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index d507e9b252463..0742db9606d66 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -1,3 +1,5 @@ +[float] +[[java_time_migration_guide]] === Java Time Migration Guide Since version 7.0 Elasticsearch is using java.time as an underlying implementation of dates parsing, formatting and calculations. From 6bed675c2d648c17155ef507b0d04f5298f4a3f4 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 26 Feb 2020 10:11:03 +0100 Subject: [PATCH 05/81] docs fix --- .../migration/migrate_7_0/java_time.asciidoc | 6 ++- .../migrate_7_0/migrate_to_java_time.asciidoc | 44 ++++++++++--------- 2 files changed, 27 insertions(+), 23 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index 03e22b6526d09..9cb57d1416a18 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -89,7 +89,7 @@ When you are running Elasticsearch 7 with Java 8, you are not able to parse the timezone `GMT0` properly anymore. The reason for this is a bug in the JDK, which has not been fixed for JDK8. You can read more in the https://bugs.openjdk.java.net/browse/JDK-8138664[official issue] -The bug in JDK version 9 and newer. +The fix is in JDK version 9 and newer. [float] ==== Scripting with dates should use java time based methods @@ -131,5 +131,7 @@ For dates before 1970, use a date format containing a year. [float] ==== Migration Guide You can find a detailed migration guide here -* <> + +* <> + include::migrate_to_java_time.asciidoc[] diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 0742db9606d66..545361d7caf1a 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -1,12 +1,12 @@ [float] -[[java_time_migration_guide]] -=== Java Time Migration Guide +[[migrate_to_java_time]] += Java Time Migration Guide Since version 7.0 Elasticsearch is using java.time as an underlying implementation of dates parsing, formatting and calculations. java.time is in many aspects very similar to joda-time which we previously used. However there are some differences and this guide is meant to help you to become compatible. -==== Identifying if you are affected +== Identifying if you are affected You are only affected if you have used any custom date formats in your mappings, pipelines or templates. Most of the custom date formats are compatible, but with some there is a change of meaning or missing. More on that below. A custom format is specified on a date field like this: @@ -29,7 +29,7 @@ https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html#multip To make sure if any of the date formats is deprecated run deprecation API or kibana upgrade assistant. * <> -include::apis/deprecation.asciidoc[] +include::../apis/deprecation.asciidoc[] Sample output of deprecation api will be as follows: [source,console] @@ -37,29 +37,29 @@ Sample output of deprecation api will be as follows: This index has date fields with deprecated formats: [[type: doc, field: param2, format: yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis, suggestion: 'y' year should be replaced with 'u'. Use 'y' for year-of-era.]]. Prefix your date format with '8' to use the new specifier. -------------------------------------------------- -==== Incompatible formats +== Incompatible formats If a pattern contains literals below, that means the pattern is incompatible and needs to be changed. -===== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is a big change because in 7.0 'Y' means week-based-year. +=== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is a big change because in 7.0 'Y' means week-based-year. If you don't update your date pattern, you will have problems with search results, parsing and date calculations. example: YYYY-MM-dd should become yyyy-MM-dd -====== The difference between year-of-era and week based year +==== The difference between year-of-era and week based year For pattern YYYY-ww and date 2019-01-01T00:00:00.000Z will give 2019-01 For pattern YYYY-ww and date 2018-12-31T00:00:00.000Z will give 2019-01 (counter intuitive) because there is >4 days of that week in 2019 -===== y - in 6.8 meaning year. It should be replaced to 'u' in 7.0. This is a subtle change, but worth fixing. +=== y - in 6.8 meaning year. It should be replaced to 'u' in 7.0. This is a subtle change, but worth fixing. As previously stated 'y' means year-of-era. 'u' means year. The difference is that year can contain non positive values, whereas 'y' can not. //clarify 0 year year-of-era can also be associated with an era field. -===== C - century of era is no longer supported in 7.0. There is no replacement, so you need to preprocess your input. +=== C - century of era is no longer supported in 7.0. There is no replacement, so you need to preprocess your input. https://stackoverflow.com/questions/38354151/how-to-force-java-time-localdate-to-assume-19th-century-as-yy-year-pattern -===== x - in 6.8 meaning week year. It should be replaced with 'Y' in 7.0. Again a big change as 'x' means zone-offset in 7.0. +=== x - in 6.8 meaning week year. It should be replaced with 'Y' in 7.0. Again a big change as 'x' means zone-offset in 7.0. You will get parsing, search problems if you won't update. -===== Z - in 6.8 meaning time zone offset/id. It has similar meaning in 7.0 with a notable difference that it expects different number of literals +=== Z - in 6.8 meaning time zone offset/id. It has similar meaning in 7.0 with a notable difference that it expects different number of literals to parse different forms. It also won't parse 'Z' for Zulu timezone. You should consider migrating to 'X' which give you more control of how your time is parsed. For instance format YYYY-MM-dd'T'hh:mm:ssZZ @@ -92,7 +92,7 @@ Pattern YYYY-MM-dd'T'hh:mm:ssZ accepting these dates: should be migrated to yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX -===== d for day. +=== d for day. This one have the same meaning in 7.0 but are less flexible. If your pattern was YYYY-MM-dd in 6.8 and you expects dates in form 2010-01-01 or 2010-01-1 @@ -101,7 +101,7 @@ yyyy-MM-dd||yyyy-MM-d More then 2 digits are not allowed anymore. If you want to parse dates like 2010-01-00001 you need to create a pattern that expects a text literal '000' before 'd' yyyy-MM-'000'dd -===== e for name of day +=== e for name of day Also has a similar meaning in 7.0. However was more flexible in 6.8 and allowed to parse short and full text forms. if you used EEE YYYY-MM for a format and expected days in form: Wed 2020-01 or Wednesday 2020-01 @@ -111,26 +111,26 @@ cccc yyyy-MM||ccc yyyy-MM E vs c. E means day-of-week vs c is for localized day-of-week. The difference is that E won't behave correctly for full text forms like Wednesday //todo confirm -===== more on text forms EEEE and similar //todo +=== more on text forms EEEE and similar //todo -===== 'z' +=== 'z' 'z' time zone text. Will print 'Z' for Zulu given UTC timezone. //todo expand -==== Test with your data +=== Test with your data All of the examples above should be tested with real data. It is likely that you do not need the flexibility joda was giving. If you expected your timezone to only be in +01:00 form (XXX in java, ZZ in joda), then there is no need to create a new java pattern with so many alternatives. Consider using this date debugging site for assistance https://esddd.herokuapp.com/ -==== Migrating affected mappings +== Migrating affected mappings Once an index is created with a mapping, it cannot change already existing fields. You need to create a new index with updated mapping and reindex your data to it. You can however update your pipelines or templates. Remember to look for templates you use with tools outside ES. If you specified a custom date format there, then you need to update it too. -===== Example migration procedure +=== Example migration procedure Let's assume that you have an index with a date field and custom format [source,console] ---- @@ -172,6 +172,8 @@ PUT my_index_2 -------------------------- - Reindex the deprecated format index to new_index_1 +[source,console] +-------------------------------------------------- POST _reindex { "source": { @@ -181,7 +183,7 @@ POST _reindex "index": "my_index_2" } } ----- +-------------------------------------------------- // NOTCONSOLE - If you were using aliases, update them to a new index @@ -197,7 +199,7 @@ POST /_aliases -------------------------------------------------- // NOTCONSOLE -===== Update before upgrading to ES7. +=== Update before upgrading to ES7. - update pipeline If your pipelines were using a joda style patterns, they also have to be updated. There is no need to create a new pipeline. Just update the already existing one. @@ -259,5 +261,5 @@ PUT _template/template_1 - Upgrade to 7.x -===== External templates +=== External templates Revisit other templates from elastic stack where you used a custom date pattern. From 44a7c7e4e07884251367d3fcb6c2deb797044254 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 26 Feb 2020 10:57:32 +0100 Subject: [PATCH 06/81] fix build --- .../migrate_7_0/migrate_to_java_time.asciidoc | 84 +++++++++---------- f | 0 2 files changed, 41 insertions(+), 43 deletions(-) create mode 100644 f diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 545361d7caf1a..e29c3010ea40f 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -1,12 +1,11 @@ -[float] [[migrate_to_java_time]] -= Java Time Migration Guide +=== Java Time Migration Guide Since version 7.0 Elasticsearch is using java.time as an underlying implementation of dates parsing, formatting and calculations. java.time is in many aspects very similar to joda-time which we previously used. However there are some differences and this guide is meant to help you to become compatible. -== Identifying if you are affected +==== Identifying if you are affected You are only affected if you have used any custom date formats in your mappings, pipelines or templates. Most of the custom date formats are compatible, but with some there is a change of meaning or missing. More on that below. A custom format is specified on a date field like this: @@ -18,48 +17,47 @@ PUT my_index "properties": { "date": { "type": "date", - "format": "YYYY-MM-dd" + "format": "YYYY/MM/dd HH:mm:ss||YYYY/MM/dd||epoch_millis" } } } } -------------------------------------------------- -See more on: -https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html#multiple-date-formats To make sure if any of the date formats is deprecated run deprecation API or kibana upgrade assistant. -* <> -include::../apis/deprecation.asciidoc[] +xx migration-api-deprecation xx +xxinclude ::../apis/deprecation.asciidoc Sample output of deprecation api will be as follows: -[source,console] +[source,text] -------------------------------------------------- This index has date fields with deprecated formats: [[type: doc, field: param2, format: yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis, suggestion: 'y' year should be replaced with 'u'. Use 'y' for year-of-era.]]. Prefix your date format with '8' to use the new specifier. -------------------------------------------------- -== Incompatible formats + +==== Incompatible formats If a pattern contains literals below, that means the pattern is incompatible and needs to be changed. -=== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is a big change because in 7.0 'Y' means week-based-year. +===== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is a big change because in 7.0 'Y' means week-based-year. If you don't update your date pattern, you will have problems with search results, parsing and date calculations. example: YYYY-MM-dd should become yyyy-MM-dd -==== The difference between year-of-era and week based year +====== The difference between year-of-era and week based year For pattern YYYY-ww and date 2019-01-01T00:00:00.000Z will give 2019-01 For pattern YYYY-ww and date 2018-12-31T00:00:00.000Z will give 2019-01 (counter intuitive) because there is >4 days of that week in 2019 -=== y - in 6.8 meaning year. It should be replaced to 'u' in 7.0. This is a subtle change, but worth fixing. +===== y - in 6.8 meaning year. It should be replaced to 'u' in 7.0. This is a subtle change, but worth fixing. As previously stated 'y' means year-of-era. 'u' means year. The difference is that year can contain non positive values, whereas 'y' can not. //clarify 0 year year-of-era can also be associated with an era field. -=== C - century of era is no longer supported in 7.0. There is no replacement, so you need to preprocess your input. +===== C - century of era is no longer supported in 7.0. There is no replacement, so you need to preprocess your input. https://stackoverflow.com/questions/38354151/how-to-force-java-time-localdate-to-assume-19th-century-as-yy-year-pattern -=== x - in 6.8 meaning week year. It should be replaced with 'Y' in 7.0. Again a big change as 'x' means zone-offset in 7.0. +===== x - in 6.8 meaning week year. It should be replaced with 'Y' in 7.0. Again a big change as 'x' means zone-offset in 7.0. You will get parsing, search problems if you won't update. -=== Z - in 6.8 meaning time zone offset/id. It has similar meaning in 7.0 with a notable difference that it expects different number of literals +===== Z - in 6.8 meaning time zone offset/id. It has similar meaning in 7.0 with a notable difference that it expects different number of literals to parse different forms. It also won't parse 'Z' for Zulu timezone. You should consider migrating to 'X' which give you more control of how your time is parsed. For instance format YYYY-MM-dd'T'hh:mm:ssZZ @@ -92,7 +90,7 @@ Pattern YYYY-MM-dd'T'hh:mm:ssZ accepting these dates: should be migrated to yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX -=== d for day. +===== d for day. This one have the same meaning in 7.0 but are less flexible. If your pattern was YYYY-MM-dd in 6.8 and you expects dates in form 2010-01-01 or 2010-01-1 @@ -101,7 +99,7 @@ yyyy-MM-dd||yyyy-MM-d More then 2 digits are not allowed anymore. If you want to parse dates like 2010-01-00001 you need to create a pattern that expects a text literal '000' before 'd' yyyy-MM-'000'dd -=== e for name of day +===== e for name of day Also has a similar meaning in 7.0. However was more flexible in 6.8 and allowed to parse short and full text forms. if you used EEE YYYY-MM for a format and expected days in form: Wed 2020-01 or Wednesday 2020-01 @@ -111,35 +109,34 @@ cccc yyyy-MM||ccc yyyy-MM E vs c. E means day-of-week vs c is for localized day-of-week. The difference is that E won't behave correctly for full text forms like Wednesday //todo confirm -=== more on text forms EEEE and similar //todo +===== more on text forms EEEE and similar //todo -=== 'z' +===== 'z' 'z' time zone text. Will print 'Z' for Zulu given UTC timezone. //todo expand -=== Test with your data +===== Test with your data All of the examples above should be tested with real data. It is likely that you do not need the flexibility joda was giving. If you expected your timezone to only be in +01:00 form (XXX in java, ZZ in joda), then there is no need to create a new java pattern with so many alternatives. Consider using this date debugging site for assistance https://esddd.herokuapp.com/ -== Migrating affected mappings +==== Migrating affected mappings Once an index is created with a mapping, it cannot change already existing fields. You need to create a new index with updated mapping and reindex your data to it. You can however update your pipelines or templates. Remember to look for templates you use with tools outside ES. If you specified a custom date format there, then you need to update it too. -=== Example migration procedure +===== Example migration procedure Let's assume that you have an index with a date field and custom format [source,console] ----- +-------------------------------------------------- GET /my_index_1/_mapping ----- -// TEST[continued] +-------------------------------------------------- -[source,console-result] ----- +[source,js] +-------------------------------------------------- { "my_index_1" : { "mappings" : { @@ -152,12 +149,13 @@ GET /my_index_1/_mapping } } } ----- +-------------------------------------------------- +todo make this test response?? -- Create my_index_2 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis +* Create my_index_2 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis [source,console] --------------------------- +-------------------------------------------------- PUT my_index_2 { "mappings": { @@ -169,9 +167,10 @@ PUT my_index_2 } } } --------------------------- +-------------------------------------------------- + -- Reindex the deprecated format index to new_index_1 +* Reindex the deprecated format index to new_index_1 [source,console] -------------------------------------------------- POST _reindex @@ -184,9 +183,8 @@ POST _reindex } } -------------------------------------------------- -// NOTCONSOLE -- If you were using aliases, update them to a new index +* If you were using aliases, update them to a new index [source,console] -------------------------------------------------- POST /_aliases @@ -197,10 +195,10 @@ POST /_aliases ] } -------------------------------------------------- -// NOTCONSOLE -=== Update before upgrading to ES7. -- update pipeline + +===== Update before upgrading to ES7. +* update pipeline If your pipelines were using a joda style patterns, they also have to be updated. There is no need to create a new pipeline. Just update the already existing one. [source,console] @@ -226,9 +224,9 @@ PUT _ingest/pipeline/mypipeline ] } -------------------------------------------------- -// NOTCONSOLE -- upgrade template + +* upgrade template If your tempalte was using joda date pattern it also should be updated before upgrading to ES7. [source,console] -------------------------------------------------- @@ -257,9 +255,9 @@ PUT _template/template_1 } } -------------------------------------------------- -// NOTCONSOLE -- Upgrade to 7.x -=== External templates +* Upgrade to 7.x + +===== External templates Revisit other templates from elastic stack where you used a custom date pattern. diff --git a/f b/f new file mode 100644 index 0000000000000..e69de29bb2d1d From 7450e9c34432b0419b1619677104325ecaa103f2 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Tue, 17 Mar 2020 15:30:42 +0100 Subject: [PATCH 07/81] formatting changes --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index e29c3010ea40f..ecb446b61a1f1 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -154,6 +154,7 @@ todo make this test response?? * Create my_index_2 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis + [source,console] -------------------------------------------------- PUT my_index_2 @@ -171,6 +172,7 @@ PUT my_index_2 * Reindex the deprecated format index to new_index_1 + [source,console] -------------------------------------------------- POST _reindex @@ -185,6 +187,7 @@ POST _reindex -------------------------------------------------- * If you were using aliases, update them to a new index + [source,console] -------------------------------------------------- POST /_aliases @@ -201,6 +204,7 @@ POST /_aliases * update pipeline If your pipelines were using a joda style patterns, they also have to be updated. There is no need to create a new pipeline. Just update the already existing one. + [source,console] -------------------------------------------------- PUT _ingest/pipeline/mypipeline @@ -227,7 +231,8 @@ PUT _ingest/pipeline/mypipeline * upgrade template -If your tempalte was using joda date pattern it also should be updated before upgrading to ES7. +If your template was using joda date pattern it also should be updated before upgrading to ES7. + [source,console] -------------------------------------------------- PUT _template/template_1 @@ -249,7 +254,7 @@ PUT _template/template_1 }, "created_at": { "type": "date", - "format": "EEE MMM dd HH:mm:ss Z yyyy" + "format": "8EEE MMM dd HH:mm:ss Z yyyy" } } } From eceb3b84433d369f465a67fcd8d520c7bebcb678 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Tue, 17 Mar 2020 17:11:31 +0100 Subject: [PATCH 08/81] cleanup and todos --- .../migrate_7_0/migrate_to_java_time.asciidoc | 60 ++++++++++++------- 1 file changed, 38 insertions(+), 22 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index ecb446b61a1f1..3c8402be41df8 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -25,6 +25,7 @@ PUT my_index -------------------------------------------------- To make sure if any of the date formats is deprecated run deprecation API or kibana upgrade assistant. +//todo fix this xx migration-api-deprecation xx xxinclude ::../apis/deprecation.asciidoc @@ -41,83 +42,98 @@ If a pattern contains literals below, that means the pattern is incompatible and ===== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is a big change because in 7.0 'Y' means week-based-year. If you don't update your date pattern, you will have problems with search results, parsing and date calculations. example: -YYYY-MM-dd should become yyyy-MM-dd +`YYYY-MM-dd` should become `yyyy-MM-dd` ====== The difference between year-of-era and week based year -For pattern YYYY-ww and date 2019-01-01T00:00:00.000Z will give 2019-01 -For pattern YYYY-ww and date 2018-12-31T00:00:00.000Z will give 2019-01 (counter intuitive) because there is >4 days of that week in 2019 +For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z` will give `2019-01` +For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter intuitive) because there is >4 days of that week in 2019 ===== y - in 6.8 meaning year. It should be replaced to 'u' in 7.0. This is a subtle change, but worth fixing. -As previously stated 'y' means year-of-era. 'u' means year. The difference is that year can contain non positive values, whereas 'y' can not. //clarify 0 year +As previously stated `y` means year-of-era. `u` means year. The difference is that year can contain non positive values, whereas `y` can not. //clarify 0 year year-of-era can also be associated with an era field. ===== C - century of era is no longer supported in 7.0. There is no replacement, so you need to preprocess your input. https://stackoverflow.com/questions/38354151/how-to-force-java-time-localdate-to-assume-19th-century-as-yy-year-pattern ===== x - in 6.8 meaning week year. It should be replaced with 'Y' in 7.0. Again a big change as 'x' means zone-offset in 7.0. -You will get parsing, search problems if you won't update. +You will get parsing and search problems if you do not update this. ===== Z - in 6.8 meaning time zone offset/id. It has similar meaning in 7.0 with a notable difference that it expects different number of literals to parse different forms. -It also won't parse 'Z' for Zulu timezone. You should consider migrating to 'X' which give you more control of how your time is parsed. -For instance format YYYY-MM-dd'T'hh:mm:ssZZ -allowed all the forms below: +It also won't parse `Z` for Zulu timezone. You should consider migrating to `X` which give you more control of how your time is parsed. +For instance format `YYYY-MM-dd'T'hh:mm:ssZZ` allowed all the forms below: + +``` 2010-01-01T01:02:03Z 2010-01-01T01:02:03+01 2010-01-01T01:02:03+01:02 2010-01-01T01:02:03+01:02:03 +``` + You cannot parse them all with one pattern in 7.0. You need to specify 3 separate patterns +``` 2010-01-01T01:02:03Z 2010-01-01T01:02:03+01 -both parsed with yyyy-MM-dd'T'hh:mm:ssX //clarify Z +both parsed with yyyy-MM-dd'T'hh:mm:ssX 2010-01-01T01:02:03+01:02 yyyy-MM-dd'T'hh:mm:ssXXX 2010-01-01T01:02:03+01:02:03 yyyy-MM-dd'T'hh:mm:ssXXXXX +``` + If you expect all these combination to occur in your data you need to combine these patterns in 7.0 +[source,txt] +-------------------------------------------------- yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX +-------------------------------------------------- + The same applies if you expect your pattern to occur without a colon: -Pattern YYYY-MM-dd'T'hh:mm:ssZ accepting these dates: +Pattern `YYYY-MM-dd'T'hh:mm:ssZ` accepting these dates: +``` 2010-01-01T01:02:03Z 2010-01-01T01:02:03+01 2010-01-01T01:02:03+0102 2010-01-01T01:02:03+010203 +``` should be migrated to +[source,txt] +-------------------------------------------------- yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX +-------------------------------------------------- ===== d for day. This one have the same meaning in 7.0 but are less flexible. -If your pattern was YYYY-MM-dd in 6.8 and you expects dates in form -2010-01-01 or 2010-01-1 +If your pattern was `YYYY-MM-dd` in 6.8 and you expects dates in form +`2010-01-01` or `2010-01-1` You will need to provide an alternative pattern for all of these. -yyyy-MM-dd||yyyy-MM-d -More then 2 digits are not allowed anymore. If you want to parse dates like 2010-01-00001 you need to create a pattern that expects a text literal '000' before 'd' -yyyy-MM-'000'dd +`yyyy-MM-dd||yyyy-MM-d` +More then 2 digits are not allowed anymore. If you want to parse dates like `2010-01-00001` you need to create a pattern that expects a text literal `'000'` before `'d'` +`yyyy-MM-'000'dd` ===== e for name of day Also has a similar meaning in 7.0. However was more flexible in 6.8 and allowed to parse short and full text forms. -if you used EEE YYYY-MM for a format and expected days in form: +if you used `EEE YYYY-MM` for a format and expected days in form: Wed 2020-01 or Wednesday 2020-01 then you have to use two combined patterns in 7.0 -cccc yyyy-MM||ccc yyyy-MM +`cccc yyyy-MM||ccc yyyy-MM` E vs c. E means day-of-week vs c is for localized day-of-week. The difference is that E won't behave correctly for full text forms like Wednesday -//todo confirm -===== more on text forms EEEE and similar //todo +===== more on text forms EEEE and similar +Full text forms are depending on locale data provided with JDK and the implementation details of java vs joda. You should test carefully before upgrading these patterns. ===== 'z' 'z' time zone text. Will print 'Z' for Zulu given UTC timezone. -//todo expand + ===== Test with your data All of the examples above should be tested with real data. It is likely that you do not need the flexibility joda was giving. -If you expected your timezone to only be in +01:00 form (XXX in java, ZZ in joda), +If you expected your timezone to only be in `+01:00` form (XXX in java, ZZ in joda), then there is no need to create a new java pattern with so many alternatives. Consider using this date debugging site for assistance https://esddd.herokuapp.com/ @@ -150,7 +166,7 @@ GET /my_index_1/_mapping } } -------------------------------------------------- -todo make this test response?? +todo make this test response?? and others? * Create my_index_2 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis From fa9b6617e486493152e10e230104c481aa35fdc1 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Tue, 17 Mar 2020 17:40:58 +0100 Subject: [PATCH 09/81] test fixes --- .../migrate_7_0/migrate_to_java_time.asciidoc | 24 ++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 3c8402be41df8..e0892e3b422c7 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -146,6 +146,23 @@ If you specified a custom date format there, then you need to update it too. ===== Example migration procedure Let's assume that you have an index with a date field and custom format +//// +[source,console] +-------------------------------------------------- +PUT /my_index_1 +{ + "mappings" : { + "properties" : { + "datetime": { + "type": "date", + "format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis" + } + } + } +} +-------------------------------------------------- + + [source,console] -------------------------------------------------- GET /my_index_1/_mapping @@ -166,7 +183,7 @@ GET /my_index_1/_mapping } } -------------------------------------------------- -todo make this test response?? and others? +// NOTCONSOLE * Create my_index_2 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis @@ -185,7 +202,7 @@ PUT my_index_2 } } -------------------------------------------------- - +// TEST[continued] * Reindex the deprecated format index to new_index_1 @@ -201,6 +218,7 @@ POST _reindex } } -------------------------------------------------- +// TEST[continued] * If you were using aliases, update them to a new index @@ -214,7 +232,7 @@ POST /_aliases ] } -------------------------------------------------- - +// TEST[continued] ===== Update before upgrading to ES7. * update pipeline From d7f877b93e55a1153134aee2a989321e289d636c Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Tue, 17 Mar 2020 17:52:04 +0100 Subject: [PATCH 10/81] test fix --- .../migrate_7_0/migrate_to_java_time.asciidoc | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index e0892e3b422c7..85c07e72befe2 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -149,7 +149,7 @@ Let's assume that you have an index with a date field and custom format //// [source,console] -------------------------------------------------- -PUT /my_index_1 +PUT my_index_1 { "mappings" : { "properties" : { @@ -165,8 +165,9 @@ PUT /my_index_1 [source,console] -------------------------------------------------- -GET /my_index_1/_mapping +GET my_index_1/_mapping -------------------------------------------------- +// TEST[continued] [source,js] -------------------------------------------------- @@ -295,7 +296,13 @@ PUT _template/template_1 } -------------------------------------------------- +//// +[source,console] +-------------------------------------------------- +DELETE /_template/template_1 +-------------------------------------------------- +// TEST[continued] * Upgrade to 7.x ===== External templates From 01d73d62eed845d8b80cfbf50c75b7ac9d7b26a5 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:36:28 +0100 Subject: [PATCH 11/81] Update docs/reference/migration/migrate_7_0/java_time.asciidoc Co-Authored-By: James Rodewig --- docs/reference/migration/migrate_7_0/java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index 9cb57d1416a18..29d21576ef5f5 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -89,7 +89,7 @@ When you are running Elasticsearch 7 with Java 8, you are not able to parse the timezone `GMT0` properly anymore. The reason for this is a bug in the JDK, which has not been fixed for JDK8. You can read more in the https://bugs.openjdk.java.net/browse/JDK-8138664[official issue] -The fix is in JDK version 9 and newer. +This bug is fixed in JDK9 and later versions. [float] ==== Scripting with dates should use java time based methods From c2673dbffe40f7b1c0843b533c53f41b73456040 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:36:44 +0100 Subject: [PATCH 12/81] Update docs/reference/migration/migrate_7_0/java_time.asciidoc Co-Authored-By: James Rodewig --- docs/reference/migration/migrate_7_0/java_time.asciidoc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index 29d21576ef5f5..6e076af47d84b 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -130,8 +130,6 @@ For dates before 1970, use a date format containing a year. [float] ==== Migration Guide -You can find a detailed migration guide here - -* <> +For a detailed migration guide, see <>. include::migrate_to_java_time.asciidoc[] From 02eca6b263e7c92eaa4d3018778ebff8d2c26235 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:36:55 +0100 Subject: [PATCH 13/81] Update docs/reference/migration/migrate_7_0/java_time.asciidoc Co-Authored-By: James Rodewig --- docs/reference/migration/migrate_7_0/java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index 6e076af47d84b..9056f976d6b99 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -129,7 +129,7 @@ For dates before 1970, use a date format containing a year. [float] -==== Migration Guide +==== Migration guide For a detailed migration guide, see <>. include::migrate_to_java_time.asciidoc[] From e0aed7804665ff1b76de944e5749fb4bfe268531 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:37:34 +0100 Subject: [PATCH 14/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 85c07e72befe2..cacfc2a088a7c 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -1,5 +1,5 @@ [[migrate_to_java_time]] -=== Java Time Migration Guide +=== Java time migration guide Since version 7.0 Elasticsearch is using java.time as an underlying implementation of dates parsing, formatting and calculations. java.time is in many aspects very similar to joda-time which we previously used. However there are some differences and this guide is meant to From 2f695b5463b4a7e17bfd1d8e0f51464034e0564f Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:38:28 +0100 Subject: [PATCH 15/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index cacfc2a088a7c..346c3a7d307c1 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -1,9 +1,9 @@ [[migrate_to_java_time]] === Java time migration guide -Since version 7.0 Elasticsearch is using java.time as an underlying implementation of dates parsing, formatting and calculations. -java.time is in many aspects very similar to joda-time which we previously used. However there are some differences and this guide is meant to -help you to become compatible. +With 7.0, {es} switch to java time from joda time for date-related parsing, +formatting, and calculations. This guide is designed to help you determine +if your cluster is impacted and, if so, prepare for the upgrade. ==== Identifying if you are affected You are only affected if you have used any custom date formats in your mappings, pipelines or templates. From 3bd83f4697da234622395d9aab1d42e283f84964 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:39:17 +0100 Subject: [PATCH 16/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 346c3a7d307c1..3f875d2f1cc44 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -5,7 +5,7 @@ With 7.0, {es} switch to java time from joda time for date-related parsing, formatting, and calculations. This guide is designed to help you determine if your cluster is impacted and, if so, prepare for the upgrade. -==== Identifying if you are affected +==== Impacted features You are only affected if you have used any custom date formats in your mappings, pipelines or templates. Most of the custom date formats are compatible, but with some there is a change of meaning or missing. More on that below. A custom format is specified on a date field like this: From 88ad15b23872a7188867975f7f992f99b52121de Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:39:42 +0100 Subject: [PATCH 17/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migrate_7_0/migrate_to_java_time.asciidoc | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 3f875d2f1cc44..03b1c8f942258 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -6,7 +6,16 @@ formatting, and calculations. This guide is designed to help you determine if your cluster is impacted and, if so, prepare for the upgrade. ==== Impacted features -You are only affected if you have used any custom date formats in your mappings, pipelines or templates. +The switch to java time only impacts custom <> and +<> formats. + + These formats are commonly used in: + +* <> +* <> +* <> + +If you don't use custom date formats, you can skip the rest of this guide. Most of the custom date formats are compatible, but with some there is a change of meaning or missing. More on that below. A custom format is specified on a date field like this: [source,console] From 79c5ada7421284384c491ccb6bdf45d6c8d25de2 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:41:06 +0100 Subject: [PATCH 18/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 03b1c8f942258..cc827d665323f 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -46,7 +46,8 @@ This index has date fields with deprecated formats: [[type: doc, field: param2, ==== Incompatible formats -If a pattern contains literals below, that means the pattern is incompatible and needs to be changed. +Custom date formats containing the following joda time literals should be +changed before upgrading. ===== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is a big change because in 7.0 'Y' means week-based-year. If you don't update your date pattern, you will have problems with search results, parsing and date calculations. From 48bfbdf503496d4b8520f2ec9371a8b602c3f612 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:41:19 +0100 Subject: [PATCH 19/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index cc827d665323f..322d036eabd76 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -171,7 +171,7 @@ PUT my_index_1 } } -------------------------------------------------- - +//// [source,console] -------------------------------------------------- From 9c154838d2d339e3c727b1a224fbbb6dcc4dd61f Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:41:28 +0100 Subject: [PATCH 20/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 322d036eabd76..aba430f8355df 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -179,7 +179,7 @@ GET my_index_1/_mapping -------------------------------------------------- // TEST[continued] -[source,js] +[source,console-result] -------------------------------------------------- { "my_index_1" : { From c10e7c15b6122809e799604e4373cf3286033192 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:41:36 +0100 Subject: [PATCH 21/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index aba430f8355df..3e0cbc5357f68 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -306,7 +306,6 @@ PUT _template/template_1 } -------------------------------------------------- -//// [source,console] -------------------------------------------------- From cf6c0eff4488fdc08a2dc56bbfbf4f1127338f7f Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:41:44 +0100 Subject: [PATCH 22/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 3e0cbc5357f68..434feeb084103 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -197,7 +197,7 @@ GET my_index_1/_mapping // NOTCONSOLE -* Create my_index_2 with mappings changed with the format set to 8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis +* Create my_index_2 with mappings changed with the format set to `8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis` [source,console] -------------------------------------------------- From 5e85c5384bb8961a2c5146818f674fa41b29e6be Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:41:52 +0100 Subject: [PATCH 23/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 434feeb084103..9254dfd61d5e0 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -194,7 +194,6 @@ GET my_index_1/_mapping } } -------------------------------------------------- -// NOTCONSOLE * Create my_index_2 with mappings changed with the format set to `8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis` From 72478aba8c76eec8378d34db1041eb7095ae745e Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 07:42:19 +0100 Subject: [PATCH 24/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 9254dfd61d5e0..f84330f703c76 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -245,7 +245,6 @@ POST /_aliases // TEST[continued] ===== Update before upgrading to ES7. -* update pipeline If your pipelines were using a joda style patterns, they also have to be updated. There is no need to create a new pipeline. Just update the already existing one. From b51fea425b6ccf972df41a06b6b9151b2efb56a9 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 09:03:10 +0100 Subject: [PATCH 25/81] review follow up --- .../migrate_7_0/migrate_to_java_time.asciidoc | 144 +++++++++++------- 1 file changed, 89 insertions(+), 55 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index f84330f703c76..b1b899d50ddfe 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -5,11 +5,20 @@ With 7.0, {es} switch to java time from joda time for date-related parsing, formatting, and calculations. This guide is designed to help you determine if your cluster is impacted and, if so, prepare for the upgrade. +Joda style indices are supported in 6.8, but indices created with joda-style patterns in 6.x are also allowed to be used in 7.0 +All new indices created in 7.x are expected to be using java-time patterns. + +We advice to prepare for this migration in 6.8, but it will also work if you decide to perform it in version 7.x + +- In 6.8 you need to prefix your new java.style pattern with `8`. Patterns without `8` are considered joda style. +- In 7.x patterns without `8` on indices *created in 6.x* are considered joda-style. +- Patterns with or without `8` on indices *created in 7.x* are considered java.style + ==== Impacted features The switch to java time only impacts custom <> and <> formats. - These formats are commonly used in: +These formats are commonly used in: * <> * <> @@ -17,60 +26,69 @@ The switch to java time only impacts custom <> and If you don't use custom date formats, you can skip the rest of this guide. Most of the custom date formats are compatible, but with some there is a change of meaning or missing. More on that below. -A custom format is specified on a date field like this: -[source,console] --------------------------------------------------- -PUT my_index -{ - "mappings": { - "properties": { - "date": { - "type": "date", - "format": "YYYY/MM/dd HH:mm:ss||YYYY/MM/dd||epoch_millis" - } - } - } -} --------------------------------------------------- - -To make sure if any of the date formats is deprecated run deprecation API or kibana upgrade assistant. -//todo fix this -xx migration-api-deprecation xx -xxinclude ::../apis/deprecation.asciidoc -Sample output of deprecation api will be as follows: -[source,text] --------------------------------------------------- -This index has date fields with deprecated formats: [[type: doc, field: param2, format: yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis, suggestion: 'y' year should be replaced with 'u'. Use 'y' for year-of-era.]]. Prefix your date format with '8' to use the new specifier. --------------------------------------------------- +To see if your date format is impacted, use the <> +or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant] ==== Incompatible formats Custom date formats containing the following joda time literals should be changed before upgrading. -===== Y - in 6.8 meaning Year of era. It should be replaced to y in 7.0 This is a big change because in 7.0 'Y' means week-based-year. -If you don't update your date pattern, you will have problems with search results, parsing and date calculations. -example: +`Y` (Year of era):: ++ +-- +Replace with `y`. + +*Example:* `YYYY-MM-dd` should become `yyyy-MM-dd` -====== The difference between year-of-era and week based year +In java time, `Y` is used for +https://docs.oracle.com/javase/8/docs/api/java/time/temporal/WeekFields.html[week-based year]. +Using `Y` in place of `y` could result in off-by-one errors in year calculation. + + +The difference between year-of-era and week based year For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z` will give `2019-01` For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter intuitive) because there is >4 days of that week in 2019 +-- + +`y` year:: ++ +-- +Replace with `u`. + +*Example:* +`yyyy-MM-dd` should become `uuuu-MM-dd` + +The difference is that year can contain non positive values, whereas `y` can not. Also year-of-era can also be associated with an era field. +-- + + +`C` century of era:: ++ +-- +No longer supported in 7.0. There is no replacement, so you need to preprocess your input. + +A possible workaround https://stackoverflow.com/questions/38354151/how-to-force-java-time-localdate-to-assume-19th-century-as-yy-year-pattern +-- -===== y - in 6.8 meaning year. It should be replaced to 'u' in 7.0. This is a subtle change, but worth fixing. -As previously stated `y` means year-of-era. `u` means year. The difference is that year can contain non positive values, whereas `y` can not. //clarify 0 year -year-of-era can also be associated with an era field. +`x` week year:: ++ +-- +Replace with `Y` -===== C - century of era is no longer supported in 7.0. There is no replacement, so you need to preprocess your input. -https://stackoverflow.com/questions/38354151/how-to-force-java-time-localdate-to-assume-19th-century-as-yy-year-pattern +This is a big change. `x` means https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html[zone-offset] in java.time +-- -===== x - in 6.8 meaning week year. It should be replaced with 'Y' in 7.0. Again a big change as 'x' means zone-offset in 7.0. -You will get parsing and search problems if you do not update this. +`Z` Zone offset/id:: ++ +-- +Replace with multiple `X` -===== Z - in 6.8 meaning time zone offset/id. It has similar meaning in 7.0 with a notable difference that it expects different number of literals -to parse different forms. -It also won't parse `Z` for Zulu timezone. You should consider migrating to `X` which give you more control of how your time is parsed. +`Z` has similar meaning in 7.0 with a notable difference that it expects different number of literals to parse different forms. + +Consider migrating to `X` which give you more control of how your time is parsed. For instance format `YYYY-MM-dd'T'hh:mm:ssZZ` allowed all the forms below: ``` @@ -80,8 +98,8 @@ For instance format `YYYY-MM-dd'T'hh:mm:ssZZ` allowed all the forms below: 2010-01-01T01:02:03+01:02:03 ``` - You cannot parse them all with one pattern in 7.0. You need to specify 3 separate patterns + ``` 2010-01-01T01:02:03Z 2010-01-01T01:02:03+01 @@ -115,35 +133,51 @@ should be migrated to -------------------------------------------------- yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX -------------------------------------------------- +-- + -===== d for day. -This one have the same meaning in 7.0 but are less flexible. -If your pattern was `YYYY-MM-dd` in 6.8 and you expects dates in form -`2010-01-01` or `2010-01-1` -You will need to provide an alternative pattern for all of these. -`yyyy-MM-dd||yyyy-MM-d` + +`d` day:: ++ +-- +The same meaning in 7.0 but is less flexible. + +If your pattern was `YYYY-MM-dd` in 6.8 and you expect dates in form `2010-01-01` or `2010-01-1` +You need to provide an alternative pattern for all of these `yyyy-MM-dd||yyyy-MM-d`. More then 2 digits are not allowed anymore. If you want to parse dates like `2010-01-00001` you need to create a pattern that expects a text literal `'000'` before `'d'` `yyyy-MM-'000'dd` +-- + + +`e` name of day:: ++ +-- +The same meaning in 7.0. However it is less flexible in 7.0. It do not allow parsing short and full text forms. -===== e for name of day -Also has a similar meaning in 7.0. However was more flexible in 6.8 and allowed to parse short and full text forms. -if you used `EEE YYYY-MM` for a format and expected days in form: -Wed 2020-01 or Wednesday 2020-01 +If you used `EEE YYYY-MM` for a format and expected days in form: +`Wed 2020-01` or `Wednesday 2020-01` then you have to use two combined patterns in 7.0 `cccc yyyy-MM||ccc yyyy-MM` E vs c. E means day-of-week vs c is for localized day-of-week. The difference is that E won't behave correctly for full text forms like Wednesday +-- -===== more on text forms EEEE and similar +Text forms EEEE and similar:: ++ +-- Full text forms are depending on locale data provided with JDK and the implementation details of java vs joda. You should test carefully before upgrading these patterns. +-- -===== 'z' -'z' time zone text. Will print 'Z' for Zulu given UTC timezone. +`z` time zone text:: ++ +-- +In 7.0 it will print 'Z' for Zulu given UTC timezone. +-- ===== Test with your data All of the examples above should be tested with real data. It is likely that you do not need the flexibility joda was giving. -If you expected your timezone to only be in `+01:00` form (XXX in java, ZZ in joda), +If you expect your timezone to only be in one form - for instance `+01:00` (XXX in java, ZZ in joda) - then there is no need to create a new java pattern with so many alternatives. Consider using this date debugging site for assistance https://esddd.herokuapp.com/ From e0f99f49edb0b4e00d4ce5d6189dc34c237e7d98 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Wed, 18 Mar 2020 09:22:20 +0100 Subject: [PATCH 26/81] known issue --- docs/reference/release-notes/7.0.asciidoc | 9 +++++++++ docs/reference/release-notes/7.1.asciidoc | 8 ++++++++ docs/reference/release-notes/7.2.asciidoc | 8 ++++++++ docs/reference/release-notes/7.3.asciidoc | 11 +++++++++++ docs/reference/release-notes/7.4.asciidoc | 5 +++++ docs/reference/release-notes/7.5.asciidoc | 5 +++++ 6 files changed, 46 insertions(+) diff --git a/docs/reference/release-notes/7.0.asciidoc b/docs/reference/release-notes/7.0.asciidoc index 751c824207db1..2dffe3da4f564 100644 --- a/docs/reference/release-notes/7.0.asciidoc +++ b/docs/reference/release-notes/7.0.asciidoc @@ -6,6 +6,15 @@ releases of 7.0.0. Also see <>. +[float] +=== Known issues + +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + + [[breaking-7.0.0]] [float] === Breaking changes diff --git a/docs/reference/release-notes/7.1.asciidoc b/docs/reference/release-notes/7.1.asciidoc index ec93927513b85..5f9df16026363 100644 --- a/docs/reference/release-notes/7.1.asciidoc +++ b/docs/reference/release-notes/7.1.asciidoc @@ -3,6 +3,14 @@ Also see <>. +[float] +=== Known issues + +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + [[bug-7.1.1]] [float] === Bug fixes diff --git a/docs/reference/release-notes/7.2.asciidoc b/docs/reference/release-notes/7.2.asciidoc index 6f6c12ed65f7a..98252068cc51a 100644 --- a/docs/reference/release-notes/7.2.asciidoc +++ b/docs/reference/release-notes/7.2.asciidoc @@ -85,6 +85,14 @@ Security:: Also see <>. +[float] +=== Known issues + +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + [[breaking-7.2.0]] [float] === Breaking changes diff --git a/docs/reference/release-notes/7.3.asciidoc b/docs/reference/release-notes/7.3.asciidoc index 975f183514d75..0e7452202fe4e 100644 --- a/docs/reference/release-notes/7.3.asciidoc +++ b/docs/reference/release-notes/7.3.asciidoc @@ -125,6 +125,7 @@ Security:: * Use system context for looking up connected nodes {pull}43991[#43991] (issue: {issue}43974[#43974]) + [[upgrade-7.3.1]] [float] === Upgrades @@ -138,6 +139,16 @@ Infra/Packaging:: Also see <>. + +[float] +=== Known issues + +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + + [[breaking-7.3.0]] [float] === Breaking changes diff --git a/docs/reference/release-notes/7.4.asciidoc b/docs/reference/release-notes/7.4.asciidoc index 80550d53c8ee3..084d5224f94d2 100644 --- a/docs/reference/release-notes/7.4.asciidoc +++ b/docs/reference/release-notes/7.4.asciidoc @@ -116,6 +116,11 @@ Any attempt to log a slow search can throw an AIOOBE due to a bug that performs concurrent modifications on a shared byte array. (issue: {issue}/48358[#48358]) +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + [[breaking-7.4.0]] [float] === Breaking changes diff --git a/docs/reference/release-notes/7.5.asciidoc b/docs/reference/release-notes/7.5.asciidoc index 1faf181b92f24..5f115314cbe1d 100644 --- a/docs/reference/release-notes/7.5.asciidoc +++ b/docs/reference/release-notes/7.5.asciidoc @@ -98,6 +98,11 @@ Also see <>. If a {transform} is running during upgrade, the {transform} audit index might disappear. (issue: {issue}/49730[#49730]) +* Indices created in 6.x with <> and <> fields using formats +that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. +https://github.com/elastic/elasticsearch/pull/52555 +This is fixed in {es} 7.7 and later versions. + [[breaking-7.5.0]] [float] === Breaking changes From f38a28d772e62beba1effbc2896295d957b7e014 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:07:13 +0100 Subject: [PATCH 27/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index b1b899d50ddfe..35076a725fd7c 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -128,7 +128,7 @@ Pattern `YYYY-MM-dd'T'hh:mm:ssZ` accepting these dates: 2010-01-01T01:02:03+0102 2010-01-01T01:02:03+010203 ``` -should be migrated to +To accept all these forms in java time, you must use the `||` delimiter: [source,txt] -------------------------------------------------- yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX From e552aa6d12387ed5bde5498879875c5e72ecaedc Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:09:00 +0100 Subject: [PATCH 28/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 35076a725fd7c..6e6c297d4652a 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -135,8 +135,6 @@ yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX -------------------------------------------------- -- - - `d` day:: + -- From 89d6cf63e52c5346dd1e235690e23ace75696e4e Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:09:20 +0100 Subject: [PATCH 29/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 6e6c297d4652a..35d95fb46068f 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -119,7 +119,6 @@ If you expect all these combination to occur in your data you need to combine th yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX -------------------------------------------------- - The same applies if you expect your pattern to occur without a colon: Pattern `YYYY-MM-dd'T'hh:mm:ssZ` accepting these dates: ``` From 5ca94dcdb86619d2ea7db6b97cc757088fd8e7ab Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:09:36 +0100 Subject: [PATCH 30/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 35d95fb46068f..204ddff173842 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -134,7 +134,7 @@ yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX -------------------------------------------------- -- -`d` day:: +`d` (Day):: + -- The same meaning in 7.0 but is less flexible. From f6a3096d6cc3e38cdb82b488a12b609cd510fa70 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:11:09 +0100 Subject: [PATCH 31/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migrate_7_0/migrate_to_java_time.asciidoc | 24 +++++++++++++++---- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 204ddff173842..9fd2d7cabdab5 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -137,12 +137,26 @@ yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX `d` (Day):: + -- -The same meaning in 7.0 but is less flexible. +In java time, `d` is still interpreted as "day" but is less flexible. -If your pattern was `YYYY-MM-dd` in 6.8 and you expect dates in form `2010-01-01` or `2010-01-1` -You need to provide an alternative pattern for all of these `yyyy-MM-dd||yyyy-MM-d`. -More then 2 digits are not allowed anymore. If you want to parse dates like `2010-01-00001` you need to create a pattern that expects a text literal `'000'` before `'d'` -`yyyy-MM-'000'dd` +For example, the joda-time date format `YYYY-MM-dd` accepts `2010-01-01` or +`2010-01-1`. + +In java time, you must use the `||` delimiter to provide specify each format: + +[source,txt] +-------------------------------------------------- +yyyy-MM-dd||yyyy-MM-d +-------------------------------------------------- + +In java time, `d` also does not accept more than 2 digits. To accept days with more +than two digits, you must include a text literal in your java-time date format. +For example, to parse `2010-01-00001`, you must use the following java-time date format: + +[source,txt] +-------------------------------------------------- +yyyy-MM-'000'dd +-------------------------------------------------- -- From 170a75be62e874891d6ad6a78f2e745d069f80de Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:11:27 +0100 Subject: [PATCH 32/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 9fd2d7cabdab5..e634324014d57 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -159,7 +159,6 @@ yyyy-MM-'000'dd -------------------------------------------------- -- - `e` name of day:: + -- From 48d8188903eddce0ba8755cf411bf497250719f7 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:11:42 +0100 Subject: [PATCH 33/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index e634324014d57..3114eeaa3a7f7 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -159,7 +159,7 @@ yyyy-MM-'000'dd -------------------------------------------------- -- -`e` name of day:: +`e` (Name of day):: + -- The same meaning in 7.0. However it is less flexible in 7.0. It do not allow parsing short and full text forms. From d230fcd6cb57c220f8f1a7eddf1aec98688547dd Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:12:16 +0100 Subject: [PATCH 34/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migrate_7_0/migrate_to_java_time.asciidoc | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 3114eeaa3a7f7..1731d0b7fe5a6 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -162,14 +162,23 @@ yyyy-MM-'000'dd `e` (Name of day):: + -- -The same meaning in 7.0. However it is less flexible in 7.0. It do not allow parsing short and full text forms. +In java time, `e` is still interpreted as "name of day" but does not parse +short- or full-text forms. -If you used `EEE YYYY-MM` for a format and expected days in form: -`Wed 2020-01` or `Wednesday 2020-01` -then you have to use two combined patterns in 7.0 -`cccc yyyy-MM||ccc yyyy-MM` +For example, the joda-time date format `EEE YYYY-MM` accepts both +`Wed 2020-01` and `Wednesday 2020-01`. -E vs c. E means day-of-week vs c is for localized day-of-week. The difference is that E won't behave correctly for full text forms like Wednesday +To accept both of these dates in java time, you must specify each format using +the `||` delimiter: + +[source,txt] +-------------------------------------------------- +cccc yyyy-MM||ccc yyyy-MM +-------------------------------------------------- + +The joda-time literal `E` is interpreted as "day of week." +The java-time literal `c` is interpreted as "localized day of week." +`E` does not accept full-text day formats, such as `Wednesday`. -- Text forms EEEE and similar:: From a6fb10da60a3873e0f6f6edd360cc85e24ae7034 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:13:14 +0100 Subject: [PATCH 35/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 1731d0b7fe5a6..29b33473bce3c 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -184,7 +184,9 @@ The java-time literal `c` is interpreted as "localized day of week." Text forms EEEE and similar:: + -- -Full text forms are depending on locale data provided with JDK and the implementation details of java vs joda. You should test carefully before upgrading these patterns. +Support for full-text forms depends on the locale data provided with your Java +Development Kit (JDK) and other implementation details. We recommend you +test formats containing these patterns carefully before upgrading. -- `z` time zone text:: From 5dda7d070d0b6ddc7e6ecddd8a97fbbab1616109 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:13:26 +0100 Subject: [PATCH 36/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 29b33473bce3c..88144ba3964f2 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -189,7 +189,7 @@ Development Kit (JDK) and other implementation details. We recommend you test formats containing these patterns carefully before upgrading. -- -`z` time zone text:: +`z` (Time zone text):: + -- In 7.0 it will print 'Z' for Zulu given UTC timezone. From 6ecee5cc65d073b1c91006a09d7c38ccd4ed853e Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:13:48 +0100 Subject: [PATCH 37/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 88144ba3964f2..5a67fc73299c3 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -30,7 +30,7 @@ Most of the custom date formats are compatible, but with some there is a change To see if your date format is impacted, use the <> or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant] - +[[java-time-migration-incompatible-date-formats]] ==== Incompatible formats Custom date formats containing the following joda time literals should be changed before upgrading. From 83f36f7befa18de744c555c6ad67f580094e49cc Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:14:02 +0100 Subject: [PATCH 38/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 5a67fc73299c3..d5b6d32f5e78d 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -31,7 +31,7 @@ To see if your date format is impacted, use the < Date: Fri, 20 Mar 2020 11:14:26 +0100 Subject: [PATCH 39/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index d5b6d32f5e78d..2e3cfb7e16620 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -14,6 +14,7 @@ We advice to prepare for this migration in 6.8, but it will also work if you dec - In 7.x patterns without `8` on indices *created in 6.x* are considered joda-style. - Patterns with or without `8` on indices *created in 7.x* are considered java.style +[[java-time-migration-impacted-features]] ==== Impacted features The switch to java time only impacts custom <> and <> formats. From 0469410061b8bfbe3d8527a1a3a3276e013ce0f5 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:15:25 +0100 Subject: [PATCH 40/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 2e3cfb7e16620..a8d1f9d4a0ee9 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -196,7 +196,7 @@ test formats containing these patterns carefully before upgrading. In 7.0 it will print 'Z' for Zulu given UTC timezone. -- - +[[java-time-migration-test]] ===== Test with your data All of the examples above should be tested with real data. It is likely that you do not need the flexibility joda was giving. If you expect your timezone to only be in one form - for instance `+01:00` (XXX in java, ZZ in joda) - From ec542859de1b6e8816e538c095b887d72513ca40 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:36:32 +0100 Subject: [PATCH 41/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index a8d1f9d4a0ee9..5f1a955f8d69c 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -198,11 +198,12 @@ In 7.0 it will print 'Z' for Zulu given UTC timezone. [[java-time-migration-test]] ===== Test with your data -All of the examples above should be tested with real data. It is likely that you do not need the flexibility joda was giving. -If you expect your timezone to only be in one form - for instance `+01:00` (XXX in java, ZZ in joda) - -then there is no need to create a new java pattern with so many alternatives. -Consider using this date debugging site for assistance https://esddd.herokuapp.com/ +We strongly recommend you test any date format changes using real data before +deploying in production. + +For help with date debugging, consider using +https://esddd.herokuapp.com/[https://esddd.herokuapp.com/.] ==== Migrating affected mappings Once an index is created with a mapping, it cannot change already existing fields. You need to create a new index with updated mapping From 2967f132169c8acb7d871a311931822515f82114 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:36:44 +0100 Subject: [PATCH 42/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 5f1a955f8d69c..629e38b2aeaf5 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -205,7 +205,8 @@ deploying in production. For help with date debugging, consider using https://esddd.herokuapp.com/[https://esddd.herokuapp.com/.] -==== Migrating affected mappings +[[java-time-migrate-update-mappings]] +==== Update index mappings Once an index is created with a mapping, it cannot change already existing fields. You need to create a new index with updated mapping and reindex your data to it. You can however update your pipelines or templates. Remember to look for templates you use with tools outside ES. From e0c325297a8df215c7a8e066616da835b2611da6 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:38:02 +0100 Subject: [PATCH 43/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 629e38b2aeaf5..5885138d777e5 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -213,7 +213,8 @@ You can however update your pipelines or templates. Remember to look for templat If you specified a custom date format there, then you need to update it too. ===== Example migration procedure -Let's assume that you have an index with a date field and custom format +The following `my_index_1` index contains a mapping for the `datetime` field, a +`date` field with a custom joda-time date format. //// [source,console] -------------------------------------------------- From 552f4cb2a8b36b814d9dafb9842be6015bcaff25 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:38:31 +0100 Subject: [PATCH 44/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 5885138d777e5..51189d0346386 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -255,7 +255,12 @@ GET my_index_1/_mapping -------------------------------------------------- -* Create my_index_2 with mappings changed with the format set to `8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis` +To change the date format for the `datetime` field, create a separate index +containing an updated mapping and date format. + +For example, the following `my_index_2` index changes the `datetime` field's +date format to `8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis`. The `8` prefix +indicates this date format uses java time. [source,console] -------------------------------------------------- From b9860708011aadfdc9b922094b7515c50609c6b3 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:38:49 +0100 Subject: [PATCH 45/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 51189d0346386..30f413f5552f6 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -294,7 +294,7 @@ POST _reindex -------------------------------------------------- // TEST[continued] -* If you were using aliases, update them to a new index +If you use index aliases, update them to point to the new index. [source,console] -------------------------------------------------- From 6d7e2c243be9159e516a2d1cd259a6f24c753f16 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:39:48 +0100 Subject: [PATCH 46/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 30f413f5552f6..91cf337236757 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -309,8 +309,8 @@ POST /_aliases // TEST[continued] ===== Update before upgrading to ES7. -If your pipelines were using a joda style patterns, they also have to be updated. There is no need to create a new pipeline. -Just update the already existing one. +If your ingest pipelines contain joda-time date formats, you can update them +using the <> API. [source,console] -------------------------------------------------- From 0e99c3cebf30d5eda10040b01ababbe0c5061534 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:40:13 +0100 Subject: [PATCH 47/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 91cf337236757..46e1fb585002b 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -368,13 +368,13 @@ PUT _template/template_1 } -------------------------------------------------- - +//// [source,console] -------------------------------------------------- DELETE /_template/template_1 -------------------------------------------------- // TEST[continued] -* Upgrade to 7.x +//// ===== External templates Revisit other templates from elastic stack where you used a custom date pattern. From 49bb8a408cdf8da99e20770d2d314f7f951825cc Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:40:27 +0100 Subject: [PATCH 48/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 46e1fb585002b..cceb3daacc1fa 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -212,7 +212,6 @@ and reindex your data to it. You can however update your pipelines or templates. Remember to look for templates you use with tools outside ES. If you specified a custom date format there, then you need to update it too. -===== Example migration procedure The following `my_index_1` index contains a mapping for the `datetime` field, a `date` field with a custom joda-time date format. //// From 4b231f76aef78c3631be08ff51ae91fe25c5a91c Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:40:49 +0100 Subject: [PATCH 49/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index cceb3daacc1fa..0ec5283937d54 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -375,5 +375,8 @@ DELETE /_template/template_1 // TEST[continued] //// -===== External templates -Revisit other templates from elastic stack where you used a custom date pattern. +[[java-time-migration-update-external-tools-templates]] +===== Update external tools and templates +Ensure you also update any date formats in templates or tools outside of {es}. +This can include tools such as {beats-ref}/getting-started.html[{beats}] or +{logstash-ref}/index.html[Logstash]. From 232d6b30a5d8abcf2501e1dfc84c871c33e87b24 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:41:01 +0100 Subject: [PATCH 50/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 0ec5283937d54..c57dd46471b94 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -336,8 +336,11 @@ PUT _ingest/pipeline/mypipeline -------------------------------------------------- -* upgrade template -If your template was using joda date pattern it also should be updated before upgrading to ES7. +[[java-time-migration-update-index-templates]] +===== Update index templates + +If your index templates contain joda-time date formats, you can update them +using the <> API. [source,console] -------------------------------------------------- From 92690fd15a355e4f14cbbf9ce7c6ebe50b2295da Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:55:31 +0100 Subject: [PATCH 51/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index c57dd46471b94..945eea804bf81 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -1,7 +1,7 @@ [[migrate_to_java_time]] === Java time migration guide -With 7.0, {es} switch to java time from joda time for date-related parsing, +With 7.0, {es} switched from joda time to java time for date-related parsing, formatting, and calculations. This guide is designed to help you determine if your cluster is impacted and, if so, prepare for the upgrade. From a482a7146e43d9c0f945dd7416e6d9ac1224d57e Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 11:56:51 +0100 Subject: [PATCH 52/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 945eea804bf81..14b1945b0d14e 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -1,4 +1,4 @@ -[[migrate_to_java_time]] +[[migrate-to-java-time]] === Java time migration guide With 7.0, {es} switched from joda time to java time for date-related parsing, From 78acb559e5ccdb08232b24c0bcc7d416dc0ae87e Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 12:01:43 +0100 Subject: [PATCH 53/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migrate_7_0/migrate_to_java_time.asciidoc | 27 ++++++++++++++----- 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 14b1945b0d14e..86df7578ab9e7 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -5,14 +5,29 @@ With 7.0, {es} switched from joda time to java time for date-related parsing, formatting, and calculations. This guide is designed to help you determine if your cluster is impacted and, if so, prepare for the upgrade. -Joda style indices are supported in 6.8, but indices created with joda-style patterns in 6.x are also allowed to be used in 7.0 -All new indices created in 7.x are expected to be using java-time patterns. +[[java-time-convert-date-formats]] +=== Convert date formats -We advice to prepare for this migration in 6.8, but it will also work if you decide to perform it in version 7.x +To upgrade to 7.0, you'll need to convert any joda-time date formats +to their java-time equivalents. -- In 6.8 you need to prefix your new java.style pattern with `8`. Patterns without `8` are considered joda style. -- In 7.x patterns without `8` on indices *created in 6.x* are considered joda-style. -- Patterns with or without `8` on indices *created in 7.x* are considered java.style +To help track this effort, you can prefix java-time date formats with an `8` +prefix in {es} 6.8 and later versions. + +For example, you can change the date format `YYYY-MM-dd` to `8yyyy-MM-dd` to +indicate the date format uses java time. + +{es} treats date formats starting with the `8` prefix differently depending on +the version: + +*6.8*: Date formats with an `8` prefix are handled as java-time formats. Date +formats without an `8` prefix are treated as joda-time formats. We recommend +converting these joda-time formats to java-time _before_ upgrading to 7.x. + +*7.x and later*: For indices created in 6.x, date formats without an `8` prefix +are treated as joda-time formats. For indices created in 7.x and later versions, +all date formats are treated as java-time formats, regardless of whether it +starts with an `8` prefix. [[java-time-migration-impacted-features]] ==== Impacted features From cd0c386b3d166c1c4e71863695adcddf849db9f0 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 12:21:57 +0100 Subject: [PATCH 54/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 86df7578ab9e7..c26ccbc952e65 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -41,7 +41,8 @@ These formats are commonly used in: * <> If you don't use custom date formats, you can skip the rest of this guide. -Most of the custom date formats are compatible, but with some there is a change of meaning or missing. More on that below. +Most custom date formats are compatible. However, several require +an update. To see if your date format is impacted, use the <> or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant] From 9229af272f920507d990b420904fc8657efb1089 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 12:22:10 +0100 Subject: [PATCH 55/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index c26ccbc952e65..4cf65d81e9ee1 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -58,7 +58,7 @@ changed before upgrading. Replace with `y`. *Example:* -`YYYY-MM-dd` should become `yyyy-MM-dd` +`YYYY-MM-dd` should become `yyyy-MM-dd`. In java time, `Y` is used for https://docs.oracle.com/javase/8/docs/api/java/time/temporal/WeekFields.html[week-based year]. From d7fb5ab91993d558ccb1647adffe0179ee7a693c Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 12:22:33 +0100 Subject: [PATCH 56/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 4cf65d81e9ee1..484ca3d51d9da 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -49,7 +49,7 @@ or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant] [[java-time-migration-incompatible-date-formats]] ==== Incompatible date formats -Custom date formats containing the following joda time literals should be +Custom date formats containing the following joda-time literals should be changed before upgrading. `Y` (Year of era):: From 22ea0f5aeb7dc41e7f76fee51a2c9ee7a9deac3c Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 12:23:31 +0100 Subject: [PATCH 57/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 484ca3d51d9da..dfb426345e1c9 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -65,7 +65,6 @@ https://docs.oracle.com/javase/8/docs/api/java/time/temporal/WeekFields.html[wee Using `Y` in place of `y` could result in off-by-one errors in year calculation. -The difference between year-of-era and week based year For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z` will give `2019-01` For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter intuitive) because there is >4 days of that week in 2019 -- From 7f7881a5580300913b531f06e3630f7c3e8497cb Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:03:13 +0100 Subject: [PATCH 58/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index dfb426345e1c9..71e13d9d7b2c7 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -69,7 +69,7 @@ For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z` will give `2019-01` For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter intuitive) because there is >4 days of that week in 2019 -- -`y` year:: +`y` (Year):: + -- Replace with `u`. From fc42904a60fd6426bdf5f6b41121742cc33e11a6 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:03:27 +0100 Subject: [PATCH 59/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 71e13d9d7b2c7..2e6cebd8eca8e 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -75,7 +75,7 @@ For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` ( Replace with `u`. *Example:* -`yyyy-MM-dd` should become `uuuu-MM-dd` +`yyyy-MM-dd` should become `uuuu-MM-dd`. The difference is that year can contain non positive values, whereas `y` can not. Also year-of-era can also be associated with an era field. -- From 8d64851bf7733f551b5ae0ca56b63d79c44323ad Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:04:41 +0100 Subject: [PATCH 60/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 2e6cebd8eca8e..1e0b557d144d1 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -77,7 +77,8 @@ Replace with `u`. *Example:* `yyyy-MM-dd` should become `uuuu-MM-dd`. -The difference is that year can contain non positive values, whereas `y` can not. Also year-of-era can also be associated with an era field. +In java time, `y` is used for year of era. `u` can contain non-positive +values, while `y` cannot. `y` can also be associated with an era field. -- From 268a8946a3d9f41792f66048d5a2cf831146507e Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:04:53 +0100 Subject: [PATCH 61/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 1e0b557d144d1..10a42d3391dc3 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -82,7 +82,7 @@ values, while `y` cannot. `y` can also be associated with an era field. -- -`C` century of era:: +`C` (Century of era):: + -- No longer supported in 7.0. There is no replacement, so you need to preprocess your input. From 08a674851f0046aec198d520b0804b8fa7c6752c Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:05:16 +0100 Subject: [PATCH 62/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 10a42d3391dc3..44791a97f67cd 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -90,7 +90,7 @@ No longer supported in 7.0. There is no replacement, so you need to preprocess y A possible workaround https://stackoverflow.com/questions/38354151/how-to-force-java-time-localdate-to-assume-19th-century-as-yy-year-pattern -- -`x` week year:: +`x` (Week year):: + -- Replace with `Y` From 314d68f2dd85daedae603540d2159ab023240386 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:06:03 +0100 Subject: [PATCH 63/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 44791a97f67cd..8767537328f43 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -93,7 +93,7 @@ A possible workaround https://stackoverflow.com/questions/38354151/how-to-force- `x` (Week year):: + -- -Replace with `Y` +Replace with `Y`. This is a big change. `x` means https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html[zone-offset] in java.time -- From 183725bd21f6ec3f247b99d8906b527f64197a51 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:13:25 +0100 Subject: [PATCH 64/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 8767537328f43..36d72594c4187 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -95,7 +95,12 @@ A possible workaround https://stackoverflow.com/questions/38354151/how-to-force- -- Replace with `Y`. -This is a big change. `x` means https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html[zone-offset] in java.time +In java time, `x` means https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html[zone-offset]. + +[WARNING] +==== +Failure to properly convert `x` (Week year) to `Y` could result in data loss. +==== -- `Z` Zone offset/id:: From 8074844089ac5811225f6fb25611a71e4eda3bee Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:13:45 +0100 Subject: [PATCH 65/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 36d72594c4187..d40cff36fb104 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -103,7 +103,7 @@ Failure to properly convert `x` (Week year) to `Y` could result in data loss. ==== -- -`Z` Zone offset/id:: +`Z` (Zone offset/id):: + -- Replace with multiple `X` From 864a4981ea16a1e496b2ebbc8125eefe19eaac28 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:15:34 +0100 Subject: [PATCH 66/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index d40cff36fb104..5629268f45fc5 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -85,9 +85,8 @@ values, while `y` cannot. `y` can also be associated with an era field. `C` (Century of era):: + -- -No longer supported in 7.0. There is no replacement, so you need to preprocess your input. - -A possible workaround https://stackoverflow.com/questions/38354151/how-to-force-java-time-localdate-to-assume-19th-century-as-yy-year-pattern +Century of era is not supported in java time. +There is no replacement. Instead, we recommend you preprocess your input. -- `x` (Week year):: From a4f7c239b1d69bec81d6e531b9b65a844bedffbf Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:15:53 +0100 Subject: [PATCH 67/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 5629268f45fc5..9f152cc9dacbe 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -105,7 +105,7 @@ Failure to properly convert `x` (Week year) to `Y` could result in data loss. `Z` (Zone offset/id):: + -- -Replace with multiple `X` +Replace with multiple `X`'s. `Z` has similar meaning in 7.0 with a notable difference that it expects different number of literals to parse different forms. From 300e800eb2df034902d3ba74550e454aadebe709 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:16:16 +0100 Subject: [PATCH 68/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 9f152cc9dacbe..fb8e3d07d7989 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -107,7 +107,8 @@ Failure to properly convert `x` (Week year) to `Y` could result in data loss. -- Replace with multiple `X`'s. -`Z` has similar meaning in 7.0 with a notable difference that it expects different number of literals to parse different forms. +`Z` has s similar meaning in java time. However, java time expects different +numbers of literals to parse different forms. Consider migrating to `X` which give you more control of how your time is parsed. For instance format `YYYY-MM-dd'T'hh:mm:ssZZ` allowed all the forms below: From 23af7e0e02e3ddc5e72cda5a743c47ccdb9ab89f Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:16:42 +0100 Subject: [PATCH 69/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index fb8e3d07d7989..fdd322e8b3b39 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -110,8 +110,8 @@ Replace with multiple `X`'s. `Z` has s similar meaning in java time. However, java time expects different numbers of literals to parse different forms. -Consider migrating to `X` which give you more control of how your time is parsed. -For instance format `YYYY-MM-dd'T'hh:mm:ssZZ` allowed all the forms below: +Consider migrating to `X`, which gives you more control over how time is parsed. +For example, the joda-time format `YYYY-MM-dd'T'hh:mm:ssZZ` accepts the following dates: ``` 2010-01-01T01:02:03Z From e84971a088eb10c2354e221266130748d8c48e37 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:16:55 +0100 Subject: [PATCH 70/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index fdd322e8b3b39..50071aa13add2 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -120,7 +120,8 @@ For example, the joda-time format `YYYY-MM-dd'T'hh:mm:ssZZ` accepts the followin 2010-01-01T01:02:03+01:02:03 ``` -You cannot parse them all with one pattern in 7.0. You need to specify 3 separate patterns +In java time, you cannot parse all these dates using a single format +Instead, you must specify 3 separate formats: ``` 2010-01-01T01:02:03Z From 1efee6712c7218de764810693b129af03aa272af Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:17:21 +0100 Subject: [PATCH 71/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 50071aa13add2..133492a22444e 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -142,7 +142,7 @@ If you expect all these combination to occur in your data you need to combine th yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX -------------------------------------------------- -The same applies if you expect your pattern to occur without a colon: +The same applies if you expect your pattern to occur without a colon (`:`): Pattern `YYYY-MM-dd'T'hh:mm:ssZ` accepting these dates: ``` 2010-01-01T01:02:03Z From ba82ebb211706fd3c83773646b882564968464f5 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:22:50 +0100 Subject: [PATCH 72/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 133492a22444e..ac34920d69e33 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -136,7 +136,7 @@ yyyy-MM-dd'T'hh:mm:ssXXXXX ``` -If you expect all these combination to occur in your data you need to combine these patterns in 7.0 +The formats must then be delimited using `||`: [source,txt] -------------------------------------------------- yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX From 4abeabf8f64c3a5069972233d272fe899d088ff5 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:23:11 +0100 Subject: [PATCH 73/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index ac34920d69e33..9916f423f664b 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -143,7 +143,7 @@ yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX -------------------------------------------------- The same applies if you expect your pattern to occur without a colon (`:`): -Pattern `YYYY-MM-dd'T'hh:mm:ssZ` accepting these dates: +For example, the `YYYY-MM-dd'T'hh:mm:ssZ` format accepts the following date forms: ``` 2010-01-01T01:02:03Z 2010-01-01T01:02:03+01 From 1b389eac203d099d0fd9351ddc1bbb2860ecf473 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:23:24 +0100 Subject: [PATCH 74/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 9916f423f664b..f391b6776405d 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -204,7 +204,7 @@ The java-time literal `c` is interpreted as "localized day of week." `E` does not accept full-text day formats, such as `Wednesday`. -- -Text forms EEEE and similar:: +`EEEE` and similar text forms:: + -- Support for full-text forms depends on the locale data provided with your Java From bad6adfb336d67bf5a1cd21ceceec49d702d76b2 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:24:02 +0100 Subject: [PATCH 75/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index f391b6776405d..43e1f56def71c 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -215,7 +215,7 @@ test formats containing these patterns carefully before upgrading. `z` (Time zone text):: + -- -In 7.0 it will print 'Z' for Zulu given UTC timezone. +In java time, `z` outputs 'Z' for Zulu when given a UTC timezone. -- [[java-time-migration-test]] From 70bc73d9620c4691b750a85d0b68519be3f5338d Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:24:17 +0100 Subject: [PATCH 76/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 43e1f56def71c..526039fba0786 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -299,7 +299,7 @@ PUT my_index_2 -------------------------------------------------- // TEST[continued] -* Reindex the deprecated format index to new_index_1 +Next, reindex the data from `my_index_1` to `my_index_2`. [source,console] -------------------------------------------------- From df97a0d8565ae9a307cf2804973c7c194ecd94a0 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:24:52 +0100 Subject: [PATCH 77/81] Update docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 526039fba0786..19a9a38840445 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -329,7 +329,8 @@ POST /_aliases -------------------------------------------------- // TEST[continued] -===== Update before upgrading to ES7. +[[java-time-migration-update-ingest-pipelines]] +===== Update ingest pipelines If your ingest pipelines contain joda-time date formats, you can update them using the <> API. From 52690b930385e1ae71343aa0bba8ae3894138c7c Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 13:29:29 +0100 Subject: [PATCH 78/81] rephrase update index mappings section --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 19a9a38840445..2b65830884b01 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -229,8 +229,8 @@ https://esddd.herokuapp.com/[https://esddd.herokuapp.com/.] [[java-time-migrate-update-mappings]] ==== Update index mappings -Once an index is created with a mapping, it cannot change already existing fields. You need to create a new index with updated mapping -and reindex your data to it. +To update joda-time date formats in index mappings, you must create a new index +with an updated mapping and reindex your data to it. You can however update your pipelines or templates. Remember to look for templates you use with tools outside ES. If you specified a custom date format there, then you need to update it too. From 56527142022d13789c2e0c8c79fd5687bbbcba4c Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 15:46:58 +0100 Subject: [PATCH 79/81] Apply suggestions from code review Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/java_time.asciidoc | 2 +- .../migrate_7_0/migrate_to_java_time.asciidoc | 17 ++++++++++------- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/java_time.asciidoc b/docs/reference/migration/migrate_7_0/java_time.asciidoc index 9056f976d6b99..8e1e2178e9e72 100644 --- a/docs/reference/migration/migrate_7_0/java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/java_time.asciidoc @@ -130,6 +130,6 @@ For dates before 1970, use a date format containing a year. [float] ==== Migration guide -For a detailed migration guide, see <>. +For a detailed migration guide, see <>. include::migrate_to_java_time.asciidoc[] diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 2b65830884b01..d6d3036637a00 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -5,6 +5,7 @@ With 7.0, {es} switched from joda time to java time for date-related parsing, formatting, and calculations. This guide is designed to help you determine if your cluster is impacted and, if so, prepare for the upgrade. +[discrete] [[java-time-convert-date-formats]] === Convert date formats @@ -12,7 +13,7 @@ To upgrade to 7.0, you'll need to convert any joda-time date formats to their java-time equivalents. To help track this effort, you can prefix java-time date formats with an `8` -prefix in {es} 6.8 and later versions. + in {es} 6.8 and later versions. For example, you can change the date format `YYYY-MM-dd` to `8yyyy-MM-dd` to indicate the date format uses java time. @@ -45,7 +46,7 @@ Most custom date formats are compatible. However, several require an update. To see if your date format is impacted, use the <> -or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant] +or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant]. [[java-time-migration-incompatible-date-formats]] ==== Incompatible date formats @@ -64,7 +65,6 @@ In java time, `Y` is used for https://docs.oracle.com/javase/8/docs/api/java/time/temporal/WeekFields.html[week-based year]. Using `Y` in place of `y` could result in off-by-one errors in year calculation. - For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z` will give `2019-01` For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter intuitive) because there is >4 days of that week in 2019 -- @@ -78,7 +78,7 @@ Replace with `u`. `yyyy-MM-dd` should become `uuuu-MM-dd`. In java time, `y` is used for year of era. `u` can contain non-positive -values, while `y` cannot. `y` can also be associated with an era field. +values while `y` cannot. `y` can also be associated with an era field. -- @@ -107,7 +107,7 @@ Failure to properly convert `x` (Week year) to `Y` could result in data loss. -- Replace with multiple `X`'s. -`Z` has s similar meaning in java time. However, java time expects different +`Z` has a similar meaning in java time. However, java time expects different numbers of literals to parse different forms. Consider migrating to `X`, which gives you more control over how time is parsed. @@ -299,7 +299,10 @@ PUT my_index_2 -------------------------------------------------- // TEST[continued] -Next, reindex the data from `my_index_1` to `my_index_2`. +Next, reindex data from the old index to the new index. + +The following <> API request reindexes data from +`my_index_1` to `my_index_2`. [source,console] -------------------------------------------------- @@ -336,7 +339,7 @@ using the <> API. [source,console] -------------------------------------------------- -PUT _ingest/pipeline/mypipeline +PUT _ingest/pipeline/my_pipeline { "description": "Pipeline for routing data to specific index", "processors": [ From 533bd1b9ee95c16ac0bd9f0572feb226930c1a53 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 15:50:49 +0100 Subject: [PATCH 80/81] remove the part of external pipelines form update index section --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 2b65830884b01..00a158cf07903 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -231,8 +231,7 @@ https://esddd.herokuapp.com/[https://esddd.herokuapp.com/.] ==== Update index mappings To update joda-time date formats in index mappings, you must create a new index with an updated mapping and reindex your data to it. -You can however update your pipelines or templates. Remember to look for templates you use with tools outside ES. -If you specified a custom date format there, then you need to update it too. +You can however update your pipelines or templates. The following `my_index_1` index contains a mapping for the `datetime` field, a `date` field with a custom joda-time date format. From ea1e0f220e9089b0d3cfcb13ec0a41e6784ac3e2 Mon Sep 17 00:00:00 2001 From: Przemyslaw Gomulka Date: Fri, 20 Mar 2020 15:54:47 +0100 Subject: [PATCH 81/81] Apply suggestions from code review Co-Authored-By: James Rodewig --- .../migration/migrate_7_0/migrate_to_java_time.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc index 5f7fb892caebd..2eab8e2de7f28 100644 --- a/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc +++ b/docs/reference/migration/migrate_7_0/migrate_to_java_time.asciidoc @@ -66,7 +66,7 @@ https://docs.oracle.com/javase/8/docs/api/java/time/temporal/WeekFields.html[wee Using `Y` in place of `y` could result in off-by-one errors in year calculation. For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z` will give `2019-01` -For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter intuitive) because there is >4 days of that week in 2019 +For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter-intuitive) because there is >4 days of that week in 2019 -- `y` (Year)::