From e55f02f3ed47b7871a86fa1e3dfdc99551f61e8f Mon Sep 17 00:00:00 2001 From: debadair Date: Tue, 26 Mar 2019 11:01:46 -0700 Subject: [PATCH] 7.0 upgrade (#242) * [DOCS] First pass at 7.0 upgrade update. * [DOCS] Fixed formatting issues & flagged the Upgrade Assistant as xpack. * Update docs/en/install-upgrade/upgrading-stack.asciidoc Co-Authored-By: debadair * [DOCS] Incorporated feedback from lcawl. * [DOCS] Fixes broken link * [DOCS] Removed tip about temporary superuser when upgrading the security index. --- .../install-upgrade/upgrading-stack.asciidoc | 295 ++++-------------- 1 file changed, 69 insertions(+), 226 deletions(-) diff --git a/docs/en/install-upgrade/upgrading-stack.asciidoc b/docs/en/install-upgrade/upgrading-stack.asciidoc index 095455f49..329237c29 100644 --- a/docs/en/install-upgrade/upgrading-stack.asciidoc +++ b/docs/en/install-upgrade/upgrading-stack.asciidoc @@ -1,254 +1,97 @@ [[upgrading-elastic-stack]] == Upgrading the Elastic Stack -When upgrading to a new version of Elasticsearch, you need to upgrade -each of the products in your Elastic Stack. Beats and Logstash 5.6 are -compatible with Elasticsearch {version} to give you flexibility in scheduling the -upgrade. +When upgrading to a new version of {es}, you need to upgrade +each of the products in your Elastic Stack. Beats and Logstash 6.7 are +compatible with {es} {version} to give you flexibility in scheduling +the upgrade. -**** -The steps you need to take to upgrade differ depending on which products you -are using. Want a list that's tailored to your stack? Try out our -{upgrade_guide}[Interactive Upgrade Guide]. -**** +{es} supports rolling upgrades between minor versions, from {es} 5.6 to 6.7, +and from 6.7 to {version}. -If you are running a pre-6.0 version, we recommend upgrading to the most -recent 5.6 before upgrading to {version}. {xpack} 5.6 -provides a free Upgrade Assistant that identifies issues you need to address -before upgrading and simplifies migrating indices that need to be reindexed -before you upgrade. The Upgrade Assistant is enabled with both Trial and -Basic licenses. You can install {xpack} solely for the purpose of upgrading. - -Rolling upgrades are supported when upgrading from Elasticsearch 5.6 and -Elasticsearch 6.0-6.2 to {version}. Upgrading from any -version prior to 5.6 requires a full cluster restart. - -IMPORTANT: 5.x and older indices are not compatible with {version}. You must -remove or reindex them on your 6.x cluster before upgrading to {version}. The internal -Kibana and {xpack} indices and the default Beats and Logstash mapping templates -also need to be updated to work with {version}. +IMPORTANT: 5.x indices are not compatible with {version}. You must +remove or reindex them to upgrade to {version}. The default Beats and +Logstash mapping templates also need to be updated to work with {version}. === Preparing to upgrade Before upgrading the Elastic Stack to {version}: -. Back up your data. You **cannot roll back** to an earlier version unless -you have a backup of your data. For information about creating snapshots, see -{ref}/modules-snapshots.html[Snapshot and Restore]. - -. Check the Elasticsearch {ref}/settings.html[deprecation log] to see if -you're using any deprecated features and update your code accordingly. -By default, deprecation log messages are enabled at the `WARN` level. - -. Review the breaking changes for each product you use -and make the necessary changes so your code is compatible with {version}. See -<>. +. Check the {es} {ref}/logging.html#deprecation-logging[deprecation log] +to see if you're using any deprecated features and update your code accordingly. +By default, deprecation warnings are logged when the log level is set to `WARN`. -. {ref}/docs-reindex.html[Reindex] or delete any indices created on 2.n. We recommend -upgrading to the most recent 5.6 and using the {xpack} Reindex Helper to reindex 2.n indices. +. Review the <> and upgrade your code to work +with {version}. -. If Kibana and {xpack} are part of your stack, upgrade the internal Kibana -and {xpack} indices. We recommend using the {xpack} 5.6 Reindex Helper to -upgrade the internal indices. If you're performing a full cluster restart upgrade -from an earlier version, you can also -<> directly to -upgrade the internal indices after you install Elasticsearch {version}. - -. If you use {stack} {security-features} to secure your cluster: -.. Make sure TLS is enabled to encrypt communications between nodes. TLS must -be enabled to upgrade to {version}. For more information, see -{stack-ov}/encrypting-communications.html[Encrypting communications]. +. Upgrade to 6.7 and use the {kibana-ref}/upgrade-assistant.html[{kib} Upgrade Assistant] to {ref}/docs-reindex.html[reindex] +any indices that are not compatible with {version}. + -NOTE: Enabling TLS requires a full cluster restart. Nodes that have TLS -enabled cannot communicate with nodes that do not have TLS enabled. You must -restart all nodes to maintain communication across the cluster. - -.. Make sure real passwords are configured for the built-in `elasticsearch`, -`kibana`, and `logstash_system` users. They cannot use the 5.n default -password (`changeme`). For more information, see -{stack-ov}/built-in-users.html[Built-in users]. - -. Consider closing {ml} jobs before you start the upgrade process. It is not -required, but there are pros and cons to leaving the jobs running. These -considerations are described in the steps related to -{ref}/setup-upgrade.html[upgrading {es}]. +[role="xpack"] +.Upgrade Assistant +****** +The Upgrade Assistant and migration APIs are enabled with both the Basic and +Trial licenses. You can install the default distribution of 6.7 to use the +Upgrade Assistant to prepare to upgrade even if you are upgrading to the OSS +distribution of {version}. +****** -IMPORTANT: Test upgrades in a dev environment before upgrading your -production cluster. +. Use the Upgrade Assistant to identify any changes you need to make to your +cluster configuration. [[upgrade-order-elastic-stack]] -=== Upgrade order - -Upgrade the Elastic Stack products you use in the following order: +=== Upgrade process -. Elasticsearch Hadoop: {hadoop-ref}/install.html[install instructions] -. Elasticsearch: {ref}/setup-upgrade.html[upgrade instructions] -. Kibana: {kibana-ref}/upgrade.html[upgrade instructions] -. Logstash: {logstash-ref}/upgrading-logstash.html[upgrade instructions] -. Beats: {beats-ref}/upgrading.html[upgrade instructions] -. APM Server: {apm-server-ref-70}/upgrading.html[upgrade instructions] - -NOTE: Logstash 5.6 and 6.n and Beats 5.6 and 6.n are compatible with all 6.n versions of -Elasticsearch. This provides flexibility in when you schedule the upgrades -for your Logstash instances and Beats agents. We recommend upgrading Logstash -and Beats as soon as possible to take advantage of performance improvements -and other enhancements. - -=== Upgrading to 6.3 -Starting in 6.3, the default distributions of {es}, {ls}, and {kib} -include {xpack} and a free Basic license that never expires. - -You can perform rolling upgrades to 6.3 from OSS-only clusters running 5.6 -or 6.0-6.2. Basic features are operational once the cluster is fully -upgraded. If you are already using {xpack}, your settings are preserved when -you upgrade. - -IMPORTANT: If you use {xpack} you must explicitly remove the old {xpack} plugin -before restarting: `bin/elasticsearch-plugin remove x-pack`. The node will fail -to start if the {xpack} plugin is present. - -You must explicitly enable data collection after the upgrade to use monitoring. -Set `xpack.monitoring.collection.enabled` to `true` with the `_cluster/settings` -API: - -[source,json] ----------------------------------------------------------- -PUT /_cluster/settings -{ - "persistent" : { - "xpack.monitoring.collection.enabled" : "true" - } -} ----------------------------------------------------------- -// CONSOLE - -To take more of the {stack} features for a spin, you can start a 30-day trial -from Kibana or with the Start Trial API: - -[source,json] ----------------------------------------------------------- -POST _license/start_trial ----------------------------------------------------------- -// CONSOLE - -The 30-day trial enables you to try out the full set of platinum features, -including security, machine learning, alerting, graph capabilities, and more. - -[role="xpack"] -[[xpack-stack-upgrade]] -=== Upgrading from 5.6 +When you've made the necessary changes and are ready to upgrade from 6.7 to +{version}: -{xpack} 5.6 provides migration and upgrade APIs for Elasticsearch and a -Upgrade Assistant UI for Kibana. These tools are included with the trial -license and the free basic license. - -To upgrade to {version} from 5.6: +. Test the upgrade in a dev environment *before* upgrading your +production cluster. -. {ref}/setup-upgrade.html[Upgrade Elasticsearch] to the most recent 5.6 and -install {xpack} on all nodes in your cluster. If you are upgrading from an -earlier 5.x release, you can perform a rolling upgrade. To upgrade from older -versions you must perform a full cluster restart. -+ -If your trial license expires, -https://register.elastic.co/[register for a free Basic license]. To apply the -license, upload the license file with the `license` API: -+ -[source,json] ----------------------------------------------------------- -license -d @license.json ----------------------------------------------------------- +. Back up your data. You **cannot roll back** to an earlier version unless +you have a snapshot of your data. For information about creating snapshots, see +{ref}/modules-snapshots.html[Snapshot and Restore]. -. If {xpack} **IS NOT** normally a part of your {stack}, disable the -{es} {security-features} in `elasticsearch.yml`: -+ -[source,yaml] ----------------------------------------------------------- -xpack.security.enabled: false ----------------------------------------------------------- +. Consider closing {ml} jobs before you start the upgrade process. While {ml} +jobs can continue to run during a rolling upgrade, it increases the overhead +on the cluster during the upgrade process. For more information, see +{ref}/rolling-upgrades.html[Rolling upgrades]. -. Upgrade {kib} to the most recent 5.6 and install {xpack}. +. Upgrade the components of your Elastic Stack in the following order: -. If you disabled the {es} {security-features}, also disable the {kib} -{security-features} in `kibana.yml`: -+ -[source,yaml] ----------------------------------------------------------- -xpack.security.enabled: false ----------------------------------------------------------- +.. {es} Hadoop: {hadoop-ref}/install.html[install instructions] +.. {es}: {ref}/setup-upgrade.html[upgrade instructions] +.. Kibana: {kibana-ref}/upgrade.html[upgrade instructions] +.. Logstash: {logstash-ref}/upgrading-logstash.html[upgrade instructions] +.. Beats: {beats-ref}/upgrading.html[upgrade instructions] +.. APM Server: {apm-server-ref}/upgrading.html[upgrade instructions] -. Use the Upgrade Assistant in Kibana to -view incompatibilities that you need to fix, identify any 2.x indices that -need to be migrated or deleted, and upgrade the internal indices to the -{major-version} index format. -+ -You can also call the Elasticsearch migration APIs directly: -+ -`/_xpack/migration/deprecations`:: Retrieves information about cluster-, node-, -and index-level settings that use deprecated features. -+ -`/_xpack/migration/assistance`:: deprecated[6.7.0] Returns a list of indices -that need to be reindexed before you can upgrade to {version}. -+ -`/_xpack/migration/upgrade`:: deprecated[6.7.0] Upgrades the indices for the -{watcher} and {security-features} to a single-type format compatible with -Elasticsearch 6.x. - -. Once you've resolved all of the migration issues, perform -a {ref}/rolling-upgrades.html[rolling upgrade] from Elasticsearch 5.6 to {version}. +NOTE: Logstash 6.7 and Beats 6.7 are compatible with all 7.x versions of +{es}. This provides flexibility in when you schedule the upgrades +for your Logstash instances and Beats agents, but we recommend upgrading as +soon as possible to take advantage of performance improvements +and other enhancements. [[oss-stack-upgrade]] -=== Upgrading from a pre-5.6 installation - -It is possible to upgrade directly to {major-version} from a pre-5.6 installation, -but it requires a {ref}/restart-upgrade.html[full cluster restart] and you must -manually reindex any 2.x indices you need to carry forward to {major-version}. - -IMPORTANT: If you use Kibana or {xpack}, you also need to upgrade the -internal Kibana and {xpack} indices. For information about upgrading them -after you install Elasticsearch {version}, see -<>. - -To manually reindex a 2.x index: - -. Create an index with 6.x compatible mappings. -. Use the {ref}/docs-reindex.html[reindex API] to copy documents from the -2.x index into the new index. You can use a script to perform any necessary -modifications to the document data and metadata during reindexing. -. Use the {ref}/indices-aliases.html[_aliases] API to add the name of the 2.x -index as alias for the new index and delete the 2.x index. - -[[upgrade-internal-indices]] -==== Upgrading internal indices for {major-version} - -The format used for the internal indices used by Kibana and {xpack} has -changed in {major-version}. Before you can run Kibana and {xpack} in {version}, -these indices must be upgraded to the new format. If you are upgrading from a -version prior to 5.6, you must upgrade them after after installing -Elasticsearch {version}. - -To get a list of the indices that need to be upgraded, use the -{ref}/migration-api-deprecation.html[deprecation info API]: - -[source,json] ----------------------------------------------------------- -GET /_xpack/migration/deprecations ----------------------------------------------------------- -// CONSOLE - -To upgrade the `.security` index: +=== Upgrading from 6.6 or earlier -. On a single node, add a temporary superuser account to the `file` realm. -. Use the {kib} Upgrade Assistant to upgrade the security index, submitting the -request with the credentials for the temporary superuser. Alternatively, you can -{ref}/reindex-upgrade.html[reindex manually]. +To upgrade directly to {es} {version} from versions 6.0-6.6, you must +{ref}/reindex-upgrade.html[manually reindex] any 5.x indices you need to +carry forward, and perform a {ref}/restart-upgrade.html[full cluster restart]. +This includes any internal indices created in 5.x, such as the `.kibana` and +`.security*` indices. -. Delete the temporary superuser account from the file realm. +Make sure all 5.x indices have been deleted before upgrading to {version}. {es} +{version} will fail to start if any 5.x indices are present. -You can use your regular administration credentials to upgrade the other -internal indices. +If you are running a version prior to 6.0, +https://www.elastic.co/guide/en/elastic-stack/6.7/upgrading-elastic-stack.html[upgrade to 6.7] +and reindex your old indices or bring up a new {version} cluster and +{ref}/reindex-upgrade-remote.html[reindex from remote]. -TIP: Once you upgrade the `.kibana` index, you can run Kibana and use the -{xpack} Reindex Helper UI to upgrade the other indices. +The recommended path is to upgrade to 6.7 before upgrading to {version}. This +makes it easier to identify the changes you need to make to upgrade and enables +you to perform a rolling upgrade with no downtime. [[upgrade-elastic-stack-for-elastic-cloud]] === Upgrading on Elastic Cloud @@ -256,17 +99,17 @@ TIP: Once you upgrade the `.kibana` index, you can run Kibana and use the A single click in the Elastic Cloud console can upgrade a cluster to a newer version, add more processing capacity, change plugins, and enable or disable high availability, all at the same time. During the upgrade process, -Elasticsearch, Kibana, {xpack} and the officially included plugins are +{es}, Kibana, {xpack} and the officially included plugins are upgraded simultaneously. Although upgrading your Elastic Cloud clusters is easy, you still need to address breaking changes that affect your application. Minor version upgrades, -upgrades from 5.6 to {major-version}, and all other cluster configuration +upgrades from 6.7 to {version}, and all other cluster configuration changes can be performed with no downtime. To avoid downtime when a full cluster restart is required: -. Provision an additional cluster with the new Elasticsearch version, reindex +. Provision an additional cluster with the new {es} version, reindex your data, and send index requests to both clusters temporarily. . Verify that the new cluster performs as expected, fix any problems, and then @@ -276,7 +119,7 @@ permanently swap in the new cluster. only for the time that the new cluster runs in parallel with your old cluster. Usage is billed on an hourly basis. -To learn more about the upgrade process on Elastic Cloud, see +To learn more about the upgrade process on Elastic Cloud, see {cloud}/ec-upgrade-cluster.html[Upgrade versions]. NOTE: Elastic Cloud only supports upgrades to released versions. Preview