Skip to content

Commit

Permalink
7.0 upgrade (#242)
Browse files Browse the repository at this point in the history 
* [DOCS] First pass at 7.0 upgrade update.

* [DOCS] Fixed formatting issues & flagged the Upgrade Assistant as xpack.

* Update docs/en/install-upgrade/upgrading-stack.asciidoc

Co-Authored-By: debadair <debadair@elastic.co>

* [DOCS] Incorporated feedback from lcawl.

* [DOCS] Fixes broken link

* [DOCS] Removed tip about temporary superuser when upgrading the security index.
  • Loading branch information
@debadair
debadair committed Mar 26, 2019
1 parent 8938dda commit e55f02f
Showing 1 changed file with 69 additions and 226 deletions.
295 changes: 69 additions & 226 deletions docs/en/install-upgrade/upgrading-stack.asciidoc
View file
Original file line number Diff line number Diff line change
@@ -1,272 +1,115 @@
[[upgrading-elastic-stack]]
== Upgrading the Elastic Stack

When upgrading to a new version of Elasticsearch, you need to upgrade
each of the products in your Elastic Stack. Beats and Logstash 5.6 are
compatible with Elasticsearch {version} to give you flexibility in scheduling the
upgrade.
When upgrading to a new version of {es}, you need to upgrade
each of the products in your Elastic Stack. Beats and Logstash 6.7 are
compatible with {es} {version} to give you flexibility in scheduling
the upgrade.

****
The steps you need to take to upgrade differ depending on which products you
are using. Want a list that's tailored to your stack? Try out our
{upgrade_guide}[Interactive Upgrade Guide].
****
{es} supports rolling upgrades between minor versions, from {es} 5.6 to 6.7,
and from 6.7 to {version}.

If you are running a pre-6.0 version, we recommend upgrading to the most
recent 5.6 before upgrading to {version}. {xpack} 5.6
provides a free Upgrade Assistant that identifies issues you need to address
before upgrading and simplifies migrating indices that need to be reindexed
before you upgrade. The Upgrade Assistant is enabled with both Trial and
Basic licenses. You can install {xpack} solely for the purpose of upgrading.

Rolling upgrades are supported when upgrading from Elasticsearch 5.6 and
Elasticsearch 6.0-6.2 to {version}. Upgrading from any
version prior to 5.6 requires a full cluster restart.

IMPORTANT: 5.x and older indices are not compatible with {version}. You must
remove or reindex them on your 6.x cluster before upgrading to {version}. The internal
Kibana and {xpack} indices and the default Beats and Logstash mapping templates
also need to be updated to work with {version}.
IMPORTANT: 5.x indices are not compatible with {version}. You must
remove or reindex them to upgrade to {version}. The default Beats and
Logstash mapping templates also need to be updated to work with {version}.

=== Preparing to upgrade

Before upgrading the Elastic Stack to {version}:

. Back up your data. You **cannot roll back** to an earlier version unless
you have a backup of your data. For information about creating snapshots, see
{ref}/modules-snapshots.html[Snapshot and Restore].

. Check the Elasticsearch {ref}/settings.html[deprecation log] to see if
you're using any deprecated features and update your code accordingly.
By default, deprecation log messages are enabled at the `WARN` level.

. Review the breaking changes for each product you use
and make the necessary changes so your code is compatible with {version}. See
<<elastic-stack-breaking-changes>>.
. Check the {es} {ref}/logging.html#deprecation-logging[deprecation log]
to see if you're using any deprecated features and update your code accordingly.
By default, deprecation warnings are logged when the log level is set to `WARN`.

. {ref}/docs-reindex.html[Reindex] or delete any indices created on 2.n. We recommend
upgrading to the most recent 5.6 and using the {xpack} Reindex Helper to reindex 2.n indices.
. Review the <<elastic-stack-breaking-changes>> and upgrade your code to work
with {version}.

. If Kibana and {xpack} are part of your stack, upgrade the internal Kibana
and {xpack} indices. We recommend using the {xpack} 5.6 Reindex Helper to
upgrade the internal indices. If you're performing a full cluster restart upgrade
from an earlier version, you can also
<<upgrade-internal-indices,use the `_migration/upgrade` API>> directly to
upgrade the internal indices after you install Elasticsearch {version}.

. If you use {stack} {security-features} to secure your cluster:
.. Make sure TLS is enabled to encrypt communications between nodes. TLS must
be enabled to upgrade to {version}. For more information, see
{stack-ov}/encrypting-communications.html[Encrypting communications].
. Upgrade to 6.7 and use the {kibana-ref}/upgrade-assistant.html[{kib} Upgrade Assistant] to {ref}/docs-reindex.html[reindex]
any indices that are not compatible with {version}.
+
NOTE: Enabling TLS requires a full cluster restart. Nodes that have TLS
enabled cannot communicate with nodes that do not have TLS enabled. You must
restart all nodes to maintain communication across the cluster.

.. Make sure real passwords are configured for the built-in `elasticsearch`,
`kibana`, and `logstash_system` users. They cannot use the 5.n default
password (`changeme`). For more information, see
{stack-ov}/built-in-users.html[Built-in users].

. Consider closing {ml} jobs before you start the upgrade process. It is not
required, but there are pros and cons to leaving the jobs running. These
considerations are described in the steps related to
{ref}/setup-upgrade.html[upgrading {es}].
[role="xpack"]
.Upgrade Assistant
******
The Upgrade Assistant and migration APIs are enabled with both the Basic and
Trial licenses. You can install the default distribution of 6.7 to use the
Upgrade Assistant to prepare to upgrade even if you are upgrading to the OSS
distribution of {version}.
******

IMPORTANT: Test upgrades in a dev environment before upgrading your
production cluster.
. Use the Upgrade Assistant to identify any changes you need to make to your
cluster configuration.

[[upgrade-order-elastic-stack]]
=== Upgrade order

Upgrade the Elastic Stack products you use in the following order:
=== Upgrade process

. Elasticsearch Hadoop: {hadoop-ref}/install.html[install instructions]
. Elasticsearch: {ref}/setup-upgrade.html[upgrade instructions]
. Kibana: {kibana-ref}/upgrade.html[upgrade instructions]
. Logstash: {logstash-ref}/upgrading-logstash.html[upgrade instructions]
. Beats: {beats-ref}/upgrading.html[upgrade instructions]
. APM Server: {apm-server-ref-70}/upgrading.html[upgrade instructions]

NOTE: Logstash 5.6 and 6.n and Beats 5.6 and 6.n are compatible with all 6.n versions of
Elasticsearch. This provides flexibility in when you schedule the upgrades
for your Logstash instances and Beats agents. We recommend upgrading Logstash
and Beats as soon as possible to take advantage of performance improvements
and other enhancements.

=== Upgrading to 6.3
Starting in 6.3, the default distributions of {es}, {ls}, and {kib}
include {xpack} and a free Basic license that never expires.

You can perform rolling upgrades to 6.3 from OSS-only clusters running 5.6
or 6.0-6.2. Basic features are operational once the cluster is fully
upgraded. If you are already using {xpack}, your settings are preserved when
you upgrade.

IMPORTANT: If you use {xpack} you must explicitly remove the old {xpack} plugin
before restarting: `bin/elasticsearch-plugin remove x-pack`. The node will fail
to start if the {xpack} plugin is present.

You must explicitly enable data collection after the upgrade to use monitoring.
Set `xpack.monitoring.collection.enabled` to `true` with the `_cluster/settings`
API:

[source,json]
----------------------------------------------------------
PUT /_cluster/settings
{
"persistent" : {
"xpack.monitoring.collection.enabled" : "true"
}
}
----------------------------------------------------------
// CONSOLE

To take more of the {stack} features for a spin, you can start a 30-day trial
from Kibana or with the Start Trial API:

[source,json]
----------------------------------------------------------
POST _license/start_trial
----------------------------------------------------------
// CONSOLE

The 30-day trial enables you to try out the full set of platinum features,
including security, machine learning, alerting, graph capabilities, and more.

[role="xpack"]
[[xpack-stack-upgrade]]
=== Upgrading from 5.6
When you've made the necessary changes and are ready to upgrade from 6.7 to
{version}:

{xpack} 5.6 provides migration and upgrade APIs for Elasticsearch and a
Upgrade Assistant UI for Kibana. These tools are included with the trial
license and the free basic license.

To upgrade to {version} from 5.6:
. Test the upgrade in a dev environment *before* upgrading your
production cluster.

. {ref}/setup-upgrade.html[Upgrade Elasticsearch] to the most recent 5.6 and
install {xpack} on all nodes in your cluster. If you are upgrading from an
earlier 5.x release, you can perform a rolling upgrade. To upgrade from older
versions you must perform a full cluster restart.
+
If your trial license expires,
https://register.elastic.co/[register for a free Basic license]. To apply the
license, upload the license file with the `license` API:
+
[source,json]
----------------------------------------------------------
license -d @license.json
----------------------------------------------------------
. Back up your data. You **cannot roll back** to an earlier version unless
you have a snapshot of your data. For information about creating snapshots, see
{ref}/modules-snapshots.html[Snapshot and Restore].

. If {xpack} **IS NOT** normally a part of your {stack}, disable the
{es} {security-features} in `elasticsearch.yml`:
+
[source,yaml]
----------------------------------------------------------
xpack.security.enabled: false
----------------------------------------------------------
. Consider closing {ml} jobs before you start the upgrade process. While {ml}
jobs can continue to run during a rolling upgrade, it increases the overhead
on the cluster during the upgrade process. For more information, see
{ref}/rolling-upgrades.html[Rolling upgrades].

. Upgrade {kib} to the most recent 5.6 and install {xpack}.
. Upgrade the components of your Elastic Stack in the following order:

. If you disabled the {es} {security-features}, also disable the {kib}
{security-features} in `kibana.yml`:
+
[source,yaml]
----------------------------------------------------------
xpack.security.enabled: false
----------------------------------------------------------
.. {es} Hadoop: {hadoop-ref}/install.html[install instructions]
.. {es}: {ref}/setup-upgrade.html[upgrade instructions]
.. Kibana: {kibana-ref}/upgrade.html[upgrade instructions]
.. Logstash: {logstash-ref}/upgrading-logstash.html[upgrade instructions]
.. Beats: {beats-ref}/upgrading.html[upgrade instructions]
.. APM Server: {apm-server-ref}/upgrading.html[upgrade instructions]

. Use the Upgrade Assistant in Kibana to
view incompatibilities that you need to fix, identify any 2.x indices that
need to be migrated or deleted, and upgrade the internal indices to the
{major-version} index format.
+
You can also call the Elasticsearch migration APIs directly:
+
`/_xpack/migration/deprecations`:: Retrieves information about cluster-, node-,
and index-level settings that use deprecated features.
+
`/_xpack/migration/assistance`:: deprecated[6.7.0] Returns a list of indices
that need to be reindexed before you can upgrade to {version}.
+
`/_xpack/migration/upgrade`:: deprecated[6.7.0] Upgrades the indices for the
{watcher} and {security-features} to a single-type format compatible with
Elasticsearch 6.x.

. Once you've resolved all of the migration issues, perform
a {ref}/rolling-upgrades.html[rolling upgrade] from Elasticsearch 5.6 to {version}.
NOTE: Logstash 6.7 and Beats 6.7 are compatible with all 7.x versions of
{es}. This provides flexibility in when you schedule the upgrades
for your Logstash instances and Beats agents, but we recommend upgrading as
soon as possible to take advantage of performance improvements
and other enhancements.

[[oss-stack-upgrade]]
=== Upgrading from a pre-5.6 installation

It is possible to upgrade directly to {major-version} from a pre-5.6 installation,
but it requires a {ref}/restart-upgrade.html[full cluster restart] and you must
manually reindex any 2.x indices you need to carry forward to {major-version}.

IMPORTANT: If you use Kibana or {xpack}, you also need to upgrade the
internal Kibana and {xpack} indices. For information about upgrading them
after you install Elasticsearch {version}, see
<<upgrade-internal-indices, Upgrading internal indices>>.

To manually reindex a 2.x index:

. Create an index with 6.x compatible mappings.
. Use the {ref}/docs-reindex.html[reindex API] to copy documents from the
2.x index into the new index. You can use a script to perform any necessary
modifications to the document data and metadata during reindexing.
. Use the {ref}/indices-aliases.html[_aliases] API to add the name of the 2.x
index as alias for the new index and delete the 2.x index.

[[upgrade-internal-indices]]
==== Upgrading internal indices for {major-version}

The format used for the internal indices used by Kibana and {xpack} has
changed in {major-version}. Before you can run Kibana and {xpack} in {version},
these indices must be upgraded to the new format. If you are upgrading from a
version prior to 5.6, you must upgrade them after after installing
Elasticsearch {version}.

To get a list of the indices that need to be upgraded, use the
{ref}/migration-api-deprecation.html[deprecation info API]:

[source,json]
----------------------------------------------------------
GET /_xpack/migration/deprecations
----------------------------------------------------------
// CONSOLE

To upgrade the `.security` index:
=== Upgrading from 6.6 or earlier

. On a single node, add a temporary superuser account to the `file` realm.
. Use the {kib} Upgrade Assistant to upgrade the security index, submitting the
request with the credentials for the temporary superuser. Alternatively, you can
{ref}/reindex-upgrade.html[reindex manually].
To upgrade directly to {es} {version} from versions 6.0-6.6, you must
{ref}/reindex-upgrade.html[manually reindex] any 5.x indices you need to
carry forward, and perform a {ref}/restart-upgrade.html[full cluster restart].
This includes any internal indices created in 5.x, such as the `.kibana` and
`.security*` indices.

. Delete the temporary superuser account from the file realm.
Make sure all 5.x indices have been deleted before upgrading to {version}. {es}
{version} will fail to start if any 5.x indices are present.

You can use your regular administration credentials to upgrade the other
internal indices.
If you are running a version prior to 6.0,
https://www.elastic.co/guide/en/elastic-stack/6.7/upgrading-elastic-stack.html[upgrade to 6.7]
and reindex your old indices or bring up a new {version} cluster and
{ref}/reindex-upgrade-remote.html[reindex from remote].

TIP: Once you upgrade the `.kibana` index, you can run Kibana and use the
{xpack} Reindex Helper UI to upgrade the other indices.
The recommended path is to upgrade to 6.7 before upgrading to {version}. This
makes it easier to identify the changes you need to make to upgrade and enables
you to perform a rolling upgrade with no downtime.

[[upgrade-elastic-stack-for-elastic-cloud]]
=== Upgrading on Elastic Cloud

A single click in the Elastic Cloud console can upgrade a cluster to a newer
version, add more processing capacity, change plugins, and enable or disable
high availability, all at the same time. During the upgrade process,
Elasticsearch, Kibana, {xpack} and the officially included plugins are
{es}, Kibana, {xpack} and the officially included plugins are
upgraded simultaneously.

Although upgrading your Elastic Cloud clusters is easy, you still need to
address breaking changes that affect your application. Minor version upgrades,
upgrades from 5.6 to {major-version}, and all other cluster configuration
upgrades from 6.7 to {version}, and all other cluster configuration
changes can be performed with no downtime.

To avoid downtime when a full cluster restart is required:

. Provision an additional cluster with the new Elasticsearch version, reindex
. Provision an additional cluster with the new {es} version, reindex
your data, and send index requests to both clusters temporarily.

. Verify that the new cluster performs as expected, fix any problems, and then
Expand All @@ -276,7 +119,7 @@ permanently swap in the new cluster.
only for the time that the new cluster runs in parallel with your old cluster.
Usage is billed on an hourly basis.

To learn more about the upgrade process on Elastic Cloud, see
To learn more about the upgrade process on Elastic Cloud, see
{cloud}/ec-upgrade-cluster.html[Upgrade versions].

NOTE: Elastic Cloud only supports upgrades to released versions. Preview
Expand Down

0 comments on commit e55f02f

Please sign in to comment.