Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Cleans up monitoring configuration steps #35111

Merged
merged 13 commits into from
Nov 8, 2018
64 changes: 33 additions & 31 deletions docs/reference/monitoring/configuring-metricbeat.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,15 @@ To learn about monitoring in general, see
{stack-ov}/xpack-monitoring.html[Monitoring the {stack}].

. Enable the collection of monitoring data. Set
`xpack.monitoring.collection.enabled` to `true` on the production cluster. +
`xpack.monitoring.collection.enabled` to `true` on each node in the production
cluster. By default, it is is disabled (`false`).
+
--
NOTE: You can specify this setting in either the `elasticsearch.yml` on each
node or across the cluster as a dynamic cluster setting. If {es}
{security-features} are enabled, you must have `monitor` cluster privileges to
view the cluster settings and `manage` cluster privileges to change them.

For example, you can use the following APIs to review and change this setting:

[source,js]
Expand All @@ -29,16 +35,21 @@ PUT _cluster/settings
}
}
----------------------------------
// CONSOLE
// CONSOLE

For more information, see <<monitoring-settings>> and <<cluster-update-settings>>.
--

. Disable the default collection of {es} monitoring metrics. Set
`xpack.monitoring.elasticsearch.collection.enabled` to `false` on the production
cluster. +
`xpack.monitoring.elasticsearch.collection.enabled` to `false` on each node in
the production cluster.
+
--
NOTE: You can specify this setting in either the `elasticsearch.yml` on each
node or across the cluster as a dynamic cluster setting. If {es}
{security-features} are enabled, you must have `monitor` cluster privileges to
view the cluster settings and `manage` cluster privileges to change them.

For example, you can use the following API to change this setting:

[source,js]
Expand Down Expand Up @@ -94,30 +105,26 @@ You must specify the following settings in the `modules.d/elasticsearch.yml` fil
- shard
period: 10s
hosts: ["http://localhost:9200"] <1>
xpack.enabled: true
xpack.enabled: true <2>
----------------------------------
<1> This setting identifies the host and port number that are used to access {es}.
<1> This setting identifies the host and port number that are used to access {es}.
<2> This setting ensures that {kib} can read this monitoring data successfully.
That is to say, it's stored in the same location and format as monitoring data
that is sent by <<es-monitoring-exporters,exporters>>.
--

.. If {security} is enabled, you must also provide a user ID and password so that
{metricbeat} can collect metrics successfully.
.. If Elastic {security-features} are enabled, you must also provide a user ID
and password so that {metricbeat} can collect metrics successfully.

... Create or identify a user that you want to use to collect the metrics.
+
--
TIP: There is a `remote_monitoring_user` built-in user that grants the privileges
necessary for {metricbeat} to monitor {stack} products. See
{stack-ov}/built-in-users.html[Built-in users].

Alternatively, you can choose a different user and give them the
`remote_monitoring_collector` {stack-ov}/built-in-roles.html[built-in role].
--
... Create a user on the production cluster that has the
{stack-ov}/built-in-roles.html[`remote_monitoring_collector` built-in role].
Alternatively, use the {stack-ov}/built-in-users.html[`remote_monitoring_user` built-in user].

... Add the `username` and `password` settings to the {es} module configuration
file.
+
--
For example, add the following settings in the `modules.d/kibana.yml` file:
For example, add the following settings in the `modules.d/elasticsearch.yml` file:

[source,yaml]
----------------------------------
Expand Down Expand Up @@ -156,19 +163,14 @@ For more information about these configuration options, see
{metricbeat-ref}/elasticsearch-output.html[Configure the {es} output].
--

.. If {security} is enabled on the monitoring cluster, you must provide a valid
user ID and password so that {metricbeat} can send metrics successfully.
.. If {es} {security-features} are enabled on the monitoring cluster, you
must provide a valid user ID and password so that {metricbeat} can send metrics
successfully.

... Create or identify a user that you want to use to send the metrics.
+
--
TIP: There is a `remote_monitoring_user` built-in user that grants the privileges
necessary for {metricbeat} to monitor {stack} products. See
{stack-ov}/built-in-users.html[Built-in users].

Alternatively, you can choose a different user and give them the
`remote_monitoring_agent` {stack-ov}/built-in-roles.html[built-in role].
--
... Create a user on the monitoring cluster that has the
{stack-ov}/built-in-roles.html[`remote_monitoring_agent` built-in role].
Alternatively, use the
{stack-ov}/built-in-users.html[`remote_monitoring_user` built-in user].

... Add the `username` and `password` settings to the {es} output information in
the {metricbeat} configuration file (`metricbeat.yml`):
Expand Down
181 changes: 114 additions & 67 deletions docs/reference/monitoring/configuring-monitoring.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
<titleabbrev>Configuring monitoring</titleabbrev>
++++

If you enable the collection of monitoring data in your cluster, you can
optionally collect metrics about {es}. By default, {monitoring} is enabled but
If you enable the Elastic {monitor-features} in your cluster, you can
optionally collect metrics about {es}. By default, monitoring is enabled but
data collection is disabled.

The following method involves sending the metrics to the monitoring cluster by
Expand All @@ -20,15 +20,51 @@ monitoring indices. You can also adjust how monitoring data is displayed.
To learn about monitoring in general, see
{stack-ov}/xpack-monitoring.html[Monitoring the {stack}].

. To collect monitoring data about your {es} cluster:
. Configure your cluster to collect monitoring data:

.. Verify that the `xpack.monitoring.enabled`,
`xpack.monitoring.collection.enabled`, and
`xpack.monitoring.elasticsearch.collection.enabled` settings are `true` on each
node in the cluster. By default `xpack.monitoring.collection.enabled` is disabled
(`false`), and that overrides `xpack.monitoring.elasticsearch.collection.enabled`,
which defaults to being enabled (`true`). Both settings can be set dynamically
at runtime. For more information, see <<monitoring-settings>>.
.. Verify that the `xpack.monitoring.enabled` setting is `true`, which is its
default value, on each node in the cluster. For more information, see
<<monitoring-settings>>.

.. Verify that the `xpack.monitoring.elasticsearch.collection.enabled` setting
is `true`, which is its default value, on each node in the cluster.
+
--
NOTE: You can specify this setting in either the `elasticsearch.yml` on each
node or across the cluster as a dynamic cluster setting. If {es}
{security-features} are enabled, you must have `monitor` cluster privileges to
view the cluster settings and `manage` cluster privileges to change them.

For more information, see <<monitoring-settings>> and <<cluster-update-settings>>.
--

.. Set the `xpack.monitoring.collection.enabled` setting to `true` on each
node in the cluster. By default, it is is disabled (`false`).
+
--
NOTE: You can specify this setting in either the `elasticsearch.yml` on each
node or across the cluster as a dynamic cluster setting. If {es}
{security-features} are enabled, you must have `monitor` cluster privileges to
view the cluster settings and `manage` cluster privileges to change them.

For example, use the following APIs to review and change this setting:

[source,js]
----------------------------------
GET _cluster/settings

PUT _cluster/settings
{
"persistent": {
"xpack.monitoring.collection.enabled": true
}
}
----------------------------------
// CONSOLE

For more
information, see <<monitoring-settings>> and <<cluster-update-settings>>.
--

.. Optional: Specify which indices you want to monitor.
+
Expand All @@ -53,107 +89,118 @@ patterns. For example, to include all indices that start with `test` except
the `xpack.monitoring.collection.interval` setting 10 seconds. See
<<monitoring-settings>>.

. Optional: Configure your cluster to route monitoring data from sources such
as {kib}, Beats, and Logstash to a monitoring cluster:

.. Verify that `xpack.monitoring.collection.enabled` settings are `true` on each
node in the cluster.

.. {stack-ov}/xpack-monitoring.html[Configure {monitoring} across the Elastic Stack].

. Identify where to store monitoring data.
+
--
By default, {monitoring} uses a `local` exporter that indexes monitoring data
on the same cluster. See <<es-monitoring-default-exporter>> and <<local-exporter>>.
By default, the data is stored on the same cluster by using a
<<local-exporter,`local` exporter>>.

Alternatively, you can use an `http` exporter to send data to a separate
monitoring cluster. See <<http-exporter>>.
Alternatively, you can use an <<http-exporter,`http` exporter>> to send data to
a separate _monitoring cluster_.

For more information about typical monitoring architectures,
see {stack-ov}/how-monitoring-works.html[How Monitoring Works].
--

. If {security} is enabled and you are using an `http` exporter to send data to
a dedicated monitoring cluster:
. If you choose to use an `http` exporter:

.. Create a user on the monitoring cluster that has the
{stack-ov}/built-in-roles.html#built-in-roles-remote-monitoring-agent[`remote_monitoring_agent` built-in role].
For example, the following request creates a `remote_monitor` user that has the
`remote_monitoring_agent` role:
.. On the cluster that you want to monitor (often called the _production cluster_),
configure each node to send metrics to your monitoring cluster. Configure an
HTTP exporter in the `xpack.monitoring.exporters` settings in the
`elasticsearch.yml` file. For example:
+
--
[source, sh]
---------------------------------------------------------------
POST /_xpack/security/user/remote_monitor
{
"password" : "changeme",
"roles" : [ "remote_monitoring_agent"],
"full_name" : "Internal Agent For Remote Monitoring"
}
---------------------------------------------------------------
// CONSOLE
// TEST[skip:needs-gold+-license]
[source,yaml]
--------------------------------------------------
xpack.monitoring.exporters:
id1:
type: http
host: ["http://es-mon-1:9200", "http://es-mon2:9200"]
--------------------------------------------------
--

.. On each node in the cluster that is being monitored, configure the `http`
exporter to use the appropriate credentials when data is shipped to the
monitoring cluster.
.. If the Elastic {security-features} are enabled on the monitoring cluster, you
must provide appropriate credentials when data is shipped to the monitoring cluster:

... Create a user on the monitoring cluster that has the
{stack-ov}/built-in-roles.html[`remote_monitoring_agent` built-in role].
Alternatively, use the
{stack-ov}/built-in-users.html[`remote_monitoring_user` built-in user].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome change!


... Add the user ID and password settings to the HTTP exporter settings in the
`elasticsearch.yml` file on each node. +
+
--
If SSL/TLS is enabled on the monitoring cluster, you must use the HTTPS protocol
in the `host` setting. You must also include the CA certificate in each node's
trusted certificates in order to verify the identities of the nodes in the
monitoring cluster.
For example:

The following example specifies the location of the PEM encoded certificate with
the `certificate_authorities` setting:
[source,yaml]
--------------------------------------------------
xpack.monitoring.exporters:
id1:
type: http
host: ["http://es-mon-1:9200", "http://es-mon2:9200"]
auth.username: remote_monitoring_user
auth.password: YOUR_PASSWORD
--------------------------------------------------
--

.. If you configured the monitoring cluster to use
<<configuring-tls,encrypted communications>>, you must use the HTTPS protocol in
the `host` setting. You must also specify the trusted CA certificates that will
be used to verify the identity of the nodes in the monitoring cluster.

*** To add a CA certificate to an {es} node's trusted certificates, you can
specify the location of the PEM encoded certificate with the
`certificate_authorities` setting. For example:
+
--
[source,yaml]
--------------------------------------------------
xpack.monitoring.exporters:
id1:
type: http
host: ["https://es-mon1:9200", "https://es-mon2:9200"]
auth:
username: remote_monitor <1>
password: changeme
username: remote_monitoring_user
password: YOUR_PASSWORD
ssl:
certificate_authorities: [ "/path/to/ca.crt" ]
id2:
type: local
--------------------------------------------------
<1> The `username` and `password` parameters provide the user credentials.

Alternatively, you can configure trusted certificates using a truststore
(a Java Keystore file that contains the certificates):
--

*** Alternatively, you can configure trusted certificates using a truststore
(a Java Keystore file that contains the certificates). For example:
+
--
[source,yaml]
--------------------------------------------------
xpack.monitoring.exporters:
id1:
type: http
host: ["https://es-mon1:9200", "https://es-mon2:9200"]
auth:
username: remote_monitor
password: changeme
username: remote_monitoring_user
password: YOUR_PASSWORD
ssl:
truststore.path: /path/to/file
truststore.password: password
id2:
type: local
--------------------------------------------------
--

. If {security} is enabled and you want to visualize monitoring data in {kib},
you must create users that have access to the {kib} indices and permission to
read from the monitoring indices.
. Configure your cluster to route monitoring data from sources such as {kib},
Beats, and {ls} to the monitoring cluster. The
`xpack.monitoring.collection.enabled` setting must be `true` on each node in the
cluster. For information about configuring each product to collect and send
monitoring data, see {stack-ov}/xpack-monitoring.html[Monitoring the {stack}].

. If you updated settings in the `elasticsearch.yml` files on your production
cluster, restart {es}. See <<stopping-elasticsearch>> and <<starting-elasticsearch>>.
+
--
You set up {monitoring} UI users on the cluster where the monitoring data is
stored, that is to say the monitoring cluster. To grant all of the necessary permissions, assign users the
`monitoring_user` and `kibana_user` roles. For more information, see
{stack-ov}/mapping-roles.html[Mapping users and groups to roles].
TIP: You may want to temporarily {ref}/modules-cluster.html[disable shard
allocation] before you restart your nodes to avoid unnecessary shard
reallocation during the install process.

--

. Optional:
Expand Down
Loading